* Replace jszip with yauzl and yazl
jszip loads a whole archive into memory to read it and buffers the
output when writing. Saves and mods can be several hundred megabytes,
so read zips with yauzl, which only reads the central directory and
streams entries on demand, and write them with yazl, which streams the
output.
lib gains a ZipArchive wrapper around yauzl with the lookup helpers the
callers need, findRoot() now takes it instead of a JSZip. Saves are
patched by streaming every kept entry into a new zip, mod info.json is
read straight from the file, exports are streamed to the controller and
Factorio zip downloads go through a temp file instead of a buffer.
Patching now inflates and deflates each entry again where jszip reused
the compressed data, on a 336 MB synthetic save that is 7 s instead of
0.7 s, with peak memory going from 416 MB to 151 MB.
Closes#1002
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Copy save entries without recompressing them
yazl inflates and deflates every entry it is given, so patching a save
took ten times longer than with jszip, which reused the compressed
data. Add a ZipWriter to lib that streams entries out as they are
added and copies entries from a ZipArchive byte for byte, with ZIP64
structures when sizes or offsets need them, and use it in patch().
On a 336 MB synthetic save patching now takes 0.4 s at 143 MB peak RSS
against 0.75 s at 417 MB with jszip. Factorio loads the result.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Handle save write errors before the patch loop awaits them
A write error on the temporary save, such as a full disk, rejected the
pipeline promise while addEntry() or addBuffer() was still pending. Nothing
was listening yet, so the host's unhandledRejection handler exited the
process instead of failing the instance start.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Keep the export zip cache local to each export
The cache was module level, so when two instances on one host exported at
once, the first to finish closed archives the other was still reading.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Start writing zips before adding files from disk
yazl reads files passed to addFile() as soon as they are added. An
unreadable file emitted an error while the directory traversal was still
running, before the output pipeline existed, and the process exited on the
uncaught error. build_mod and zipDirectory now start the pipeline first and
reject instead. build_mod also removes the partial zip on failure.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
place_items no longer creates script inventories for what does not fit. Each serialized inventory is split at the size of its destination: the slots that fit are restored slot for slot and the rest become serialized stacks, which are inserted into the free space of the same inventory and then main, guns, ammo and armor. What is still left over is returned as a list and spill_items puts it into a corpse sized to hold it. This keeps 1.0 out of reach of game.create_inventory.
The armor-first fill order is gone, as the bonus slots from armor equipment only apply on the next tick anyway (#1106). Filters are no longer stripped for destinations, they are the same inventory type the data came from.
deserialize_inventory concatenated the filter table in its error print on 2.0, which errored out the whole restore when an inventory did not support filters.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Download each Factorio version only once at a time
Instances starting together each downloaded the new version into the same
.tmp file and extracted into the same version directory, corrupting the
install. Keep the downloads in progress in a map by target directory and
let later callers wait on the existing download.
Fixes#1105
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RhMv1AxB8EZdyZoSJhyNmY
* Wait for a download in progress whose version looks installed
The changelog is extracted partway through the archive, so a server
checking during the extraction saw the version as installed and could
start the half-extracted install. Treat a version with a download in
progress as not installed, so the server waits on that download.
Hold the mocked download open in the concurrency tests and wait for the
second server to reach the waiting path instead of relying on a delay.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RhMv1AxB8EZdyZoSJhyNmY
---------
Co-authored-by: ILLISIS <jayson.harris01@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
serialize_item_stack did not store the spoil percent and
deserialize_item_stack did not pass it to set_stack, so spoilable items
arrived fresh after a transfer. Store it when above 0 and restore it. The
percent is used instead of spoil_tick since tick counts differ between servers.
Co-authored-by: ILLISIS <jayson.harris01@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The plugin syncs more than inventories now (settings, appearance, recipe
notifications), so name it for what it does. The package becomes
@clusterio/plugin-player_sync and every inventory_sync identifier follows.
A migration entrypoint renames the inventory_sync config fields, including
load_plugin, and the inventory view permission. The plugin moves
inventories.json in the database directory to player_sync.json and
storage.inventory_sync in saves to storage.player_sync itself, and closes
dialogs left open under the old GUI names on startup.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The staging copy made by Config.set() went through the constructor, so
plugin migrations ran a second time on the live fields. A version-gated
migration was applied twice when setPluginVersions() changed the version.
Staging now copies the fields with update() instead.
A config saved without <plugin>.version got the running version as its
default, so setPluginVersions() saw no change and nothing saved the stamp.
The constructor now writes the registered plugin versions after the
migrations run and marks the config dirty when that changes a loaded value.
Controller.bootstrap() saves the instances that came out dirty, and
setPluginVersions() is gone.
Restructure the SyncHook and plugin config tests by method, and suggest the
controller config version for role migrations in the docs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Plugins had no way to update the config fields and permissions core stores
for them, so renaming inventory_sync needed migrations in core packages.
A plugin can now declare a migrationEntrypoint. It's loaded on the
controller and hosts before configs and roles are read, for every plugin in
the plugin list, and attaches handlers to sync hooks for the controller,
host and instance configs and roles. lib gets helpers to rename config
fields and permissions, and each config gets a hidden <plugin>.version field
set after load so migrations can tell which version saved the data.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Fix running Clusterio from published packages
Since the move to ES modules and pnpm 12, packages installed from npm
fail to start:
- The package imports of @clusterio/lib point Node at TypeScript
sources, which Node refuses to strip types from inside node_modules.
- The controller, host and ctl import their package.json from next to
the compiled entrypoint, a copy pnpm leaves out of the package.
- ctl, host and web_ui have no .npmignore, so pnpm packs them using the
repository .gitignore and leaves out nearly all of dist/.
Resolve the package imports to the compiled files at runtime while
keeping the sources for type checking, import package.json by package
name so it resolves to the real file, and add .npmignore files so pnpm
packs the same files as npm.
* Load plugins installed from npm by package name
Plugins installed from npm are listed by package name, but the plugin
loader joined that name onto the main and entrypoint paths and imported
the result as a file, which resolves relative to the working directory.
Every npm installed plugin then failed to load. Resolve these paths from
the directory of the plugin's package.json instead.
HostRouter.wakeInstance() logged err.stack, which RequestError leaves
undefined, so a failed start was logged as "Error starting instance:
undefined" and the reason, such as no game port left in the range, was
lost.
Claude-Session: https://claude.ai/code/session_01F7obMEi4L4PzUQ8w8q9Uyz
Co-authored-by: ILLISIS <jayson.harris01@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Since the move to ES modules and pnpm 12, packages installed from npm
fail to start:
- The package imports of @clusterio/lib point Node at TypeScript
sources, which Node refuses to strip types from inside node_modules.
- The controller, host and ctl import their package.json from next to
the compiled entrypoint, a copy pnpm leaves out of the package.
- ctl, host and web_ui have no .npmignore, so pnpm packs them using the
repository .gitignore and leaves out nearly all of dist/.
Resolve the package imports to the compiled files at runtime while
keeping the sources for type checking, import package.json by package
name so it resolves to the real file, and add .npmignore files so pnpm
packs the same files as npm.
Covers the icon_size default of 64 in the export icon loaders and the
host recreating saves and script-output when an instance is assigned.
Both tests fail when the original fix is reverted.
Closes#710
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A handler that only updates gauges has nothing to yield. collect() already
dropped undefined results at runtime, now its type says so too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Add a request to mark hosts and the controller as needing a restart
Not every plugin arrives through PluginInstallRequest. Plugins checked
out into external_plugins or installed by hand on the machine never
flag a restart, so the web UI shows nothing and tooling that acts on
the flag never fires.
HostMarkRestartRequiredRequest and ControllerMarkRestartRequiredRequest
set config.restartRequired, guarded by the existing restart
permissions. They are exposed as "Mark Restart Required" in the control
dropdown of the host and controller pages, hidden once the flag is set,
and as mark-restart-required ctl commands. The mark shows up on the
next system metrics poll.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Cover the mark-restart-required ctl commands
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Not every plugin arrives through PluginInstallRequest. Plugins checked
out into external_plugins or installed by hand on the machine never
flag a restart, so the web UI shows nothing and tooling that acts on
the flag never fires.
HostMarkRestartRequiredRequest and ControllerMarkRestartRequiredRequest
set config.restartRequired, guarded by the existing restart
permissions. They are exposed as "Mark Restart Required" in the control
dropdown of the host and controller pages, hidden once the flag is set,
and as mark-restart-required ctl commands. The mark shows up on the
next system metrics poll.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The behaviour of creating the config file if it's missing while
convenient causes confusion and problems when the tools are run
in the wrong folder. They end up creating a shadow installation
and misdirect the user with nonsensical error messages, instead
pointing to the real problem which is that the tool can't find
the configuration needed to actually run.
Require a config file to be present at startup and refuse to run if this
file is missing. To support the odd use-case of creating the config
when missing a --create-config flag is added. To conveniently generate
a config file to edit out from the "config create" command is added.
_waitForRconReadyOrExit resolved with an Error to signal the process
exited before RCON came up, which is an odd shape for a promise. It now
resolves when RCON is ready and rejects otherwise, with the error emitted
with the exit, or "RCON connection lost" when there was none.
Observable behaviour is unchanged. sendRcon rejects with the same
messages as before and stop ignores the rejection, as stopping is done
once the process is gone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Waiting for exit alongside rcon-ready made sendRcon reject with the flat
"RCON connection lost" for every case, including the unexpected exits
that used to reject with the EnvironmentError saying why the server went
away, as events.once rejects on error events.
Have the paired wait capture the error emitted with the exit and hand it
back, so sendRcon throws that when there is one and falls through to the
old message when there is not, which is the kill path where no error is
emitted. stop() ignores the error as its job is done once the process is
gone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The fieldChanged handler repeated the hook invoke in each branch, and
factorio.shutdown_timeout had no invoke at all, so plugins were never
told about a change to it. Do the field specific side effect, then
invoke the hook once for every field.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_watchServerLogActions dereferenced two regex results with ! and
threw on null. The name regex /^([^ ]+)/ returns null when a line
classified as an action starts with a space, and the BAN reason
regex /\. Reason: (.+)\.$/ returns null for a [BAN] line with no
". Reason: X." tail. The output listener runs synchronously off the
stdout LineSplitter, so the throw becomes an uncaughtException and
takes the whole host down with every instance on it.
Repro: instance with enable_save_patching off and enable_script_commands
on; a control client sends /sc print("2000-01-01 00:00:00 [BAN] evil").
The raw print reaches stdout, parses as a dated BAN action, and the
reason deref throws.
Skip the line when the name regex misses and default the ban reason
to empty when the tail is absent. Apply the same name guard to
_watchPlayerPromote. The autosave ![1] in server.ts is safe because
its filter regex already guarantees the capture.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
sendRcon waits for the rcon-ready event when RCON has not connected yet,
which is the same root cause as the stop() hang. events.once rejects on
error events, but _watchExit skips emitting error while the state is
stopping, and kill() sets that state before sending SIGKILL. exportData
does exactly that, so an RCON call racing the export never settles and
the request stays pending until the session closes.
Wait for exit as well, which leaves the existing RCON client check to
reject the call with "RCON connection lost" instead. Both call sites now
share a helper as the paired wait is identical.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The config changed handler ran updateFactorioSettings and
updateFactorioWhitelist as soon as the field changed. While the instance
is starting, stopping, creating_save or exporting_data the server is not
ready for RCON, so the update threw and was only logged. The three list
update events pushed from the controller had the same problem, except
Link._processEvent swallowed the rejection and the update was lost
without a trace.
Add _waitForSettledStatus, which resolves once notifyStatus lands on
running or stopped. notifyStatus is the only place _status changes and
notifyExit always ends on stopped, so a waiter cannot hang. The settings,
whitelist and list updates now go through it.
Nothing is applied when the instance settles on stopped, because there is
no server to apply it to. factorio.settings is written to
server-settings.json and the lists to the list files on the next start,
and enableWhitelist is read from the config when the next Instance is
constructed.
The plugin hook now runs immediately for the two config fields instead of
being chained onto the update. Every other field already invokes it right
away, and waiting for the instance to settle would hold the notification
back for the whole startup.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
stop() waits for the rcon-ready event when RCON has not connected yet,
but nothing emits it once the process is gone, so the promise never
settles and the stop request stays pending until the session closes.
Wait for exit as well and drop the other listener when either fires.
Repro: send InstanceExportDataRequest to a stopped instance, then
InstanceStopRequest while the status is exporting_data. Instance.stop()
sees the server as running and calls server.stop(), then exportData
kills the process before RCON is up. The export finishes and the
instance reports stopped, but the stop request never gets a response.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The fieldChanged handler in the Instance constructor ran
updateFactorioSettings and updateFactorioWhitelist with only a finally
attached. A rejection from either went unhandled, and the host exits on
unhandled rejections, taking every instance on it down.
Two ways to hit it against a real host:
- Start an instance and immediately toggle factorio.enable_whitelist.
updateFactorioWhitelist calls sendRcon while the server state is still
"new" and it throws "Expected state running,stopping but state is new".
- Set factorio.settings tags to [1] on a running instance. applyAsConfig
called tag.replace outside its try block and threw
"tag.replace is not a function".
Catch and log both on the instance logger before invoking the hook, and
convert tags with String() so non-string entries no longer throw.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
yargs, @types/yargs, semver and @types/semver are used by several packages at the same version. Also point the remaining direct references to @swc/core and react-router-dom at their existing catalog entries.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Raise every caret range to the current latest within its major, in the
root package.json, the pnpm-workspace.yaml catalog, the per-package
package.json files and the create plugin template.
Majors taken: eslint 10, @eslint/js 10, globals 17, nyc 18,
@types/express 5, @types/node 24, finalhandler 2, pidusage 4,
style-loader 4, css-loader 7, yargs 18 (all packages), chalk 6.
Majors skipped: mocha 12 (integration suite fails under it, 59 tests,
starting with a 30 s timeout in clusteriocontroller run; kept at
^11.8.0), inquirer (10+ is a rewrite on @inquirer/prompts where
ui.Prompt.prototype.close no longer exists, so the win32 readline
workaround in create.js would have to go and cannot be verified here),
react/antd/react-router-dom/@types/react, typescript 7,
@sinclair/typebox, webpack-merge (exact pin), webpack-dev-middleware,
webpack-manifest-plugin, winston-daily-rotate-file.
Code adaptations: the two /api/stream/:id handlers in routes.ts are
typed Request<{ id: string }> because @types/express 5 types
req.params values as string | string[]; eslint.config.mjs adds
newIsCapExceptions: ["constructor"] because eslint 10.10 new-cap now
reports `new this.constructor()`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Resolves the ESM conflicts in patch.ts, Instance.ts and host.ts. The
module gathering in Instance now goes through patch.loadModules with
the instance's loadedPlugins, and imports use explicit extensions.
Review changes: the existing-output check only treats ENOENT as
"does not exist" and rethrows anything else, a --zip option keeps the
output as a zip file for transfer (extracting stays the default), the
--scenario option describes that it takes a path and what the default
is, and the docs note that IPC is not available outside Clusterio.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Rethrow errors other than ENOENT and ENOTDIR when reading the package
spec as a directory, mark the tests that run npm as slow so FAST_TEST
skips them, and use ./external_plugins/my_plugin in the README examples.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The mod pack's Factorio version ends up in mod-settings.dat. Factorio
refuses to read that file when the version is newer than the game and
falls back to default settings, which can wreck the save. Check the
versions in syncMods before touching the mods folder.
Closes#1006
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Replace invokeHook and the per plugin on* methods with AsyncHook
collections on the controller, host, instance, clusterioctl and web
control objects. Plugin entrypoints export a default function that
receives a load context and attaches handlers to the hooks it needs.
Class exports still load through fromContext with a deprecation
warning. The create templates and docs use the entrypoint function.
This is clusterio/clusterio#921 recreated on top of master after the
ESM migration and the PluginDeclaration permissions change.
Closes#730
Co-Authored-By: Cooldude2606 <25043174+Cooldude2606@users.noreply.github.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Adds a permissions array to PluginDeclaration, a type-level Permissions
registry that plugins extend with module augmentation, and
registerPluginPermissions() called from all entrypoints. checkPermission
and the web UI hasPermission helpers now take PermissionName so typos are
compile errors. Core definitions move into a corePermissions array that
the interface derives its keys from.
Closes#987
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Cooldude2606 <25043174+Cooldude2606@users.noreply.github.com>
Migrate the usage of CommonJS specific code and imports to ESM. Roughly
summarised this:
- Changes all require() calls to import statement or expressions.
- Changes usage of module.exports to export statements.
- Replaces require.resolve with import.meta.resolve.
- Replaces conditional imports with conditional subpath imports.
- Replaces module === require.main with import.meta.main.
- Adds the full filename with extention to all relative imports.
- Changes all usage of __dirname and __filename to their import.meta
equivalents.
- Sets "type": "module" in package.json files.
- Removes usage of "use strict".
- Adds .ts an an alias of .js for the webpack build.
Due to differences in how require and import resolves files the plugin
entrypoints will have to include the filename and extension after this
migration in order to work.
The ESM import of yargs does not support the singleton way of using
yargs, instead we have to call the yargs function to get a parser that
can be augmented with the options.
The .argv property has also been deprecated. Replace it with a call to
.parse() instead.
Cooldude2606