InstanceDetails now carries factorio.mod_pack_id as modPackId, so the
instance list and pages can show it without fetching each config. The
web UI resolves an unset value to the controller default and tags it
"Default". The name links to the mod pack page.
Closes#852
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The templates table shows name, inheritor count and a create instance
action instead of the instance status and start/stop columns.
factorio.rcon_password joins the ports as never inherited.
Part of #200
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review changes for #1142:
- base becomes template everywhere (is_template, template_id,
--template, --is-template, --templates, InstanceDetails.isTemplate).
- Object fields inherit per property. inherited_fields lists the
overridden properties for object fields and the template object is
merged with them.
- Linking a template also inherits fields that hold the default value.
The server name stays per instance.
- Ports are never inherited.
- InstanceDetails carries templateId.
Web UI: inherited values are locked until unlocked, overridden values
can be inherited again, templates are listed separately, the create
modal can create templates and instances from templates, and the
instance page has "Create template" which can link the source.
Part of #200
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Instances can be marked as a base with instance.is_base and other
instances inherit config values from it through instance.base_id.
The controller copies base values into inheriting configs and keeps
track of inherited fields in the hidden instance.inherited_fields, so
hosts receive resolved configs and need no changes. Setting a field
stops inheriting it, `instance config inherit` undoes that.
Bases cannot be assigned to a host, are hidden from `instance list`
unless --bases is given, and cannot be deleted or unmarked while
other instances inherit from them. Chains of bases are allowed and
cycles are rejected.
Part of #200
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"safe" suggested the data being written survives a crash, it only
guarantees the old file is either kept or fully replaced. The new name
says what it does. No alias, plugin API breaks are allowed in minor
versions and the 2.1 migration guide lists the rename.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Cooldude2606 <25043174+Cooldude2606@users.noreply.github.com>
Revoking a host now unassigns its instances instead of being refused
while it has any, and removes its system metrics.
Approving and revoking no longer force a save. Hosts are saved by
autosave and on exit like everything else, which also removes the
approving state from pairing requests.
The approval code and link are logged as warnings so they show up in
the systemd service, which only prints warnings and above. The pairing
limits are now the controller.host_pairing_timeout and
controller.host_pairing_max_pending config values, 0 removes the limit.
Subscribers take an optional permitted function, used for the pairing
requests instead of overrides in a subclass.
- Saves of a datastore run one at a time, and changes made while saving
are saved by the next save. Overlapping saves from approving and
revoking hosts could write an older snapshot after a newer one.
- Map subscribers fetch everything again when a new session starts, as
the controller only sends values that exist. A host revoked while a
browser was disconnected stayed in its list.
- The host token check accepts the same pre-rename tokens as the
WebSocket, so upgrading a host with such a token does not make it ask
to join again.
Drops tests that restate message declarations, check that removed
commands are gone, test internal helpers or repeat the same behaviour
at another layer.
Hosts join the cluster by running the installer, which shows a code and
waits until the request is approved under Hosts > Add host in the web
interface or with clusterioctl host approve. An admin can pass their
user token to the installer to approve the host right away. The token
is only used for that and is not saved.
The controller only accepts hosts that have a record, and only
approving a host creates one. Revoking a host deletes its record, so
its token can no longer be used to reconnect. Hosts that have connected
before keep working.
Removes host generate-token, revoke-token and create-config from
clusterioctl, bootstrap generate-host-token, the host token messages and
the core.host.generate_token, revoke_token and create_config
permissions, which are also stripped from stored roles. Adds
clusterioctl host pending, approve, reject and revoke, and the
core.host.approve and core.host.revoke permissions.
Standalone installs add the host to the controller database directly,
since the controller is not running during the install.
loaded, enabled, pendingInstall, removed, noEntrypoint and nameTaken
allowed far more combinations than can happen, and every consumer had
to know which took priority. PluginDetails now has one state, with
loaded, loadsOnRestart and removed as getters over it.
2.0.3 web UIs and ctl send PluginListRequest straight to hosts and
require loaded and enabled, so those are still sent next to the
optional state. Plugins listed by 2.0.3 hosts get their state from
loaded and enabled.
The web UI planner gets the same treatment: MachinePlugin has one
state in place of disabled, noEntrypoint, dependencyTaken and
nameTaken. ctl host plugin list shows the state.
Closes#1134
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Host not connected, host shutting down and instance not running are
now UnavailableError (503) instead of ConflictError. The request error
migration notes move to a new 2.1 guide, since this ships in 2.1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Error responses now carry the registered name of the thrown error's
class, and the requester rejects with an instance of that class instead
of a generic RequestError. Error classes are registered like message
classes, and plugins list their own in `errors`.
RequestError is replaced by a hierarchy following the HTTP status codes:
ClientError (ValidationError, ForbiddenError, NotFoundError,
TimeoutError, ConflictError) and ServerError (InternalServerError).
PermissionError is PermissionDeniedError and the config errors
InvalidValue, InvalidField and InvalidAccess gain an Error suffix.
Link logs only errors whose class is not registered. These arrive on the
other side as InternalServerError with the original message and stack,
so an error forwarded through the controller is logged once, where it
was thrown.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Controller.bootstrap and Host.bootstrap returned constructor arguments
that the caller had to spread into new Controller/Host, and
DatastoreProvider.bootstrap did the same for datastores. They are
replaced by Controller.fromConfig, Host.fromConfig and
KeyValueDatastore/SubscribableDatastore.fromProvider, which load the
data and return the finished instance.
Closes#1126
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The name it returns may already be taken by a concurrently running process
by the time the caller uses it. Callers now try the name and retry with a
-2, -3, ... suffix on EEXIST instead.
Save transfers previously renamed with fs.rename, which silently overwrites
a save that appears at the target, and copied without COPYFILE_EXCL. Moves
now use link + unlink like save rename does, and copies pass COPYFILE_EXCL,
so both fail with EEXIST rather than overwrite.
Supersedes #880.
Co-authored-by: Hornwitser <code@hornwitser.no>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Factorio 2.1.18 removed PollutionSettings::max_pollution_to_restore_trees,
but readPollution still read it for every version, so every map setting
after it was read 9 bytes late: enemy_attack_pollution_consumption_modifier
came out as garbage, the enemy_evolution factors shifted by one field and
enemy_expansion.enabled and both expansion distances came out null.
Read the field only for strings older than 2.1.18. Add modified_v2_1_17
and modified_v2_1_18 fixtures generated with headless 2.1.17 and 2.1.18
from the modified 2.1 settings (base game only), unit tests asserting
their full decoded output, and use the 2.1.18 string for the
load-scenario integration test on 2.1.18 and later.
Fixes#1131
Co-authored-by: ColumbusLabs <287001685+ColumbusLabs@users.noreply.github.com>
Co-authored-by: Cooldude2606 <25043174+Cooldude2606@users.noreply.github.com>
The send buffer limit in controller.max_send_buffer_size is set in MiB, but
buffered messages were measured with String.length, which counts UTF-16 code
units. Messages with non-ASCII text could go over the configured size in bytes
before the session was ended.
Measure each serialised message with Buffer.byteLength when it is buffered,
keep that size with the entry, and subtract the same size when it is
acknowledged. The web UI build already provides Buffer through the buffer
polyfill.
Fixes#1108
Co-authored-by: ColumbusLabs <287001685+ColumbusLabs@users.noreply.github.com>
Browsers have no setImmediate. The web bundle only had one because jszip
depended on the setimmediate package, which installs it as a global.
Since jszip was replaced in #1011 every component using a subscription
threw on unmount, breaking page changes in the web interface.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Replace jszip with yauzl and yazl
jszip loads a whole archive into memory to read it and buffers the
output when writing. Saves and mods can be several hundred megabytes,
so read zips with yauzl, which only reads the central directory and
streams entries on demand, and write them with yazl, which streams the
output.
lib gains a ZipArchive wrapper around yauzl with the lookup helpers the
callers need, findRoot() now takes it instead of a JSZip. Saves are
patched by streaming every kept entry into a new zip, mod info.json is
read straight from the file, exports are streamed to the controller and
Factorio zip downloads go through a temp file instead of a buffer.
Patching now inflates and deflates each entry again where jszip reused
the compressed data, on a 336 MB synthetic save that is 7 s instead of
0.7 s, with peak memory going from 416 MB to 151 MB.
Closes#1002
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Copy save entries without recompressing them
yazl inflates and deflates every entry it is given, so patching a save
took ten times longer than with jszip, which reused the compressed
data. Add a ZipWriter to lib that streams entries out as they are
added and copies entries from a ZipArchive byte for byte, with ZIP64
structures when sizes or offsets need them, and use it in patch().
On a 336 MB synthetic save patching now takes 0.4 s at 143 MB peak RSS
against 0.75 s at 417 MB with jszip. Factorio loads the result.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Handle save write errors before the patch loop awaits them
A write error on the temporary save, such as a full disk, rejected the
pipeline promise while addEntry() or addBuffer() was still pending. Nothing
was listening yet, so the host's unhandledRejection handler exited the
process instead of failing the instance start.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Keep the export zip cache local to each export
The cache was module level, so when two instances on one host exported at
once, the first to finish closed archives the other was still reading.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Start writing zips before adding files from disk
yazl reads files passed to addFile() as soon as they are added. An
unreadable file emitted an error while the directory traversal was still
running, before the output pipeline existed, and the process exited on the
uncaught error. build_mod and zipDirectory now start the pipeline first and
reject instead. build_mod also removes the partial zip on failure.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Add opt-in UDP messaging between the host and the Factorio server using
the --enable-lua-udp option of Factorio 2.0.59+. When
factorio.enable_lua_udp is set the instance binds a localhost UDP socket,
starts Factorio with --enable-lua-udp on factorio.lua_udp_port (random
if null, like the RCON port) and tells the clusterio module the host port
via clusterio_private.update_instance.
Data from Factorio is sent with clusterio_api.send_udp(channel, data) in
the same IPC format as send_json, so plugins receive it through the
existing ipc-<channel> events. Packets not sent from the Lua UDP port
are ignored, as that is the only unauthenticated way into the IPC
handler. Data to Factorio is sent with the new sendUdp method on
instances and instance plugins, and is dispatched in-game as
on_udp_packet_received events by polling helpers.recv_udp every tick
from the clusterio module.
Plugins requiring UDP declare the new LuaUdp feature flag, which like
SavePatching and ScriptCommands refuses to load the plugin on instances
without the option enabled.
Verified against a headless 2.1.17 server: send_udp and recv_udp are
silent no-ops when the option is off, so a stale port in a save that is
moved to an instance without UDP does no harm. The integration test
matrix now starts the instance with UDP enabled on Factorio 2.0.59+ and
round trips a player event over it.
Closes#969
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The staging copy made by Config.set() went through the constructor, so
plugin migrations ran a second time on the live fields. A version-gated
migration was applied twice when setPluginVersions() changed the version.
Staging now copies the fields with update() instead.
A config saved without <plugin>.version got the running version as its
default, so setPluginVersions() saw no change and nothing saved the stamp.
The constructor now writes the registered plugin versions after the
migrations run and marks the config dirty when that changes a loaded value.
Controller.bootstrap() saves the instances that came out dirty, and
setPluginVersions() is gone.
Restructure the SyncHook and plugin config tests by method, and suggest the
controller config version for role migrations in the docs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Plugins had no way to update the config fields and permissions core stores
for them, so renaming inventory_sync needed migrations in core packages.
A plugin can now declare a migrationEntrypoint. It's loaded on the
controller and hosts before configs and roles are read, for every plugin in
the plugin list, and attaches handlers to sync hooks for the controller,
host and instance configs and roles. lib gets helpers to rename config
fields and permissions, and each config gets a hidden <plugin>.version field
set after load so migrations can tell which version saved the data.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The controller logs the url and query of every HTTP request. Plugins
doing OAuth get the authorization code as a query parameter on their
callback route, so it ended up in the log. redactUrl now accepts a
path, code joins the redacted parameter names, and redactQuery covers
the parsed query in the log metadata.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Plugins installed from npm are listed by package name, but the plugin
loader joined that name onto the main and entrypoint paths and imported
the result as a file, which resolves relative to the working directory.
Every npm installed plugin then failed to load. Resolve these paths from
the directory of the plugin's package.json instead.
EventSubscriber registers with the control's onAccountUpdate when it
has one and resends its subscription through handleAccountEvent. A lost
permission makes the request fail, a granted one syncs it again.
The web Control also reports the account on connect, so resends are
queued and sent once per tick. A failed subscription request now sets
synced to false and notifies callbacks, before the error never reached
the UI.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The old name read as an unconditional unsubscribe. Log following has
its own subscription outside SubscriptionController, so check
core.log.follow on permission changes too. Move the per-connection
work into one helper and split the tests into userPermissionsUpdated
and rolesUpdated, checking the account update event is sent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add controller.max_send_buffer_size in MiB, default 64 and 0 for no
limit. Close with 1013 instead of 1008 on overflow: clients treat 1008
as fatal, so a host would shut down and the web UI would log out. With
1013 the client resumes, the controller answers with invalidate since
the session has ended, and the client registers a new session.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lowering the limit at runtime would leave connections holding more
filters than allowed, so read it once when the controller starts.
Move the limit tests into the subscribe and replace groups.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every message sent over a link is kept in the send buffer until the
other side acknowledges it with a heartbeat, so it can be resent if
the session resumes. Nothing forced a peer to acknowledge, so a client
that kept heartbeating without a sequence number while subscribed to a
busy event made the controller buffer messages until it ran out of
memory.
Store buffered messages serialised, track their total size, and close
the connection once it goes over maxSendBufferSize. The controller
sets this to 64 MiB for the connections it accepts. A client that far
behind could not resume in a reasonable time anyway.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The permission for a subscription was only checked when subscribing,
so a control connection kept receiving events after the user's roles
or role permissions changed to no longer allow it.
Add SubscriptionController.unsubscribeUnpermitted which rechecks every
subscription of a link and removes the ones the user is no longer
permitted, and call it for the affected control connections from
userPermissionsUpdated and rolesUpdated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A subscriber could keep sending subscribe requests with new filters and
the controller would add every one of them to the subscriber's filter
set without bound, growing memory until it ran out.
Add controller.max_subscription_filters (default 10000) and reject
subscribe and replace requests that would leave a subscriber holding
more filters than that for one event.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On Windows this test throws "EPERM: operation not permitted, rename
<path>" presumably because Windows prevents concurrent renames. This
does not affect the test outcome, so ignore the error(s).
These are used outside of the integration tests, but including the
integration index file causes the very slow setup of the contorller
and host. Avoid the integration setup costs when running commands
such as `pnpm mocha test/lib` by moving these utility functions to
the common test functions.
Without a transport Winston will print annoying "Attempt to write logs
with no transports" warnings all over the place. Setup a transport in
tests that cause logging to suppress this.
The behaviour of creating the config file if it's missing while
convenient causes confusion and problems when the tools are run
in the wrong folder. They end up creating a shadow installation
and misdirect the user with nonsensical error messages, instead
pointing to the real problem which is that the tool can't find
the configuration needed to actually run.
Require a config file to be present at startup and refuse to run if this
file is missing. To support the odd use-case of creating the config
when missing a --create-config flag is added. To conveniently generate
a config file to edit out from the "config create" command is added.
Ctl ended up calling its main class for "Control" for historical
reasons, along with its config file and plugin entrypoint.
The plugin entrypoint was renamed to ctl in 808eda0513. This
finishes that renaming by also renaming the main class, default config
file, config-control command, and main entrypoint.
The config-control command is renamed to config to be consistent with
the host and controller's config commands.
The term "control" is still kept in the config and message locations as
a common specifier for both ctl and web_ui.
A permission function runs during ingress permission checking, before
the request body is validated. UserBulkImportRequest.permission read the
raw, unvalidated message.data and threw a bare Error for an unknown
importType. That error is not a PermissionError, so _processMessage
rethrew it to the connector "message" listener, which emitted "error" on
the connector. Controller-side connectors have no "error" listener, so
Node rethrew and handleUnhandledErrors exited the process.
Repro: a control client with client-side validateSent disabled sends
UserBulkImportRequest with importType "foo"; the client gets a normal
RequestError response and the controller then logs a fatal uncaught
exception and exits.
Fix both layers: _processMessage now logs and drops an unexpected error
from the permission check (control connections only, where a response
was already sent) instead of rethrowing, and UserBulkImportRequest
throws a PermissionError for an unknown importType so it is denied
cleanly. A genuine PermissionError still denies silently as before.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Two writes to the same file at the same time shared one temporary name, so the second write clobbered the first's temporary and the first rename left the second call with nothing to rename, failing with ENOENT. The host does exactly that when it saves an instance config while the instance saves it too.
The temporary now carries a random tag and is created exclusively, so concurrent writers never touch each other's file. Since the temporary is always new, the mode is applied on creation and the chmod for reused leftovers goes away. A failed rename removes its temporary instead of leaving it for the next write to overwrite.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Bastiaan