Add controller.max_send_buffer_size in MiB, default 64 and 0 for no
limit. Close with 1013 instead of 1008 on overflow: clients treat 1008
as fatal, so a host would shut down and the web UI would log out. With
1013 the client resumes, the controller answers with invalidate since
the session has ended, and the client registers a new session.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every message sent over a link is kept in the send buffer until the
other side acknowledges it with a heartbeat, so it can be resent if
the session resumes. Nothing forced a peer to acknowledge, so a client
that kept heartbeating without a sequence number while subscribed to a
busy event made the controller buffer messages until it ran out of
memory.
Store buffered messages serialised, track their total size, and close
the connection once it goes over maxSendBufferSize. The controller
sets this to 64 MiB for the connections it accepts. A client that far
behind could not resume in a reasonable time anyway.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Not every plugin arrives through PluginInstallRequest. Plugins checked
out into external_plugins or installed by hand on the machine never
flag a restart, so the web UI shows nothing and tooling that acts on
the flag never fires.
HostMarkRestartRequiredRequest and ControllerMarkRestartRequiredRequest
set config.restartRequired, guarded by the existing restart
permissions. They are exposed as "Mark Restart Required" in the control
dropdown of the host and controller pages, hidden once the flag is set,
and as mark-restart-required ctl commands. The mark shows up on the
next system metrics poll.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A permission function runs during ingress permission checking, before
the request body is validated. UserBulkImportRequest.permission read the
raw, unvalidated message.data and threw a bare Error for an unknown
importType. That error is not a PermissionError, so _processMessage
rethrew it to the connector "message" listener, which emitted "error" on
the connector. Controller-side connectors have no "error" listener, so
Node rethrew and handleUnhandledErrors exited the process.
Repro: a control client with client-side validateSent disabled sends
UserBulkImportRequest with importType "foo"; the client gets a normal
RequestError response and the controller then logs a fatal uncaught
exception and exits.
Fix both layers: _processMessage now logs and drops an unexpected error
from the permission check (control connections only, where a response
was already sent) instead of rethrowing, and UserBulkImportRequest
throws a PermissionError for an unknown importType so it is denied
cleanly. A genuine PermissionError still denies silently as before.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Migrate the usage of CommonJS specific code and imports to ESM. Roughly
summarised this:
- Changes all require() calls to import statement or expressions.
- Changes usage of module.exports to export statements.
- Replaces require.resolve with import.meta.resolve.
- Replaces conditional imports with conditional subpath imports.
- Replaces module === require.main with import.meta.main.
- Adds the full filename with extention to all relative imports.
- Changes all usage of __dirname and __filename to their import.meta
equivalents.
- Sets "type": "module" in package.json files.
- Removes usage of "use strict".
- Adds .ts an an alias of .js for the webpack build.
Due to differences in how require and import resolves files the plugin
entrypoints will have to include the filename and extension after this
migration in order to work.
Updating a single plugin can leave the installation with multiple
versions of @clusterio/lib. Passing the core package and every npm
installed plugin to a single npm update call lets npm resolve them
together.
UpdateAllRequest targets the controller or a host, requires both the
core update permission and core.plugin.update, and is gated by both
allow_remote_updates and allow_plugin_updates. The web UI Updates modal
gets an Update All action and ctl gets --all on the update commands.
Closes#910
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Generic error handlers such as FactorioServer.handle() and the event
handler catch in Link only log err.stack, so the ajv errors attached to
an InvalidMessage were lost unless the catch site checked for the class
explicitly. Appending the errors to the stack in the constructor makes
every stack-logging handler show which field failed validation without
special casing each one.
Fixes#986
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The create instance modal can now copy the active save of the instance
whose config is copied, optionally saving the running game first, and
start the new instance once the copy is done. The new instance is
assigned to the host of the source so the save can be transferred.
Saving a running game is a new InstanceSaveGameRequest handled by the
host through /server-save, which waits for the save to finish. It is
also exposed as clusterioctl instance save-game.
Closes#405
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
An event broadcast from an instance to all hosts only reached the host
running that instance. The host side of an instance connection has the
host's own address as its source, so a broadcast aimed at hosts counts as
addressed to it, and a link that a message is addressed to handles it and
stops. The event was delivered locally and never routed onwards.
A broadcast is addressed to every link of the type it targets, so arriving
at one of them does not mean it has arrived everywhere. Keep routing such a
message after handling it locally. Only links that route are affected, as
instances and controls have no router and would otherwise throw on every
broadcast they receive.
The only links whose source type matches a broadcast target and which route
are the host and its instance connections, both matching a broadcast to
hosts, so add the missing case for that to the host router. It forwards to
the controller, which already passes broadcasts on to every other host, and
stops when the message came from the controller to begin with.
Drop the exclusion this bug had in the routing test matrix, which now
covers it along with the plugin filtering variants.
Fixes#575
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
forwardRequest records a request so that its response can be sent back the
way it came, but recorded it before handing the message to the connector.
When the connector threw, because the session to the next hop was gone, the
record stayed behind for a request that had never left.
Routers answer the origin themselves when forwarding throws, so the request
was already accounted for. The leftover record then had it answered a
second time when the link cleared its pending requests on disconnect,
producing a duplicate response and "Received response N without a pending
request" at the far end.
Remove the record when the forward throws. Doing it here rather than in the
routers covers both the host, which catches and responds, and the
controller router, which does not.
Fixes#650
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An instance's factorio.version can now target a release channel such as
"stable" or "experimental" (any key from factorio.com/api/latest-releases)
in addition to "latest" and specific X.Y / X.Y.Z versions.
The controller proxies the latest-releases API through a ValueCache and
persists the last successful response to latest-releases.json, serving that
copy when the API is unreachable. On instance start the host resolves a
channel to the concrete version it points to and feeds it into the existing
checkForUpdates/findVersion path, so a channel auto-installs and runs like a
pinned version. The version selector in the web UI lists the available
channels alongside "latest".
Also clarifies the host log/error when a newer version cannot be downloaded
automatically because the host uses a direct (single-version) Factorio
install rather than a versioned directory.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The migration from phin to fetch in #755 was hampred by the lack of an
option to set the CA when using fetch. This meant places that used phin
to talk to the controller with the tlsCa option could not be migrated to
use fetch.
Remove the tlsCa option to unblock this dependency. The replacement for
this option is to set the NODE_EXTRA_CA_CERTS environment variable to a
path to the certificate file instead, which works with fetch.
* Refactor UserManager to use Datastore
* Rename lib.User to UserDetails also adds to/fromUserDetails
* Fix existing tests
* Add user class tests
* Add tests for finaliseUsers
* Move HostDetails to own file with tests
* Move InstanceDetails to its own file with tests
* Rename HostInfo and add tests
* Rename InstanceInfo and add tests
- Also changes the names of the host properties related to instances
- Also removes the need to specify the location of InstanceInfo
* Make json round trip test validate schema
* Add InstanceManager
* Fix lint errors
* Extra tests
* Add final test coverage
* Add one last test to InstanceManager
* Response to comments
* Use generator rather than return anonumous generator
Co-authored-by: Hornwitser <github@hornwitser.no>
* Fix lint issue with generator
---------
Co-authored-by: Hornwitser <github@hornwitser.no>
* Add ModDependency and version types
* Add warnings for mod dependencies
* Add warning for wrong factorio version
* Add fix issues button
* Add automatic dependency resolution and download
* Add abilty to check for updates
* Add extra test coverage
* One extra test
* Fix copy paste error
* Event types for controller
* Clean up ctl event emitters
* Event types for server
* Event types for lib
* Remove TypedEventEmitter
* Event types for webui
* Fix compile issues
* Small cleanup of connectors
Add config entries to controller and host for the Factorio username and
token to use when talking to the Factorio APIs. If set an username and
token is not set in factorio.settings then these settings are used,
making it easy to configure one set of credentials for a cluster/host.
For running more public provide your own hardware clusters, there's an
option to disable controller sharing the credential with the hosts so
that each host have to set their own credentials.
If a link is closed while the prepareDisconnect action is in progress
the link will still try to signal the disconnect ready over the socket
when the prepareDisconnect action completes, which throws due to the
socket no longer being present. This error is not caught due to the
catch happening before the finally on the promise and triggers an
unhandled rejection.
Fix by rearranging the finally to come before the catch on the promise
so that errors are always handled, and checking that the link still has
a session before attempting to send the disconnect ready signal.
When messages were forwarded they were sent using the low level
connector .send method which accepts a message and sends it over the
link. This method sends the message as is with no modification which
means that at no point during the forwarding logic was the message
sequence number that originated from Link A renumbered to the correct
sequence number to send out on Link B which it was forwarded on.
This in turn caused the logic which drops acked messages from the send
buffer to break which in turn meant the resume could cause duplication
of message and/or messages to be lost. With enough duplicated messages
the resume of one connection could cause other connections to drop and
also initiate resume.
Fix by renumbering the sequence on forwarded messages. In the future
the sequences sent and received should also be checked to make sure this
kind of problem does not happen again, this is not done here because
partially updating a running cluster with sequence problems to one that
checks and enforces sequences numbers will be problematic.
When a plugin broadcasts an event to instances there may be hosts and
instances in the cluster which does not have the plugin loaded. This
would cause an error to be logged for every instance which does not have
the plugin loaded.
Filter out broadcast events sent to hosts and instances such that
instances and hosts that does not have the event do not received them.
When a request is forwarded from a virtual link to a WebSocket link as
is the case when an instance sends a request to the controller the
origin for the request is a virtual link. Handle this case in
_clearPendingRequests instead of throwing an assertion error.
Track the number of requests currently awaiting a response from a
request handler both from interally issued requests and requests
forwarded from other parties.
Add suffixes like Ms and Ns to variables whos time unit is not in
seconds. This is aligning with a new code style of putting units
on variables that are not in SI base units.
Implement remotely initiated stopping and restarting of the controller
and hosts. Restarting is implemented by exiting with a non-zero exit
code and relying on a process manager to start the procress again. To
prevent accidentally trying to restart on a system that doesn't have a
process manager present the --can-restart command line option needs to
be passed to enable the restart operation.
