Add suffixes like Ms and Ns to variables whos time unit is not in
seconds. This is aligning with a new code style of putting units
on variables that are not in SI base units.
The logical order of clusterio's components are in most places
controller, host, instance, control. It is confusing that Address
breaks this convention and I do not remember I wrote it that way.
Reorder and renumber the host and instance members of Address to
be consistent with most other places.
The entrypoint of plugins were changed in 50f12c7 and b0f6c95 but the
installer's loading of plugin info to get the correct plugin name was
not updated for these changes. Update installer plugin loading to use
the correct entrypoint.
The place for storing mods was changed from sharedMods to mods when mod
management was implemented. Rename the old sharedMods folder to mods
when the migration script is invoked.
Change the expected exported name of plugin declarations to be "plugin"
at the main entrypoint instead of the default export. This makes the
plugin's code organisation more flexible especially when using CommonJS
JavaScript modules as they can export additional symbols from the
plugin declaration file.
Implement remotely initiated stopping and restarting of the controller
and hosts. Restarting is implemented by exiting with a non-zero exit
code and relying on a process manager to start the procress again. To
prevent accidentally trying to restart on a system that doesn't have a
process manager present the --can-restart command line option needs to
be passed to enable the restart operation.
The create.js script in development mode used to use hard coded paths to
the other cluster commands (ctl, controller etc.). This means that in
dev mode a cluster could only be created from the root directory of the
project.
By using relative paths one can also create a cluster in a subdirectory.
For example by using the following:
mkdir test-cluster
cd test-cluster
node ../packages/create --dev
Record the time a player was first seen joining an instance in
PlayerStats. For existing stats this value is set to the last
joined time when it is first loaded.
Add custom input component support to the Web UI config in order to
provide a user friendly dropdown selector for the Mod Pack that is
configured as default on the controller and overridden instances.
Provide the kernel, machine, CPU Model, hostname, and Node.js version
fields to SystemInfo and display them in the Web UI. These were added
as metrics exposed to prometheus in b46efeb.
There are more pieces of information about a system than just its CPU,
Memory and disk metrics. To avoid having to build a separate collection
and subscription for metrics and system info, rename the existing system
metrics to system info in anticipation of providing additonal static
system information such as CPU model in it.
If the controller is behind a proxy then the socket.remoteAddress will
be that of the proxy instead of the end client that connected. In this
case the X-Forwarded-For header will contain the address of the end
client, but this should only be relied upon if the party connecting is a
trusted proxy, otherwise a malicious user could spoof the IP address
that is logged by providing a forged X-Forwarded-For header.
Implement a trusted proxy list consisting of a configured list of IP
addresses and/or CIDR blocks and use the last address[1] in the
X-Forwarded-For header if the remoteAddress address of the connecting
party is in the trusted proxy list.
[1]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For
Split out host specific data from RegisterHostData that is not related
to the establishment of a connection to the controller into it's own
HostInfoUpdate event. This allows this data to be updated when the
config values they are taken from are changed.
The agent string was inspired by the HTTP User-Agent header to serve as
a way to identify what's connecting to the cluster. While there may
have been vision of other software connecting to a controller than the
Clusterio provided ones this seems highly unlikely to ever happen.
Remove redundant agent string for now.
Memory sizes are usually written using the prefixes with a base of 1024
instead of 1000. Implment formatting of the power of two based prefixes
for displaying memory sizes.
Implement simple collection of basic real time system metrics of hosts
and the controller for the purpose of displaying them in the Web UI. The
metrics are collected by the controller at a configurable interval and
pushed out to web clients currently subscribed to the metrics.
The Prometheus library has an optimisation for metrics that expose
multiple time series such as Summary that provide two values in the
time series consisting of the metric name suffixed with _sum and _count.
These are provided as a series of sufixes with corresponding samples such
that a serialised summary metric named some_value would have a samples
mapping looking something like this:
[
["_sum", ["label=\"value\"", 3]],
["_count", ["lavel=\"value\"", 2]],
]
If multiple hosts expose the same metric then these need to be merged,
which is the job of mergeSamples, which merges the samples from each
sufix into the destination.
One corner case that may occur is if the source and destination metric
that's being merged have different lists of suffixed samples. In this
case the suffixes that are present in the source but not in the
destination needs to be added verbatim to the destination. The code for
handling this case incorrectly added these samples to the source metric
instead of the destination metric.
This corner case only occrs if a metric's type differs between two
hosts connected to the same controller. For example if a Gauge metric
is turned into a Summary metric.
When ws was updated to version 8 it came with a breaking change where
message data was no longer utf8 decoded from binary to a string. The
main codepath handling messages was updated to handle this but the ws
dump path was not resulting in the controller crashing when receiving
a message that is then forwarded to the dumper without decoding.
Fix by decoding the message in the dump-ws path.
The render method for columns in Antd Table take the three arguments
value, record, index. If dataIndex is set then value is the property
given by dataIndex in record, otherwise value and record is the same.
Unfortunately only record has the proper type while value is the type
any. Change usage of `render: value =>` to `render: (_, value) =>` to
get proper typing for the render methods defined on columns.
Use a separate ES Module based build for the web version of the lib
package. This fixes the rebuild on watched files changing in lib
failing with a TS1479 error.
Add host.factorio_port_range config to hosts that specify a pool of
ports that gets automatically assigned to instances which do not have a
game_port set on them. This allows setting up a port range that's
forwarded to the host on the router and then configure that port range
on the host and have instances automatically get assigned an available
port when they start up.
Resolves: #448
Due to configs needing a location to be deserialised it's not possible
to cleanly replace RawInstanceInfo with the InstanceInfo class. Rename
it to HostInstanceUpdate to better represent its purpose.
Refactor away the config group concept and simplyfy the way config
fields are defined. Fields are defined directly as properties of the
static field fieldDefinitions on the class itself with a separate
interface to declare the types of fields.
This makes it possible to easily make accessing the config fields type
checked in a way that lets plugins extend the interfaces with their own
config options by adding additional fields to the interface via
declation merging.
Remove circular dependency between inventory_sync/message.ts and
inventory_sync/instance.ts by moving the IpcPlayerData type from
message.ts to instance.ts. This solves a problem with the web
build checking types in instance.ts that it dosen't use.
Refactor the Config and ConfigGroup classes to match the JSON data class
pattern with toJSON, toRemote and fromJSON methods replacing the load,
init, and serialize methods. Since fromJSON requires extra information
the messages structures passes around the plain JSON representation of
the configs instead of serialising instances of the config classes
automatically.
If a new save is added to the saves list of an instance the current save
selected to be loaded by default changes its state without updating the
mtime of the file on disk, which in turn means basing the updatedAt
property on mtime will not work with caching. Fix by having a separate
updatedAt property that is set by the controller whenever the save
details changes from the controller's perspective.
Noticed-By: @Cooldude2606
Avoid possible use of an unintended empty mod pack by not setting the
activeModPack on the Instance until it's loaded in syncMods. This was a
leftover from when a newly created empty mod pack was supposed to act as
the default when no mod pack was set.
Process updates of the resource EventSubscriber are subscribed to and
have it keep a mapping of id to value syncronised with event provider as
long as there are subscription callbacks registered with it. The
mapping is kept around after the last callback is unsubscribed so that
it can be efficiently resynced with the resource based on the updatedAt
timestaps of individual elements.
This allows the EventSubscriber to provide the right building blocks to
provide its values to React components via the the useSyncExternalStore
hook. This in turn greatly simplifies the web_ui models for the
subscribed resources.
Reduce complexity in the implementation of EventSubscriber by requiring
a control link be passed to its constructor. This is possible after the
change in 8c724e3 made the web interface control instance available to
plugins at init time.
Since subscriptions are handlede per link there's no need for the
eventReplay to be returned as a part of the SubscriptionRequest, it can
be sent and processed normally as an event. Simplify the logic by
sending eventReplay as an event and remove SubscriptionResponse.
Remove the reliance a reference to the Controller to get the link that a
subscription is tied to by explicitly handling the SubscriptionRequest
and passing the link to the SubscriptionController.
Notify the SubscriptionController that a connection closed from the
Connection instead of having the subscription controller set up an event
to do so. This simplifies the logic on the SubscriptionController's
side.
Entites that didn't have an updatedAt time at the time of creation will
default to having one set to 0. Since entities with an updatedAt time
greater than lastResponseTime is expected to be returned this means that
these entities would not be included when a client does its initial sync
subscribing to a resource. Change the default lastResponseTime
requested so that these entities are included.
Skip calling the subscription function that should provide an event
replay if the client is unsubscribing from the event. This avoids
having to check if the client is subscribing in every subscription
function.
Keep a copy of the instance saves details on the controller so that
saves can be queried even when the host is disconnected and the
controller can properly track and provide updates on the saves to
its clients.
Refactor HostUpdatesEvent, InstanceDetailsUpdatesEvent, ModUpdatesEvent,
ModPackUpdatesEvent and UserUpdatesEvent to have multiple updates of the
entities they support in each event. This is intended to make the events
capable of syncing the entire list that they represent.
Download and cache needed mods to run instance mod packs on instance
startup. To avoid opening too many download streams at once only one
mod is downloaded at a time.
Structure the storage of mods on the controller into into a reusable
component in preparation for the host using the same kind of mod storage
to retrieve and cache the mods it needs to run mod packs on instances.
Log errors messages to the log when setting up an instance fails in the
preparation phase, creating save phase, starting up phase, and so on.
Previously these were only sent back as a an error response to the
request that iniatied the wakeup of the instance.
Build the clusterio_lib mod from source and download and copy the
subspace_storage mod used for testing if it's not present in the mods
directory. This removes a manual step needed to run tests on a new
clone of the repository.
Add helper to serialise the calls made to an async function such that no
overlapping invocations happen to the async function by queue up calls
made while it's running to continue after it's running.
Replace PluginWebApi's requirePath member with npmPackage which is only
populated if the package is not marked as private and the requirePath
matches the name. This prevents the public /api/plugins endpoint from
leaking filesystem paths into the controller.
Use the JSON data class pattern for User, Role, and PlayerStats and move
them into lib/src/data. This means that User is purely data now and
can't reference role objects directly. To make it possible easily check
permissions attached to roles and store data only visible on the
controller on user objects a ControllerUser type is introduced which is
linked with the UserManager and can therefore reference role objects
directly.
Resolves#525.
Load hosts, instances, mod packs, mods and users before constructing the
Controller instances. This allows these resources to be correctly typed
as always available and prevents any possibility of save being initiated
on these resources before they are loaded.
Some updates for instances trigger instance details updates, while other
updates to instances are not visible on instance details. Rename
instanceUpdated to instanceDetailsUpdated to better represent what it
does and only set instancesDirty when an update is made to the
instances.
Separate out the serialisation of the invocations to saveData with an
AsyncSerialMergingCallback helper than has unit tests covering its
functionality.
Exit the Space Exploration remote view if its active to prevent the
empty remote view "character" being serialized as the inventory instead
of the real character for the player.
Original-patch-by: @Reavershark
Fixes: #513
When creating temporary files use the pattern `.tmp.ext` instead of
`.ext.tmp` to be consistent with Factorio which creates `.tmp.zip` files
during saving.
If patching fails this will leave a `.tmp.zip` file in the instance saves
directory which is ignored by the default save selector.
Resolves#486.
Add controller.autosave_interval config that specifies the frequency
which controller config, hosts, instances, users, mod packs and plugin
specific data which is held in memory should be automatically flushed to
disk if they have changed since the last time they were saved. This
prevents data loss in the event of a server crash.
Plugins can use the onSaveData hook to save its own data at the same
time as the controller does and this obsoletes making an autosave
interval in the plugin.
Resolves#523.
Catch uncaught errors and display the error message on the page itself
if bootstraping fails. This should prevent most cases of the of the web
interface being stuck on "Loading web interface..." after an error was
thrown and the loading failed.
By default webpack-dev-middleware will refuse to delete files outside of
its working directory. This becomes dist/web in the configuration
currently set up, which means that a rebuild triggered on the watched
files will fail when declaration files are output to to dist/web_build.
Fix by moving dist/web_build to dist/web/build.
Clean up the configuration for compiling TypeScript for the Web
interface to us its own tsconfig.web.json config files and reference the
TypeScript source files directly in the configuration and imports
instead of files compiled to dist.
To properly support recompliation with TypeScript's projects feature
each package's tsconfig.web.json file is set to output the build to
dist/web_build which is then ignored in .npmignore. Webpack is still
used to create the proper build in dist/web, so this is mainly to make
TypeScipt's compiler happy but it also allows the root tsconfig.json
file to include the web interface code, meaning running the build script
now also checks the typing of the web interface.
Final part of #491.
The control interface is required for adding handlers for message types
a plugin defines, but the plugins were loaded before the Control
instance was created. This meant there was no sensible way for web
plugins to register handlers for their own message types.
Move the creating of the Control instance to before plugins are loaded
and set this.control for web plugins before init is called.
Output message_validate.js to both dist/src/data and src/data. This
allows tools to directly consume the TypeScript sources in src/data
without throwing an error due to message_validate being missing.
These variables were intended to be globals declare in the global
namespace. That this ends up on the window object is an implementation
detail of JavaScript. Remove the window. prefix put in front of
staticRoot and webRoot and include the global declaration in the
TypeScript module.
Resolve circular dependency of controller depending on web_ui and web_ui
depending on controller by moving the type from controller that was used
in web_ui to lib.
Add strong typing of the invokeHook function to prevent incorrect
paramaters being passed to plugin hooks. This detected a few places
where the parameters passed were incorrect.
Move BaseControllerPlugin to controller/src/BaseControllerPlugin.ts
Move BaseInstancePlugin to host/src/BaseInstancePlugin.ts
Move BaseControlPlugin to ctl/src/BaseCtlPlugin.ts
Move BaseWebPlugin to web_ui/src/BaseWebPlugin.ts
This solves the circular package dependency problem of the base plugins
needing types that are available only in the dependencies to lib.
Add configuration for VSCode to use tabs for indent, unix style endings
and leave a newline at the last line of the file. This will make
editing from VSCode less error-prone as it will select the right
settings by default.
Be consistent with the usage of tabs as indents in all JSON files in the
project. This makes it easier to configure editors to emit the right
indents when manually editing these files.
Leave a warning in case someone opens and tries to modify the
server-settings.json file in the instance folder, as any changes
made will be lost when it's overwritten.
When defining a generic type where a type argument is never used
as a part of the type itself, TypeScript will consider two instances of
the generic type with different values for the type to be the same.
This caused the handle method to accept any handler for any
Request/Event, ignoring the event/request type completely.
Fix by specifying the constructor to return the intersection of the type
itself and the Request/Event type.
Turn on source map deduction in stack traces from thrown errors. This
feature is off by default due to potential performance degration in
certain corner cases. For the dev environment add instructions for
passing the option to node.
Keep a handle to the Instance in the connection the host has to the
instance so that methods can be called directly on the Instance. This
simplifies logic that exists outside of the link system like the host
shutting down all of its instances when it exits.
Fixes validation error thrown on host shutdown with running instances.
Remove extra whitespace, use more consistent spacing, add missing
semicolons, using dangling commas, break up long lines, consistent
multiline operator placement, add missing JSDoc strings and so on.
* allow to revoke / restore host access from web ui
* refector to what is expected
* remove core.admin check on HostViewPage.tsx
* update core.host.revoke_token description
* .jsx to .tsx
* controller: tsc --build before webpack-cli
* Type Annotation
* disabled move into the <Input/> and <Checkbox/>
* cleanup debug left over
* documentation
* default ModPack
* modPackUpdateHandlers no undefined
* PluginInfo changes proposition
* PluginInfo changes proposition miss
* ModRecord & ModInfo feedback loop at const mods = [ ?
* ["connect", "drop", "resume", "close"] as const
* .tsx to .ts when possible
Allow modules to include a module_exports.lua file in their root which
will cause the file save/modules/module_name.lua to be generated
pointing to the exports file.
Map locale files located at locale/{lang}/{file} in a module to
locale/{lang}/{module-name}-{file} in the save. This additionally
support a shorthand notation by mapping locale/{lange}.cfg to
locale/{lang}/{module-name}.cfg.
Resolves#500.
* .js to .ts
* Allow access to _items entries; getItemCount return number
* add permission field to PluginPage interface
* port plugins to typescript
* update Request classes
* fix IpcFlowType
* .js to .ts
* translate
* update paths to dist/plugin
* add "tsc --build" to prepare
* use field main in package.json de find info
* tsc --build need to append before webpack
* update test and use for plugins package.main
The unknown status happens when an instance is on a host that has
disconnected, but since alt-host is not spawned in the tests on
Windows this status is not observed.
On Windows require paths use a mix of forward and backward slashes due
to require path possibly being a package path which can't have backward
slashes, or a Windows file path which can have have either. Fix plugin
loader test failing on Windows due to not using the same combination of
forward and backward slashes for require file paths.
Fix updates to Node.js timers types breaking the TypeScript build by
deducing the correct type from the return type of the
setTimeout/setInterval functions.
Fix coverage checking not being able to deduce the original TypeScript
files due to lacking a way to link the compiled .js output files with
the .ts input files.
Messages are forwareded when received from another link, this means it's
too late to throw an error that bubbles up to the code trying to send
the incorrect message thus defeating the purpose of the validation.
Only the controller has access to the secret needed to sign user tokens
so it makes sense for the user signing code to live in the controller.
This saves about 20ms on the time ctl takes to start up due to lib no
longer needing to import jsonwebtoken.
Move all functions and classes currently available through lib/name and
libName to be under a single flat namespace in lib. This make
reorganising the lib namespace much simpler as the location of symbols
can move without breaking the consumers of lib.
The multi-entrypoint design was already problematic for the web plugin
system which couldn't effectively use it due to how webpack's shared
module system is awkward to set up with multiple entrypoints per package.
- react: Update to new root rendering method
- react-router: Replace useHistory with useNavigate
- react-router: Replace <Switch> with <Routes>
- reimplement sticky notice from scratch in ModPackViewPage
- remove isomorphic-ws as it's latest version doesn't work and only
saves one line of code.
- remove import of antd.dark.css
- fix misc css breakages.
Add type object to schemas specifying properties and minItems to schemas
specifying tuples. Disable strictTuples because it's desirable to be
able to add additional items to the tuples in the future.
Update non-web related dependencies to their latest version were
possible. It appears yargs has changed its parsing of command line
options to no longer remove a layer of quotes.
The LTS releases for v12 and v14 are already end of life and the v16 is
end of life in under two months from now on. Bump supported version up
to v18 as it does not make sense to continue to support EOL releases.
Even if we wanted to keep supporting them, our dependencies will not.
Note that Node.js v18 is currently not avaliable via the NodeSource PPA
on Ubuntu 19.10 and below or Debian 9 and below.
It makes more sense to put the address first then have the large
structure to send as the second argument to sendTo. The opposite order
was originally chosen to be similar to that of sendto in BSD sockets.
Format arguments to have nested structures start on a new line instead
of inline to make it easier to see where the structures starts and what
structure it is.
Keep track of requests that have been forwarded over a link in a similar
manner to requests that were initiated on a link in order to properly
return an error response to the sender of the request if the link goes
down while the request is still pending at the forwarded node.
Validate messages received on the controller from hosts and control
nodes before handling or forwarding them. This should prevent
performing actions that are not allowed by the role system and block
simple attacks such as spoofing the source address of a message.
The data class pattern used for request and event payloads require the
class names to be preserved as they are used for identifying the type of
the requests and events.
Split up the action of registring an event or request data class to be
decoded by the Link class and the action of registring a handler for
those requests and events. This moves the validation work from Link
instantiation time to startup initialisation which makes logic errors
fail on startup instead of failing instantiation of the link when a
connection attempt is made.
Additionally implement a partial validation logic for messages received
using the newly refactored message registration. This unfortunately
ended up merged with the registration refactor.
Reverse the direction of the API for sending messages from
messageDefinition.send(link, { content });
to
link.send(new MessageClass(content));
and convert all message definitions to use the JSON serialisable class
pattern and explicit registration of message handlers, along with a
end-to-end based routing of messages instead of link to link.
The experience I gained from the existing way messages were defined and
sent over links was that it was difficult to understand for people
unfamiliar with the system. The automatic registration of message
handlers ment that code was being called as if by magic from nowhere
with no easy way to discover it. The manual forwarding and rules for
what can be sent where and how obscured its working.
The use of pure JSON as both input and output from message handlers made
it tedious to work with. Data had to be manually converted between the
program representation and the over the wire format, which made handler
code unplesant to write. Particularly because the over the wire format
used snake_case while the program code uses camelCase for variables.
The idea behind the decision to do it this way was to make it more clear
which data came from the wire and which from the program, and while this
worked it didn't make message handlers any nicer to write.
The direction of the API and magic registration of message handlers also
meant that a future migration to TypeScript would be troublesome. While
it may be possible to implement, the degree of TypeScript magic required
would go starkly against the principle of keep it simple stupid.
By refactoring to using the JSON serialisable class pattern most of
these issues go away. There is no longer a need to distinguish the on
wire format of messages with the program representation because the only
place that deals with the wire format of messages are the toJSON and
fromJSON methods on the classes themselves. Message handlers will work
with proper instances of classes instead of raw data, something that
makes them easier to write, and trivial to type with TypeScript.
End-to-end routing also simplifies sending messages that need to do
mulitple hops to reach the destination. And removes the need for having
all messages be specified in plugin's info module, as they can be routed
without having to be parsed completely.
The explicit registration of message handlers is something that I
thought would be tedious boilerplate code that could be automated away.
But experience now shows it to be the opposite. Allthough dull
repetition, the purpose it serves is to link together the code in a way
that humans can easily find. With explicit registration searching the
name of the message handler in the editor will show the registration that
ties it together with the type of message it handles and the kind of
registration. With implicit registration searching the name of the
message handler turns up only the handler with no clue as to what calls
it.
Change the JSON seriasiable data classes to use an explicit static
fromJSON method and an optional toJSON method for turning them to and
from JSON.
This improves over the previous pattern of passing the JSON to the
constructor as it frees up the constructor to be able to take whichever
argument types suits the code instead of being beholden to the JSON
serialised form.
Add migration code to installer fixing the names used in configs,
logs, database, and packages to reflect the new controller/host names
for people updating from alpha-13.
The role of a slave as a host for Factorio servers is not well reflected
in its name and has caused confusion among users. Rename slave to host
to better reflect its role as a host of Factorio servers.
The role of the master server as the party controlling every part of the
cluster is not well reflected in its name and has caused confusion among
users on several occasions. Rename master to controller to better
reflects its role.
Check patch number in update_instance step to ensure the clusterio_api
is properly initialized if the game started tick paused and no ticks
to trigger the patch check have happened yet.
Technologies that were disabled were ignored completely by research sync
due to a missunderstanding of what disabled means. Disabled does not
mean the tech is somehow inaccessible and completely out of play, it
only means it can't be selected in the technology screen.
Ignore the enabled status of technologies in research_sync to fix a
crash that happened when a mod enables a disabled by default technology
and the player attempts to research it.
If a system crash occurs after a simple write file + rename operation
the target file that was renamed over may end up being empty if the data
for the file written had not yet been committed to disk. This is
possible because no part of the open file, write data to it, close it
and then rename it ensure the data was actually written to disk prior to
the rename happening. See this discussion on the btrfs wiki[1].
Fix this possible data loss scenario by calling fsync on the temporary
file that is written prior to calling close on it.
[1] https://btrfs.wiki.kernel.org/index.php/FAQ#What_are_the_crash_guarantees_of_overwrite-by-rename.3F
All possible Promise rejections and exception bubbling should be handled
by either the code creating the exception or the code invoking the
operation that could fail. Treat therefore uncaught exceptions and
unhandled promise rejetions as fatal errors that log a traceback and
immedately terminate the Node.js process.
The format of mod-list.json written on syncMods would incorrectly write
a base mod without an enableds key which is required, causing Factorio
to ignore the file entirerly. Furthermore the enabled status was not
written causing all mods to be loaded even if they were disabled in the
mod pack.
The logging format provides both the name and id of slaves and
instances. Use the name when formatting for the web console just like
the TerminalFormat does.
Track summary statistic of total HTTP response durations using a
middleware that applies for all requests instead of having every request
needing to update the hit counter manually. Deprecates
endpointHitCounter.
The rationale for not allowing players to list users in the cluster by
default was that this would be an information disclosure of banned users
in the cluster, but given that a player is a user that can connect to a
server, and anyone that can connect to a server get the full banlist
from the server during the connection phase this does not make much
sense. Grant permission to list users by default for players.
Check that Master/Instance/Control plugins export the correct class and
that this class is a subclass of the correct base class. This gives a
better user experience when plugins are incorrectly set up and loads a
duplicated clusterio/lib, and when accidentally using the wrong name
when developing plugins.
The directory which all files are placed under in a mod is commonly
named <mod-name>_<mod-version>, but this is not a requirement as it can
be named anything. Fix code incorrectly assuming the zipped mod's root
folder follows this pattern during data export.
Use exported settings prototypes and locale to format the settings UI to
with mod names, settings names, and select dropdowns. This is a crude
approximation of how the game presents these settings to the user.
Add missing mod settings set to their default values to a mod pack when
performing a data export based on the settings prototypes that were
extracted from the game. This mirrors the behaviour of Factorio when new
setting prototypes are added by a mod.
Updates to Node.js error reporting have caused tests to fail due to them
being hardcoded to expect a certain format that has since changed. Fix
by making Node.js create the error message for us.
Store the export manifest in ModPack to allow mulitple mod packs to have
different data exported. This is necessary as the export contains the
settings prototypes needed to create a proper UI for mod settings.
For now the useLocale and useItemMetadata interfaces uses the manifest
for a default mod pack set in the master config.
Add settings and all prototypes to the export of data done. Settings
are needed for a settings GUI to function, while prototype data would be
useful to display any sort of information about things from the game.
Mark mods in the UI and the mod-pack show ctl command that are missing
with a question mark and mark mods with a missmatched checksum with an
exclamation mark.
Validate sha1 sum matches for a mod used in ModsTable if it is present.
This prevents showing details for a zip file where the version matches
but the cheksum does not.
Wipe and reconstruct the instances' mods/ folder using the configured
ModPack for the case were the mods folder is shared with the master
server. This assumes the master server will check the mod checksums
before sending the start request.
Remove ugly hack to generate and use an insecure certificate on the fly
in the HttpCloser's https tests and replace it with the already existing
properly generated test certificate. Removes the dependency on node-forge.
Make all properties on the mod pack object except sha1 required and add
the base mod if it does not exist in order to align the JSON
representation with the proposed Mod Pack String format I published on
the Factorio Forums (https://forums.factorio.com/103578)
Enforce the filename of uploaded mods match {name}_{version}.zip with
name and version taken from the zip file's info.json entry. This allows
it to be used as a unique key for the mod and solves part of the issues
with mods having different number of leading zeros in their version.
The Factorio Mod Portal uses and provides SHA1 hashes of mods, so it
makes sense to align the ModPack and ModInfo structures to sha1 by
renaming the genericly named hash property to sha1.
Make the extra content for section headers flow under the heading when
there is not enough space to show both the heading and the extra content
in a single line.
Use overflow clip instead of hidden in the page area of the site layout
to avoid creating a formatting context that would make sticky not work
as expected inside the page area.
Prevent tables cells containing byte sizes from having a line break
inserted between the number and the byte unit when they get squished by
using a no-break space instead of a regular space.
Add ModPack data type with corresponding storage on the master, messages
to create, update and delete them, as well as commands to invoke them.
The ModPack data type contains a list of mods with versions and optional
sha1 hashes, and mod settings to be used.
Add the ability to manage a set of mods to be shared by the master
server to the rest of the cluster. This implements storage, uploads,
downloads, deletetion, and listing details of mods but does not actually
do anything with the stored mods.
A file name con.zip is also treated in the same way as the special CON
redirection and may not be used as a normal filesystem path in Windows.
This includes any extension, even with multiple dots. Fix checkFilename
and cleaneFilename not handling these cases.
Fix the installer wiping the package.json file due to a call to
outputFile that needed to target the original file being incorrectly
changed to safeOutputFile.
Be a bit more accurate in the online time stat name and call it for what
it is. It counts time spend online on the server, not necessarily
time spent playing.
Fix broken support for retrying uploads and background downloads by
making crafting queue serialisation non-destructive and only performing
the serialisation and deserialisation while the player is online.
This removes the /save-crafts and /restore-crafts commands because the
existing implementation would voids the non-saved crafting queue items
if /save-crafts is performed and then new items are added to the
crafting queue before switching server.
Fixes#419.
If only the bot was clever enough to see that node-forge is only used
for a test that has no user input it might have had the foresight to
realize that crying wolf about completely irrelevant security issues
will only lead to legitimate issues being ignored in the future. But
alas, the AI overlords are not taking over any time soon.
On Linux the installer offers to correctly download and install the
latest version of Factorio. Remove no longer needed instructions for
how to do it incorrectly.
I have seen multiple people have installed Clusterio as the root account
and then run it as root when tasked with setting up Clusterio on Linux.
Make this mistake harder to do by refusing to install as root by
default.
Since the admin and whitelist status is shown with a tag next to the
name it's no longer necessary to show the admin and whitelisted switches
when the user does not have permission to use them.
Add command that extracts play time (and players) from the currently
running save to the recorded instance statistics. This is useful for
getting correct play time statistics when an existing save game is
imported.
In the case of shutdown the Factorio server will not notify the
currently online players as leaving the game. Notify the master server
that all remaining online players quit the game in this case.
Count up to 100 empty slots past the last seen slot with data in it
before stopping to look for personal logistics slots in inventory_sync.
Allows unlimeted amount of logistics request slots to be used for
heavily modded games.
Factorio 1.1.40 added a feature to use a centralized banlist maintained
by Wube. For the dedicated server this needs to be turned on with the
command line flag --use-authserver-bans.
Void excess slots when deserializing an inventory to a target that has
fewer slots than what was serialized. This fixs inventory sync failing
when a stored inventory has trash slots in use but the player does not
have the slots unlocked.
Store exported data files with a hash in the filename to allow the new
caching stategy for static files to work with it and provide a manifest
for the web interface to find the exported files with.
Render the web interface as early as possible with a black background
and show a message to indicate whether the interface is actually loading
or if an error occured loading the main bundle.
Put all web assets including plugin assets under a shared static/ route
in the web interface that's declared as immutable and cacheable for 1
year. This allows all JavaScript bundles to be loaded from disk instead
of having to check with the web server each time and significantly
speeds up the web interface load time. It also fixes the public path
issue when using the --dev-plugin option behind a proxy.
To prevent collisions and stale data all assets now include a hash in
the filename. This means the entrypoint is now dynamic and specified in
a manifest.json file created from the web build. For plugins this means
the filename attribute needs to be removed from webpack.config.js.
Mocha's recursive search feature finds all files and then removes paths
that match the exclution filter, except that the globstar filter doesn't
match directories with a dot in them which is created by pnpm. Avoid
this madness by removing the test package.
The prepare script runs before publishing a package, after installation
of a package from source, and when running npm install without
arguments in the repository. These are the times where a build of the
web interface files are required to make the web interface work and
removes a manual step from using the package(s). The "build" script on
the other hand is never automatically invoked by npm.
Save disk space, time and sanity by replacing npm and lerna with pnpm.
This is more performant and doesn't have developer experience ruining
bugs like npm link removing all previous links before creating the
requested one.
Add ability to move or copy saves between instances. For instances on
the same slave the saves are transferred with filesystem operations, for
instances on two different slaves the proxy stream interface is used to
send the save via the master server.
Travis seems to be going away from supporting Open Source Software for
free and their proceedure to gaining free tokens is getting harder and
hander to find for each year that passes. Remove Travis CI integration
and migrate the coverage reporting previously provided by the Travis
build over to the GitHub Actions.
Allow sidebar entries to be grouped together by specifiying a name under
which they will be grouped. To avoid confusion groups are rendered at
the bottom of the list.
Convert the master metrics to void instead of storing them in a variable
as they are completely self contained, and this makes it more clear that
there are no use of references to the metrics.
There are some corner cases with streams that cause them to not emit
finish close or end events which in turn means that code liseting for
these events directly may end up hanging forever. Use stream.finished
to detect end of streams which will handle these cases correctly.
The interface provided by Winston is unusably slow for querying large
logs. Reimplement the log querying with both forwards and backwards
line reading and a simple file based index to skip files which do not
contain any lines matching the filter.
Move the default location of the log files into a configurable logs
folder and split the written logs up by UTC date to more easily allow
deleting of old logs to be performed. The log rotation is implemented
by winston-daily-rotate-file.
The old log format will automatically be migrated to the new format and
location if the target folder for the new logs don't yet exists.
The burner equipment serialization code was never tested and
consequently never worked. Wasn't noticed before now because burner
equipment isn't in the vanilla game and few mods use it.
Fixes#442.
Turns out objects in the second argument to useEffect is compared by
Object.is(), this means if the array containing the instances or slaves
to show logs for get re-created during an update the LogConsole
component would send a new request for the logs.
Fix by converting the slave_ids and instance_ids arrays into strings
before passing them on to useEffect.
Add splitter based on LineSplitter which splits lines in reverse from a
reverse stream created from the added createReverseReadStream function.
Enables line processing of files starting from the last line and going
towards the first.
Pull the LineSplitter class out of lib/factorio to its own stream module
and refactor it to be based on stream.Transform. This allows composing
with other streams by piping it around.
Instances require a number of modifications leading to it not being
able to be converted into a function. The main one is that there is a
list of config elements that cannot be set via the edit command and
either have to be set via different means or cannot be set. To stop
these fields constantly sending error messages, we disregard adding them
to the file in the first place.
Some systems may resolve localhost to ::1 while Factorio doesn't respond
to RCON connection over IPv6 loopback, this causes instances startup to
fail. Fix by using 127.0.0.1 instead of localhost.
Using an event emmiter, the fields can now be set correctly in the
master config. Null values are now handled correctly, and numbers are also set
correctly.
Also now using the system described by Hornwitser, in using
libConfig.MasterConfig.groups to get definitions. Now all that's needed
is to re-read the updated data back into clustorio.
Write new file contents to a temporary file and then rename over the
target file to avoid data loss in the unlikely event there's an error
writing the new file contents or the process is interrupted.
Resolves#416.
ctl master config edit [editor]
is the new structure; it can take an optional editor, and will prio this
editor over any other editor, such as one set by env vars. Generally,
the prio goes like this: argv supplied editor > EDITOR env var > VISUAL
env var. Things needing to be done are marked with a TODO.
Implimented framework for edit command, including ability to get a temp
file, open an editor on the file, then resume code execution once the
editor exits.
Somehow the step of automatically obtaining the right address to apply
to slave.public_address, and asking for the master url and token for ctl
installations was implemented but the code to actually set it in the
relevant config was forgotten.
The serialization and deserialization of editor mode is not supported by
the current code, and while the download correctly checked for the
player being in editor mode, the upload did not check for the player
being in editor mode causing players in editor mode to break. Fix by
adding check in upload_inventory.
Fix commands syncing researces being sent before the instance is started
up, and potential desync of research progress as a result of out of
order execution of RCON commands sent.
If the session to a slave is lost while gathering metrics the error
would propagate out to the metrics route handler and break collection of
statistics. Fix by handling errors thrown from sending the getMetrics
request to slaves.
Don't bother doing the prepareDisconnect dance if the connection is
already broken. This avoids having to wait until the connection times
out when a slave gives up on a current connection and starts a new one.
If a connection has timed out the most likely cause is the connection
going stale, in which case calling close could take up to 30 seconds
before the ws lib gives up on waiting for the close message. Terminate
the connection instead on the Node.js side. Note that Firefox does not
permit applications from closing connections with a code other than
1000, or in the range 3000-4999.
The INI format Factorio use differs from what the ini package parse and
produce. Most notably no escape mechanism exists in Factorio's parser
which causes instance paths containing among other thing the # character
to completely break. Fix by replacing the ini package with a Factorio
compatible implementation of INI parsing and serialization.
Resolves the problem in #414.
Move the save list subscription permission to be after the save list
permission as these two permissions are logically related to each other
and both need to be granted to view the save list in the web interface.
Add the roles and their permissions for the user the control connection
is connected as along with updates when these change. This enables the
web interface to know which permissions it has available to it.
Fix has_data not being checked by sync_player, dialog being stuck open if
no data is received by the master and master server refusing to server
empty data.
Add optional limit to the number of log records to return in the
log query request to avoid having the web interface instance console
download the entire log for only to show a few hundred lines from it.
To make it possible to query the end of the log instead of just the
start add an order option as well.
With the instance status unknown now meaning assigned slave is offline
the save list buttons can now easily be disabled using this status.
Disable the upload save, download save and delete save buttons when the
slave is offline or the instance is not assigned to a slave.
The system for preventing items being sent without a connection to the
master and (and thus voiding the items) is to only do so if a ping from
the plugin has been received in the last 5 seconds of in-game time.
Don't send pings without being fully connected to the master to make
sure this system actually activates during connection downtime.
Ignore inventory updates and orders while the instance is not in the
running state to avoid sending commands before the server has started
up, and also ignore orders if the connection to the master server isn't
isn't online to avoid piling up requests in the buffer during session
resume attempts.
Since the connector refactor the action of sending a message on a
connector that is in the process of connecting changed from queuing up
the message to send it to throwing an error. To avoid events forwarded
by the slave to to the master when the connection is down from being
logged as an error forwardEventToMaster has to explicitly drop them.
Prevent ordering issues with save notifications on stop by making the
save list update part of the stop action instead of some unspecificed
amount of time later.
This also fixes the issue of the connection being closed before the last
instance sends it save list on slave shutdown.
Provide a convenience method for checking if the Master-Slave link is
fully online via a .connected property. This is true if the link has
been establised, is not in the dropped state, and not in the process of
disconnecting.
The activeConnectors mapping is from session id to connector, but the
master refactor in 67219ed accidentally changed it to try deleting the
entry by the connector reference instead. Possibly mistaking it for a
Set instead of a Map.
Complete rework of the inventory synchronization to reduce the number of
times the inventory has to be downloaded and the possibilities for item
loss/duplication.
The basic concept is to acquire exclusive access to the player data
before downloading the data. This prevents the player from logging in
to two servers at the same time and getting the synced inventory twice.
The acquisition will in most cases execute faster than the player can
provide input to the game on login, and if the player data on the master
isn't what was previously uploaded then the player inventory is deleted
and the player is turned into a spectator while the updated inventory is
being transferred into the game. This model allows for the downloading
of the synced inventory to be skipped in the case the player joins the
same server they left from.
The synced data is expanded to include more than just the character and
inventory to properly handle sync of dead players. If the player leaves
dead on one server this state will be saved and when the player joins
another server they continue off as dead. The ghost, god, and spectator
controllers are supported, while the cutscene and editor controllers are
not. For players in a cutscene when joining the acquisition will be
attempted immediately, but the scenario waits until the player is out of
the cutscene before acting upon the response to the acquisition. For
players in the editor mode the scenario toggles them out of the mode
before syncing data.
Since uploads to the master can fail handling of this is also added.
The scenario will retry uploading the player data at exponentially
increasing intervals until it gets the message back that the upload
succeeded. Since it's possible for servers to go down and stay down for
a long time the exclusive access held by these servers are released after
a timeout. This means that a player could be playing and or have
uploaded newer data by the time the instance comes back online and tries
to upload the inventory, in this case the master will report the upload
succeeded but discard the data. Instances may also upload old data if
they load back to an earlier save, which will also be discarded by the
master.
Other small improvements made by this rework:
- Add a retry button to the failed download dialog
- Update the existing progress dialog instead of destroying and
recreating it every time.
- Automatically stuff the items from the temporary inventory into the
synced inventory if they fit.
- Move player back to the position they started the download from if
they were turned into a spectator for the download.
- Ensure the player has a character if the stored data is wiped.
- Add error checking of the RCON commands sent by the instance.
Refine the state flow used in the connector abstractions to better
handle close while connecting and resuming. The new logic is to close
the connection immediately in these two states instead of continuing
trying to connect and then giving up some point later.
For the connectors the handshake state is split into connecting
and resuming, new is replaced by closed, and the closing state is turned
into a boolean flag as it's possible that closing is desired while
connecting, connected or resuming. The connecting state is not present
in the server connector because there's nothing for the server to do in
that connector state.
The timeout logic is completely removed, once a connection attempt is
started for the client it will continue to attempt to connect, attempt
to resume dropped connections and attempt reconnects until either an
authentication error occurs or the client explicitly requests to close
the connection.
A new session timeout is added to control how long to attempt to resume
the connection until it's considered dead and invalidated. This allows
stale connections to give up earlier, and swallow up less data while in
the dropped but not yet invalidated state.
For the client side sending messages while it's connecting has been
changed to throw an error instead of queuing up the message and
(possibly) sending it after the connection is established. Code
shouldn't try to send messages without knowing if the connection it's
trying to send them over is active or not.
The ajv module was made shared to reduce package size, but it turned out
the package size was large because of a missunderstanding in how imports
to sub-paths works with module federation. Since that was fixed the ajv
module was no longer shared, but plugins incorrectly still specified it.
In order to import a path inside packages/lib/ in the webpack shared
module environment it has to be declared as shared in both the web_ui
webpack config and the plugin's webpack config. This is rather error
prone and easy to miss. Move the burden of specifying which entrypoints
are valid in lib to lib itself by changing all imports in the web
interface code to import from @clusterio/lib instead sub paths.
For web code this means imports is done like this from now on:
import { libConfig, libLink } from "@clusterio/lib";
For info.js which has to be loadable by both Node.js and web code the
imports are done as follows:
const { libConfig, libLink } = require("@clusterio/lib");
While the Node.js only code can either continue to import sub paths
directly or use the info.js style.
Create entries for players that are present in the save on server
startup to preserving their inventory and fix attempting to reference the
player object after it had been removed in on_player_removed.
Make sure the GUI element in the on_gui_click event is valid before
attempting to access this, as another module may have destroyed it
during its event handler.
When upgrading or downgrading the inventory_sync plugin it's possible
an different structure of the global.inventory_sync table is left behind
which could cause the code to crash when certain sub tables are not
present.
Be more defensive in the initialization of the table to make sure all
sub-tables are present on startup.
Fix setting a nullable numeric field to blank in the web UI return error
that the value cannot be a string due to the field returning null and
this being converted to the string "null".
Ignore stop if instance status is already stopped. Fixes instance status
getting stuck at stopping if Factorio exits during startup due to the
stopped status having already been sent at the time stop is invoked.
Fixes#400
Convert code indents to tabs to be consistent with the rest of the
JavaScript code. This commits covers all of the lua code, some of the
hand written JSON code and the lone HTML and CSS files that were not
using tabs.
The jsonwebtoken accept strings as input for secrets which is
nonsensical from a cryptographic standpoint as the algorithms all
operate on byte sequences. Worse still, strings are encoded into bytes
using UTF-8, which disallows certain sequences of bytes from appearing
and potentially weakening the cryptography.
For legacy reasons the master secret was encoded into base64 _and passed
as a string to jsonwebtoken_. This further weakened the crypto as the
bytes that can appear in base 64 is only 1/4 of the 256 possible values
for bytes. This could significantly weaken the cryptographic strength
of the tokens.
I have known of this issue for quite some time but has put off actually
fixing it for too long now. Since the master secret is already
documented as being in base 64, decode it as base 64 and pass it as a
byte Buffer to jsonwebtoken. This will ensure every byte value is
possible in the byte sequence passed to the jsonwebtoken crypto.
The impact of this change is unfortunately that all tokens will become
invalid and will have to be reissued.
* New method avoids all dataloss and prefectly replicates the queue
* Made proper description in readme
* Added packages to make lerna build happy
* Added reference to main readme
It's possible to put leading zeroes in the names of mod zip files which
cause the existing algorithm to find the mod zip from the mod version to
fail. Fix by listing the mods directory and fixing the version numbers
for any of the mods with leading zeros in them.
Fix incorrect serialization of control characters to \xHH escapes since
JSON only support \uHHHH escapes, and serialization of infinite and NaN
numbers.
Browser like Chrome doesn't use the application/zip mime type when
uploading .zip files. Add support for application/x-zip-compressed to
the save upload endpoint.
Provide a create package for the clusterio npm org which gathers the
necessary details to install a working cluster either as standalone or
as separate nodes.
Add a generate slave token button to the web interface and modify the
generateSlaveToken message and corresponding ctl command to have the
slave id be optional. The random id is the desired behavior when adding
a slave with the installer, as it'll ask for the token for the new slave
and deduce the id from the token.
Gracefully handle invalid IDs in useSlave and useInstance as this is
necessary when when the ID isn't known by the component using the hook,
since it can't conditionally call the hook.
When factorio starts up with with the require_user_verification setting
turn on it tries to obtain a server padlock from the matchmaking server
and will fail if this does not succeed. Turn off user verification and
public visibility which require user verification both of which are on
by default for the integration tests so that they work without internet.
Add logic to stop sending commands and wait for any pending commands to
complete during instance stop in subspace_storage. This should fix the
RCON connection lost errors printed in the console during stop.
Prevent potential data loss when starting an autosave by copying it
first. This avoids overwriting a random autosave slot when the server
shuts down after having loaded an autosave.
Add /api/upload-save endpoint that streams the data via the slave
assigned to the target instance to the instance saves directory, along
with a ctl command and web interface button to do the upload.
Add link messages for listing saves and listening for changes to the
list of saves for instances along with a table in the instance view page
showing the save list for the instance and a list-saves command to ctl.
Add options to pass map seed, map exchange string, map gen settings
and/or map settings when creating a save either through the create save
request or the load scenario request.
Check if the instance status in the instanceStatusChanged event actually
changed and ignore the event if it didn't. This prevents bogus updates
from being pushed to control clients and plugins.
Provide the control link with the ability to subscribe to updates when
instance status and properties changes. This allows the web interface
to respond in real time to instance starts and stops that are triggered
from somewhere else.
Add readMapExchangeString to lib/factorio which parses and decodes the
data in Map Exchange into a format suitable for giving to the Factorio
server command line.
The plugins are available as a property on the Control provided by the
ControlContext which makes the PluginsContext redundant. Remove it and
use ControlContext to retreive the plugin list in the app.
The Node.js specific rules for Eslint were deprecated in favour of the
node plugin for Eslint. Add node plugin for eslint and switch over to
using its rules, fixing any issues detected.
Adjust the regex for detecting startup to also work with 0.16 as
updateTick was called mapTick before 0.17. Note: This commit does not
imply 0.16 is going to be supported by Cluestio.
Notify when an instance is in the process of stopping with a separate
status for it. This makes handling shutdown gracefully a bit simpler in
plugin code.
During Travis integration tests a BrokenPipeError is emitted on stderr
after the Factorio server has quit. I have no idea what's going on here
so for now just log it and ignore it.
Move the depenedncies needed to run the tests into test/package.json and
manage it using lerna. This should make it easier to selectively
install only what's needed to run the software.
Fix path stripping logic to take into account the possibility of the
writePath being relative, and that Windows uses \ as a path separator
for some paths but not all.
Add reference to the Control instance in BaseWebPlugin for web plugin
code to use in a similar manner to the instance reference in instance
plugin code.
For master plugins a message can be received from any of the clients
corrently connected to the master. In order for plugins to be able to
handle link specific requests they need a reference to the link messages
was received from. Add link as the third argument to plugin message
handlers.
The automatic determination of the public path for plugins works
correctly when they are build for production and loaded from their
respective paths. But when the dev middelware compiles it, it doesn't
know the correct path to host it at withouth the publicPath being set.
Fix by setting publicPath explicitly when compiling with the dev
middleware.
Add the name of the account that's connected to the control link on the
web interface via an additional property on the ready payload for the
control link.
Add a button on the right of the top nav bar to access account related
functions and add a log out button there. To make it fit on the navbar
the text and version is hidden on narrow layouts.
Simplify the the way markdown files are formatted to make them easier to
author without editors that do auto line breaking, and make the diffs
from them simpler to read.
Fix ServerConnector reaching an invalid state if a continue operation is
performed on an active connection. The continue would cause the
connection established logic to be runned before the connection lost
logic of the pervious connection is perfomed, resulting in a leaked
heartbeat interval and the connection lost logic being performed on the
new connection instead of the old. Prevent this by closing both the new
and the old connection in this case.
Add authentication plugin that does a simple challenge response protocol
to verify that the player trying to log in is who they claim to be. The
process goes as follows:
1. A player code is generated for a player that wants to log in and is
displayed in-game.
2. The player code is input into the web interface and the master
generates a verification code as a challenge back along with a token.
3. The player inputs the verification code in-game and is logged in to
the web interface in doing so.
To do the actual login the web interface polls a verify endpoint every 2
seconds until the code has either been verified or expired. The
challenge comes from being unable to control the verification code and
having to input it as a player logging into one of the servers in order
to login.
Some additional steps are taken in order to increase the difficulty for
an attacker to subvert the challenge:
- The login process must be started from in-game.
- Generated codes are only valid for a limited timespan which is
configurable and 2 minutes by default.
- The verification code comes with a token that only works with one
player code and verification code combo and must be provided to pass
validation.
Change to "auto" as the publicPath to have webpack deduce the correct
path at runtime. This fixes an issue with plugins being unable to be
loaded due to trying to load bundles from the wrong path when the web
interface is hosted under a proxied sub-path.
Remove basename in favor of making webRoot absolute. The distinction
between the two were supposed to be that basename was absolute and did
not have a trailing slash while webRoot was relative and had a traling
slash. This was not how it ended up working in the code, which lead to
double forward slashes in requests sent out when webRoot was not equal
to './'.
Due to the way the imports and exports works with module fedderation
each require path needs it's own shared module in the plugin. To avoid
having to specify every possible import in the module federation plugin
config, export all useful components to plugins in the browser
entrypoint for web_ui.
Note that for some reason the module wouldn't exist for import by
plugins unless some part of the build imported it. Which is the reason
for the master index.jsx to import bootstrap via it.
Add a complementary total hit counter for all HTTP requests. The
endpointHitCounter doesn't cover all endpoints and fixing it to cover
all endpoints without making it possible for it to create an infinite
number of time seriers is not straight forward.
If data from the database is not loaded but a shutdown is triggered then
the master server will overwrite the database with empty data, causing
unexpected data loss. This occurs for example when the --dev-plugin
option throws.
Fix by setting the database entries to null before they are loaded so
that should an attempt be made to save them without having been loaded
first it will throw an error instead.
The web files for plugins were hosted at /plugin/plugin_name/ while the
view page for a plugin is hosted at /plugins/plugin_name/view. Reduce
confusion by making both urls use the plural version.
Add hidden development option to run the webpack compiller for the given
plugin(s) similar to the --dev option for the built-in web interface.
This requires publicPath to be set in the webpack config and an updated
version of webpack-dev-middleware to function correctly.
Add separate BaseWebPlugin class which plugins can extend and expose
through a webEntrypoint in a similar fasion to the existing master,
instance and control entrypoints.
To simplify the logic for the rest of the web interface code the plugins
context is changed to be a mapping of loaded WebPlugin instances. This
in turn means the plugins and plugin view pages fetches the plugin list
upon load in order to show plugins that failed to load (which
incidentally also gives them an up to date list).
During startup the plugins are loaded one by one and init is called for
each of them. But due to the already loaded plugins being discarded if
one of the plugins throw an error an unclean shutdown that might hang
on timers started by plugins occurs. This leads to the master server
starting up, printing an error, displaying the shutdown sequence and
then hang forever.
Fix by storing the plugins loaded into masterPlugins as they are loaded
instead only after all of them have been loaded.
Use relative imports in lib to avoid an issue with webpack thinking
absolute imports inside @clusterio/lib are for a separate module and
giving out warnings for it being missing in package.json.
The module sharing in Webpack works on a per import basis and therefore
doesn't trigger when the value in the config is @clusterio/lib while the
module imported is @clusterio/lib/config. This in turn caused a copy of
lib to be bundled with each plugin. Fix by making sure each lib import
is shared and specified in the configs.
Rename the config and status indicating whether to load a plugin or not
from "Enabled" to "Load Plugin". The latter better describes what it
does, and unlike the former doesn't imply plugins can be enabled or
disabled on the fly.
When setting up the tests there were issues with code that didn't
properly clean up after itself and left lingering connections, intervals
or timers that kept the event loop running after the tests where done.
To fix this the --exit flag was passed as a temporary measure but then
subsequently forgotten about. The issues with keeping the event loop
alive have long since been fixed on Linux, but on Windows there was a
problem with the control connector not exiting because the master server
was killed instead of shutten down properly.
With that issue fixed there are no longer any connections or timers
keeping the event loop alive and --exit can thus be removed.
If the connect close method is called while the connection is waiting
for a reconnection attempt then the attempt is cancelled but the close
method then erroneously waited for a close event that would never
happen. Fix by resetting the connector and rasing the close event in
the close method if it's in the waiting for reconnect state.
The removal of the automatic TLS certificate generation removed the use
of node-forge in the master server, while the addition of the HttpCloser
added it as dependency of the tests. Fix node-forge dependency being
listed in lib and master instead of as a dev dependency in package.json.
Add page listing the plugins installed on the master along with a
detailed page for each plugin with fields from their package.json.
Failure to load the module and version missmatch is also reported.
This adds ./packages.json as required part of the webpack module.
Add a React context to access the plugins loaded in the web interface.
The info, /api/plugins metadata and webpack container is provided for
each plugin.
b538960 added an error boundnary for pages not directly linked on the
sidebar, but forgot to add it for pages on the sidebar. This add the
missing error boundnary for those pages.
Log the stack of the error caught during failure to load the web
interface. This ensure the error is printed to the console instead of
"[fatal] undefined".
Provide the loaded file of the tls_ca config available to slave,
instance and ctl code. This ensures they can easily use it when
connecting to the master server.
Remove the need for installing babel and the webpack loaders used in the
webpack.common.js file separately by resolving them from the web_ui
package. This allows future changes to done to the modules used without
having to update them in each plugin that uses the web_ui common config.
Pull out the instance specific logic from InstanceConfigTree and turn it
into a general BaseConfigTree component that can be used for master and
slave configs.
Prevent the page from turning into a black screen when page rendering
fails by adding an error boundary to the display of the page content in
SiteLayout.jsx.
Add a per field access parameter to the config class which specifies
which of master, slave and control can access the given field. A
location is added to the config class to specify which party owns it and
fields which do not have an access parameter are not present in the
config and will throw if attempted to be read from or writen to.
The serialize, load, update, get, set and setProp are ammended with a
location parameter which specifiy where the data is from or destined to
and the access checks are performed to ensure remote access to the
config is still limited by the access parameter of field.
With this system in place the master config can be shared with control
and slave connections without compromising secrets that should stay
private. This is primarily the master authentication secret, but other
plugins may define secrets it does not want to have shared with slave
and/or control connections.
Raise "resume" instead of "connect" when a connection is re-established
after a connection drop. This prevents logic meant to resynchronize
state after a full reconnection from being invoked whenever the
connection was temporarily dropped.
Implement --add-perms and --remove-perms to add and remove permissions
from the role without having to list up all of the permissions in the
role again.
Rename --permissions to --set-perms to be consistent with the two new
options added.
Replace old paragraph in the developing for clusterio document about the
removed HTTP interface with a notice of the new protocol running over
WebSocket.
Remove the pluginInfo parameter from lib/plugin attachPluginMessages as
the plugin info object is available in the `.info` property of the
plugin. While a plugin instance doesn't exist for the slave side and
was the reason for these to be separate arguments, I expect slave
plugins to be implemented eventually.
Move the target folder for the built web interface to dist/web/ instead
of static/. This better follows the norm for npm packages to include
built files in the dist/ folder and also allows putting other built
files into that folder in the future.
Exit immediately if shutdown has already been called. This is due to
the possibility of it being called multiple times from combinations of
signals and fatal errors.
On windows the terminal output is asynchronous by default which leads to
log lines being lost if text is written to stdout before calling
process.exit. Fix by using setBlocking before doing the log call.
Add --stdin option to the config set commands which reads the value to
set via stdin instead of on the command line. This allows a more secure
way of entering values and avoids having to deal with multiple layers of
quoting and escapes for special characters.
Add optional arguments to Request and Event to control which properties
are required to be present in the payload. This allows gracefully
extending the protocol, reducing overhead for optional data, and leaving
properties out for additional behaviour.
Add settings of interest to server administrators that appear in
server-settings.json to the factorio.settings config. This makes it
easier to see which settings can be set there.
When parsing using the .argv interface to yargs the parser state is not
restored back to its pre-parsing state after the operation completes.
This causes a susequent reparse with argv to forget its commands and
respond with an unknown arguments error when trying to use the
control-config command to clusterioctl. Fix by switching to use the
parse() interface which does restore the parser state after completing.
The automatic creation of a self-signed certificate while nice presents
several problems. To make connections not reject the self-signed
certificate the validation of TLS certificates are wholesale disabled,
decreasing the security even when properly set up and trusted
certificates are used.
To make the created self-signed certificate pass validation it is
necessary for it to list the host names it's valid for and to copy it to
each machine that intends to connect to the server. These two steps
complicate automating the process in a reliable way and will likely lead
to confused users seeking to disable TLS certificate validation as the
easy way out to problems.
For these reasons, remove the automating creation of a broke self-signed
certificate, fix validation of certificates and document how to set up
TLS properly. This makes HTTPS recommended when accessing over the
internet, but still something which must be explicitly configured.
If the server decides to exit on its own during the stop operation to
the FactorioServer class after the operation has started but before the
signal is sent to the process then the exit handler will have wiped the
server child process and cause the kill call to fail. This in turn
would let the hang timer running which would bring down the Node.js
process due to an unhandled error for the same reason.
Fix by making sure the stop operation watches for the server exiting and
returning early if the server already quit before getting to the send
stop signal part.
When closing down the HTTP and HTTPS servers existing connections are
not closed by Node.js, and may live on forever if they are actively in
use. A common case where this happens is if Prometheus is set up with a
short interval for collections.
To close down the active connections they need to be tracked and closed
when the server is closed, add the HttpCloser class which encapsulates
the tracking and closure process behind a simple interface and use this
to close down the HTTP and HTTPS server on the master.
Fix the validation loging for sending responses to requests not
functioning due to the message data property being passed incorrectly to
the validator.
When reducing the collection interval in Prometheus it may be necessary
to to reduce the metrics gathering timeout to ensure it's less than the
interval. Add config options to adjust the timeouts used for metrics
gathering so that these can be set accordingly.
Special case PluginError printing during startup in slave.js in the same
way it's done in master.js. This makes it so that the stack of the
error thrown by a plugin is printed instead of the stack trace of the
place it was wrapped into a PluginError.
Based on docker and github actions
Builds a docker image for each commit as well as the lua lib mod
Builds the master branch of subspace_storage for use with the tests
Tests are based on latest stable factorio branch
Fix the logic for determining which entries in the banlist changed being
incorrect and causing the banlist sync to be broken. Add unit tests for
the syncUserListsEventHandler to ensure it functions as exxpected.
Add exported metrics that map instance id and slave id to their
respective names. This allows getting the instance name of a metric by
doing a left join using query of the form 'metric * on(instance_id)
group_left(instance_name) clusterio_master_instance_mapping'.
Fix callback typo in Collector constructor jsdocs, add notice that
the collector is passed as the first argument, and use this first
arugument in the built-in collector implementations.
Add a variants array which can specify overriden properties of the
info.json for building multiple versions of the mod at the same time and
fix the additonal_files to allow specifying under what path the file is
put.
Together these two changes allow setting up the build to do all
supported versions at the same time and add specialized code for each
version of Factorio supported.
Add log query and log follow commands which allows inspection of the
combined cluster log. Past entries can be queried via the new queryLog
request. The log is stored into a single flat file for now, though the
performance of this may good enough.
Add winston based logger for master, slave, ctl, lib, and web_ui code.
Logging done from slaves and instances are sent to the master server and
tagged with the instance and/or slave it originated from. This allows
for example error messages to propagate back to the source that iniated
the action. The master server also stores the log to allow it to be
queried at a later point. For slaves a copy of the slave's log is also
stored locally.
Add a sendRcon method to BaseInstancePlugin and the slave's Instance
class. This servers two purposes, one is to reduce the repetition when
send RCON commands particularly from plugins, and the other is to
instrument the RCON commands sent with metrics about duration and size.
Move common functionality between labeled collectors out of
ValueCollector and into a new LabeledCollector class, and use this as
the basis for implementing a Histogram collector.
Fix the console view of the instance web interface losing the output
subscription after a reconnect to the master server due to not
re-subscribing to the outputs.
Under the current system there's no distinction made on the slave side
between instance config files that are present in the filesystem and
instances actually assigned to the slave. Which could lead to problems
with determing if an instance is actually assigned to the slave.
Resolve this issue by separating discovered instances with assigned
instances. Upon startup discovered instances are put into the
discoveredInstanceInfos mapping on the Slave object and all of these are
sent to the master server. The master server in turn sends assignment
requests for all the instances which are assigned to the slave. These
are put into the instanceInfos mapping of the Slave object which can
then be relied up to include only assigned instances.
Fix trying to use non-existent method on Slave class for forwading
events to the master. Also update request forward to be consistent with
event forwarding.
Add option to set the the maximum amount of commands transmitted in
parallel to the game. Increasing this value allows a greater rate of
commands to be executed.
When statistics_exporter handles getMetrics it may take a long time for
the result of command it sends to be returned if the instance's command
stream is overloaded, this in turn would cause the slave to miss its
window to reply on the getMetrics request and no metrics for the slave
would be available. Ensure statistics_exporter does not use more than 5
seconds to respond by adding a timeout to the metrics gathering.
When the slave metrics request takes more than 10 seconds the response
to the /metrics endpoint ends up being discarded by Prometheus which
defaults to a timeout of 10 seconds. Fix by ignoring slaves that take
longer than 8 seconds to respond to the metrics request.
Amend the error message shown when a message handler is missing from a
plugin with the expected name of the handler and the name of the plugin
it's missing from.
Use 45 seconds as the default heartbeat interval instead of 60. Using
60 causes connections to drop when proxied through Nginx as the default
timeout there is also 60 seconds.
Print only the error message in the console when a connection error is
encountered on the link connector. This reduces log spam on reconnect
for slaves.
Detect and colour the Script and Verbose levels. The format of the
source file for the Script level is slightly different and needs an
update to the output parsing.
Set breakpoint for the sidebar so that it hides by default when the page
width is too small to show it with the content and keep the width of the
content constant on small screens while sliding the sidebar out to
prevent it getting crushed.
Add command to generate a user token for connecting to the master server
web interface. This servers as an alternative to generating a control
config for use with clusterioctl.
Move the definition and implementation of the config command to
lib/shared_comhmands.js, this removes the need for lib config to depend
on the fs module, which is not accessible in web build.
Remove the echoing out of the test log from test/integration/server.js
used for debugging weird hang on shutdown immediately after startup that
would only happen on Travis.
Move the code for loading plugins infos from Node.js packages into a
separate module in order to break it out from lib/plugin.js which is
also used on the web interface side. This resolves a warning of using
an expression with require on the webpack build.
Fix the issue with paths resolving incorrectly for the webpack config
when running the master server with the --dev option from the project
root folder through either npx clusteriomaster or node packages/master
Add detailed instructions for setting the https port the master server
runs on, setting the external_address option and how to set up a reverse
proxy to the master server.
Mave the ignore entry for packages/master/static out of
packages/master/.gitignore to avoid npm using it and causing the web
build to be excluded from the package.
Use lib* prefix for the modules impored from the lib package to help
distinguish them more from ordinary variables. Fix up the naming
convention for files to be consistently lowercase_underscore for
JavaScript files.
Fix the path to plugin modules using the old plugins dir instead of
resolving to the actual path of the plugin, which in turn caused no
plugin modules to be loaded.
Avoid any potential for accidentally installing multiple copies of
clusterio/lib by specifiying it as a peer dependency instead of a
regular dependency.
Fix the plugin add command not storing the resolved path for relative
paths passed to it into plugin-list.json and incorrectly resolving any
path containing ./ or ../ instead of only paths starting with these.
Check if the additional file to add exists before trying to pack it.
Note that there may be other ways reading the files could break, but
this should suffice for most cases of getting it wrong.
Move the export and clusterio mod code to the slave package and fix the
path used for them. Refactor tests to use temp/tests/ as the cwd when
running integration tests in order to ensure paths to lua code are
correctly handled.
Add a separate plugin-list.json file for storing which plugins are
available and the path to pass to require in order to load it. This
allows plugins to be published and installed as npm packages. And also
replaces the old method of installing plugins by putting them in the
plugins folder.
Move the code generating the master authentication secret from the
inital value function of the master.auth_secret config entry to the
master startup. Having the authentication secret generated in the
config library poses issues with using the config library in browser.
Send the status of the instance in the repsonse to the listInstances
request and use this status in the integration tests to check if the
instance was started/stopped.
Add the statuses unknown and unassigned for when the status has not yet
been set by a slave and replace the initialized status with starting,
creating_save and exporting_data.
Add all routes the web interface is served on to the express app and
serve the index.html file on. Due to the index html not knowing which
path it's hosted under, a path to the root of the web interface is
replaced using string substitution from the master server.
Replace usage of .constructor.name with a separate property to avoid
issue with JavaScript terser mangling the class name during minification
for the web build.
Set up webpack for the master server's web interface and fix the issues
from the imported interface source files. The .jsx files in
package/master/web are bundled together into package/master/static and
served by the master server via express.
Make the config and link libraries compatible with packing for web with
webpack. This would fail due to ws not being web compatible and config
using a node function not available in the web shim for util.
Mark the "should not create the user if not instructed to" and "should
create the user if instructed to" tests as slow as these take more than
2 seconds to execute on travis.
Disable ci mode for lerna in travis environment as this requires
package-lock.json files to be commited to the repository and we do not
for the time being do this.
Split up codebase into the packages @clsterio/lib, @clusterio/master,
@clusterio/slave, and @clusterio/ctl using lerna to manage them. This
will allow deployment to use only the parts needed reducing install
size, and better separate the code into shared library, master, slave
and control.
Move the integration tests for the admin, user, and ban list sync to be
done while an instance is running in order to hit more lines in the
coverage with them.
Add integration of the server adminlist, whitelist and banlist into
clusterio. Membership is stored as booleans in the user acconuts and
changes are synchronized live to slaves which pass it on to instances
which in turn will update the running server's lists with RCON commands.
Remove the unused dependencies to bcrypt and bcrypt-promise. These were
used by the playerManager plugin to implement password based
authentication, but this plugin is likely not going to be ported to 2.0.
Turn the flat list of commands into a hierarchy of commands, converting
most required parameters into positional parameters and add plugin
loading support for clusterctl.
Add alternative method of starting an instance by loading a scenario.
Since there's no save when starting a scenario it can only be used if
the save patching is disabled.
Add factorio.enable_save_patching config entry to allow disabling the
save patching done on startup. This is for running vannila games or
scenarios that are not compatible with Clusterio.
Add ESLint as a dev dependency and configure it to match the preferred
style for the project. The config options set to warn are tentative and
may not be sensible as they haven't been fully investigated yet.
Add users, roles and permissions and rework the authentication to use
separate user and slave tokens for control and slave connections
respectively.
Add commands for listing permissions, roles and users, and creating,
editing and deleting roles and users.
Upload the created export data zip file to a new /api/upload-export
endpoint on the master server which unpacks it into the appropriate
directory on the master.
If a slave that is already connected attempts connecting the existing
connection for that slave is disconnected. The most common scenario for
this is that the existing connection is stale and the slave is
attempting to connect agan. However if the connection is stale then the
disconnection will take 15 minutes to time out and the slave will appear
unresponsive during this time. Fix this problem by setting the timeout
to 15 seconds before disconnecting the existing connection.
Fix WebSocketClientConnector not sending the send buffer queued up
between starting the connection and the master server sending the ready
signal for it.
Listen for the close event and connect again if the connection was
closed from proper disconnect or initiate a shutdown if the connection
was unexpectedly closed. Unexpected closure happens from authentication
failure and if a connection fails to reconnect before the timeout.
Implement notification to plugins for disconnect request, connect, drop,
and close. This allows plugins to properly handle slaves and the master
server shutting down without sending messages that will get lost.
Fix missing module declarations, broken lings, broken formating and
incorrect scopes in the documentation. For modules split over several
files it appears the only viable option is to use the memberof directive
on each documented member and only declare the index file as a module.
Send a mapping of the plugins the master server has loaded to slaves
vice versa. Limit the plugins that may be loaded by instances to only
those that the master server has loaded.
Fix crash when technologiy level stored on the master is greater than
the max_level for the technology in Factorio. This can happen with mods
that for example reduce the max_level from inifity to a finite level.
Fix research_sync sending out contribution events for the previous level
of a researched technology and sending finished events for the wrong
level when synchronizing technologies. Both of these issues are related
to failing to accunt for the unlocked level being one less than the
level stored when the researched property is false.
Fix log spam when the dole distribution method is used on
subspace_storage plugin. This was caused by a bogus if condition being
translated into a sensible but wrong one.
Fix integration tests using the default certificate paths instead of
having them located in the dedicated test database directory. Also
increment key size to the minimum allowed of 1024.
Add config entry for the number of bits to use in the auto generate TLS
certificate for the master server and reduce it to 512 for the
integration test. This reduces the time needed to generate the
certificate in the integration test.
Ensure that the replay is disabled after the game is patched by calling
game.disable_replay(). It's not possible to replay a patched save, and
it also stores every command sent into save, causing it to quickly grow
into an unplayably large size.
Gate the onMetrics plugin hook behind a boolean that's set only after
the Factorio server has fully started, and reset when it stops. This
prevents errors due to plugins trying to get metrics while the server
is creating a new save or stopped.
Ensure that notifyExit is called if errors occur during startInstance
and createSave request, and rework startInstance save logic to create
a new savegame if no save is present and allow specifying the save to
load.
Pull the exit handler into its own method that's invoked if the startup
fails before the Factorio server is created. Also move the metrics
clearing to the exit notification method to ensure it's also invoked.
Add option to bump patch number and override factorio_version in the
generated mod and update the serialize library to support the inventory
bar access methods of both 0.17 and 0.18.
Add web_root option to the master server and use it create URLs for
resources hosted by it. This permits hosting the master server web
interface behind reverse proxy under a dedicated path.
Add section to contributing guidelines detailing that single quoted
strings should not be used. The use of it has only resulted in the
addition of more inconsistency to the codebase.
Allow the factorio_directory specified in the slave config to contain
multiple Factorio installs and add factior.version instance option to
select which install to use for the given instance.
When using the shell for spawning the master and slave integration
processes the shell becomes the process that is returned by the spawn
command. This in turn means that the SIGINT that was supposed to be
sent to the master and slave processes got ignored by the shell instead.
Fix by spawning the node process directly instead of through a shell.
Shutdown running instances and close the connection to the master server
on slave stop. Remove call to process.exit on shutdown as it's no
longer needed.
Use SIGINT instead of the default SIGTERM when signalling to shut down
the integration master and slave subprocesses. This should let nyc
properly write the coverage report before it exits.
Spawn master, slave, and clusterctl in separate processes in the
clusterio integration tests and leave the test master and slave process
available for the whole test duration to allow plugins to use them as
well. This removes the need for reimplementing the startup logic in the
test code and makes the existing startup code covered by tests.
Replace socket.io with plain WebSocket using the ws library as it
doesn't really provide any useful tools over plain WebSocket when it
comes to implementing reliable communication that works across
reconnects. The widespread browser support for WebSocket also means
it has little value to add for the web interface over WebSocket.
Validate messages sent over links with the json schema provided from
them. This aids debugging as an exception will be thrown from the code
sending the bad content.
Catch errors thrown when constructing or initializing master pluigns and
wrap them in PluginError to ensure that they are properly reported as
plugin errors.
Catch errors and shut down the instance in question if the startup
failed. This happens if for example the Factorio server errors out on a
script during startup.
Add a library mod for wrapping the Clusterio remote API and providing
related utilities. This eases development of Clusterio related Lua code
by providing the same interface for both mods and modules and also makes
the serialize library available without having to duplicate it.
It is not possible to implement this through save patching as mods
cannot load Lua code from the level code like they can with code from
other mods.
Add library for serializing items and inventories to the clusterio
module. This will provide a common codebase for plugins like
playerManager and trainTeleports to use in order to serialize
inventories to the master server.
Remove the instructions for installing and using pm2 with clusterio, as
I don't know how to use this tool properly and can't keep the README up
to date on it.
Replace current configuration mess with a trimmed down version of the
original vision of #200. Configs are collection of config groups that
have fields that can be set. These groups allow organizing the fields
as well as opertunity for plugins to add their own groups. By default
plugins have an auto generated group with the enabled field.
Remove the remoteCommand specific config allowRemoteCommandExecution.
The plugin has been removed and with the move to master managing the
instances/slaves it makes no sense to deny remote commands.
Add startup event to the Clusterio module API which is raised at startup
of a Factorio server. This is to server as a replacement for the
on_init and on_configuration_changed events which cannot be used in
modules.
Adds clusterio_patch_number global from the patcher in order to
implement this. This is a number that's incremented every time the save
is patched.
Add option to FactorioServer.sendRcon to throw on non-empty responses
from Factorio. This is useful for detecting errors returned by commands
that should not have a response.
Change module loading to use a module.json file describing the module
and patch in all files in a directory path passed to the patcher instead
of having to pass the files names and their content. This allows the
specification of which files should be passed to the event_handler lib
and an easy implementation of loading stand alone modules.
Add an implementation of a Prometheus client. This is because the
prom-client implementation seems to have missunderstood how Prometheus
is supposed to work and doesn't allow updating collectors via
asynchronous callback.
Close active connections and http servers during master shutdown and let
node exit on its own. If this fails with an error set output streams to
blocking and report it then shutdown with process.exit.
Also add double SIGINT handling to immediately shutdown.
Change Instance class to be a subclass of Link and refactor the
communication to and establisment of instances to use a virtual
connection between the slave and the instance. This is simplifies the
implementation of plugin communication for instances.
Replace InstanceRequest with a forwardTo option to Request which does
the task of forwarding in a much cleaner way.
Change .gitignore to only ignore the config files in the top level
directory of repository. If these files exist elsewhere it's likely
either intended or an error.
The maintainers of bcrypt have in their infinite wisdom decided to stop
distributing Windows binaries. Their inner workings remain a mystery.
This is a port of 5fe98a0.
Remove the obsolete and broken statistics gather from the clusterioMod
plugin. The statisticsExporter plugin already exports all of the
statistics (and more) that was exported here.
This is a port of 3e7d4a8.
Replace the sources/targets concept with a links array in the message
definitions. Fixes the problem of the actual links the messages applies
to being ambigious when there's multiple sources and targets.
Remove connection and client subclasses of Link and put the connection
logic into a "connector" class. The connector deals with sending
and receiving messages, handling the connection and reconnection of lost
connections and giving sequence numbers to messages.
This move the logic of the actual connection away from the link
sub-classes, which in turn means the connection should be a lot easier
to mock in tests.
Merge the events and requests categories in lib/link into messages and
use the class name of the message to determine which handler to call.
Also swaps the order of the arguments for forwardInstanceRequest to be
consistent with the rest of the lib/link code and fixes up the docs for
requests and events.
Remove the unimplemented "binary" option from all plugin config files.
Setting this option to something other than "nodePackage" would
supposedly run a binary and feed it with the stdout of the factorio
proccess, but that was never implemented.
Add instance output event that is forwarded by the master server to
control connections that are subscribing to them. Add code subscribing
to the relevant instance output when dealing with instances in
clusterctl.
Check if Factorio terminated as a result of SIGKILL and throw an error
suggesting the system may be low on memory as a cause. Also print exit
code if the unexpected shutdown occured without a signal.
Rework client.js to support running multiple instances at once and be
managed remotely and implement the neccessary communication
infrastructure for this.
Due to circular dependencies on the implementation of this the master
branch is significantly broken after this.
Remove the reference to the out of repo Windows install script and list
updated commands to install Clusterio stable in the readme itself.
This is a port of e1771ec.
Fix the inventoryImports client side code never receiving the script
output by specifying fileReadDelay and thereby getting the old
behaviour. This broke with the introduction of the tail mode script
output mode which requires a newline to be written at the end of
the output.
This is a port of d810b56.
Fix the inventory imports command trying to count up the removed quick
bar slots as well as using player_main instead of character_main.
Add warning that the lua commands file containing the unminified vesion
of the command has not been kept up to date. There are multiple changes
that has been done previously so these have already diverged.
This is a port of 10414eb.
The default limit of 100kb to JSON payloads of bodyParser can be
exceeded when running modded games with researchSync. Increase the
limit to 10mb as a bandaid to fix this issue.
This is a port of 25eb5e5.
Set uploadModsToMaster to false in config.json.dist. This feature is
not necessary as the integrated mod portal in Factorio provides a better
method to synchronize mods.
This is a port of 3c5b108.
If the RCON client is currently in the process of connecting to the
Factorio server then wait for it to complete the connection to avoid it
emmiting out an error connecting in FactorioServer.stop().
Move item database, tests, endpoint definitions for /api/place,
/api/remove, /api/inventory, /api/inventoryAsObject, and the web
interface view for the storage page into the clusterioMod plugin.
Remove the .js suffix from help texts of node invocations as well as
actual invocations as this is not necessary. Removing it opens up the
possibility of restructuring the code into a directory with the same
name.
Remove the unused and badly behaved getCommand and getLua functions from
clusterTools. The main issue with these is that encourage the use of
large commands that have negative impact on the performance of the
servers, as such large commands have to be transmitted to all players
that are connected. These functions also have poor handling of lua
comments and will fail on strings containing for example "--".
Remove Hotpatch and implement a save patcher in its place. The save
patcher takes a normal Factorio save and adds modules to it that's
loaded by a generated control.lua file. This greatly reduces the
difficulty of adding an existing game to Clusterio and using scenario
code in plugins.
The mechanism for selecting what code to patch into the saves from
plugins is temorary and expected to change with #220.
Add a hashStream method to lib/hash that does the same as hashFile only
with a stream instead of a file path and implement tests for both of
these methods.
Add entry to changelog detailing the change to null by default entries
for the ports and password in the instance config, and update the
comment in it to reflect the recent changes.
Move getFactorioLocale to the new lib/factorio namespace and refactor it
to use async functions instead of callbacks. Split tests into
integration with factorio and testing of the function itself.
Also removes the cacheing from the function itself as this breaks the
isolated testing of it.
Break out the server process management to a new FactorioServer class in
lib/factorio. This class takes responsibility for managing a factorio
server, communicating with it and handling the stopping of it. It also
replaces the previous effort to do this in lib/factorioServer.
This change also updates the version of rcon-client, and since the
author of it has updated it to use UTF-8 the fork is no longer
necessary. It also removes the messageInterface implementation
in favor of just calling the rcon-client library directly since
this has rate limiting logic built in.
The lib/factorio library is intended to abstract all the interfacing
that happens with Factorio.
Create a random instance ID using Math.random instead of deriving it
from the RCON password with a hombrew hash algorithm. Remove
stringUtils as it's no longer needed.
Simplify getNewestFile by converting it to an async function, use the
fs-extra promise based fs functions and remove the files argument as
well as the callback.
Refactors the tests as well to use the test/temp folder.
Add symlinkMods function which takes an instance and a shared mods
folder as input and updates or creates the instance mods folder from it
by symlinking the mods in the shared mods folder to the instance mods
folder.
On Windows this uses hard links as the symlinks require elevated
privileges.
Change back the instructions in README.md to install the master branch
of clusterio and add warnings for this in the relevant locations
directing users to the stable branch for instructions on how to install
the stable version.
Reorder the install pm2 step to be after the npm install step in the
instructions. This is necessary to avoid npm creating some files in the
home directory owned as root and then failing the npm install step
because of this.
This is a port of c25bf23.
Check that the relevant research is present in the research mapping
before accessing it in recount_cluster_research_progress.
This is a port of 0cdb5d0.
Fix the next level of an infinite research getting updated with the
progress of the previous level from other nodes in the cluster by
filtering out infinite researches with a lower level than the local
level.
This is a port of 075d54f.
Set the progress for the next level of an infinite research when
unlocking it via a script. This is necessary when the research is
actively being researched as Factorio will carry over the progress for
the current level into the next one when set via a script.
It's possible for the new progress to be non-zero if the next level was
on the research queue.
This is a port of 62f3a99.
Fix naive checks for properties being set of the form obj[key] into the
proper form Object.hasOwnProperty.call(obj, key). The reason for this
is that if the key has a value of a built-in property like 'constructor'
then a check of the form obj[key] will always return true even if that
property hasn't been set on the object.
This is a port of 6a6274e.
Sending ^C via the terminal on Linux may cause SIGINT to be sent twice
to the factorio process, causing it to exit without completing the
saving. This happens because the shell sends the interupt to all
members of the process group for the job and this includes both the
node client process and the factorio server process.
Avoid this situation by passing detached to spawn when the platform is
Linux, which causes the process to be created with its own process group
and therefore not targeted by a SIGINT from the shell.
This unfortunately does not work on Windows as we can't send SIGINT or
^C to a process there. Doing subprocces.kill on Windows terminates the
process similarly to how SIGKILL works on Linux.
This is a port of 6b090cd.
Move the ejs templates to the default Express.js views/ folder and
change their extension to .ejs. Reconfigure Express.js to look for
templates in the views folders as well as the plugins folder. This
enables plugins to render templates using paths of following format
"pluginName/path/to/page-or-view".
Add missing.js test files that requires any files that currently does
not have any unit tests defined for them. This should hopefully make it
so that when unit tests are added for parts of these files the coverage
will increase instead of decreasing.
Note that this does not include CLI Tools/ as those are not testable.
Remove getCommand, getInstanceName, getSafeLua, and
checkHotpatchInstallation definitions from plugins and use the
implementation in lib/clusterTools for these functions instead.
Remove lib/manager/configManager.js and the config management command
line interface as well as the serverManager's web interface for config
management. The configuration rewrite will work differently and is not
compatible with configManager.js.
After 651b09b the instance ID would be treated differently depending on
wheteher it was a string or a number in the master server. This caused
researchSync to break as it would send the ID as a number while the
master server would have it stored as a string.
Fix by coercing instanceID to a string in the master HTTP API.
Replace the instancedirectory variable with an Instance class that holds
the directory and has a .path() to get paths relative to the instance
directory.
Authenticate connections made to the socket.io server via a token query
parameter that's validated in a similar manner to the x-access-token for
HTTP requests.
This is a port of 0c45e57.
Move the various actions performed by client.js into separate async
functions and do the command action logic if client.js is invoked as a
module directly.
Merge the masterIP and masterPort configuration entries into a single
masterURL entry. Fixes the problem with connecting to an HTTPS version
of the master server and also makes it easier to host the master server
behind a reverse proxy.
This breaks all known code interfacing Clusterio.
Remove the compiled output of veladon's clusterioModel project. The
source code can be found at https://github.com/veladon/clusterioModel
and doesn't really belong in Clusterio as it's specific to the train
teleports plugin.
Remove the explicit ignoring of the exit status of the test scripts.
This makes the Travis CI build be considered a failure if one of the
tests fail. Also rewrite the fast-test script to use the test script.
Change the ugly global slaves object to be a member of the db object and
implement the items database as a separate integrety verifying class in
lib/database.
Add missing attach calls to the master socket.io object that was
accidentally removed by the refactoring in a3fc1dd. Unfortunately I've
been unable to determine if socket.io actually supports listening on
multiple servers at the same time, this needs further investigation.
Add StartupError that differentiates unhandled errors from errors with
known causes and use it to only print the "please report" banner when an
unhandled error occurs.
Also makes the error reporting more visible in the console.
Remove remoteMap specific code left behind after the removal of the
remoteMap plugin in 6415784. The plugin interface should be versatile
enough to allow all of this code to reside inside the plugin itself.
Fix the incorrect error validation, be more lenient about empty values
and convert the checks to use async functions instead of error prone
callbacks.
Remove excessive getFactorioLocale checks in master.spec.
Remove code pushing metrics to a hardcoded Prometheus pushgateway that
is no longer in operation. There's no point supporting the use of a
pushgateway as there's already a full client implemented and since the
master server is not ephemeral or batch job it shouldn't be using a
pushgateway in the first place.
GitHub requests that the user agent be set to either the name of the
user or the application[1]. Set user agent to the name of this
application as they request.
[1] https://developer.github.com/v3/#user-agent-required
Rewrite the generateSSLcert lib to be async, use the sslCert and
sslPrivKey config entries and invoke it only if needed during master
server startup instead of in the post install script. Also add note to
the changelog covering both this and the fix in c645fac.
Update old dependencies. Fix change in mocha causing it to not exist
afer the tests are done and a change in the command parsing causing the
tests to no longer be detected.
Update instructions, Dockerfile and Travis CI to use Node.js 10.
Specify that Node.js version 10 or greater is needed in the engines
section of package.json (note that this is only advisory.) And document
this as a breaking change in CHANGELOG.md.
Do not install git, tar and curl as these packages are installed by
default on Travis CI. Move the codecov upload to the travis script and
do not include the codecov token as it's not needed when uploaded from
Travis CI. Make the cover npm script into something that can be runned
locally and move the Travis CI version to a new ci-cover script. Remove
the verbose flag from the tar extration of the factorio headless server.
Add config.json.test as the the config for running the tests on travis
and add script command to copy it to config.json there. Should fix the
unit test for master failing at loading the config.
Move server initialization code to startServer and call it only when
master.js is being runned directly. Also refactor the server listen
step to work properly when https is configured and only listen after
plugins have completed loading.
Remove the async function wrapper around the master.js code. This is
just dumb and makes the code untestable.
Ensure exceptions are properly passed down to the unit test framework
with promises, check the return status of requests, use config to
generate an access token, reduce code duplication and reformat to be
consistent.
The masterPlugins array should not be a global. The ejs templates
already have it provided through locals.masterPlugins and plugins get it
through onLoadFinish. Make masterPlugins local to master.js again and
fix the ejs templates to use locals.masterPlugins.
2019-08-21 12:43:09 +02:00
4257 changed files with 56535 additions and 14337 deletions
# This workflow will do a clean install of node dependencies, cache/restore them, build the source code and run tests across different versions of node
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-nodejs-with-github-actions
name:Node.js CI
on:
push:
pull_request:
jobs:
test:
runs-on:ubuntu-latest
strategy:
matrix:
node-version:[18.x, 20.x]
# See supported Node.js release schedule at https://nodejs.org/en/about/releases/
- Disabled uploading mods to the master server by default as this is mostly
just a waste of bandwidth with the mod portal being integrated into the
game now.
- Fixed researchSync breaking with heavily modded games due to the tech tree
exceeding the default 100kb limit on JSON payloads.
-Fixed the command inventoryImport sends referencing player instead of
character, and trying to count the non-existant quick bar slots.
-Fixed inventoryImport never receiving the script output after the default
mode for reading script output changed to tail mode.
- Fixed desync caused by the mods and loaded mods arrays in Hotpatch getting
out of sync when plugins update their scenario mod code. To fix existing
games you will need copy lib/scenarios/Hotpatch/hotpatch/mod-tools.lua
over the existing mod-tools.lua in the save's hotpatch folder.
This release does not retain backwards compatibility with 2.0.0-alpha.13 meaning all parts of the cluster needs to updated at the same time.
See the [migration guide](/docs/migrating-from-2.0.0-alpha.13.md) for how to update.
### Major Features
-Added mod management. Mods can be uploaded to the controller and included in mod packs that can be assigned to instances and run.
- Added host port ranges which the instance ports will be automatically assigned a port from if factorio.game_port is not set.
-Added system info such as CPU model, hostname and kernel along with basic metrics for CPU usage, memory usage and disk usage for the controller and hosts.
- Added ability to remotely view and modify the configuration of hosts.
### Features
- Added logging of Clusterio version on startup
- Added `clusterio_controller_log_bytes` metric counting the size of the log files on the controller.
- Added warning logged when plugins on a connecting host does not match the ones installed on the controller.
- Added ability to revoke user tokens.
- Added restart required indicator to config fields in the Web UI.
- Added ability to revoke host tokens.
- Added support for plugins to run in the context of a host.
- Added more feedback when the web UI and or its plugins fails to load.
- Added autosaving of data held in memory on the controller such as configs, instances, users, mods to prevent large data loss on crashing.
- Added `system_info` metric with kernel, instruction set, CPU Model, hostname and Node.js version of the controller.
- Added `clusterio_host_system_info` metric with kernel, instruction set, CPU Model, hostname and Node.js version of hosts.
- Added `system_cpu_seconds_total`, `system_cpu_idle_seconds_total`, `system_memory_capacity_bytes`, `system_memory_available_bytes`, `system_disk_capacity_bytes`, `system_disk_available_bytes` metrics for CPU, memory and disk usage on the controller.
- Added `clusterio_host_system_cpu_seconds_total`, `clusterio_host_system_cpu_idle_seconds_total`, `clusterio_host_system_memory_capacity_bytes`, `clusterio_host_system_memory_available_bytes`, `clusterio_host_system_disk_capacity_bytes`, `clusterio_host_system_disk_available_bytes` metrics for CPU, memory and disk usage on hosts.
- Added support for reading the client's IP address from the `X-Forwarded-For` header when the source address is in the `controller.trusted_proxies` config option.
- Added field showing the IP address a host is connecting from.
- Added per player statistic tracking when they were first seen joining a particular server.
- Added ability to remotely restart and/or stop the controller and hosts.
### Changes
- Fixed filenames with extensions that are invalid on Windows such as "con.zip" not being treated as invalid.
- Fixed generate host token modal not using a random id when the optional id field is unset.
- Exported Factorio data is now tied to mod packs instead of being per cluster.
- Fixed export failing if a mod uses a different root folder mod_version in its zip file.
- Fixed sorting of user last seen column in the Web UI.
- The list users permission is now granted by default to players.
- Fixed incorrect handling of hosts and instance with an id of 0 causing them to not update properly.
- Fixed unable to unassign instances in the Web UI.
- Fixed host crashing if the Factorio server is not marked executable.
- Fixed errors logged from instance not being attributed to the instance in certain situations.
- Hosts no longer print messages from instances into their console.
- Clusterio now treat unhandled Node.js errors and promise rejections as fatal and exit immediately.
- Fixed a potential for data loss after a hard system crash due to fsync not being called on files written.
- Fixed Factorio server crashing in the research_sync plugin when a mod enables a previously disabled technology and a player attempts to research it.
- Fixed Factorio server crashing when the game starts with tick_paused = true
- Removed migration of alpha 10's single file log format.
- Log errors linking mods on Windows instead of preventing startup.
- Added locale support for modules patched into to the game save.
- Added support for defining a module_exports.lua file for modules patched into the game save.
- Fixed controller logs being split on server's local time day boundary instead of UTC day boundary.
- Fixed Unicode BOM in mod locale files breaking instance data export.
- Added a warning to the server-setting.json that is automatically written to instance folders stating changes will be overwritten.
- Changed save patching to use a `.tmp.zip` extension for the temporary file created instead of `zip.tmp`.
- Fixed Inventory Sync wiping the player's inventory when logging out while in the Space Exploration remote view mode.
- Fixed the public /api/plugins endponit leaking filesystem paths on the controller.
- Refactored list subscriptions in the Web UI to cache and only fetch updated entries when navigating to previously seen list.
- Added caching of save lists on the controller, enabling the saves of instances on offline hosts to still be seen.
- Hosts in the assign instance dropdown is now sorted by name.
- Added `non_blocking_saving` as one of the default settings added to `factorio.settings` on new instances.
### Breaking Changes
- Renamed usage of the terms master and slave to controller and host in all places they occur.
- Mods located in `sharedMods` are no longer copied to instances' mods folder.
- The instances' mods folder is now managed through mod packs, mods not part of the mod pack will be automatically deleted.
- Removed ability to import sub packages like `@clusterio/lib/link` from.
- Replaced `clusterio_controller_http_hits_total` metric with `clusterio_controller_http_endpoint_duration_seconds`.
- Replaced `clusterio_controller_last_query_log_duration_seconds` metric with `clusterio_controller_query_log_duration_seconds` summary.
- Reworked the network protocol to be simpler to work and reason with.
- Bumped minimum supported Node.js version to v18.
- Ported the codebase to TypeScript
- Changed the config format stored on disk to be much simpler.
### Contributors
Many thanks to the following for contributing code in this release:
Removed the Hotpatch scenario and the depency on it for getting code into the game.
Added a save patcher than runs before starting up Factorio that patches in lua modules based on the event_loader lib into the savegame.
Regular freeplay games can now be used with Clusterio and will be compatible without having to convert them to Hotpatch.
- Daemonized hosts.
Hosts now have the ability to run multiple Factorio instances, starting and stopping them individually.
To manage this the local command line interface has been replaced with a remote interface on the controller that can be accessed through the clusterioctl cli tool.
- Rewritten the communication between hosts and the controller from scratch.
The new system is based on a WebSocket connection between the hosts and the controller and provides efficient validated bi-directonal communication.
- Rewritten the plugin system from scratch.
Plugins now inherit from a base class and use the same WebSocket connection Clusterio uses to communicate.
- New configuration system with support for initializing configs when needed, modifying config entries using built-in commands, and updating instance configuration remotely.
- Packaged and uploaded to npm again, greatly simplifying distribution and installation of both Clusterio and plugins.
- Added centralized logging for the cluster.
All logs from both Clusterio and Factorio is stored in a shared log which can be inspected and queried.
- Added remote save management with the ability to list, create, rename, copy, upload, download, transfer, delete and load saves for instances.
### Features
- Added export of pollution statitics.
- Reconnection logic that esures no data is dropped talking to the controller provided the session can be resumed.
- Added support for having multiple Factorio installs and selecting which version to use on a per instance basis.
- Added config to specify which path the controller interface is accessed under, allowing it to be proxied behind web-server.
- Added WebSocket usage statistics.
- Added commands sent statistic.
- Added Factorio server CPU and memory usage statistic.
- Added Factorio autosave size statistic.
- Added support for extracting locale and item icons from Factorio and mods.
- Added integrated user management for controlling access and storing per player data.
- Added synchronization of in-game adminlist, banlist and whitelist to Clusterio core.
Previously this was handled by the playerManager plugin.
- Added stripping of long paths in the Factorio server log.
- Added option to configure maximum number of commands to send in parallel.
- Added option to auto start instances on host startup.
- Added metric mapping of host and instance ids to their names, allowing them to be displayed by name in queries that join with the mapping.
- Added option to configure timeout for exported Prometheus metrics.
- Added option to set address to bind HTTP(S) port to.
- Added an overview of the installed plugins to the web interface.
- Added clusterioctl command to list plugins on the controller.
- Added Player Auth plugin for logging in to the web interface by proving the ability to log in to a server as a given user.
- Added Node.js based installer to simplify setting up a cluster.
### Changes
- Added error handling during controller startup.
- Factorio game and rcon port now defaults to a random port above 49151.
- Removed unimplemented mods update command.
- Fixed rcon password being generated with Math.random().
- Added plugins directory to the views path.
This makes it possible for plugins to render their own ejs views or pages in their own folders by using paths of the format "pluginName/path/to/page-or-view".
- Replaced the per instance copy of the shared Factorio mods with symlinks.
On Windows hard links are used instead due to the privileges requirements of symlinks.
- Changed the per instance scenario folder to be linked instead of copied on instance creation.
- Instance id is no longer derived from the rcon password, instead it's generated upon instance creation and stored in the instance config.
- The game port, RCON port and RCON password instance config entries are are now null by default, indicating that a random one will be generated every time the instance is started.
- Removed the broken client download command as this would download pre-releases.
- Moved the item database and HTTP definitions for /api/place, /api/remove, /api/inventory, /api/inventoryAsObject and the web interface view for the storage page into the clusterioMod plugin.
If you disable this plugin then these things will not be available.
- Removed undocumented --port and --rcon-port arguments to client.
- Removed undocumented FACTORIOPORT and RCONPORT enviroment variable handling in client.
- Removed redundant call to /api/slaves in globalChat plugin.
- Removed broken serverManager plugin.
- Removed factorio_version from config.
The version installed is auto detected and used instead.
- Removed the playerManager specific command CLI tools/delete_player.js.
- Creating an instance, assigning it to a host, creating a save for an instance and starting an instance is now four separate commands.
- Removed oddball limits to hosts.json size.
- Moved host specific and instance specific configuration into their own configuration files.
- Removed unused binary option from plugin config.
- Removed info and shout command from globalChat plugin - Removed mirrorAllChat and enableCrossServerShout configuration options for globalChat plugin.
- Removed UPSdisplay plugin.
UPS statistics is exported by the statistics exporter plugin.
- Controller now defaults to hosting on https on port 8443.
- Renamed client to slave.
- Removed rotation of factorio-current.log files.
### Breaking Changes
- Removed lib/authenticate.
Breaks playerManager.
- The masterIP and masterPort config entries has been merged into masterURL.
Breaks discordChat, playerManager, serverSelect, trainTeleports and clusterioModel.
- Removed config management from the command line and the server manager.
- Moved ejs templates into views folder and changed their extension to .ejs.
Breaks playerManager.
- Mods are no longer copied from the per instance instanceMods directory.
If you need per instance mods you can now place them directly in the mods directory inside the instance folder.
- Hotpatch scenarios and code loading is no longer compatible with Clusterio.
Breaks playerManager, serverSelect, and tranTeleports.
- Removed getLua and getCommand from lib/clusterTools.
If you need to run more than the most trivial of code in commands use the save patcher and add in a remote interface.
- Removed mod uploading and distributing from the HTTP interface.
Breaks the old unmaintained and no longer needed factorioClusterioClient.
- Removed the remoteCommands plugin and the old runCommand interface.
Breaks playerManager and external tools sending commands.
- Removed broken serverManager plugin.
- Removed fields info, time, rconPort, rconPassword, serverPort, unique, mods, instanceName, playerCount, mac, and meta from the hosts in the host database.
- Removed getInstanceName and lib/clusterTools.
Breaks playerManager, and discordChat.
- Removed the /api/rconPasswords, and /api/slaves endpoints.
Breaks web interface, trainTeleports, and discordChat.
- Removed the /api/getSlaveMeta and /api/editSlaveMeta endpoints.
Breaks researchSync, and UPSdisplay.
- Removed the hello event from the socket connection handshake.
Breaks playerManager, trainTeleports, serverSelect, and discordChat
- Changed the format of database/hosts.json.
- Removed the output file subscription system.
Breaks inventoryImports, playerManager, trainTeleports, serverSelect and researchSync.
- Removed the factorioOutput hook from instance plugins.
The onOutput hook provides parsed output instead.
- Removed the onLoadFinish hook from master plugins.
- Moved plugins from the sharedPlugins directory to plugins directory.
- Implemented a new plugin system that replaces the old.
- Removed clusterio_connected_instaces_gauge and added clusterio_master_connected_clients_count in its place.
- Renamed clusterio_player_count_gauge to clusterio_statistics_exporter_instance_player_count
- Removed clusterio_UPS_gauge and added clusterio_statistics_exporter_instance_game_ticks_total in its place.
- Renamed clusterio_endpoint_hit_gauge to clusterio_http_enpoint_hits_total
- Renamed clusterio_statistics_gauge to clusterio_statistics_exporter_instance_force_flows
- Renamed clusterio_nn_dole_gauge to clusterio_subspace_storage_nn_dole_gauge.
- Renamed clusterio_dole_factor_gauge to clusterio_subspace_storage_dole_factor_gauge.
- Renamed clusterio_import_gauge to clusterio_subspace_storage_import_total.
- Renamed clusterio_export_gauge to clusterio_subspace_storage_export_total.
- Renamed clusterio_master_inventory_gauge to clusterio_subspace_storage_master_inventory.
- Removed clusterioMod plugin specific config options logItemTransfers, disableFairItemDistribution useNeuralNetDoleDivider, autosaveInterval, and disableImportsOfEverythingExceptElectricity from the master config.
- Removed msBetweenCommands config option.
The RCON is instead limited to 5 concurrent commands.
Remote commands are always allowed with the move to controller managing hosts/instances.
- Removed `--databaseDirectory`, `--controllerPort`, and `--sslPort` command line arguments from the controller.
- Implemented a new config system that replaces the old.
Breaks all plugins.
- Removed usage of socket.io entirely in favor of a plain WebSocket connection.
- Renamed clusterioMod plugin to subspace_storage.
- Controller no longer creates secret-api-token.txt.
- Removed automatic creation of self-signed TLS certificate.
- Node.js versions below 12 are no longer supported.
## Version 1.2.4
- Removed broken remote combinator signaling.
- Fixed research sync endlessly updating already researched technologies
- Removed obsolete item/fluid statistics from clusterioMod
- Removed per mod upload logging when config.uploadModsToMaster disabled
- Fixed bcrypt failing to install on windows due to the new version not having Windows binaries.
## Version 1.2.3
- Disabled uploading mods to the master server by default as this is mostly just a waste of bandwidth with the mod portal being integrated into the game now.
- Fixed researchSync breaking with heavily modded games due to the tech tree exceeding the default 100kb limit on JSON payloads.
- Fixed the command inventoryImport sends referencing player instead of character, and trying to count the non-existant quick bar slots.
- Fixed inventoryImport never receiving the script output after the default mode for reading script output changed to tail mode.
- Fixed desync caused by the mods and loaded mods arrays in Hotpatch getting out of sync when plugins update their scenario mod code.
To fix existing games you will need copy lib/scenarios/Hotpatch/hotpatch/mod-tools.lua over the existing mod-tools.lua in the save's hotpatch folder.
- Updated the Windows install instructions.
Version 1.2.2
-------------
## Version 1.2.2
- Fixed possible crash with modded technologies named the same as a built-in
Object prototype property in researchSync.
- Fixed progress of a current infinite tech carrying over to the next one
when researching it and another node completes it in researchSync.
- Fixed progress of a previous infinite tech from another node being applied
to the current one in researchSync.
-Fixed crash in researchSync when modded technologies are present only on some
nodes.
- Fixed install failing due to bcrypt version less than 3 not being supported
on node v10.
- Reordered install instructions to avoid problem with npm creating files owned
by root in the home directory.
- Swapped curl out with wget in the install instructions as the latter comes
pre-installed on Ubuntu.
- Fixed possible crash with modded technologies named the same as a built-in Object prototype property in researchSync.
- Fixed progress of a current infinite tech carrying over to the next one when researching it and another node completes it in researchSync.
- Fixed progress of a previous infinite tech from another node being applied to the current one in researchSync.
- Fixed crash in researchSync when modded technologies are present only on some nodes.
- Fixed install failing due to bcrypt version less than 3 not being supported on node v10.
- Reordered install instructions to avoid problem with npm creating files owned by root in the home directory.
-Swapped curl out with wget in the install instructions as the latter comes pre-installed on Ubuntu.
### Breaking Changes
- Node.js versions below 10 are no longer supported.
Version 1.2.1
-------------
## Version 1.2.1
- Updated node-factorio-api to v0.3.8 to fix mod downloads randomly breaking
([#229][#229].)
- Fixed SIGINT being sent twice to Factorio server when interrupted by CTRL+C
on Linux ([#217][#217].)
- Updated node-factorio-api to v0.3.8 to fix mod downloads randomly breaking ([#229][#229]).
- Fixed SIGINT being sent twice to Factorio server when interrupted by CTRL+C on Linux ([#217][#217]).
- Fixed package.json incorrectly reporting the license as ISC.
letkillBiters=`/silent-command local surface=game.surfaces[1] for key, entity in pairs(surface.find_entities_filtered({force="enemy"})) do entity.destroy() end`;
* [Setting up shared storage](#setting-up-shared-storage)
* [Running Clusterio](#running-clusterio)
* [Setting up remote hosts](#setting-up-remote-hosts)
* [Setting up clusterioctl](#setting-up-clusterioctl)
* [Common problems](#Common-problems)
* [Command cheatsheet](#cheatsheet)
## Introduction
Features:
Clusterio is a clustered Factorio server manager that provides the tooling for implementing cross server interactions in Factorio.
It was previously best known for implementing cross server transfer and cloud storage of items via teleporter chests.
But this functionality has been pulled out of Clusterio into its own plugin for Clusterio named [Subspace Storage](https://github.com/clusterio/subspace_storage).
- Entities to send/recieve items
- Cross dimensional storage
- Sending of liquids
- Sending of circuit network signals
- Inventory combinator to display item levels in the cluster (and epoch time)
- Reporting of graphs and UPS on master interface (Also has extensive Prometheus reporting)
Optional extras (see [Plugins](#Plugins))
- Have your inventory synchronize across servers
- Teleport trans from the border of one world to te next
- Show in-game chat in discord
By itself Clusterio doesn't change the gameplay in any way, you could even use Clusterio to manage completely vanilla Factorio servers.
Plugins do the work of modding in the visible changes into the game, see the [Plugins section](#plugins) for ready-made plugins you can install into a Clusterio cluster.
- Clustered Factorio server management allowing you manage the running of Factorio servers across a fleet of physical servers from both a web interface and a command line interface.
There can be any number of clients connected to each slave, and any number of slaves connected to a master but there can only be one master server in each cluster.
- User list management for synchronizing in-game admins, whitelisted users, and bans to all the servers in the cluster.
**How does it work?**
- Integrated support for exporting statistics for the whole cluster to Prometheus via a single metrics endpoint.
Traditional factorio mods have always been limited by the games deterministic design. This gives us a very bug free and predictable game, but doesn't allow us cool stuff such as internet communication.
Clusterio exploits one of the games logging features, game.write_file and RCON to communicate between servers. Sending an item from one server to another takes this path:
1. server1: Chest has stuff in it, write the contents to a file and delete them from the game world
2. client.js on server1: File has stuff in it, parse and send to the master for storage
3. master: server1 gave us stuff, store it in the storage and write some statistics
4. server2: get-chest is empty, write a request to file
5. client.js on server2: Request file has stuff in it, parse and send a request to master for more items of that type
6. master: server2 asked for stuff, check if we have enough and how much demand there is, then send however much is appropriate back
7. client.js on server2: We were allowed to import x of item y, run command /c remote.call("clusterio", "importMany", "{'copper-plate':120}")
This process works the same for both items and liquids, independent on what mods are used. Yes, modded items are fully supported.
Clusterio can also do a few other neat things, such as giving you access to epoch time, syncing player inventories between servers, keeping track of playtime (playerManager plugin), teleporting trains between servers (trainTeleports) and exporting tons of factorio related statistics to Prometheus for graphing in grafana.
## Ubuntu setup
NodeJS does not support EOL ubuntu releases. Make sure you are on the most recent LTS release or newer.
downloads and installs nodejs, git and clusterio. To specify a version, change "latest" in the link to a version number like 0.14.21.
Optional step (if you want to use pm2):
sudo npm install pm2 -g
Now you need to edit the `config.json` file. If you skip this step nothing will work.
Pretty much all the blank fields should be filled in, except on the master where a few can be omitted.
* You get the `masterAuthToken` from `secret-api-token.txt` in the master install dir after running the master twice.
* You get your factorio matchmaking token from factorio.com
* The `masterAuthSecret` should never be touched unless you want to invalidate everyones authentication tokens
**Master**
node master.js
Or with pm2 (it's recommened to run it without pm2 first):
pm2 start master.js --name master
- Extensive plugin support for adding your own cross server features to Factorio using Clusterio's communication backbone.
**Server Host**
## Plugins
To download the mod for all its non vanilla features and items, (optional, but very recommended)
The heart of Clusterio is its plugin system.
Plugins add functionality to Factorio servers, Clusterio itself or both.
These are the plugins supported and maintained by the Clusterio developers:
node client.js manage shared mods add clusterio
- [Global Chat](/plugins/global_chat/README.md): share the in-game chat between servers.
- [Research Sync](/plugins/research_sync/README.md): synchronize research progress and technologies unlocked between servers.
- [Statistics Exporter](/plugins/statistics_exporter/README.md): collect in-game statistics from all the servers and makes it available to the Prometheus endpoint on the controller.
- [Subspace Storage](https://github.com/clusterio/subspace_storage): Provide shared storage that can transport items between servers via teleport chests.
- [Player Auth](/plugins/player_auth/README.md): Provides authentication to the cluster via logging into a Factorio server.
- [Inventory Sync](/plugins/inventory_sync/README.md): Synchronizes your inventory between servers.
To create a new instance (its own save, set of mods and config files)
There's also plugins developed and maintained by the community:
node client.js start [instancename]
- [Discord Bridge](https://github.com/Hornwitser/discord_bridge) (@hornwitser/discord_bridge): Bridges chat between instances and Discord.
By Hornwitser.
- [Server Select](https://github.com/Hornwitser/server_select/tree/clusterio-2.0) (@hornwitser/server_select): In-game GUI for connecting to other servers in the cluster.
Originally by Godmave, ported to 2.0 by Hornwitser.
To launch an instance with pm2
Want to make your own plugin?
Check out the documentation on [Writing Plugins](/docs/writing-plugins.md) for where to start.
The -v flag is used to specify the instance directory. Your instances (save files etc) will be stored there.
The -v flag is used to specify the instance directory.
Your instances (save files etc) will be stored there.
-->
## Windows setup
### Windows setup
Clusterio is built up of multiple parts. Here is a quick guide:
1. Download and install the latest LTS release from http://nodejs.org.
Master = master.js
2. Create a new empty directory for the installation and navigate into it.
Open a PowerShell window in this new directory by Shift+Right clicking inside it and choosing "Open PowerShell window here" and then run the following command.
Server host (Slave) = client.js + factorio server
npm init "@clusterio"
Game Client = The people connecting to the server
Make sure to note down the admin authentication token it provides at the end as you will need it later.
**Requirements**
download and install nodeJS 10 from http://nodejs.org
download and install git from https://git-scm.com/
reboot when you are done, then proceed to the next steps. *reboots matter*
**Master**
1. Open PowerShell or Command prompt in the directory you want to install to and run the following commands.
2. Obtain Factorio by either of these two methods:
- Via the stand alone version on from their website
1. Download the MS Windows (64-bit zip package) from https://www.factorio.com/download
2. Open the zip file and drag the folder called "Factorio_0.17.x" into the factorioClusterio folder
3. Rename the folder to "factorio"
- Via steam installation
1. Locate the game files by right clicking the game in steam, selecting properties, then Local Files, then Browse local files.
2. Go to the parent folder of the folder that Steam opened and copy the Factorio folder into the factorioClusterio folder
3. Rename the folder to "factorio"
3. If you chose to use local factorio directory for the Factorio installation, download the Windows 64-bit zip package from https://www.factorio.com/download and extract it to the `factorio` folder in your clusterio installation folder.
3. Open `config.json` with a text editor and configure as desired.
### MacOS Setup
4. Run `node master.js` to generate the athentication token into secret-api-token.txt
1. Install the latest Node.js LTS release from http://nodejs.org or use brew (`brew install node`).
5. Run `node master.js` again to start the master server.
2. Open Terminal or Command prompt in the directory you want to install to and run the following commands.
**Server Host**
mkdir clusterio
cd clusterio
npm init "@clusterio"
1. Do step 1 and 2 of the Master section above *OR* use the same folder that was created in that section.
Make sure to note down the admin authentication token it provides at the end as you will need it later.
2. Open `config.json` with a text editor and configure as desired. You will need to set masterAuthToken to string found in secret-api-token.txt on the master server.
3. If you chose to use local factorio directory for the Factorio installation, you will need to obtain and copy a mac version of Factorio and unpack it to the `factorio` folder in your clusterio installation folder.
3. Optionally run the command `node client.js manage shared mods add clusterio` to add the clusterio mod (needed for item teleports.)
4. Run `node client.js start [instancename]` to create a new instance. Repeat it again to start the instance.
### Installing Plugins
To connect to a master server running on a remote machine, open config.json with your favourite text editor (notepad++). You can also set it up to use the official server browser.
For well-known plugins you can select them during installation and no further steps are necessary to make them work.
Installing plugins that are not offered by the installer consists of two steps.
First install the package the plugin is provided by via npm, for example
Change `masterIP `to something like `31.152.123.14` (provided by master server owner)
npm install @clusterio/plugin-subspace_storage
Change `masterPort` to something like `8080` (provided by master server owner)
Then tell clusterio that this plugin exists by adding it as a plugin.
Change `masterAuthToken` to the value found in `secret-api-token.txt` on the master server
Repeat step 4 for more servers on one machine. You should be able to find its port by looking at the slave section on master:8080 (the web interface)
This adds it to the default `plugin-list.json` file which in the shared folder setup is loaded by controller, host and ctl.
If you have hosts or ctl installed on separate computers (or directories) then you need to repeat the plugin install process for all of them.
The clusteriocontroller, clusteriohost and clusterioctl commands has the plugin sub-command so you do not need to install clusteriocontroller to add plugins, instead use the clusterio command you have available.
**GameClient**
For development purposes the `plugin add` command supports adding plugins by the absolute path to them, or a relative path that must start with either . or .. (which will then be resolved to an absolute path).
Fancy game client that does the following steps automatically, but is really old so be warned: [clusterioClient](https://github.com/Danielv123/factorioClusterioClient)
1. Download the same version of the mod as the slave is running from [the mod portal](https://mods.factorio.com/mods/Danielv123/clusterio) or [github](https://github.com/Danielv123/factorioClusterioMod
## Configure Controller
2. Drop it into ./factorio/mods
By default the controller will listen for HTTP on port 8080.
You can change the port used with the command
3. Run factorio and connect to slave as a normal MP game. You will find the port number to connect to at http://[masterAddress]:8080
npx clusteriocontroller config set controller.http_port 1234
When changing the port you will also need to change the address hosts connect with.
For the standalone installation mode you can use
npx clusteriohost config set host.controller_url http://localhost:1234/
If you plan to make your cluster available externally set the address
that it will be accessible under with, for example
npx clusteriocontroller config set controller.external_address http://203.0.113.4:1234/
Change the url to reflect the IP, protocol, and port the controller is accessible under, dns names are also supported.
If you're planning on making the controller accessible on the internet it's recommended to set up TLS, see the [Setting Up TLS](/docs/setting-up-tls.md) document for more details.
You can list the config of the controller with the `npx clusteriocontroller config list` command.
See the [readme for @clusterio/controller](/packages/controller/README.md) for more information.
## Managing Factorio mods
Clusterio currently only provides a manual method for running mods on servers by copying them either to each instance's individual `mods` folder, or by copying them into a host installation's `sharedMods` folder which will link/copy them to the `mods` folder of each instance on that host if the file doesn't already exist in the `mods` folder for that instance.
Mods are copied from `sharedMods` on Windows and symlinked on Linux, this means that when mods are removed from `sharedMods` the mods will also be removed from the instance's `mods` folder on Linux if it got there by being linked by Clusterio, but not on Windows.
For managing the mod settings, the `mod-settings.dat` and `mod-list.json` files will also be linked/copied over if present in `sharedMods`, but since Factorio creates these files automatically it might be necessary to delete them from each instance's `mods` folder for changes to them in `sharedMods` to take effect.
Better remote management of mods is a planned feature.
## Setting up shared storage
The chests teleporting items to and from the shared storage on the controller has been moved into the Subspace Storage mod and plugin.
To get this to work you will need to install the Subspace Storage plugin and copy the [Subspace Storage mod](https://mods.factorio.com/mod/subspace_storage) and the [Clusterio Lib mod](https://mods.factorio.com/mod/clusterio_lib) into the `sharedMods` folder on each host installation.
Do not install the [Clusterio mod](https://mods.factorio.com/mod/clusterio), this was for the old 1.2.x version of Clusterio and is not compatible with Clusterio 2.0.
## Running Clusterio
After completing the installation start up the controller and at least one host separately.
The installer provides the `run-controller` and `run-host` scripts to make this simple.
Once the controller process is running you can log into the web interface which is hosted by default on http://localhost:8080/ (adjust the port number if you changed it), use the admin authentication token provided from the installation to log in.
The basics of setting up a Factorio server from the web interface is to create an instance, assign it to a host and then click start.
### Running via Systemd
The install script creates systemd service scripts for clusteriocontroller and clusteriohost (if applicable) to start up as the user than ran the installer.
If you have copied these files over to `/etc/systemd/system/` then you can startup Clusterio as a background service using:
#### To start as a service run:
sudo systemctl start clusteriocontroller # for the controller
sudo systemctl start clusteriohost # for each physical server intended to host Factorio servers
#### To automatically get it to start on boot:
sudo systemctl enable clusteriocontroller # for the controller
sudo systemctl enable clusteriohost # for each physical server intended to host Factorio servers
## Setting up remote hosts
Run the installer as described in the installation section and choose "Host only" as the operating mode to install, it'll ask for a controller URL and an authentication token.
The URL is the same as what is needed to connect to the web interface, and the authentication token can be generated on the Hosts page in the web interface.
Once you start up the host it should show up in the Hosts list and be available for assigning and running instances on.
## Setting up clusterioctl
There's a command line interface available for Clusterio which is installed separately with the same installer as for the controller and host.
Run the installer as described in the installation section and choose "Ctl only" as the operating mode to install, you can do this in the same directory as you have installed other Clusterio component(s) to.
The installer will ask for a controller URL and an authentication token, these are the same as you would use to connect to the web interface.
If you want to use a different user for the command line interface, you can generate an authentication token for an existing user with
1. [Player Manager](https://github.com/Danielv123/playerManager) - Adds player management to the Web UI and shared inventory handling (beta)
2. [DiscordChat](https://github.com/jakedraddy/ClusterioDiscordChat) - Logs in-game chat/joins/leave messages on all instances to a Discord webhook.
3. [TrainTeleports](https://github.com/Godmave/clusterioTrainTeleports) - Allows you to teleport cargotrains between servers.
## Common problems
### Cannot find module: `/../../config`
Copy your config.json.dist to config.json and configure it.
### EACCESS [...] LISTEN 443
Some systems don't let non root processes listen to ports below 1000. Either run with `sudo` or change config.json to use higher port numbers.
Some systems don't let non root processes listen to ports below 1000.
Either run with `sudo` or change config.json to use higher port numbers.
According to [this link](https://askubuntu.com/questions/839520/open-port-443-for-a-node-web-app) if you manually installed node.js following the above instructions, you may need to run the following command to fix this issue:
### Portforwarding doesn't work on the master server when running under WSL
### Portforwarding doesn't work on the controller when running under WSL
If you follow the ubuntu guide on WSL (Windows Subsystem for Linux, Bash on Ubuntu on Windows specifically), you will find that the website works on localhost and on your local ip, but not on the global ip. This is also true when you correctly port-forwarded the correct ports. Even when routing this server through nginx in WSL, the issue persists. Then, on a hunch, I tried to run nginx from windows itself and found that this DID work. It came to me that the only usage difference between the 2 versions of nginx is that I got a Windows Firewall popup.
If you follow the ubuntu guide on WSL (Windows Subsystem for Linux, Bash on Ubuntu on Windows specifically), you will find that the website works on localhost and on your local ip, but not on the global ip.
This is also true when you correctly port-forwarded the correct ports.
Even when routing this server through nginx in WSL, the issue persists.
Then, on a hunch, I tried to run nginx from windows itself and found that this DID work.
It came to me that the only usage difference between the 2 versions of nginx is that I got a Windows Firewall popup.
TLDR: the tested fix is:
@@ -274,36 +295,3 @@ TLDR: the tested fix is:
- select TCP and select specific local ports and type in the ports that you want to open (comma separated) and click next > 3 times
- give the rule a name (like 'web server' or something), give it a description (optionally) and click finish
### Other fixes for other potential problems:
Sometimes the install fails. Try `node ./lib/npmPostinstall.js` to complete it.
## Cheatsheet
**To create a new instance/start it**
node client.js start [instanceName]
**Other instance management tools:**
```
node client.js delete [instanceName]
node client.js list
```
**To update clusterio to the latest version:**
1. Download the latest zip version of factorio for your platform manually from factorio.com. Place it in the project root folder and call it "factorio" (folder name is specified in config.json)
2. Grab the latest version of the repo
```
git pull
npm install --only=production
```
3. Download the latest version of the factorioClusterioMod from its github repo
fileOps.deleteFolderRecursiveSync(path.resolve(config.instanceDirectory,process.argv[3]));// TODO: Check if this can cause i-craft users to format their server by using wrong paths
console.log("Deleted instance "+process.argv[3]);
process.exit(0);
}else{
console.error("Instance not found: "+process.argv[3]);
process.exit(0);
}
}elseif(command=="download"){
console.log("Downloading mods...");
// get JSON data about releases
letres=syncRequest('GET','https://api.github.com/repos/Danielv123/factorioClusterioMod/releases',{"headers":{"User-Agent":"Fuck you for requiring user agents!"}});
// fs.symlinkSync('../../../sharedMods', instancedirectory + "/mods", 'junction') // This is broken because it can only take a file as first argument, not a folder
"__comment": "Where factorio folder. Should be in the same folder as this config file.",
"factorioDirectory": "./factorio",
"instanceDirectory": "./instances",
"databaseDirectory": "./database",
"__comment1": "IP players should connect to to join your servers, should be portforwarded and stuff",
"publicIP": "localhost",
"__comment2": "IP the master web server runs at, can be ignored on the master system",
"masterIP": "localhost",
"__commentMasterPort": "HTTP port for the master to listen to",
"masterPort": 8080,
"__commentsslPort": "HTTPS port for the master to listen to. Set to 0 or an empty string to not listen for HTTPS.",
"sslPort": 0,
"sslCert": "database/certificates/cert.crt",
"sslPrivKey": "database/certificates/cert.key",
"__commentMasterAuthToken": "Authentication token to connect to the master server. Find it in secret-api-token.txt on the master.",
"masterAuthToken": "",
"__commentAuthSecret": "This is a long random string used for generating authentication tokens. Do not share. Regenerates when secret-api-token.txt is deleted.",
"masterAuthSecret": "",
"__comment3": "Your factorio.com username for factorio MP services, also username will be admin on new instances",
"username": "",
"__comment4": "Token from your factorio.com profile page. Required for MP and mod manager services.",
"token": "",
"__comment7": "Used for communication with the mod portal, not needed if you manage mods by hand",
"factorio_version": "0.16.41",
"__comment5": "General user settings",
"verify_user_identity": true,
"auto_pause": false,
"allow_commands": "admins-only",
"game_password": "",
"description": "This server runs Clusterio",
"visibility": {
"public": true,
"lan": true
},
"__comment6": "Statistic gathering statistics, may affect server performance (master memory usage especially)",
"itemStats": {
"maxEntries":60,
"entriesPerSecond": 0.01666666667
},
"logItemTransfers": false,
"__comment6.5": "When the master is undersupplied, we have 3 different modes to 'fairly' distribute items. First to ask gets all, dole division using weird formula or experimental neural net tries to make stuff as fair as possible. Feel free to play around.",
"disableFairItemDistribution": false,
"useNeuralNetDoleDivider": true,
"uploadModsToMaster": false,
"msBetweenCommands": 10,
"allowRemoteCommandExecution": true,
"enableCrossServerShout": true,
"mirrorAllChat": false,
"__comment7": "Autosave interval for items in the master storage in milliseconds. Defaults to one minute when set to 0 or not set.",
"autosaveInterval": 600000,
"disablePrometheusPushgateway": false,
"__comment8": "This setting only affects the master. It makes every import for things that arent 'electricity' return a maximum of 0. Effectively disables teleport chests.",
<sub>This document describes the communication layer in Clusterio and unless you're developing Clusterio it's probably not going to be very useful for you.</sub>
Communication is facilitated by the lib/link library and is based on three core abstractions: links, connectors and messages.
Links represents one side of a two way communication pipe, connectors deal with connecting and establishing communication, and messages are the content sent over the link.
## Link
The Link class represents one side of a two way communication channel where the actual communication is done by a connector associated with the link.
The link also knows what the endpoint types it's connected to are via the source and target attributes.
The endpoint types currently implemented are controller, host, instance, and control.
Although host and instances run in the same Node.js program they use virtual links to communicate between them in order to simplify the communication architecture.
Upon creation the link will register with the connector in order to receive and process messages from it.
The messages received are validated using the validator registered for the type of the message.
A message for which there is no validator for is considered an error.
## Connector
The connector used with a link is responsible for establishing and/or maintain the connection that messages are sent over.
Reconnect logic is expected to be implemented in the connector, though this has not yet been done.
There's also the virtual connector which lets two links in the same program be connected to each other without having to establish a network connection.
## Message
There's two types of messages currently implemented apart from the Link builtin close message and that's Request and Event.
The builtin messages are defined in [packages/lib/link/messages.js](/packages/lib/link/messages.js) and are attached to links via the attachAllMessages function, which is usally called in the Link subclass constructor.
The messages define which links they are valid on and only attaches handlers for those links.
There's also a forwarding mechanism for when a request or event is to be forwarded another hop.
<sub>This document describes the implementation and inner working of the configuration system in Clusterio, and unless you're developing Clusterio it's probably not going to be very useful for you.</sub>
Clusterio uses a field based configuration system to manage settings for the controller, instances, hosts, and the clusterioctl utility.
Plugins can add their own config fields for the controller and instances configs.
The fields are defined early in the startup, and fields for disabled plugins will still be created.
The built-in config fields are defined in [packages/lib/config/definitions.js](/packages/lib/config/definitions.js) while plugins can add their own config field definitions to the `controllerConfigFields` and `instanceConfigFields` properties of the `plugin` export.
A `<plugin_name>.load_plugin` field is automatically created for each plugin and it's used to enable/disable the loading plugins on startup.
## Defing Fields
Fields are defined as POJS object properties where the name of the property correspond to the name of the configuration field.
The property name is the primary way the field is accessed both in code and on the command line, as such it should follow the lower\_case\_underscore style and consist of a group name and a field name separated by a dot.
For plugins the group name is the name of the plugin, e.g., the `level` field of the `foo_frobber` plugin would be named `foo_frobber.lavel`.
**access**:
The locations this config field can be read and modified from.
This is an array of strings which can contain the values `"controller"`, `"host"` and `"control"`.
If a party is missing in this array then that party will not receive the value of this field when the config is shared, and will be unable to modify the field.
If not set any party can read and modify this config entry.
Note that instance config fields should have both `"controller"` and `"host"` or unexpected behaviour may occur.
**type**:
The type of config value this field will support.
The supported values are boolean, string, number, and object, and if an attept is made to set the field to a value that is not of the right type it will be reject unless it's a string that can be converted to the right type.
**title** (optional):
Text used to identify the config field in user interfaces.
This should not contain the grouping used in `name`.
**description** (optional):
Text used to describe the config field in user interfaces.
**enum** (optional):
Array of acceptable values this field can have.
Attempts to set a value not in this array will be rejected, which makes it useful for making enumareted list of choices.
**optional** (optional):
True if this field can be set to null.
Defaults to false.
**initialValue** (optional if optional is true):
Value this config field should take in a newly initialized config.
This can also be an async function returning the value to use.
If the value for a field stored in on disk somehow ends up being invalid the field will take on the initial\_value instead when the config is loaded.
## Accessing Fields
On the Config instance there's two ways to access fields: through the set and get methods.
E.g.:
let value = configInstance.get("group.field");
Or by through the raw fields object (which is best avoided as it doesn't check types or whether the accessed field exists).
E.g.:
let value = configInstance.fields["group.field"]
The set method also takes an extra argument, which is the value to set the field to.
If the value is not valid this will throw an InvalidValue exception, if the field doesn't exist it will throw an InvalidField exception.
## Config Lifecycle
Values for all config fields are created when the config is initialized, as well as for any fields with missing values when a config is loaded.
Once a field has a value it is never automatically removed, even if the field it was part of no longer exists.
This ensures that plugin configuration is not lost should the server be started up without the plugins that defined the configuration installed.
For instance configs, they are kept on the controller and serialized and sent over to the relevant hosts on startup.
This makes it possible to edit instance configs when the host the instance is on is offline.
A copy is also stored in the instance directory on the host, and this config will be deliverd to the controller on host startup, which enables instances to be copied between clusters as long as their IDs are unique.
Used to distinguish it from other clusters and is prepended to the name of servers in the server list.
Defaluts to "Your Cluster".
### controller.database_directory
Directory used to store cluster data to.
Needs to be writeable by the controller.
This is also used by plugins to store its data.
Defaults to "database".
### controller.http_port
Port to host HTTP server on.
If set to null no HTTP server will be exposed.
At least one of this and controller.https_port needs to be set to a port.
Defaults to 8080.
### controller.https_port
Port to host HTTPS server on.
Uses controller.tls_certificate and controller.tls_private_key as the certificate and private key for the HTTPS server.
If set to null no HTTPS server will be exposed.
At least one of this and controller.http_port needs to be set to a port.
See the [Setting up TLS](/docs/setting-up-tls.md) document for a guide to setting up HTTPS with Clusterio.
Defaults to null.
### controller.bind_address
IP address to bind the HTTP and HTTPS ports on.
Useful to limit which interface to accept connections from.
If set to null the unspecified IPv6 address will be used, which on most systems means it will also listen on the unspecified IPv4 address and accept connections from all
interfaces.
Defaults to null.
### controller.external_address
External address the controller is accessible on.
Currently only used for `clusteriocontroller bootstrap create-ctl-config` in order to give the right url to connect to.
This should be a full URL ending with a /.
Defaults to null meaning assume localhost.
### controller.tls_certificate
Path to TLS certificate to use for the HTTPS server when controller.https_port is configured.
Defaults to "database/certificates/cert.crt".
### controller.tls_private_key
Path to TLS private key to use for the HTTPS server when controller.https_port is configured.
Defaults to "database/certificates/cert.key".
### controller.auth_secret
Base64 encoded authentication secret used to verify tokens issued to hosts, users, and sessions.
This should contain 256 bytes of random data, and if changed will cause all tokens become invalid.
Keep this secret if leaked an attacker could easily compromise the cluster.
Defaults to null which means generate a secure secret on startup.
### controller.heartbeat_interval
Interval in seconds heartbeats are sent out at for WebSocket connections.
If a WebSocket connection hasn't received a heartbeat in 2 times the heartbeat interval it will be considered stale and closed.
A lower value means less time between connections going stale and them being closed.
Defaults to 15.
### controller.session_timeout
Time in seconds from the connection is closed following a heartbeat timeout until the session is invalidated and data loss occurs.
A higher timeout gives more time for a client to recconect and resume an active session over a connection that was dropped, but also leads to stale connections taking longer to clear out.
In the case of the controller or client crashing the data loss is unavoidable.
Value should not be less than the configured `max_reconnect_delay` of the clients and it should be greater than 2 times `controller.heartbeat_interval`.
Defaults to 60.
### controller.metrics_timeout
Timeout in seconds before a call to gather metrics from a host times out.
This should be less than both the timeout and collection interval configured for the collection job in Prometheus.
Defaults to 8.
### controller.stream_timeout
Timeout in seconds for proxy streams through the controller that has been created but is not yet sending data to be automatically closed.
Streams that have started sending data before this timeout is not affected.
Defaults to 15.
### controller.default_role_id
Role to automatically grant to new users.
If null no role is granted.
Defaults to 1 which correspond to the default Player role.
### <plugin_name>.load_plugin
Whether to load the plugin on the controller.
Plugins not loaded on the controller will not be loaded on instances.
Defaults to true.
## Host Configuration
### host.name
Name of the host.
Shows up in host lists and is used to reference this host in the clusterioctl command line interface.
Defaults to "New Host".
### host.id
Immutable numeric id of the host which uniquely identifies this host in the cluster.
Defaults to a random 31 bit number.
### host.factorio_directory
Directory to look for the Factorio server(s) in.
This can either point to a Factorio server directory, or to a directory containing multiple versions of the Factorio server.
Defaults to "factorio".
### host.instances_directory
Directory to store instances in.
One sub-directory will be created in it for each instance assigned to this host and these sub-directories will contain logs, saves, and various auto generated configuration files for the instances.
Defaults to "instances".
### host.controller_url
URL to connect to the controller to.
Defaults to "http://localhost:8080/".
### host.controller_token
Access token used for authenticating with the controller.
You can generate an access token with `clusterioctl host generate-token --id <host-id>`, or use the `clusterioctl host create-config` to create a new host config with the correct url and token in it.
Defaults to "enter token here".
### host.tls_ca
Path to certificate in PEM format to use for validating a TLS connection to the controller.
If you have a self signed certificate on the controller you will need to copy the certificate to your hosts and set it as the certificate authority with this option.
Defaults to null meaning use Node.js's default set of trusted certificate authorities.
### host.public_address
External address instances hosted on this host can be accessed on.
This is used by plugins like Server Select to give the correct address to switch between instances in-game.
Defaults to "localhost".
### host.max_reconnect_delay
Maximum delay in seconds to wait after connection to the controller is dropped before attempting to reconnect to it.
The actual delay on each reconnect will be a random number between 0 and this configured value to avoid all clients trying to reconnect at the same time.
Defaults to 60.
## Instance Configuration
### instance.name
Name of the instance, shown in instance lists and used to reference to this instance in the clusterioctl command line interface.
Defaults to "New Instance".
### instance.id
Imutable numeric id of the instance which uniquely identifies this instance in the cluster.
Defaluts to a random 31 bit integer.
### instance.assigned_host
Host this instance is assigned to.
To change this you need to use the separate assign instance interface.
After being assigned to a host the directory for the instance is created on the host.
Defaults to null meaning not assigned.
### instance.auto_start
If enabled start this instance when the host it is assigned to is started up.
Does not affect start or stop of the instance while the host is running.
Defaults to false.
### factorio.version
Version of Factorio to use for this instance.
Can be a version like "1.0.0" or special string "latest" meaning the latest version of Factorio found on the host.
Defaults to "latest".
### factorio.game_port
UDP port to run the Factorio server on.
If null a random port in the dynamyc range assigned each time the instance starts.
Defaults to null.
### factorio.rcon_port
TCP port to start the RCON interface on the Factorio server on.
If null a random port in the dynamic range is assigned each time the instance starts.
Defaults to null.
### factorio.rcon_password
Password to use for the RCON interafec.
If null a random secure password is generated and used each time the instance starts.
Defaults to null.
### player_online_autosave_slots
Keeps a separate pool for autosaves where at least on player has been online since the previous autosave was done.
This requires regular autosaving to be enabled and will rename autosaves to `_autosave_poN.zip` (where N is a slot number) whenever a player has been online for that save.
The slot numbers start at 1, are incremented by 1 after each autosave that is renamed and reset back to 1 whenever this value is exceeded.
Note that this value should be set to a higher value than the value of `autosave_slots` in `factorio.settings`.
Useful for preventing the loss of all progress when nobody is around on always online servers with auto pause disabled.
If 0 no autosave renaming takes place.
Defaults to 5.
### factorio.enable_save_patching
Whether to use save patching or not.
When enabled lua code is patched into the game save before starting it.
Most plugins require the use of save patching to function.
Turning it off makes it possible to use Clusterio to run and manage regular vanilla games and scenarios not compatible with Clusterio.
Defaults to true.
### factorio.enable_whitelist
Turn on the whitelist on the server.
Defaults to false.
### factorio.settings
Object with the settings to put into `server-settings.json` for the Factorio server.
The settings in `data/server-settings.example.json` of the Factorio installation is used as the base and properties specified here overrides the properties there.
Changes to the following properties will be applied live if the instance is running while it is changed: `afk_autokick_interval`, `allow_commands`, `autosave_interval`, `autosave_only_on_server`, `description`, `ignore_player_limit_for_returning_players`, `max_players`, `max_upload_slots`, `max_upload_in_kilobytes_per_second`, `name`, `only_admins_can_pause_the_game`, `game_password`, `require_user_verification`, `tags`, `visibility`.
Defaults to {"tags":["clusterio"],"auto_pause":false}.
### factorio.verbose_logging
Pass the `--verbose` flag to Factorio when starting the server.
This prints more messages to the log.
Defaults to false.
### factorio.strip_paths
Strip down absolute paths in the server log going to files inside the instance directory such that they are relative to the instance directory.
This improves the signal to noise ratio of the log.
0.614 Checksum for script /srv/clusterio/instances/test/temp/currently-playing/control.lua: 2390553941
1.277 Script @/srv/clusterio/instances/test/temp/currently-playing/modules/example/test.lua:7: Example log line.
Sample output when enabled:
0.487 Loading map saves/world.zip: 2615888 bytes.
0.614 Checksum for script temp/currently-playing/control.lua: 2390553941
1.277 Script @modules/example/test.lua:7: Example log line.
Defaults to true.
### factorio.sync_adminlist
Synchronize in-game admin list with admin status of the users in cluster.
If enabled Clusterio will generate a new `server-adminlist.json` on instance startup based on the users in the cluster with admin status set to true, and keep the admins in sync with the users that have admin status while the instance is running.
Defaults to true.
### factorio.sync_whitelist
Synchronize in-game whitelist with the whitelisted status of the users in the cluster.
If enabled Clusterio will generate a new `server-whitelist.json` on instance startup based on the users in the cluster with whitelisted status set to true, and keep the whitelisted users in sync with the users that have whitelisted status while the instance in running.
Defaults to true.
### factorio.sync_banlist
Synchronize in-game banlist with the banned status of users in the cluster.
If enabled Clusterio will generate a new `server-banlist.json` on instance startup based on users in the cluster with banned status set to true, and keep the banned users in sync with the users that have the banned status while the instance is running.
Defaults to true.
### factorio.max_concurrent_commands
Maximum number of commands being transmitted into the game in parallel.
Since the rate at which long commands are streamed into the game is by default limeted to 100 bytes/tick and scales down to 25 bytes/tick when there's 20 players connected, long commands can hold up the command interface and cause large latencies in getting data pushed into the game.
(The rates are controlled by the `segment_size` options in the server settings.)
To counteract this Clusterio sends up to this configured value number of commands in parallel over the RCON interface.
This causes the game updates sent to clients to have this many command streams in parallel contained in them, drastically increasing the maxiumum size they can become.
Since commands are only processed so ofter the maximum rate of commands that can be sustained is roughly 30 times this configured value per second, though note that large commands will lower this value.
The game updates sent by the server is split into roughly 500 byte packets, and if any one of those parts are lost the entire update is resent.
This means a high number of concurrent commands can amplify resends due to packet loss and degrade the player experience up to the point of becomming completely unplayable.
Prior to 0.18.7 the maximum practical size of game update sent where arround 8 kB due to the UDP receive buffer on windows defaulting to this value, larger updates would not get through the receive buffer and cause the connection to drop.
To stay below the 8 kB limit the maximum safe value for this option would be around 7000 / (3 * game_speed * maximum_segment_size), which is around 20 in normal circumstances.
Defaults to 5.
### <plugin_name>.load_plugin
Whether to load the given plugin on the instance.
Note that plugins need to be loaded on the controller in order for them to be loaded on instances.
Defaults to true.
## Control Configuration
### control.controller_url
URL to connect to the controller to.
Defaults to null meaning complain about it not being set and exit.
### control.controller_token
Access token used for authenticating with the controller.
You can generate an access token with `clusteriocontroller bootstrap generate-user-token <username>`, or use the `clusteriocontroller bootstrap create-ctl-config <username>` to create a new ctl config with the correct url and token in it.
Defaults to null meaning complain about it not being set and exit.
### control.tls_ca
Path to certificate in PEM format to use for validating a TLS connection to the controller.
If you have a self signed certificate on the controller you will need to copy the certificate to the computer you run clusterctl from and set it as the certificate authority with this option.
Defaults to null meaning use Node.js's default set of trusted certificate authorities.
### control.max_reconnect_delay
Max duration in seconds to wait before attempting to reconnect with the controller after the connection is dropped.
The actual delay on each reconnect will be a random number between 0 and this configured value to avoid all clients trying to reconnect at the same time.
- [Updating Your GitHub Fork](#updating-your-github-fork)
- [Changelog](#changelog)
- [Supported Node.js and Factorio Version](#supported-node.js-and-factorio-version)
- [Code Style](#code-style)
- [Naming Style](#naming-style)
- [Units](#units)
- [Indenting](#indenting)
- [Line Length](#line-length)
- [Markdown](#markdown)
- [Pushing releases to npm](#pushing-releases-to-npm)
## Workflow
While we don't require you to follow this workflow, it makes things much simpler for both you and us if you choose to do so.
Especially when it comes to making the commit history reflect what happened and not be a giant ball of mess.
GitHub's git workflow is less than ideal for dealing with small feature-branches so you might see things being done here a little diffrent than the usual.
It is recommended that you use the command line git version for doing the operations described here.
GUI wrappers tend to have different ideas of what is what and you might find it difficult to make it do the things described here.
For doing the actual commits though, your faviorit git GUI should work just fine.
If you pressed the fork button on GitHub and used `git clone`_on your own fork_ you can skip to the next paragraph.
Otherwise you will have to click the fork button and clone your fork.
If you already cloned the main GitHub repository you can change your clone to point at your own fork using the `git remote` command, but it's probably easier to delete your local clone and clone it again.
To keep your local repository up to to date with the main repository add the main repository as a remote.
By default node does not enable source maps, to get stack traces pointing to the `.ts` files you will need to pass `--enable-source-maps` to `node` or set the environment variable `NODE_OPTIONS=--enable-source-maps`.
Note that to use `clusterioctl` you will have to create a control config first:
Once you've set up the cluster you can use `node packages/controller run` to run the controller and `node packages/host run` to run the host which connects to the controller.
For web development on the controller there are also the following flags:
`--dev` - Recompile the `web_ui` package on the fly providing live reloading of it.
`--dev-plugin [name]` - Recompile the web module for the given plugin on the fly proving live reloading of it.
These options can be combined and multiple plugins can be actived for dev mode.
Keep in mind if you're working on external plugins the `--dev-plugin` option is likely to only work if the plugin is placed inside the `external_plugins` directory and its dependencies are installed with pnpm as part of the clusterio repository workspace.
Note that webpack is configured to delete the web interface build when using these options, so you will have to run `pnpm run -r --if-present prepare` before you start the controller without these flags again.
### Starting a Feature Branch
Development is done exclusively in feature branches.
This means the first thing to do before starting any development at all is to make a new branch starting from the master branch of the main repository.
To do this check out that branch from the main repository.
git fetch upstream
git checkout upstream/master
This will give a warning about detached HEAD which means you did it correctly.
Now you can make a feature branch based on this with.
git checkout -b my-feature-branch
The name of the branch is not important, as long as it's not a name you've used before.
But it's nice to have its name be somewhat descriptive of what it implements.
### Checking your code for errors
There's automated tests and linting setup for the project.
Please do add tests for any code you add and run both:
pnpm run test
pnpm run lint
To check that your changes pass the integration tests as well as the ESLint rules set up for the project.
### Submitting a Pull Request
Once you've commited changes to your feature branch that you want to have included into the main repository push it to you fork with
git push origin my-feature-branch
and make a pull request through the GitHub web interface.
You can also pass the `--set-upstream` (or `-u`) flag to make `git push` do the same thing for this branch.
After your feature branch have been merged you should consider it final.
If there are more changes you want to have merged or mistakes you need to fix you must start a new feature branch and submit a new pull request.
### Editing your Feature Branch
Until your feature branch is merged you can edit it as you see fit by using the rebase tool.
If your feature branch has been merged however it's final and you will have to start a new feature branch and submit that in a new pull request.
The base usage of rebase assuming you are checked out on your feature branch is the following command
git rebase -i upstream/master
This will open up an editor with the commits of your feature branch and let you reorder, edit, and remove commits as you see fit.
If the main repository has been updated and you have fetched down those updates with `git fetch upstream` this will also move your feature branch so that it is based on the latest commit from the main repository.
If you have already pushed your feature branch to your fork a simple `git push origin my-feature-branch` will no longer work as the histories are different, instead you will have to run
git push origin +my-feature-branch
The + sign signifies a force push and should be used with care.
If you have a pull request that has not yet been merged submitted for this feature branch this will update the pull request too.
### Resolving Merge Conflicts
The best way to resolve merge conflicts in feature braches is to rebase it on top of the updated upstream.
Run the following commands
git fetch upstream
git rebase upstream/master
This will stop on the commit(s) that caused the merge conflict and let you resolve them manually before continuing.
Once you're done force push your updated feature branch with
git push origin +my-feature-branch
### Updating Your GitHub Fork
No.
This is pointless busywork that accomplishes nothing.
Fetch the main repository directly as described above and base your feature branches on the refs from it.
GitHub has no interface or functionality for keeping the branches in your fork up to date with the branches in the main repository and there is no point in doing this for our feature branch based workflow.
## Changelog
There's a [Changelog](../CHANGELOG.md) in the root of the project.
If you make changes that are visible to users of this project you should add an entry to the changelog describing what has changed.
The format of this log should be self explanatory, please follow it carefully.
## Supported Node.js and Factorio Versions
Clusterio run on Node.js v18 and up, make sure to check that the Node builtin API's and npm packages you use are supported on these versions.
The GitHub Actions tests are run on whatever version of Node.js v18 and v20 is the default.
For Factorio Clusterio 2.0 aims to support version 0.17.69 and up, including the latest experimental release.
It's recommended that you use the lua API reference for 0.17.69, as there's no information on what version Factorio API's were introduced in.
The GitHub Actions tests are run against the latest experimental release.
## Code Style
The style of the code is a bit of a mixed bag at the moment.
But there are at least a few things that have been agreed upon.
### Naming style
JavaScript variables in general uses camelCase.
The exception is for classes/constructors which use PascalCase and variables destructured from over the wire objects.
Fields in configs uses lowercase_underscore.
JavaScript files are named using lowercase_underscore for files that export multiple items.
Files containing and exporting a single class should be named the same as the class in PascalCase.
Relative imports in lib package are prefixed with lib to make them easier to distinguish, as the names tend to be very generic.
E.g.:
const libLink = require("./link");
const libLuaTools = require("./lua_tools");
Note that no sub-modules of the lib package is imported directly from the outside of it with the exception of `build_mod`.
For lua code lowercase_underscore is used for everything.
### Units
When naming variables whos unit is not a base SI unit like seconds or metres the unit used should be suffixed in the variable name:
E.g.:
const startTimeMs = date.now();
const endNs = process.hrtime();
This makes it easier to spot different unit bases are being mixed in the code.
### Strings
JavaScript and Lua code should use double quoted strings (`"string"`) for all strings with the only exception that strings that contain double quotes may use single quotes (i.e., `'String with "quotes"'`).
JavaScript template literals may also be used where appropriate.
### Indenting
JavaScript and Lua code should be indented with tabs.
Other files types should prefer tabs for indents where reasonable with the exception of Markdown.
### Line Length
Lines are limited to 120 characters where tabs count as 4 characters.
If the argument list for a function or expressino for compound statement gets very long break them out on their own line(s).
For example:
function foo(
really, long, argument, list,
over, multiple, lines
) {
if (
really && long && expression
|| over && multiple && lines
) {
// code
}
})
### Markdown
Markdown files are formatted using one line per sentence to improve diff reports when making changes.
Headings have two spaces before them and on space after them to make them stand out, the heading at the top of the file does not have any spaces before it, and heading immediadly followed by another heading have one space between them.
For example:
```md
# A markdown File
Sentence of text which end in a period.
Each sentence is on it's own line.
And there's two spaces before the next heading.
## Subheading
### Sub-subheading
Headings use the `# heading` format.
```
## Pushing releases to npm
The releases are done by updating packages versions and then publishing them with pnpm.
You will need commit access to the repository as well publish access to the @clusterio org on npm to do this.
1. To do a release switch to the master branch and make sure it's up to date with the repository before proceeding.
```sh
git switch master
git pull --ff-only upstream master
```
Check and make sure there are no untracked files with `git status`.
2. Update the version in package files.
Make sure to update the command for the right version jump.
```sh
sed -i 's/2\.0\.0-alpha\.12/2\.0\.0-alpha\.13/' {packages,plugins}/*/package.json
```
3. Inspect, commit and tag the changes.
Again, make sure to be tagging the right version.
```sh
git add -p
git commit -m v2.0.0-alpha.13
git tag v2.0.0-alpha.13
```
4. Publish the packages.
```sh
# Warning: Double check your current working directory is the repository before running this
This document describes the various different types of content that can be created for Factorio and/or Clusterio and how this is made compatible with Clusterio.
- [Communicating with Clusterio](#communicating-with-clusterio)
- [Clusterio Modules](#clusterio-modules)
- [Clusterio Lua API](#clusterio-lua-api)
- [Clusterio Plugins](#clusterio-plugins)
## Factorio Mods
Factorio mods that do not interact with Clusterio should not have any considerations that need to be taken into account for them to work with Clusterio.
## Factorio Scenarios
Clusterio patches saves and because of this scenarios have a few limitations and must explicitly mark themselves as compatible with Clusterio to work.
The main limitation is that the [event_handler](https://github.com/wube/factorio-data/blob/master/core/lualib/event_handler.lua) lib must be used, and control.lua cannot contain any code other than calls to event_handler loading the relevant libs of the scenario.
This is because the control.lua file will be overwritten by the save patcher.
Scenarios also cannot use the event registering functions exposed by `script` such as `script.on_event` or `script.on_nth_tick` as using these will overwrite the handlers registered by the event_handler lib.
A brief description of the usage of the event_handler library is provided in the [event_handler interface](#event_handler-interface) section.
Once a scenario is made using the event_handler it should have a control.lua file that looks something like this:
```lua
-- control.lua
localevent_handler=require("event_handler")
event_handler.add_lib(require("module_foo"))
event_handler.add_lib(require("module_bar"))
```
To make Clusterio recognize and correctly handle this scenario it needs to include a clusterio.json file with following content:
```json
{
"scenario":{
"name":"example-scenario",
"modules":[
"module_foo",
"module_bar"
]
}
}
```
This tells Clusterio that it should load both module_foo and module_bar by passing the result of requiring them to the `add_lib` function when writing a new control.lua file to the save.
### event_handler interface
<sub>**Note:** At the time of writing the event_handler lib has been a part of Factorio since at least 0.17.4, and was moved from base to core in 0.17.63 but is still not documented anywhere.
A brief description is provided here for this reason.</sub>
The event_handler lib provides a simple interface for registering multiple callbacks to the same events without having to be concerned with callback chaining or overwriting callbacks defined elsewhere.
It works by taking over the task of registering the actual callbacks with Factorio, providing its own interface for the rest of the code to use.
The sole function of interest exported by event_handler is `add_lib` which accepts a table with event handler callbacks definitions that it will register.
The following entries are recognized by `add_lib`, all of which are optional:
-`on_init`:
Callback called when the callback of `script.on_init` is invoked.
-`on_load`:
Callback called when the callback of `script.on_load` is invoked.
-`on_configuration_changed`:
Callback called when the callback of `script.on_configuration_changed` is invoked.
-`events`:
Table mapping event ids as defined in `defines.events` to callbacks.
For example the table `{ [defines.events.on_player_died] = foo }` will call bar with the usual `event` table as argument every time a player dies.
-`on_nth_tick`:
Table mapping the nth tick number to a callback.
For example the table `{ [30] = bar }` will call bar every 30th tick.
-`add_remote_interface`:
Callback called before on_init/on_load callbacks<sup>[1]</sup>.
It has no special meaning and receives no arguments but should be used for registering remote interfaces.
-`add_commands`:
Callback called before on_init/on_load callbacks<sup>[1]</sup>.
It has no special meaning and receives no arguments but should be used for registering commands.
<sub>1: Before 0.17.69 these callbacks were called after on_init/on_load.</sub>
The usual way to use `add_lib` is to define the table of events to register in a separate file and return it, then load it in control.lua via `require` before passing it to `add_lib`.
And then in control.lua the following used to load it with `add_lib`:
```lua
-- control.lua
localevent_handler=require("event_handler")
event_handler.add_lib(require("example"))
```
Because the event_handler lib registers events itself you may not use `script.on_load` or `script.on_init` at all in your code and any usage of `script.on_configuration_changed`, `script.on_nth_tick` and/or `script.on_event` will cause the corresponding events registered with event_handler to break and should therefore not be used.
There's also the `add_libraries` function exported by event_handler, which accepts a table and calls `add_lib` for each value in the table.
## Communicating with Clusterio
Clusterio uses a homebrew protocol based on sending JSON payloads over a WebSocket connection.
See [socket.md](socket.md) for the implementation details of it.
It's also possible to write a plugin for Clusterio that exposes a custom interface over HTTP, WebSocket or any other technology supported by Node.js.
## Clusterio Modules
Modules are primarily used by plugins to inject code into Factorio games with the save patcher, though it's also possible to make stand alone modules that are loaded into the game if you don't need the capabilites of the plugin system.
The save patcher puts modules into the `modules` folder of the Factorio save and adds code to `control.lua` to load the module according to the `load` and `require` options to the module.json file.
Like with scenarios for Clusterio the [event_handler interface](#event_handler-interface) has to be used for any event subscriptions in modules.
Stand alone modules are placed into the modules folder of Clusterio, plugin modules are located in the module folder of the plugin.
In either case a `module.json` file is required and has the following structure:
```json
{
"name":"my_module",
"version":"1.2.0",
"dependencies":{
"clusterio":"*",
"foo":">=0.4.2"
},
"require":["bar.lua"],
"load":["foo.lua"]
}
```
The following entries are supported in the module.json file:
-`name`:
Name of the module, must match the folder the module is located in for stand alone modules or the name of the plugin the module is located in.
-`version`:
Version of the module.
Must be compatible with [Semantic Versioning 2.0.0](https://semver.org/).
In plugin modules this defaults to the version of the plugin otherwise it's required.
-`dependencies`:
Optional mapping of modules this module depends on and their version.
In the example `>=0.4.2` means that the foo module must be at least version 0.4.2.
See the [Ranges syntax](https://www.npmjs.com/package/semver#ranges) of the node-semver package for a full description of what operators are supported.
The events for dependencies is invoked before the events of the dependent, and starting an instance will fail if the dependencies cannot be satisfied.
If not specified it will default to depending on `clusterio`.
Make sure to add `clusterio` to your dependencies if you add other dependencies and depend on the Lua API.
-`require`:
Lua files to require in `control.lua`.
This should only really be needed if you want to make a global function available for use in Lua commands.
The paths are relative to the module's own folder and should be specified using forward slashes as directory sepparators if it's located in a sub directory in the module.
-`load`:
Lua files to load with the `event_handler` lib, the result of requiring the file will be passed to the `add_lib` function of the `event_handler` lib in `control.lua`.
See the section on the [event_handler interface](#event_handler-interface) for a detailed description on how the `event_handler` lib works.
The paths are relative to the module's own folder and should be specified using forward slashes as directory sepparators if it's located in a sub directory in the module.
It's recommended to use the new style of defining modules in Lua, as well as avoid the use of global variables as much as possible.
This means always declearing your top level variables and functions local and exporting only the things you need in other files.
You can require other files in your module by prefixing your require paths with `modules/your_module_name/`.
It's also possible to require files from other modules this way too.
The global variables and data stored in the global table is shared by all Clusterio modules as well as the scenario, so it's important that you use unique names.
It's recommended to prefix the global variables and data in the global table that your module has with the name of your module.
Because modules can be patched into an existing game you cannot rely on the `on_init` callback to be called in Clusterio Modules.
Nor can you rely on the `on_configuration_changed` callback, as this is not called when level code changes.
The Clusterio Lua API provides the custom `on_server_startup` event that can be used as a substitute, see the next section.
## Clusterio Lua API
Clusterio provides a save patched module as well as a regular Factorio mod for interfacing with Clusterio.
This includes tools sending data to plugins, script events to listen for, and an item serialization module.
Provides the main interface to Clusterio from within the game.
When using the mod version you must call `init` before it'll function properly.
#### init
<sub>Mod version only</sub>
Initialize the API.
This should be called from both the `on_init` and `on_load` events in your mod, and before this is called certain features will not be available, most notably the events table.
#### clusterio_api.events
Table of events raised by clusterio.
If the game is started outside of Clusterio then none of these events will be raised.
In the mod version this table does not exist before the `init` function is called.
##### on_instance_updated
Raised after the name and id of an instance has been updated.
This may occur even if the id and name didn't change.
Event data:
-`instance_id`: the id of the instance.
-`instance_name`: the name of the instance.
##### on_server_startup
Raised when Clusterio is starting up the server.
It can be used to initialize clusterio related features in a mod, and as a stand-in for the `on_init` and `on_configuration_changed` event for modules.
It is invoked on the first tick the server runs after the save has been patched, before most other events.
Use this event to initialize and/or migrate the data structures you need.
Keep in mind that both mods and Clusterio modules can switch from any version to any version so it should be able to handle both forwards and backwards migrations gracefully.
#### clusterio_api.send_json(channel, data)
Send JSON data to Clusterio over the given channel.
The `data` argument should be table that can be serialized with `game.table_to_json`.
Clusterio plugins can listen to channels and will receive an event with the data sent here.
See the [Communicating with Factorio section](writing-plugins.md#communicating-with-factorio) in the Writing Plugins document for more information.
If the game is started outside of Clusterio the data will be sent, but since there's no code following stdout to pick it up it will be lost.
**Note**: Payloads greater than 4 MB will cause stuttering in the game.
**Note**: This is not a binary safe way of sending data.
Strings embedded into the tables sent must be valid UTF-8 text.
Parameters:
-`channel`: string identifying which channel to send it on.
-`data`: table that can be converted to JSON with game.table_to_json
To serialise and deserialise data the Clusterio codebase uses a JSON serialisable data class pattern.
Instances of these classes can be serialised to JSON by passing them directly to `JSON.serialise` and deserialised by passing the serialised JSON representation to its static `fromJSON` method.
The data class also has a static `jsonSchema` property describing the serialised JSON representation for validation purposes.
See the following example code for a hypothetical `Data` class with a single property `member` defaulting to a value of `"foo"`.
```js
// Data.js
"use strictc";
/**
* A Data class example
* @alias module:Data
*/
classData{
/** @type {string} */
member="foo";
constructor(member){
if(member!==undefined){this.member=member;}
}
staticjsonSchema={
type:"object",
properties:{
"member":{type:"string"},
},
};
staticfromJSON(json){
returnnewthis(json.member);
}
// toJSON is called by JSON.stringify when serialising this class
Serialising an instance of `Data` for JSON is as simple as doing `serialised = JSON.stringify(data)` as the `toJSON` method is automatically called by `JSON.stringify` to create a suitable object.
Deserialising is done by calling the static `fromJSON` method with the serialised JSON data, like `data = Data.fromJSON(JSON.parse(serialised))`.
By structuring the classes like this they can be easily composed together.
For example a class holding a data property with an instance of the Data class from the previous example can be created as follows.
// toJSON is optional and not needed for this particular class
}
module.exports=Composed;
```
It's easy to create a new instance of Composed by calling `new Composed(new Data("spam"))` and serialising to and from JSON is the same as for Data.
Meaning `serialised = JSON.stringify(composed)` and `composed = Composed.fromJSON(JSON.parse(serialised))` does as expected.
## Interface description
Data classes are ES6 style class declarations with the following methods and fields:
### toJSON(): any (optional)
Method returning an object or primitive suitable for passing to JSON.stringify in order to create a JSON representation of the class instance.
The returned object may contain objects that have their own toJSON methods which JSON.stringify will then call recursively.
This method is optional if calling `JSON.stringify` on an instance without it would produce the desired serialised form, however this is rarely the case as often you'll have to provide it to convert the values fram Map/Set properites, omit default values or do similar actions.
Note that the primary use case for serialising data classes is to send them over the network, this method should therefore try to reduce the size of the resulting JSON by for example omitting properties who's values are the default value.
### static fromJSON(json: any): this
Method constructing an instance of this class from one that was previously serialised by passing an instance of it to `JSON.stringify`.
While not a requirement it is preferrable if the input is not mutated and that no references to any any structures in the input ends up in the returned object.
In other words calling fromJSON twice with the same input should produce two independent objects.
Note that directly passing the output of `toJSON` to `fromJSON` is not valid, as the output of `toJSON` may require further expansion calling `toJSON` on objects returned by it.
It has to go through `JSON.stringify` or an equivalent recursive expansion of `toJSON`.
### static jsonSchema
A [JSON schema](http://json-schema.org/) describing the format of the JSON resulting from passing an instance of this class to `JSON.stringify`.
This should be lenient towards adding additional properties that would be ignored by `fromJSON` as it makes it easier to extend these classes in the future without breaking compatibility.
Between version 2.0.0-alpha.13 and 2.0.0-alpha.14 the codebase underwent major refactoring and was ported to TypeScript.
In order to support alpha 14 there are several major changes that need to be done to existing plugins, these are in no particular order
## Rename master and slave
All references to slave and master in the code was renamed to controller and host respectively which means that code referencing these in plugins must be renamed.
In short this means performing a global search and replace on the codebase replacing:
-`master` with `controller`
-`Master` with `Controller`
-`slave` with `host`
-`Slave` with `Host`
in all places it occurs, even as part of longer names such as changing `BaseMasterPlugin` to `BaseControllerPlugin`.
## Change lib imports to import directly from the top level package
Previously code could import sub packages of lib such as lib/lua_tools.
These imports need to be changed to all import from lib directly.
Previously `BaseMasterPlugin` and `BaseInstancePlugin` were imported from `@clusterio/lib`, these have been moved to `@clusterio/controller` and `@clusterio/host` respectively.
Previously the info of a plugin would be loaded by requiring `./plugin_dir/info`, this has been changed to require `./plugin_dir` which in turn means that `info.js` is no longer the default entrypoint.
To solve this you can either add `"main": "info.js",` to package.json or rename `info.js` to `index.js` (the rest of this guide assumes the entrypoint was renamed to `index.js`).
The motivation behind this change was to allow TypeScript to output its build to a sub-folder of the plugin, if you're using TypeScript you'll likely have the entrypoint be located in `dist/index.js` instead.
The expected place where the declaration of a plugin is exported has also changed from `module.exports` to `module.exports.plugin`:
**Alpha 13**
```js
// info.js
module.exports={
name:"plugin_name",
...
}
```
**Alpha 14 (JS)**
```js
// index.js
constplugin={
name:"plugin_name",
...
}
module.exports={
plugin,
};
```
**Alpha 14 (TS)**
```ts
// index.ts
importtype*aslibfrom"@clusterio/lib";
exportconstplugin: lib.PluginDeclaration={
name:"plugin_name",
...
};
```
### Modify exposed path in webpack.config.js
The `./info` import path exposed in the ModuleFederationPlugin in the `webpack.config.js` needs to be changed to `./`.
E.g.:
**Alpha 13**
```js
// In webpack.config.js
newwebpack.container.ModuleFederationPlugin({
...
exposes:{
"./info":"./info.js",
...
},
}),
```
**Alpha 14**
```js
// In webpack.config.js
newwebpack.container.ModuleFederationPlugin({
...
exposes:{
"./":"./index.js",
...
},
}),
```
## Remove usage of ConfigGroup
The Config system was refactored and the `ConfigGroup` concept was completely removed.
Config fields are now defined directly on the exported plugin info.
// Event is an instance of the ForwardedEvent class
if(event.stringValue){
// ...
}
// src and dst describe where the message came from and who it was addressed to.
}
...
}
```
### Refactor message sending
Message are now addressed end to end instead of using the confusing forwardTo and broadcastTo properties of the message definition.
When broadcasting events the special `"allInstances"`, `"allHosts"`, and `"allControls"` shorthand addresses can be used.
There's two methods for sending messages `.send` and `.sendTo` where `.send` is a shorthand for `.sendTo` with an address filled in as the other side of the link, this is mostly only useful in control plugins where the other side of the link is the controller.
Clusterio communicates using a simple homebrew message passing protocol based on JSON over WebSocket with limited support for reliable transport of messages.
The protocol is implemented in [packages/lib/link](/packages/lib/link) and uses the `ws` library to handle the WebSocket connections on the Node.js side and the native WebSocket API on the browser side.
Two general kinds of messages are supported: events and requests.
Events are messages sent by one party and received by the other and offer no notification for the sender on whether it was actually received.
Requests are pairs of request-response messages where one side sends the requests and the other side replies with a response.
Apart from the handshake used to establish connections the protocol is directionless, and either party can send or receive both requests and events.
Events and requests are defined as instances of the `Request` and `Event` classes, and specify a unique type string, who can send and receive the message, optional forwarding of the message and the format of the data payload as a JSON schema.
All messages sent and received are validated with JSON schemas to catch potential errors in sending messages and avoid having to write lots of validation code in message handlers.
Messages are sent by calling the `.send()` method of the message instance and passing a `Link` instance to send it over and the data to send.
For the controller there's a `Link` instance for each host and control connection stored in the `WsServer` class.
For hosts the `Host` instance is a link to the controller, and each `Instance` instance is an in-memory link to the `Host` instance and has a corresponding `InstanceConnection` link stored on the `Host` instance.
For ctl and web_ui there's a `Control` instance that acts as the link to the controller.
Messages received invokes message handlers that were registered when the link was brought up.
These handlers are named after the name of the property the message is stored under with `EventHandler` appended for events and `RequestHandler` appended for requests.
The handlers are defined on the link class itself for core messages, and on the plugin class in the case of plugin defined messages.
## Connection flow
Clients connect via WebSocket to the `/api/socket` endpoint and upon establishing connection to the server the connection is in the handshake phase and the server will send a `hello` message to the client.
If the client has no previous session with the server it sends a `register_host` or `register_control` message depending on what kind of client it is to the server.
If the register is valid then the server will respond with a `ready` message with a session token and the connection has left the handshake phase, otherwise the server will close the connection with a WebSocket close code indicating the kind of failure.
Upon leaving the handshake phase both parties should start sending heartbeat messages with the sequence number of the last received message.
Messages sent over the link should be stored in a send buffer and only cleared from the send buffer when a heartbeat with a `last_seq` equal or or higher is received or the session is invalidated.
If the connection goes stale or is unexpectedly closed the client can reastablish the same session by connecting to the server again and sending a `resume` message after received the `hello` from the server.
If the session resume is successful then the server will respond with a `continue` message and the connection is active again.
The client should then re-send all messages it has in its send buffer that is greater than the `last_seq` number in the `continue` message.
Otherwise the server will respond with `invalidate` and the client is expected to drop its send buffer start a new session with `register_host` or `register_control`.
## Connector flow
To manage the complexity of connecting, reconnecting, resuming or invalidating dropped sessions the WebSocket connection for a `Link` class is managed by a connector.
There's three kinds of connectors defined: `WebSocketClientConnector`, `WebSocketServerConnector` and `VirtualConnector`.
The virtual connector is for in-memory connections and doesn't have any connection state logic.
For the WebSocket connectors there's four states they can exist in:
-`closed` - The connector has just been created or has been explicitly closed.
-`connecting` (client only) - The connector is in the process of trying to connect to the server.
-`connected` - The connector is successfully connected.
-`resuming` - The connection dropped and the connector is in the process of attempting to resume it.
Along with the following events emitted when transition between states:
-`connect` From `connecting` (client) or `closed` (server) to `connected`
-`drop` - From `connected` to `resuming`
-`resume` - From `resuming` to `connected`
-`invalidate` - From `resuming` to `connecting` or `closed`
The client connector will keep trying to establish and mantain a connection after `.connect()` is called on it, stopping only in the case of authentication failure or `.close()`/`.setClosing()` being called.
Other than that, the logic is pretty straight forward.
All communication in the protocol is done via WebSocket text frames containing a JSON payload with the following properties:
-`seq: integer | null` - The sequence number of the message, can only be null for handshake and heartbeats.
-`type: string` - The type of message
-`data: object` - Message data
Before a connection is established it's in the handshake phase and the `seq` property is `null`.
The `type` property is used to distinguish which of handshake, heartbeat, events or requests this message is and determines the expected format of the `data` property.
### Handshake Messages
The following messages are the only valid messages that can be sent during the handshake phase.
#### `hello`
Sent by the controller when the WebSocket connection has been opened.
-`version: string` - The version of the controller, e.g. "2.0.0".
-`plugins: Object<string, string>` -
Object mapping plugin names to plugin version for plugins that are loaded on the controller.
#### `register_host`
Host handshake for establishing a new connection session.
-`token: string` - Authentication token.
-`version: string` - The protocol version of the host, e.g. "2.0.0".
-`id: integer` - ID of the host.
-`plugins: Object<string, string>` -
Object mapping plugin names to plugin version for plugins that are available on the host.
#### `register_control`
Control handshake for enstablishing a new connection session.
-`token: string` - Authentication token.
-`version: string` - The protocol version of the control, e.g. "2.0.0".
#### `resume`
Handshake for resuming an existing connection session where the connection had dropped unexpectedly.
-`session_token: string` - Session token to resume session with.
-`last_seq: integer | null` - Last message seq received by the client.
#### `invalidate`
Sent by the controller in reply to a resume for a session that cannot be resumed.
The client should start a new session with a register\_\* message in response.
#### `continue`
Sent by the controller to signal resuming the session was successful.
-`last_seq: integer | null` - Last message seq received by the server.
-`session_timeout: integer` - Session timeout in seconds.
-`heartbeat_interval: number` - Rate client should send heartbeats at.
#### `ready`
Sent by the controller to signal the connection has been established with a new session.
-`session_token: string` - Token to resume the session with.
-`session_timeout: integer` - Session timeout in seconds.
-`heartbeat_interval: number` - Rate client should send heartbeats at.
### Heartbeat message
Once connected as indicated by the `ready` and `continue` messages both sides of the connection starts sending heartbeats at the inverval given in the `ready` and `continue` messages.
The heartbeat should contain the `seq` of the last received message, and receiving a heartbeat with a given sequence number means that the other party of the link has received all messages up to that seq number.
#### `heartbeat`
Sent by both the controller and clients at the interval given in the continue/ready messages.
-`last_seq: integer` - Last message seq received over the link.
### Events
Events are messages send in one direction over the link, that the receiver is expected to act upon, but not make any replies (for that there's [Requests](#requests).
By convention message types ending in `*_event` are events.
The data payload of an event consists solely of event specific properties.
See [packages/lib/link/messages.js](/packages/lib/link/messages.js) for the recognized core events and their contents.
Plugins may defined their own events as well.
#### `*_event`
- any properties defined by the event type.
### Requests
Requests are messages which expects a response in return.
They function a lot like HTTP requests except that both parties of a connection can inniate a request that is expected to be responded to.
By convention message types ending in `*_request` are requests, and is expected to be replied to with a corresponding `*_response` message.
If an error occured while processing the request an error response containing an error property in the data payload is sent instead.
See [packages/lib/link/messages.js](/packages/lib/link/messages.js) for the recognized core requests and their contents.
Plugins may define their own requests as well.
#### `*_request`
- any properties defined by the request type.
#### `*_response` success
-`seq: integer` - The seq number of the request message.
- any additional properties defined by the request type.
#### `*_response` error
-`seq: integer` - The seq number of the request message.
-`error: string` - Human readable text message describing the error.
### Graceful Disconnect
To ensure no data is lost when closing the link it's important to send a prepare disconnect request and wait for it to complete before closing the WebSocket.
Either party of the link can send this request and the party sending the request should be the one that initates the close of the WebSocket.
#### `prepare_disconnect_request`
Request graceful disconnect of the connection.
The receiving party should stop tasks that send events and requests over the link and wait for pending requests it has over the link to complete, but it should not stop responding to requests sent to it.
Only after this process is done should the response be sent.
#### `prepare_disconnect_response`
Signals the preparation for disconnect is complete and that the connection can be closed.
The party that initiatied the prepare disconnect request may still send other requests/events over the link to finish its part of the disconnection proceedure.
If the error response variant is received it should be logged if possible and treated as a success.
### Example handshake
Server hello after opening the WebSocket.
```json
{
"seq":null,
"type":"hello",
"data":{
"version":"2.0.0"
}
}
```
Host registering a new connection session.
```json
{
"seq":null,
"type":"register_host",
"data":{
"token":"xyzabc123",
"version":"2.0.0",
"id":123
}
}
```
Controller reply signaling the session is established and ready.
```json
{
"seq":null,
"type":"ready",
"data":{
"session_token":"xyz.azy.bzy",
"session_timeout":60,
"heartbeat_interval":15
}
}
```
Host and controller can now send events and requests over the connection.
In order to efficiently get data in and out of running Factorio servers and to provide custom interfaces Clusterio uses save patching<sup>[1]</sup> to add code to the game.
The save patching is based on the built in [event_handler](https://github.com/wube/factorio-data/blob/master/core/lualib/event_handler.lua) library to Factorio and requires scenarios to be coded specifically for it.
Mods on the other hand work without modifications.
This can be disabled with the `factorio.enable_save_patching` config on a per instance basis, however disabling it means that modules are not loaded into the game and most plugins will not function.
<sub>1: Before version 2.0 of Clusterio this was done at runtime with a patcher called Hotpatch, but due to the difficulty in supporting run time patching and the lack of documentation it was replace it with save patching instead.</sub>
## User Accounts
The controller creates and stores accounts for all Factorio users that log into instances under its control.
These accounts are used to store per player data and authorize management of the cluster.
By default users accounts are assigned a Player role that gives a limited read access to the cluster resources, and admins of the cluster can be assigned a Cluster Admin role to get access to manage it.
Clusterio clusters are managed through the controller by using the `clusterioctl` command line interface which is invoked by running `npx clusterioctl <command>` in the clusterio directory.
This document uses the shorthand `ctl> foo` to indicate `npx clusterioctl foo` should be executed.
Mandatory parameters are shown in `<angles bracket>` and optional pameters are in `[square brackets]`.
Before `clusterioctl` can be used it needs to be configured for the cluster it will connect to.
The easiest way to do this is to run `npx clusteriocontroller bootstrap create-ctl-config <username>` on the controller, which creates the necessary `config-control.json` for managing the cluster as the given user.
## Hosts
### List hosts
ctl> host list
Lists all hosts.
### Generate token
ctl> host generate-token [--id <host id>]
Generate a token for a host. `--id` is the host id.
Create a host config. `--id` id the host id. `--name` is the host name. `--generate-token` decides whether to also generate a token, defaulting to `0`. `--output` is the filename for the output of the config; Default is "config-host.json", "-" for stdout.
## Instances
### List Instances
ctl> instance list
List instances.
### Create instance
ctl> instance create <name> [--id <instance id>]
Create a new instance. `name` is the instance name. `--id` is the optional id.
### Kill instance
ctl> instance kill <name>
Terminates the running Factorio server without giving it a chance to save or cleanup after itself.
Note: This may cause loss of data.
### Extract players from instance
ctl> instance extract-players <name>
Creates a user account in the cluster for each player that has been online on the currently running save of the instance and sets the online time of that account on this instance to the online time recorded in the save.
Useful when importing a save to a cluster.
Note that the online time in a save is recorded differntly from how Clusterio records online time.
Most notably Clusterio records wall clock time while the save time is recorded in ticks and this extraction of player time assumes the game runs at 60 ticks a second.
### Config management
#### List Configuration
ctl> instance config list <instance>
List config values for an instance. `instance` is the instance whose config values to list.
#### Set config value
ctl> instance config set <instance> <field> [value] [--stdin]
Set field in instance config. `instance` is instance to set. `field` is field to set. `value` is value to set in field. `--stdin` signals to read value from stdin.
#### Set config property of field
To be written.
#### Edit instance config
ctl> instance config edit <instance> [editor]
Edit instance config in a text editor. Once file is saved and editor has been exited, config values will be read back in. `instance` is the instance to edit, `editor` is the editor to use. If `editor` is not given, clusterioctl will read from the `EDITOR` or `VISUAL` enviroment variables.
### Assign instance to host
ctl> instace assign <instance> [host]
Assigns instance to host. `instance` is instance to assign, `host` is host to assign to. If `host` is not provided, instance will be unassigned.
## Users
Clusterio automatically creates user accounts for all players that join an instance when save patching is enabled.
These accounts are used to store per player data shared between instances, like if the player should be whitelisted, admin or banned.
### List users
ctl> user list
Lists all user accounts in the cluster along with some data for each.
### Create User
ctl> user create <name>
Creates a new empty user account for the given Factorio player.
Note that case matters here.
This is usually not required as user accounts are created automatically for players that join instances with save patching enabled.
### Promote User to Server Admin
ctl> user set-admin <name> [--revoke] [--create]
Promotes the user given to in-game admin on instances with the `sync_adminlist` option enabled.
If the `--revoke` switch is used the user is removed from the adminlist.
Since admin status is a part of the account data the account must exist for this to succeed, passing `--create` will create the account if it does not exist.
**Note:** Being a server admin does not grant any access to manage the cluster.
See [set-roles](#set-cluster-roles) for adding roles which grant access to managing the clusetr.
### Whitelist User
ctl> user set-whitelisted <name> [--remove] [--create]
Add the user given to the whitelist on instances with the `sync_whitelist` option enabled.
If the `--remove` switch is used the user is removed from the whitelist.
Since whitelisted status is a part of the account data the account must exist for this to succeed, passing `--create` will create the account if it does not exist.
### Ban User
ctl> user set-banned <name> [--reason <message>] [--pardon] [--create]
Ban user in-game from instances with the `sync_banlist` option enabled.
Reason is a message that will be shown to the user when attemption to log in.
If the `--pardon` switch is used it removes the ban.
Since ban status is a part of the account data the account must exist for this to succeed, passing `--create` will create the account if it does not exist.
Note: This bans the user from logging in to Factorio servers in the cluster, it does not revoke access to any cluster management they might have, see next section on setting roles for revoking tha.
### Bulk importing admin, whitelist and ban lists
Whitelists and admin lists can be bulk imported with the following commands:
ctl> user whitelist-bulk-import <file> [--create]
ctl> user admin-bulk-import <file> [--create]
The file should simply be newline seperated names:
joe
bob
allan
The `--create` option will create the user if they do not exist, and then set the relevent value.
Banlists can be bulk imported with the following command:
ctl> user banned-bulk-import <file> [--create]
The file should be names, followed by a ban reason:
joe greifed copper mines
bob stole materials
allan rude language
The first string on the line is used as the username, the others are used as the ban reason.
The `--create` option will create the user if they do not exist, then ban them.
### Set Cluster Roles
ctl> user set-roles <name> [roles...]
Replaces the roles the user has in the cluster with a new list of roles.
Calling with an empty roles argument will remove all roles from the user.
By default there's a roled named Cluster Admin which grants access to everything and a role named Player which grants a limited read access to the cluster, see [the section on roles](#roles) for more information about setting up roles and permissions.
### Delete user
ctl> user delete <name>
Deletes everything stored on the controller for this user.
Note: If the player is banned from the cluster this will effectively unban them, as the ban status is stored with the user account.
Note: If the player joins the cluster again a new account will be made for them automatically.
Alpha 14 comes with significant breaking changes for existing clusters, most notably the rename of master and slave to controller and host.
Please study the entire migration guide carefully before attempting to update an existing cluster.
## Before updating
If you're using mods you need to create mod pack string for the mod configurations you're currently running.
You can do this by running the follow RCON command on an instance that is currently running the mod configuration you want to use (you can also run this command in single player and find the string printed in Factorio's [Log file](https://wiki.factorio.com/Log_file)):
/sc local m={name="pack",description="",factorio_version=game.active_mods.base,mods={},settings={startup={},["runtime-global"]={},["runtime-per-user"]={}}} for n,v in pairs(game.active_mods) do table.insert(m.mods,{name=n,enabled=true,version=v}) end local function set(s,c) for n,v in pairs(s) do m.settings[c][n]=v end end set(settings.startup,"startup") set(settings.global,"runtime-global") if game.player then set(game.player.mod_settings,"runtime-per-user") else set(settings.player,"runtime-per-user") end local e=game.encode_string(game.table_to_json(m)) log(e) rcon.print(e)
This should produce a [Mod Pack String](https://forums.factorio.com/viewtopic.php?f=96&t=103578) which you need to copy and paste somewhere to keep it for later.
If you have multiple mod configuration create a mod pack string for each and note them down.
## Migrate installations
**Note:** Check that plugins you are using have updates avaialble for alpha 14 before attempting to update, if they do not either wait for the plugin to be updated or uninstall the plugin from all places it is installed on first.
Shutdown all slaves and the master server and then migrate each installations by running the following command:
This will convert the existing installation's usage of the master and slave names in packages, configs, database and logs to the new controller and host terms and update the installed packages.
If you're using systemd then copy the generated clusteriocontroller.service and/or clusteriohost.service files from the systemd folder to /etc/systemd/system/ and remove the old clusteriomaster.service and clusterioslave.service files.
Once the installations have been migrated start up the controller and hosts again.
## Post installation
Import the Mod Pack strings that was created beforehand on the controller and upload the mods needed by those mod packs to the controller.
You can assign a default mod pack in the controller config which instances use by default or set the mod pack on the per instance basis in the instance config.
Once you have assigned a mod pack to an instance run a data export on the instance in order to properly populate the mod settings in the mod and get item icons and item names resolved in the web interface.
By default Clusterio will listen for connections over HTTP which is fine for local networks but over the Internet it should idealy be over HTTPS as there are some credentials that could be exposed to eavesdroppers.
The recommended setup is to have an ordinary HTTPS server software like Apache or Nginx to act as a reverse proxy in front of the Clusterio controller.
The reverse proxy would then be set up with a TLS certificate from a Certificate Authoritiy like [Let's Encrypt](https://letsencrypt.org).
Alternatively you can have Clusterio listen for connections over HTTPS directly by providing it with a TLS certificate and private key.
Note that a TLS certificate signed by a recognized Certificate Authority in general requires that you own a domain.
See the last section for how to create and use a self-signed certificate for an IP address if you don't have a domain.
## Using a reverse proxy
Configure the proxy to forward requests over HTTP to the port the controller is listening to so that for example a request to `https://example.com/instances/` is forwarded to `http://localhost:8080/instances/`.
It is possible to host the interface under a sub-path, in that case a request under the sub-path should be mapped to the controller with the sub-path stripped out.
For example `https://example.com/sub/path/instances/` would be mapped to `http://localhost:8080/instances`.
Additianlly WebSocket connections to `/api/socket` also needs to be forwarded.
Typically this requires explicit configuration and support for forwarding WebSocket connections.
If the interface hosted under a sub-path the `/api/socket` address is relative to the sub-path and the sub-path also needs to be stripped out when forwarded.
You can also optionally set the controller to listen only on the loopback interface to prevent it being directly connected to from the outside.
npx clusteriocontroller config set controller.bind_address 127.0.0.1
### Apache Config
To proxy with Apache make sure that `proxy_module`, `proxy_http_module` and `proxy_wstunnel_module` is loaded in the config, then add the following directives to wherever is the approriate place for your flavour of Apache config organization.
```apache
<Location/sub/path/>
ProxyPass"http://localhost:8080/"
</location>
<Location/sub/path/api/socket>
ProxyPass"ws://localhost:8080/api/socket"
</Location>
```
Replace `/sub/path/` with the location on the proxy you want to host the Clusterio interface under.
If the whole domain is to be dedicated to the Clusterio interface then `/sub/path/` should be just `/`.
Pay attention to the use of trailing slashes, as they are important.
## Hosting HTTPS directly from Node.js
Listening on HTTPS is supported by the Clusterio controller via the `controller.https_port` option provided paths to a certificate file and a private key file is configured for the `controller.tls_certificate` and `controller.tls_private_key` options.
Both of these are expected to be unencrypted PEM files.
## Creating and using a self-signed certificate
Create a self-signed certificate with OpenSSL by running
substituting `<ip>` with the IP address the server will be accessed under, for example `IP:203.0.113.4`.
Multiple addresses can be given by supplying a comma sepparated list of `IP:address` pairs and DNS names can also be specified using the `DNS:example.com` format.
You need to supply every name you want to connect to the server under the subjectAltName field, for example to connect to the HTTPS port locally via an address like `https://localhost:4443/` you need to add `DNS:localhost`.
This creates a private key file named `key.pem` and a public certificate file named `cert.pem` and these are used for the `controller.tls_certificate` and `controller.tls_private_key` config options respectively on the controller:
npx clusteriocontroller config set controller.tls_certificate cert.pem
npx clusteriocontroller config set controller.tls_private_key key.pem
Additionally you will need copy the `cert.pem` file to all of the computers you want to set up as hosts and configure it as the `host.tls_ca` option:
npx clusteriohost config set host.tls_ca cert.pem
You will also need to copy the `cert.pem` file to all of the computers you want to remotely manage the cluster via clusterctl and configure it as `control.tls_ca` option:
npx clusterioctl control-config set control.tls_ca cert.pem
- [Adding Custom Commands to clusterioctl](#adding-custom-commands-to-clusterioctl)
## Plugin Structure
The basic file structure of a plugin is the following.
plugin_name/
+- index.js
+- package.json
+- controller.js
+- instance.js
+- control.js
+- test/
| +- plugin.js
+- module/
+- module.json
+- plugin.lua
Clusterio plugins are Node.js packages that can be published on npm for ease of installation and distribution.
The usual guides for creating such packages apply.
At minimum the `package.json` file must contain a version entry.
A possible workflow for developing plugins is to place the plugin in a sub-directory of where clusterio has been installed, and rely on Node.js searching up the folder heirarchy for it to find `@clusterio/lib`.
To add it to `plugin-list.json` so that it gets loaded use the `plugin add <path>` sub-command to either clusteriocontroller, clusteriohost or clusterioctl.
Note that it's important that the path starts with ./ or ../ (use .\ or ..\ on Windows).
For a plugin to be recognised by Clusterio it needs to export an entry named `plugin` from its main entrypoint.
By default the main entrypoint is the `index.js` file, but this may be changed by setting the `"main"` entry to a different file in `package.json`.
Here's an example of `index.js`:
```js
module.exports.plugin={
name:"foo_frobber",
title:"Foo Frobber",
description:"Does advanced frobnication",
instanceEntrypoint:"instance",
controllerEntrypoint:"controller",
messages:{
/* See below */
},
};
```
The following properties are recognized:
**name**:
Internal name of the plugin.
Must be the same as the plugin's directory name
**title**:
Name of the plugin as shown to users.
Currently not used.
**description**:
Brief description of what the plugin does.
Currently not used.
**instanceEntrypoint**:
Path to a Node.js module relative to the plugin directory which contains the InstancePlugin class definition for this plugin.
This is an optional paramater.
A plugin may have code only for instances but it must still be loaded on the controller in order for it to be possible to load it on an instance.
**instanceConfigFields**:
Object mapping instance configuration fields to `FieldDefinitions`for this plugin.
See [Plugin Configuration](#plugin-configuration)
**controllerEntrypoint**:
Path to a Node.js module relative to the plugin directory which contains the ControllerPlugin class definiton for this plugin.
This is an optional parameter.
A plugin can be made that only runs on the controller.
**ControllerConfigGroup**:
Object mapping controller configuration fields to `FieldDefinitions`for this plugin.
See [Plugin Configuration](#plugin-configuration)
**ctlEntrypoint**:
Path to a Node.js module relative to the plugin directory which contains the CtlPlugin class definition for this plugin.
This is an optional paramater.
A plugin can be made that only runs on the clusterioctl side.
**messages**:
Object with link messages definitions for this plugin.
See guide for [defining link messages](#defining-link-messages) below.
The optional module folder contains a Clusterio module that will be patched into the save when the plugin is loaded.
See the section on [Clusterio Modules](developing-for-clusterio.md) in the Developing for Clusterio document.
The only restriction imposed on modules embedded into plugins is that they must be named the same as the plugin.
While there is no standard for how to organize a plugin it's recommended to put the ControllerPlugin class definition into controller.js and the InstancePlugin class definition into instance.js.
You can put them into whatever file you want (even the same one for both).
For both instanceEntrypoint and controllerEntrypoint the path should not end with .js and it should use forward slashes for directory separators if any.
## Defining the plugin class
The plugin class should derive from its respective base class defined in `lib/plugin`.
For example, to define a ControllerPlugin class the following code can be used:
For the instance plugin it's exactly the same except "Controller" is replaced with "Instance", for host plugins "Host" is used and for the clusterioctl plugin "Ctl" is used.
The available hooks that you can override are documented in the base classes.
It's best to avoid defining a constructor, but if you insist on defining one, forward all arguments to the base class.
E.g.:
```js
constructor(...args){
super(...args);
// Code here
}
```
The arguments passed may change, and attempting to modify them will result in unpredicatable behaviour.
The async init method is always called immediatly after the constructor, so there's little reason to do this.
## Logging Messages
The base plugin classes provide a winston logger for logging messages to the shared cluster log.
For instances a copy of the log is also stored on the host the instance is on.
To use it, pass a string to one of the log levels functions, for example:
```js
asyncinit(){
this.logger.info("Initializing frobbing");
}
```
Metadata covering which plugin and instance (for instance plugin classes) the log event happened on will automatically be attached to the log message.
The available levels are fatal, error, warn, audit, info, and verbose.
These levels have roughly the following meanings:
- fatal: Unrecoverable error that prevents continuing execution of the Node.js process.
Note that Plugins shouldn't have fatal errors.
- error: An unexpected error condition occured, usually an exception thrown when it wasn't expected.
Should be investigated and fixed.
- warn: An potential error was detected, but operation continued gracefully.
These may indicate that something is running sub-optimally, or is not working as intedend.
- audit: Record of actions performed by users of the cluster.
This is usually emitted in response to management requests by a control link.
- info: A general log message.
May indicae a status or operation completed.
This should not be emitted by periodic tasks that run often.
- verbose: A verbose log message which might be useful when investigating issues, does not show by default.
Note that logging messages is not a cheap operation even when the messages do not show up and spammy logging should be avoided for the verbose level.
If the logs are still valuable in certain cases consider putting them behind a config option.
For guarding unexpected errors the best option is to log a short description along with the`stack` property of the Error:
```js
try{
// Operation that should not throw but may end up throwing
For plugin hooks expections thrown are automatically catched and logged, but for event handlers registered on EventEmitters it's critical that exceptions are catched and handled appropriately.
## Plugin Configuration
Clusterio provides a configuration system that handles storing, distributing, editing and validating config fields for you.
You can take advantage of it by declaring config fields under either `controllerConfigFields` or `instanceConfigFields` in the `plugin` export.
For example in index.js:
```ts
importtype*aslibfrom"@clusterio/lib";
declaremodule"@clusterio/lib"{
// Extend the interface defining available fields so that instance.config.get(...)
// will have our own options.
exportinterfaceControllerConfigFields{
"foo_frobber.level":number,// If optional is true then this should have | null in addition.
}
}
exportdefault{
...
controllerConfigFields:{
"foo_frobber.level":{
description:"Level of frobnication done",
type:"number",
initialValue: 2,
},
}
}satisfieslib.PluginDeclaration;
```
Code inside the `ControllerPlugin` class will then be able to access the level config field through the `Config` object at `this.controller.config`, for example in the ControllerPluginClass:
thrownewError("foo_frobber plugin requires save patching.");
}
}
```
## Communicating with Factorio
For pushing data into Factorio there's RCON, which lets you send arbitrary Lua commands to invoke whatever code you want in the game.
This is done by calling the `sendRcon` method on the plugin object.
For example:
```js
asynconStart(){
letresponse=awaitthis.sendRcon(
"/sc rcon.print('data')"
);
// Do stuff with response.
}
```
Because data into Factorio is streamed at a rate of 3-6 kB/s by default, it is recommended to avoid sending large commands as much as possible, and to strip down the data on the ones you send to only what's strictly necessary.
You can have lua code injected into the game via the module system and call that from RCON to avoid having to send code through the commands.
For getting data out from Factorio there's both RCON and the `send_json` API of the Clusterio module.
Returning data via RCON is prefered if the action is initiated from the Node.js side.
The `send_json` API allows sending JSON payloads on channels that plugins can listen to.
From a plugin you listen for an event named `ipc-channel_name` in order to get data sent by `send_json`.
It's recommended to either use the plugin name as the channel name or to prefix the channel name with the name of the plugin if you need multiple channels.
It's also important to catch any errors that might occur as they will otherwise be propogated to the instance code and kill the server.
Data out from Factorio does not have the same limits as data into Factorio, RCON responses can be in 100kB range without causing issues, and payloads to the `send_json` API can be in the 4MB range provided the server has a fast enough storage system.
**Note:** both `send_json` and RCON can operate out of order.
For `send_json` it's possible that payloads greater than 4kB are received after payloads that were sent at a later point in time.
For RCON, commands longer than 50 characters may end up being executed after shorter commands sent after it.
## Defining Link Messages
You will most likely have to communicate with the controller or other instances in your plugin for it to do anything useful.
For this there's a WebSocket communication channel established between the hosts and the controller that plugins can define their own messages to send over it.
This channel is bi-directional and all messages sent over it are validated with a JSON schema (see [this guide][guide] for an introduction to writing JSON schema).
There are currently two kinds of messages that can be defined: events and requests.
Events are simple one-way notifications that invoke a handler on the target it's sent to.
Requests are pairs of request and response messages where the request is sent to the target and the response is the reply back from the target.
The requests are similar to HTTP requests, except both parties of a link may initiate one.
### Defining Events
Events are defined as properties of the messages object exported on `plugin` that map to instances of the `Event` class from `lib/link`.
The name of the property correspond to the handler invoked on the plugin class.
The Event constructor takes an object of properties that define the event, for example the following could be defined in `plugin`:
```js
messages:{
startFrobnication:newlib.Event({
type:"foo_frobber:start_frobnication",
links:["controller-host","host-instance"],
forwardTo:"instance",
eventRequired:["frobnication_type"],
eventProperties:{
"frobnication_type":{type:"string"},
"urgent":{type:"boolean"},
},
}),
},
```
This specifies an event that can be sent from the controller to a host, and from a host to an instance.
It also specifies that the event must contain the property `frobnication_type`, with a string value in the data payload and that it may optionally contain a boolean `urgent` property.
It will also be forwarded by hosts to a specific instance.
The following properties are recognized by the Event constructor:
#### type
The message type sent over the wire.
This must start with the name of the plugin followed by colon and and be unique for the plugin.
The type of the message sent over the socket will have the suffix `_event` appended to it.
#### links
An array of strings describing which links this event can be sent over.
Direction matters, `"controller-host"` means the event can be sent from the controller to the host, but can't be sent back the other way, unless `"host-controller"` is also present in the links array.
The available endpoints are `controller`, `host`, `instance`, and `control`.
Controller talks with host and control, and host talks to instance.
The full chain must be specified as the individual links in order for a message to travers multiple hops, (i.e., for a message to go from controller to instance it must have both `"controller-host"` and `"host-instance"` in the links array).
See `forwardTo` and `broadcastTo` for ways to forward an event to the next link in a chain.
#### forwardTo
Target to forward an event to.
Can either be `"controller"`, to indicate a host should forward it to the controller, or `"instance"`, to indicate it should be forwarded to the instances specified by the `instance_id` event property.
This works by using a default handler for the event at the links that forward it.
#### broadcastTo
Target to broadcast this message towards.
A value of "instance" means the event will be broadcast to all instances downstream of the target it's sent to, but not back from where it came from.
Currently, only "instance" is supported.
This means that sending the event to a host from an instance will cause it to be broadcast to all instances of that host except for the instance it came from.
#### eventRequired
By default all properties defined in `eventProperties` are required to be present in the payload for the event.
This can be overriden by specifying an array of properties which are required here.
This is equivalent to using the `required` keyword in JSON schema.
#### eventProperties
Object with properties mapping to a JSON schema of that property that specifies what's valid to send in the event.
This is equivalent to using the `properties` keyword in JSON schema, except that the properties specified are by default required and additional properties are not allowed.
See [this guide][guide] for an introduction to writing JSON schemas.
The forwardTo and broadcastTo can be combined such that specifying `"controller"` as the forwardTo value and `"instance"` as the broadcastTo value will cause the event to be broadcast to all instances in the cluster.
For this to work, you will need to specify `instance-host`, `host-controller`, `controller-host`, and `host-instance` as the links.
Keep in mind when forwarding events that if the target an event is being forwarded to is not online, the event will be dropped.
Use a request if you need a confirmation that the message was received.
### Definining Requests
Requests are defined as properties of the messages object exported by `plugin` that map to instances of the `Request` class from `lib/link`.
The name of the property corresponds to the handler invoked on the plugin class.
The Request constructor takes an object of properties that define the event.
For example, the following could be defined in `plugin`:
```js
messages:{
reportFrobnication:newlib.Request({
type:"foo_frobber:report_frobnication",
links:["controller-host","host-instance"],
forwardTo:"instance",
requestRequired:["verbosity"],
requestProperties:{
"verbosity":{type:"integer"},
"special":{type:"boolean"},
},
responseProperties:{
"report":{
type:"array",
items:{type:"string"},
},
},
}),
},
```
This specifies a request that can be sent from the controller to a host, and from a host to an instance.
The request data must contain the property `verbosity` with an integer number as the value, as well as the `instance_id` property (implied by `forwardTo: "instance"`) and it may also contain a boolean `special` property.
It also defines that the response sent must contain a `report` property mapping to an array of strings.
When received by a host, it will also be forwarded to the instance specified by `instance_id`.
The following properties are recognized by the Request constructor:
#### type
The message type sent over the wire.
This must start with the name of the plugin followed by colon and and be unique for the plugin.
The type of the message sent over the socket will have the suffix `_request` appended to it for the request and `_response` appended to it for the response.
#### links
An array of strings describing which links this request can be sent over.
Direction matters; `"controller-host"` means the request can be sent from the controller to the host and the host can reply to the controller, but the host can't send a request to the controller unless `"host-controller"` is also present in the links array.
The available endpoints are `controller`, `host`, `instance`, and `control`.
Controller talks with host and control, and host talks to instance.
The full chain must be specified as the individual links in order for a message to travers multiple hops, (i.e., for a message to go from controller to instance it must have both `"controller-host"` and `"host-instance"` in the links array).
See `forwardTo` for ways to forward a request to the next link in a chain.
#### forwardTo
Target to forward the request to.
Can either be `"controller"` to indicate a host should forward it to the controller when receiving it from an instance, or `"instance"` to indicate it should be forwarded to the instances specified by the `instance_id` request property.
This works by using a default handler for the request by the links that forward it.
#### requestRequired
By default all properties defined in `requestProperties` are required to be present in the payload for the request.
This can be overriden by specifying an array of properties which are required here.
This is equivalent to using the `required` keyword in JSON schema.
#### requestProperties
Object with properties mapping to a JSON schema of that property that specifies what's valid to send in the request.
This is equivalent to using the `properties` keyword in JSON schema, except that the properties specified are by default required and additional properties are not allowed.
See [this guide][guide] for an introduction to writing JSON schemas
#### responseRequired
Same as the requestRequired only for the response sent back by the target.
#### responseProperties
Same as the requestProperties only for the response sent back by the target.
## Sending Link Messages
Link messages are sent by calling the `.send()` method on the Event/Request instance with the link you want to send it over and the data to send.
For `InstancePlugin` code the link to the host is the `instance` itself, which is accessible through the `.instance` property of the `InstancePlugin`.
The `.info` property of the plugin class exposes the data exported from the plugin's `plugin` export.
For the Request class the send method is async and returns the response data received from the target it was sent to, or throws an error if the request failed.
### Handling connection events
There are a few connection related events that plugins neeed to repsond to in order to avoid data loss and connection problems.
The most important is the prepare disconnect for the link between controller and host.
This is signaled to `ControllerPlugin` classes via the `onPrepareHostDisconnect` hook and to `InstancePlugin` classes via the `onPrepareControllerDisconnect` hook.
After the prepare disconnect the connection will be closed, which will result in pending requests and events being dropped.
Plugins must respond to the prepare disconnect by stopping any processess it does that send events or requests over the link in question.
This can be accomplished either through listening for the prepare disconnect hook, or by checking the `connected` property of the `HostConnection` class and `Host` class on the controller and host respectively.
For example the sending of an event from an `InstancePlugin` class can be stopped while the connection is not connected, not in the dropped state, and not in the process of discunnecting by using the following code:
If the event or request needs to be sent to the controller it can be put into a queue stored on the plugin instance and sent out when the connection is established again.
The re-establishement of the connection is notified to plugins via the `connect` event to the `onControllerConnectionEvent` and `onHostConnectionEvent` hooks.
The second connection event which is of lesser importance to respond to is the `drop` connection event served through `onControllerConnectionEvent` for `InstancePlugin` classes and through `onHostConnectionEvent` for `ControllerPlugin` classes.
This is raised when the connection between the controller and host in question is lost, most likely due to networking issues.
When in the dropped state the host will keep trying to reconnect to the controller in order to re-establish it, and if successful no events or requests will be lost.
However while in the dropped state any requests and events sent gets queued up in memory until the connection is either re-established or the session times out.
This means that if your plugin sends a lot of events or requests, they can end up being queued up in a buffer and sent out all at once the connection is re-estabished.
To avoid this you should be throtteling and/or stopping your requests/events after `drop` has been raised, and continue back as normal when `resume` is raised.
## Collecting Statistics
Clusterio comes with its own Prometheus client implementation, one part due to Not Invented Here and another part due to collectors in prom-client being difficult to get to work nicely with collecting data from plugins optionally loaded at runtime on different computers.
In its simplest form collecting data from plugins consists of defining the metric and updating it somewhere in the plugin code.
For example:
```js
const{Counter}=require("@clusterio/lib");
constfooMetric=newCounter(
"clusterio_foo_frobber_foo_metric","Measures the level of foo",
);
// Somewhere in the controller plugin code
fooMetric.inc();
```
This works for controller plugins, and the metric will be automatically available through the /metric HTTP endpoint.
It's recommended that plugin metrics follow `clusterio_<plugin_name>_<metric_name>` as the naming scheme.
For metrics that are per-instance, you must define an `instance_id` label and set it accordingly, for example:
Metrics are automatically registered to the default registry, and this default registry is automatically polled by the controller on hosts.
This means that it's important that you place the definition of the metric at module level so that it's not created more than once over the lifetime of a host.
Since the metrics remember their values and would continue to be exported after an instance is shutdown, there's code at instance shutdown that removes all the values where the `instance_id` label matches the id of the instance shut down.
For statistics you need to update on collection there's an `onMetrics` hook on both controller and instance plugins that is run before the metrics in the default registry are collected.
## Adding Custom Commands to clusterioctl
The control entrypoint for plugins allows you to extend clustectl with your own commands.
The creation of custom commands typically starts with defining a command tree for the plugin:
For a command the `definition` is the arguments to pass to [yargs.command](http://yargs.js.org/docs/#api-reference-commandcmd-desc-builder-handler) (see also [yargs.positional](http://yargs.js.org/docs/#api-reference-positionalkey-opt) and [yargs.options](http://yargs.js.org/docs/#api-reference-optionskey-opt) for setting up positional and optional arguments to commands).
The `handler` is an async function that's invoked when the command is executed and it's passed the parsed command line arguments and a reference to the `Control` class of clusterioctl.
It's possible to optain a reference to the plugin class with `control.plugins.get(info.name)`.
Note that messages sent from clusterioctl needs to have `"control-controller"` as a part of the links array for it to be accepted by the controller, see [Defining Link Messages](#defining-link-messages) for how to define the messages that can be sent to the controller.
To have the command tree become part of clusterioctl it needs to be added to the rootCommand tree in `addCommands` callback of the Ctl plugin:
Functions frequently needed from different places in the cluster. Ex:
* Loading Lua scripts
* Checking if hotpatch is installed
* Getting an instance's displayname from their ID
*/
module.exports=({config}={})=>({
/**
async function to get Lua from a file, remove comments, minify and return.
arg1 is the file path
arg2 is whether or not to compress the Lua. Makes debugging harder, but reduces bandwidth when many players are online. Recommended for polling commands. Defaults to true.
Promise resolves with a string of Lua.
*/
getLua:require("./getLua.js"),
/**
async function to check whether hotpatch is installed in a server.
First argument is the servers async messageInterface for rcon return values
//Numreq - number of items that are requested for item X for slave Y
//Instore - number of items for item X in master gauge (currently)
//store_last_tick - Number of items for item X in master gauge at time of last tick
//dole - Current dole for item X
//carry - carry for item X slave Y
//prev_req - Numreq for previous request for item X slave Y
//* doleinfo_last - Object with 2 values for last "tick period": numreq - total number or requests for item, numslave - number of slaves requesting that item
res_capped=res_capped||0;//Safeguard against wrong values
debt=res1-res_capped;
debt_cap=Math.round(numreq*1.5)+5;
if(debt>debt_cap)debt=debt_cap;
carry=sigmoid0(outputs[2]*carry+outputs[0])||0;
return[res_capped,dole+outputs[1],carry,debt];
}
//TICK - Updates values for item X, should be called once for each item for each round of all slaves requesting that item
//Instore - number of items for item X in master gauge (currently)
//dole - Current dole for item X
//store_last_tick - Number of items for item X in master gauge at time of last tick (not this one)
//* doleinfo_last - Object with 2 values for last "tick period": numreq - total number or requests for item, numslave - number of slaves requesting that item
//* doleinfo_new - Object with 2 values for new "tick period": numreq - total number or requests for item, numslave - number of slaves requesting that item
//Returns: Array of 2 elements
//0.New dole for item X
//1.Guessed percentage average of items supplied in last 10 ticks
it("Returns false if input is not a factorio command",function(){
assert(isFactorioCommand({})==false,"This is not even a string so should return false");
assert(isFactorioCommand("Howdy boys!")==false,"This is not a command so should return false");
assert(isFactorioCommand("/c this ain't a command because its invalid LUA")==false,"This is not valid LUA so it should return false");
assert(isFactorioCommand("Trying to trick the system /c game.print('Hello world!')")==false,"this is still not a command");
});
it("Returns true if it is a valid command",function(){
assert(isFactorioCommand("/c game.print('hello world')"),"This is a valid command, should output true not false")
});
it("Fails to understand that leading spaces are valid",function(){
assert(isFactorioCommand(" /c game.print('Hello world!')")==false,"This should be true but often returns false unintentionally. If you see this you probably fixed the bug so change this test, OK?")
/*let res = syncRequest('GET', 'https://api.github.com/repos/Danielv123/factorioClusterioMod/releases', {"headers":{"User-Agent":"Fuck you for requiring user agents!"}});
let url = JSON.parse(res.getBody())[0].assets[0].browser_download_url;
let name = JSON.parse(res.getBody())[0].assets[0].name;
console.log(JSON.parse(res.getBody()))
*/
});
},
/**
Searches mods.factorio.com for mods matching a search term and prints
if(e.contains("Insufficient information"))console.log("We are missing some parameters required for logging in! Make sure your factorio username and token are specified in config.json.");
reject(newError(e));
});
});
},
addMod:function(term,instance){
returnnewPromise((resolve,reject)=>{
if(instance=="shared"){// modDirectory has to end in /
console.log(" Downloaded in "+moment.utc(moment.duration(Date.now()-downloadStartTime).as('milliseconds')).format('mm:ss'));// because sLog.clear() doesn't seem to do the job of ending the line properly
resolve();
});
});
}
}else{
// treat it as a search and download the top result
let res = syncRequest('GET', 'https://api.github.com/repos/Danielv123/factorioClusterioMod/releases', {"headers":{"User-Agent":"Fuck you for requiring user agents!"}});
let url = JSON.parse(res.getBody())[0].assets[0].browser_download_url;
let name = JSON.parse(res.getBody())[0].assets[0].name;
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
]]
-- MIT License, https://opensource.org/licenses/MIT
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
]]
-- MIT License, https://opensource.org/licenses/MIT
local hotpatch_tools = require 'hotpatch.mod-tools'
--load private API
-- mod installation/uninstallation support functions
-- These take a mod NAME as a first argument
local install_mod = hotpatch_tools.install_mod
local find_installed_mod = hotpatch_tools.find_installed_mod
local install_mod_file = hotpatch_tools.install_mod_file
local uninstall_mod = hotpatch_tools.uninstall_mod
-- mod interaction functions
-- These take a LOADED INDEX as a first argument, except load_mod, which takes an INSTALLED INDEX
local load_mod = hotpatch_tools.load_mod
local find_loaded_mod = hotpatch_tools.find_loaded_mod
local run_mod = hotpatch_tools.run_mod
local reset_mod = hotpatch_tools.reset_mod
local reset_mod_events = hotpatch_tools.reset_mod_events
local register_mod_events = hotpatch_tools.register_mod_events
local unload_mod = hotpatch_tools.unload_mod
-- internal callbacks when a mod registers events
local register_event = hotpatch_tools.register_event
local register_nth_tick = hotpatch_tools.register_nth_tick
local register_on_tick = hotpatch_tools.register_on_tick
-- mod bootstrap functions
-- These take a LOADED INDEX as a first argument
local mod_on_init = hotpatch_tools.mod_on_init
local mod_on_load = hotpatch_tools.mod_on_load
local mod_on_configuration_changed = hotpatch_tools.mod_on_configuration_changed
local static_mods = hotpatch_tools.static_mods
local console = hotpatch_tools.console
local debug_log = hotpatch_tools.debug_log
local loaded_mods = hotpatch_tools.loaded_mods
local installed_mods = hotpatch_tools.installed_mods
local help_commands = {
help = '/hotpatch help\nYou got this far, you probably know all you need to know',
list = '/hotpatch list [type]\nLists \'installed\', \'loaded\' or \'running\' mods',
}
local sub_commands
sub_commands = {
help = function(player_index, param)
local caller = (player_index and game.players[player_index]) or console
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
]]
-- MIT License, https://opensource.org/licenses/MIT
local hotpatch_tools = require 'hotpatch.mod-tools'
local mod_gui = require 'mod-gui'
--load private API
-- mod installation/uninstallation support functions
-- These take a mod NAME as a first argument
local install_mod = hotpatch_tools.install_mod
local find_installed_mod = hotpatch_tools.find_installed_mod
local install_mod_file = hotpatch_tools.install_mod_file
local uninstall_mod = hotpatch_tools.uninstall_mod
-- mod interaction functions
-- These take a LOADED INDEX as a first argument, except load_mod, which takes an INSTALLED INDEX
local load_mod = hotpatch_tools.load_mod
local find_loaded_mod = hotpatch_tools.find_loaded_mod
local run_mod = hotpatch_tools.run_mod
local reset_mod = hotpatch_tools.reset_mod
local reset_mod_events = hotpatch_tools.reset_mod_events
local register_mod_events = hotpatch_tools.register_mod_events
local unload_mod = hotpatch_tools.unload_mod
-- internal callbacks when a mod registers events
local register_event = hotpatch_tools.register_event
local register_nth_tick = hotpatch_tools.register_nth_tick
local register_on_tick = hotpatch_tools.register_on_tick
-- mod bootstrap functions
-- These take a LOADED INDEX as a first argument
local mod_on_init = hotpatch_tools.mod_on_init
local mod_on_load = hotpatch_tools.mod_on_load
local mod_on_configuration_changed = hotpatch_tools.mod_on_configuration_changed
local static_mods = hotpatch_tools.static_mods
local console = hotpatch_tools.console
local debug_log = hotpatch_tools.debug_log
local loaded_mods = hotpatch_tools.loaded_mods
local installed_mods = hotpatch_tools.installed_mods
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
]]
-- MIT License, https://opensource.org/licenses/MIT
local hotpatch_tools = require 'hotpatch.mod-tools'
--load private API
-- mod installation/uninstallation support functions
-- These take a mod NAME as a first argument
local install_mod = hotpatch_tools.install_mod
local find_installed_mod = hotpatch_tools.find_installed_mod
local install_mod_file = hotpatch_tools.install_mod_file
local uninstall_mod = hotpatch_tools.uninstall_mod
-- mod interaction functions
-- These take a LOADED INDEX as a first argument, except load_mod, which takes an INSTALLED INDEX
local load_mod = hotpatch_tools.load_mod
local find_loaded_mod = hotpatch_tools.find_loaded_mod
local run_mod = hotpatch_tools.run_mod
local reset_mod = hotpatch_tools.reset_mod
local reset_mod_events = hotpatch_tools.reset_mod_events
local register_mod_events = hotpatch_tools.register_mod_events
local unload_mod = hotpatch_tools.unload_mod
-- internal callbacks when a mod registers events
local register_event = hotpatch_tools.register_event
local register_nth_tick = hotpatch_tools.register_nth_tick
local register_on_tick = hotpatch_tools.register_on_tick
-- mod bootstrap functions
-- These take a LOADED INDEX as a first argument
local mod_on_init = hotpatch_tools.mod_on_init
local mod_on_load = hotpatch_tools.mod_on_load
local mod_on_configuration_changed = hotpatch_tools.mod_on_configuration_changed
local static_mods = hotpatch_tools.static_mods
local console = hotpatch_tools.console
local debug_log = hotpatch_tools.debug_log
local loaded_mods = hotpatch_tools.loaded_mods
local installed_mods = hotpatch_tools.installed_mods
local remote_interface = {}
--IMPORTANT: WHEN COPY PASTING CODE TO CONSOLE THE CODE MUST HAVE SINGLE LINE COMMENTS REMOVED
-- FACTORIO CONSOLE STRIPS LINEFEEDS WHICH MAKES ALL THE CODE BECOME COMMENTED OUT
--TODO: gate these behind admin permissions; if (game.player and game.player.admin) should work for console commands?
msg-intro=Hierdie is Factorio vryspel. Jou taak is om 'n vuurpyl in die ruimte in te stuur. Doen dit deur 'n vuurpyl te bou en hom te lanseer. Jy sal baie gevorderde tegnologiëe moet navors om die Vuurpyl Silo te ontsluit. Begin klein, werk jou pad op met outomatisering en moenie vergeet om jouself te beskerm teen die inboorlinge nie.
msg-intro=Гэта рэжым свабоднай гульні Factorio. Вашая мэта - запусціць ракету ў космас. Дзеля гэтага вам трэба пабудаваць ракетную шахту і саму ракету са спадарожнікам. Вы мусіце даследаваць перадавыя тэхналогіі, каб навучыцца будаваць ракетную шахту. Пачынайце з малога, працуйце над аўтаматызацыяй вытворчасці, даследуйце новыя тэхналогіі і не забывайцеся на абарону ад «тутэйшых».
scenario-name=Свабодная гульня
Loaded 100 of 4257 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Hornwitser
