factorio-clusterio/docs/managing-a-cluster.md

Managing a Cluster

Clusterio clusters are managed through the controller by using the clusterioctl command line interface which is invoked by running npx clusterioctl <command> in the clusterio directory. This document uses the shorthand ctl> foo to indicate npx clusterioctl foo should be executed. Mandatory parameters are shown in <angles bracket> and optional pameters are in [square brackets].

Before clusterioctl can be used it needs to be configured for the cluster it will connect to. The easiest way to do this is to run npx clusteriocontroller bootstrap create-ctl-config <username> on the controller, which creates the necessary config-ctl.json for managing the cluster as the given user.

Hosts

List hosts

ctl> host list

Lists all hosts.

Hosts are added to the cluster by running the installer on the computer the host runs on and approving the code it shows, see Setting up remote hosts. Listing, approving and rejecting hosts waiting for approval needs the core.host.approve permission.

List pending hosts

ctl> host pending

Lists hosts waiting for approval with the code they show, the name they asked for, the address they connected from and how long they have been waiting. A request goes away when its host stops waiting, and after the time set by controller.host_pairing_timeout, an hour by default, after which the host asks again with a new code.

Approve host

ctl> host approve <code> [--name <host name>]

Approves the host showing the code and prints the id of the new host. The code is not case sensitive and the dash can be left out. --name gives the host a different name than the one it asked for. Approving gives the host full access to the cluster, check with host pending that the name and address match the computer you are setting up.

Reject host

ctl> host reject <code>

Rejects the host showing the code, the host then exits with an error.

Revoke host

ctl> host revoke <host>

Specify the host by name or ID. Removes the host from the cluster, disconnects it and makes its token stop working. Instances assigned to the host become unassigned, their files stay on the computer. Requires the core.host.revoke permission. To add the computer back to the cluster, start the host again and approve it.

Update Clusterio

ctl> host update <host> [--all] [--restart]

Specify the host by name or ID. If remote updates are enabled on the host, this will attempt to update @clusterio/host on the machine. Add --all to also update all npm-installed plugins. Changes will not be applied until the host is restarted. If the --restart flag is provided, the host will automatically restart once the update is complete.

Remote updates are enabled by default but can be disabled on a per-machine basis at the owner's discretion. For security reasons, this configuration setting cannot be changed remotely. To modify it, run the following command locally while the host is offline:

host> config set host.allow_remote_updates true/false

The controller also supports remote updates, following the same logic. The corresponding commands are:

ctl> controller update [--all] [--restart]
controller> config set controller.allow_remote_updates true/false

Mark restart required

ctl> host mark-restart-required <host>

Marks the host as requiring a restart, the same way a plugin update does. Use it after installing or changing a plugin by other means than host plugin install, such as a plugin checked out into external_plugins. The mark shows up in the web interface on the next system metrics poll. The controller has the same command:

ctl> controller mark-restart-required

Instances

List Instances

ctl> instance list

List instances.

Create instance

ctl> instance create <name> [--id <instance id>]

Create a new instance. name is the instance name. --id is the optional id.

Kill instance

ctl> instance kill <name>

Terminates the running Factorio server without giving it a chance to save or cleanup after itself.

Note: This may cause loss of data.

Save game

ctl> instance save-game <name>

Saves the running game of the instance to the save it was started from and waits for the save to finish.

Extract players from instance

ctl> instance extract-players <name>

Creates a user account in the cluster for each player that has been online on the currently running save of the instance and sets the online time of that account on this instance to the online time recorded in the save.

Useful when importing a save to a cluster. Note that the online time in a save is recorded differntly from how Clusterio records online time. Most notably Clusterio records wall clock time while the save time is recorded in ticks and this extraction of player time assumes the game runs at 60 ticks a second.

Config management

List Configuration

ctl> instance config list <instance>

List config values for an instance. instance is the instance whose config values to list.

Set config value

ctl> instance config set <instance> <field> [value] [--stdin]

Set field in instance config. instance is instance to set. field is field to set. value is value to set in field. --stdin signals to read value from stdin.

Set config property of field

To be written.

Edit instance config

ctl> instance config edit <instance> [editor]

Edit instance config in a text editor. Once file is saved and editor has been exited, config values will be read back in. instance is the instance to edit, editor is the editor to use. If editor is not given, clusterioctl will read from the EDITOR or VISUAL enviroment variables.

Assign instance to host

ctl> instance assign <instance> [host]

Assigns instance to host. instance is instance to assign, host is host to assign to. If host is not provided, instance will be unassigned.

Users

Clusterio automatically creates user accounts for all players that join an instance when save patching is enabled. These accounts are used to store per player data shared between instances, like if the player should be whitelisted, admin or banned.

List users

ctl> user list

Lists all user accounts in the cluster along with some data for each.

Create User

ctl> user create <name>

Creates a new empty user account for the given Factorio player. Note that case matters here. This is usually not required as user accounts are created automatically for players that join instances with save patching enabled.

Promote User to Server Admin

ctl> user set-admin <name> [--revoke] [--create]

Promotes the user given to in-game admin on instances with the sync_adminlist option enabled. If the --revoke switch is used the user is removed from the adminlist.

Since admin status is a part of the account data the account must exist for this to succeed, passing --create will create the account if it does not exist.

Note: Being a server admin does not grant any access to manage the cluster. See set-roles for adding roles which grant access to managing the clusetr.

Whitelist User

ctl> user set-whitelisted <name> [--remove] [--create]

Add the user given to the whitelist on instances with the sync_whitelist option enabled. If the --remove switch is used the user is removed from the whitelist.

Since whitelisted status is a part of the account data the account must exist for this to succeed, passing --create will create the account if it does not exist.

Ban User

ctl> user set-banned <name> [--reason <message>] [--pardon] [--create]

Ban user in-game from instances with the sync_banlist option enabled. Reason is a message that will be shown to the user when attemption to log in. If the --pardon switch is used it removes the ban.

Since ban status is a part of the account data the account must exist for this to succeed, passing --create will create the account if it does not exist.

Note: This bans the user from logging in to Factorio servers in the cluster, it does not revoke access to any cluster management they might have, see next section on setting roles for revoking tha.

Bulk importing admin, whitelist and ban lists

Whitelists and admin lists can be bulk imported with the following commands:

ctl> user whitelist-bulk-import <file> [--create]
ctl> user admin-bulk-import <file> [--create]

The file should simply be newline seperated names:

joe
bob
allan

The --create option will create the user if they do not exist, and then set the relevent value.

Banlists can be bulk imported with the following command:

ctl> user banned-bulk-import <file> [--create]

The file should be names, followed by a ban reason:

joe greifed copper mines
bob stole materials
allan rude language

The first string on the line is used as the username, the others are used as the ban reason. The --create option will create the user if they do not exist, then ban them.

Set Cluster Roles

ctl> user set-roles <name> [roles...]

Replaces the roles the user has in the cluster with a new list of roles. Calling with an empty roles argument will remove all roles from the user.

By default there's a roled named Cluster Admin which grants access to everything and a role named Player which grants a limited read access to the cluster, see the section on roles for more information about setting up roles and permissions.

Delete user

ctl> user delete <name>

Deletes everything stored on the controller for this user.

Note: If the player is banned from the cluster this will effectively unban them, as the ban status is stored with the user account.

Note: If the player joins the cluster again a new account will be made for them automatically.

Roles

To be written.

Notes

Free text notes can be attached to hosts, instances, users, roles, mod packs, mods and saves. They are stored on the controller and shown to everyone with access to the resource in the web interface, and can for example be used to record who runs a host or what an instance is meant for.

The web interface renders notes as Markdown. A mention of the form @<type>/<id> becomes a link to that resource showing its name, for example @instance/778528005 or @user/cooldude2606. This works for hosts, instances, users, roles and mod packs, and the id is shown if the name is not available. The note itself stores the mention as written.

List notes

ctl> note list [--type <type>]

Lists all notes, optionally limited to one resource type.

Set note

ctl> note set <type> <id> <content>

Sets the note of a resource. The type is one of host, instance, user, role, mod_pack, mod or save. The id is the numeric id for hosts, instances, roles and mod packs, the lower case name for users, <name>_<version> for mods and <instance id>/<file name> for saves.

Delete note

ctl> note delete <type> <id>

Removes the note of a resource. Notes are also removed when the resource they are attached to is deleted.

Plugins (Local)

To be written.

Plugins (Remote)

The installer asks "Allow remote updates and plugin installs from the web interface?" and the answer sets allow_remote_updates, allow_plugin_updates and allow_plugin_install together on that machine, pass --remote-npm or --no-remote-npm to answer it from the command line. Remote updates allow ctl and the web UI to update Clusterio and plugins on machines within the cluster. Remote installs allow them to install any package from npm onto the machine, which is a risk of arbitrary code execution through malicious npm packages, so only allow them if you trust everyone with access to install plugins. In a config created without the installer remote updates are enabled and remote installs are disabled by default. See below for changing these settings after installing.

List

ctl> controller plugin list
ctl> host plugin list <host>

This command lists all plugins currently known to the remote. Newly installed plugins will not be displayed until the remote is restarted. Plugins that are distributed as an npm package can be updated remotely.

Update

ctl> controller plugin update <name>[@<version>] [--restart]
ctl> host plugin update <host> <name>[@<version>] [--restart]

If remote plugin updates are enabled on the remote, this will execute npm update --save <name>, which stays within the version range saved in the remote's package.json. To update to a specific version, including one outside that range, pass it as <name>@<version>, for example @clusterio/plugin-example@2.1.0. The version must be published on npm, and the remote then executes npm install --save <name>@<version>. The plugin must already be installed on the remote, and only remote plugin updates need to be enabled. Changes will not be applied until the remote is restarted. If the --restart flag is provided, the remote will automatically restart once the update is complete.

Update All

ctl> controller update --all [--restart]
ctl> host update <host> --all [--restart]

Updates the Clusterio package and all plugins installed from npm in a single npm update --save command. Updating everything at once avoids ending up with multiple versions of @clusterio/lib in the installation. Requires both remote updates and remote plugin updates to be enabled on the remote.

Install

ctl> controller plugin install <name>[@<version>] [--restart]
ctl> host plugin install <host> <name>[@<version>] [--restart]

If remote plugin installs are enabled on the remote, this will execute npm install --save <name>. Pass <name>@<version> to install a specific version, for example to match the version used by the rest of the cluster. The package, and the version if given, must be published on npm. Changes will not be applied until the remote is restarted. If the --restart flag is provided, the remote will automatically restart once the update is complete.

Search

ctl> controller plugin search [query] [--page <n>] [--page-size <n>]

Searches the npm registry for packages tagged with the clusterio-plugin keyword. The search is done by the controller and does not require remote installs to be enabled. The web UI has a plugin browser on the plugins page that uses the same search.

Concurrent Operations

Only one npm operation runs at a time on a machine. A standalone controller and host share their working directory, so the operation holds the npm-operation.lock file there while npm runs. Other installs and updates on the same machine or installation are rejected with "Another package operation is already in progress on this machine" until it finishes. A lock file left behind by a process that is no longer running is removed automatically.

Enable/Disable Updates

host> config set host.allow_plugin_updates true/false
controller> config set controller.allow_plugin_updates true/false

Remote updates are enabled by default but can be disabled on a per-machine basis at the owner's discretion. For security reasons, this configuration setting cannot be changed remotely. To modify it, run the above commands locally while the remote is offline.

Enable/Disable Installs

host> config set host.allow_plugin_install true/false
controller> config set controller.allow_plugin_install true/false

Remote installs are disabled by default but can be enabled on a per-machine basis at the owner's discretion. For security reasons, this configuration setting cannot be changed remotely. To modify it, run the above commands locally while the remote is offline.

Prometheus Metrics

The controller exports metrics for the whole cluster on the /metrics endpoint. Fetching it requires a token for a user with the core.controller.metrics permission, sent either as an Authorization: Bearer <token> header or as an X-Access-Token header. Requests without a valid token get a 401 response and requests for users lacking the permission get a 403 response.

Create a role with just this permission, a user to hold it, and generate a token for that user. Give the user a name that is not a valid Factorio player name, such as one starting with #, so a player joining the cluster can never claim the account and its token:

ctl> role create Prometheus --permissions core.controller.metrics
ctl> user create #prometheus
ctl> user set-roles #prometheus Prometheus
$ npx clusteriocontroller bootstrap generate-user-token '#prometheus'

Then pass the token in the scrape job for the controller in prometheus.yml:

scrape_configs:
  - job_name: clusterio
    authorization:
      credentials: <token>
    static_configs:
      - targets: ["localhost:8080"]

Concurrent scrapes share a single round of metrics gathering from the hosts, so multiple scrapers polling the controller do not multiply the load on the hosts.