v2.0.0-alpha.16

When messages were forwarded they were sent using the low level
connector .send method which accepts a message and sends it over the
link.  This method sends the message as is with no modification which
means that at no point during the forwarding logic was the message
sequence number that originated from Link A renumbered to the correct
sequence number to send out on Link B which it was forwarded on.

This in turn caused the logic which drops acked messages from the send
buffer to break which in turn meant the resume could cause duplication
of message and/or messages to be lost.  With enough duplicated messages
the resume of one connection could cause other connections to drop and
also initiate resume.

Fix by renumbering the sequence on forwarded messages.  In the future
the sequences sent and received should also be checked to make sure this
kind of problem does not happen again, this is not done here because
partially updating a running cluster with sequence problems to one that
checks and enforces sequences numbers will be problematic.

.eslintrc.js

@@ -35,6 +35,7 @@ module.exports = {
 			"rules": {
 				"prefer-arrow-callback": "off",
 				"no-invalid-this": "off",
+				"valid-jsdoc": "off",
 				"node/no-unpublished-import": "off",
 				"node/no-unpublished-require": "off",
 			},

CHANGELOG.md

@@ -1,5 +1,33 @@
 # Changelog
 
+## Version 2.0.0-alpha.16
+
+## Changes
+
+- Fixed message sequences not being renumbered when forwarded between links causing reconnects to send duplicated messages.
+
+
+## Version 2.0.0-alpha.15
+
+### Features
+
+- Added support for `color-settings` Mod settings [#568](https://github.com/clusterio/clusterio/issues/568).
+- Added `clusterio_controller_pending_requests` and `clusterio_host_pending_requests` metrics.
+
+### Changes
+
+- Fixed generated systemd file for clusteriohost not enabling source maps.
+- Fixed ctl not exporting types used for adding commands to ctl plugins.
+- Fixed controller only plugins throwing an error about missing a config field on startup out when added to hosts.
+- Fixed installer incorrectly claiming the options --factorio-dir and --no-download-headless conflicts with each other.
+- Fixed a memory leak on every request sent over the WebSocket.
+- Fixed assertion error on closing or invalidating a link with messages forwarded from a virtual link.
+- Fixed host hanging if stopped while an instance is in the process of creating a save during startup.
+- Fixed `instance.auto_start` option.
+- Fixed host log spammed with errors when plugins broadcast messages to instances with the plugin disabled.
+- Fixed properties of object config fields not being possible to remove.
+
+
 ## Version 2.0.0-alpha.14
 
 This release does not retain backwards compatibility with 2.0.0-alpha.13 meaning all parts of the cluster needs to updated at the same time.

README.md

@@ -39,6 +39,7 @@ If you are starting a new cluster, it's highly recommended to use the 2.0 alpha.
 * [Running Clusterio](#running-clusterio)
 * [Setting up remote hosts](#setting-up-remote-hosts)
 * [Setting up clusterioctl](#setting-up-clusterioctl)
+* [Known issues](#known-issues)
 * [Common problems](#Common-problems)
 
 
@@ -265,6 +266,20 @@ If you want to use a different user for the command line interface, you can gene
     npx clusteriocontroller bootstrap generate-user-token <username>
 
 
+## Known Issues
+
+- When the `server_select` plugin is installed you may see `[error] Unhandled event server_select:UpdateInstancesEvent` logged when starting an instance.
+  This should be ignored as it does not indicate an error occured.
+- After migrating from Alpha 13 to Alpha 14 running an instance data export fails with the error `Upload failed: 401 Unauthorized` due to the host token being issued to a slave.
+  To workaround this find the id of the host in the host config and generate a new host token with the same id.
+  Stop the host and replace the value of the `host.controller_token` field in `config-host.json` with the new host token before starting it up again.
+- After migrating from Alpha 13 to Alpha 14 you get kicked out from the web interface due to the master to controller rename.
+  If you don't have the authentication token readily available you can manually migrate the token by opening the Inspect/Developer Tools on the page and running the following piece of JavaScript in the Console:
+  ```
+  localStorage.controller_token = localStorage.master_token
+  ```
+
+
 ## Common problems
 
 ### EACCESS [...] LISTEN 443

docs/migrating-from-2.0.0-alpha.13.md

@@ -17,6 +17,8 @@ Mod Pack strings looks something like this:
 
 If you have multiple mod configuration create a mod pack string for each and note them down.
 
+Double check that you are on a supported version of Node.js by running `node --version`, the version should be v18 or later, if not you will need to update Node.js before continuing.
+
 
 ## Migrate installations
 
@@ -24,7 +26,7 @@ If you have multiple mod configuration create a mod pack string for each and not
 
 Shutdown all slaves and the master server and then migrate each installations by running the following command:
 
-    npm exec @clusterio/create --migrate-rename --log-level=verbose
+    npm exec @clusterio/create -y -- --migrate-rename --log-level=verbose
 
 This will convert the existing installation's usage of the master and slave names in packages, configs, database and logs to the new controller and host terms and update the installed packages.
 

packages/controller/controller.ts

@@ -37,6 +37,24 @@ import { version } from "./package.json";
 // globals
 let controller: Controller;
 
+void new lib.Gauge(
+	"clusterio_controller_pending_requests",
+	"Count of pending link requests currently waiting in memory on the controller.",
+	{
+		callback: (gauge) => {
+			let count = 0;
+			if (controller) {
+				for (const host of controller.wsServer.hostConnections.values()) {
+					count += host.pendingRequestCount;
+				}
+				for (const control of controller.wsServer.controlConnections.values()) {
+					count += control.pendingRequestCount;
+				}
+			}
+			gauge.set(count);
+		},
+	}
+);
 
 void new lib.Gauge(
 	"clusterio_controller_host_mapping",

packages/controller/index.ts

@@ -1,7 +1,9 @@
 import { bootstrap } from "./controller";
 export { default as Controller } from "./src/Controller";
 export { default as ControllerUser } from "./src/ControllerUser";
+export { default as ControllerRouter } from "./src/ControllerRouter";
 export { default as ControlConnection } from "./src/ControlConnection";
+export { default as HostConnection } from "./src/HostConnection";
 export { default as InstanceInfo } from "./src/InstanceInfo";
 export { default as BaseControllerPlugin } from "./src/BaseControllerPlugin";
 export { default as UserManager } from "./src/UserManager";

packages/controller/package.json

@@ -3,7 +3,7 @@
 	"description": "Implementation of Clusterio controller",
 	"author": "Hornwitser",
 	"license": "MIT",
-	"version": "2.0.0-alpha.14",
+	"version": "2.0.0-alpha.16",
 	"main": "dist/index.js",
 	"scripts": {
 		"prepare": "tsc --build && webpack-cli --env production",
@@ -19,7 +19,7 @@
 		"node": ">=18"
 	},
 	"dependencies": {
-		"@clusterio/lib": "^2.0.0-alpha.14",
+		"@clusterio/lib": "^2.0.0-alpha.16",
 		"@sinclair/typebox": "^0.30.4",
 		"busboy": "^1.6.0",
 		"compression": "^1.7.4",
@@ -35,7 +35,7 @@
 		"yargs": "^17.7.2"
 	},
 	"devDependencies": {
-		"@clusterio/web_ui": "^2.0.0-alpha.14",
+		"@clusterio/web_ui": "^2.0.0-alpha.16",
 		"@types/busboy": "^1.5.0",
 		"@types/compression": "^1.7.2",
 		"@types/express": "^4.17.17",

packages/controller/src/BaseConnection.ts

@@ -6,7 +6,6 @@ import type WsServerConnector from "./WsServerConnector";
 
 import * as lib from "@clusterio/lib";
 const { logger } = lib;
-import ControllerRouter from "./ControllerRouter";
 import * as routes from "./routes";
 
 
@@ -24,7 +23,7 @@ export default class BaseConnection extends lib.Link {
 		public _controller: Controller
 	) {
 		super(connector);
-		this.router = new ControllerRouter(this._controller);
+		this.router = _controller.router;
 		for (let [Request, handler] of this._controller._registeredRequests) { this.handle(Request, handler); }
 		for (let [Request, handler] of this._controller._fallbackedRequests) { this.fallbackRequest(Request, handler); }
 		for (let [Event, handler] of this._controller._registeredEvents) { this.handle(Event, handler); }

packages/controller/src/Controller.ts

@@ -26,6 +26,7 @@ import WsServer from "./WsServer";
 import HostConnection from "./HostConnection";
 import HostInfo from "./HostInfo";
 import BaseControllerPlugin from "./BaseControllerPlugin";
+import ControllerRouter from "./ControllerRouter";
 
 const endpointDurationSummary = new Summary(
 	"clusterio_controller_http_endpoint_duration_seconds",
@@ -74,6 +75,7 @@ export default class Controller {
 
 	/** WebSocket server */
 	wsServer: WsServer;
+	router = new ControllerRouter(this);
 	trustedProxies: BlockList;
 	debugEvents: events.EventEmitter = new events.EventEmitter();
 	private _events: events.EventEmitter = new events.EventEmitter();
@@ -1374,7 +1376,11 @@ export default class Controller {
 				}
 
 			} else if (dst.id === lib.Address.instance || dst.id === lib.Address.host) {
+				const plugin = event.constructor.plugin;
 				for (let hostConnection of this.wsServer.hostConnections.values()) {
+					if (plugin && !hostConnection.plugins.has(plugin)) {
+						continue;
+					}
 					hostConnection.sendEvent(event, dst);
 				}
 

packages/controller/src/ControllerRouter.ts

@@ -7,8 +7,6 @@ import ControlConnection from "./ControlConnection";
 const { logger } = lib;
 
 export default class ControllerRouter {
-	pendingRequests: Map<any, any> = new Map();
-
 	constructor(
 		public controller: Controller
 	) {}
@@ -16,12 +14,14 @@ export default class ControllerRouter {
 	/**
 	 * @param origin - Source link of the message.
 	 * @param message - Link the message originated from.
+	 * @param entry - message entry if message is a request or an event.
 	 * @param hasFallback - true if fallback handling is available.
 	 * @returns true if the message was handled
 	 */
 	forwardMessage(
 		origin: ControlConnection | HostConnection,
 		message: lib.MessageRoutable,
+		entry: lib.RequestEntry | lib.EventEntry | undefined,
 		hasFallback: boolean
 	): boolean {
 		if (!["request", "response", "responseError", "event"].includes(message.type)) {
@@ -32,7 +32,7 @@ export default class ControllerRouter {
 		let nextHop: ControlConnection | HostConnection | undefined;
 		let msg: string = "";
 		if (dst.type === lib.Address.broadcast) {
-			this.broadcastMessage(origin, message);
+			this.broadcastMessage(origin, message, entry as lib.EventEntry);
 			return true;
 		} else if (dst.type === lib.Address.host) {
 			nextHop = this.controller.wsServer.hostConnections.get(dst.id);
@@ -84,7 +84,7 @@ export default class ControllerRouter {
 			if (message.type === "request") {
 				nextHop.forwardRequest(message as lib.MessageRequest, origin);
 			} else {
-				nextHop.connector.send(message);
+				nextHop.connector.forward(message);
 			}
 		} else {
 			this.warnUnrouted(message, msg);
@@ -96,20 +96,24 @@ export default class ControllerRouter {
 	broadcastMessage(
 		origin: lib.Link,
 		message: lib.MessageRoutable,
+		entry: lib.EventEntry,
 	) {
 		let dst = message.dst;
 		if (message.type !== "event") {
 			this.warnUnrouted(message, `Unexpected broadcast of ${message.type}`);
-		} else if (dst.id === lib.Address.host || dst.id === lib.Address.instance) {
+			return;
+		}
+		const plugin = entry.Event.plugin;
+		if (dst.id === lib.Address.host || dst.id === lib.Address.instance) {
 			for (let hostConnection of this.controller.wsServer.hostConnections.values()) {
-				if (hostConnection !== origin) {
-					hostConnection.connector.send(message);
+				if (hostConnection !== origin && (!plugin || hostConnection.plugins.has(plugin))) {
+					hostConnection.connector.forward(message);
 				}
 			}
 		} else if (dst.id === lib.Address.control) {
 			for (let controlConnection of this.controller.wsServer.controlConnections.values()) {
 				if (controlConnection !== origin) {
-					controlConnection.connector.send(message);
+					controlConnection.connector.forward(message);
 				}
 			}
 		} else {
@@ -126,5 +130,3 @@ export default class ControllerRouter {
 		logger.warn(msg ? `${baseMsg}: ${msg}.` : `${baseMsg}.`);
 	}
 }
-
-module.exports = ControllerRouter;

packages/create/create.js

@@ -488,6 +488,7 @@ Group=${await groupIdToName(os.userInfo().gid)}
 WorkingDirectory=${process.cwd()}
 KillMode=mixed
 KillSignal=SIGINT
+Environment=NODE_OPTIONS=--enable-source-maps
 ExecStart=${process.cwd()}/node_modules/.bin/clusteriohost run --log-level=warn --can-restart
 Restart=on-failure
 RestartPreventExitStatus=8
@@ -611,7 +612,7 @@ async function inquirerMissingArgs(args) {
 				},
 			], answers);
 
-			if (answers.downloadHeadless) {
+			if (!answers.factorioDir && answers.downloadHeadless) {
 				answers.factorioDir = "factorio";
 			}
 		}
@@ -759,7 +760,7 @@ async function main() {
 					"Can be set to false using --no-download-headless.",
 				type: "boolean",
 			})
-			.conflicts("factorio-dir", "download-headless");
+		;
 	}
 
 	args = args.argv;
@@ -779,6 +780,9 @@ async function main() {
 
 	let answers = await inquirerMissingArgs(args);
 	if (answers.downloadHeadless) {
+		if (answers.factorioDir !== "factorio") {
+			throw new InstallError("--download-headless option requires --factorio-dir to be set to factorio");
+		}
 		await downloadLinuxServer();
 	}
 

packages/create/package.json

@@ -3,7 +3,7 @@
 	"description": "Installer for clusterio",
 	"author": "Hornwitser",
 	"license": "MIT",
-	"version": "2.0.0-alpha.14",
+	"version": "2.0.0-alpha.16",
 	"main": "create.js",
 	"bin": {
 		"clusteriocreate": "./create.js"

packages/ctl/index.ts

@@ -1,4 +1,5 @@
 import { bootstrap } from "./ctl";
+export * from "./ctl";
 export { default as BaseCtlPlugin } from "./src/BaseCtlPlugin";
 
 if (module === require.main) {

packages/ctl/package.json

@@ -3,7 +3,7 @@
 	"description": "Clusterio command line management tool",
 	"author": "Hornwitser",
 	"license": "MIT",
-	"version": "2.0.0-alpha.14",
+	"version": "2.0.0-alpha.16",
 	"main": "dist/index.js",
 	"scripts": {
 		"prepare": "tsc --build",
@@ -19,7 +19,7 @@
 		"node": ">=18"
 	},
 	"dependencies": {
-		"@clusterio/lib": "^2.0.0-alpha.14",
+		"@clusterio/lib": "^2.0.0-alpha.16",
 		"as-table": "^1.0.55",
 		"fs-extra": "^11.1.1",
 		"phin": "^3.7.0",

packages/ctl/src/commands.ts

@@ -1097,7 +1097,13 @@ modPackCommands.add(new lib.Command({
 
 function setModPackSettings(
 	modPack: lib.ModPack,
-	args: { boolSetting?: string[], intSetting?: string[], doubleSetting?: string[], stringSetting?: string[] },
+	args: {
+		boolSetting?: string[],
+		intSetting?: string[],
+		doubleSetting?: string[],
+		stringSetting?: string[],
+		colorSetting?: string[],
+	},
 ) {
 	function doSettings(settings: string[], cast: (value: string) => (string | number | boolean)) {
 		for (let i = 0; i + 2 < settings.length; i += 3) {
@@ -1134,6 +1140,13 @@ function setModPackSettings(
 		return number;
 	});
 	doSettings(args.stringSetting || [], value => value);
+	doSettings(args.colorSetting || [], value => {
+		let color = JSON.parse(value);
+		if (typeof color !== "object" || color === null) {
+			throw new lib.CommandError("color value must be a JSON object with r, g, b, a properties");
+		}
+		return color;
+	});
 }
 
 function setModPackMods(modPack: lib.ModPack, mods: string[] | undefined) {
@@ -1173,6 +1186,7 @@ modPackCommands.add(new lib.Command({
 			"int-setting": { describe: "Set int setting", array: true, nargs: 3, type: "string" },
 			"double-setting": { describe: "Set double setting", array: true, nargs: 3, type: "string" },
 			"string-setting": { describe: "Set string setting", array: true, nargs: 3, type: "string" },
+			"color-setting": { describe: "Set color setting", array: true, nargs: 3, type: "string" },
 		});
 	}],
 	handler: async function(
@@ -1186,6 +1200,7 @@ modPackCommands.add(new lib.Command({
 			intSetting?: string[],
 			doubleSetting?: string[],
 			stringSetting?: string[],
+			colorSetting?: string[],
 		},
 		control: Control,
 	) {
@@ -1244,6 +1259,7 @@ modPackCommands.add(new lib.Command({
 			"int-setting": { describe: "Set int setting", array: true, nargs: 3, type: "string" },
 			"double-setting": { describe: "Set double setting", array: true, nargs: 3, type: "string" },
 			"string-setting": { describe: "Set string setting", array: true, nargs: 3, type: "string" },
+			"color-setting": { describe: "Set color setting", array: true, nargs: 3, type: "string" },
 			"remove-setting": { describe: "Remove a setting", array: true, nargs: 2, type: "string" },
 		});
 	}],
@@ -1261,6 +1277,7 @@ modPackCommands.add(new lib.Command({
 			intSetting?: string[],
 			doubleSetting?: string[],
 			stringSetting?: string[],
+			colorSetting?: string[],
 			removeSetting?: string[],
 		},
 		control: Control,

packages/host/host.ts

@@ -26,6 +26,27 @@ import { ConsoleTransport, levels, logger } from "@clusterio/lib";
 
 import Host from "./src/Host";
 
+let host: Host | undefined;
+
+void new lib.Gauge(
+	"clusterio_host_pending_requests",
+	"Count of pending link requests currently waiting in memory on the host.",
+	{
+		labels: ["host_id"],
+		callback: (gauge) => {
+			if (!host) {
+				return;
+			}
+
+			let count = host.pendingRequestCount;
+			for (const connection of host.instanceConnections.values()) {
+				count += connection.pendingRequestCount;
+				count += connection.instance.pendingRequestCount;
+			}
+			gauge.labels(String(host.config.get("host.id"))).set(count);
+		},
+	}
+);
 
 export class HostConnector extends lib.WebSocketClientConnector {
 	constructor(
@@ -205,7 +226,7 @@ async function startHost() {
 	}
 
 	let hostConnector = new HostConnector(hostConfig, tlsCa, pluginInfos);
-	let host = new Host(
+	host = new Host(
 		hostConnector,
 		args.config,
 		hostConfig,
@@ -227,7 +248,7 @@ async function startHost() {
 
 		secondSigint = true;
 		logger.info("Caught interrupt signal, shutting down");
-		host.shutdown();
+		host!.shutdown();
 	});
 	let secondSigterm = false;
 	process.on("SIGTERM", () => {
@@ -240,11 +261,11 @@ async function startHost() {
 
 		secondSigterm = true;
 		logger.info("Caught termination signal, shutting down");
-		host.shutdown();
+		host!.shutdown();
 	});
 	process.on("SIGHUP", () => {
 		logger.info("Terminal closed, shutting down");
-		host.shutdown();
+		host!.shutdown();
 	});
 
 	try {
@@ -255,7 +276,7 @@ async function startHost() {
 	}
 
 	hostConnector.once("connect", () => {
-		logger.add(new lib.LinkTransport({ link: host }));
+		logger.add(new lib.LinkTransport({ link: host! }));
 	});
 
 	await hostConnector.connect();

packages/host/index.ts

@@ -1,6 +1,9 @@
 import { bootstrap } from "./host";
 export { default as BaseHostPlugin } from "./src/BaseHostPlugin";
 export { default as BaseInstancePlugin } from "./src/BaseInstancePlugin";
+export { default as Host, HostRouter } from "./src/Host";
+export { default as Instance } from "./src/Instance";
+export { default as InstanceConnection } from "./src/InstanceConnection";
 
 if (module === require.main) {
 	bootstrap();

packages/host/package.json

@@ -3,7 +3,7 @@
 	"description": "Implementation of Clusterio host server",
 	"author": "Hornwitser",
 	"license": "MIT",
-	"version": "2.0.0-alpha.14",
+	"version": "2.0.0-alpha.16",
 	"main": "dist/index.js",
 	"scripts": {
 		"prepare": "tsc --build",
@@ -19,7 +19,7 @@
 		"node": ">=18"
 	},
 	"dependencies": {
-		"@clusterio/lib": "^2.0.0-alpha.14",
+		"@clusterio/lib": "^2.0.0-alpha.16",
 		"@sinclair/typebox": "^0.30.4",
 		"fs-extra": "^11.1.1",
 		"jimp": "^0.22.8",

packages/host/src/Host.ts

@@ -82,7 +82,7 @@ const instanceStartingMessages = new Set([
 	lib.InstanceExportDataRequest.name,
 ]);
 
-class HostRouter {
+export class HostRouter {
 	constructor(
 		public host: Host
 	) { }
@@ -93,16 +93,22 @@ class HostRouter {
 	 * @param origin -
 	 *    Link the message originated from.
 	 * @param message - Message to process.
+	 * @param entry - message entry if message is a request or an event.
 	 * @param hasFallback - is fallback available
 	 * @returns true if the message was handled, false if fallback
 	 *     is requested
 	 */
-	forwardMessage(origin: lib.Link, message: lib.MessageRoutable, hasFallback: boolean) {
+	forwardMessage(
+		origin: lib.Link,
+		message: lib.MessageRoutable,
+		entry: lib.RequestEntry | lib.EventEntry | undefined,
+		hasFallback: boolean,
+	) {
 		let dst = message.dst;
 		let nextHop: lib.Link | undefined;
 		let msg: string | undefined;
 		if (dst.type === lib.Address.broadcast) {
-			this.broadcastMessage(origin, message);
+			this.broadcastMessage(origin, message, entry as lib.EventEntry);
 			return true;
 		} else if (
 			dst.type === lib.Address.controller
@@ -153,27 +159,36 @@ class HostRouter {
 		return true;
 	}
 
-	broadcastMessage(origin: lib.Link, message: lib.MessageRoutable) {
+	broadcastMessage(
+		origin: lib.Link,
+		message: lib.MessageRoutable,
+		entry: lib.EventEntry,
+	) {
 		let dst = message.dst;
 		if (message.type !== "event") {
 			this.warnUnrouted(message, `Unexpected broadcast of ${message.type}`);
-		} else if (dst.id === lib.Address.host || dst.id === lib.Address.instance) {
+			return;
+		}
+		const plugin = entry.Event.plugin;
+		if (dst.id === lib.Address.instance) {
 			for (let instanceConnection of this.host.instanceConnections.values()) {
-				if (instanceConnection !== origin) {
-					instanceConnection.connector.send(message);
+				if (instanceConnection !== origin && (!plugin || instanceConnection.plugins.has(plugin))) {
+					instanceConnection.connector.forward(message);
 				}
 			}
-			if (this.host !== origin) {
-				this.host.connector.send(message);
+			if (this.host !== origin && (!plugin || this.host.serverPlugins.has(plugin))) {
+				this.host.connector.forward(message);
 			}
 		} else if (dst.id === lib.Address.control) {
 			if (this.host !== origin) {
-				this.host.connector.send(message);
+				if (!plugin || this.host.serverPlugins.has(plugin)) {
+					this.host.connector.forward(message);
+				}
 			} else {
-				logger.warn(`Received control broadcast of ${(message as lib.MessageEvent).name} from master.`);
+				logger.warn(`Received control broadcast of ${(message as lib.MessageEvent).name} from controller.`);
 			}
 		} else {
-			this.warnUnrouted(message, `Unexpected broacdast target ${dst.id}`);
+			this.warnUnrouted(message, `Unexpected broadcast target ${dst.id}`);
 		}
 	}
 
@@ -193,7 +208,7 @@ class HostRouter {
 		}
 
 		this.host._connectInstance(dst.id).then(instanceConnection => {
-			instanceConnection.connector.send(message);
+			instanceConnection.connector.forward(message);
 		}).catch(err => {
 			logger.error(`Error starting instance:\n${err.stack}`, this.host.instanceLogMeta(dst.id));
 			origin.connector.sendResponseError(
@@ -207,7 +222,7 @@ class HostRouter {
 			if (message.type === "request") {
 				nextHop.forwardRequest(message as lib.MessageRequest, origin);
 			} else {
-				nextHop.connector.send(message);
+				nextHop.connector.forward(message);
 			}
 		} catch (err: any) {
 			if (message.type === "request") {
@@ -377,7 +392,10 @@ export default class Host extends lib.Link {
 
 	async loadPlugins() {
 		for (let pluginInfo of this.pluginInfos) {
-			if (!this.config.get(`${pluginInfo.name}.load_plugin` as keyof lib.HostConfigFields)) {
+			if (
+				!pluginInfo.hostEntrypoint && !pluginInfo.instanceEntrypoint
+				|| !this.config.get(`${pluginInfo.name}.load_plugin` as keyof lib.HostConfigFields)
+			) {
 				continue;
 			}
 
@@ -1053,7 +1071,7 @@ export default class Host extends lib.Link {
 				if (instanceInfo.config.get("instance.auto_start")) {
 					try {
 						let instanceConnection = await this._connectInstance(instanceId);
-						await instanceConnection.send(new lib.InstanceStartRequest());
+						await instanceConnection.instance.handleInstanceStartRequest(new lib.InstanceStartRequest());
 					} catch (err: any) {
 						logger.error(
 							`Error during auto startup for ${instanceInfo.config.get("instance.name")}:\n${err.stack}`,

packages/host/src/Instance.ts

@@ -955,6 +955,11 @@ rcon.print(game.table_to_json(players))`.replace(/\r?\n/g, " ");
 			await lib.invokeHook(this.plugins, "onStop");
 			await this.server.stop();
 			await this.sendSaveListUpdate();
+		} else if (
+			this.server._state === "stopping"
+			|| this.server._state === "create"
+		) {
+			await this.server.kill();
 		}
 	}
 

packages/host/src/server.ts

@@ -865,7 +865,9 @@ export class FactorioServer extends events.EventEmitter {
 
 		try {
 			let [code, signal] = await events.once(this._server, "exit");
-			void signal;
+			if (signal) {
+				throw new Error(`Factorio exited with signal ${signal}`);
+			}
 			if (code !== 0) {
 				throw new Error(`Factorio exited with status ${code}`);
 			}
@@ -1092,7 +1094,7 @@ export class FactorioServer extends events.EventEmitter {
 	 *     process.
 	 */
 	async kill(unexpected = false) {
-		this._check(["running", "stopping"]);
+		this._check(["running", "stopping", "create"]);
 		this._killed = true;
 		if (!unexpected) {
 			this._state = "stopping";

packages/lib/package.json

@@ -1,7 +1,7 @@
 {
 	"name": "@clusterio/lib",
 	"description": "Shared library for Clusterio",
-	"version": "2.0.0-alpha.14",
+	"version": "2.0.0-alpha.16",
 	"repository": "https://github.com/clusterio/clusterio",
 	"license": "MIT",
 	"scripts": {

packages/lib/src/config/classes.ts

@@ -459,7 +459,7 @@ export class Config<
 	setProp<Field extends keyof Fields & string>(
 		name: Field,
 		prop: string,
-		value: unknown,
+		value?: unknown,
 		location = this.location,
 	) {
 		let def = this.constructor.fieldDefinitions[name];

packages/lib/src/data/ModPack.ts

@@ -8,16 +8,24 @@ import { integerFactorioVersion } from "./version";
 import type { Logger } from "../logging";
 
 
+export const ModSettingColor = Type.Object({
+	r: Type.Number(),
+	g: Type.Number(),
+	b: Type.Number(),
+	a: Type.Number(),
+});
+export type ModSettingColor = Static<typeof ModSettingColor>;
+
 /**
  * A setting for a mod.
  */
 export interface ModSetting {
 	/** Value of the given mod setting. */
-	value: boolean | number | string;
+	value: boolean | number | string | ModSettingColor;
 }
 
 const ModSettingJsonSchema = Type.Object({
-	"value": Type.Union([Type.Boolean(), Type.Number(), Type.String()]),
+	"value": Type.Union([Type.Boolean(), Type.Number(), Type.String(), ModSettingColor]),
 });
 
 const ModSettingsJsonSchema = Type.Record(Type.String(), ModSettingJsonSchema);
@@ -335,7 +343,7 @@ export default class ModPack {
 	 * @param logger - Logger used to report warnings on.
 	 */
 	fillDefaultSettings(settingPrototypes: Record<string, object>, logger: Logger) {
-		const knownTypes = ["bool-setting", "int-setting", "double-setting", "string-setting"];
+		const knownTypes = ["bool-setting", "int-setting", "double-setting", "string-setting", "color-setting"];
 		let prototypes = Object.entries(settingPrototypes)
 			.filter(([type, _]) => knownTypes.includes(type))
 			.flatMap(([_, settings]) => Object.values(settings))

packages/lib/src/data/index.ts

@@ -5,7 +5,7 @@
  */
 export { default as ExportManifest } from "./ExportManifest";
 export { default as ModInfo } from "./ModInfo";
-export { default as ModPack, ModSetting, ModRecord } from "./ModPack";
+export { default as ModPack, ModSetting, ModSettingColor, ModRecord } from "./ModPack";
 export { default as ModuleInfo } from "./ModuleInfo";
 export { default as Permission } from "./Permission";
 export { default as PlayerStats } from "./PlayerStats";

packages/lib/src/data/messages_core.ts

@@ -87,7 +87,7 @@ export class Address {
 	}
 
 	toString() {
-		const typeMap = ["controller", "instance", "host", "control"];
+		const typeMap = ["controller", "host", "instance", "control"];
 		const type = typeMap[this.type] || this.type;
 		if (this.type === Address.broadcast) {
 			return `[Address ${type}:${typeMap[this.id] || this.id}]`;

packages/lib/src/data/messages_instance.ts

@@ -208,7 +208,7 @@ export class InstanceConfigSetPropRequest {
 		"instanceId": Type.Integer(),
 		"field": Type.String(),
 		"prop": Type.String(),
-		"value": Type.Unknown(),
+		"value": Type.Optional(Type.Unknown()),
 	});
 
 	static fromJSON(json: Static<typeof this.jsonSchema>) {

packages/lib/src/link/connectors.ts

@@ -33,7 +33,30 @@ export abstract class BaseConnector extends events.EventEmitter {
 		this._seq = 1;
 	}
 
-	abstract send(message: libData.Message): void;
+	protected abstract send(message: libData.MessageRoutable): void;
+
+	forward(message: libData.MessageRoutable) {
+		const seq = this._seq;
+		this._seq += 1;
+		// The message type currently can't be inferred like a tagged union
+		// TODO fix typing of message so it can be deduced from .type
+		if (message.type === "request") {
+			const request = message as libData.MessageRequest;
+			this.send(new libData.MessageRequest(seq, request.src, request.dst, request.name, request.data));
+		} else if (message.type === "response") {
+			const response = message as libData.MessageResponse;
+			this.send(new libData.MessageResponse(seq, response.src, response.dst, response.data));
+		} else if (message.type === "responseError") {
+			const error = message as libData.MessageResponseError;
+			this.send(new libData.MessageResponseError(seq, error.src, error.dst, error.data));
+		} else if (message.type === "event") {
+			const event = message as libData.MessageEvent;
+			this.send(new libData.MessageEvent(seq, event.src, event.dst, event.name, event.data));
+		} else {
+			throw new Error(`Cannot forward message type ${(message as any).type}`);
+		}
+		return seq;
+	}
 
 	sendRequest<Req, Res>(
 		request: Request<Req, Res>,
@@ -242,7 +265,7 @@ export abstract class WebSocketBaseConnector extends BaseConnector {
 	 *
 	 * @param message - Message to send.
 	 */
-	send(message: libData.MessageRoutable) {
+	protected send(message: libData.MessageRoutable) {
 		if (!["connected", "resuming"].includes(this._state)) {
 			throw new libErrors.SessionLost("No session");
 		}
@@ -295,6 +318,13 @@ export abstract class WebSocketBaseConnector extends BaseConnector {
 		await this.close(1000, "Disconnect");
 	}
 
+	/**
+	 * Notify other end that the link is ready to disconnect
+	 */
+	sendDisconnectReady() {
+		this._sendInternal(new libData.MessageDisconnect("ready"));
+	}
+
 	/**
 	 * True if the connection is established and active
 	 */

packages/lib/src/link/link.ts

@@ -72,7 +72,12 @@ export type RequestHandler<Req, Res> = (request: Req, src: libData.Address, dst:
 export type EventHandler<T> = (event: T, src: libData.Address, dst: libData.Address) => Promise<void>;
 
 interface Router {
-	forwardMessage(origin: Link, message: MessageRoutable, fallback: boolean): boolean,
+	forwardMessage(
+		origin: Link,
+		message: MessageRoutable,
+		entry: RequestEntry | EventEntry | undefined,
+		fallback: boolean,
+	): boolean,
 }
 
 // Some definitions for the terminology used here:
@@ -127,7 +132,8 @@ export class Link {
 				}
 			).finally(
 				() => {
-					this.connector.send(new libData.MessageDisconnect("ready"));
+					// Only WebSocket connectors issue disconnection events.
+					(this.connector as WebSocketBaseConnector).sendDisconnectReady();
 				},
 			);
 		});
@@ -141,17 +147,30 @@ export class Link {
 		});
 	}
 
+	/**
+	 * Count of requests currently waiting for a response on this link
+	 */
+	get pendingRequestCount() {
+		return this._pendingRequests.size + this._forwardedRequests.size;
+	}
+
 	_clearPendingRequests(err: Error & { code?: string }) {
 		for (let pending of this._pendingRequests.values()) {
 			pending.reject(err);
 		}
 		for (let pending of this._forwardedRequests.values()) {
-			assert(pending.origin.connector instanceof WebSocketBaseConnector);
-			if (pending.origin.connector.hasSession) {
-				pending.origin.connector.sendResponseError(
-					new libData.ResponseError(err.message, err.code), pending.src, pending.dst
-				);
+			// Drop response if the origin is a WebSocket and the session
+			// has expired.  The other end of the link will have sent an
+			// error response to the request in this case.
+			if (
+				pending.origin.connector instanceof WebSocketBaseConnector
+				&& !pending.origin.connector.hasSession
+			) {
+				continue;
 			}
+			pending.origin.connector.sendResponseError(
+				new libData.ResponseError(err.message, err.code), pending.src, pending.dst
+			);
 		}
 		this._pendingRequests.clear();
 		this._forwardedRequests.clear();
@@ -330,7 +349,7 @@ export class Link {
 		let fallback =
 			message.type === "request" ? this._requestFallbacks.get((entry as RequestEntry).Request) : undefined
 		;
-		if (this.router.forwardMessage(this, message, Boolean(fallback))) {
+		if (this.router.forwardMessage(this, message, entry, Boolean(fallback))) {
 			return;
 		}
 		if (!fallback) {
@@ -421,6 +440,7 @@ export class Link {
 			throw new libErrors.InvalidMessage(`Received reply from ${message.src} for message sent to ${pending.dst}`);
 		}
 
+		this._pendingRequests.delete(message.dst.requestId!);
 		try {
 			pending.resolve(pending.request.responseFromJSON(message.data));
 		} catch (err: any) {
@@ -437,6 +457,7 @@ export class Link {
 			);
 		}
 
+		this._pendingRequests.delete(message.dst.requestId!);
 		pending.reject(new libErrors.RequestError(message.data.message, message.data.code, message.data.stack));
 	}
 
@@ -541,7 +562,7 @@ export class Link {
 			dst: message.dst,
 		};
 		this._forwardedRequests.set(message.src.index(), pending);
-		this.connector.send(message);
+		this.connector.forward(message);
 	}
 
 	sendEvent<T>(event: Event<T>, dst: libData.Address) {

packages/lib/src/logging.ts

@@ -73,7 +73,7 @@ export class WebConsoleFormat {
 	transform(info: any, _options: object) {
 		let src = " ";
 		if (info.host_id !== undefined) {
-			src += `s:${info.host_name} - `;
+			src += `h:${info.host_name} - `;
 		}
 		if (info.instance_id !== undefined) {
 			src += `i:${info.instance_name}`;

packages/lib/src/logging_utils.ts

@@ -163,7 +163,7 @@ export class TerminalFormat {
 			src += `i:${info.instance_name} `;
 
 		} else if (info.host_name) {
-			src += `s:${info.host_name} `;
+			src += `h:${info.host_name} `;
 		}
 		if (info.plugin) {
 			src += `${info.plugin}: `;

packages/web_ui/package.json

@@ -3,7 +3,7 @@
 	"description": "Clusterio web interface implementation",
 	"author": "Hornwitser",
 	"license": "MIT",
-	"version": "2.0.0-alpha.14",
+	"version": "2.0.0-alpha.16",
 	"browser": "src/index.ts",
 	"main": "dist/src/index.js",
 	"scripts": {
@@ -19,7 +19,7 @@
 		"@ant-design/icons": "^5.1.4",
 		"@babel/core": "^7.22.9",
 		"@babel/preset-react": "^7.22.5",
-		"@clusterio/lib": "^2.0.0-alpha.14",
+		"@clusterio/lib": "^2.0.0-alpha.16",
 		"antd": "^5.13.0",
 		"assert": "^2.0.0",
 		"babel-loader": "^9.1.3",

packages/web_ui/src/components/ControllerConfigTree.tsx

@@ -27,6 +27,8 @@ export default function ControllerConfigTree() {
 			} catch (err) {
 				return;
 			}
+		} else {
+			value = undefined;
 		}
 		await control.send(new lib.ControllerConfigSetPropRequest(field, prop, value));
 	}

packages/web_ui/src/components/HostConfigTree.tsx

@@ -24,6 +24,8 @@ export default function HostConfigTree(props: { id: number, available: boolean }
 			} catch (err) {
 				return;
 			}
+		} else {
+			value = undefined;
 		}
 		await control.sendTo({ hostId: props.id }, new lib.HostConfigSetPropRequest(field, prop, value));
 	}

packages/web_ui/src/components/InstanceConfigTree.tsx

@@ -28,6 +28,8 @@ export default function InstanceConfigTree(props: InstanceConfigTreeProps) {
 			} catch (err) {
 				return;
 			}
+		} else {
+			value = undefined;
 		}
 		await control.send(new lib.InstanceConfigSetPropRequest(props.id, field, prop, value));
 	}

packages/web_ui/src/components/ModPackViewPage.tsx

@@ -1,7 +1,7 @@
 import React, { Fragment, memo, useCallback, useEffect, useContext, useRef, useState } from "react";
 import { useNavigate, useParams } from "react-router-dom";
 import {
-	Button, Card, Checkbox, Col, ConfigProvider, Descriptions, Form, Input, Pagination,
+	Button, Card, ColorPicker, Checkbox, Col, ConfigProvider, Descriptions, Form, Input, Pagination,
 	Popconfirm, Row, Table, Tag, Typography, Select, Skeleton, Space, Spin, Switch, Modal, Tooltip,
 	TablePaginationConfig,
 } from "antd";
@@ -404,7 +404,7 @@ function hasChange(changes: ModChange[], field:Partial<ModChange>) {
 }
 
 // picks out the value property from settings.
-function pickValue(map: Map<string, lib.ModSetting>): [string, string|number|boolean][] {
+function pickValue(map: Map<string, lib.ModSetting>): [string, lib.ModSetting["value"]][] {
 	return [...map].map(([name, value]) => [name, value.value]);
 }
 
@@ -420,6 +420,25 @@ function groupToMap(array: any[], fn: (...args:any) => any) {
 	return map;
 }
 
+function InputColor(props: {
+	value: lib.ModSettingColor,
+	onChange: (value: lib.ModSettingColor) => void,
+}) {
+	const value = props.value;
+	return <ColorPicker
+		defaultFormat="rgb"
+		defaultValue={`rgba(${value.r * 255}, ${value.g * 255}, ${value.b * 255}, ${value.a}`}
+		onChangeComplete={(color) => {
+			const rgb = color.toRgb();
+			props.onChange({
+				r: rgb.r / 255,
+				g: rgb.g / 255,
+				b: rgb.b / 255,
+				a: rgb.a,
+			});
+		}}
+	/>;
+}
 type SettingsTableFieldProps = {
 	name: string;
 	type: string;
@@ -431,11 +450,15 @@ type SettingsTableFieldProps = {
 const SettingsTableField = memo((props: SettingsTableFieldProps) => {
 	const name = props.name;
 	const isBoolean = props.type === "bool-setting";
+	const isColor = props.type === "color-setting";
 	const isNumber = ["int-setting", "double-setting"].includes(props.type);
 	const isSelect = Boolean(props.allowedValues);
 	let input;
 	if (isBoolean) {
 		input = <Checkbox />;
+	} else if (isColor) {
+		const PartialInputColor = InputColor as () => React.JSX.Element;
+		input = <PartialInputColor />;
 	} else if (isSelect) {
 		const options = Object.values(props.allowedValues).map(value => ({
 			value,
@@ -484,7 +507,7 @@ function SettingsTable(props: SettingsTableProps) {
 		.map(mod => [mod.name, mod.title])
 	);
 
-	const types = ["bool-setting", "int-setting", "double-setting", "string-setting"];
+	const types = ["bool-setting", "int-setting", "double-setting", "string-setting", "color-setting"];
 	let prototypes = Object.entries(props.prototypes)
 		.filter(([type, _]) => types.includes(type))
 		.flatMap<any>(([_, settingPrototypes]: [string, any]) => Object.values(settingPrototypes))
@@ -492,7 +515,7 @@ function SettingsTable(props: SettingsTableProps) {
 
 	function controls(
 		scope: "startup"|"runtime-global"|"runtime-per-user",
-		storedFields: [string, string|number|boolean][]
+		storedFields: [string, lib.ModSetting["value"]][]
 	) {
 		let fields = new Map(prototypes
 			.filter(p => p.setting_type === scope)
@@ -508,6 +531,8 @@ function SettingsTable(props: SettingsTableProps) {
 					type = "double-setting";
 				} else if (typeof value === "string") {
 					type = "string-setting";
+				} else if (typeof value === "object" && "r" in value) {
+					type = "color-setting";
 				} else {
 					throw Error(`Unhandled setting type ${typeof value}`);
 				}

plugins/global_chat/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@clusterio/plugin-global_chat",
-	"version": "2.0.0-alpha.14",
+	"version": "2.0.0-alpha.16",
 	"description": "Clusterio plugin forwarding between Factorio servers",
 	"author": "Hornwitser <github@hornwitser.no>",
 	"homepage": "https://github.com/clusterio/clusterio#readme",
@@ -21,8 +21,8 @@
 		"@clusterio/lib": "^2.0.0-alpha.0"
 	},
 	"devDependencies": {
-		"@clusterio/lib": "^2.0.0-alpha.14",
-		"@clusterio/web_ui": "^2.0.0-alpha.14",
+		"@clusterio/lib": "^2.0.0-alpha.16",
+		"@clusterio/web_ui": "^2.0.0-alpha.16",
 		"@types/node": "^20.4.5",
 		"@types/react": "^18.2.21",
 		"typescript": "^5.1.6",

plugins/inventory_sync/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@clusterio/plugin-inventory_sync",
-	"version": "2.0.0-alpha.14",
+	"version": "2.0.0-alpha.16",
 	"description": "Clusterio plugin forwarding between Factorio servers",
 	"author": "Danielv <danielv@danielv.no>",
 	"homepage": "https://github.com/clusterio/factorioClusterio#readme",
@@ -25,8 +25,8 @@
 		"fs-extra": "^8.1.0"
 	},
 	"devDependencies": {
-		"@clusterio/lib": "^2.0.0-alpha.14",
-		"@clusterio/web_ui": "^2.0.0-alpha.14",
+		"@clusterio/lib": "^2.0.0-alpha.16",
+		"@clusterio/web_ui": "^2.0.0-alpha.16",
 		"@types/fs-extra": "^11.0.1",
 		"@types/node": "^20.4.5",
 		"@types/react": "^18.2.21",

plugins/player_auth/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@clusterio/plugin-player_auth",
-	"version": "2.0.0-alpha.14",
+	"version": "2.0.0-alpha.16",
 	"description": "Clusterio plugin authenticating logged in players to the web interface",
 	"author": "Hornwitser <github@hornwitser.no>",
 	"homepage": "https://github.com/clusterio/clusterio#readme",
@@ -26,8 +26,8 @@
 		"jsonwebtoken": "^9.0.1"
 	},
 	"devDependencies": {
-		"@clusterio/lib": "^2.0.0-alpha.14",
-		"@clusterio/web_ui": "^2.0.0-alpha.14",
+		"@clusterio/lib": "^2.0.0-alpha.16",
+		"@clusterio/web_ui": "^2.0.0-alpha.16",
 		"@types/express": "^4.17.17",
 		"@types/jsonwebtoken": "^9.0.2",
 		"@types/node": "^20.4.5",

plugins/research_sync/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@clusterio/plugin-research_sync",
-	"version": "2.0.0-alpha.14",
+	"version": "2.0.0-alpha.16",
 	"description": "Clusterio plugin for synchronizing research between Factorio servers",
 	"author": "Hornwitser <github@hornwitser.no>",
 	"homepage": "https://github.com/clusterio/clusterio#readme",
@@ -25,8 +25,8 @@
 		"fs-extra": "^8.1.0"
 	},
 	"devDependencies": {
-		"@clusterio/lib": "^2.0.0-alpha.14",
-		"@clusterio/web_ui": "^2.0.0-alpha.14",
+		"@clusterio/lib": "^2.0.0-alpha.16",
+		"@clusterio/web_ui": "^2.0.0-alpha.16",
 		"@types/fs-extra": "^11.0.1",
 		"@types/node": "^20.4.5",
 		"@types/react": "^18.2.21",

plugins/statistics_exporter/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@clusterio/plugin-statistics_exporter",
-	"version": "2.0.0-alpha.14",
+	"version": "2.0.0-alpha.16",
 	"description": "Clusterio plugin for gathering statistics from Factorio servers",
 	"author": "Hornwitser <github@hornwitser.no>",
 	"homepage": "https://github.com/clusterio/clusterio#readme",
@@ -21,8 +21,8 @@
 		"@clusterio/lib": "^2.0.0-alpha.0"
 	},
 	"devDependencies": {
-		"@clusterio/lib": "^2.0.0-alpha.14",
-		"@clusterio/web_ui": "^2.0.0-alpha.14",
+		"@clusterio/lib": "^2.0.0-alpha.16",
+		"@clusterio/web_ui": "^2.0.0-alpha.16",
 		"@types/node": "^20.4.5",
 		"@types/react": "^18.2.21",
 		"typescript": "^5.1.6",

plugins/subspace_storage/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@clusterio/plugin-subspace_storage",
-	"version": "2.0.0-alpha.14",
+	"version": "2.0.0-alpha.16",
 	"description": "Clusterio plugin for sharing storage between Factorio servers",
 	"author": "Hornwitser <github@hornwitser.no>",
 	"homepage": "https://github.com/clusterio/clusterio#readme",
@@ -25,8 +25,8 @@
 		"fs-extra": "^8.1.0"
 	},
 	"devDependencies": {
-		"@clusterio/lib": "^2.0.0-alpha.14",
-		"@clusterio/web_ui": "^2.0.0-alpha.14",
+		"@clusterio/lib": "^2.0.0-alpha.16",
+		"@clusterio/web_ui": "^2.0.0-alpha.16",
 		"@types/express": "^4.17.17",
 		"@types/fs-extra": "^11.0.1",
 		"@types/node": "^20.4.5",

test/integration/clusterio.js

@@ -17,6 +17,7 @@ const {
 } = require("./index");
 
 
+/** @returns {Promise<Map<number, lib.InstanceDetails>>} */
 async function getInstances() {
 	let instances = await getControl().send(new lib.InstanceDetailsListRequest());
 	return new Map(instances.map(instance => [instance.id, instance]));
@@ -27,6 +28,10 @@ async function checkInstanceStatus(id, status) {
 	assert.equal(instances.get(44).status, status, "incorrect instance status");
 }
 
+async function spawnAltHost(config = "alt-host-config.json") {
+	return await spawn("alt-host:", `node ../../packages/host run --config ${config}`, /Started host/);
+}
+
 async function startAltHost() {
 	let config = "alt-host-config.json";
 	let configPath = path.join("temp", "test", config);
@@ -38,7 +43,7 @@ async function startAltHost() {
 	await exec(`node ../../packages/host --config ${config} config set host.instances_directory alt-instances`);
 	await exec(`node ../../packages/host --config ${config} config set host.factorio_directory ../../factorio`);
 	await exec(`node ../../packages/host --config ${config} config set host.mods_directory alt-mods`);
-	return await spawn("alt-host:", `node ../../packages/host run --config ${config}`, /Started host/);
+	return await spawnAltHost(config);
 }
 
 async function stopAltHost(hostProcess) {
@@ -81,6 +86,10 @@ async function getUsers() {
 	return new Map(users.map(user => [user.name, user]));
 }
 
+function jsonArg(value) {
+	return `"${JSON.stringify(value).replace(/"/g, process.platform === "win32" ? '""' : '\\"')}"`;
+}
+
 describe("Integration of Clusterio", function() {
 	describe("clusteriocontroller", function() {
 		describe("bootstrap generate-user-token", function() {
@@ -196,6 +205,38 @@ describe("Integration of Clusterio", function() {
 				assert(await fs.pathExists(clusterioLib), "mod was not downloaded by the test");
 			});
 		});
+		it("should auto start instances with auto_start enabled", async function() {
+			slowTest(this);
+
+			let hostProcess;
+			try {
+				hostProcess = await startAltHost();
+				await execCtl("instance create alt-start --id 97");
+				await execCtl("instance assign alt-start 5");
+				await execCtl("instance config set alt-start instance.auto_start true");
+				await stopAltHost(hostProcess);
+				hostProcess = await spawnAltHost();
+				// Stop the host immediatly to test the handling of stopping
+				// instances while automatically started and a save is being created
+				await stopAltHost(hostProcess);
+				hostProcess = await spawnAltHost();
+				let status;
+				// Wait for instance status to become running.
+				for (let i = 0; i < 100; i++) {
+					const instances = await getInstances();
+					status = instances.get(97)?.status ?? "not_present";
+					if (status === "running") {
+						break;
+					}
+					await wait(100); // Try again 10 times per second.
+				}
+				assert.equal(status, "running");
+				await execCtl("instance stop alt-start");
+				await execCtl("instance delete alt-start");
+			} finally {
+				await stopAltHost(hostProcess);
+			}
+		});
 	});
 
 	describe("clusterioctl", function() {
@@ -502,8 +543,7 @@ describe("Integration of Clusterio", function() {
 				];
 
 				for (let [prop, value] of testConfigs) {
-					value = `"${JSON.stringify(value).replace(/"/g, process.platform === "win32" ? '""' : '\\"')}"`;
-					let args = `test factorio.settings ${prop} ${value}`;
+					let args = `test factorio.settings ${prop} ${jsonArg(value)}`;
 					await execCtl(`instance config set-prop ${args}`);
 				}
 
@@ -517,6 +557,15 @@ describe("Integration of Clusterio", function() {
 				slowTest(this);
 				await checkInstanceStatus(44, "running");
 			});
+			it("should allow creating and removing props", async function() {
+				slowTest(this);
+				await execCtl("instance config set-prop test factorio.settings new_property new-string-value");
+				let config = await getControl().send(new lib.InstanceConfigGetRequest(44));
+				assert.equal(config["factorio.settings"]["new_property"], "new-string-value");
+				await execCtl("instance config set-prop test factorio.settings new_property");
+				config = await getControl().send(new lib.InstanceConfigGetRequest(44));
+				assert.equal(config["factorio.settings"]["new_property"], undefined);
+			});
 		});
 
 		describe("user set-admin/whitelisted/banned", function() {
@@ -876,6 +925,7 @@ describe("Integration of Clusterio", function() {
 				assert(modPacks.some(modPack => modPack.name === "empty-pack"), "created pack is not in the list");
 			});
 			it("should allow setting all fields", async function() {
+				const color = { r: 1, g: 0, b: 1, a: 0 };
 				await execCtl(
 					"mod-pack create full-pack 0.17.59 " +
 					"--description Description " +
@@ -883,7 +933,8 @@ describe("Integration of Clusterio", function() {
 					"--bool-setting startup MyBool true " +
 					"--int-setting runtime-global MyInt 1235 " +
 					"--double-setting runtime-global MyDouble 12.25 " +
-					"--string-setting runtime-per-user MyString a-string"
+					"--string-setting runtime-per-user MyString a-string " +
+					`--color-setting runtime-per-user MyColor ${jsonArg(color)}`
 				);
 				let modPacks = await getControl().send(new lib.ModPackListRequest());
 				let modPack = modPacks.find(entry => entry.name === "full-pack");
@@ -898,6 +949,7 @@ describe("Integration of Clusterio", function() {
 				reference.settings["runtime-global"].set("MyInt", { value: 1235 });
 				reference.settings["runtime-global"].set("MyDouble", { value: 12.25 });
 				reference.settings["runtime-per-user"].set("MyString", { value: "a-string" });
+				reference.settings["runtime-per-user"].set("MyColor", { value: color });
 				reference.updatedAtMs = modPack.updatedAtMs;
 				assert.deepEqual(modPack, reference);
 			});

test/integration/routing.js

@@ -0,0 +1,422 @@
+"use strict";
+const assert = require("assert").strict;
+
+const lib = require("@clusterio/lib");
+const { Host, Instance, InstanceConnection } = require("@clusterio/host");
+const { ControlConnection, Controller, HostConnection, InstanceInfo } = require("@clusterio/controller");
+
+const addr = lib.Address.fromShorthand;
+
+class Control extends lib.Link { }
+
+/**
+ * @param {Controller} controller
+ * @param {number} hostId
+ */
+function connectHost(controller, hostId, plugins) {
+	const [controllerSide, hostSide] = lib.VirtualConnector.makePair(addr("controller"), addr({ hostId }));
+	const registerData = new lib.RegisterHostData("", "0.0.0", hostId, plugins);
+	const hostConnection = new HostConnection(registerData, controllerSide, controller, "host-a.test");
+	controller.wsServer.hostConnections.set(hostId, hostConnection);
+	const hostConfig = new lib.HostConfig("host");
+	hostConfig.set("host.id", hostId);
+	return new Host(hostSide, "invalid", hostConfig, undefined, []);
+}
+
+/**
+ * @param {Controller} controller
+ * @param {Host} host
+ * @param {number} instanceId
+ */
+function connectInstance(controller, host, instanceId) {
+	const hostAddress = addr({ hostId: host.config.get("host.id")});
+	const instanceAddress = addr({ instanceId });
+	const [instanceSide, hostSide] = lib.VirtualConnector.makePair(instanceAddress, hostAddress);
+	const instanceConfig = new lib.InstanceConfig("host");
+	instanceConfig.set("instance.id", instanceId);
+	instanceConfig.set("instance.assigned_host", host.config.get("host.id"));
+	host.instanceInfos.set(instanceId, { path: "invalid", config: instanceConfig });
+	controller.instances.set(instanceId, new InstanceInfo(instanceConfig, "stopped"));
+	const instance = new Instance(host, instanceSide, "invalid", "invalid", instanceConfig);
+	const instanceConnection = new InstanceConnection(hostSide, host, instance);
+	host.instanceConnections.set(instanceId, instanceConnection);
+	return instance;
+}
+
+/**
+ * @param {Controller} controller
+ * @param {number} hostId
+ */
+function connectControl(controller, controlId) {
+	const [controllerSide, controlSide] = lib.VirtualConnector.makePair(addr("controller"), addr({ controlId }));
+	const registerData = new lib.RegisterControlData("", "0.0.0");
+	let user = controller.userManager.users.get("test");
+	if (!user) {
+		user = controller.userManager.createUser("test");
+	}
+	const controlConnection = new ControlConnection(registerData, controllerSide, controller, user, controlId);
+	controller.wsServer.controlConnections.set(controlId, controlConnection);
+	return new Control(controlSide);
+}
+
+class TestRequest {
+	static type = "request";
+	static src = ["controller", "host", "instance", "control"];
+	static dst = ["controller", "host", "instance", "control"];
+	static permission = null;
+}
+
+class TestEvent {
+	static type = "event";
+	static src = ["controller", "host", "instance", "control"];
+	static dst = ["controller", "host", "instance", "control"];
+	static permission = null;
+}
+
+class PluginEvent {
+	static type = "event";
+	static src = ["controller", "host", "instance", "control"];
+	static dst = ["controller", "host", "instance", "control"];
+	static permission = null;
+	static plugin = "test";
+}
+
+lib.Link.register(TestRequest);
+lib.Link.register(TestEvent);
+lib.Link.register(PluginEvent);
+
+describe("Integration of link routing", function() {
+	/** @type {Controller} */
+	let controller;
+	/** @type {Host} */
+	let hostA;
+	/** @type {Host} */
+	let hostB;
+	/** @type {Instance} */
+	let instanceA1;
+	/** @type {Instance} */
+	let instanceA2;
+	/** @type {Instance} */
+	let instanceB1;
+	/** @type {Control} */
+	let controlA;
+	/** @type {Control} */
+	let controlB;
+
+	beforeEach(function() {
+		controller = new Controller(
+			{},
+			[],
+			"invalid",
+			new lib.ControllerConfig("controller")
+		);
+		hostA = connectHost(controller, 10, []);
+		hostB = connectHost(controller, 20, []);
+		instanceA1 = connectInstance(controller, hostA, 11);
+		instanceA2 = connectInstance(controller, hostA, 12);
+		instanceB1 = connectInstance(controller, hostB, 21);
+		controlA = connectControl(controller, 100);
+		controlB = connectControl(controller, 101);
+	});
+
+	/** @returns {Controller | Host | Instance} */
+	function get(name) {
+		return {
+			controller,
+			hostA,
+			hostB,
+			instanceA1,
+			instanceA2,
+			instanceB1,
+			controlA,
+			controlB,
+		}[name];
+	}
+
+	// Combinations to try. Equivalent paths have been removed.
+	const parties = [
+		["controller", ["hostA", "instanceA1", "controlA"]],
+		["hostA", ["controller", "hostB", "instanceA1", "instanceB1", "controlA"]],
+		["instanceA1", ["controller", "hostA", "hostB", "instanceA2", "instanceB1", "controlA"]],
+		["controlA", ["controller", "hostA", "instanceA1", "controlB"]],
+	];
+
+	function handle(dst, Message, handler) {
+		if (dst instanceof Controller) {
+			for (const host of dst.wsServer.hostConnections.values()) {
+				host.handle(Message, handler);
+			}
+			for (const control of dst.wsServer.controlConnections.values()) {
+				control.handle(Message, handler);
+			}
+		} else if (dst instanceof Host) {
+			for (const instance of dst.instanceConnections.values()) {
+				instance.handle(Message, handler);
+			}
+			dst.handle(Message, handler);
+		} else {
+			dst.handle(Message, handler);
+		}
+	}
+
+	/**
+	 * @param {Controller | Host | Instance} link
+	 * @param {Address} dst
+	 */
+	function send(link, dst, message) {
+		// This logic exists becasue .sendTo doesn't forward all kinds of
+		// messages to the correct location from all places.  Idealy this
+		// would not be neccessary and .sendTo would do the right thing.
+		if (link instanceof Host) {
+			const instance = link.instanceConnections.get(dst.id);
+			if (instance) {
+				return instance.sendTo(dst, message);
+			}
+		}
+		return link.sendTo(dst, message);
+	}
+
+	describe("request routing", function() {
+		for (const [srcName, dstNames] of parties) {
+			it(`should route from ${srcName}`, async function() {
+				for (const dstName of dstNames) {
+					let called;
+					const dst = get(dstName);
+					handle(dst, TestRequest, async () => { called = true; });
+					const dstAddr = dst instanceof Controller ? addr("controller") : dst.connector.src;
+					await send(get(srcName), dstAddr, new TestRequest());
+					assert(called, `${dstName} handler was not called`);
+				}
+			});
+		}
+	});
+
+	describe("event routing", function() {
+		for (const [srcName, dstNames] of parties) {
+			it(`should route from ${srcName}`, async function() {
+				for (const dstName of dstNames) {
+					let called;
+					const dst = get(dstName);
+					handle(dst, TestEvent, async () => { called = true; });
+					const dstAddr = dst instanceof Controller ? addr("controller") : dst.connector.src;
+					send(get(srcName), dstAddr, new TestEvent());
+					assert(called, `${dstName} handler was not called`);
+				}
+			});
+		}
+	});
+
+	/**
+	 * @param {Controller | Host | Instance | Control} link
+	 * @param {"allHosts" | "allInstances" | "allControls"} dst
+	 */
+	function bcast(link, dst, message) {
+		// This logic exists becasue .sendTo doesn't forward all kinds of
+		// messages to the correct location from all places.  Idealy this
+		// would not be neccessary and .sendTo would do the right thing.
+		if (link instanceof Host) {
+			const plugin = message.constructor.plugin;
+			if (dst === "allInstances") {
+				for (const instance of link.instanceConnections.values()) {
+					if (plugin && !instance.plugins.has(plugin)) {
+						continue;
+					}
+					instance.sendTo(dst, message);
+				}
+			}
+			if (plugin && !link.serverPlugins.has(plugin)) {
+				return;
+			}
+		}
+		link.sendTo(dst, message);
+	}
+
+	function bhandle(srcName, dst, Event, canReach = (dstB) => true) {
+		const asserts = [];
+		const dsts = {
+			allHosts: ["hostA", "hostB"],
+			allInstances: ["instanceA1", "instanceA2", "instanceB1"],
+			allControls: ["controlA", "controlB"],
+		};
+		for (const dstName of dsts[dst]) {
+			if (srcName === dstName) { continue; }
+			// TODO remove when broadcasting from instance to hosts is fixed, see #575
+			if (srcName === "instanceA1" && dstName === "hostB") { continue; }
+			let called;
+			handle(get(dstName), Event, async () => { called = true; });
+			const should = canReach(dstName);
+			asserts.push(() => assert(
+				!should ^ called,
+				`${dstName} handler was ${should ? "not " : ""}called from ${srcName}`,
+			));
+		}
+		return asserts;
+	}
+
+	describe("event broadcast", function() {
+		for (const srcName of ["controller", "hostA", "instanceA1", "controlA"]) {
+			it(`should broadcast from ${srcName}`, async function() {
+				for (const dstName of ["allHosts", "allInstances", "allControls"]) {
+					const dstAsserts = bhandle(srcName, dstName, TestEvent);
+					bcast(get(srcName), dstName, new TestEvent());
+					for (const dstAssert of dstAsserts) {
+						dstAssert();
+					}
+				}
+			});
+		}
+	});
+
+	const graph = {
+		controller: ["hostA", "hostB", "controlA", "controlB"],
+		hostA: ["controller", "instanceA1", "instanceA2"],
+		hostB: ["controller", "instanceB1"],
+		instanceA1: ["hostA"],
+		instanceA2: ["hostA"],
+		instanceB1: ["hostB"],
+		controlA: ["controller"],
+		controlB: ["controller"],
+	};
+
+	// Returns true if the path from src to dst goes through via
+	function goesVia(src, dst, via) {
+		// Places we've been to
+		const checked = new Set();
+		// Places left to check
+		const paths = new Set([src]);
+		while (paths.size) {
+			const [candidate] = paths;
+			paths.delete(candidate);
+			if (candidate === via) {
+				continue;
+			}
+			if (checked.has(candidate)) {
+				continue;
+			}
+			if (candidate === dst) {
+				return false;
+			}
+			checked.add(candidate);
+			for (const newCandidate of graph[candidate]) {
+				paths.add(newCandidate);
+			}
+		}
+		return true;
+	}
+	assert(goesVia("instanceA1", "instanceB1", "controller"));
+	assert(!goesVia("instanceA1", "instanceA2", "controller"));
+	assert(goesVia("instanceA1", "hostB", "hostA"));
+	assert(!goesVia("instanceA1", "hostB", "controlA"));
+	assert(goesVia("controlA", "controlB", "controller"));
+	assert(goesVia("controlA", "hostA", "controller"));
+
+	function addPluginExceptFor(exName) {
+		const ex = get(exName);
+		const exAddr = ex instanceof Controller ? addr("controller") : ex.connector.src;
+		for (const name of [
+			"controller", "hostA", "hostB", "instanceA1", "instanceA2", "instanceB1", "controlA", "controlB",
+		]) {
+			const party = get(name);
+			if (party instanceof Controller) {
+				for (const host of party.wsServer.hostConnections.values()) {
+					if (!host.connector.dst.equals(exAddr)) {
+						host.plugins.set("test", "0.0.0");
+					}
+				}
+				for (const control of party.wsServer.controlConnections.values()) {
+					if (!control.connector.dst.equals(exAddr)) {
+						// No way to set plugins
+					}
+				}
+			} else if (party instanceof Host) {
+				for (const instance of party.instanceConnections.values()) {
+					if (!instance.connector.dst.equals(exAddr)) {
+						instance.plugins.set("test", "0.0.0");
+					}
+				}
+				if (!party.connector.dst.equals(exAddr)) {
+					party.serverPlugins.set("test", "0.0.0");
+				}
+			} else if (party instanceof Instance) {
+				// Currently does not handle plugins
+			} else if (party instanceof Control) {
+				// Currently does not handle plugins
+			}
+		}
+	}
+
+	// When plugin events are broadcast the party sending the event should
+	// filter it out if the next hop it is forwarded to does not have plugin
+	// the event was defined in loaded.
+	describe("event broadcast with controller missing plugin", function() {
+		for (const srcName of ["hostA", "instanceA1", "controlA"]) {
+			it(`should broadcast from ${srcName}`, async function() {
+				if (srcName === "controlA") {
+					this.skip(); // Not implemented
+				}
+				addPluginExceptFor("controller");
+				for (const dstName of ["allHosts", "allInstances", "allControls"]) {
+					const dstAsserts = bhandle(
+						srcName, dstName, PluginEvent, dstB => !goesVia(srcName, dstB, "controller"),
+					);
+					bcast(get(srcName), dstName, new PluginEvent());
+					for (const dstAssert of dstAsserts) {
+						dstAssert();
+					}
+				}
+			});
+		}
+	});
+
+	describe("event broadcast with host missing plugin", function() {
+		for (const srcName of ["controller", "hostA", "instanceA1", "controlA"]) {
+			it(`should broadcast from ${srcName}`, async function() {
+				addPluginExceptFor("hostB");
+				for (const dstName of ["allHosts", "allInstances", "allControls"]) {
+					const dstAsserts = bhandle(
+						srcName, dstName, PluginEvent, dstB => !goesVia(srcName, dstB, "hostB"),
+					);
+					bcast(get(srcName), dstName, new PluginEvent());
+					for (const dstAssert of dstAsserts) {
+						dstAssert();
+					}
+				}
+			});
+		}
+	});
+
+	describe("event broadcast with instance missing plugin", function() {
+		for (const srcName of ["controller", "hostA", "instanceA1", "controlA"]) {
+			it(`should broadcast from ${srcName}`, async function() {
+				addPluginExceptFor("instanceA2");
+				for (const dstName of ["allHosts", "allInstances", "allControls"]) {
+					const dstAsserts = bhandle(
+						srcName, dstName, PluginEvent, dstB => !goesVia(srcName, dstB, "instanceA2"),
+					);
+					bcast(get(srcName), dstName, new PluginEvent());
+					for (const dstAssert of dstAsserts) {
+						dstAssert();
+					}
+				}
+			});
+		}
+	});
+
+	describe("event broadcast with control missing plugin", function() {
+		for (const srcName of ["controller", "hostA", "instanceA1", "controlA"]) {
+			it(`should broadcast from ${srcName}`, async function() {
+				this.skip(); // Currently not implemented.
+				addPluginExceptFor("controlB");
+				for (const dstName of ["allHosts", "allInstances", "allControls"]) {
+					const dstAsserts = bhandle(
+						srcName, dstName, PluginEvent, dstB => !goesVia(srcName, dstB, "controlB"),
+					);
+					bcast(get(srcName), dstName, new PluginEvent());
+					for (const dstAssert of dstAsserts) {
+						dstAssert();
+					}
+				}
+			});
+		}
+	});
+});

test/lib/data/ModPack.js

@@ -39,6 +39,7 @@ describe("lib/data/ModPack", function() {
 				},
 				"runtime-per-user": {
 					"string-setting": { "value": "a string" },
+					"color-setting": { "value": { "r": 1, "g": 1, "b": 0, "a": 1 } },
 				},
 			}}));
 			check(ModPack.fromJSON({ export_manifest: { assets: { setting: "settings.json" }}}));
@@ -60,6 +61,7 @@ describe("lib/data/ModPack", function() {
 					},
 					"runtime-per-user": {
 						"string-setting": { "value": "a string" },
+						"color-setting": { "value": { "r": 1, "g": 1, "b": 0, "a": 1 } },
 					},
 				},
 				export_manifest: { assets: { setting: "settings.json" }},
@@ -101,6 +103,14 @@ describe("lib/data/ModPack", function() {
 						default_value: "a string",
 					},
 				},
+				"color-setting": {
+					"color": {
+						name: "color",
+						type: "color-setting",
+						setting_type: "runtime-per-user",
+						default_value: { "r": 1, "g": 1, "b": 1, "a": 1 },
+					},
+				},
 			};
 			const mockLogger = { warn: () => {} };
 			it("should fill in defaults for settings", function() {
@@ -115,6 +125,7 @@ describe("lib/data/ModPack", function() {
 					},
 					"runtime-per-user": {
 						"string": { "value": "a string" },
+						"color": { "value": { "r": 1, "g": 1, "b": 1, "a": 1 } },
 					},
 				});
 			});
@@ -128,6 +139,7 @@ describe("lib/data/ModPack", function() {
 					},
 					"runtime-per-user": {
 						"string": { "value": "spam" },
+						"color": { "value": { "r": 0, "g": 0.5, "b": 1, "a": 1 } },
 					},
 				}});
 				pack.fillDefaultSettings(prototypes, mockLogger);
@@ -140,6 +152,7 @@ describe("lib/data/ModPack", function() {
 					},
 					"runtime-per-user": {
 						"string": { "value": "spam" },
+						"color": { "value": { "r": 0, "g": 0.5, "b": 1, "a": 1 } },
 					},
 				});
 			});
@@ -204,6 +217,7 @@ describe("lib/data/ModPack", function() {
 					},
 					"runtime-per-user": {
 						"string-setting": { "value": "a string" },
+						"color-setting": { "value": { "r": 1, "g": 0.5, "b": 0, "a": 1 } },
 					},
 				}});
 
@@ -248,7 +262,7 @@ describe("lib/data/ModPack", function() {
 
 							...istr("runtime-per-user"), // name
 							Uint8Array.from([5, 0]), // dictionary
-							new Uint8Array(Uint32Array.from([1]).buffer), // items
+							new Uint8Array(Uint32Array.from([2]).buffer), // items
 
 								...istr("string-setting"), // name
 								Uint8Array.from([5, 0]), // dictinary
@@ -257,6 +271,30 @@ describe("lib/data/ModPack", function() {
 									...istr("value"), // name
 									Uint8Array.from([3, 0, 0, "a string".length]), // string
 									Buffer.from("a string"),
+
+								...istr("color-setting"), // name
+								Uint8Array.from([5, 0]), // dictinary
+								new Uint8Array(Uint32Array.from([1]).buffer), // items
+
+									...istr("value"), // name
+									Uint8Array.from([5, 0]), // dictinary
+									new Uint8Array(Uint32Array.from([4]).buffer), // items
+
+										...istr("r"), // name
+										Uint8Array.from([2, 0]), // number
+										new Uint8Array(Float64Array.from([1]).buffer),
+
+										...istr("g"), // name
+										Uint8Array.from([2, 0]), // number
+										new Uint8Array(Float64Array.from([0.5]).buffer),
+
+										...istr("b"), // name
+										Uint8Array.from([2, 0]), // number
+										new Uint8Array(Float64Array.from([0]).buffer),
+
+										...istr("a"), // name
+										Uint8Array.from([2, 0]), // number
+										new Uint8Array(Float64Array.from([1]).buffer),
 					])
 				);
 				/* eslint-enable indent */

test/lib/link/link.js

@@ -16,6 +16,7 @@ describe("lib/link/link", function() {
 
 	describe("class Link", function() {
 		let testConnector;
+		/** @type {lib.Link} */
 		let testLink;
 		let src = addr({ controlId: 1 });
 		let dst = addr("controller");
@@ -281,6 +282,7 @@ describe("lib/link/link", function() {
 				let srcReq = new lib.Address(lib.Address.control, 1, 1);
 				testConnector.emit("message", new lib.MessageResponse(1, dst, srcReq));
 				await request;
+				assert.deepEqual(testLink._pendingRequests, new Map());
 			});
 			it("should send request and reject with error when error response is received", async function() {
 				let message = events.once(testConnector, "send");
@@ -297,6 +299,7 @@ describe("lib/link/link", function() {
 					request,
 					{ message: "Error" }
 				);
+				assert.deepEqual(testLink._pendingRequests, new Map());
 			});
 			it("should send request and resolve with data when response is received", async function() {
 				let message = events.once(testConnector, "send");
@@ -305,6 +308,7 @@ describe("lib/link/link", function() {
 				let srcReq = new lib.Address(lib.Address.control, 1, 1);
 				testConnector.emit("message", new lib.MessageResponse(1, dst, srcReq, 44));
 				assert.deepEqual(await request, new NumberRequest.Response(44));
+				assert.deepEqual(testLink._pendingRequests, new Map());
 			});
 			it("should send event to the other side of the link", async function() {
 				let message = events.once(testConnector, "send");

test/mock.js

@@ -80,6 +80,10 @@ class MockConnector extends lib.BaseConnector {
 		this.sentMessages.push(message);
 		setImmediate(() => this.emit("send", message));
 	}
+
+	sendDisconnectReady() {
+		this.send(new lib.MessageDisconnect("ready"));
+	}
 }
 
 class MockServer extends events.EventEmitter {