Add Factorio to Clusterio communication API

Add send_json to the Clusterio module's API which allows sending
arbitrary tables serialized as JSON to plugins that listen on the
channel specified.

docs/developing-for-clusterio.md

@@ -12,7 +12,6 @@ Contents
 - [Factorio Mods](#factorio-mods)
 - [Factorio Scenarios](#factorio-scenarios)
     - [event_handler interface](#event_handler-interface)
-- [Communicating between Clusterio and Factorio](#communicating-between-clusterio-and-factorio)
 - [Communicating with Clusterio](#communicating-with-clusterio)
 - [Clusterio Modules](#clusterio-modules)
 - [Clusterio Plugins](#clusterio-plugins)
@@ -158,15 +157,6 @@ There's also the `add_libraries` function exported by event_handler,
 which accepts a table and calls `add_lib` for each value in the table.
 
 
-Communicating between Clusterio and Factorio
---------------------------------------------
-
-There is currently no standard interface for moving data between
-Clusterio and Factorio.  It's mostly a mix of RCON commands initiated
-from Clusterio plugins and writing out files with `game.write_file` that
-is picked up and parsed by plugins.
-
-
 Communicating with Clusterio
 ----------------------------
 

docs/writing-plugins.md

@@ -11,6 +11,7 @@ Contents
 
 - [Plugin Structure](#plugin-structure)
 - [Defining the plugin class](#defining-the-plugin-class)
+- [Communicating with Factorio](#communicating-with-factorio)
 - [Defining Link Messages](#defining-link-messages)
     - [Defining Events](#defining-events)
     - [Defining Requests](#defining-requests)
@@ -141,6 +142,74 @@ result in unpredicatable behaviour.  The async init method always called
 immediatly after the constructor so there's little reason to do this.
 
 
+Communicating with Factorio
+---------------------------
+
+For pushing data into Factorio there's RCON which lets you send
+arbitrary Lua command to invoke whatever code you want in the game.
+This is done by calling sendRcon method on the instance's server object.
+For example:
+
+    async onStart() {
+        let response = await this.instance.server.sendRcon(
+            "/sc rcon.print('data')"
+        );
+
+        // Do stuff with response.
+    }
+
+
+Because data into Factorio is streamed at a rate of 3-6 kB/s by default
+it is recommended to avoid sending large commands as much as possible,
+and to strip down the data on the ones you send to only what's strictly
+necessary.  You can have lua code injected into the game via the module
+system and call that from RCON to avoid having to send code through the
+commands.
+
+For getting data out from Factorio there's both RCON and the `send_json`
+API of the Clusterio module.  Returning data via RCON is prefered if the
+action is initiated from the Node.js side.  The `send_json` API allows
+sending JSON payloads on channels that plugins can listen to.  From a
+plugin you listen for an event named ipc-channel_name in order to get
+data sent by `send_json`.  For example in the plugin code:
+
+    async init() {
+        this.server.on('ipc-my_plugin_foo', content =>
+            this.handleFoo(content).catch(err => console.log(
+                "Error handling foo:", foo
+            ))
+        );
+    }
+
+    async handleFoo(content) {
+        // Do stuff with content
+    }
+
+And then in the module for the plugin:
+
+    local clusterio_api = require('modules/clusterio/api')
+
+    -- inside some event handler
+    clusterio_api.send('my_plugin_foo', { data = 123 })
+
+It's recommended to either use the plugin name as the channel name or to
+prefix the channel name with the name of the plugin if you need multiple
+channels.  It's also important to catch any errors that might occur as
+they will otherwise be propogated to the instance code and kill the
+server.
+
+Data out from Factorio does not have the same limits as data into
+Factorio, RCON responses can be in 100kB range without causing issues,
+and payloads to the `send_json` API can be in the 4MB range provided the
+server has a fast enough storage system.
+
+**Note:** both `send_json` and RCON can operate out of order.  For
+`send_json` it's possible that payloads greater than 4kB are received
+after payloads that was sent at a later point in time.  For RCON,
+commands longer than 50 characters may end up being executed after
+shorter commands sent after it.
+
+
 Defining Link Messages
 ----------------------
 

lib/factorio/server.js

@@ -377,7 +377,56 @@ class FactorioServer extends events.EventEmitter {
 		}
 	}
 
+	async _handleIpc(line) {
+		let channelEnd = line.indexOf('?');
+		if (channelEnd === -1) {
+			throw new Error(`Malformed IPC line "${line.toString()}"`);
+		}
+
+		let channel = line
+			.slice(6, channelEnd)
+			.toString('utf-8')
+			.replace(/\\x([0-9a-f]{2})/g, (match, p1) =>
+				String.fromCharCode(parseInt(p1, 16))
+			)
+		;
+
+		let type = line.slice(channelEnd + 1, channelEnd + 2).toString('utf-8');
+		let content;
+		if (type === 'j') {
+			content = JSON.parse(line.slice(channelEnd + 2).toString('utf-8'));
+
+		} else if (type === 'f') {
+			let fileName = line.slice(channelEnd + 2).toString('utf-8');
+			let filePath = this.writePath('script-output', fileName);
+
+			// Prevent malicious names
+			if (/[\\/\0]/.test(fileName)) {
+				throw new Error(`Invalid IPC file name '${fileName}'`)
+			}
+
+			if (fileName.endsWith('.json')) {
+				content = JSON.parse(await fs.readFile(filePath, 'utf-8'));
+				await fs.unlink(filePath);
+
+			} else {
+				let format = fileName.slice(fileName.lastIndexOf('.') + 1);
+				throw new Error(`Unknown IPC file format '${format}'`)
+			}
+
+		} else {
+			throw new Error(`Unknown IPC type '${type}'`);
+		}
+
+		this.emit(`ipc-${channel}`, content);
+	}
+
 	_handleOutput(line, source) {
+		if (line.slice(0, 6).equals(Buffer.from("\f$ipc:"))) {
+			this._handleIpc(line).catch(err => this.emit('error', err));
+			return;
+		}
+
 		this.emit(source, line);
 
 		let output = parseOutput(line.toString('utf-8'), source);

modules/clusterio/api.lua

@@ -12,6 +12,42 @@ api.events = {
     on_instance_updated = script.generate_event_name(),
 }
 
+-- Send a table as json to Clusterio
+-- Send data to Clusterio over the given channel.  Clusterio plugins can
+-- listen to channels and will receive an event with the data sent here.
+-- See docs/writing-plugins.md for more information.
+
+-- Note: Payloads greater than 4 MB will cause stuttering.
+
+-- Note: This is not a binary safe way of sending data.  Strings embedded
+-- into the tables sent must be valid UTF-8 text.
+
+-- Parameters:
+--   channel: string identifying which channel to send it on.
+--   data: table that can be converted to json with game.table_to_json
+function api.send_json(channel, data)
+
+    -- Escape bad characters.  The question mark is used for separating the
+    -- channel name from the payload.
+    channel = channel:gsub("([\x00-\x1f?\\])", function(match)
+        return "\\x" .. string.format("%02x", match:byte())
+    end)
+
+    data = game.table_to_json(data)
+
+    -- If there's more than about 4kB of data users running with the Windows
+    -- console open will start to experience stuttering, otherwise we could
+    -- output about 1 MB of data at a time through stdout.
+    if #data < 4000 then
+        print("\f$ipc:" .. channel .. "?j" .. data)
+    else
+        global.clusterio_file_no = (global.clusterio_file_no or 0) + 1
+        local file_name = "clst_" .. global.clusterio_file_no .. ".json"
+        game.write_file(file_name, data, false, 0)
+        print("\f$ipc:" .. channel .. "?f" .. file_name)
+    end
+end
+
 
 return api
 

test/lib/factorio/server.js

@@ -1,4 +1,5 @@
 const assert = require("assert").strict;
+const events = require("events");
 const fs = require("fs-extra");
 const path = require("path");
 
@@ -122,5 +123,53 @@ describe("lib/factorio/server", function() {
 				assert.equal(server.version, "0.1.1");
 			});
 		});
+
+		describe("._handleIpc()", function() {
+			it("should emit the correct ipc event", async function() {
+				let waiter = events.once(server, 'ipc-channel');
+				await server._handleIpc(Buffer.from('\f$ipc:channel?j"value"'));
+				let result = await waiter;
+				assert.equal(result[0], "value");
+			});
+			it("should handle special characters in channel name", async function() {
+				let waiter = events.once(server, 'ipc-$ ?\x00\x0a:');
+				await server._handleIpc(Buffer.from('\f$ipc:$ \\x3f\\x00\\x0a:?j"value"'));
+				let result = await waiter;
+				assert.equal(result[0], "value");
+			});
+			it("should throw on malformed ipc line", async function() {
+				await assert.rejects(
+					server._handleIpc(Buffer.from('\f$ipc:blah')),
+					new Error('Malformed IPC line "\f$ipc:blah"')
+				);
+			})
+			it("should throw on unknown type", async function() {
+				await assert.rejects(
+					server._handleIpc(Buffer.from('\f$ipc:channel??')),
+					new Error("Unknown IPC type '?'")
+				);
+			})
+			it("should throw on unknown file type", async function() {
+				await assert.rejects(
+					server._handleIpc(Buffer.from('\f$ipc:channel?ffoo.invalid')),
+					new Error("Unknown IPC file format 'invalid'")
+				);
+			})
+			it("should throw on file name with slash", async function() {
+				await assert.rejects(
+					server._handleIpc(Buffer.from('\f$ipc:channel?fa/b')),
+					new Error("Invalid IPC file name 'a/b'")
+				);
+			})
+			it("should load and delete json file", async function() {
+				let filePath = server.writePath('script-output', 'data.json');
+				await fs.outputFile(filePath, '{"data":"spam"}');
+				let waiter = events.once(server, 'ipc-channel');
+				await server._handleIpc(Buffer.from('\f$ipc:channel?fdata.json'));
+				let result = await waiter;
+				assert.deepEqual(result[0], { "data": "spam" });
+				assert(!await fs.pathExists(filePath), "File was not deleted");
+			})
+		});
 	});
 });