Release 1.2.4

Mark version 1.2.4 for release.

CHANGELOG.md

@@ -1,6 +1,36 @@
 Changelog
 =========
 
+Version 1.2.4
+-------------
+
+- Removed broken remote combinator signaling.
+- Fixed research sync endlessly updating already researched technologies
+- Removed obsolete item/fluid statistics from clusterioMod
+- Removed per mod upload logging when config.uploadModsToMaster disabled
+- Fixed bcrypt failing to install on windows due to the new version not
+  having Windows binaries.
+
+
+Version 1.2.3
+-------------
+
+- Disabled uploading mods to the master server by default as this is mostly
+  just a waste of bandwidth with the mod portal being integrated into the
+  game now.
+- Fixed researchSync breaking with heavily modded games due to the tech tree
+  exceeding the default 100kb limit on JSON payloads.
+- Fixed the command inventoryImport sends referencing player instead of
+  character, and trying to count the non-existant quick bar slots.
+- Fixed inventoryImport never receiving the script output after the default
+  mode for reading script output changed to tail mode.
+- Fixed desync caused by the mods and loaded mods arrays in Hotpatch getting
+  out of sync when plugins update their scenario mod code.  To fix existing
+  games you will need copy lib/scenarios/Hotpatch/hotpatch/mod-tools.lua
+  over the existing mod-tools.lua in the save's hotpatch folder.
+- Updated the Windows install instructions.
+
+
 Version 1.2.2
 -------------
 

README.md

@@ -11,7 +11,7 @@ Discord for development/support/play: https://discord.gg/5XuDkje
 ## Important notice
 
 This is the stable branch of the project.  The
-[master branch][http://github.com/clusterio/factorioClusterio/tree/master]
+[master branch](http://github.com/clusterio/factorioClusterio/tree/master)
 is undergoing significant changes that breaks compatibility to plugins and
 existing installations and is not recommended for use.
 
@@ -177,28 +177,47 @@ reboot when you are done, then proceed to the next steps. *reboots matter*
 
 **Master**
 
-1. download and run https://puu.sh/toFHl/01eebbb333.bat
+1. Open PowerShell or Command prompt in the directory you want to install to and run the following commands.
 
-2. Copy config.json.dist to config.json
+        git clone -b 1.2.x https://github.com/clusterio/factorioClusterio
+        cd factorioClusterio
+        npm install --only=production
+        copy config.json.dist config.json
 
-3. Follow the instructions in the bat file
-3. Some of the instructions are outdated. If you get stuck somewhere, look at the Ubuntu section.
+2. Obtain Factorio by either of these two methods:
 
-4. Run `node client.js manage shared mods add clusterio`
+    - Via the stand alone version on from their website
 
-5. type `node master.js` to start the server
+        1. Download the MS Windows (64-bit zip package) from https://www.factorio.com/download
+
+        2. Open the zip file and drag the folder called "Factorio_0.17.x" into the factorioClusterio folder
+
+        3. Rename the folder to "factorio"
+
+    - Via steam installation
+
+        1. Locate the game files by right clicking the game in steam, selecting properties, then Local Files, then Browse local files.
+
+        2. Go to the parent folder of the folder that Steam opened and copy the Factorio folder into the factorioClusterio folder
+
+        3. Rename the folder to "factorio"
+
+
+3. Open `config.json` with a text editor and configure as desired.
+
+4. Run `node master.js` to generate the athentication token into secret-api-token.txt
+
+5. Run `node master.js` again to start the master server.
 
 **Server Host**
 
-1. download and run https://puu.sh/toFHl/01eebbb333.bat
+1. Do step 1 and 2 of the Master section above *OR* use the same folder that was created in that section.
 
-2. Copy config.json.dist to config.json
+2. Open `config.json` with a text editor and configure as desired.  You will need to set masterAuthToken to string found in secret-api-token.txt on the master server.
 
-3. Follow the instructions given.
+3. Optionally run the command `node client.js manage shared mods add clusterio` to add the clusterio mod (needed for item teleports.)
 
-3.5 Some of the instructions are outdated. If you get stuck somewhere, look at the Ubuntu section.
-
-4. Type `node client.js start [instancename]` to create a new instance.
+4. Run `node client.js start [instancename]` to create a new instance.  Repeat it again to start the instance.
 
 To connect to a master server running on a remote machine, open config.json with your favourite text editor (notepad++). You can also set it up to use the official server browser.
 

client.js

@@ -299,7 +299,6 @@ if (!command || command == "help" || command == "--help") {
 	fs.mkdirSync(instancedirectory + "/saves/");
 	fs.writeFileSync(instancedirectory + "/script-output/output.txt", "");
 	fs.writeFileSync(instancedirectory + "/script-output/orders.txt", "");
-	fs.writeFileSync(instancedirectory + "/script-output/txbuffer.txt", "");
 	fs.mkdirSync(instancedirectory + "/mods/");
 	fs.mkdirSync(instancedirectory + "/instanceMods/");
     fs.mkdirSync(instancedirectory + "/scenarios/");
@@ -738,19 +737,19 @@ async function instanceManagement(instanceconfig) {
 	
 	setTimeout(function(){hashMods(instance, uploadMods)}, 5000);
 	function uploadMods(modHashes) {
-		// [{modName:string,hash:string}, ... ]
-		for(let i=0;i<modHashes.length;i++){
-			let payload = {
-				modName: modHashes[i].modName,
-				hash: modHashes[i].hash,
-			};
-			needle.post(config.masterIP + ":" + config.masterPort + '/api/checkMod', payload, needleOptionsWithTokenAuthHeader, function (err, response, body) {
-				if(err) console.error("Unable to contact master server /api/checkMod! Please check your config.json.");
-				if(response && body && body == "found") {
-					console.log("master has mod "+modHashes[i].modName);
-				} else if (response && body && typeof body == "string") {
-					let mod = response.body;
-					if(config.uploadModsToMaster){
+		if (config.uploadModsToMaster) {
+			// [{modName:string,hash:string}, ... ]
+			for(let i=0;i<modHashes.length;i++){
+				let payload = {
+					modName: modHashes[i].modName,
+					hash: modHashes[i].hash,
+				};
+				needle.post(config.masterIP + ":" + config.masterPort + '/api/checkMod', payload, needleOptionsWithTokenAuthHeader, function (err, response, body) {
+					if(err) console.error("Unable to contact master server /api/checkMod! Please check your config.json.");
+					if(response && body && body == "found") {
+						console.log("master has mod "+modHashes[i].modName);
+					} else if (response && body && typeof body == "string") {
+						let mod = response.body;
 						console.log("Sending mod: " + mod);
 						// Send mods master says it wants
 						// response.body is a string which is a modName.zip
@@ -767,11 +766,11 @@ async function instanceManagement(instanceconfig) {
 						});
 						var form = req.form();
 						form.append('file', fs.createReadStream(config.instanceDirectory+"/"+instance+"/mods/"+mod));
-					} else {
-						console.log("Not sending mod: " + mod + " to master because config.uploadModsToMaster is not enabled")
 					}
-				}
-			});
+				});
+			}
+		} else {
+			console.log("Bypassing sending mods to master because config.uploadModsToMaster is not enabled")
 		}
 	}
 } // END OF INSTANCE START ---------------------------------------------------------------------

config.json.dist

@@ -50,7 +50,7 @@
 	"__comment6.5": "When the master is undersupplied, we have 3 different modes to 'fairly' distribute items. First to ask gets all, dole division using weird formula or experimental neural net tries to make stuff as fair as possible. Feel free to play around.",
 	"disableFairItemDistribution": false,
 	"useNeuralNetDoleDivider": true,
-	"uploadModsToMaster": true,
+	"uploadModsToMaster": false,
 	"msBetweenCommands": 10,
 	"allowRemoteCommandExecution": true,
 	"enableCrossServerShout": true,

lib/scenarios/Hotpatch/hotpatch/mod-tools.lua

@@ -461,8 +461,9 @@ install_mod = function(mod_name, mod_version, mod_code, mod_files)
     local index = find_installed_mod(mod_name)
     local mod = {}
     if index then
-        -- TODO: notify about installing over top of existing mod
-        mod = global.mods[index]
+        debug_log('attempt to install mod that is already installed, reinstalling: ' .. mod_name)
+        mod = table.remove(global.mods, index)
+        table.insert(global.mods, mod)
     else
         --next free index
         table.insert(global.mods, mod)

master.js

@@ -1,7 +1,6 @@
 /**
 Clusterio master server. Facilitates communication between slaves through
-a webserver, storing data related to slaves like production graphs and
-combinator signals.
+a webserver, storing data related to slaves like production graphs.
 
 @module clusterioMaster
 @author Danielv123
@@ -83,7 +82,9 @@ const fileUpload = require('express-fileupload');
 var app = express();
 
 app.use(cookieParser());
-app.use(bodyParser.json());
+app.use(bodyParser.json({
+	limit: '10mb',
+}));
 app.use(bodyParser.urlencoded({
 	parameterLimit: 100000,
 	limit: '10mb',
@@ -874,14 +875,6 @@ class wsSlave {
 			prometheusWsUsageCounter.labels('heartbeat', this.instanceID).inc();
 			this.lastBeat = Date.now();
 		});
-		this.socket.on("combinatorSignal", circuitFrameWithMeta => {
-			prometheusWsUsageCounter.labels('combinatorSignal', this.instanceID).inc();
-			if(circuitFrameWithMeta && typeof circuitFrameWithMeta == "object"){
-				Object.keys(wsSlaves).forEach(instanceID => {
-					wsSlaves[instanceID].socket.emit("processCombinatorSignal", circuitFrameWithMeta);
-				});
-			}
-		});
 		// handle command return values
 		this.commandsWaitingForReturn = {};
 		this.socket.on("runCommandReturnValue", resp => {

package.json

@@ -1,6 +1,6 @@
 {
   "name": "clusterio",
-  "version": "1.2.2",
+  "version": "1.2.4",
   "description": "Interserver communication for factorio",
   "repository": "https://github.com/Danielv123/factorioClusterio",
   "main": "master.js",
@@ -23,7 +23,7 @@
     "as-table": "^1.0.31",
     "averaged-timeseries": "^1.4.0",
     "base64url": "^3.0.0",
-    "bcrypt": "^3.0.6",
+    "bcrypt": "=3.0.6",
     "bcrypt-promise": "^2.0.0",
     "body-parser": "^1.18.2",
     "cli": "^1.0.1",

sharedPlugins/clusterioMod/index.js

@@ -35,61 +35,6 @@ module.exports = class remoteCommands {
 			}
 		})().catch(e => console.log(e));
 		
-		// flow/production statistics ------------------------------------------------------------
-		var oldFlowStats = false;
-		var oldTimestamp;
-		var oldFlowStats;
-		setInterval(function(){
-			fs.readFile(instancedirectory + "/script-output/flows.txt", {encoding: "utf8"}, function(err, data) {
-				if(!err && data) {
-					let timestamp = Date.now();
-					data = data.split("\n");
-					let flowStats = [];
-					for(let i = 0; i < data.length; i++) {
-						// try catch to remove any invalid json
-						try{
-							flowStats[flowStats.length] = JSON.parse(data[i]);
-						} catch (e) {
-							// console.log(" invalid json: " + i);
-							// some lines of JSON are invalid but don't worry, we just filter em out
-						}
-					}
-					// fluids
-					let flowStat1 = flowStats[flowStats.length-1].flows.player.input_counts
-					// items
-					let flowStat2 = flowStats[flowStats.length-2].flows.player.input_counts
-					// merge fluid and item flows
-					let totalFlows = {};
-					for(let key in flowStat1) totalFlows[key] = flowStat1[key];
-					for(let key in flowStat2) totalFlows[key] = flowStat2[key];
-					if(oldFlowStats && totalFlows && oldTimestamp) {
-						let payload = objectOps.deepclone(totalFlows);
-						// change from total reported to per time unit
-						for(let key in oldFlowStats) {
-							// get production per minute
-							payload[key] = Math.floor((payload[key] - oldFlowStats[key])/(timestamp - oldTimestamp)*60000);
-							if(payload[key] < 0) {
-								payload[key] = 0;
-							}
-						}
-						for(let key in payload) {
-							if(payload[key] == '0') {
-								delete payload[key];
-							}
-						}
-						console.log("Recorded flows, copper plate since last time: " + payload["copper-plate"]);
-						needle.post(that.config.masterIP + ":" + that.config.masterPort + '/api/logStats', {timestamp: timestamp, instanceID: that.config.unique, data: payload}, needleOptionsWithTokenAuthHeader, function (err, response, body) {
-							// we did it, keep going
-						});
-					}
-					oldTimestamp = timestamp;
-					oldFlowStats = totalFlows;
-					fs.writeFileSync(instancedirectory + "/script-output/flows.txt", "");
-				}
-			});
-			// we don't need to update stats quickly as that could be expensive
-		}, 60000*5);
-		
 		// provide items --------------------------------------------------------------
 		// trigger when something happens to output.txt
 		fs.watch(instancedirectory + "/script-output/output.txt", function (eventType, filename) {
@@ -232,56 +177,6 @@ module.exports = class remoteCommands {
 		setTimeout(function(){
 			messageInterface("/silent-command remote.call('clusterio','setWorldID',"+that.config.unique+")")
 		}, 20000);
-		/* REMOTE SIGNALLING
-		 * send any signals the slave has been told to send
-		 * Fetch combinator signals from the server
-		*/
-		this.socket.on("processCombinatorSignal", circuitFrameWithMeta => {
-			if(circuitFrameWithMeta && typeof circuitFrameWithMeta == "object" && circuitFrameWithMeta.frame && Array.isArray(circuitFrameWithMeta.frame)){
-				messageInterface("/silent-command remote.call('clusterio', 'receiveFrame', '"+JSON.stringify(circuitFrameWithMeta.frame)+"')");
-			}
-		});
-		// get outbound frames from file and send to master
-		// get array of lines in file, each line should correspond to a JSON encoded frame
-		let signals = fs.readFileSync(instancedirectory + "/script-output/txbuffer.txt", "utf8").split("\n");
-		// if we actually got anything from the file, proceed and reset file
-		let readingTxBufferSoon = false;
-		let txBufferClearCounter = 0;
-		fs.watch(instancedirectory + "/script-output/txbuffer.txt", "utf-8", (eventType, filename) => {
-			if(!readingTxBufferSoon){ // use a 100ms delay to avoid messing with rapid sequential writes from factorio (I think that might be a problem maybe?)
-				readingTxBufferSoon = true;
-				setTimeout(()=>{
-					txBufferClearCounter++;
-					fs.readFile(instancedirectory + "/script-output/txbuffer.txt", "utf-8", (err, signals) => {
-						signals = signals.split("\n");
-						if (signals[0]) {
-							//if(txBufferClearCounter > 500){
-								fs.writeFileSync(instancedirectory + "/script-output/txbuffer.txt", "");
-							//	txBufferClearCounter = 0;
-							//}
-							
-							// loop through all our frames
-							for (let i = 0; i < signals.length; i++) {
-								if (signals[i] && objectOps.isJSON(signals[i])) {
-									// signals[i] is a JSON array called a "frame" of signals. We timestamp it for storage on master
-									// then we unpack and RCON in this.frame to the game later.
-									let framepart = JSON.parse(signals[i]);
-									let doneframe = {
-										time: Date.now(),
-										frame: framepart, // thats our array of objects(single signals);
-									}
-									// send to master using socket.io, opened at the top of instanceManagement()
-									this.socket.emit("combinatorSignal", doneframe);
-								} else {
-									// console.log("Invalid jsony: "+typeof signals[i])
-								}
-							}
-						}
-					});
-					readingTxBufferSoon = false;
-				},100);
-			}
-		});
 	}
 	async getCommand(file){
 		this.commandCache = this.commandCache || {};

sharedPlugins/inventoryImports/config.js

@@ -17,4 +17,5 @@ module.exports = {
 	description: "Clusterio plugin to fill character logistic slots with master imports",
 	// We'll send everything in this file to your stdin. Beware.
 	scriptOutputFileSubscription: "inventoryImports.txt",
+	fileReadDelay: 100,
 }

sharedPlugins/inventoryImports/interfacingFunctions.js

@@ -141,7 +141,10 @@ function pollInventories(outputFile){
 	//console.log("/silent-command "+'local a=game.players["Danielv123"].get_inventory(defines.inventory.player_main).get_contents()local b=game.players["Danielv123"].get_quickbar().get_contents()local c={}for d=1,game.players["Danielv123"].force.character_logistic_slot_count do c[d]=game.players["Danielv123"].character.get_request_slot(d)end;local e="{inventory:{"for f,g in pairs(a)do e=e.."['+"'"+'"..f.."'+"'"+']:"..g..","end;for f,g in pairs(b)do e=e.."['+"'"+'"..f.."'+"'"+']:"..g..","end;e=e.."},requestSlots:{"for f,g in pairs(c)do e=e.."['+"'"+'"..g["name"].."'+"'"+']:"..g["count"]..", "end;e=e.."}}"game.write_file("'+pluginConfig.scriptOutputFileSubscription+'",e)')
 	// console.log("/silent-command "+'local a="{"for b,c in pairs(game.players)do local d=game.players[b].get_inventory(defines.inventory.player_main).get_contents()local e=game.players[b].get_quickbar().get_contents()local f={}for g=1,game.players[b].force.character_logistic_slot_count do f[g]=game.players[b].character.get_request_slot(g)end;a=a..b..":{inventory:{"for h,i in pairs(d)do a=a.."['+"'"+'"..h.."'+"'"+']:"..i..","end;for h,i in pairs(e)do a=a.."['+"'"+'"..h.."'+"'"+']:"..i..","end;a=a.."},requestSlots:{"for h,i in pairs(f)do a=a.."['+"'"+'"..i["name"].."'+"'"+']:"..i["count"]..", "end;a=a.."}},"end;a=a.."}"game.write_file("'+pluginConfig.scriptOutputFileSubscription+'",a)')
 	//return "/silent-command "+'local a="{"for b,c in pairs(game.players)do if c.connected then local d=game.players[b].get_inventory(defines.inventory.player_main).get_contents()local e=game.players[b].get_quickbar().get_contents()local f={}for g=1,game.players[b].force.character_logistic_slot_count do f[g]=game.players[b].character.get_request_slot(g)end;a=a..b..":{inventory:{"for h,i in pairs(d)do a=a.."['+"'"+'"..h.."'+"'"+']:"..i..","end;for h,i in pairs(e)do a=a.."['+"'"+'"..h.."'+"'"+']:"..i..","end;a=a.."},requestSlots:{"for h,i in pairs(f)do a=a.."['+"'"+'"..i["name"].."'+"'"+']:"..i["count"]..", "end;a=a.."}},"end end;a=a.."}"game.write_file("'+outputFile+'",a)'
-	return '/silent-command '+'local a='+"'"+'{"players":{'+"'"+'local b=false;for c,d in pairs(game.players)do if d.connected and d.character then if b then a=a..'+"'"+','+"'"+'else b=true end;local e=game.players[c].get_inventory(defines.inventory.player_main).get_contents()local f=game.players[c].get_quickbar().get_contents()local g={}local z={}for j,k in pairs(e)do if z[j] ~= nil then z[j] = z[j] + k else z[j] = k end end for j,k in pairs(f)do if z[j] ~= nil then z[j] = z[j] + k else z[j] = k end end for h=1,game.players[c].force.character_logistic_slot_count do g[h]=game.players[c].character.get_request_slot(h)end;a=a..'+"'"+'"'+"'"+'..c..'+"'"+'":{"inventory":{'+"'"+'local i=false;for j,k in pairs(z)do if i then a=a..'+"'"+','+"'"+'else i=true end;a=a..'+"'"+'"'+"'"+'..j..'+"'"+'":'+"'"+'..k end;a=a..'+"'"+'},"requestSlots":{'+"'"+'i=false;for j,k in pairs(g)do if i then a=a..'+"'"+','+"'"+'else i=true end;a=a..'+"'"+'"'+"'"+'..k['+"'"+'name'+"'"+']..'+"'"+'":'+"'"+'..k['+"'"+'count'+"'"+']end;a=a..'+"'"+'}}'+"'"+'end end;a=a.."}}"game.write_file("'+outputFile+'",a,true,0)'
+	//return '/silent-command '+'local a='+"'"+'{"players":{'+"'"+'local b=false;for c,d in pairs(game.players)do if d.connected and d.character then if b then a=a..'+"'"+','+"'"+'else b=true end;local e=game.players[c].get_inventory(defines.inventory.player_main).get_contents()local f=game.players[c].get_quickbar().get_contents()local g={}local z={}for j,k in pairs(e)do if z[j] ~= nil then z[j] = z[j] + k else z[j] = k end end for j,k in pairs(f)do if z[j] ~= nil then z[j] = z[j] + k else z[j] = k end end for h=1,game.players[c].force.character_logistic_slot_count do g[h]=game.players[c].character.get_request_slot(h)end;a=a..'+"'"+'"'+"'"+'..c..'+"'"+'":{"inventory":{'+"'"+'local i=false;for j,k in pairs(z)do if i then a=a..'+"'"+','+"'"+'else i=true end;a=a..'+"'"+'"'+"'"+'..j..'+"'"+'":'+"'"+'..k end;a=a..'+"'"+'},"requestSlots":{'+"'"+'i=false;for j,k in pairs(g)do if i then a=a..'+"'"+','+"'"+'else i=true end;a=a..'+"'"+'"'+"'"+'..k['+"'"+'name'+"'"+']..'+"'"+'":'+"'"+'..k['+"'"+'count'+"'"+']end;a=a..'+"'"+'}}'+"'"+'end end;a=a.."}}"game.write_file("'+outputFile+'",a,true,0)'
+
+	// Warning: This command is not in sync with lua commands.lua
+	return `/silent-command local a='{"players":{'local b=false;for c,d in pairs(game.players)do if d.connected and d.character then if b then a=a..','else b=true end;local e=game.players[c].get_inventory(defines.inventory.character_main).get_contents()local g={}for h=1,game.players[c].force.character_logistic_slot_count do g[h]=game.players[c].character.get_request_slot(h)end;a=a..'"'..c..'":{"inventory":{'local i=false;for j,k in pairs(e)do if i then a=a..','else i=true end;a=a..'"'..j..'":'..k end;a=a..'},"requestSlots":{'i=false;for j,k in pairs(g)do if i then a=a..','else i=true end;a=a..'"'..k['name']..'":'..k['count']end;a=a..'}}'end end;a=a.."}}"game.write_file("${outputFile}",a,true,0)`
 	/*
 	needle.post(config.masterIP+':'+config.masterPort+'/api/editSlaveMeta', {instanceID: config.unique, password: config.clientPassword, meta: {UPS:UPS}}, function(err, resp) {
 		// success?

sharedPlugins/inventoryImports/lua commands.lua

@@ -1,3 +1,4 @@
+-- WARNING: The commands in this file is not in sync with interfacingFunctions.js
 -- Lua cheatsheet
 
 /c game.write_file("test.txt",serpent.block(game.players["Danielv123"].get_quickbar().get_contents()))

sharedPlugins/researchSync/index.js

@@ -284,6 +284,11 @@ class ResearchSync {
             if (isNaN(cluster_techs[name].progress))
                 continue
 
+			// Ignore researched techs
+			if (!tech.infinite && tech.researched) {
+				continue;
+			}
+
 			// Do not update progress for infinite techs with a local level that's higher.
 			if (tech.infinite && tech.level > cluster_techs[name].level) {
 				continue;