Compare commits

...
13 Commits
Author SHA1 Message Date
Cooldude2606 a49841f3fc Merge pull request #1237 from bbassie/fix/stream-slow-receiver-test
Measure transfer_min_speed in KiB/s and raise it to 64
2026-11-20 22:33:08 +00:00
bbassieandClaude Opus 5.5 2e273dc630 Take a burst allowance off what a transfer check requires
controller.transfer_min_speed is now the speed that is never closed.
A check requires that speed over 60 s less 2 MiB, which covers the
bursts the OS sends data in, so at the default of 64 KiB/s the cutoff
is about 30 KiB/s. The requirement is at least 1 byte, so a transfer
that moved nothing is still closed at low settings.

The slow receiver test sends 16 MiB so that it spans real checks, and
two upload tests with set bursts cover the allowance.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 22:13:51 +00:00
bbassieandClaude Opus 5.5 42bc31e406 Check transfer speed over 60 seconds
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 20:59:37 +00:00
bbassieandClaude Opus 5.5 19569aefa4 Measure transfer_min_speed in KiB/s and raise it to 64
The guard measures a proxy stream on completed writes to the receiver.
The OS buffers a couple of MiB per connection and only takes more once
a large part has been sent, so a slow receiver can go a long time with
no write completing. At the old default of 30 KiB per 30 s, downloads
well above that could be closed, and the slow receiver test failed
every run on Linux.

controller.transfer_min_progress (KiB per 30 s) is replaced by
controller.transfer_min_speed (KiB/s), defaulting to 64. At that speed
the measured 1.6 MB bursts come within the 30 second check. Configs
on the old default move to the new one, other values are converted.
The slow receiver test checks every 3 s instead of 500 ms, as with a
500 ms check it fails at any minimum.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 20:59:37 +00:00
Cooldude2606 e9a11013a4 Merge pull request #1236 from bbassie/chore/web-bundle-zip-zlib
Keep yauzl, browserify-zlib and assert out of the web interface
2026-11-20 20:55:12 +00:00
Cooldude2606andClaude Opus 5.5 59fa8403de Keep yauzl and yazl types out of lib's published declarations (#1235)
* Keep yauzl and yazl types out of lib's published declarations

ZipArchive exposed yauzl.ZipFile and yauzl.Entry in its public members
and zipOutputStream took a yazl.ZipFile, so @types/yauzl and @types/yazl
had to be runtime dependencies of @clusterio/lib. They pull in
@types/node and undici-types, about 2.7 MB in every server install.

Entries are now typed with a local ZipEntry interface, the yauzl
ZipFile is private, and zipOutputStream takes any object with an
outputStream and an error event. ZipWriter.addEntry reads the
compressed bytes through the new ZipArchive.openRawStream. Both type
packages move to devDependencies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Explain why ZipEntry is a local type

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 20:54:56 +00:00
Bastiaan 1c18dc3661 Merge pull request #1234 from bbassie/chore/inline-daily-rotate
Replace winston-daily-rotate-file with an in-repo transport
2026-11-20 20:54:42 +00:00
Bastiaan 81c7d24867 Merge pull request #1233 from bbassie/chore/drop-chalk
Replace chalk with util.styleText
2026-11-20 20:54:30 +00:00
bbassieandClaude Opus 5.5 c4a9545d28 Explain why ZipEntry is a local type
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 16:11:36 +00:00
bbassieandClaude Opus 5.5 6dc48b6732 Replace node:assert in link with a local assert
The assert polyfill is 28 KB in the web interface bundle and link only
uses it for plain truthiness checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 10:19:04 +00:00
bbassieandClaude Opus 5.5 4d7c1020b0 Use fflate for zlib in the web interface
ModPack and map exchange strings need synchronous inflate and deflate.
In the browser these came from the browserify-zlib polyfill, which
brings pako, assert and util along. The browser build now imports
fflate through a #zlib_sync subpath import, Node.js keeps node:zlib.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 10:18:37 +00:00
bbassieandClaude Opus 5.5 909f6ab08b Move ModInfo.fromModFile to ModStore.readModInfo
Reading a mod zip needs zip_ops, which pulled yauzl into the web
interface bundle through the data classes. ModStore is node only, so
the zip reading now lives there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 10:17:40 +00:00
bbassieandClaude Opus 5.5 aaafce4ace Keep yauzl and yazl types out of lib's published declarations
ZipArchive exposed yauzl.ZipFile and yauzl.Entry in its public members
and zipOutputStream took a yazl.ZipFile, so @types/yauzl and @types/yazl
had to be runtime dependencies of @clusterio/lib. They pull in
@types/node and undici-types, about 2.7 MB in every server install.

Entries are now typed with a local ZipEntry interface, the yauzl
ZipFile is private, and zipOutputStream takes any object with an
outputStream and an error event. ZipWriter.addEntry reads the
compressed bytes through the new ZipArchive.openRawStream. Both type
packages move to devDependencies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-19 20:39:58 +00:00
30 changed files with 485 additions and 180 deletions

No files matched your search

+7 -4
View File
@@ -241,17 +241,20 @@ Must be greater than 0.
Defaults to 15.
### controller.transfer_min_progress
### controller.transfer_min_speed
Amount in KiB a request to the controller has to move every 30 seconds, requests moving less are closed.
Speed in KiB/s at which a request to the controller is never closed for moving too slowly.
This closes uploads and downloads that have stalled or are sent too slowly to ever finish, while there is no limit on how long a transfer making progress can take.
It applies to saves, mods and exports uploaded to the controller, proxy streams through it, and the body of any other request.
Time spent waiting for a proxy stream to start is not counted, see controller.proxy_stream_timeout.
Data is passed on in chunks of up to 64 KiB and progress is counted per chunk, so a proxy stream moving less than about that in 30 seconds may also be closed.
Requests are checked every 60 seconds and must have moved this speed times 60 seconds, less 2 MiB, since the last check.
The 2 MiB covers the OS sending data in bursts, it can buffer a couple of MiB per connection and only accept more once a large part of that has gone out.
So at the default a request moving slower than about 30 KiB/s is closed, and one moving between that and 64 KiB/s may be.
Set below about 34 KiB/s, only requests that moved nothing in 60 seconds are closed.
Set to 0 for no limit.
Must be 0 or greater.
Defaults to 30.
Defaults to 64.
### controller.factorio_username
+4 -7
View File
@@ -21,7 +21,6 @@ import crypto from "crypto";
import yargs from "yargs";
import util from "util";
import winston from "winston";
import DailyRotateFile from "winston-daily-rotate-file";
// homebrew modules
@@ -355,19 +354,17 @@ async function initialize(): Promise<InitializeParameters> {
level: "verbose",
levels,
});
clusterLogger.add(new DailyRotateFile({
clusterLogger.add(new lib.DailyFileTransport({
format: winston.format.json(),
filename: "cluster-%DATE%.log",
utc: true,
dirname: path.join(args.logDirectory, "cluster"),
prefix: "cluster",
}));
// Log stream for the controller.
logger.add(new DailyRotateFile({
logger.add(new lib.DailyFileTransport({
format: winston.format.json(),
filename: "controller-%DATE%.log",
utc: true,
dirname: path.join(args.logDirectory, "controller"),
prefix: "controller",
}));
logger.add(new winston.transports.Stream({
stream: clusterLogger,
-1
View File
@@ -45,7 +45,6 @@
"jsonwebtoken": "catalog:",
"semver": "catalog:",
"winston": "catalog:",
"winston-daily-rotate-file": "catalog:",
"ws": "catalog:",
"yargs": "catalog:"
},
+35 -23
View File
@@ -419,7 +419,7 @@ export interface ProxyStream {
timeout: ReturnType<typeof setTimeout>;
closed: boolean;
/** Bytes the receiver has taken, set once the stream is flowing */
progress?: () => number;
bytesTaken?: () => number;
}
export async function createProxyStream(app: Application): Promise<ProxyStream> {
@@ -473,18 +473,23 @@ const answeredUploadDrainMs = 1000;
* so the connection is closed if the body hasn't ended shortly after the
* response. Closing it right away could make the client lose the response.
*
* The connection is also closed if the request moved less than
* controller.transfer_min_progress during a check interval, so that a
* The connection is also closed if the request moved too little during a
* check interval for controller.transfer_min_speed, so that a
* request which stalled or is trickled in does not stay open forever while
* a slow one can take the time it needs. For a request body this is what
* was received of it. For both ends of a proxy stream it is what the
* receiver has taken, as the sender is held back when the receiver is the
* slow one. Time spent waiting for a proxy stream to start is not counted,
* controller.proxy_stream_timeout limits that.
*
* The OS hands data on in bursts, so a transfer at the minimum speed can
* move up to a burst less than that in a check. The burst allowance is
* taken off what a check requires so such a transfer is never closed.
* @param checkIntervalMs - Interval to check the progress of requests on.
* @param burstAllowanceBytes - Bytes taken off what a check requires.
* @returns Express middleware.
*/
function guardRequest(checkIntervalMs: number) {
function guardRequest(checkIntervalMs: number, burstAllowanceBytes: number) {
return function(req: Request, res: Response, next: any) {
const socket = req.socket;
res.once("finish", () => {
@@ -501,33 +506,36 @@ function guardRequest(checkIntervalMs: number) {
});
});
const received = () => socket.bytesRead;
let measure: (() => number) | undefined = received;
let last = received();
const bytesReceived = () => socket.bytesRead;
let measuredBytes: (() => number) | undefined = bytesReceived;
let lastBytes = bytesReceived();
const check = setInterval(() => {
const stream: ProxyStream | undefined = res.locals.proxyStream;
let current: (() => number) | undefined;
let currentBytes: (() => number) | undefined;
if (stream && !stream.closed) {
current = stream.progress;
currentBytes = stream.bytesTaken;
} else if (!req.complete) {
current = received;
currentBytes = bytesReceived;
}
const minProgress = req.app.locals.controller.config.get("controller.transfer_min_progress");
if (current !== measure || !current || !minProgress) {
const minSpeedKiBps = req.app.locals.controller.config.get("controller.transfer_min_speed");
if (currentBytes !== measuredBytes || !currentBytes || !minSpeedKiBps) {
// What to measure changed, start over from here.
measure = current;
last = current ? current() : 0;
measuredBytes = currentBytes;
lastBytes = currentBytes ? currentBytes() : 0;
return;
}
const progress = current();
if (progress - last < minProgress * 1024) {
const bytes = currentBytes();
const intervalS = checkIntervalMs / 1000;
// At least 1 so a transfer that moved nothing is closed at any speed setting.
const requiredBytes = Math.max(minSpeedKiBps * 1024 * intervalS - burstAllowanceBytes, 1);
if (bytes - lastBytes < requiredBytes) {
logger.warn(
`Closing ${req.method} ${req.path}, it moved less than ${minProgress} KiB ` +
`in ${checkIntervalMs / 1000} seconds`
`Closing ${req.method} ${req.path}, it moved ${bytes - lastBytes} bytes in ${intervalS} seconds, ` +
`less than the ${requiredBytes} required for ${minSpeedKiBps} KiB/s`
);
socket.destroy();
}
last = progress;
lastBytes = bytes;
}, checkIntervalMs);
res.once("close", () => clearInterval(check));
next();
@@ -567,7 +575,7 @@ function startStream(res: Response, stream: ProxyStream) {
source.pipe(res);
stream.flowing = true;
// Counts completed writes, so data waiting in a buffer is not progress.
stream.progress = () => socket.bytesWritten - socket.writableLength;
stream.bytesTaken = () => socket.bytesWritten - socket.writableLength;
// Pipe does not pass on a source that was cut short, the receiver would wait forever.
const onSourceClose = () => {
if (!res.writableEnded) {
@@ -828,7 +836,7 @@ async function uploadMod(req: Request, res: Response) {
// Pipe does not pass on an upload that was cut short.
await Promise.all([finished(file.stream), finished(writeStream)]);
const modInfo = await lib.ModInfo.fromModFile(tempFilePath);
const modInfo = await lib.ModStore.readModInfo(tempFilePath);
checkModName(modInfo.filename);
await fs.rename(tempFilePath, path.join(modsDirectory, modInfo.filename));
req.app.locals.controller.modStore.addMod(modInfo);
@@ -856,8 +864,12 @@ async function uploadMod(req: Request, res: Response) {
}
export function addRouteHandlers(app: Application, progressCheckIntervalMs = 30_000) {
app.use(guardRequest(progressCheckIntervalMs));
export function addRouteHandlers(
app: Application,
progressCheckIntervalMs = 60_000,
burstAllowanceBytes = 2 * 1024 * 1024,
) {
app.use(guardRequest(progressCheckIntervalMs, burstAllowanceBytes));
app.get("/metrics",
validateUserToken,
(req:Request, res:Response, next:any) => getMetrics(req, res).catch(next)
+2 -5
View File
@@ -1,5 +1,5 @@
/* eslint-disable no-console */
import chalk from "chalk";
import { styleText } from "node:util";
export const levels = Object.freeze({
fatal: 0,
@@ -27,10 +27,7 @@ for (let [name, level] of Object.entries(levels)) {
if (level > maxLogLevel) {
return;
}
let style = chalk;
for (let part of colors[name].split(" ")) {
style = style[part];
}
const style = text => styleText(colors[name].split(" "), text);
if (["fatal", "error"].includes(name)) {
console.error(`${style(name)} ${message}`);
} else if (["warn"].includes(name)) {
-1
View File
@@ -36,7 +36,6 @@
"@inquirer/confirm": "^6.3.5",
"@inquirer/input": "^5.1.9",
"@inquirer/select": "^5.2.8",
"chalk": "catalog:",
"yargs": "catalog:"
}
}
+2 -4
View File
@@ -17,7 +17,6 @@ import path from "path";
import yargs from "yargs";
import packageConfig from "@clusterio/host/package.json" with { type: "json" };
import winston from "winston";
import "winston-daily-rotate-file";
// internal libraries
// Reduce startup time by lazy compiling schemas.
@@ -147,11 +146,10 @@ async function startHost() {
.parseSync()
;
logger.add(new winston.transports.DailyRotateFile({
logger.add(new lib.DailyFileTransport({
format: winston.format.json(),
filename: "host-%DATE%.log",
utc: true,
dirname: path.join(args.logDirectory, "host"),
prefix: "host",
}));
if (args.logLevel !== "none") {
logger.add(new ConsoleTransport({
-1
View File
@@ -49,7 +49,6 @@
"semver": "catalog:",
"set-blocking": "catalog:",
"winston": "catalog:",
"winston-daily-rotate-file": "catalog:",
"yargs": "catalog:",
"yazl": "catalog:"
},
+10 -3
View File
@@ -31,6 +31,13 @@
},
"import": "./src/link/WebSocket.browser.ts"
},
"#zlib_sync": {
"node": {
"types": "./src/zlib_sync.node.ts",
"default": "./dist/node/src/zlib_sync.node.js"
},
"import": "./src/zlib_sync.browser.ts"
},
"#is_deep_strict_equal": {
"node": {
"types": "./src/is_deep_strict_equal.node.ts",
@@ -51,11 +58,9 @@
"dependencies": {
"@sinclair/typebox": "catalog:",
"@types/yauzl": "catalog:",
"@types/yazl": "catalog:",
"ajv": "^8.20.0",
"chalk": "catalog:",
"fast-deep-equal": "^3.1.3",
"fflate": "catalog:",
"set-blocking": "catalog:",
"triple-beam": "^1.4.1",
"winston": "catalog:",
@@ -71,6 +76,8 @@
"@types/triple-beam": "^1.3.5",
"@types/ws": "catalog:",
"@types/yargs": "catalog:",
"@types/yauzl": "catalog:",
"@types/yazl": "catalog:",
"typescript": "catalog:"
}
}
+39 -1
View File
@@ -4,6 +4,8 @@ import events from "events";
import { Static } from "@sinclair/typebox";
import { logger } from "./logging.js";
import { downloadFile, atomicOverwriteFile } from "./file_ops.js";
import { hashFile } from "./hash.js";
import { ZipArchive, findRoot } from "./zip_ops.js";
import { ModPortalReleaseSchema, ModPortalDetailsSchema, ModNameVersionPair } from "./data/messages_mod.js";
import { ModInfo, ModPack, MajorMinorVersion, ModVersionEquality, SourceVersion } from "./data/index.js";
@@ -54,8 +56,44 @@ export default class ModStore extends events.EventEmitter<ModStoreEvents> {
mods() { return this.files.values(); }
/**
* Read the info.json of a mod zip file along with its size, mtime and hash
*
* @param modPath - Path to the mod zip file.
* @returns Mod info for the file.
*/
static async readModInfo(modPath: string) {
let modInfo: Static<typeof ModInfo.jsonSchema>;
let zip = await ZipArchive.fromFile(modPath);
try {
let content = await zip.readFile(path.posix.join(findRoot(zip), "info.json"));
if (content === null) {
throw new Error("Mod contains no info.json file");
}
modInfo = JSON.parse(content.toString("utf8"));
} finally {
zip.close();
}
let valid = ModInfo.validateInfo(modInfo);
if (!valid) {
throw new Error("Mod's info.json is not valid");
}
const stat = await fs.stat(modPath);
return ModInfo.fromJSON({
...modInfo,
size: stat.size,
sha1: await hashFile(modPath),
mtime_ms: stat.mtimeMs,
updated_at_ms: stat.mtimeMs,
is_deleted: false,
});
}
static async loadFile(modsDirectory: string, file: string) {
let modInfo = await ModInfo.fromModFile(path.join(modsDirectory, file));
let modInfo = await ModStore.readModInfo(path.join(modsDirectory, file));
if (modInfo.filename !== file) {
throw new Error(`filename does not match the expected name ${modInfo.filename}.`);
}
+13 -5
View File
@@ -31,7 +31,7 @@ export interface ControllerConfigFields {
"controller.max_subscription_filters": number;
"controller.system_metrics_interval": number;
"controller.proxy_stream_timeout": number;
"controller.transfer_min_progress": number;
"controller.transfer_min_speed": number;
"controller.factorio_username": string | null;
"controller.factorio_token": string | null;
"controller.share_factorio_credentials_with_hosts": boolean;
@@ -63,6 +63,14 @@ export class ControllerConfig extends classes.Config<ControllerConfigFields> {
config["controller.public_url"] = config["controller.external_address"];
delete config["controller.external_address"];
}
if (config.hasOwnProperty("controller.transfer_min_progress")) {
// Was KiB per 30 s, the old default is left out so the new one applies
const kib = config["controller.transfer_min_progress"];
if (kib !== 30) {
config["controller.transfer_min_speed"] = typeof kib === "number" ? kib / 30 : kib;
}
delete config["controller.transfer_min_progress"];
}
return config;
}
@@ -224,11 +232,11 @@ export class ControllerConfig extends classes.Config<ControllerConfigFields> {
initialValue: 15,
validator: validators.greaterThanZero,
},
"controller.transfer_min_progress": {
title: "Transfer Minimum Progress",
description: "KiB an upload or download has to move every 30 seconds to not be closed, 0 for no limit.",
"controller.transfer_min_speed": {
title: "Transfer Minimum Speed",
description: "KiB/s at which an upload or download is never closed for moving too slowly, 0 for no limit.",
type: "number",
initialValue: 30,
initialValue: 64,
validator: validators.greaterThanEqualZero,
},
"controller.session_timeout": {
-34
View File
@@ -1,10 +1,6 @@
import fs from "node:fs/promises";
import path from "path";
import { Type, Static } from "@sinclair/typebox";
import * as libHash from "../hash.js";
import * as libSchema from "../schema.js";
import { ZipArchive, findRoot } from "../zip_ops.js";
import { ModRecord } from "./ModPack.js";
import {
@@ -349,34 +345,4 @@ export default class ModInfo {
if (this.isDeleted) { json.is_deleted = this.isDeleted; }
return json;
}
static async fromModFile(modPath: string) {
let modInfo: Static<typeof ModInfo.jsonSchema>;
let zip = await ZipArchive.fromFile(modPath);
try {
let content = await zip.readFile(path.posix.join(findRoot(zip), "info.json"));
if (content === null) {
throw new Error("Mod contains no info.json file");
}
modInfo = JSON.parse(content.toString("utf8"));
} finally {
zip.close();
}
let valid = this.validateInfo(modInfo);
if (!valid) {
throw new Error("Mod's info.json is not valid");
}
const stat = await fs.stat(modPath);
return this.fromJSON({
...modInfo,
size: stat.size,
sha1: await libHash.hashFile(modPath),
mtime_ms: stat.mtimeMs,
updated_at_ms: stat.mtimeMs,
is_deleted: false,
});
}
}
+2 -2
View File
@@ -1,4 +1,4 @@
import zlib from "zlib";
import * as zlib from "#zlib_sync";
import { Type, Static } from "@sinclair/typebox";
import * as libSchema from "../schema.js";
@@ -306,7 +306,7 @@ export default class ModPack {
// eslint-disable-next-line node/no-sync
let buf = zlib.deflateSync(JSON.stringify(json));
return buf.toString("base64");
return Buffer.from(buf).toString("base64");
}
static fromModPackString(modPackString: string) {
+1 -1
View File
@@ -1,4 +1,4 @@
import zlib from "zlib";
import * as zlib from "#zlib_sync";
class MapReaderState {
pos = 0;
+6
View File
@@ -0,0 +1,6 @@
/** Throws if condition is falsy, node:assert is not available in the browser. */
export function assert(condition: unknown): asserts condition {
if (!condition) {
throw new Error("Assertion failed");
}
}
+1 -1
View File
@@ -1,5 +1,4 @@
// Connection adapters for Links
import { strict as assert } from "assert";
import events from "events";
import { WebSocket } from "#WebSocket";
@@ -7,6 +6,7 @@ import * as libData from "../data/index.js";
import * as libErrors from "../errors.js";
import { logger } from "../logging.js";
import ExponentialBackoff from "../ExponentialBackoff.js";
import { assert } from "./assert.js";
import type { Request, Event } from "./link.js";
/**
+1 -1
View File
@@ -5,7 +5,7 @@ import { logger } from "../logging.js";
import * as libSchema from "../schema.js";
import { dataClasses, errorClasses } from "./messages.js";
import { BaseConnector, WebSocketBaseConnector } from "./connectors.js";
import { strict as assert } from "assert";
import { assert } from "./assert.js";
import type { PluginNodeEnvInfo, PluginWebEnvInfo } from "../plugin.js";
import type {
AddressType, JSONDeserialisable, MessageRoutable, MessageRequest, MessageEvent, IUser,
+61 -12
View File
@@ -5,14 +5,14 @@
* @module lib/logging_utils
*/
import fs from "node:fs/promises";
import type { ReadStream } from "node:fs";
import { type ReadStream, type WriteStream, createWriteStream, mkdirSync } from "node:fs";
import os from "node:os";
import path from "path";
import stream from "stream";
import util from "util";
import util, { styleText } from "util";
import winston from "winston";
import Transport from "winston-transport";
import { LEVEL, MESSAGE } from "triple-beam";
import chalk from "chalk";
import * as libData from "./data/index.js";
import * as libErrors from "./errors.js";
@@ -108,30 +108,30 @@ export type ParsedFactorioOutput =
function formatServerOutput(parsed: ParsedFactorioOutput) {
let time = "";
if (parsed.format === "seconds") {
time = `${chalk.yellow(parsed.time.padStart(8))} `;
time = `${styleText("yellow", parsed.time.padStart(8))} `;
} else if (parsed.format === "date") {
time = `${chalk.yellow(parsed.time)} `;
time = `${styleText("yellow", parsed.time)} `;
}
let info = "";
if (parsed.type === "log") {
let level = parsed.level;
if (level === "Script") {
level = chalk.bold.greenBright(level);
level = styleText(["bold", "greenBright"], level);
} else if (level === "Verbose") {
level = chalk.bold.gray(level);
level = styleText(["bold", "gray"], level);
} else if (level === "Info") {
level = chalk.bold.blueBright(level);
level = styleText(["bold", "blueBright"], level);
} else if (parsed.level === "Warning") {
level = chalk.bold.yellowBright(level);
level = styleText(["bold", "yellowBright"], level);
} else if (parsed.level === "Error") {
level = chalk.bold.redBright(level);
level = styleText(["bold", "redBright"], level);
}
info = `${level} ${chalk.gray(parsed.file)}: `;
info = `${level} ${styleText("gray", parsed.file)}: `;
} else if (parsed.type === "action") {
info = `[${chalk.yellow(parsed.action)}] `;
info = `[${styleText("yellow", parsed.action)}] `;
}
return time + info + parsed.message;
@@ -226,6 +226,55 @@ export class LinkTransport extends Transport {
}
}
/**
* Writes log entries as lines to dirname/prefix-YYYY-MM-DD.log, opening
* a new file when the UTC date changes.
*/
export class DailyFileTransport extends Transport {
dirname: string;
prefix: string;
now: () => number;
date?: string;
stream?: WriteStream;
constructor(
options: Transport.TransportStreamOptions & {
dirname: string,
prefix: string,
now?: () => number,
}
) {
super(options);
this.dirname = options.dirname;
this.prefix = options.prefix;
this.now = options.now ?? Date.now;
// eslint-disable-next-line node/no-sync
mkdirSync(this.dirname, { recursive: true });
}
log(info: any, callback: () => void) {
const date = new Date(this.now()).toISOString().slice(0, 10);
if (!this.stream || date !== this.date) {
this.stream?.end();
this.date = date;
this.stream = createWriteStream(path.join(this.dirname, `${this.prefix}-${date}.log`), { flags: "a" });
this.stream.on("error", err => this.emit("error", err));
}
this.stream.write(info[MESSAGE] + os.EOL);
return callback();
}
close(callback?: () => void) {
const fileStream = this.stream;
this.stream = undefined;
if (fileStream) {
fileStream.end(callback);
} else {
callback?.();
}
}
}
const logFileGlob = /^[a-z]+-(\d{4}-\d{2}-\d{2})\.log$/;
const logIndexVersion = 2;
+42 -10
View File
@@ -7,10 +7,31 @@ import { PassThrough, type Readable } from "stream";
import util from "util";
import zlib from "zlib";
import yauzl from "yauzl";
import type yazl from "yazl";
const deflateRaw = util.promisify(zlib.deflateRaw);
/**
* Entry in a ZipArchive
*
* Local subset of yauzl.Entry so the published declarations do not import
* yauzl, which would make @types/yauzl a runtime dependency of lib.
*/
export interface ZipEntry {
fileName: string;
fileComment: string;
versionMadeBy: number;
versionNeededToExtract: number;
generalPurposeBitFlag: number;
compressionMethod: number;
lastModFileTime: number;
lastModFileDate: number;
crc32: number;
compressedSize: number;
uncompressedSize: number;
externalFileAttributes: number;
getLastModDate(options?: { timezone?: "local" | "UTC" | null, forceDosFormat?: boolean }): Date;
}
/**
* Zip archive opened for random access reading
@@ -21,10 +42,9 @@ const deflateRaw = util.promisify(zlib.deflateRaw);
*/
export class ZipArchive {
private constructor(
/** Underlying yauzl zip file. */
public zipfile: yauzl.ZipFile,
private zipfile: yauzl.ZipFile,
/** Entries by file name in the order they appear in the central directory. */
public entries: Map<string, yauzl.Entry>,
public entries: Map<string, ZipEntry>,
) { }
/**
@@ -77,8 +97,17 @@ export class ZipArchive {
*
* @param entry - Entry to read, must belong to this archive.
*/
openStream(entry: yauzl.Entry): Promise<Readable> {
return this.zipfile.openReadStreamPromise(entry);
openStream(entry: ZipEntry): Promise<Readable> {
return this.zipfile.openReadStreamPromise(entry as yauzl.Entry);
}
/**
* Open a stream of the content of an entry as stored in the archive
*
* @param entry - Entry to read, must belong to this archive.
*/
openRawStream(entry: ZipEntry): Promise<Readable> {
return this.zipfile.openReadStreamPromise(entry as yauzl.Entry, { decodeFileData: false });
}
/**
@@ -86,7 +115,7 @@ export class ZipArchive {
*
* @param entry - Entry to read, must belong to this archive.
*/
async readEntry(entry: yauzl.Entry) {
async readEntry(entry: ZipEntry) {
const chunks: Buffer[] = [];
for await (const chunk of await this.openStream(entry)) {
chunks.push(chunk);
@@ -153,7 +182,10 @@ export function findRoot(zip: ZipArchive) {
*
* @param zipFile - zip file being written.
*/
export function zipOutputStream(zipFile: yazl.ZipFile) {
export function zipOutputStream(zipFile: {
outputStream: NodeJS.ReadableStream,
on(event: "error", listener: (err: Error) => void): unknown,
}) {
const output = zipFile.outputStream as Readable;
zipFile.on("error", err => output.destroy(err));
return output;
@@ -224,7 +256,7 @@ export class ZipWriter {
* @param zip - Archive to copy from.
* @param entry - Entry to copy, must belong to zip.
*/
async addEntry(zip: ZipArchive, entry: yauzl.Entry) {
async addEntry(zip: ZipArchive, entry: ZipEntry) {
const writerEntry: WriterEntry = {
fileName: Buffer.from(entry.fileName, "utf8"),
fileComment: Buffer.from(entry.fileComment, "utf8"),
@@ -242,7 +274,7 @@ export class ZipWriter {
offset: this.offset,
};
await this.write(this.localFileHeader(writerEntry));
const stream = await zip.zipfile.openReadStreamPromise(entry, { decodeFileData: false });
const stream = await zip.openRawStream(entry);
for await (const chunk of stream) {
await this.write(chunk);
}
+15
View File
@@ -0,0 +1,15 @@
/* eslint-disable node/no-sync */
import { strToU8, unzlibSync, zlibSync } from "fflate";
export function inflateSync(data: Uint8Array): Uint8Array {
try {
return unzlibSync(data);
} catch (err: any) {
// Match the error code node:zlib gives for invalid data.
throw Object.assign(new Error(err.message), { code: "Z_DATA_ERROR" });
}
}
export function deflateSync(data: Uint8Array | string): Uint8Array {
return zlibSync(typeof data === "string" ? strToU8(data) : data);
}
+11
View File
@@ -0,0 +1,11 @@
import zlib from "zlib";
export function inflateSync(data: Uint8Array): Uint8Array {
// eslint-disable-next-line node/no-sync
return zlib.inflateSync(data);
}
export function deflateSync(data: Uint8Array | string): Uint8Array {
// eslint-disable-next-line node/no-sync
return zlib.deflateSync(data);
}
+1
View File
@@ -19,6 +19,7 @@
"./src/hash.ts",
"./src/ini.ts",
"./src/is_deep_strict_equal.browser.ts",
"./src/zlib_sync.browser.ts",
"./src/link/*.ts",
"./src/lua_tools.ts",
"./src/logging.ts",
+1 -2
View File
@@ -21,8 +21,8 @@ catalog:
'@types/yazl': ^3.3.1
'@vitejs/plugin-react': ^6.1.2
antd: ^5.29.3
chalk: ^6.0.1
express: ^5.3.0
fflate: ^0.8.3
jsonwebtoken: ^9.0.3
react: ^18.3.1
react-dom: ^18.3.1
@@ -34,7 +34,6 @@ catalog:
typescript: ^7.0.2
vite: ^8.3.4
winston: ^3.19.0
winston-daily-rotate-file: ^5.0.0
winston-transport: ^4.9.0
ws: ^8.22.0
yargs: ^18.2.0
+89 -28
View File
@@ -22,8 +22,8 @@ describe("controller/src/routes", function() {
let port;
beforeEach(async function() {
controller = new mock.MockController();
// Check the progress of requests every 500 ms instead of every 30 seconds.
routes.addRouteHandlers(controller.app, 500);
// Check the progress of requests every 500 ms without a burst allowance.
routes.addRouteHandlers(controller.app, 500, 0);
server = http.createServer(routes.httpServerOptions, controller.app);
server.listen(0, "localhost");
await events.once(server, "listening");
@@ -179,7 +179,7 @@ describe("controller/src/routes", function() {
return status;
}
it("should close a transfer that stops making progress", async function() {
controller.mockConfigEntries.set("controller.transfer_min_progress", 1);
controller.mockConfigEntries.set("controller.transfer_min_speed", 2);
const stream = await routes.createProxyStream(controller.app);
const request = http.request(
`${endpoint}/${stream.id}`, { method: "PUT", headers: { "Content-Length": 1024 * 1024 } }
@@ -196,7 +196,7 @@ describe("controller/src/routes", function() {
});
it("should close a transfer that is trickled in", async function() {
this.timeout(10000);
controller.mockConfigEntries.set("controller.transfer_min_progress", 4);
controller.mockConfigEntries.set("controller.transfer_min_speed", 8);
const stream = await routes.createProxyStream(controller.app);
const start = Date.now();
const upload = putPaced(stream.id, 200, 50);
@@ -208,7 +208,7 @@ describe("controller/src/routes", function() {
});
it("should not close a slow transfer that makes progress", async function() {
this.timeout(10000);
controller.mockConfigEntries.set("controller.transfer_min_progress", 1);
controller.mockConfigEntries.set("controller.transfer_min_speed", 2);
const stream = await routes.createProxyStream(controller.app);
const upload = putPaced(stream.id, 1024, 20);
const download = fetch(`${endpoint}/${stream.id}`).then(response => response.arrayBuffer());
@@ -216,30 +216,41 @@ describe("controller/src/routes", function() {
assert.equal((await download).byteLength, 20 * 1024);
});
it("should not close a transfer held back by a slow receiver", async function() {
this.timeout(20000);
controller.mockConfigEntries.set("controller.transfer_min_progress", 16);
const size = 4 * 1024 * 1024;
const stream = await routes.createProxyStream(controller.app);
const upload = fetch(`${endpoint}/${stream.id}`, { method: "PUT", body: Buffer.alloc(size) });
const received = await new Promise((resolve, reject) => {
http.get(`${endpoint}/${stream.id}`, response => {
let bytes = 0;
response.on("data", chunk => {
bytes += chunk.length;
response.pause();
setTimeout(() => response.resume(), 50);
});
response.on("end", () => resolve(bytes));
response.on("error", reject);
}).on("error", reject);
});
assert.equal(received, size);
assert.equal((await upload).status, 200);
this.timeout(40000);
// The receiver takes about 1.2 MB/s, which the OS hands on in bursts of over a MB.
const slowController = new mock.MockController();
slowController.mockConfigEntries.set("controller.transfer_min_speed", 1024);
routes.addRouteHandlers(slowController.app, 3000);
const slowServer = http.createServer(routes.httpServerOptions, slowController.app);
slowServer.listen(0, "localhost");
await events.once(slowServer, "listening");
const slowEndpoint = `http://localhost:${slowServer.address().port}/api/stream`;
try {
const size = 16 * 1024 * 1024;
const stream = await routes.createProxyStream(slowController.app);
const upload = fetch(`${slowEndpoint}/${stream.id}`, { method: "PUT", body: Buffer.alloc(size) });
const received = await new Promise((resolve, reject) => {
http.get(`${slowEndpoint}/${stream.id}`, response => {
let bytes = 0;
response.on("data", chunk => {
bytes += chunk.length;
response.pause();
setTimeout(() => response.resume(), 50);
});
response.on("end", () => resolve(bytes));
response.on("error", reject);
}).on("error", reject);
});
assert.equal(received, size);
assert.equal((await upload).status, 200);
} finally {
slowServer.close();
}
});
it("should not count the time waiting for the stream to start", async function() {
this.timeout(10000);
controller.mockConfigEntries.set("controller.proxy_stream_timeout", 3);
controller.mockConfigEntries.set("controller.transfer_min_progress", 1);
controller.mockConfigEntries.set("controller.transfer_min_speed", 2);
const stream = await routes.createProxyStream(controller.app);
const download = fetch(`${endpoint}/${stream.id}`);
await lib.wait(1500);
@@ -1257,7 +1268,7 @@ describe("controller/src/routes", function() {
});
it("should close an upload that is trickled in", async function() {
this.timeout(10000);
controller.mockConfigEntries.set("controller.transfer_min_progress", 4);
controller.mockConfigEntries.set("controller.transfer_min_speed", 8);
const modsDirectory = await fs.mkdtemp(path.join(os.tmpdir(), "clusterio-mods-"));
try {
controller.mockConfigEntries.set("controller.mods_directory", modsDirectory);
@@ -1277,15 +1288,65 @@ describe("controller/src/routes", function() {
});
it("should close a request to another route that is trickled in", async function() {
this.timeout(10000);
controller.mockConfigEntries.set("controller.transfer_min_progress", 4);
controller.mockConfigEntries.set("controller.transfer_min_speed", 8);
controller.app.post("/test", express.raw({ type: "*/*", limit: "2mb" }), (req, res) => res.sendStatus(200));
const result = await trickle(`http://localhost:${port}/test`, { "Content-Type": "text/plain" });
assert.equal(result.status, undefined);
assert(result.duration < 2000, `closed after ${result.duration} ms`);
});
describe("with a burst allowance", function() {
let burstServer, burstUrl;
beforeEach(async function() {
// Each check requires 256 KiB/s * 0.5 s - 64 KiB = 64 KiB.
const burstController = new mock.MockController();
burstController.mockConfigEntries.set("controller.transfer_min_speed", 256);
routes.addRouteHandlers(burstController.app, 500, 64 * 1024);
burstController.app.post("/test", express.raw({ type: "*/*", limit: "2mb" }), (req, res) => {
res.sendStatus(200);
});
burstServer = http.createServer(routes.httpServerOptions, burstController.app);
burstServer.listen(0, "localhost");
await events.once(burstServer, "listening");
burstUrl = `http://localhost:${burstServer.address().port}/test`;
});
afterEach(function() {
burstServer.close();
});
async function sendBursts(burstBytes) {
const bursts = 12;
const request = http.request(burstUrl, {
method: "POST", headers: { "Content-Type": "text/plain", "Content-Length": burstBytes * bursts },
});
let status;
request.on("response", response => {
status = response.statusCode;
response.resume();
});
request.on("error", () => {});
const closed = new Promise(resolve => request.on("close", resolve));
for (let i = 0; i < bursts && !request.closed; i++) {
request.write(Buffer.alloc(burstBytes));
await lib.wait(300);
}
if (!request.closed) {
request.end();
}
await closed;
return status;
}
it("should not close a request at the minimum speed arriving in bursts", async function() {
this.timeout(10000);
// 78 KiB every 300 ms is 260 KiB/s, a check can see only one burst.
assert.equal(await sendBursts(78 * 1024), 200);
});
it("should close a request below the minimum speed arriving in bursts", async function() {
this.timeout(10000);
assert.equal(await sendBursts(39 * 1024), undefined);
});
});
it("should not close a trickled in request when the limit is 0", async function() {
this.timeout(10000);
controller.mockConfigEntries.set("controller.transfer_min_progress", 0);
controller.mockConfigEntries.set("controller.transfer_min_speed", 0);
controller.app.post("/test", express.raw({ type: "*/*", limit: "2mb" }), (req, res) => res.sendStatus(200));
const result = await trickle(`http://localhost:${port}/test`, { "Content-Type": "text/plain" });
assert(result.duration >= 5000, `closed after ${result.duration} ms`);
+34 -1
View File
@@ -6,7 +6,8 @@ import stream from "node:stream";
// Capture native fetch before any potential mocks are applied
const nativeFetch = global.fetch;
import { ModStore, ModInfo, ModVersionEquality, zipOutputStream } from "@clusterio/lib";
import { ModStore, ModInfo, ModVersionEquality, zipOutputStream, hashFile } from "@clusterio/lib";
import * as libBuildMod from "@clusterio/lib/build_mod.js";
import { externalTest, createZip, createZipBuffer } from "../common.js";
import "../setup_logging.js";
@@ -706,6 +707,38 @@ describe("lib/ModStore", function () {
});
// === Instance Methods ===
describe("readModInfo (static)", function () {
it("should load from a mod zip file", async function () {
await libBuildMod.build({
build: true,
pack: true,
sourceDir: path.join("test", "file", "empty_mod"),
outputDir: path.join("temp", "test"),
});
const filePath = path.join("temp", "test", "empty_mod_1.0.0.zip");
const hash = await hashFile(filePath);
const stat = await fs.stat(filePath);
const mod = await ModStore.readModInfo(filePath);
assert.deepEqual(
mod,
ModInfo.fromJSON({
name: "empty_mod",
version: "1.0.0",
title: "An Empty Mod",
author: "Me",
description: "An empty mod for testing",
factorio_version: "1.1",
dependencies: [],
filename: "empty_mod_1.0.0.zip",
size: stat.size,
mtime_ms: stat.mtimeMs,
updated_at_ms: stat.mtimeMs,
sha1: hash,
})
);
});
});
describe("Instance Methods", function () {
let modStore;
let testMod1Name = "test-mod-1";
+10
View File
@@ -15,6 +15,16 @@ describe("lib/config/definitions/migrations", function() {
assert.equal(config.fields["controller.external_address"], undefined,
"external_address was not removed");
});
it("should migrate 'transfer_min_progress' to 'transfer_min_speed'", function() { // 2.1.2
const migrate = value => lib.ControllerConfig.fromJSON({
"controller.transfer_min_progress": value,
}, "controller");
assert.equal(migrate(30).get("controller.transfer_min_speed"), 64, "old default was kept");
assert.equal(migrate(0).get("controller.transfer_min_speed"), 0, "no limit was not kept");
assert.equal(migrate(3000).get("controller.transfer_min_speed"), 100, "value was not converted");
assert.equal(migrate(30).fields["controller.transfer_min_progress"], undefined,
"transfer_min_progress was not removed");
});
});
describe("Host Config", function() {
-32
View File
@@ -1,9 +1,6 @@
import assert from "node:assert/strict";
import fs from "node:fs/promises";
import path from "node:path";
import * as lib from "@clusterio/lib";
import * as libBuildMod from "@clusterio/lib/build_mod.js";
import { ModInfo, ModDependency } from "@clusterio/lib";
import { testMatrix, testRoundTripJsonSerialisable } from "../../common.js";
@@ -413,35 +410,6 @@ describe("lib/data/ModInfo", function() {
["UltraMod", "SuperLib >= 1.00", "! bad-mod"].map(dep => new ModDependency(dep))
);
});
it("should load from a mod zip file", async function() {
await libBuildMod.build({
build: true,
pack: true,
sourceDir: path.join("test", "file", "empty_mod"),
outputDir: path.join("temp", "test"),
});
const filePath = path.join("temp", "test", "empty_mod_1.0.0.zip");
const hash = await lib.hashFile(filePath);
const stat = await fs.stat(filePath);
const mod = await ModInfo.fromModFile(filePath);
assert.deepEqual(
mod,
ModInfo.fromJSON({
name: "empty_mod",
version: "1.0.0",
title: "An Empty Mod",
author: "Me",
description: "An empty mod for testing",
factorio_version: "1.1",
dependencies: [],
filename: "empty_mod_1.0.0.zip",
size: stat.size,
mtime_ms: stat.mtimeMs,
updated_at_ms: stat.mtimeMs,
sha1: hash,
})
);
});
describe("checkDependencySatisfaction()", function() {
it("should return undefined when satisfied", function() {
const mod = ModInfo.fromJSON({ dependencies: ["! a", "b = 1.0.0", "? c = 2.0.0"] });
+72
View File
@@ -1,5 +1,6 @@
import assert from "node:assert/strict";
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { testLines } from "./factorio/lines.js";
@@ -66,6 +67,77 @@ describe("lib/logging_utils.js", function() {
});
});
describe("class DailyFileTransport", function() {
let tempDir = path.join("temp", "test", "daily_file");
beforeEach(async function() {
await fs.rm(tempDir, { recursive: true, force: true });
});
function create(now, dirname = tempDir) {
return new lib.DailyFileTransport({ dirname, prefix: "test", now });
}
function write(transport, message) {
transport.log({ [Symbol.for("message")]: JSON.stringify({ message }) }, () => {});
}
async function close(transport) {
await new Promise(resolve => transport.close(resolve));
}
async function read(file) {
let content = await fs.readFile(path.join(tempDir, file), "utf8");
assert(content.endsWith(os.EOL));
return content.split(os.EOL).slice(0, -1).map(line => JSON.parse(line).message);
}
it("should create the directory and write lines to the file for the UTC date", async function() {
let transport = create(() => Date.parse("2026-11-19T12:00:00Z"), path.join(tempDir, "nested"));
write(transport, "first");
write(transport, "second");
await close(transport);
assert.deepEqual(await read(path.join("nested", "test-2026-11-19.log")), ["first", "second"]);
});
it("should append to an existing file", async function() {
await fs.mkdir(tempDir, { recursive: true });
await fs.writeFile(path.join(tempDir, "test-2026-11-19.log"), `{"message":"old"}${os.EOL}`);
let transport = create(() => Date.parse("2026-11-19T12:00:00Z"));
write(transport, "new");
await close(transport);
assert.deepEqual(await read("test-2026-11-19.log"), ["old", "new"]);
});
it("should switch files when the UTC date changes", async function() {
let time = Date.parse("2026-11-19T23:59:59.999Z");
let transport = create(() => time);
write(transport, "before");
time += 1;
write(transport, "after");
await close(transport);
assert.deepEqual((await fs.readdir(tempDir)).sort(), ["test-2026-11-19.log", "test-2026-11-20.log"]);
assert.deepEqual(await read("test-2026-11-19.log"), ["before"]);
assert.deepEqual(await read("test-2026-11-20.log"), ["after"]);
});
it("should use the UTC date rather than the local one", async function() {
let transport = create(() => Date.parse("2026-11-20T00:30:00+02:00"));
write(transport, "message");
await close(transport);
assert.deepEqual(await fs.readdir(tempDir), ["test-2026-11-19.log"]);
});
it("should emit write errors", async function() {
await fs.mkdir(path.join(tempDir, "test-2026-11-19.log"), { recursive: true });
let transport = create(() => Date.parse("2026-11-19T12:00:00Z"));
let error = new Promise(resolve => transport.once("error", resolve));
write(transport, "message");
assert.equal((await error).code, "EISDIR");
});
it("should close when nothing was written", async function() {
await close(create(Date.now));
assert.deepEqual(await fs.readdir(tempDir), []);
});
});
describe("deleteOldLogs", function() {
let tempDir = path.join("temp", "test", "log_retention");
let files = [
+25
View File
@@ -0,0 +1,25 @@
/* eslint-disable node/no-sync */
import assert from "node:assert/strict";
import zlib from "node:zlib";
import * as browserZlib from "../../packages/lib/src/zlib_sync.browser.ts";
describe("lib/zlib_sync.browser", function() {
const text = JSON.stringify({ name: "pack", mods: [{ name: "base", version: "2.0.0" }] });
it("should inflate data deflated by node:zlib", function() {
const inflated = browserZlib.inflateSync(zlib.deflateSync(text));
assert.equal(Buffer.from(inflated).toString(), text);
});
it("should deflate data node:zlib can inflate", function() {
assert.equal(zlib.inflateSync(browserZlib.deflateSync(text)).toString(), text);
const bytes = Buffer.from(text);
assert.equal(zlib.inflateSync(browserZlib.deflateSync(bytes)).toString(), text);
});
it("should throw with a Z_ code on invalid data", function() {
assert.throws(
() => browserZlib.inflateSync(Buffer.from("not zlib data")),
err => err.code === "Z_DATA_ERROR"
);
});
});
+1 -1
View File
@@ -175,7 +175,7 @@ export class MockController {
["controller.public_url", "test"],
["controller.auth_secret", "TestSecretDoNotUse"],
["controller.proxy_stream_timeout", 1],
["controller.transfer_min_progress", 30],
["controller.transfer_min_speed", 64],
["controller.metrics_timeout", 1],
["controller.host_pairing_timeout", 3600],
["controller.host_pairing_max_pending", 50],