Replace jszip with yauzl and yazl (#1011)

* Replace jszip with yauzl and yazl

jszip loads a whole archive into memory to read it and buffers the
output when writing.  Saves and mods can be several hundred megabytes,
so read zips with yauzl, which only reads the central directory and
streams entries on demand, and write them with yazl, which streams the
output.

lib gains a ZipArchive wrapper around yauzl with the lookup helpers the
callers need, findRoot() now takes it instead of a JSZip.  Saves are
patched by streaming every kept entry into a new zip, mod info.json is
read straight from the file, exports are streamed to the controller and
Factorio zip downloads go through a temp file instead of a buffer.

Patching now inflates and deflates each entry again where jszip reused
the compressed data, on a 336 MB synthetic save that is 7 s instead of
0.7 s, with peak memory going from 416 MB to 151 MB.

Closes #1002

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Copy save entries without recompressing them

yazl inflates and deflates every entry it is given, so patching a save
took ten times longer than with jszip, which reused the compressed
data.  Add a ZipWriter to lib that streams entries out as they are
added and copies entries from a ZipArchive byte for byte, with ZIP64
structures when sizes or offsets need them, and use it in patch().

On a 336 MB synthetic save patching now takes 0.4 s at 143 MB peak RSS
against 0.75 s at 417 MB with jszip.  Factorio loads the result.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* Handle save write errors before the patch loop awaits them

A write error on the temporary save, such as a full disk, rejected the
pipeline promise while addEntry() or addBuffer() was still pending. Nothing
was listening yet, so the host's unhandledRejection handler exited the
process instead of failing the instance start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep the export zip cache local to each export

The cache was module level, so when two instances on one host exported at
once, the first to finish closed archives the other was still reading.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Start writing zips before adding files from disk

yazl reads files passed to addFile() as soon as they are added. An
unreadable file emitted an error while the directory traversal was still
running, before the output pipeline existed, and the process exited on the
uncaught error. build_mod and zipDirectory now start the pipeline first and
reject instead. build_mod also removes the partial zip on failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
BastiaanandClaude Opus 5.5 authored and GitHub committed 2026-10-17 21:44:19 +03:00
1 parent e810020db5
commit c28e617b22
23 files changed
+976 -298

No files matched your search

+39 -21
View File
@@ -1,8 +1,9 @@
// Implementation of the create-scenario command
import fs from "node:fs/promises";
import os from "node:os";
import JSZip from "jszip";
import path from "path";
import { pipeline } from "node:stream/promises";
import yazl from "yazl";
import * as lib from "@clusterio/lib";
import { logger } from "@clusterio/lib";
@@ -61,33 +62,50 @@ export async function createScenario(
async function zipDirectory(dirPath: string, zipPath: string) {
const root = path.basename(path.resolve(dirPath));
const zip = new JSZip();
let dirs: [string, string][] = [[dirPath, root]];
while (dirs.length) {
let [dir, relativeDir] = dirs.pop()!;
for (let entry of await fs.readdir(dir, { withFileTypes: true })) {
let fsPath = path.join(dir, entry.name);
let relativePath = path.posix.join(relativeDir, entry.name);
if (entry.isFile()) {
zip.file(relativePath, await fs.readFile(fsPath));
} else if (entry.isDirectory()) {
dirs.push([fsPath, relativePath]);
const zip = new yazl.ZipFile();
const output = lib.zipOutputStream(zip);
const fd = await fs.open(zipPath, "w");
// addFile() can fail before the traversal is done, so start writing first.
const writing = pipeline(output, fd.createWriteStream());
writing.catch(() => {});
try {
let dirs: [string, string][] = [[dirPath, root]];
while (dirs.length) {
let [dir, relativeDir] = dirs.pop()!;
for (let entry of await fs.readdir(dir, { withFileTypes: true })) {
let fsPath = path.join(dir, entry.name);
let relativePath = path.posix.join(relativeDir, entry.name);
if (entry.isFile()) {
zip.addFile(fsPath, relativePath);
} else if (entry.isDirectory()) {
dirs.push([fsPath, relativePath]);
}
}
}
zip.end();
} catch (err) {
output.destroy();
await writing.catch(() => {});
throw err;
}
await fs.writeFile(zipPath, await zip.generateAsync({ type: "nodebuffer" }));
await writing;
}
async function extractZip(zipPath: string, dirPath: string) {
const zip = await JSZip.loadAsync(await fs.readFile(zipPath));
const root = lib.findRoot(zip);
for (let file of Object.values(zip.files)) {
if (file.dir || !file.name.startsWith(`${root}/`)) {
continue;
const zip = await lib.ZipArchive.fromFile(zipPath);
try {
const root = lib.findRoot(zip);
for (let entry of zip.entries.values()) {
if (entry.fileName.endsWith("/") || !entry.fileName.startsWith(`${root}/`)) {
continue;
}
const fsPath = path.join(dirPath, entry.fileName.slice(root.length + 1));
await fs.mkdir(path.dirname(fsPath), { recursive: true });
const fd = await fs.open(fsPath, "w");
await pipeline(await zip.openStream(entry), fd.createWriteStream());
}
const fsPath = path.join(dirPath, file.name.slice(root.length + 1));
await fs.mkdir(path.dirname(fsPath), { recursive: true });
await fs.writeFile(fsPath, await file.async("nodebuffer"));
} finally {
zip.close();
}
}