* Replace jszip with yauzl and yazl
jszip loads a whole archive into memory to read it and buffers the
output when writing. Saves and mods can be several hundred megabytes,
so read zips with yauzl, which only reads the central directory and
streams entries on demand, and write them with yazl, which streams the
output.
lib gains a ZipArchive wrapper around yauzl with the lookup helpers the
callers need, findRoot() now takes it instead of a JSZip. Saves are
patched by streaming every kept entry into a new zip, mod info.json is
read straight from the file, exports are streamed to the controller and
Factorio zip downloads go through a temp file instead of a buffer.
Patching now inflates and deflates each entry again where jszip reused
the compressed data, on a 336 MB synthetic save that is 7 s instead of
0.7 s, with peak memory going from 416 MB to 151 MB.
Closes#1002
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Copy save entries without recompressing them
yazl inflates and deflates every entry it is given, so patching a save
took ten times longer than with jszip, which reused the compressed
data. Add a ZipWriter to lib that streams entries out as they are
added and copies entries from a ZipArchive byte for byte, with ZIP64
structures when sizes or offsets need them, and use it in patch().
On a 336 MB synthetic save patching now takes 0.4 s at 143 MB peak RSS
against 0.75 s at 417 MB with jszip. Factorio loads the result.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Handle save write errors before the patch loop awaits them
A write error on the temporary save, such as a full disk, rejected the
pipeline promise while addEntry() or addBuffer() was still pending. Nothing
was listening yet, so the host's unhandledRejection handler exited the
process instead of failing the instance start.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Keep the export zip cache local to each export
The cache was module level, so when two instances on one host exported at
once, the first to finish closed archives the other was still reading.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Start writing zips before adding files from disk
yazl reads files passed to addFile() as soon as they are added. An
unreadable file emitted an error while the directory traversal was still
running, before the output pipeline existed, and the process exited on the
uncaught error. build_mod and zipDirectory now start the pipeline first and
reject instead. build_mod also removes the partial zip on failure.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Since the move to ES modules and pnpm 12, packages installed from npm
fail to start:
- The package imports of @clusterio/lib point Node at TypeScript
sources, which Node refuses to strip types from inside node_modules.
- The controller, host and ctl import their package.json from next to
the compiled entrypoint, a copy pnpm leaves out of the package.
- ctl, host and web_ui have no .npmignore, so pnpm packs them using the
repository .gitignore and leaves out nearly all of dist/.
Resolve the package imports to the compiled files at runtime while
keeping the sources for type checking, import package.json by package
name so it resolves to the real file, and add .npmignore files so pnpm
packs the same files as npm.
yargs, @types/yargs, semver and @types/semver are used by several packages at the same version. Also point the remaining direct references to @swc/core and react-router-dom at their existing catalog entries.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Raise every caret range to the current latest within its major, in the
root package.json, the pnpm-workspace.yaml catalog, the per-package
package.json files and the create plugin template.
Majors taken: eslint 10, @eslint/js 10, globals 17, nyc 18,
@types/express 5, @types/node 24, finalhandler 2, pidusage 4,
style-loader 4, css-loader 7, yargs 18 (all packages), chalk 6.
Majors skipped: mocha 12 (integration suite fails under it, 59 tests,
starting with a 30 s timeout in clusteriocontroller run; kept at
^11.8.0), inquirer (10+ is a rewrite on @inquirer/prompts where
ui.Prompt.prototype.close no longer exists, so the win32 readline
workaround in create.js would have to go and cannot be verified here),
react/antd/react-router-dom/@types/react, typescript 7,
@sinclair/typebox, webpack-merge (exact pin), webpack-dev-middleware,
webpack-manifest-plugin, winston-daily-rotate-file.
Code adaptations: the two /api/stream/:id handlers in routes.ts are
typed Request<{ id: string }> because @types/express 5 types
req.params values as string | string[]; eslint.config.mjs adds
newIsCapExceptions: ["constructor"] because eslint 10.10 new-cap now
reports `new this.constructor()`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Migrate the usage of CommonJS specific code and imports to ESM. Roughly
summarised this:
- Changes all require() calls to import statement or expressions.
- Changes usage of module.exports to export statements.
- Replaces require.resolve with import.meta.resolve.
- Replaces conditional imports with conditional subpath imports.
- Replaces module === require.main with import.meta.main.
- Adds the full filename with extention to all relative imports.
- Changes all usage of __dirname and __filename to their import.meta
equivalents.
- Sets "type": "module" in package.json files.
- Removes usage of "use strict".
- Adds .ts an an alias of .js for the webpack build.
Due to differences in how require and import resolves files the plugin
entrypoints will have to include the filename and extension after this
migration in order to work.
These packages are obsoleted by the promises API to the built-in Node fs
module. The fs-extra package is also causing problems with broken APIs
that don't behave the same as Node's fs module, and throwing errors
without code properties on them.
Use a separate folder for the TypeScript build for Node.js in the dist
folder. This makes each type of build consistently have its own output
folder instead of being interleaved with the Node.js build.
Clean up the configuration for compiling TypeScript for the Web
interface to us its own tsconfig.web.json config files and reference the
TypeScript source files directly in the configuration and imports
instead of files compiled to dist.
To properly support recompliation with TypeScript's projects feature
each package's tsconfig.web.json file is set to output the build to
dist/web_build which is then ignored in .npmignore. Webpack is still
used to create the proper build in dist/web, so this is mainly to make
TypeScipt's compiler happy but it also allows the root tsconfig.json
file to include the web interface code, meaning running the build script
now also checks the typing of the web interface.
Final part of #491.
Be consistent with the usage of tabs as indents in all JSON files in the
project. This makes it easier to configure editors to emit the right
indents when manually editing these files.
Only the controller has access to the secret needed to sign user tokens
so it makes sense for the user signing code to live in the controller.
This saves about 20ms on the time ctl takes to start up due to lib no
longer needing to import jsonwebtoken.
- react: Update to new root rendering method
- react-router: Replace useHistory with useNavigate
- react-router: Replace <Switch> with <Routes>
- reimplement sticky notice from scratch in ModPackViewPage
- remove isomorphic-ws as it's latest version doesn't work and only
saves one line of code.
- remove import of antd.dark.css
- fix misc css breakages.
Update non-web related dependencies to their latest version were
possible. It appears yargs has changed its parsing of command line
options to no longer remove a layer of quotes.
Bastiaan