* Replace jszip with yauzl and yazl
jszip loads a whole archive into memory to read it and buffers the
output when writing. Saves and mods can be several hundred megabytes,
so read zips with yauzl, which only reads the central directory and
streams entries on demand, and write them with yazl, which streams the
output.
lib gains a ZipArchive wrapper around yauzl with the lookup helpers the
callers need, findRoot() now takes it instead of a JSZip. Saves are
patched by streaming every kept entry into a new zip, mod info.json is
read straight from the file, exports are streamed to the controller and
Factorio zip downloads go through a temp file instead of a buffer.
Patching now inflates and deflates each entry again where jszip reused
the compressed data, on a 336 MB synthetic save that is 7 s instead of
0.7 s, with peak memory going from 416 MB to 151 MB.
Closes#1002
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Copy save entries without recompressing them
yazl inflates and deflates every entry it is given, so patching a save
took ten times longer than with jszip, which reused the compressed
data. Add a ZipWriter to lib that streams entries out as they are
added and copies entries from a ZipArchive byte for byte, with ZIP64
structures when sizes or offsets need them, and use it in patch().
On a 336 MB synthetic save patching now takes 0.4 s at 143 MB peak RSS
against 0.75 s at 417 MB with jszip. Factorio loads the result.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Handle save write errors before the patch loop awaits them
A write error on the temporary save, such as a full disk, rejected the
pipeline promise while addEntry() or addBuffer() was still pending. Nothing
was listening yet, so the host's unhandledRejection handler exited the
process instead of failing the instance start.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Keep the export zip cache local to each export
The cache was module level, so when two instances on one host exported at
once, the first to finish closed archives the other was still reading.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Start writing zips before adding files from disk
yazl reads files passed to addFile() as soon as they are added. An
unreadable file emitted an error while the directory traversal was still
running, before the output pipeline existed, and the process exited on the
uncaught error. build_mod and zipDirectory now start the pipeline first and
reject instead. build_mod also removes the partial zip on failure.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
These are used outside of the integration tests, but including the
integration index file causes the very slow setup of the contorller
and host. Avoid the integration setup costs when running commands
such as `pnpm mocha test/lib` by moving these utility functions to
the common test functions.
Without a transport Winston will print annoying "Attempt to write logs
with no transports" warnings all over the place. Setup a transport in
tests that cause logging to suppress this.
Migrate the usage of CommonJS specific code and imports to ESM. Roughly
summarised this:
- Changes all require() calls to import statement or expressions.
- Changes usage of module.exports to export statements.
- Replaces require.resolve with import.meta.resolve.
- Replaces conditional imports with conditional subpath imports.
- Replaces module === require.main with import.meta.main.
- Adds the full filename with extention to all relative imports.
- Changes all usage of __dirname and __filename to their import.meta
equivalents.
- Sets "type": "module" in package.json files.
- Removes usage of "use strict".
- Adds .ts an an alias of .js for the webpack build.
Due to differences in how require and import resolves files the plugin
entrypoints will have to include the filename and extension after this
migration in order to work.
Adds an externalTest(this) helper alongside slowTest that skips the test
when NO_EXTERNAL_TEST is set and raises the timeout to 60s, plus a
no-external-test script. Marks the tests hitting factorio.com, the mod
portal and GitHub with it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The mod portal "ApiVersion" enum was validated strictly when loading a
mod's info.json, which rejected versions the game itself accepts. Versions
are now normalised the same lenient way the game reads them: leading
whitespace is allowed, the major and minor are required, the patch is
optional, any trailing content is ignored, and numbers are read as integers.
This applies to both a mod's version and its factorio_version, validated by
a regex schema rather than the fixed enum. The enum is no longer used to
validate versions at all - the mod portal is the source of truth, so its
error is now surfaced when a version is unsupported. The known-version list
kept for the web interface fallback gains 2.1.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
These packages are obsoleted by the promises API to the built-in Node fs
module. The fs-extra package is also causing problems with broken APIs
that don't behave the same as Node's fs module, and throwing errors
without code properties on them.
A lot of code is repeated in various different ways in order to handle
streaming content to files without overwriting them, whether it's from
web resources or local streams.
Refactor the handling of these streams to all use the same utility
function for creating a write stream, and for the places downloading
urls use the same downloadFile function.
* Add ModDependency and version types
* Add warnings for mod dependencies
* Add warning for wrong factorio version
* Add fix issues button
* Add automatic dependency resolution and download
* Add abilty to check for updates
* Add extra test coverage
* One extra test
* Fix copy paste error
