mirror of
https://github.com/clusterio/clusterio.git
synced 2026-11-13 07:04:53 +09:00
Document tile data storage
This commit is contained in:
Danielv123 committed
1 parent
c5c26bfad1
commit
ab8321ff4b
2 files changed
+140
-47
No files matched your search
+26
-47
@@ -3,59 +3,38 @@
|
||||
A Clusterio plugin that provides interactive minimaps of Factorio instances with real-time tile and entity data visualization.
|
||||
|
||||
## Features
|
||||
- Live updating map in web UI
|
||||
- Map per instance, force and surface - also available when instance is offline
|
||||
- Live updating terrain map in web UI
|
||||
- Recipe overlay showing active crafting machines
|
||||
- Timelapse mode for viewing historical changes
|
||||
- Chart tag visualization and management
|
||||
- Per instance, force and surface support
|
||||
- Works when instances are offline
|
||||
|
||||
## Implementation
|
||||
## Documentation
|
||||
|
||||
The flow is roughly:
|
||||
- Factorio sends rgb565 of a 32x32 chunk
|
||||
- The controller adds the chunk to the chunk saving queue for its tile
|
||||
- The chunk is broadcast to all web clients viewing the map to allow live updates
|
||||
- **[Tile Data Storage](./docs/tile_data_storage.md)** - Binary format for terrain visualization data
|
||||
- **[Recipe Overlay](./docs/minimap_recipe_overlay.md)** - Compact storage for crafting machine recipes
|
||||
|
||||
Tile saving has a separate flow:
|
||||
- The tile is opened and processed to the current time
|
||||
- For each chunk in saving queue:
|
||||
- Calculate changed pixels. If there are any, append a pixel change with tick information to the tile file
|
||||
- If any changes are made, save tile
|
||||
## Architecture Overview
|
||||
|
||||
When a user opens the map:
|
||||
- All visible tiles are requested from the server with either a specific tick or latest
|
||||
- The server sends the last tile file saved before the specified tick
|
||||
- The web client parses each tile file and replays changes up to the specified tick
|
||||
- Broadcasted chunks are overlayed on top of the tile files in memory directly when in live mode
|
||||
The plugin consists of three main data streams:
|
||||
|
||||
### File formats
|
||||
### Terrain Data
|
||||
- Factorio sends RGB565 data for 32×32 chunks
|
||||
- Controller queues chunks and generates incremental pixel changes
|
||||
- Data stored in append-only `.bin` files for efficient timelapse replay
|
||||
- Web UI renders live updates and historical data
|
||||
|
||||
Map colors in factorio are stored as rgb565 internally in the engine. That is 5 bits for red, 6 bits for green, 5 bits for blue.
|
||||
### Recipe Data
|
||||
- Tracks which recipe each crafting machine is running
|
||||
- Compact binary format with per-tile dictionaries (~8x smaller than JSON)
|
||||
- Live streaming of recipe changes with timeline support
|
||||
|
||||
From factorio we are sending 32x32x2 byte pixels encoded with encode_string() which applies deflate and base64.
|
||||
### Chart Tags
|
||||
- User-created map markers with text and icons
|
||||
- JSONL storage format for simplicity
|
||||
- Real-time synchronization across all connected clients
|
||||
|
||||
From the controller we are sending either chunks with live updates or larger tiles with 8x8 chunks.
|
||||
## Usage
|
||||
|
||||
The tile format is as follows:
|
||||
- 1 byte: Type identifier
|
||||
- 1 = chunk
|
||||
- 2 = pixels
|
||||
|
||||
For chunk:
|
||||
- 4 byte: math.floor(tick / 60)
|
||||
- 4 bits: chunk relative X coordinate
|
||||
- 4 bits: chunk relative Y coordinate
|
||||
- 2 byte: length, number of data bytes
|
||||
- n bytes of 32x32 rgb565, with deflate compression
|
||||
|
||||
Type 1 chunks are assumed to only occur upon first exploration - further changes usually happen as individual pixels. For that reason, there is no rewind information tied to chunks, and they revert to black when rewinded.
|
||||
|
||||
For pixels:
|
||||
- 4 byte: math.floor(tick / 60)
|
||||
- 2 byte: length, number of pixels
|
||||
- Each pixel:
|
||||
- 1 byte X coordinate
|
||||
- 1 byte Y coordinate
|
||||
- 2 bytes for new rgb565 value
|
||||
- 2 bytes for previous rgb565 value (used for fast rewinding in timeline)
|
||||
|
||||
Multiple subsequent chunk/pixel blocks can appear with the same tick, although they should be merged to save space if possible.
|
||||
|
||||
Tiles are compressed with deflate and base64 encoded to fit in json payloads.
|
||||
Access the minimap through the Clusterio web interface. Select an instance to view its terrain, toggle recipe overlays, enable chart tags, or switch to timelapse mode to review historical changes.
|
||||
@@ -0,0 +1,114 @@
|
||||
# Minimap Tile Data Storage Format
|
||||
|
||||
## Overview
|
||||
The minimap plugin stores terrain visualization data in a compact binary format optimized for both live updates and historical timelapse playback. The system uses append-only tiles that capture pixel changes over time.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Data Flow
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant L as Factorio Lua
|
||||
participant H as Host Plugin
|
||||
participant C as Controller
|
||||
participant W as Web UI
|
||||
|
||||
L->>H: chart_data (32x32 chunk)
|
||||
H->>C: TileDataEvent
|
||||
C->>W: TileDataEvent (broadcast)
|
||||
W->>C: GetRawTileRequest
|
||||
C->>W: tile data (.bin)
|
||||
```
|
||||
|
||||
### Storage Organization
|
||||
- **Tile size**: 256×256 pixels (8×8 chunks of 32×32 each)
|
||||
- **Filename**: `<instance>_<surface>_<force>_<tileX>_<tileY>.bin`
|
||||
- **Location**: `database_directory/minimap_tiles/`
|
||||
- **Format**: Append-only binary records
|
||||
|
||||
## Binary Format
|
||||
|
||||
### Color Encoding
|
||||
Map colors use **RGB565** format (Factorio's internal representation):
|
||||
- 5 bits red, 6 bits green, 5 bits blue
|
||||
- 16 bits (2 bytes) per pixel
|
||||
|
||||
### Record Types
|
||||
All timestamps use `⌊tick / 60⌋` (seconds) as uint32.
|
||||
|
||||
| Bytes | Field | Description |
|
||||
| ----- | ------ | --------------------------- |
|
||||
| 1 | `type` | `1=Chunk`, `2=PixelChanges` |
|
||||
|
||||
#### Chunk Record (Type 1)
|
||||
Initial exploration data for a 32×32 area.
|
||||
|
||||
| Bytes | Field | Description |
|
||||
| ----- | ----------------- | -------------------------------- |
|
||||
| 4 | `tick` (uint32) | ⌊tick / 60⌋ timestamp |
|
||||
| 1 | `coords` (uint8) | `(chunkX << 4) \| chunkY` |
|
||||
| 2 | `length` (uint16) | Compressed data length |
|
||||
| n | `data` | Deflate-compressed RGB565 pixels |
|
||||
|
||||
- **Chunk coordinates**: 4-bit X and Y (0-7 within tile)
|
||||
- **Data**: 32×32×2 bytes (2048 bytes uncompressed)
|
||||
- **Compression**: Deflate (zlib)
|
||||
|
||||
#### Pixel Changes Record (Type 2)
|
||||
Incremental updates for individual pixels.
|
||||
|
||||
| Bytes | Field | Description |
|
||||
| ----- | ---------------- | ----------------------- |
|
||||
| 4 | `tick` (uint32) | ⌊tick / 60⌋ timestamp |
|
||||
| 2 | `count` (uint16) | Number of pixel changes |
|
||||
|
||||
**Per pixel change (6 bytes each):**
|
||||
| Bytes | Field | Description |
|
||||
| ----- | -------------------- | ---------------------------- |
|
||||
| 1 | `x` (uint8) | X coordinate in tile (0-255) |
|
||||
| 1 | `y` (uint8) | Y coordinate in tile (0-255) |
|
||||
| 2 | `new_color` (uint16) | New RGB565 value |
|
||||
| 2 | `old_color` (uint16) | Previous RGB565 value |
|
||||
|
||||
## Processing Strategy
|
||||
|
||||
### Live Updates
|
||||
1. **Exploration**: New chunks sent as Type 1 records
|
||||
2. **Changes**: Individual pixel updates as Type 2 records
|
||||
3. **Optimization**: Multiple changes at same tick can be merged
|
||||
|
||||
### Storage Efficiency
|
||||
- **Chunks**: ~100-500 bytes after compression (vs 2048 uncompressed)
|
||||
- **Pixels**: 6 bytes per change (compact for sparse updates)
|
||||
- **Timeline**: Old color values enable fast rewind without recalculation
|
||||
|
||||
### Timelapse Playback
|
||||
1. **Forward**: Apply chunks and pixel changes up to target tick
|
||||
2. **Backward**: Revert pixel changes using old_color values
|
||||
3. **Chunks**: Revert to black when rewinding past exploration time
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
### Saving Process
|
||||
```
|
||||
1. Queue chunk data in chunkSavingQueue
|
||||
2. Every 5 seconds, process queued chunks:
|
||||
- Load existing tile (if exists)
|
||||
- Compare new chunk pixels with current tile state
|
||||
- Generate pixel change records for differences
|
||||
- Append new data to tile file (never modify existing)
|
||||
```
|
||||
|
||||
### Web Client Rendering
|
||||
```
|
||||
1. Request tile data with optional maxTick parameter
|
||||
2. Parse binary records sequentially
|
||||
3. Apply chunks and pixel changes up to target time
|
||||
4. Convert RGB565 to RGBA for Canvas rendering
|
||||
5. Overlay live broadcasts for real-time updates
|
||||
```
|
||||
|
||||
### Cross-Platform Compatibility
|
||||
- **Node.js**: Uses Buffer with native endian operations
|
||||
- **Browser**: Uses Uint8Array with manual bit manipulation
|
||||
- **Shared utilities**: `tile-utils.ts` handles platform differences
|
||||
Reference in new issue
Block a user