Add a variants array which can specify overriden properties of the
info.json for building multiple versions of the mod at the same time and
fix the additonal_files to allow specifying under what path the file is
put.
Together these two changes allow setting up the build to do all
supported versions at the same time and add specialized code for each
version of Factorio supported.
Add log query and log follow commands which allows inspection of the
combined cluster log. Past entries can be queried via the new queryLog
request. The log is stored into a single flat file for now, though the
performance of this may good enough.
Add winston based logger for master, slave, ctl, lib, and web_ui code.
Logging done from slaves and instances are sent to the master server and
tagged with the instance and/or slave it originated from. This allows
for example error messages to propagate back to the source that iniated
the action. The master server also stores the log to allow it to be
queried at a later point. For slaves a copy of the slave's log is also
stored locally.
Add a sendRcon method to BaseInstancePlugin and the slave's Instance
class. This servers two purposes, one is to reduce the repetition when
send RCON commands particularly from plugins, and the other is to
instrument the RCON commands sent with metrics about duration and size.
Move common functionality between labeled collectors out of
ValueCollector and into a new LabeledCollector class, and use this as
the basis for implementing a Histogram collector.
Fix the console view of the instance web interface losing the output
subscription after a reconnect to the master server due to not
re-subscribing to the outputs.
Under the current system there's no distinction made on the slave side
between instance config files that are present in the filesystem and
instances actually assigned to the slave. Which could lead to problems
with determing if an instance is actually assigned to the slave.
Resolve this issue by separating discovered instances with assigned
instances. Upon startup discovered instances are put into the
discoveredInstanceInfos mapping on the Slave object and all of these are
sent to the master server. The master server in turn sends assignment
requests for all the instances which are assigned to the slave. These
are put into the instanceInfos mapping of the Slave object which can
then be relied up to include only assigned instances.
Fix trying to use non-existent method on Slave class for forwading
events to the master. Also update request forward to be consistent with
event forwarding.
Add option to set the the maximum amount of commands transmitted in
parallel to the game. Increasing this value allows a greater rate of
commands to be executed.
When statistics_exporter handles getMetrics it may take a long time for
the result of command it sends to be returned if the instance's command
stream is overloaded, this in turn would cause the slave to miss its
window to reply on the getMetrics request and no metrics for the slave
would be available. Ensure statistics_exporter does not use more than 5
seconds to respond by adding a timeout to the metrics gathering.
When the slave metrics request takes more than 10 seconds the response
to the /metrics endpoint ends up being discarded by Prometheus which
defaults to a timeout of 10 seconds. Fix by ignoring slaves that take
longer than 8 seconds to respond to the metrics request.
Amend the error message shown when a message handler is missing from a
plugin with the expected name of the handler and the name of the plugin
it's missing from.
Use 45 seconds as the default heartbeat interval instead of 60. Using
60 causes connections to drop when proxied through Nginx as the default
timeout there is also 60 seconds.
Print only the error message in the console when a connection error is
encountered on the link connector. This reduces log spam on reconnect
for slaves.
Detect and colour the Script and Verbose levels. The format of the
source file for the Script level is slightly different and needs an
update to the output parsing.
Set breakpoint for the sidebar so that it hides by default when the page
width is too small to show it with the content and keep the width of the
content constant on small screens while sliding the sidebar out to
prevent it getting crushed.
Add command to generate a user token for connecting to the master server
web interface. This servers as an alternative to generating a control
config for use with clusterioctl.
Move the definition and implementation of the config command to
lib/shared_comhmands.js, this removes the need for lib config to depend
on the fs module, which is not accessible in web build.
Remove the echoing out of the test log from test/integration/server.js
used for debugging weird hang on shutdown immediately after startup that
would only happen on Travis.
Hornwitser