Due a bad order of operations when a host connection was closed the
event handler listening on "close" to remove the host from the
hostConnections mapping was added after the HostConnection's event
handler listening on "close" and invoking plugin hooks in response to
the host going offline. This in turn causes plugins that send
broadcasts in repsonse to thos hooks to throw an error due to the
broadcast sending code assuming connections that exist are valid target
to send messages to.
Fix by ordering the listener for the removal of the host connection
before the listener invoking plugin events.
Use a separate folder for the TypeScript build for Node.js in the dist
folder. This makes each type of build consistently have its own output
folder instead of being interleaved with the Node.js build.
Always send the instance details update when an instance is reassigned,
not just when it is unassigned. The idea for not sending an update when
assigned to a host was that the host would send an update on the status
that in turn updates the instance details. However this is not actually
the case; when a stopped instance is reassigned its status does not
change so the update is ignored.
Set the save's updatedAtMs timestamp when clearing the save list of a
reassigned or deleted instance. This fixes the Web UI not updating the
saves list to remove the saves when an instance is reassigned.
When messages were forwarded they were sent using the low level
connector .send method which accepts a message and sends it over the
link. This method sends the message as is with no modification which
means that at no point during the forwarding logic was the message
sequence number that originated from Link A renumbered to the correct
sequence number to send out on Link B which it was forwarded on.
This in turn caused the logic which drops acked messages from the send
buffer to break which in turn meant the resume could cause duplication
of message and/or messages to be lost. With enough duplicated messages
the resume of one connection could cause other connections to drop and
also initiate resume.
Fix by renumbering the sequence on forwarded messages. In the future
the sequences sent and received should also be checked to make sure this
kind of problem does not happen again, this is not done here because
partially updating a running cluster with sequence problems to one that
checks and enforces sequences numbers will be problematic.
When a plugin broadcasts an event to instances there may be hosts and
instances in the cluster which does not have the plugin loaded. This
would cause an error to be logged for every instance which does not have
the plugin loaded.
Filter out broadcast events sent to hosts and instances such that
instances and hosts that does not have the event do not received them.
Add suffixes like Ms and Ns to variables whos time unit is not in
seconds. This is aligning with a new code style of putting units
on variables that are not in SI base units.
Change the expected exported name of plugin declarations to be "plugin"
at the main entrypoint instead of the default export. This makes the
plugin's code organisation more flexible especially when using CommonJS
JavaScript modules as they can export additional symbols from the
plugin declaration file.
Implement remotely initiated stopping and restarting of the controller
and hosts. Restarting is implemented by exiting with a non-zero exit
code and relying on a process manager to start the procress again. To
prevent accidentally trying to restart on a system that doesn't have a
process manager present the --can-restart command line option needs to
be passed to enable the restart operation.
There are more pieces of information about a system than just its CPU,
Memory and disk metrics. To avoid having to build a separate collection
and subscription for metrics and system info, rename the existing system
metrics to system info in anticipation of providing additonal static
system information such as CPU model in it.
If the controller is behind a proxy then the socket.remoteAddress will
be that of the proxy instead of the end client that connected. In this
case the X-Forwarded-For header will contain the address of the end
client, but this should only be relied upon if the party connecting is a
trusted proxy, otherwise a malicious user could spoof the IP address
that is logged by providing a forged X-Forwarded-For header.
Implement a trusted proxy list consisting of a configured list of IP
addresses and/or CIDR blocks and use the last address[1] in the
X-Forwarded-For header if the remoteAddress address of the connecting
party is in the trusted proxy list.
[1]: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-For
Split out host specific data from RegisterHostData that is not related
to the establishment of a connection to the controller into it's own
HostInfoUpdate event. This allows this data to be updated when the
config values they are taken from are changed.
The agent string was inspired by the HTTP User-Agent header to serve as
a way to identify what's connecting to the cluster. While there may
have been vision of other software connecting to a controller than the
Clusterio provided ones this seems highly unlikely to ever happen.
Remove redundant agent string for now.
Implement simple collection of basic real time system metrics of hosts
and the controller for the purpose of displaying them in the Web UI. The
metrics are collected by the controller at a configurable interval and
pushed out to web clients currently subscribed to the metrics.
The Prometheus library has an optimisation for metrics that expose
multiple time series such as Summary that provide two values in the
time series consisting of the metric name suffixed with _sum and _count.
These are provided as a series of sufixes with corresponding samples such
that a serialised summary metric named some_value would have a samples
mapping looking something like this:
[
["_sum", ["label=\"value\"", 3]],
["_count", ["lavel=\"value\"", 2]],
]
If multiple hosts expose the same metric then these need to be merged,
which is the job of mergeSamples, which merges the samples from each
sufix into the destination.
One corner case that may occur is if the source and destination metric
that's being merged have different lists of suffixed samples. In this
case the suffixes that are present in the source but not in the
destination needs to be added verbatim to the destination. The code for
handling this case incorrectly added these samples to the source metric
instead of the destination metric.
This corner case only occrs if a metric's type differs between two
hosts connected to the same controller. For example if a Gauge metric
is turned into a Summary metric.
When ws was updated to version 8 it came with a breaking change where
message data was no longer utf8 decoded from binary to a string. The
main codepath handling messages was updated to handle this but the ws
dump path was not resulting in the controller crashing when receiving
a message that is then forwarded to the dumper without decoding.
Fix by decoding the message in the dump-ws path.
Refactor away the config group concept and simplyfy the way config
fields are defined. Fields are defined directly as properties of the
static field fieldDefinitions on the class itself with a separate
interface to declare the types of fields.
This makes it possible to easily make accessing the config fields type
checked in a way that lets plugins extend the interfaces with their own
config options by adding additional fields to the interface via
declation merging.
Refactor the Config and ConfigGroup classes to match the JSON data class
pattern with toJSON, toRemote and fromJSON methods replacing the load,
init, and serialize methods. Since fromJSON requires extra information
the messages structures passes around the plain JSON representation of
the configs instead of serialising instances of the config classes
automatically.
If a new save is added to the saves list of an instance the current save
selected to be loaded by default changes its state without updating the
mtime of the file on disk, which in turn means basing the updatedAt
property on mtime will not work with caching. Fix by having a separate
updatedAt property that is set by the controller whenever the save
details changes from the controller's perspective.
Noticed-By: @Cooldude2606
Since subscriptions are handlede per link there's no need for the
eventReplay to be returned as a part of the SubscriptionRequest, it can
be sent and processed normally as an event. Simplify the logic by
sending eventReplay as an event and remove SubscriptionResponse.
Remove the reliance a reference to the Controller to get the link that a
subscription is tied to by explicitly handling the SubscriptionRequest
and passing the link to the SubscriptionController.
Notify the SubscriptionController that a connection closed from the
Connection instead of having the subscription controller set up an event
to do so. This simplifies the logic on the SubscriptionController's
side.
Keep a copy of the instance saves details on the controller so that
saves can be queried even when the host is disconnected and the
controller can properly track and provide updates on the saves to
its clients.
Refactor HostUpdatesEvent, InstanceDetailsUpdatesEvent, ModUpdatesEvent,
ModPackUpdatesEvent and UserUpdatesEvent to have multiple updates of the
entities they support in each event. This is intended to make the events
capable of syncing the entire list that they represent.
Structure the storage of mods on the controller into into a reusable
component in preparation for the host using the same kind of mod storage
to retrieve and cache the mods it needs to run mod packs on instances.
Replace PluginWebApi's requirePath member with npmPackage which is only
populated if the package is not marked as private and the requirePath
matches the name. This prevents the public /api/plugins endpoint from
leaking filesystem paths into the controller.
Use the JSON data class pattern for User, Role, and PlayerStats and move
them into lib/src/data. This means that User is purely data now and
can't reference role objects directly. To make it possible easily check
permissions attached to roles and store data only visible on the
controller on user objects a ControllerUser type is introduced which is
linked with the UserManager and can therefore reference role objects
directly.
Resolves#525.
Hornwitser