aperx/factorio-clusterio
1
0 Fork 0
mirror of https://github.com/clusterio/clusterio.git synced 2026-11-15 15:24:52 +09:00
Code Issues Packages Projects Releases Wiki Activity

Rewrite reconnection logic

Browse Source 
Implement proper session tracking and resume on reconnect for the
WebSocket server.
This commit is contained in:
Hornwitser committed 2020-04-02 21:58:44 +02:00
1 parent 026b7f1306
commit d4e072f376
24 files changed
+1403 -319
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/socket.md
+214 -50
View File
@@ -1,4 +1,4 @@
Socket.io protocol
WebSocket protocol
==================
<sub>This document describes the protocol used between the master and
@@ -7,64 +7,211 @@ to be very useful for you.</sub>
The protocol is built on WebSocket and uses JSON encoded text messages
to communicate to and from the master server. These messages contain
the following three properties:
- seq - integer - The sequence number of the message
a JSON encoded object with the following three properties:
- seq - integer|null - The sequence number of the message
- type - string - The type of message
- data - object - Message data
For the handshake and heartbeats the seq property is null.
Handshake
---------
Upon connecting to the WebSocket endpoint the server will respond with a
`hello` message containing the master server version as the data. The
client is then expected to send a `register_slave` or `register_control`
message back depending on what kind of client it is. If the register
operation is successful the master will reply with a `ready` message
with an empty data payload.
client is then expected to send either `register_*` or a `resume`
message to set up the connection with the master. The exact flow is
described below.
**Master `hello` data**
### Client connection flow
For a client making a new connecting to the master server the first
course of action after obtaining a token is to connect with WebSocket to
the /api/socket endpoint and wait for the server hello. After the
hello, the client sends a register\_\* message corresponding to the type
of the client. If successful the server will reply with a ready message
and the connection has been established. See the flowchart below.
![Client connection flowchart](assets/connection_flow.svg)
If for any reason the connection is closed or lost during the connection
attempt the process is started over after a random delay, with the
exception of the connection being closed with a 4003 code which means
authentication failed and the client should stop trying to connect. The
wait for hello and wait for ready states should also have a timeout,
after which the connection is closed if the hello/reply wasn't received
in time.
### Client reconnection flow
If the connection is lost after the client has received the ready
message from the server it should attempt a reconnect instead. This is
very similar to a new connection, but instead of a register\_\* messsage
the client sends a resume message and the server replies with a continue
message if resuming the session was successful. This continue message
has sequence number of the last message the master received and the
client is expected to imediately send all messages from its send buffer
that occured after the sequence number given.
The server will respond with an invalidate message if resuming the
session is not possible. In this case the client should continue from
the new session state of the connection flow (see the previous section).
![Client reconnection flowchart](assets/reconnection_flow.svg)
Like the connection flow the process starts over after a delay if the
connection is closed or lost. The same exception for connection closed
with 4003 applies as with the connection flow. Additionally after an
invalidate message has been received and the connection flow fails, it
should start over from the connection flow.
### Server connection flow
From the master server side new connections and reconnection attempts
start out the same, a new connection has been made and the server sends
the hello messoge to the client. If a vaild register\_\* message is
received a new connection is set up for the client and the ready message
is sent in reply. If a valid resume message is received and the session
that's attempted to be resumed is active it'll signal that the session
is resumed with a continue message and the client upon receiving the
continue message should send the messages from its send buffer from
after the last\_seq given in the continue message.
If the session token used to resume is not valid it'll send an invalidate
message and expect the client to continue with a registr\_\* message. If an
invalid or unexpected message is received during the handshake the connection
is closed. See the flowchart below.
![Client connection flowchart](assets/server_flow.svg)
For authentication failure the connection is closed with code 4003,
which the connection client should interpret as connecting will never
succeed and stop trying to reconnect.
### Connection tracking and heartbeats
Once connected as indicated by the ready and continue messages from the
master server the client must start sending heartbeats at the interval
given in the ready/continue message, and storing messages sent over the
link in a sending side buffer. The heartbeats sent must have the seq of
the last received message on the link and acts as an ack for received
messages. The master server also sends heartbeats in the same manner
and the client should remove messages up to and including the sequence
number received in the heartbeat from its send buffer.
Handshake messages
------------------
### `hello`
Sent by the master server when the WebSocket connection has been opened.
- version - string - The version of the master, e.g. "2.0.0".
**Slave `register_slave` response data**
### `register_slave`
- agent - string - Human readable identifying the software connecting.
Slave handshake for establishing a new connection session.
- token - string - Authentication token.
- agent - string - Human readable string of the software connecting.
- version - string - The protocol version of the slave, e.g. "2.0.0".
- name - string - Name of the slave.
- id - integer - ID of the slave.
**Control `register_control repsonse data**
### `register_control`
- agent - string - Human readable identifying the software connecting.
Control handshake for enstablishing a new connection session.
- token - string - Authentication token.
- agent - string - Human readable string of the software connecting.
- version - string - The protocol version of the control, e.g. "2.0.0".
**Example handshake**
```js
// Server hello
({
type: 'hello',
data: {
version: "2.0.0",
}
})
### `resume`
// Client response
({
type: 'register_slave',
data: {
agent: 'Clusterio Slave',
version: "2.0.0",
name: "Foo",
id: 123,
}
})
Handshake for resuming an existing connection session where the
connection had dropped unexpectedly.
// Server repsonse
({
type: 'ready',
data: {}
})
- session_token - string - Session token to resume session with.
- last_seq - integer|null - Last message seq received by the client.
### `invalidate`
Sent by the master server in reply to a resume for a session that cannot
be resumed. The client should start a new session with a register\_\*
message in response.
### `continue`
Sent by the master server to signal resuming the session was successful.
- last_seq - integer|null - Last message seq received by the server.
- heartbeat_interval - number - Rate client should send heartbeats at.
### `ready`
Sent by the master server to signal the connection has been established
with a new session.
- session_token - string - Token to resume the session with.
- heartbeat_interval - number - Rate client should send heartbeats at.
### `heartbeat`
Sent by both the master server and clients at the interval given in the
continue/ready messages.
- last_seq - integer - Last message seq received over the link.
Example handshake
-----------------
Server hello after opening the WebSocket.
```json
{
"seq": null,
"type": "hello",
"data": {
"version": "2.0.0"
}
}
```
Slave registering a new connection session.
```json
{
"seq": null,
"type": "register_slave",
"data": {
"token": "xyzabc123",
"agent": "Clusterio Slave",
"version": "2.0.0",
"name": "Foo",
"id": 123
}
}
```
Master server reply signaling the session is established and ready.
```json
{
"seq": null,
"type": "ready",
"data": {
"session_token": "xyz.azy.bzy",
"heartbeat_interval": 45
}
}
```
@@ -73,11 +220,11 @@ Events
Events are messages send in one direction over the link, that the
receiver is expected to act upon, but not make any replies (for that
there's [Requests](#requests). By convention a message types ending in
`*_event` is an event. The data payload of an event consists solely of
there's [Requests](#requests). By convention message types ending in
`*_event` are events. The data payload of an event consists solely of
event specific properties.
See [lib/link/messages.js](lib/link/messages.js) for the recognized
See [lib/link/messages.js](/lib/link/messages.js) for the recognized
events and their contents.
@@ -87,30 +234,47 @@ Requests
Requests are messages which expects a response in return. They function
a lot like HTTP requests except that both parties of a connection can
inniate a request that is expected to be responded to. By convention
message types ending in `*_request` is a request, and is expected to be
message types ending in `*_request` are requests, and is expected to be
replied to with a corresponding `*_response` message. If an error
occured while processing the request an error response containing an
error property in the data payload is sent instead.
**`*_response` data**
See [lib/link/messages.js](/lib/link/messages.js) for the recognized
requests and their contents.
### `*_response`
- seq - integer - The seq number of the request message.
- additional properties may be present depending on the type of request.
**`*_resopnse` error data**
### `*_response` error
- seq - integer - The seq number of the request message.
- error - string - Human readable text message describing the error.
See [lib/link/messages.js](lib/link/messages.js) for the recognized
requests and their contents.
Graceful shutdown
-----------------
Closing the connection
----------------------
To ensure no data is lost when closing the link it's important to
send connection shutdown request and wait for it to complete before
closing the WebSocket. Either party of the link can send this request
and the party sending the request should be the one that initates the
close of the WebSocket.
The party that innitates the shut down of the connection should send a
close message containing the reason for closing before disconnecting.
### `connection_shutdown_request`
**`close` data**
Request graceful shutdown of the connection. The receiving party should
stop tasks that send events and requests over the link and wait for
pending requests it has over the link to complete, but it should not
stop responding to requests sent to it. Only after this process is done
should the response be sent.
- reason - string - Reason for closing the connection
### `connection_shutdown_response`
Signals the shutdown is complete and that the connection can be closed.
The party that initiatied the shutdown request may still send other
requests/events over the link to finish its part of the shutdown.
If the error response variant is received it should be logged if
possible and treated as a success.