Fix message seq reuse on forwarding (#579)

When messages were forwarded they were sent using the low level
connector .send method which accepts a message and sends it over the
link.  This method sends the message as is with no modification which
means that at no point during the forwarding logic was the message
sequence number that originated from Link A renumbered to the correct
sequence number to send out on Link B which it was forwarded on.

This in turn caused the logic which drops acked messages from the send
buffer to break which in turn meant the resume could cause duplication
of message and/or messages to be lost.  With enough duplicated messages
the resume of one connection could cause other connections to drop and
also initiate resume.

Fix by renumbering the sequence on forwarded messages.  In the future
the sequences sent and received should also be checked to make sure this
kind of problem does not happen again, this is not done here because
partially updating a running cluster with sequence problems to one that
checks and enforces sequences numbers will be problematic.
This commit is contained in:
Hornwitser authored and GitHub committed 2024-01-26 13:02:26 +01:00
1 parent 7d6eac6558
commit f5255d6f76
6 files changed
+54 -12

No files matched your search

+3 -3
View File
@@ -84,7 +84,7 @@ export default class ControllerRouter {
if (message.type === "request") {
nextHop.forwardRequest(message as lib.MessageRequest, origin);
} else {
nextHop.connector.send(message);
nextHop.connector.forward(message);
}
} else {
this.warnUnrouted(message, msg);
@@ -107,13 +107,13 @@ export default class ControllerRouter {
if (dst.id === lib.Address.host || dst.id === lib.Address.instance) {
for (let hostConnection of this.controller.wsServer.hostConnections.values()) {
if (hostConnection !== origin && (!plugin || hostConnection.plugins.has(plugin))) {
hostConnection.connector.send(message);
hostConnection.connector.forward(message);
}
}
} else if (dst.id === lib.Address.control) {
for (let controlConnection of this.controller.wsServer.controlConnections.values()) {
if (controlConnection !== origin) {
controlConnection.connector.send(message);
controlConnection.connector.forward(message);
}
}
} else {