Prove a second relay instance actually works (relay#31) #100

2026-09-04T15:15:49Z

paulhitt commented

2026-09-04 15:15:49 +00:00

(Migrated from gitlab.com)

"Wake tokens are per device in Redis, so any instance can serve any waiter
and horizontal scaling needs no protocol change" was reasoning that had never
been executed. This runs two Routes against one SQLite file and asks the
questions directly.

The belief holds where it was aimed. A device registered on one instance
exists on the other; a token minted against one authenticates against the
other; a message sent through one is collected through the other; both serve
the same key log, leaf hashes included. The interesting one is the held poll:
with no Redis at all, a poll waiting on B is released by a send that lands on
A, because the fallback interval runs a cursor query against the shared
database. That is the actual scaling answer and it now has a test.

What the reasoning missed is the state the relay keeps in memory, and two
pieces of it are not caches. Replay protection is what makes a
proof-of-possession signature single-use — two instances keep two caches, so
a captured X-FN-Signature refused by the relay that first saw it is accepted
by the other one. Proof-of-possession silently becomes a bearer credential
for the length of the skew window, with no protocol change to blame. The
sign-in lockout and the attachment quota are per-process counters, so N
replicas give an attacker N times the guess budget and the load balancer
distributes the attempts for them.

So FN_RELAY_REPLICAS > 1 with no FN_REDIS_URL now refuses to boot, naming
what would stop holding. Refused rather than warned: every request would
succeed and nothing would error — the only symptom is a guarantee this
project states in public quietly becoming false.

Wake-ups are deliberately not a reason to refuse. Without Redis the poll
falls back to the interval, which works across instances; the cost is latency,
not correctness, and refusing over that would teach an operator to route
around the check.

What this does NOT prove: it is an in-process pair on one SQLite file, not
two containers behind Traefik on Postgres and Redis. It covers the protocol
and data questions the issue asked; the deployment rehearsal is still owed.

The harness gained withRelayPair, which puts both instances on a shared
FILE — a pair on .memory would each get their own database and every
cross-instance assertion would pass while proving nothing.

623 relay tests in 55 suites green, core paths 7/7.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com

"Wake tokens are per device in Redis, so any instance can serve any waiter and horizontal scaling needs no protocol change" was reasoning that had never been executed. This runs two Routes against one SQLite file and asks the questions directly. The belief holds where it was aimed. A device registered on one instance exists on the other; a token minted against one authenticates against the other; a message sent through one is collected through the other; both serve the same key log, leaf hashes included. The interesting one is the held poll: with no Redis at all, a poll waiting on B is released by a send that lands on A, because the fallback interval runs a cursor query against the shared database. That is the actual scaling answer and it now has a test. What the reasoning missed is the state the relay keeps in memory, and two pieces of it are not caches. Replay protection is what makes a proof-of-possession signature single-use — two instances keep two caches, so a captured X-FN-Signature refused by the relay that first saw it is accepted by the other one. Proof-of-possession silently becomes a bearer credential for the length of the skew window, with no protocol change to blame. The sign-in lockout and the attachment quota are per-process counters, so N replicas give an attacker N times the guess budget and the load balancer distributes the attempts for them. So FN_RELAY_REPLICAS > 1 with no FN_REDIS_URL now refuses to boot, naming what would stop holding. Refused rather than warned: every request would succeed and nothing would error — the only symptom is a guarantee this project states in public quietly becoming false. Wake-ups are deliberately not a reason to refuse. Without Redis the poll falls back to the interval, which works across instances; the cost is latency, not correctness, and refusing over that would teach an operator to route around the check. What this does NOT prove: it is an in-process pair on one SQLite file, not two containers behind Traefik on Postgres and Redis. It covers the protocol and data questions the issue asked; the deployment rehearsal is still owed. The harness gained `withRelayPair`, which puts both instances on a shared FILE — a pair on `.memory` would each get their own database and every cross-instance assertion would pass while proving nothing. 623 relay tests in 55 suites green, core paths 7/7. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

paulhitt commented

2026-09-04 15:15:49 +00:00

(Migrated from gitlab.com)

assigned to @paulhitt

paulhitt commented

2026-09-04 20:40:49 +00:00

(Migrated from gitlab.com)

deleted the feat/capacity-metrics-status-page branch. This merge request now targets the main branch

deleted the `feat/capacity-metrics-status-page` branch. This merge request now targets the `main` branch

paulhitt commented

2026-09-04 20:40:51 +00:00

(Migrated from gitlab.com)

mentioned in commit ee55d46344

mentioned in commit ee55d46344cceac72226084005588bb8029e5e11

paulhitt (Migrated from gitlab.com) merged commit ee55d46344 into main

2026-09-04 20:40:51 +00:00

paulhitt commented

2026-09-09 18:59:11 +00:00

(Migrated from gitlab.com)

changed the description

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: extant2000/flamenet-relay#100