Publish capacity metrics and a status page (relay#34) #99

2026-09-04T15:08:44Z

paulhitt commented

2026-09-04 15:08:44 +00:00

(Migrated from gitlab.com)

Nobody knew the peak concurrent held connections, so every scaling
conversation about this relay was guesswork. Requests-per-second does not
answer it: a poll that returns instantly and a poll that waits thirty
seconds are one request each and cost wildly different amounts. This
measures the concurrency directly, keeps the high-water mark, and publishes
both at /status and /status.json.

The filed trade was that a status page is a commitment to stay honest during
an outage, which is when it is hardest. Three decisions carry that:

Nothing is declared. The state is derived from checks taken on the request
and never stored, so there is no switch an operator can leave set to green.
Degraded outranks draining, because reporting a planned state over an
unplanned one is how a real outage gets read as routine.
A failed check is a rendered row, not a thrown error. Both endpoints answer
200 with the trouble in the body; /health remains the endpoint that exists
to be a status code. A page that 500s during an outage shows the reader a
browser error at the one moment it had something worth saying. Published
detail carries no host, bucket or connection string.
"Not checked" is a third state. Object storage is not probed from a page
anyone can load — that would be a bill the operator did not agree to and a
way for a stranger to generate traffic against their bucket — and declining
to check is printed rather than folded into a green tick.

There is deliberately no default ceiling. A held poll does not pin a database
connection, so max_connections is not the number of polls this can hold, and
a guess carrying a status page's credibility is worse than an admission.
Unset, the page says it can report the load but not whether it is a lot.

Counters only, never identifiers. The number of connections is safe to
publish and the set of them is not, so the set is never assembled.

Two faults found while building it:

The gauge is a lock rather than an actor so the decrement can live in a
defer. The cursor query runs once a second for as long as a poll is held,
so a database wobble throws out of the middle of the loop — and a trailing
leave() would lose every held connection at that moment, permanently
inflating the reading by exactly the incident somebody came to understand.
Proven by dropping the table under a live poll: red without the defer.
The uptime was a lie. RelayCapacity.shared is lazily created, so its clock
started on the first access — which is somebody loading /status. Caught by
the first real request returning a negative uptime. Marked at boot now.

612 relay tests in 54 suites green, core paths 7/7.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com

Nobody knew the peak concurrent held connections, so every scaling conversation about this relay was guesswork. Requests-per-second does not answer it: a poll that returns instantly and a poll that waits thirty seconds are one request each and cost wildly different amounts. This measures the concurrency directly, keeps the high-water mark, and publishes both at /status and /status.json. The filed trade was that a status page is a commitment to stay honest during an outage, which is when it is hardest. Three decisions carry that: - Nothing is declared. The state is derived from checks taken on the request and never stored, so there is no switch an operator can leave set to green. Degraded outranks draining, because reporting a planned state over an unplanned one is how a real outage gets read as routine. - A failed check is a rendered row, not a thrown error. Both endpoints answer 200 with the trouble in the body; /health remains the endpoint that exists to be a status code. A page that 500s during an outage shows the reader a browser error at the one moment it had something worth saying. Published detail carries no host, bucket or connection string. - "Not checked" is a third state. Object storage is not probed from a page anyone can load — that would be a bill the operator did not agree to and a way for a stranger to generate traffic against their bucket — and declining to check is printed rather than folded into a green tick. There is deliberately no default ceiling. A held poll does not pin a database connection, so max_connections is not the number of polls this can hold, and a guess carrying a status page's credibility is worse than an admission. Unset, the page says it can report the load but not whether it is a lot. Counters only, never identifiers. The number of connections is safe to publish and the set of them is not, so the set is never assembled. Two faults found while building it: - The gauge is a lock rather than an actor so the decrement can live in a `defer`. The cursor query runs once a second for as long as a poll is held, so a database wobble throws out of the middle of the loop — and a trailing leave() would lose every held connection at that moment, permanently inflating the reading by exactly the incident somebody came to understand. Proven by dropping the table under a live poll: red without the defer. - The uptime was a lie. RelayCapacity.shared is lazily created, so its clock started on the first access — which is somebody loading /status. Caught by the first real request returning a negative uptime. Marked at boot now. 612 relay tests in 54 suites green, core paths 7/7. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

paulhitt commented

2026-09-04 15:08:45 +00:00

(Migrated from gitlab.com)

assigned to @paulhitt

paulhitt (Migrated from gitlab.com) changed target branch from feat/rotate-identity-keys to main

2026-09-04 20:40:45 +00:00

paulhitt commented

2026-09-04 20:40:47 +00:00

(Migrated from gitlab.com)

mentioned in commit 7261167f6e

mentioned in commit 7261167f6e00feff4faa04d7b576eb01a03c404d

paulhitt (Migrated from gitlab.com) merged commit 7261167f6e into main

2026-09-04 20:40:47 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: extant2000/flamenet-relay#99