Optimize and stabilize the server #21

New Issue

2026-09-01T23:11:46Z

paulhitt commented

2026-09-01 23:11:46 +00:00

(Migrated from gitlab.com)

Ask: optimize and stabilize the server.

What is known about its limits (measured 2026-09-01)

Postgres max_connections = 100 — the real ceiling. 8 in use at rest. Every held
long-poll that runs a cursor query draws from it, and the ack path shares it.
Redis maxclients = 10000, file descriptors 65,536 — neither is close.
8 cores and 22 GB shared with every other service on the box, including a media stack.

Scaling direction is tracked separately in #14. This ticket is the code and the runtime.

Stabilisation items

Connection pooling. PgBouncer in front of Postgres is the cheapest large win and
needs no protocol change. Currently absent.
Dead tuples. users 20 dead against 2 live, groups 7 against 1, group_members 9
against 0. Autovacuum is presumably running but the ratios suggest it rarely triggers on a
quiet table. Relevant to #17 as well, since old row versions retain overwritten plaintext.
No sweeper for delivered envelopes. They accumulate indefinitely. FN_DELETE_ON_ACK
exists and is deliberately OFF because it destroys ciphertext unrecoverably — but that is
not the only alternative to keeping everything forever. See #18.
Boot-order races. The 2026-08-28 outage was a sweeper touching Redis before Vapor
finished booting, and it only fired when boot ran slow. Scripts/staging-gate.sh now
requires a stability window for exactly this reason. Worth auditing whatever else runs at
startup for the same shape.
Rate limits are set to debugging values — #20.
The relay runs as a Postgres superuser — #15.

Suggested first move

PgBouncer, then measure concurrent held connections at peak. Everything else in the scaling
conversation is guesswork until that number exists.

⚠️ Freeze in effect. Related: #14, #15, #17, #18, #20.

**Ask:** optimize and stabilize the server. ## What is known about its limits (measured 2026-09-01) - **Postgres `max_connections = 100`** — the real ceiling. 8 in use at rest. Every held long-poll that runs a cursor query draws from it, and the ack path shares it. - Redis `maxclients = 10000`, file descriptors 65,536 — neither is close. - 8 cores and 22 GB shared with every other service on the box, including a media stack. Scaling direction is tracked separately in #14. This ticket is the code and the runtime. ## Stabilisation items - **Connection pooling.** PgBouncer in front of Postgres is the cheapest large win and needs no protocol change. Currently absent. - **Dead tuples.** `users` 20 dead against 2 live, `groups` 7 against 1, `group_members` 9 against 0. Autovacuum is presumably running but the ratios suggest it rarely triggers on a quiet table. Relevant to #17 as well, since old row versions retain overwritten plaintext. - **No sweeper for delivered envelopes.** They accumulate indefinitely. `FN_DELETE_ON_ACK` exists and is deliberately OFF because it destroys ciphertext unrecoverably — but that is not the only alternative to keeping everything forever. See #18. - **Boot-order races.** The 2026-08-28 outage was a sweeper touching Redis before Vapor finished booting, and it only fired when boot ran slow. `Scripts/staging-gate.sh` now requires a stability window for exactly this reason. Worth auditing whatever else runs at startup for the same shape. - **Rate limits** are set to debugging values — #20. - **The relay runs as a Postgres superuser** — #15. ## Suggested first move PgBouncer, then measure concurrent held connections at peak. Everything else in the scaling conversation is guesswork until that number exists. ⚠️ Freeze in effect. Related: #14, #15, #17, #18, #20.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: extant2000/flamenet-relay#21