flamenet-relay/docs/CAPACITY.md

Capacity, measured

Numbers from flamenet-e2e/test/load.mjs against this image on an 8-core OVH VPS (shared with other workloads), Postgres 17 + Redis 7. Each pair is a sender and a recipient holding a wait=20 long-poll; latency is send-accepted → poll-returned.

held polls	send rate	p50	p95	p99	relay CPU	relay RSS
200	50/s	18 ms	58 ms	4.0 s	~idle	33 MB
500	100/s	17 ms	30 ms	325 ms	~1.0 core	44 MB
1000	100/s	18 ms	38 ms	1.0 s	~1.1 cores	92 MB
2000	150/s	22 ms	1.5 s	2.6 s	~1.2 cores	126 MB

Delivery was 100% of accepted sends in every run. The p95 growth at 2000 held polls is the next knee approaching; the p99 tails at every scale are wake-ups that landed between two polls and waited out a cycle.

The table above predates in-process wake-ups (2026-10-14) and has not been re-run since. Re-run it before quoting these numbers for the current image.

The two ceilings this measurement found, and their fixes

1. The Redis client pool. Each held long-poll used to pin one Redis connection for its whole BRPOP wait, and the pool is per event loop. At 32 per loop, ~250 held polls saturated the pool and p50 went from 18 ms to seconds while the CPU burned three cores queueing. The first fix raised the pool to 512 per loop.

   Since 2026-10-14 one relay keeps its waiters in memory (WakeHub in RoutesWake.swift): a poll registers once per request, before its first cursor query, and Delivery.write signals it after the commit. A held poll holds no Redis connection and no database connection, so the pool is back to 32 per loop, for short commands only. A wake that lands between the query and the wait is kept on the waiter, which removes the p99 tail of wake-ups that waited out a cycle.

   With FN_RELAY_REPLICAS above 1 a send can land on another relay, so wake-ups still go through BRPOP on a Redis list per device, the pool default stays 512, and 512 per loop is the held-poll ceiling. Either default is settable with FN_REDIS_MAX_CONNECTIONS_PER_LOOP. A failed Redis wait sleeps one second before the next cursor query.

2. File descriptors. A held poll costs one fd (the HTTP socket) on one relay, and two (plus its Redis connection) with several replicas. Docker's default soft limit of 1024 capped the relay at ~450 open apps at two fds each, and past it connection setup fails hard. The compose files set ulimits: nofile: 65536, and the relay warns at boot under a low limit.

A poll reads id and payload length first, then loads payloads only for rows inside the 2 MiB budget (Routes.pollRows). 500 padded envelopes waiting cost one 2 MiB read, not 32 MB.

Also worth knowing: FN_PG_MAX_CONNECTIONS_PER_LOOP (default 4). Fluent's own default is ONE connection per event loop. And the SQLite tier locks under parallel registration: one file is for one household, not concurrency.

Pushes run on one bounded worker per process: 8 at a time, 1024 queued (PushWorker). Past that a push is dropped and counted, and the relay logs the running total; the envelope waits for the next poll. Sends to one device take a row lock on that device until commit (docs/FANOUT.md), so concurrent sends to the same device are serialized; sends to different devices are not.

Reproducing

Never against production. Boot a throwaway stack whose issuer keypair you minted, then:

LOAD_URL=http://target:8080 \
LOAD_ISSUER_PRIVATE_KEY=<base64 raw ed25519 seed> \
LOAD_PAIRS=1000 LOAD_RATE=100 LOAD_DURATION=60 node test/load.mjs