Document in-process wake-ups and the Redis pool defaults

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

.env.example

@@ -246,16 +246,17 @@ FN_S3_PREFIX=attachments/
 # Rolling 30-day per-account attachment budget. Totals only, never per-blob.
 FN_ATTACHMENT_BYTES_PER_MONTH=524288000
 
-# Redis. docker-compose.yml runs without it: a single relay holds rate limits
-# and replay protection in memory, and a held poll wakes on its own timer
-# rather than the moment something arrives. docker-compose.prod.yml and
+# Redis. docker-compose.yml runs without it: a single relay holds rate limits,
+# replay protection and held-poll wake-ups in memory. docker-compose.prod.yml and
 # .federated.yml run a Redis and set this themselves. Set it here only if you
 # point the base file at a Redis of your own.
 # FN_REDIS_URL=redis://redis:6379
 
-# Concurrency knobs, measured in docs/CAPACITY.md. Held long-polls each pin a
-# Redis connection; capacity for open apps is roughly event-loops x this.
-FN_REDIS_MAX_CONNECTIONS_PER_LOOP=512
+# Concurrency knobs, measured in docs/CAPACITY.md. Redis connections per event
+# loop: 32 by default for one relay, which waits in memory. With
+# FN_RELAY_REPLICAS above 1 each held poll pins one for its BRPOP, the default
+# is 512, and capacity for open apps is roughly event-loops x this.
+# FN_REDIS_MAX_CONNECTIONS_PER_LOOP=512
 #
 # Postgres connections are different, and a held poll does NOT pin one. It
 # borrows one per cursor query for a few milliseconds. What draws on this is

docker-compose.federated.yml

@@ -13,9 +13,9 @@
 #   caddy   ,  TLS on 443, automatic certificates. Federation peers dial your
 #             public hostname over HTTPS; a relay on 127.0.0.1:8080 federates
 #             with nobody.
-#   redis   ,  wake-ups for held polls. Optional in principle; at federated
-#             scale the interval fallback turns delivery latency into a coin
-#             toss, so it ships on in this profile.
+#   redis   ,  rate limits and replay protection that survive a restart,
+#             and wake-ups for held polls when FN_RELAY_REPLICAS is above 1.
+#             One relay wakes held polls in memory.
 #   coturn  ,  voice/video call relaying for peers behind NAT. Without it,
 #             calls connect only when both phones can reach each other
 #             directly, which across the public internet is "sometimes".

docs/CAPACITY.md

@@ -16,19 +16,41 @@ Delivery was 100% of accepted sends in every run. The p95 growth at 2000
 held polls is the next knee approaching; the p99 tails at every scale are
 wake-ups that landed between two polls and waited out a cycle.
 
+The table above predates in-process wake-ups (2026-10-14) and has not been
+re-run since. Re-run it before quoting these numbers for the current image.
+
 ## The two ceilings this measurement found, and their fixes
 
-1. **The Redis client pool.** Every held long-poll pins one Redis connection
-   for its whole `BRPOP` wait, and the pool is per event loop. At the old 32
-   per loop, ~250 held polls saturated the pool and p50 went from 18 ms to
-   seconds while the CPU burned three cores queueing. Now 512 per loop,
-   settable with `FN_REDIS_MAX_CONNECTIONS_PER_LOOP`.
+1. **The Redis client pool.** Each held long-poll used to pin one Redis
+   connection for its whole `BRPOP` wait, and the pool is per event loop. At
+   32 per loop, ~250 held polls saturated the pool and p50 went from 18 ms to
+   seconds while the CPU burned three cores queueing. The first fix raised the
+   pool to 512 per loop.
 
-2. **File descriptors.** A held poll costs two fds (HTTP socket + Redis
-   connection). Docker's default soft limit of 1024 caps the relay at ~450
-   open apps, and past it connection setup fails hard. The compose files set
+   Since 2026-10-14 one relay keeps its waiters in memory (`WakeHub` in
+   `RoutesWake.swift`): a poll registers once per request, before its first
+   cursor query, and `Delivery.write` signals it after the commit. A held poll
+   holds no Redis connection and no database connection, so the pool is back
+   to 32 per loop, for short commands only. A wake that lands between the
+   query and the wait is kept on the waiter, which removes the p99 tail of
+   wake-ups that waited out a cycle.
+
+   With `FN_RELAY_REPLICAS` above 1 a send can land on another relay, so
+   wake-ups still go through `BRPOP` on a Redis list per device, the pool
+   default stays 512, and 512 per loop is the held-poll ceiling. Either
+   default is settable with `FN_REDIS_MAX_CONNECTIONS_PER_LOOP`. A failed
+   Redis wait sleeps one second before the next cursor query.
+
+2. **File descriptors.** A held poll costs one fd (the HTTP socket) on one
+   relay, and two (plus its Redis connection) with several replicas. Docker's
+   default soft limit of 1024 capped the relay at ~450 open apps at two fds
+   each, and past it connection setup fails hard. The compose files set
    `ulimits: nofile: 65536`, and the relay warns at boot under a low limit.
 
+A poll reads id and payload length first, then loads payloads only for rows
+inside the 2 MiB budget (`Routes.pollRows`). 500 padded envelopes waiting cost
+one 2 MiB read, not 32 MB.
+
 Also worth knowing: `FN_PG_MAX_CONNECTIONS_PER_LOOP` (default 4). Fluent's
 own default is ONE connection per event loop. And the SQLite tier locks
 under parallel registration: one file is for one household, not concurrency.

docs/CONFIGURATION.md

@@ -70,8 +70,8 @@ store keeps its own copy of each.
 
 | Variable | Default | Notes |
 |---|---|---|
-| `FN_REDIS_URL` | unset | Shared store for the state above. Set but unusable, held polls fall back to polling and the log says so. |
-| `FN_REDIS_MAX_CONNECTIONS_PER_LOOP` | `512` | Redis connections per event loop, which is also the ceiling on held polls. |
+| `FN_REDIS_URL` | unset | Shared store for the state above. Set but unusable, the log says so. With `FN_RELAY_REPLICAS` above 1 it also carries held-poll wake-ups between relays. |
+| `FN_REDIS_MAX_CONNECTIONS_PER_LOOP` | `32`, or `512` with `FN_RELAY_REPLICAS` above 1 | Redis connections per event loop. With several replicas each held poll pins one, so it is also the ceiling on held polls. One relay wakes held polls in memory. |
 | `FN_RELAY_REPLICAS` | `1` | How many instances share this database. More than one with no `FN_REDIS_URL` is refused at boot: a single-use proof signature would be accepted once per instance, and an attacker would get that many times the sign-in attempts. |
 
 ## The contact graph

docs/FANOUT.md

@@ -133,7 +133,8 @@ global envelope id, so clients see no change. A group fan-out locks one
 device set per request, in sorted order, so two group sends cannot deadlock.
 SQLite has one writer and skips the lock.
 
-After the commit it wakes held polls and queues one push job on the push
+After the commit it wakes held polls (in memory on one relay, through Redis
+with several, see `RoutesWake.swift`) and queues one push job on the push
 worker: 8 jobs run at a time and 1024 wait. A full queue drops the push and
 counts it; the envelope is still collected on the next poll. Shutdown flushes
 the queue for up to 10 seconds. Federated deposits now wake and push the same

docs/POOLING.md

@@ -12,10 +12,10 @@ because it changes what the fix is for.
 
 A held long-poll does **not** pin a database connection. `Routes.poll` borrows
 one from the pool for each cursor query, a few milliseconds, and gives it
-back between iterations. That was a deliberate choice, and `RoutesWake.swift`
-says why: Postgres `LISTEN` *would* have pinned a backend per waiter, which is
-what caps simultaneous waiters at about 86, and it was rejected for exactly that
-reason.
+back between iterations, and waits in memory between them (`RoutesWake.swift`).
+That was a deliberate choice: Postgres `LISTEN` *would* have pinned a backend
+per waiter, which is what caps simultaneous waiters at about 86, and it was
+rejected for exactly that reason.
 
 So the pressure on `max_connections` is not proportional to **held polls**. It
 is proportional to the **query rate**, and `docs/COST.md` measures that at nine