Paul HittandClaude Opus 5.5
e4378bce51
FN_DELETE_ON_ACK was built, tested and off: production has it set but empty (it was 1 from 2026-08-29 to 2026-09-01, turned off with a backup named .env.bak-deleteonack-20260901 and no reason written down anywhere I could find). Every delivered envelope then sat for a day, with its recipient, sender (unsealed sends), size and times. It is now on unless the flag says 0, false, no or off. Postgres only, as before: on SQLite it stays off whatever the flag says, because SQLite reuses row ids. A SQLite relay on the default is quietly off; only an explicit request on SQLite gets the critical "REFUSED" line, so the SQLite test fleet does not log a critical line on every boot. Why it is safe for the two apps as they are: - The cursor is the receipt. iOS advances lastInboxID only after an envelope is ingested or given up on after 12 stalled polls (E2EInbox.pollOnce); Android advances its watermark after consume, which holds refused-identity envelopes locally (Conversations.readInbox). Neither lowers its cursor to collect again; a reset cursor of 0 deletes nothing. - Per device: envelopes are addressed to_device and the delete is scoped to the polling device, so another device's inbox is untouched. - The iOS badge counts undelivered rows; only rows already returned and marked delivered are deleted. - Sealed-sender rows, read receipts and call signals are ordinary envelopes. Attachments are a separate table with their own 30 days. - The one-day sweep stays as the backstop for a device that collects and never polls again. What it costs: Scripts/call-health.sh measured ringing speed from collected envelopes, which are now gone within seconds, so that line mostly prints 0 calls; the script says so. And a client that ever advanced its cursor wrongly would lose the message for good instead of having a day. Docs: CONFIGURATION.md, README, COLDSTORAGE.md (which said it should stay off), the prod compose comment, and the metadata and privacy pages in both languages now say the day is the backstop. configuration.leaf and relay.leaf regenerated against the pinned e2e ref. Deploying this turns it on in production, because the box's .env has FN_DELETE_ON_ACK= (empty). Set FN_DELETE_ON_ACK=0 there first to keep it off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4.8 KiB
Cold storage for delivered envelopes: the spike
relay#28 proposed moving delivered envelopes to a cold table on a schedule, because "delivered envelopes stay in the same table serving live polls, forever. The hot path gets slower as history grows, for rows nobody will read again."
They do not stay forever, and the hot path does not get slower. Both halves of the premise are wrong, and both are now checkable. The recommendation is to decline, and the one useful thing the spike produced is the measurement that would tell you if the answer ever changed.
The table does not grow without bound
Retention deletes delivered envelopes after FN_DELIVERED_RETENTION_DAYS,
which defaults to one day, on an hourly sweep. Uncollected envelopes go
after FN_UNDELIVERED_RETENTION_DAYS, default thirty days.
So the steady state of the envelopes table is roughly:
one day of collected mail + whatever is still uncollected
Not an archive. EnvelopeBacklogTests pins both windows and asserts the sweep
actually removes what is past them, so this document cannot go stale behind a
changed default, the tests fail if either window moves.
There is also FN_DELETE_ON_ACK, which deletes an envelope the moment a
client's cursor moves past it. It is on by default on Postgres (since
2026-10), so on a production relay the day of collected mail above is mostly
gone within seconds and the one-day window is the backstop for a device that
collected and never polled again. Both apps advance the cursor only past an
envelope they have finished with, and neither lowers it to collect something
again, so nothing they do depends on the day. It is
still unrecoverable by design: a client that advanced its cursor wrongly
would lose the message for good, where before it had a day. relay#28
correctly warned about being confused with it. Nothing here touches it.
The hot path does not get slower either
Measured in docs/COST.md: a send-and-collect costs nine database round
trips behind 20 delivered envelopes and nine behind 120. Flat.
The cursor query is id > after filtered by to_device, against
envelopes(to_device, id). It returns one page whatever is sitting behind it,
and it never touches another device's rows. Retained history costs index size
and disk, it does not cost the poll extra work.
The one thing that is not flat is bytes, and the client decides it: a poll
with no after asks from zero and gets a page of up to a hundred retained
envelopes. Carrying the cursor collapses 21 KB back to 381 bytes. That is a
client behaviour, not a storage-layout problem, and cold storage would not
change it.
What uncollected mail is
The rows that actually accumulate are the undelivered ones, and they are not waste. Each is a message for somebody whose phone is off, and the thirty-day window is the promise that it will still be there when they turn it on.
Moving those to slower storage would make the first poll after a holiday the slowest request the relay serves, the worst possible place to put the latency, because it lands on the person who has been away and is expecting everything at once. Any cold-storage design has to leave them alone, which removes most of what would have been moved.
What the spike produced
The sweep reported what it deleted and never what remained, and that gap is why this could be filed at all: a sweep removing a thousand rows an hour looks identical whether the table is holding steady at ten thousand or climbing past a million.
The hourly retention swept line now carries the level as well as the flow:
held_delivered=1483 held_undelivered=207 oldest_undelivered_days=6
oldest_undelivered_days is the one to watch. Sitting at 30 means the sweeper
is dropping somebody's mail every hour for devices that never came back; sitting
near zero means everybody is collecting and the window is slack nobody uses.
Two COUNT(*)s and one indexed lookup, once an hour, inside the sweeper,
which is already the slowest thing the relay does on purpose and is already
leader-elected, so one replica pays it. Deliberately not on a route and not on
the status page: a COUNT(*) over a large table is a scan, and the status page
is public.
Recommendation
Decline relay#28 as filed. The archive it exists to move does not exist, the hot path it exists to protect is already flat, and the rows that do accumulate are the ones that must stay where they are.
What would change this answer is now something you can read off a log line
rather than reason about. If held_delivered climbs across sweeps, the sweeper
has stopped, which is a bug in retention, not an argument for a second table.
If held_undelivered climbs steadily, the relay is holding mail for a growing
population of dead devices, and the honest fix is the device-expiry path, not
cold storage. Either way the answer is somewhere other than here.