Spike: cold storage for delivered envelopes, and why it is declined (relay#28) #106

2026-09-04T17:01:52Z

paulhitt commented

2026-09-04 17:01:52 +00:00

(Migrated from gitlab.com)

Both halves of the premise are wrong, and both are now checkable.

Delivered envelopes do not "stay in the same table forever". Retention
deletes them after FN_DELIVERED_RETENTION_DAYS, which defaults to ONE day, on
an hourly sweep. The steady state of the table is about a day of collected mail
plus whatever is still uncollected — not an archive. EnvelopeBacklogTests pins
both windows and asserts the sweep removes what is past them, so the document
cannot go stale behind a changed default.

And the hot path does not slow down behind history. docs/COST.md measures
nine round trips behind 20 delivered and nine behind 120: the cursor query is
id > after filtered by to_device against envelopes(to_device, id), returning
one page whatever is behind it and never touching another device's rows.

The rows that DO accumulate are the undelivered ones, and they are not
waste. Each is a message for somebody whose phone is off, and the thirty-day
window is the promise it will still be there. Moving those to slower storage
would put the latency on the first poll after a holiday — the worst possible
place, because it lands on the person who has been away and expects everything
at once. Any cold-storage design has to leave them alone, which removes most of
what would have been moved.

What the spike produced is the measurement whose absence let this be filed. The
sweep reported what it DELETED and never what REMAINED, and a sweep removing a
thousand rows an hour looks identical whether the table is steady at ten
thousand or climbing past a million. The hourly line now carries the level:

held_delivered=1483 held_undelivered=207 oldest_undelivered_days=6

oldest_undelivered_days is the one to watch. At 30 the sweeper is dropping
somebody's mail every hour for devices that never came back; near zero the
window is slack nobody uses.

Two COUNT()s and one indexed lookup, once an hour, inside the sweeper —
already the slowest thing the relay does on purpose and already leader-elected,
so one replica pays it. Deliberately not on a route and not on the status page:
a COUNT() over a large table is a scan, and that page is public.

The oldest row is found by primary key rather than timestamp — ids are
database-assigned and monotonic and envelopes(to_device, id) already indexes
them — with a test pinning that the two orderings agree.

655 relay tests in 59 suites green, core paths 7/7.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com

Both halves of the premise are wrong, and both are now checkable. Delivered envelopes do not "stay in the same table forever". Retention deletes them after FN_DELIVERED_RETENTION_DAYS, which defaults to ONE day, on an hourly sweep. The steady state of the table is about a day of collected mail plus whatever is still uncollected — not an archive. EnvelopeBacklogTests pins both windows and asserts the sweep removes what is past them, so the document cannot go stale behind a changed default. And the hot path does not slow down behind history. docs/COST.md measures nine round trips behind 20 delivered and nine behind 120: the cursor query is `id > after` filtered by to_device against envelopes(to_device, id), returning one page whatever is behind it and never touching another device's rows. The rows that DO accumulate are the undelivered ones, and they are not waste. Each is a message for somebody whose phone is off, and the thirty-day window is the promise it will still be there. Moving those to slower storage would put the latency on the first poll after a holiday — the worst possible place, because it lands on the person who has been away and expects everything at once. Any cold-storage design has to leave them alone, which removes most of what would have been moved. What the spike produced is the measurement whose absence let this be filed. The sweep reported what it DELETED and never what REMAINED, and a sweep removing a thousand rows an hour looks identical whether the table is steady at ten thousand or climbing past a million. The hourly line now carries the level: held_delivered=1483 held_undelivered=207 oldest_undelivered_days=6 oldest_undelivered_days is the one to watch. At 30 the sweeper is dropping somebody's mail every hour for devices that never came back; near zero the window is slack nobody uses. Two COUNT(*)s and one indexed lookup, once an hour, inside the sweeper — already the slowest thing the relay does on purpose and already leader-elected, so one replica pays it. Deliberately not on a route and not on the status page: a COUNT(*) over a large table is a scan, and that page is public. The oldest row is found by primary key rather than timestamp — ids are database-assigned and monotonic and envelopes(to_device, id) already indexes them — with a test pinning that the two orderings agree. 655 relay tests in 59 suites green, core paths 7/7. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

paulhitt commented

2026-09-04 17:01:52 +00:00

(Migrated from gitlab.com)

assigned to @paulhitt

paulhitt commented

2026-09-04 17:02:07 +00:00

(Migrated from gitlab.com)

mentioned in issue #28

paulhitt commented

2026-09-04 20:41:09 +00:00

(Migrated from gitlab.com)

deleted the spike/relay-side-sealed-fanout branch. This merge request now targets the main branch

deleted the `spike/relay-side-sealed-fanout` branch. This merge request now targets the `main` branch

paulhitt commented

2026-09-04 20:41:11 +00:00

(Migrated from gitlab.com)

mentioned in commit 97257bfb92

mentioned in commit 97257bfb927d2bef635e47ce0bae73d22f3d6a31

paulhitt (Migrated from gitlab.com) merged commit 97257bfb92 into main

2026-09-04 20:41:11 +00:00

paulhitt commented

2026-09-09 18:59:04 +00:00

(Migrated from gitlab.com)

changed the description

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: extant2000/flamenet-relay#106