Standing: minimize metadata — what is stored, what actually uses it #18

New Issue

2026-09-01T23:02:40Z

paulhitt commented

2026-09-01 23:02:40 +00:00

(Migrated from gitlab.com)

Ask: keep looking for metadata to remove. Standing work, not a one-off.

There is already a public commitment at /metadata (Resources/Views/metadata.leaf)
listing what the relay keeps. That page is a promise, so anything added to the schema or
the wire has to land there too — which makes it the natural checklist for this ticket.

What the relay holds today, from the live schema (2026-09-01)

envelopes is where the interesting metadata is:

to_user, to_device, from_user, from_device, payload, delivered, created_at, envelope_id

payload is opaque ciphertext. Fine.
to_user + to_device are unavoidable — something has to be told where to deliver.
from_user / from_device default to 0 and '', which is sealed sender working:
a sealed deposit records no sender at all. Worth confirming what fraction of real traffic
is actually sealed, because the column existing means the unsealed path still does.
created_at gives timing, and the envelopes_inbox index on (to_device, id) makes
per-device history cheap to walk.

Other tables carrying metadata rather than content: devices, push_tokens,
delivery_keys, delivery_tokens, conversation_state, group_tags, group_members,
blocks, key_log. 25 tables total.

Specific things worth pulling on

1. silent is new and is NOT on /metadata. Added 2026-09-01: the sender tells the
relay an envelope needs no human attention, so it can send a background push rather than an
alert. The page mentions push tokens but not this. Either document it or reconsider it —
an undocumented disclosure is the worst of both. It is also genuinely reconsiderable: the
client could send nothing and let the app decide after decrypting, at the cost of relying
on silent pushes that iOS throttles.

2. urgent has the same shape and predates it — the relay learns an envelope is
time-critical, in practice "probably a call". Documented in Wire.swift as the one thing a
sender tells the relay about content; that is now two things.

3. Retention. delivered is set but nothing sweeps. Envelopes accumulate, and their
created_at is a conversation timeline. FN_DELETE_ON_ACK exists and is deliberately OFF
because it destroys ciphertext unrecoverably — but "keep everything forever" is not the only
alternative to it. A time-based sweep of delivered envelopes is a different trade and
worth costing.

4. Timing correlation. Sealed sender hides who sent a message; created_at plus
delivery order plus the per-device index may not. Worth asking what an operator watching the
table for a week could reconstruct.

5. Unsealed fallback. Sealing depends on holding the peer's delivery key. When that is
missing the send falls back to unsealed — and the relay learns the sender. How often that
happens in practice is measurable and currently unmeasured.

Suggested shape

Standing ticket. Each pass produces: what is stored, what actually reads it, and whether the
feature survives without it. Anything that survives without it should go; anything that does
not should appear on /metadata in plain words.

⚠️ Freeze in effect — investigate and document, do not remove.

Related: #16 (security review), #17 (plaintext inventory).

**Ask:** keep looking for metadata to remove. Standing work, not a one-off. There is already a public commitment at `/metadata` (`Resources/Views/metadata.leaf`) listing what the relay keeps. **That page is a promise, so anything added to the schema or the wire has to land there too** — which makes it the natural checklist for this ticket. ## What the relay holds today, from the live schema (2026-09-01) `envelopes` is where the interesting metadata is: ``` to_user, to_device, from_user, from_device, payload, delivered, created_at, envelope_id ``` - `payload` is opaque ciphertext. Fine. - `to_user` + `to_device` are unavoidable — something has to be told where to deliver. - `from_user` / `from_device` **default to `0` and `''`**, which is sealed sender working: a sealed deposit records no sender at all. Worth confirming what fraction of real traffic is actually sealed, because the column existing means the unsealed path still does. - `created_at` gives timing, and the `envelopes_inbox` index on `(to_device, id)` makes per-device history cheap to walk. Other tables carrying metadata rather than content: `devices`, `push_tokens`, `delivery_keys`, `delivery_tokens`, `conversation_state`, `group_tags`, `group_members`, `blocks`, `key_log`. 25 tables total. ## Specific things worth pulling on **1. `silent` is new and is NOT on `/metadata`.** Added 2026-09-01: the sender tells the relay an envelope needs no human attention, so it can send a background push rather than an alert. The page mentions push tokens but not this. **Either document it or reconsider it** — an undocumented disclosure is the worst of both. It is also genuinely reconsiderable: the client could send *nothing* and let the app decide after decrypting, at the cost of relying on silent pushes that iOS throttles. **2. `urgent` has the same shape** and predates it — the relay learns an envelope is time-critical, in practice "probably a call". Documented in `Wire.swift` as the one thing a sender tells the relay about content; that is now two things. **3. Retention.** `delivered` is set but nothing sweeps. Envelopes accumulate, and their `created_at` is a conversation timeline. `FN_DELETE_ON_ACK` exists and is deliberately OFF because it destroys ciphertext unrecoverably — but "keep everything forever" is not the only alternative to it. A time-based sweep of *delivered* envelopes is a different trade and worth costing. **4. Timing correlation.** Sealed sender hides who sent a message; `created_at` plus delivery order plus the per-device index may not. Worth asking what an operator watching the table for a week could reconstruct. **5. Unsealed fallback.** Sealing depends on holding the peer's delivery key. When that is missing the send falls back to unsealed — and the relay learns the sender. How often that happens in practice is measurable and currently unmeasured. ## Suggested shape Standing ticket. Each pass produces: what is stored, what actually reads it, and whether the feature survives without it. Anything that survives without it should go; anything that does not should appear on `/metadata` in plain words. ⚠️ Freeze in effect — investigate and document, do not remove. Related: #16 (security review), #17 (plaintext inventory).

paulhitt commented

2026-09-01 23:06:34 +00:00

(Migrated from gitlab.com)

mentioned in issue #19

paulhitt commented

2026-09-01 23:11:46 +00:00

(Migrated from gitlab.com)

mentioned in issue #21

paulhitt commented

2026-09-01 23:27:46 +00:00

(Migrated from gitlab.com)

mentioned in issue #28

paulhitt commented

2026-09-04 22:00:24 +00:00

(Migrated from gitlab.com)

Pass done 2026-09-04, deployed and live (!47). Leaving this open — it is standing work, and the point below about drift is the argument for keeping it open rather than the argument for closing it.

Both specific asks: documented, with a correction to the framing

urgent and silent are now on /metadata, each as its own row. 🔴 But neither is stored, and the rows say so in those words. They are not columns on envelopes — they are read to decide the push and gone with the request. The issue called silent "an undocumented disclosure"; it is a disclosure, and it is transient, and documenting it as kept would have been its own inaccuracy. The rows say what the relay learns, that it acts on it, and that it never writes it down.

The wording keeps the trade visible rather than burying it: between them the two flags let the relay tell a call from a message — not who, not about what, but that this one is time-critical. That is the most informative thing a sender chooses to reveal, and the silent row states the alternative (send nothing, let the app decide after decrypting) and why it was not taken (iOS throttles silent pushes, so notifications stop arriving).

⚠️ And the page had drifted — in the safe direction, but drifted

The group-roster row read "every group that exists right now is still the old kind, held in the clear", waiting on the apps. The apps caught up and nobody re-measured. Production today: both live groups carry a sealed roster_blob at roster_version 1, and the plaintext group_members table holds zero live rows.

The page was claiming worse than reality, which is the direction of error you want on a privacy page — but a promise that has drifted is still wrong, and a page nobody re-measures is how the drift goes the other way next time. Corrected, keeping the nuance intact because it did not change: a sealed roster earns "a dump of this database does not reveal your groups", not "the server cannot see who you talk to". Messages are still addressed per device.

Measured while here

46 of 58 queued envelopes are sealed sends storing no sender at all; 12 are identified. So the unsealed path is real traffic rather than a vestige, and from_user/from_device defaulting to 0/'' is sealed sender genuinely working for the majority — but not all — of what is in flight.

For the next pass

The natural checklist is the one this issue already names: anything added to the schema or the wire has to land on /metadata. What would make that reliable rather than remembered is a test that fails when an envelope field exists with no row on the page — the drift above was found by reading, and reading is not a control.

**Pass done 2026-09-04, deployed and live** (!47). Leaving this open — it is standing work, and the point below about drift is the argument for keeping it open rather than the argument for closing it. ### Both specific asks: documented, with a correction to the framing `urgent` and `silent` are now on `/metadata`, each as its own row. 🔴 **But neither is stored, and the rows say so in those words.** They are not columns on `envelopes` — they are read to decide the push and gone with the request. The issue called `silent` "an undocumented disclosure"; it is a disclosure, and it is transient, and documenting it as *kept* would have been its own inaccuracy. The rows say what the relay learns, that it acts on it, and that it never writes it down. The wording keeps the trade visible rather than burying it: between them the two flags let the relay tell a call from a message — not who, not about what, but that this one is time-critical. That is the most informative thing a sender chooses to reveal, and the `silent` row states the alternative (send nothing, let the app decide after decrypting) and why it was not taken (iOS throttles silent pushes, so notifications stop arriving). ### ⚠️ And the page had drifted — in the safe direction, but drifted The group-roster row read *"every group that exists right now is still the old kind, held in the clear"*, waiting on the apps. **The apps caught up and nobody re-measured.** Production today: both live groups carry a sealed `roster_blob` at `roster_version 1`, and the plaintext `group_members` table holds **zero** live rows. The page was claiming worse than reality, which is the direction of error you want on a privacy page — but a promise that has drifted is still wrong, and a page nobody re-measures is how the drift goes the other way next time. Corrected, keeping the nuance intact because it did not change: a sealed roster earns *"a dump of this database does not reveal your groups"*, **not** *"the server cannot see who you talk to"*. Messages are still addressed per device. ### Measured while here **46 of 58 queued envelopes are sealed sends storing no sender at all; 12 are identified.** So the unsealed path is real traffic rather than a vestige, and `from_user`/`from_device` defaulting to `0`/`''` is sealed sender genuinely working for the majority — but not all — of what is in flight. ### For the next pass The natural checklist is the one this issue already names: anything added to the schema or the wire has to land on `/metadata`. What would make that reliable rather than remembered is a test that fails when an envelope field exists with no row on the page — the drift above was found by reading, and reading is not a control.

paulhitt commented

2026-09-07 12:47:51 +00:00

(Migrated from gitlab.com)

mentioned in merge request !75

paulhitt commented

2026-09-07 13:32:10 +00:00

(Migrated from gitlab.com)

mentioned in issue #59

paulhitt commented

2026-09-07 13:32:40 +00:00

(Migrated from gitlab.com)

Pass 2 done and written up in docs/METADATA-PASSES.md (MR !75). Summary of what changed in this ticket's picture:

🔴 New finding, now split out as #59. length(payload) is visible on every row and is not on /metadata. Fourteen envelopes, fourteen distinct lengths, clustering by kind. Padding has to happen sender-side inside the sealed envelope — the relay cannot pad what it cannot read — so it is an e2e + apps change, not a relay one. The ticket carries measured ladder costs and, more usefully, the argument for why total-overhead is the wrong metric to pick a ladder by.

✅ Two bullets here are stale and I have stopped repeating them.

"delivered is set but nothing sweeps" — not true. Retention is an in-process supervised loop, not a cron somebody has to visit (it was a cron once and stopped for eight weeks unnoticed, which is why it is not one now). It logs every cycle:

retention swept [attachments: 0, delivered: 0, devices: 0, held_delivered: 0,
  held_undelivered: 14, oldest_undelivered_days: 2, resets: 0,
  sends_sealed: 0, sends_unsealed: 0, undelivered: 0]

⚠️ Note what that does not prove: delivered: 0 means nothing was old enough to sweep, not that sweeping works. The oldest envelope on the box is 2 days old and undelivered. The deletion path is covered by RetentionTests; production has not exercised it.

"The unsealed fallback is unmeasured" — it is measured twice over. sends_sealed/sends_unsealed appear in the sweep line above, and the table answers directly: 14/14 sealed, from_user = 0 on every live row. ⚠️ Fourteen envelopes from two accounts is not a sample to generalise from — it says the unsealed path is not the common case and nothing about how it behaves at scale.

✅ Already landed: silent and urgent are both on /metadata, each with what it reveals and why the trade was taken.

Timing correlation, answered. An operator with select * from envelopes and patience learns, per recipient device: message count, exact arrival time, size, and order. Enough to infer waking hours, time zone, which two devices are in a conversation, roughly when a call happened, and whether a message was a photo or a sentence. Not enough to read anything, name a sender, or attribute a conversation to an identity not already known.

⚠️ The honest summary: sealed sender does what it claims and less than it sounds like. It removes attribution from the row. It does not remove the pattern, and a long enough window is made of the pattern.

Next pass should start with a real corpus — every number in this one comes from 14 rows.

Pass 2 done and written up in `docs/METADATA-PASSES.md` (MR !75). Summary of what changed in this ticket's picture: **🔴 New finding, now split out as #59.** `length(payload)` is visible on every row and is **not on `/metadata`**. Fourteen envelopes, fourteen distinct lengths, clustering by kind. Padding has to happen sender-side inside the sealed envelope — the relay cannot pad what it cannot read — so it is an e2e + apps change, not a relay one. The ticket carries measured ladder costs and, more usefully, the argument for why total-overhead is the wrong metric to pick a ladder by. **✅ Two bullets here are stale and I have stopped repeating them.** *"`delivered` is set but nothing sweeps"* — not true. Retention is an in-process supervised loop, not a cron somebody has to visit (it was a cron once and stopped for eight weeks unnoticed, which is why it is not one now). It logs every cycle: ``` retention swept [attachments: 0, delivered: 0, devices: 0, held_delivered: 0, held_undelivered: 14, oldest_undelivered_days: 2, resets: 0, sends_sealed: 0, sends_unsealed: 0, undelivered: 0] ``` ⚠️ Note what that does *not* prove: `delivered: 0` means nothing was old enough to sweep, not that sweeping works. The oldest envelope on the box is 2 days old and undelivered. The deletion path is covered by `RetentionTests`; production has not exercised it. *"The unsealed fallback is unmeasured"* — it is measured twice over. `sends_sealed`/`sends_unsealed` appear in the sweep line above, and the table answers directly: **14/14 sealed**, `from_user = 0` on every live row. ⚠️ Fourteen envelopes from two accounts is not a sample to generalise from — it says the unsealed path is not the common case and nothing about how it behaves at scale. **✅ Already landed:** `silent` and `urgent` are both on `/metadata`, each with what it reveals and why the trade was taken. **Timing correlation, answered.** An operator with `select * from envelopes` and patience learns, per recipient device: message count, exact arrival time, size, and order. Enough to infer waking hours, time zone, which two devices are in a conversation, roughly when a call happened, and whether a message was a photo or a sentence. Not enough to read anything, name a sender, or attribute a conversation to an identity not already known. ⚠️ The honest summary: **sealed sender does what it claims and less than it sounds like.** It removes attribution from the row. It does not remove the pattern, and a long enough window is made of the pattern. Next pass should start with a real corpus — every number in this one comes from 14 rows.

paulhitt commented

2026-09-09 21:43:09 +00:00

(Migrated from gitlab.com)

Re-checked one claim in this issue: "delivered is set but nothing sweeps."

Something does sweep. Retention.swift:53 has sweepOnce(), and it deletes on both windows:

.filter(\.$createdAt < deliveredCutoff).delete()     // Retention.swift:65
.filter(\.$createdAt < undeliveredCutoff).delete()   // Retention.swift:73

driven by FN_DELIVERED_RETENTION_DAYS (default 1) and FN_UNDELIVERED_RETENTION_DAYS (default 30), and /metadata publishes both from the running configuration.

Flagging it because this issue is standing work that others will read for a to-do list, and that line reads as an open gap. The rest of the issue stands.

Re-checked one claim in this issue: *"`delivered` is set but nothing sweeps."* **Something does sweep.** `Retention.swift:53` has `sweepOnce()`, and it deletes on both windows: ```swift .filter(\.$createdAt < deliveredCutoff).delete() // Retention.swift:65 .filter(\.$createdAt < undeliveredCutoff).delete() // Retention.swift:73 ``` driven by `FN_DELIVERED_RETENTION_DAYS` (default 1) and `FN_UNDELIVERED_RETENTION_DAYS` (default 30), and `/metadata` publishes both from the running configuration. Flagging it because this issue is standing work that others will read for a to-do list, and that line reads as an open gap. The rest of the issue stands.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: extant2000/flamenet-relay#18