extant2000/flamenet-messenger-android
1
0
Fork 0
Code Issues 7 Pull Requests Actions Packages Projects Releases Wiki Activity

Expired messages no longer leave their plaintext on disk (android#11)

Browse Source 
`attachmentBytes` wrote DECRYPTED bytes to `cacheDir/att-<id>` and nothing
ever removed them. So a disappearing message expired, the bubble went, and
the photo or voice note stayed on disk indefinitely — the exact failure the
feature exists to prevent. Spec §6.11: attachments MUST be purged with the
message that referenced them.

🔴 **A sweep, not a purge at each deletion point, and that changed what the
fix had to be.** FOUR paths drop a row — message expiry, story expiry,
delete-for-me and clear-thread — so threading "here is what I removed" out
of each would still miss anything a future path drops, and would never
reach the orphans already on disk. Asking instead "what is still
referenced?" is complete by construction and self-healing.

🔴 **Refs are found by SHAPE.** An attachment reference serialises with
`id`, `key` and `digest` together and nothing else in the vault has all
three, so `attachmentIdsInUse` walks every record and matches on that. A
hand-written list of stores is wrong the day somebody adds a store and
does not update it — and the sweep would then delete attachments still in
use, which is far worse than the leak.

⚠️ **That is not hypothetical: enumerating by hand gets it wrong today.**
The obvious sources are history and stories. The vault ALSO holds refs in
the story archive, in per-contact avatars and in this account's own
avatar. A sweep built on the obvious two would have deleted every avatar
on the device the first time it ran, and blanked the user's own story
archive — which is the one thing that archive exists to prevent.

Three more safety properties, each with a test that fails without it:
fails closed when the in-use set cannot be built (a partial answer is what
turns a cleanup into data loss); a grace period so a sweep racing a fetch
cannot delete bytes on their way to the screen; and a throttle inside the
function, since it reads every vault record and is called from the poll
loop.

⚠️ The cost is that a file survives to the next sweep rather than going at
the instant of expiry — named in the code rather than left implied.

17 tests. Every guard proved by breaking it. ⚠️ One did NOT hold at first:
removing the `/` -> `_` filename mangling broke nothing, because every id
in the suite was slash-free — the guard was untested by the tests meant to
be guarding it. Added the case; it now fails as it should.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Paul Hitt
2026-09-04 18:29:30 -04:00
co-authored by Claude Opus 5
parent 3caaa2172d
commit 46fb23dbe1
5 changed files with 512 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
engine/src/main/kotlin/io/flamenet/e2e/Vault.kt
+59
View File
@@ -1665,6 +1665,65 @@ class Vault(private val storage: Storage, private val key: SecretKey) {
"${m.optBoolean("mine")}|${m.optLong("at")}|${m.optString("text")}|" +
(m.optJSONObject("att")?.optString("id") ?: "")
/**
* Every attachment id this device still has a reason to hold bytes for.
*
* 🔴 **Refs are found by SHAPE, not by looking in the places we remember.**
* An attachment reference serialises with `id`, `key` and `digest`
* together, and nothing else in the vault has all three — so this walks
* every record and matches on that, wherever a ref happens to sit. The
* alternative, a hand-written list of stores, is wrong the day somebody
* adds a store and does not update the list: the sweep that uses this would
* then delete attachments that are still in use, which is a far worse
* failure than the one it exists to fix.
*
* ⚠️ It really is every record, and that is not paranoia. Enumerating by
* hand, the obvious sources are message history and stories — and the vault
* also keeps refs in the story ARCHIVE, in per-contact avatars and in this
* account's own avatar. A sweep built on the obvious two would have deleted
* every avatar on the device the first time it ran.
*
* ⚠️ **Throws rather than returning a partial answer.** A record that
* cannot be opened means this set is incomplete, and an incomplete set of
* "still in use" is exactly the input that turns a cleanup into data loss.
* The caller must treat a throw as "do not sweep".
*/
fun attachmentIdsInUse(): Set<String> {
val found = mutableSetOf<String>()
for (name in storage.names()) {
val record = readJson(name) ?: continue
collectAttachmentIds(record, found)
}
return found
}
/**
* ⚠️ Depth-bounded. The walk is over data this device wrote, but a record
* is a file on disk and a cycle or a pathological nesting must not become
* a stack overflow inside a cleanup.
*/
private fun collectAttachmentIds(node: Any?, into: MutableSet<String>, depth: Int = 0) {
if (depth > 24) return
when (node) {
is JSONObject -> {
val id = node.optString("id")
// The three together are what makes it a reference to bytes.
// A sticker carries `pack` + `id`; a story SET carries
// `id` + `i` + `n` and its id names the sequence, not a blob.
// Neither has a key or a digest, and neither is matched here.
if (id.isNotEmpty() &&
node.optString("key").isNotEmpty() &&
node.optString("digest").isNotEmpty()
) {
into += id
}
for (key in node.keys()) collectAttachmentIds(node.opt(key), into, depth + 1)
}
is JSONArray ->
for (i in 0 until node.length()) collectAttachmentIds(node.opt(i), into, depth + 1)
}
}
/** Every thread this device holds, group or person. */
fun threads(): List<String> =
storage.names().filter { it.startsWith(HISTORY_PREFIX) }
engine/src/test/kotlin/io/flamenet/e2e/AttachmentsInUseTest.kt
+155
View File
@@ -0,0 +1,155 @@
package io.flamenet.e2e
import org.json.JSONObject
import javax.crypto.KeyGenerator
import javax.crypto.SecretKey
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertFailsWith
import kotlin.test.assertFalse
import kotlin.test.assertTrue
/**
* Which attachment ids the device still has a reason to hold bytes for.
*
* 🔴 **This set is the input to a DELETE, so the dangerous error is the one
* that omits something.** A missing id means a cleanup removes a file that is
* still in use — a worse failure than the leak it was written to fix, because
* the leak only wastes space and this loses somebody's photo.
*
* So the cases below are mostly about the stores an enumeration would forget.
*/
class AttachmentsInUseTest {
private class MapStorage : Vault.Storage {
val records = LinkedHashMap<String, ByteArray>()
override fun read(name: String) = records[name]
override fun write(name: String, bytes: ByteArray) { records[name] = bytes }
override fun delete(name: String) { records.remove(name) }
override fun names() = records.keys.toList()
}
private fun key(): SecretKey = KeyGenerator.getInstance("AES").apply { init(256) }.generateKey()
private fun ref(id: String) = Attachment.Ref(
kind = Attachment.IMAGE, id = id,
key = ByteArray(32) { 7 }, digest = ByteArray(32) { 9 },
bytes = 100, mime = "image/jpeg", width = 2, height = 2,
)
private fun story(id: String, attId: String, expiresAt: Double) = Vault.Story(
id = id, authorId = 1L, mine = true, ref = ref(attId),
caption = "", postedAt = 0L, seen = false, expiresAt = expiresAt,
)
// ---- The stores an enumeration forgets --------------------------------
@Test
fun `a message's attachment counts as in use`() {
val storage = MapStorage(); val k = key()
val v = Vault(storage, k)
v.append("user:1", JSONObject().put("text", "").put("at", 1L)
.put("att", ref("in-a-message").toJson()))
assertTrue("in-a-message" in Vault(storage, k).attachmentIdsInUse())
}
@Test
fun `a live story's attachment counts`() {
val storage = MapStorage(); val k = key()
Vault(storage, k).saveStories(listOf(story("s1", "in-a-story", 4_000_000_000.0)), 1.0)
assertTrue("in-a-story" in Vault(storage, k).attachmentIdsInUse())
}
/**
* 🔴 **The archive is the case that makes a naive sweep destructive.**
* An expired story is dropped from the live set AND folded into the private
* archive, which keeps its ref. Purging on expiry without looking here
* would leave somebody's own archive full of blank pictures — the exact
* thing the archive exists to prevent.
*/
@Test
fun `an ARCHIVED story's attachment still counts, long after it expired`() {
val storage = MapStorage(); val k = key()
val v = Vault(storage, k)
// Expired: gone from the live set, kept in the archive.
v.saveStories(listOf(story("s1", "expired-blob", 1.0)), 1_000_000.0)
v.saveStoryArchive(listOf(story("s1", "expired-blob", 1.0)))
val inUse = Vault(storage, k).attachmentIdsInUse()
assertTrue("expired-blob" in inUse,
"an archived story's attachment was reported unused — the archive would go blank")
}
/**
* 🔴 **Avatars are attachments too.** They are fetched and cached by the
* same code path, and they live in a store nobody thinks of when listing
* "places messages are". A sweep that enumerated history and stories would
* have deleted every avatar on the device the first time it ran.
*/
@Test
fun `avatars count — both a contact's and this account's own`() {
val storage = MapStorage(); val k = key()
val v = Vault(storage, k)
v.saveAvatar(42L, ref("their-face"))
v.saveMyAvatar(ref("my-face"))
val inUse = Vault(storage, k).attachmentIdsInUse()
assertTrue("their-face" in inUse, "a contact's avatar was reported unused")
assertTrue("my-face" in inUse, "this account's own avatar was reported unused")
}
// ---- Things that look like refs and are not ---------------------------
/**
* ⚠️ A sticker carries `pack` + `id`, and a story SET carries `id` + `i` +
* `n` where the id names the SEQUENCE, not a blob. Matching on `id` alone
* would pull both in — harmless for a set, but it would keep files alive
* forever under ids that name nothing.
*/
@Test
fun `ids that are not attachment references are not collected`() {
val storage = MapStorage(); val k = key()
val v = Vault(storage, k)
v.append("user:1", JSONObject().put("text", "").put("at", 1L)
.put("sticker", JSONObject().put("pack", "p").put("id", "sticker-id"))
.put("set", JSONObject().put("id", "sequence-id").put("i", 0).put("n", 3)))
val inUse = Vault(storage, k).attachmentIdsInUse()
assertFalse("sticker-id" in inUse, "a sticker id was mistaken for an attachment")
assertFalse("sequence-id" in inUse, "a story SET id was mistaken for an attachment")
}
/** The discriminator is all three fields, so two of them is not enough. */
@Test
fun `an object with an id and a key but no digest is not a reference`() {
val storage = MapStorage(); val k = key()
Vault(storage, k).append("user:1", JSONObject().put("at", 1L)
.put("odd", JSONObject().put("id", "two-of-three").put("key", "x")))
assertFalse("two-of-three" in Vault(storage, k).attachmentIdsInUse())
}
// ---- Failing safe -----------------------------------------------------
/**
* 🔴 **An unreadable record must throw, not be skipped.** Skipping it
* returns a set that is missing whatever it held, and a caller that deletes
* everything outside that set would then delete live attachments. The
* cleanup has to refuse to run rather than run on a partial view.
*/
@Test
fun `a record that cannot be opened stops the whole answer`() {
val storage = MapStorage(); val k = key()
Vault(storage, k).append("user:1", JSONObject().put("at", 1L)
.put("att", ref("real").toJson()))
storage.records["history/user:2"] = byteArrayOf(1, 2, 3, 4, 5, 6, 7, 8,
9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26,
27, 28, 29, 30)
assertFailsWith<Vault.Corrupt> { Vault(storage, k).attachmentIdsInUse() }
}
@Test
fun `an empty vault reports nothing in use rather than failing`() {
assertEquals(emptySet(), Vault(MapStorage(), key()).attachmentIdsInUse())
}
}