Expired messages no longer leave their plaintext on disk (android#11)

`attachmentBytes` wrote DECRYPTED bytes to `cacheDir/att-<id>` and nothing ever removed them. So a disappearing message expired, the bubble went, and the photo or voice note stayed on disk indefinitely — the exact failure the feature exists to prevent. Spec §6.11: attachments MUST be purged with the message that referenced them. 🔴 **A sweep, not a purge at each deletion point, and that changed what the fix had to be.** FOUR paths drop a row — message expiry, story expiry, delete-for-me and clear-thread — so threading "here is what I removed" out of each would still miss anything a future path drops, and would never reach the orphans already on disk. Asking instead "what is still referenced?" is complete by construction and self-healing. 🔴 **Refs are found by SHAPE.** An attachment reference serialises with `id`, `key` and `digest` together and nothing else in the vault has all three, so `attachmentIdsInUse` walks every record and matches on that. A hand-written list of stores is wrong the day somebody adds a store and does not update it — and the sweep would then delete attachments still in use, which is far worse than the leak. ⚠️ **That is not hypothetical: enumerating by hand gets it wrong today.** The obvious sources are history and stories. The vault ALSO holds refs in the story archive, in per-contact avatars and in this account's own avatar. A sweep built on the obvious two would have deleted every avatar on the device the first time it ran, and blanked the user's own story archive — which is the one thing that archive exists to prevent. Three more safety properties, each with a test that fails without it: fails closed when the in-use set cannot be built (a partial answer is what turns a cleanup into data loss); a grace period so a sweep racing a fetch cannot delete bytes on their way to the screen; and a throttle inside the function, since it reads every vault record and is called from the poll loop. ⚠️ The cost is that a file survives to the next sweep rather than going at the instant of expiry — named in the code rather than left implied. 17 tests. Every guard proved by breaking it. ⚠️ One did NOT hold at first: removing the `/` -> `_` filename mangling broke nothing, because every id in the suite was slash-free — the guard was untested by the tests meant to be guarding it. Added the case; it now fails as it should. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-04 18:29:30 -04:00
parent 3caaa2172d
commit 46fb23dbe1
5 changed files with 512 additions and 1 deletions
@@ -0,0 +1,155 @@
+package io.flamenet.e2e
+
+import org.json.JSONObject
+import javax.crypto.KeyGenerator
+import javax.crypto.SecretKey
+import kotlin.test.Test
+import kotlin.test.assertEquals
+import kotlin.test.assertFailsWith
+import kotlin.test.assertFalse
+import kotlin.test.assertTrue
+
+/**
+ * Which attachment ids the device still has a reason to hold bytes for.
+ *
+ * 🔴 **This set is the input to a DELETE, so the dangerous error is the one
+ * that omits something.** A missing id means a cleanup removes a file that is
+ * still in use — a worse failure than the leak it was written to fix, because
+ * the leak only wastes space and this loses somebody's photo.
+ *
+ * So the cases below are mostly about the stores an enumeration would forget.
+ */
+class AttachmentsInUseTest {
+
+    private class MapStorage : Vault.Storage {
+        val records = LinkedHashMap<String, ByteArray>()
+        override fun read(name: String) = records[name]
+        override fun write(name: String, bytes: ByteArray) { records[name] = bytes }
+        override fun delete(name: String) { records.remove(name) }
+        override fun names() = records.keys.toList()
+    }
+
+    private fun key(): SecretKey = KeyGenerator.getInstance("AES").apply { init(256) }.generateKey()
+
+    private fun ref(id: String) = Attachment.Ref(
+        kind = Attachment.IMAGE, id = id,
+        key = ByteArray(32) { 7 }, digest = ByteArray(32) { 9 },
+        bytes = 100, mime = "image/jpeg", width = 2, height = 2,
+    )
+
+    private fun story(id: String, attId: String, expiresAt: Double) = Vault.Story(
+        id = id, authorId = 1L, mine = true, ref = ref(attId),
+        caption = "", postedAt = 0L, seen = false, expiresAt = expiresAt,
+    )
+
+    // ---- The stores an enumeration forgets --------------------------------
+
+    @Test
+    fun `a message's attachment counts as in use`() {
+        val storage = MapStorage(); val k = key()
+        val v = Vault(storage, k)
+        v.append("user:1", JSONObject().put("text", "").put("at", 1L)
+            .put("att", ref("in-a-message").toJson()))
+        assertTrue("in-a-message" in Vault(storage, k).attachmentIdsInUse())
+    }
+
+    @Test
+    fun `a live story's attachment counts`() {
+        val storage = MapStorage(); val k = key()
+        Vault(storage, k).saveStories(listOf(story("s1", "in-a-story", 4_000_000_000.0)), 1.0)
+        assertTrue("in-a-story" in Vault(storage, k).attachmentIdsInUse())
+    }
+
+    /**
+     * 🔴 **The archive is the case that makes a naive sweep destructive.**
+     * An expired story is dropped from the live set AND folded into the private
+     * archive, which keeps its ref. Purging on expiry without looking here
+     * would leave somebody's own archive full of blank pictures — the exact
+     * thing the archive exists to prevent.
+     */
+    @Test
+    fun `an ARCHIVED story's attachment still counts, long after it expired`() {
+        val storage = MapStorage(); val k = key()
+        val v = Vault(storage, k)
+        // Expired: gone from the live set, kept in the archive.
+        v.saveStories(listOf(story("s1", "expired-blob", 1.0)), 1_000_000.0)
+        v.saveStoryArchive(listOf(story("s1", "expired-blob", 1.0)))
+
+        val inUse = Vault(storage, k).attachmentIdsInUse()
+        assertTrue("expired-blob" in inUse,
+            "an archived story's attachment was reported unused — the archive would go blank")
+    }
+
+    /**
+     * 🔴 **Avatars are attachments too.** They are fetched and cached by the
+     * same code path, and they live in a store nobody thinks of when listing
+     * "places messages are". A sweep that enumerated history and stories would
+     * have deleted every avatar on the device the first time it ran.
+     */
+    @Test
+    fun `avatars count — both a contact's and this account's own`() {
+        val storage = MapStorage(); val k = key()
+        val v = Vault(storage, k)
+        v.saveAvatar(42L, ref("their-face"))
+        v.saveMyAvatar(ref("my-face"))
+
+        val inUse = Vault(storage, k).attachmentIdsInUse()
+        assertTrue("their-face" in inUse, "a contact's avatar was reported unused")
+        assertTrue("my-face" in inUse, "this account's own avatar was reported unused")
+    }
+
+    // ---- Things that look like refs and are not ---------------------------
+
+    /**
+     * ⚠️ A sticker carries `pack` + `id`, and a story SET carries `id` + `i` +
+     * `n` where the id names the SEQUENCE, not a blob. Matching on `id` alone
+     * would pull both in — harmless for a set, but it would keep files alive
+     * forever under ids that name nothing.
+     */
+    @Test
+    fun `ids that are not attachment references are not collected`() {
+        val storage = MapStorage(); val k = key()
+        val v = Vault(storage, k)
+        v.append("user:1", JSONObject().put("text", "").put("at", 1L)
+            .put("sticker", JSONObject().put("pack", "p").put("id", "sticker-id"))
+            .put("set", JSONObject().put("id", "sequence-id").put("i", 0).put("n", 3)))
+
+        val inUse = Vault(storage, k).attachmentIdsInUse()
+        assertFalse("sticker-id" in inUse, "a sticker id was mistaken for an attachment")
+        assertFalse("sequence-id" in inUse, "a story SET id was mistaken for an attachment")
+    }
+
+    /** The discriminator is all three fields, so two of them is not enough. */
+    @Test
+    fun `an object with an id and a key but no digest is not a reference`() {
+        val storage = MapStorage(); val k = key()
+        Vault(storage, k).append("user:1", JSONObject().put("at", 1L)
+            .put("odd", JSONObject().put("id", "two-of-three").put("key", "x")))
+        assertFalse("two-of-three" in Vault(storage, k).attachmentIdsInUse())
+    }
+
+    // ---- Failing safe -----------------------------------------------------
+
+    /**
+     * 🔴 **An unreadable record must throw, not be skipped.** Skipping it
+     * returns a set that is missing whatever it held, and a caller that deletes
+     * everything outside that set would then delete live attachments. The
+     * cleanup has to refuse to run rather than run on a partial view.
+     */
+    @Test
+    fun `a record that cannot be opened stops the whole answer`() {
+        val storage = MapStorage(); val k = key()
+        Vault(storage, k).append("user:1", JSONObject().put("at", 1L)
+            .put("att", ref("real").toJson()))
+        storage.records["history/user:2"] = byteArrayOf(1, 2, 3, 4, 5, 6, 7, 8,
+            9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26,
+            27, 28, 29, 30)
+
+        assertFailsWith<Vault.Corrupt> { Vault(storage, k).attachmentIdsInUse() }
+    }
+
+    @Test
+    fun `an empty vault reports nothing in use rather than failing`() {
+        assertEquals(emptySet(), Vault(MapStorage(), key()).attachmentIdsInUse())
+    }
+}