Android writes and reads the shared backup format
Section 16 pins the envelope and says nothing about what is sealed, so each app sealed its own storage verbatim and stamped the file with the platform that wrote it. Each then refused the other's, in as many words: "import it on a device of the same kind". Somebody moving between an Android phone and an iPhone lost every conversation they had. The iPhone half landed first. This is the other half. `BackupPayload` translates rather than dumps, which is the entire point. This app stores a message under the keys its wire format uses (`secs`, `fwd`, `rx`), keeps one-to-one and group threads in one map under prefixed ids, hangs the attachment kind beside `att` rather than inside it, abbreviates the sizes, and holds expiries in unix seconds. The shared shape spells all of that out and counts in milliseconds. Exporting storage as-is is what made the two formats incompatible in the first place, so a copy would have reproduced the bug with more steps. Federated threads are the one thing that does not fit. This app keys them by address and the iPhone's history is keyed by numeric user id, with a decoder that skips anything else, so an address left in `threads` would be dropped in silence. They travel under `addrThreads` instead: an iPhone ignores the key, an Android phone gets them back, and nothing disappears without somebody having decided it should. Import tells the two formats apart by SHAPE, not by file name, because a name is whatever the person saving it chose. Section 16 announces itself with `alg` and `ct`; the older export announces itself with `magic`. A wrong passphrase on a section 16 file reports a wrong passphrase and does NOT fall through to the legacy reader, which would have answered "that file is not a Flamenet history export" for a file that is one and sent somebody hunting for corruption instead of a typo. Nothing writes the old format any more, and everything still reads it. Seventeen tests: the shared side against the key names the iPhone writes, the round trip back into this app's own storage shape, the unit conversions in both directions, and a payload transcribed from the iPhone's encoder. That last one is the weak point and the test says so: it proves this app reads the shape the iPhone is BELIEVED to write. Only a vector both apps are built against proves they agree, and `backup-v1.json` pins the envelope alone, which is exactly how the two shapes drifted apart. Full suite: 1271 tests, no failures. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Paul Hitt
co-authored by
Claude Opus 5
Claude Opus 5
parent
3bcb7d5900
commit
5a5e41bfa2
@@ -0,0 +1,379 @@
|
||||
package io.flamenet.e2e
|
||||
|
||||
import org.json.JSONArray
|
||||
import org.json.JSONObject
|
||||
|
||||
/**
|
||||
* This device's history, translated into the shape both apps agree on.
|
||||
*
|
||||
* Section 16 pins the envelope: the KDF, the canonical header used as AAD and
|
||||
* the AES-GCM sealing. It says nothing about what is sealed. Until now each app
|
||||
* sealed its own storage verbatim and stamped the file with the platform that
|
||||
* wrote it, and each then refused the other's, in as many words: "import it on
|
||||
* a device of the same kind". Somebody moving between an Android phone and an
|
||||
* iPhone lost every conversation they had.
|
||||
*
|
||||
* The iPhone half of the shared shape landed first. This is the other half.
|
||||
*
|
||||
* It is a translation and not a dump, which is the whole point. This app stores
|
||||
* a message under the keys its wire format uses (`secs`, `fwd`, `rx`), keeps
|
||||
* one-to-one and group threads in a single map under prefixed ids, and holds
|
||||
* expiries in unix seconds. The shared shape spells those out, separates the
|
||||
* two kinds of thread and counts in milliseconds. Exporting the storage as-is
|
||||
* is exactly what made the two formats incompatible in the first place.
|
||||
*
|
||||
* Three rules hold it together.
|
||||
*
|
||||
* **Times are epoch milliseconds, as integers.** This app already keeps `at` in
|
||||
* millis, so messages pass straight through; expiries do not, and are converted
|
||||
* at the boundary. A backup that disagreed about the unit would restore every
|
||||
* story at the wrong time without ever failing.
|
||||
*
|
||||
* **A key this file does not name is not carried.** Both sides build a fresh
|
||||
* object rather than copying, so adding a field to storage and expecting it to
|
||||
* appear in a backup does not work. `BackupPayloadTest` reads this source and
|
||||
* fails if a known key stops being mentioned.
|
||||
*
|
||||
* **What the other app cannot represent is kept apart rather than faked.**
|
||||
* Federated threads are the case that matters: this app keys them by address,
|
||||
* and the iPhone's history has no room for one, so they travel under their own
|
||||
* key. An iPhone ignores it and an Android phone gets them back.
|
||||
*/
|
||||
object BackupPayload {
|
||||
const val VERSION = 1
|
||||
|
||||
/** What a section 16 backup is called on disk. Shared with the iPhone. */
|
||||
const val FILE_EXTENSION = "fnbackup"
|
||||
|
||||
private const val GROUP_PREFIX = "group:"
|
||||
private const val ADDRESS_PREFIX = "addr:"
|
||||
|
||||
/** Federated threads, which the iPhone has nowhere to put. */
|
||||
private const val ADDRESS_THREADS = "addrThreads"
|
||||
|
||||
// -- Writing -------------------------------------------------------------.
|
||||
|
||||
/** Turn [internal], as [Vault.exportPayload] produces it, into the shared shape. */
|
||||
fun toShared(internal: JSONObject): JSONObject {
|
||||
val root = JSONObject().put("v", VERSION)
|
||||
|
||||
val threads = JSONObject()
|
||||
val groups = JSONObject()
|
||||
val addressed = JSONObject()
|
||||
internal.optJSONObject("threads")?.let { all ->
|
||||
for (tid in all.keys()) {
|
||||
val messages = all.optJSONArray(tid) ?: continue
|
||||
when {
|
||||
tid.startsWith(GROUP_PREFIX) -> {
|
||||
groups.put(tid.removePrefix(GROUP_PREFIX), mapArray(messages, ::messageToShared))
|
||||
}
|
||||
|
||||
tid.startsWith(ADDRESS_PREFIX) -> {
|
||||
addressed.put(tid, mapArray(messages, ::messageToShared))
|
||||
}
|
||||
|
||||
else -> {
|
||||
threads.put(tid, mapArray(messages, ::messageToShared))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if (threads.length() > 0) root.put("threads", threads)
|
||||
if (groups.length() > 0) root.put("groups", groups)
|
||||
if (addressed.length() > 0) root.put(ADDRESS_THREADS, addressed)
|
||||
|
||||
internal.optJSONArray("callLog")?.takeIf { it.length() > 0 }?.let { log ->
|
||||
root.put(
|
||||
"calls",
|
||||
mapArray(log) { c ->
|
||||
JSONObject()
|
||||
.put("peer", c.optLong("peer"))
|
||||
.put("at", c.optLong("at"))
|
||||
.put("in", c.optBoolean("in"))
|
||||
// The only rename here: this app writes `ok`.
|
||||
.put("connected", c.optBoolean("ok"))
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
// Stories are one flat list here and are keyed by author over there.
|
||||
internal.optJSONObject("stories")?.optJSONArray("items")?.let { items ->
|
||||
val byAuthor = JSONObject()
|
||||
for (i in 0 until items.length()) {
|
||||
val s = items.optJSONObject(i) ?: continue
|
||||
val author = s.optLong("author").toString()
|
||||
val list = byAuthor.optJSONArray(author) ?: JSONArray().also { byAuthor.put(author, it) }
|
||||
list.put(storyToShared(s))
|
||||
}
|
||||
if (byAuthor.length() > 0) root.put("stories", byAuthor)
|
||||
}
|
||||
|
||||
internal.optJSONObject("roster")?.let { r ->
|
||||
r.optJSONArray("items")?.takeIf { it.length() > 0 }?.let { root.put("roster", it) }
|
||||
if (r.has("offServer")) root.put("rosterOffServer", r.optBoolean("offServer"))
|
||||
}
|
||||
|
||||
internal.optJSONObject("groupNames")?.takeIf { it.length() > 0 }?.let { root.put("groupNames", it) }
|
||||
|
||||
internal.optJSONObject("groupSecrets")?.takeIf { it.length() > 0 }?.let { all ->
|
||||
val out = JSONObject()
|
||||
for (gid in all.keys()) {
|
||||
val s = all.optJSONObject(gid) ?: continue
|
||||
val one =
|
||||
JSONObject()
|
||||
.put("tag", s.optString("tag"))
|
||||
.put("key", s.optString("key"))
|
||||
// The relay refuses a roster rewrite whose version does
|
||||
// not increase, so a restore that reset it would lock
|
||||
// the group out of its own next rekey.
|
||||
.put("version", s.optInt("version"))
|
||||
s.optString("owner").takeIf { it.isNotEmpty() }?.let { one.put("ownerPrivate", it) }
|
||||
out.put(gid, one)
|
||||
}
|
||||
root.put("groupSecrets", out)
|
||||
}
|
||||
|
||||
return root
|
||||
}
|
||||
|
||||
private fun messageToShared(m: JSONObject): JSONObject {
|
||||
val out =
|
||||
JSONObject()
|
||||
.put("mine", m.optBoolean("mine"))
|
||||
.put("at", m.optLong("at"))
|
||||
.put("text", m.optString("text"))
|
||||
m.optString("mid").takeIf { it.isNotEmpty() }?.let { out.put("mid", it) }
|
||||
// The author rides along on one-to-one threads too. The iPhone reads it
|
||||
// only for groups and ignores it elsewhere, which costs nothing and
|
||||
// keeps an Android-to-Android restore whole.
|
||||
if (m.has("author")) out.put("author", m.optLong("author"))
|
||||
m.optJSONObject("att")?.let {
|
||||
out.put("att", attachmentToShared(it, m.optString("kind", Attachment.IMAGE)))
|
||||
}
|
||||
m.optJSONObject("rx")?.takeIf { it.length() > 0 }?.let { out.put("reactions", it) }
|
||||
m.optLong("edited", 0L).takeIf { it > 0L }?.let { out.put("edited", it) }
|
||||
// Both shapes, for the reason `Content.isForwarded` accepts both:
|
||||
// `optBoolean` does not coerce a number, so a `"fwd":1` from another
|
||||
// engine would read as false.
|
||||
if (m.optBoolean("fwd", false) || m.optInt("fwd", 0) != 0) out.put("forwarded", true)
|
||||
m.optInt("secs", 0).takeIf { it > 0 }?.let { out.put("lifetime", it) }
|
||||
m.optJSONObject("re")?.let { out.put("re", it) }
|
||||
m.optJSONObject("stk")?.let { out.put("stk", it) }
|
||||
m.optJSONObject("lnk")?.let { out.put("lnk", it) }
|
||||
m.optString("delivery").takeIf { it.isNotEmpty() }?.let { out.put("delivery", it.lowercase()) }
|
||||
return out
|
||||
}
|
||||
|
||||
/**
|
||||
* The kind lives beside `att` here and inside it over there, and the sizes
|
||||
* and duration are spelled out rather than abbreviated.
|
||||
*/
|
||||
private fun attachmentToShared(
|
||||
a: JSONObject,
|
||||
kind: String,
|
||||
): JSONObject {
|
||||
val out =
|
||||
JSONObject()
|
||||
.put("kind", kind)
|
||||
.put("id", a.optString("id"))
|
||||
.put("key", a.optString("key"))
|
||||
.put("digest", a.optString("digest"))
|
||||
.put("bytes", a.optInt("bytes"))
|
||||
.put("mime", a.optString("mime"))
|
||||
.put("width", a.optInt("w"))
|
||||
.put("height", a.optInt("h"))
|
||||
a.optDouble("dur", 0.0).takeIf { it > 0.0 }?.let { out.put("duration", it) }
|
||||
// Seconds here, milliseconds there.
|
||||
a.optDouble("exp", 0.0).takeIf { it > 0.0 }?.let { out.put("expires", secondsToMillis(it)) }
|
||||
return out
|
||||
}
|
||||
|
||||
private fun storyToShared(s: JSONObject): JSONObject {
|
||||
val out =
|
||||
JSONObject()
|
||||
.put("id", s.opt("id"))
|
||||
.put("author", s.optLong("author"))
|
||||
.put("mine", s.optBoolean("mine"))
|
||||
.put("caption", s.optString("caption"))
|
||||
.put("at", s.optLong("at"))
|
||||
s.optDouble("expires", 0.0).takeIf { it > 0.0 }?.let { out.put("expires", secondsToMillis(it)) }
|
||||
s.optJSONObject("ref")?.let { out.put("att", attachmentToShared(it, Attachment.STORY)) }
|
||||
s.optString("gid").takeIf { it.isNotEmpty() }?.let { out.put("gid", it) }
|
||||
s.optString("stream").takeIf { it.isNotEmpty() }?.let { out.put("stream", it) }
|
||||
s.optString("key").takeIf { it.isNotEmpty() }?.let { out.put("key", it) }
|
||||
s.optJSONObject("set")?.let { out.put("set", it) }
|
||||
// The iPhone has no unread mark on a story, so this is ours alone. It
|
||||
// travels anyway: coming back here it is the difference between a strip
|
||||
// that remembers what you watched and one that shows everything as new.
|
||||
if (s.optBoolean("seen", false)) out.put("seen", true)
|
||||
return out
|
||||
}
|
||||
|
||||
// -- Reading -------------------------------------------------------------.
|
||||
|
||||
class NewerBackup(
|
||||
val version: Int,
|
||||
) : Exception("This backup was written by a newer version of the app.")
|
||||
|
||||
/** Turn a shared payload back into what [Vault.mergeImportedPayload] expects. */
|
||||
fun fromShared(shared: JSONObject): JSONObject {
|
||||
val version = shared.optInt("v", 0)
|
||||
if (version > VERSION) throw NewerBackup(version)
|
||||
|
||||
val threads = JSONObject()
|
||||
shared.optJSONObject("threads")?.let { all ->
|
||||
for (tid in all.keys()) {
|
||||
all.optJSONArray(tid)?.let { threads.put(tid, mapArray(it, ::messageFromShared)) }
|
||||
}
|
||||
}
|
||||
shared.optJSONObject("groups")?.let { all ->
|
||||
for (gid in all.keys()) {
|
||||
all.optJSONArray(gid)?.let {
|
||||
threads.put(GROUP_PREFIX + gid, mapArray(it, ::messageFromShared))
|
||||
}
|
||||
}
|
||||
}
|
||||
shared.optJSONObject(ADDRESS_THREADS)?.let { all ->
|
||||
// Already prefixed when written, so they go back under their own id.
|
||||
for (tid in all.keys()) {
|
||||
all.optJSONArray(tid)?.let { threads.put(tid, mapArray(it, ::messageFromShared)) }
|
||||
}
|
||||
}
|
||||
|
||||
val out = JSONObject().put("threads", threads)
|
||||
|
||||
shared.optJSONArray("calls")?.takeIf { it.length() > 0 }?.let { calls ->
|
||||
out.put(
|
||||
"callLog",
|
||||
mapArray(calls) { c ->
|
||||
JSONObject()
|
||||
.put("peer", c.optLong("peer"))
|
||||
.put("at", c.optLong("at"))
|
||||
.put("in", c.optBoolean("in"))
|
||||
.put("ok", c.optBoolean("connected"))
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
shared.optJSONObject("stories")?.let { byAuthor ->
|
||||
val items = JSONArray()
|
||||
for (author in byAuthor.keys()) {
|
||||
val list = byAuthor.optJSONArray(author) ?: continue
|
||||
for (i in 0 until list.length()) {
|
||||
val s = list.optJSONObject(i) ?: continue
|
||||
items.put(storyFromShared(s, author.toLongOrNull() ?: 0L))
|
||||
}
|
||||
}
|
||||
if (items.length() > 0) out.put("stories", JSONObject().put("items", items))
|
||||
}
|
||||
|
||||
shared.optJSONArray("roster")?.takeIf { it.length() > 0 }?.let {
|
||||
out.put(
|
||||
"roster",
|
||||
JSONObject().put("items", it).put("offServer", shared.optBoolean("rosterOffServer", false)),
|
||||
)
|
||||
}
|
||||
|
||||
shared.optJSONObject("groupNames")?.takeIf { it.length() > 0 }?.let { out.put("groupNames", it) }
|
||||
|
||||
shared.optJSONObject("groupSecrets")?.takeIf { it.length() > 0 }?.let { all ->
|
||||
val secrets = JSONObject()
|
||||
for (gid in all.keys()) {
|
||||
val s = all.optJSONObject(gid) ?: continue
|
||||
val one =
|
||||
JSONObject()
|
||||
.put("tag", s.optString("tag"))
|
||||
.put("key", s.optString("key"))
|
||||
.put("version", s.optInt("version"))
|
||||
s.optString("ownerPrivate").takeIf { it.isNotEmpty() }?.let { one.put("owner", it) }
|
||||
secrets.put(gid, one)
|
||||
}
|
||||
out.put("groupSecrets", secrets)
|
||||
}
|
||||
|
||||
return out
|
||||
}
|
||||
|
||||
private fun messageFromShared(m: JSONObject): JSONObject {
|
||||
val out =
|
||||
JSONObject()
|
||||
.put("mine", m.optBoolean("mine"))
|
||||
.put("at", m.optLong("at"))
|
||||
.put("text", m.optString("text"))
|
||||
.put("author", m.optLong("author"))
|
||||
m.optString("mid").takeIf { it.isNotEmpty() }?.let { out.put("mid", it) }
|
||||
m.optJSONObject("att")?.let { att ->
|
||||
out.put("kind", att.optString("kind", Attachment.IMAGE))
|
||||
out.put("att", attachmentFromShared(att))
|
||||
}
|
||||
m.optJSONObject("reactions")?.takeIf { it.length() > 0 }?.let { out.put("rx", it) }
|
||||
m.optLong("edited", 0L).takeIf { it > 0L }?.let { out.put("edited", it) }
|
||||
if (m.optBoolean("forwarded", false) || m.optInt("forwarded", 0) != 0) out.put("fwd", true)
|
||||
m.optInt("lifetime", 0).takeIf { it > 0 }?.let { out.put("secs", it) }
|
||||
m.optJSONObject("re")?.let { out.put("re", it) }
|
||||
m.optJSONObject("stk")?.let { out.put("stk", it) }
|
||||
m.optJSONObject("lnk")?.let { out.put("lnk", it) }
|
||||
// Stored upper case here, lower case on the wire and in the file.
|
||||
m.optString("delivery").takeIf { it.isNotEmpty() }?.let { out.put("delivery", it.uppercase()) }
|
||||
return out
|
||||
}
|
||||
|
||||
private fun attachmentFromShared(a: JSONObject): JSONObject {
|
||||
val out =
|
||||
JSONObject()
|
||||
.put("id", a.optString("id"))
|
||||
.put("key", a.optString("key"))
|
||||
.put("digest", a.optString("digest"))
|
||||
.put("bytes", a.optInt("bytes"))
|
||||
.put("mime", a.optString("mime"))
|
||||
.put("w", a.optInt("width"))
|
||||
.put("h", a.optInt("height"))
|
||||
a.optDouble("duration", 0.0).takeIf { it > 0.0 }?.let { out.put("dur", it) }
|
||||
a.optDouble("expires", 0.0).takeIf { it > 0.0 }?.let { out.put("exp", millisToSeconds(it)) }
|
||||
return out
|
||||
}
|
||||
|
||||
private fun storyFromShared(
|
||||
s: JSONObject,
|
||||
author: Long,
|
||||
): JSONObject {
|
||||
val out =
|
||||
JSONObject()
|
||||
// The iPhone numbers its stories and this app names them, so an
|
||||
// id that arrives as a number becomes its own digits rather
|
||||
// than being dropped. `mergeImportedPayload` de-duplicates on
|
||||
// this string, so it only has to be stable, not numeric.
|
||||
.put("id", s.opt("id")?.toString().orEmpty())
|
||||
.put("author", if (s.has("author")) s.optLong("author") else author)
|
||||
.put("mine", s.optBoolean("mine"))
|
||||
.put("caption", s.optString("caption"))
|
||||
.put("at", s.optLong("at"))
|
||||
.put("seen", s.optBoolean("seen", false))
|
||||
s.optDouble("expires", 0.0).takeIf { it > 0.0 }?.let { out.put("expires", millisToSeconds(it)) }
|
||||
// `ref` carries the expiry as well, because that is where the strip
|
||||
// reads it back from when deciding whether the bytes may still render.
|
||||
s.optJSONObject("att")?.let { out.put("ref", attachmentFromShared(it)) }
|
||||
s.optString("gid").takeIf { it.isNotEmpty() }?.let { out.put("gid", it) }
|
||||
s.optString("stream").takeIf { it.isNotEmpty() }?.let { out.put("stream", it) }
|
||||
s.optString("key").takeIf { it.isNotEmpty() }?.let { out.put("key", it) }
|
||||
s.optJSONObject("set")?.let { out.put("set", it) }
|
||||
return out
|
||||
}
|
||||
|
||||
// -- Small shared pieces -------------------------------------------------.
|
||||
|
||||
private fun secondsToMillis(seconds: Double): Long = Math.round(seconds * 1000.0)
|
||||
|
||||
private fun millisToSeconds(millis: Double): Double = millis / 1000.0
|
||||
|
||||
private fun mapArray(
|
||||
source: JSONArray,
|
||||
transform: (JSONObject) -> JSONObject,
|
||||
): JSONArray {
|
||||
val out = JSONArray()
|
||||
for (i in 0 until source.length()) {
|
||||
source.optJSONObject(i)?.let { out.put(transform(it)) }
|
||||
}
|
||||
return out
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user