From 8a3770f1d14a48eb64500d2b52392cdd3d7a2d61 Mon Sep 17 00:00:00 2001 From: Paul Hitt Date: Sun, 23 Aug 2026 15:12:14 -0400 Subject: [PATCH] Kotlin: the session layer, verified across the wire An app does not use X3DH and a ratchet, it uses "send this string to that person". This is that layer: device identities with prekeys and rotation, envelopes as the relay actually carries them, and sessions that survive being put away and taken out again. 11 unit tests, and the interop harness grows from 23 assertions to 37 -- the new ones at the envelope level, which is where this protocol's one shipped bug lived. A client and a relay disagreed about a field name and nothing caught it until a live round trip; hand-assembled arguments to X3DH would not have. Kotlin now builds a whole envelope, the web engine's own session layer parses it, and both directions agree on the associated data. Three decisions worth stating. Retired signed prekeys are kept for longer than the relay keeps undelivered mail. A sender fetches a bundle, encrypts against that prekey id, and the envelope can then sit in a queue for the relay's full retention window. Discarding the private half at rotation turns every one of those into an AEAD failure -- which the receiving client cannot distinguish from an attack. A prekey retired past retention is refused outright rather than guessed at, for the same reason: guessing derives a garbage secret and reports it as tampering. The handshake block rides every message until one is answered, not until one is sent. Any message may be the first to arrive. Sessions archive with their skipped message keys, and the harness proves it against the peer rather than against our own bytes. Those keys are the messages already in flight; an archive that drops them loses exactly the mail that was in the air when the app was killed. The failure this whole path exists to prevent is a client that loses session state, re-registers a device, and shows every contact "security keys changed" -- the warning that is supposed to mean an attack. `org.json` is compileOnly: it is in the Android framework, so it costs nothing on a phone and bundling a copy would shadow the platform's. Safe only because the two encodings that must match byte-for-byte -- the ratchet header and the canonical log leaf -- are built by hand; Android's implementation preserves key order and the reference one does not. Also adds PARITY.md: what the iOS app does, what Android does, what is left. It is the definition of done, and a row moves only when something tests it. Co-Authored-By: Claude Opus 5 --- PARITY.md | 140 +++++++++++ engine/build.gradle.kts | 15 +- .../kotlin/io/flamenet/e2e/DeviceIdentity.kt | 218 ++++++++++++++++ .../kotlin/io/flamenet/e2e/DoubleRatchet.kt | 77 ++++++ .../main/kotlin/io/flamenet/e2e/E2ESession.kt | 237 ++++++++++++++++++ .../main/kotlin/io/flamenet/e2e/Ed25519.kt | 10 +- .../src/main/kotlin/io/flamenet/e2e/X25519.kt | 13 + .../kotlin/io/flamenet/e2e/E2ESessionTest.kt | 211 ++++++++++++++++ .../main/kotlin/io/flamenet/interop/Main.kt | 78 ++++++ 9 files changed, 993 insertions(+), 6 deletions(-) create mode 100644 PARITY.md create mode 100644 engine/src/main/kotlin/io/flamenet/e2e/DeviceIdentity.kt create mode 100644 engine/src/main/kotlin/io/flamenet/e2e/E2ESession.kt create mode 100644 engine/src/test/kotlin/io/flamenet/e2e/E2ESessionTest.kt diff --git a/PARITY.md b/PARITY.md new file mode 100644 index 0000000..4aa23ee --- /dev/null +++ b/PARITY.md @@ -0,0 +1,140 @@ +# Android ↔ iOS parity + +What the iOS app does, what Android does, and what is left. Read from the +sources listed, not from memory — every row names the iOS file it came from so +it can be re-checked rather than trusted. + +**Standing rule: this file is the definition of done for the Android app.** A +row moves to ✅ when the behaviour exists *and* something tests it, not when the +screen renders. + +--- + +## Where things stand + +| | iOS | Android | +|---|---|---| +| Protocol engine | `Sources/E2E`, 15 files | `engine`, 10 files | +| App layer | ~9,700 lines, 45 files | none yet | +| Tests | XCUITest + JS interop | 67 unit + 37 cross-engine interop | +| Ships to | nobody (no signing team) | nobody (no app module yet) | + +Android's engine is **verified against the web engine in both directions** +(`engine/test/interop-kotlin.mjs`), which is a stronger claim than the iOS app +could make for most of its history. What Android has no version of at all is +everything above the protocol. + +--- + +## Protocol engine + +| Feature | iOS | Android | Note | +|---|:--:|:--:|---| +| X25519, HKDF, AES-GCM | ✅ | ✅ | Android verified against RFC 5869 / NIST vectors | +| Ed25519 | ✅ | ✅ | RFC 8032 vectors; keygen-from-seed done here, signing via the provider | +| X3DH v1 | ✅ | ✅ | cross-engine verified | +| Double Ratchet | ✅ | ✅ | cross-engine verified, including reordering and loss | +| Safety numbers | ✅ | ✅ | cross-engine verified | +| Transparency log (`KeyLog`) | ✅ | ✅ | canonical leaf + inclusion proofs, cross-engine verified | +| Account KDF / fnid | ✅ | ✅ | both pinned to the published vector file | +| Sealed sender | ✅ | ⚪ | `E2E/SealedSender.swift` | +| Envelope + wire codec | ✅ | ✅ | cross-engine verified at the envelope level | +| Session archive / restore | ✅ | ✅ | cross-engine verified, in-flight messages included | +| Device identity + prekey rotation | ✅ | ✅ | retired prekeys retained past the relay window | +| Vault (encryption at rest) | ✅ | ⚪ | `E2E/KeychainE2EStore.swift` → Android Keystore | +| Attachment encryption | ✅ | ⚪ | `E2E/E2EAttachment.swift` | +| ML-KEM-768 (PQ agreement) | ⚠️ | ⚪ | see below | +| ML-DSA-65 (PQ identity) | ⚠️ | ⚪ | see below | + +⚠️ **The post-quantum gap is not what it looks like.** iOS has `PQKEM.swift` and +`PQSign.swift`, and they are inert: CryptoKit exposes ML-KEM only on iOS 26 and +the app targets iOS 16, so no shipped iOS build does post-quantum anything. +Android has no platform ML-KEM at any API level. So the two clients are +*currently equal in effect* and the honest parity target is v1/v2 negotiation +that degrades cleanly — which the Android engine already does, and the interop +harness asserts. Closing this properly is a separate decision (implement, or +take a vetted library) and it should be made once, for both clients. + +--- + +## Accounts and identity + +From `Sources/Auth/*` and `Sources/Views/Sign*`. + +| Feature | Android | +|---|:--:| +| Sign up (managed, shared self-hosted, local self-hosted) | ⚪ | +| Sign in, session restore | ⚪ | +| Sign out | ⚪ | +| Password reset request + confirm | ⚪ | +| Change password | ⚪ | +| Delete account | ⚪ | +| Relay picker / `SiteConfig` equivalent | ⚪ | +| Deterministic local-relay KDF salt | ⚪ | +| Multi-homing (publish + read home records) | ⚪ | + +⚠️ Sign-up is where the three deployment modes become visible to a user, and +each derives its salt differently. Getting this wrong does not error — it +derives a *different account* from the same passphrase. + +--- + +## Messaging + +From `E2E/E2EInbox.swift`, which is the real feature list. + +| Feature | Android | +|---|:--:| +| Send / receive text | ⚪ | +| Inbox polling | ⚪ | +| Reactions | ⚪ | +| Photos | ⚪ | +| Voice notes (record + play) | ⚪ | +| Stories, and seen-state | ⚪ | +| Groups: create, list, members, leave | ⚪ | +| Group text and images | ⚪ | +| Delivery keys + rotation | ⚪ | +| Trust state, verification, identity-change acceptance | ⚪ | +| Key-log enforcement and head gossip | ⚪ | +| Clear thread / all threads / caches | ⚪ | +| History at rest (encrypted, device-only key) | ⚪ | +| Presence and status | ⚪ | +| Buddy list, add, remove, block | ⚪ | +| Member lookup | ⚪ | +| Buddy commands | ⚪ | +| Voice calls (WebRTC, DTLS-SRTP, TURN) | ⚪ | +| Notifications list, read state | ⚪ | +| Services (AIM/ICQ via `flamenet-oscar`) | ⚪ | + +⚠️ Calls need a WebRTC dependency on Android as they do on iOS. That is the one +place parity forces a third-party library rather than a choice about one. + +--- + +## Settings + +Every key in `Views/Settings/SettingsKeys.swift` has to exist, because each one +is a promise the UI already makes: push on/off, per-kind notification toggles, +in-app sounds, previews, read receipts, last-seen, online status, app lock and +its grace period, send-on-return, timestamps, larger type, appearance, recent +emoji, photo auto-download. + +App lock also needs a lock screen that covers **everything including sheets**, +and obscures on backgrounding rather than only on return. + +--- + +## Build order + +Each step is useless without the one before it, which is why they are in this +order and not a nicer-looking one. + +1. ~~**Session layer** — envelope codec, device identity, archive/restore.~~ ✅ + Cross-engine verified at the envelope level, restarts included. +2. **Vault** — Android Keystore, encryption at rest for the archive above. +3. **`:client`** — REST, relay auth, device registration, prekey upload, inbox + polling, attachments, groups. +4. **`:app`** — Compose UI, in the order above: sign-in, buddy list, chat, then + the rest. +5. **Instrumented tests** — the JVM tests prove nothing about Conscrypt. This + engine has already been bitten once by exactly that. diff --git a/engine/build.gradle.kts b/engine/build.gradle.kts index 5bf283b..6e28df4 100644 --- a/engine/build.gradle.kts +++ b/engine/build.gradle.kts @@ -6,10 +6,17 @@ repositories { mavenCentral() } dependencies { testImplementation(kotlin("test")) - // Only to read the published vector file. The engine itself has no - // dependencies at all so far, which is worth keeping for as long as it - // lasts: every jar here is one more thing between the protocol and someone - // auditing it. + // `org.json` is part of the Android framework, so on a phone this costs + // zero shipped bytes and is `compileOnly` for that reason -- bundling a + // second copy would shadow the platform's and is a classic duplicate-class + // build failure. On the JVM (tests, the interop driver) it has to be a real + // dependency. + // + // Safe here only because nothing security-relevant depends on key order: + // Android's implementation preserves insertion order and the reference one + // does not, and the two bytes-must-match encodings in this engine -- the + // ratchet header and the canonical log leaf -- are both built by hand. + compileOnly("org.json:json:20240303") testImplementation("org.json:json:20240303") } diff --git a/engine/src/main/kotlin/io/flamenet/e2e/DeviceIdentity.kt b/engine/src/main/kotlin/io/flamenet/e2e/DeviceIdentity.kt new file mode 100644 index 0000000..db12720 --- /dev/null +++ b/engine/src/main/kotlin/io/flamenet/e2e/DeviceIdentity.kt @@ -0,0 +1,218 @@ +package io.flamenet.e2e + +import org.json.JSONObject +import java.security.KeyPair + +/** + * One device's long-term keys and its published prekeys (spec §3). + * + * A device has two identity keys, and conflating them is the classic mistake: + * [identityDH] does key agreement and [identitySigning] does signatures. + * X25519 and Ed25519 are related curves, but a single key used for both invites + * cross-protocol attacks and neither JCA nor CryptoKit will hand you the same + * object for the two jobs anyway. + * + * ⚠️ **Retired signed prekeys must be kept, not discarded.** A sender fetches a + * bundle, encrypts against that `spk_id`, and the envelope can then sit + * undelivered for the relay's full 30-day retention window. Deleting the + * private half at rotation turns every one of those queued messages into an + * AEAD failure — which is indistinguishable, to the receiving client, from an + * attack. That is why [retiredSignedPreKeys] exists and why the retention + * window here is longer than the relay's. + * + * A port of `engine/src/e2e.js`, which is normative. No post-quantum material + * yet: this engine has no ML-KEM, so it publishes a v1 bundle and the peer's + * `negotiate()` sees a v1 device, which is true. + */ +class DeviceIdentity private constructor( + val deviceId: String, + val identityDH: KeyPair, + val identitySigning: KeyPair, + var signedPreKey: KeyPair, + var signedPreKeyId: Int, + var signedPreKeyCreatedAt: Long, +) { + + /** A retired signed prekey, and when it stopped being published. */ + private class Retired(val keyPair: KeyPair, val retiredAt: Long) + + private val retiredSignedPreKeys = LinkedHashMap() + private val oneTimePreKeys = LinkedHashMap() + private var nextPreKeyId = 1 + + /** One published prekey: the id a sender quotes back, and the public half. */ + data class PreKey(val id: Int, val publicKey: ByteArray) + + companion object { + /** The spec's "every few weeks", made a number so something can enforce it. */ + const val ROTATE_AFTER_MS = 30L * 24 * 60 * 60 * 1000 + + /** + * How long a retired prekey stays usable. **Must exceed the relay's + * undelivered-envelope retention** (30 days by default), with margin + * for a sender that fetched a bundle and then went offline. Too short + * does not fail loudly — it drops old messages. + */ + const val RETAIN_AFTER_MS = 45L * 24 * 60 * 60 * 1000 + + fun create(now: Long = System.currentTimeMillis()): DeviceIdentity = DeviceIdentity( + deviceId = java.util.UUID.randomUUID().toString().lowercase(), + identityDH = X25519.generate(), + identitySigning = Ed25519.generate(), + signedPreKey = X25519.generate(), + signedPreKeyId = 1, + signedPreKeyCreatedAt = now, + ) + + fun fromArchive(archive: JSONObject): DeviceIdentity { + fun x(o: JSONObject) = KeyPair( + X25519.publicKeyFromRaw(Bytes.unb64(o.getString("pub"))), + X25519.privateKeyFromPkcs8(Bytes.unb64(o.getString("priv"))), + ) + val identity = DeviceIdentity( + deviceId = archive.getString("deviceId"), + identityDH = x(archive.getJSONObject("ikDH")), + identitySigning = Ed25519.fromSeed( + Bytes.unb64(archive.getJSONObject("ikSig").getString("seed"))), + signedPreKey = x(archive.getJSONObject("spk")), + signedPreKeyId = archive.getInt("spkId"), + // 0 means "written before this field existed", and the honest + // reading of an unknown age is "old" — a device that rotates + // once on first launch after an upgrade is harmless, one that + // never rotates because it defaulted to `now` is not. + signedPreKeyCreatedAt = archive.optLong("spkAt", 0), + ) + identity.nextPreKeyId = archive.optInt("nextId", 1) + + val otp = archive.optJSONObject("otp") ?: JSONObject() + for (key in otp.keys()) identity.oneTimePreKeys[key.toInt()] = x(otp.getJSONObject(key)) + + val previous = archive.optJSONObject("prevSpk") ?: JSONObject() + for (key in previous.keys()) { + val entry = previous.getJSONObject(key) + identity.retiredSignedPreKeys[key.toInt()] = + Retired(x(entry), entry.optLong("retiredAt", 0)) + } + return identity + } + } + + val identityDHPublic: ByteArray get() = X25519.rawFromPublicKey(identityDH.public) + val identitySigningPublic: ByteArray get() = Ed25519.rawFromPublicKey(identitySigning.public) + + /** + * The signature a peer checks before trusting the signed prekey. + * + * This is the whole reason a relay cannot quietly substitute its own prekey + * and read everything — provided the peer actually verifies it, which is + * why [E2ESession.initiate] does, unconditionally. + */ + fun signedPreKeySignature(): ByteArray = + Ed25519.sign(identitySigning.private, X25519.rawFromPublicKey(signedPreKey.public)) + + fun generateOneTimePreKeys(count: Int): List = (0 until count).map { + val id = nextPreKeyId++ + val pair = X25519.generate() + oneTimePreKeys[id] = pair + PreKey(id, X25519.rawFromPublicKey(pair.public)) + } + + /** Consume a one-time prekey. Consumed exactly once — that is the point of it. */ + fun takeOneTimePreKey(id: Int): KeyPair? = + if (id < 0) null else oneTimePreKeys.remove(id) + + val oneTimePreKeyCount: Int get() = oneTimePreKeys.size + + /** + * The signed prekey a prekey envelope refers to: the current one, or a + * retired one still inside the retention window. + * + * An id of 0 means the sender did not state one — true of every envelope + * built before rotation existed — and those can only have used the current + * key, so falling back is correct rather than lax. An id that IS stated and + * unknown returns null, because guessing would derive a garbage shared + * secret and report it as tampering. + */ + fun signedPreKeyFor(id: Int): KeyPair? = when { + id == 0 || id == signedPreKeyId -> signedPreKey + else -> retiredSignedPreKeys[id]?.keyPair + } + + fun shouldRotateSignedPreKey(now: Long = System.currentTimeMillis()): Boolean = + now - signedPreKeyCreatedAt >= ROTATE_AFTER_MS + + /** + * Retire the current signed prekey and mint a new one. + * + * ⚠️ The caller must republish the bundle and persist afterwards. Rotating + * without republishing leaves the relay handing out a bundle whose private + * half has moved; rotating without persisting loses the retired key on the + * next launch, which is the message-loss case above. + */ + fun rotateSignedPreKey(now: Long = System.currentTimeMillis()): KeyPair { + retiredSignedPreKeys[signedPreKeyId] = Retired(signedPreKey, now) + signedPreKey = X25519.generate() + signedPreKeyId += 1 + signedPreKeyCreatedAt = now + pruneSignedPreKeys(now) + return signedPreKey + } + + /** Drop retired prekeys no queued envelope can still refer to. */ + fun pruneSignedPreKeys(now: Long = System.currentTimeMillis()) { + retiredSignedPreKeys.entries.removeAll { now - it.value.retiredAt > RETAIN_AFTER_MS } + } + + /** The body of `POST /e2e/devices`. */ + fun registrationWire(oneTime: List = emptyList()): JSONObject = JSONObject() + .put("device_id", deviceId) + .put("ik_dh", Bytes.b64(identityDHPublic)) + .put("ik_sig", Bytes.b64(identitySigningPublic)) + .put("spk_id", signedPreKeyId) + .put("spk", Bytes.b64(X25519.rawFromPublicKey(signedPreKey.public))) + .put("spk_sig", Bytes.b64(signedPreKeySignature())) + .put("proto", 1) + .put("opks", org.json.JSONArray().also { array -> + oneTime.forEach { + array.put(JSONObject().put("id", it.id).put("pub", Bytes.b64(it.publicKey))) + } + }) + + /** + * Everything that must survive a restart. + * + * ⚠️ **This is private key material.** It is the vault's job to encrypt it; + * this returns it in the clear because a store that cannot see the plaintext + * cannot store it. Never log this, never write it to shared preferences. + * + * Retired prekeys travel with it, for the reason at the top of this file. + */ + fun archive(): JSONObject { + fun x(pair: KeyPair) = JSONObject() + .put("priv", Bytes.b64(pair.private.encoded)) + .put("pub", Bytes.b64(X25519.rawFromPublicKey(pair.public))) + + val otp = JSONObject() + oneTimePreKeys.forEach { (id, pair) -> otp.put(id.toString(), x(pair)) } + val previous = JSONObject() + retiredSignedPreKeys.forEach { (id, entry) -> + previous.put(id.toString(), x(entry.keyPair).put("retiredAt", entry.retiredAt)) + } + + return JSONObject() + .put("deviceId", deviceId) + .put("ikDH", x(identityDH)) + // Ed25519 is stored as its 32-byte seed rather than a PKCS#8 blob, + // because that is the one form both the JCA and a passphrase-derived + // account key agree on. + .put("ikSig", JSONObject() + .put("seed", Bytes.b64(Ed25519.seedOf(identitySigning.private))) + .put("pub", Bytes.b64(identitySigningPublic))) + .put("spk", x(signedPreKey)) + .put("spkId", signedPreKeyId) + .put("spkAt", signedPreKeyCreatedAt) + .put("nextId", nextPreKeyId) + .put("otp", otp) + .put("prevSpk", previous) + } +} diff --git a/engine/src/main/kotlin/io/flamenet/e2e/DoubleRatchet.kt b/engine/src/main/kotlin/io/flamenet/e2e/DoubleRatchet.kt index da336d9..8c2e8ac 100644 --- a/engine/src/main/kotlin/io/flamenet/e2e/DoubleRatchet.kt +++ b/engine/src/main/kotlin/io/flamenet/e2e/DoubleRatchet.kt @@ -1,5 +1,6 @@ package io.flamenet.e2e +import org.json.JSONObject import javax.crypto.Mac import javax.crypto.spec.SecretKeySpec @@ -84,6 +85,16 @@ class DoubleRatchet private constructor( */ const val MAX_SKIP = 1000 + /** + * How many skipped keys are kept across the whole session. + * + * `MAX_SKIP` bounds one header; without a second bound, a peer sending + * a thousand-message gap over and over grows this map forever. Oldest + * out first — a key that has waited through two thousand later messages + * is not arriving. + */ + const val MAX_STORE = 2000 + /** Matches `Aead`'s GCM nonce length; the ratchet strips it on the wire. */ private const val NONCE_LEN = 12 @@ -111,6 +122,35 @@ class DoubleRatchet private constructor( sent = 0, received = 0, previousChainLength = 0, ) + /** The other half of [archive]. */ + fun fromArchive(archive: JSONObject): DoubleRatchet { + fun opt(key: String) = archive.optString(key, "").takeIf { it.isNotEmpty() }?.let(Bytes::unb64) + val selfPrivate = opt("dhs") + val selfPublic = opt("dhsPub") + val ratchet = DoubleRatchet( + rootKey = Bytes.unb64(archive.getString("rk")), + sendingChain = opt("cks"), + receivingChain = opt("ckr"), + selfRatchet = if (selfPrivate != null && selfPublic != null) { + java.security.KeyPair( + X25519.publicKeyFromRaw(selfPublic), + X25519.privateKeyFromPkcs8(selfPrivate), + ) + } else null, + remoteRatchet = opt("dhr"), + sent = archive.getInt("ns"), + received = archive.getInt("nr"), + previousChainLength = archive.getInt("pn"), + ) + val skipped = archive.optJSONArray("skipped") ?: org.json.JSONArray() + for (i in 0 until skipped.length()) { + val entry = skipped.getJSONObject(i) + ratchet.skipped[SkippedKey(entry.getString("dh"), entry.getInt("n"))] = + Bytes.unb64(entry.getString("mk")) + } + return ratchet + } + private fun rootStep(rootKey: ByteArray, dhOutput: ByteArray): Pair { val out = Hkdf.derive(dhOutput, rootKey, RATCHET_INFO, 64) return out.copyOfRange(0, 32) to out.copyOfRange(32, 64) @@ -197,6 +237,9 @@ class DoubleRatchet private constructor( while (n < target) { val (mk, next) = chainStep(ck) skipped[SkippedKey(remote, n)] = mk + while (skipped.size > MAX_STORE) { + skipped.remove(skipped.keys.first()) + } ck = next n += 1 } @@ -224,4 +267,38 @@ class DoubleRatchet private constructor( rootKey = stepped.first sendingChain = stepped.second } + + /** + * Everything the conversation needs to continue after a restart. + * + * ⚠️ A ratchet that is not persisted is not a bug you notice — it is a bug + * your *contacts* notice. A client that loses session state falls back to + * minting a new device identity, and everyone who has ever talked to that + * account sees "security keys changed", which is the warning that is + * supposed to mean an attack. + * + * Skipped message keys travel too. They are the keys for messages that have + * not arrived yet, and dropping them at a restart loses exactly the + * messages that were already in flight. + */ + fun archive(): JSONObject { + val skippedJson = org.json.JSONArray() + skipped.forEach { (key, messageKey) -> + skippedJson.put(JSONObject() + .put("dh", key.ratchetPublic) + .put("n", key.index) + .put("mk", Bytes.b64(messageKey))) + } + return JSONObject() + .put("rk", Bytes.b64(rootKey)) + .put("cks", sendingChain?.let { Bytes.b64(it) }) + .put("ckr", receivingChain?.let { Bytes.b64(it) }) + .put("dhs", selfRatchet?.let { Bytes.b64(it.private.encoded) }) + .put("dhsPub", selfRatchet?.let { Bytes.b64(X25519.rawFromPublicKey(it.public)) }) + .put("dhr", remoteRatchet?.let { Bytes.b64(it) }) + .put("ns", sent) + .put("nr", received) + .put("pn", previousChainLength) + .put("skipped", skippedJson) + } } diff --git a/engine/src/main/kotlin/io/flamenet/e2e/E2ESession.kt b/engine/src/main/kotlin/io/flamenet/e2e/E2ESession.kt new file mode 100644 index 0000000..586e51e --- /dev/null +++ b/engine/src/main/kotlin/io/flamenet/e2e/E2ESession.kt @@ -0,0 +1,237 @@ +package io.flamenet.e2e + +import org.json.JSONObject + +/** + * One conversation with one device (spec §8.1). + * + * This is the layer an app actually uses: [initiate] or [respond] once, then + * [encrypt] and [decrypt] strings. Everything below it — the handshake, the + * ratchet, the associated data — is set up here and then never thought about + * again, which is the point. + * + * ⚠️ **The first messages are `prekey` envelopes and must stay that way until + * the peer replies.** Not until the first one is *sent* — until one is + * *answered*. Any single message may be the first to arrive, so all of them + * have to carry the handshake block; sending it once means losing that one + * message strands the whole conversation, and the symptom is a contact whose + * messages simply never appear. + * + * A port of `engine/src/e2e.js`, which is normative, and verified against it by + * `engine/test/interop-kotlin.mjs` rather than by reading. + */ +class E2ESession private constructor( + private val ratchet: DoubleRatchet, + /** Bound into every message's AEAD tag: the two identity keys, initiator first. */ + val associatedData: ByteArray, + private var pendingHandshake: JSONObject?, + val version: Int, +) { + + companion object { + + /** + * Start a conversation from a bundle fetched for a peer. + * + * ⚠️ **The signed prekey's signature is verified here, unconditionally.** + * The bundle arrives from a relay, and a relay that can substitute an + * unverified prekey can read the entire conversation. This is the single + * check that stops that, so it is not optional, not configurable, and + * not the caller's job to remember. + */ + fun initiate(identity: DeviceIdentity, bundle: PeerBundle): E2ESession { + if (!Ed25519.verify(bundle.identitySigning, bundle.signedPreKeySignature, bundle.signedPreKey)) { + throw SecurityException("the peer's signed prekey is not signed by its identity key") + } + + val initiated = X3DH.initiate( + identityPrivateDH = identity.identityDH.private, + bundle = X3DH.Bundle( + identityKeyDH = bundle.identityDH, + signedPreKey = bundle.signedPreKey, + oneTimePreKey = bundle.oneTimePreKey, + ), + ) + + val handshake = JSONObject() + .put("ik_dh", Bytes.b64(identity.identityDHPublic)) + .put("ik_sig", Bytes.b64(identity.identitySigningPublic)) + .put("ek", Bytes.b64(initiated.ephemeralPublic)) + .put("spk_id", bundle.signedPreKeyId) + .put("opk_id", bundle.oneTimePreKeyId) + + return E2ESession( + ratchet = DoubleRatchet.initiator(initiated.sharedSecret, bundle.signedPreKey), + associatedData = identity.identitySigningPublic + bundle.identitySigning, + pendingHandshake = handshake, + version = 1, + ) + } + + /** + * Answer a `prekey` envelope. + * + * ⚠️ The ratchet seeds from the signed prekey **X3DH actually used**, + * which may be a retired one. Seeding from the current key while X3DH + * used a retired one desynchronises the very first DH step, and the + * failure appears on the peer's screen, not here. + */ + fun respond(identity: DeviceIdentity, envelope: Envelope): E2ESession { + val handshake = envelope.handshake + ?: throw IllegalArgumentException("not a prekey envelope: no x3dh block") + + val signedPreKeyId = handshake.optInt("spk_id", 0) + val signedPreKey = identity.signedPreKeyFor(signedPreKeyId) + ?: throw IllegalArgumentException( + "unknown signed prekey $signedPreKeyId: retired past retention, or fabricated") + + val theirIdentityDH = Bytes.unb64(handshake.getString("ik_dh")) + val theirIdentitySigning = Bytes.unb64(handshake.getString("ik_sig")) + + val sharedSecret = X3DH.respond( + identityPrivateDH = identity.identityDH.private, + signedPreKeyPrivate = signedPreKey.private, + oneTimePrivate = identity.takeOneTimePreKey(handshake.optInt("opk_id", -1))?.private, + initiatorIdentityDH = theirIdentityDH, + initiatorEphemeral = Bytes.unb64(handshake.getString("ek")), + ) + + val session = E2ESession( + ratchet = DoubleRatchet.responder(sharedSecret, signedPreKey), + associatedData = theirIdentitySigning + identity.identitySigningPublic, + pendingHandshake = null, + version = 1, + ) + return session + } + + /** Restore a session written by [archive]. */ + fun fromArchive(archive: JSONObject): E2ESession = E2ESession( + ratchet = DoubleRatchet.fromArchive(archive.getJSONObject("ratchet")), + associatedData = Bytes.unb64(archive.getString("ad")), + pendingHandshake = archive.optJSONObject("pending"), + version = archive.optInt("version", 1), + ) + } + + fun encrypt(text: String): Envelope { + val message = ratchet.encrypt(text.toByteArray(Charsets.UTF_8), associatedData) + return Envelope( + version = version, + type = if (pendingHandshake != null) "prekey" else "msg", + header = message.header, + handshake = pendingHandshake, + ciphertext = message.ciphertext, + ) + } + + fun decrypt(envelope: Envelope): String { + val plaintext = ratchet.decrypt( + DoubleRatchet.Message(envelope.header, envelope.ciphertext), associatedData) + // Only now — a reply proves the peer has the handshake, which is the + // one moment it is safe to stop repeating it. + pendingHandshake = null + return String(plaintext, Charsets.UTF_8) + } + + /** + * ⚠️ [pendingHandshake] has to be archived too. It is the only copy of the + * ephemeral public key the handshake block contains, and regenerating it + * after a restart would produce a different block — a different session, + * from the peer's point of view. + */ + fun archive(): JSONObject = JSONObject() + .put("ratchet", ratchet.archive()) + .put("ad", Bytes.b64(associatedData)) + .put("pending", pendingHandshake) + .put("version", version) +} + +/** + * A peer's published keys, as fetched from `GET /e2e/keys`. + * + * ⚠️ `oneTimePreKey` being null is normal, not an error: the pool drains and + * the relay serves what is left. It costs initial forward secrecy for that one + * session and nothing else. + */ +data class PeerBundle( + val deviceId: String, + val identityDH: ByteArray, + val identitySigning: ByteArray, + val signedPreKeyId: Int, + val signedPreKey: ByteArray, + val signedPreKeySignature: ByteArray, + val oneTimePreKeyId: Int, + val oneTimePreKey: ByteArray?, + val proto: Int, +) { + companion object { + /** + * Read the relay's wire shape. + * + * ⚠️ The one bug this protocol has actually shipped was a client and a + * relay disagreeing about field names, and nothing caught it until a + * live round trip. So the translation lives in exactly one place, and + * that place is tested. + */ + fun fromWire(wire: JSONObject): PeerBundle { + val oneTime = wire.optJSONObject("opk") + return PeerBundle( + deviceId = wire.getString("device_id"), + identityDH = Bytes.unb64(wire.getString("ik_dh")), + identitySigning = Bytes.unb64(wire.getString("ik_sig")), + signedPreKeyId = wire.optInt("spk_id", 0), + signedPreKey = Bytes.unb64(wire.getString("spk")), + signedPreKeySignature = Bytes.unb64(wire.getString("spk_sig")), + oneTimePreKeyId = oneTime?.optInt("id", -1) ?: -1, + oneTimePreKey = oneTime?.let { Bytes.unb64(it.getString("pub")) }, + proto = wire.optInt("proto", 1), + ) + } + } +} + +/** One message on the wire (spec §8.2). */ +data class Envelope( + val version: Int, + val type: String, + val header: DoubleRatchet.Header, + val handshake: JSONObject?, + val ciphertext: ByteArray, +) { + fun toJson(): JSONObject = JSONObject() + .put("v", version) + .put("type", type) + .put("header", JSONObject() + .put("dh", Bytes.b64(header.dh)) + .put("pn", header.pn) + .put("n", header.n)) + .put("x3dh", handshake) + .put("ct", Bytes.b64(ciphertext)) + + /** + * The relay carries an envelope as one opaque base64 string, so it never + * sees these fields at all. + */ + fun toPayload(): String = Bytes.b64(toJson().toString().toByteArray(Charsets.UTF_8)) + + companion object { + fun fromJson(json: JSONObject): Envelope { + val header = json.getJSONObject("header") + return Envelope( + version = json.optInt("v", 1), + type = json.optString("type", "msg"), + header = DoubleRatchet.Header( + dh = Bytes.unb64(header.getString("dh")), + pn = header.getInt("pn"), + n = header.getInt("n"), + ), + handshake = json.optJSONObject("x3dh"), + ciphertext = Bytes.unb64(json.getString("ct")), + ) + } + + fun fromPayload(payload: String): Envelope = + fromJson(JSONObject(String(Bytes.unb64(payload), Charsets.UTF_8))) + } +} diff --git a/engine/src/main/kotlin/io/flamenet/e2e/Ed25519.kt b/engine/src/main/kotlin/io/flamenet/e2e/Ed25519.kt index 10b6958..4b1278e 100644 --- a/engine/src/main/kotlin/io/flamenet/e2e/Ed25519.kt +++ b/engine/src/main/kotlin/io/flamenet/e2e/Ed25519.kt @@ -124,8 +124,14 @@ object Ed25519 { private fun publicFromPrivate(privateKey: java.security.PrivateKey): java.security.PublicKey = publicKeyFromRaw(Rfc8032.publicKeyFromSeed(seedOf(privateKey))) - /** The 32 seed bytes back out of a PKCS#8 encoding. */ - private fun seedOf(privateKey: java.security.PrivateKey): ByteArray { + /** + * The 32 seed bytes back out of a PKCS#8 encoding. + * + * Public because the vault stores a seed, not a DER blob: it is the one + * representation the JCA, CryptoKit and a passphrase-derived account key + * all agree on. + */ + fun seedOf(privateKey: java.security.PrivateKey): ByteArray { val encoded = privateKey.encoded require(encoded.size == PKCS8_PREFIX.size + KEY_LEN) { "unexpected Ed25519 PKCS#8 length ${encoded.size}" diff --git a/engine/src/main/kotlin/io/flamenet/e2e/X25519.kt b/engine/src/main/kotlin/io/flamenet/e2e/X25519.kt index ec3d513..887b082 100644 --- a/engine/src/main/kotlin/io/flamenet/e2e/X25519.kt +++ b/engine/src/main/kotlin/io/flamenet/e2e/X25519.kt @@ -3,6 +3,7 @@ package io.flamenet.e2e import java.security.KeyFactory import java.security.KeyPair import java.security.KeyPairGenerator +import java.security.spec.PKCS8EncodedKeySpec import java.security.spec.X509EncodedKeySpec import javax.crypto.KeyAgreement @@ -52,6 +53,18 @@ object X25519 { .generatePublic(X509EncodedKeySpec(SPKI_PREFIX + raw)) } + /** + * A private key back from its PKCS#8 encoding, which is what the vault + * stores. + * + * PKCS#8 rather than the raw 32 scalar bytes for the same reason public + * keys go through DER: Conscrypt will not build an XDH key from raw bytes. + * Storing raw and wrapping at load time would work on a laptop and throw on + * a phone, which is the exact bug this engine has already had once. + */ + fun privateKeyFromPkcs8(der: ByteArray): java.security.PrivateKey = + KeyFactory.getInstance("XDH").generatePrivate(PKCS8EncodedKeySpec(der)) + /** The raw 32 bytes the wire carries, from a JCA public key. */ fun rawFromPublicKey(key: java.security.PublicKey): ByteArray { val encoded = key.encoded diff --git a/engine/src/test/kotlin/io/flamenet/e2e/E2ESessionTest.kt b/engine/src/test/kotlin/io/flamenet/e2e/E2ESessionTest.kt new file mode 100644 index 0000000..9aeb20b --- /dev/null +++ b/engine/src/test/kotlin/io/flamenet/e2e/E2ESessionTest.kt @@ -0,0 +1,211 @@ +package io.flamenet.e2e + +import org.json.JSONObject +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertFailsWith +import kotlin.test.assertNotEquals +import kotlin.test.assertNull +import kotlin.test.assertTrue + +class E2ESessionTest { + + /** A bundle for [identity], as its relay would serve one. */ + private fun bundleFor(identity: DeviceIdentity, withOneTime: Boolean = true): PeerBundle { + val oneTime = if (withOneTime) identity.generateOneTimePreKeys(1).first() else null + return PeerBundle.fromWire(identity.registrationWire().also { wire -> + if (oneTime != null) { + wire.put("opk", JSONObject() + .put("id", oneTime.id) + .put("pub", Bytes.b64(oneTime.publicKey))) + } + }) + } + + @Test + fun `a conversation runs both ways`() { + val alice = DeviceIdentity.create() + val bob = DeviceIdentity.create() + + val a = E2ESession.initiate(alice, bundleFor(bob)) + val first = a.encrypt("hello") + val b = E2ESession.respond(bob, first) + assertEquals("hello", b.decrypt(first)) + + assertEquals("hi back", a.decrypt(b.encrypt("hi back"))) + assertEquals("and again", b.decrypt(a.encrypt("and again"))) + } + + /** + * ⚠️ The handshake has to ride EVERY message until one is answered, not + * just the first. Any message may be the one that arrives first, and a + * conversation where the peer never received the handshake looks, from the + * outside, like a contact who has stopped replying. + */ + @Test + fun `the handshake repeats until the peer answers`() { + val alice = DeviceIdentity.create() + val bob = DeviceIdentity.create() + val a = E2ESession.initiate(alice, bundleFor(bob)) + + val queued = (1..3).map { a.encrypt("m$it") } + assertTrue(queued.all { it.type == "prekey" && it.handshake != null }) + + // Bob receives the THIRD one first. It must be enough on its own. + val b = E2ESession.respond(bob, queued[2]) + assertEquals("m3", b.decrypt(queued[2])) + assertEquals("m1", b.decrypt(queued[0])) + assertEquals("m2", b.decrypt(queued[1])) + + // Once Bob replies and Alice reads it, the block stops. + a.decrypt(b.encrypt("got them")) + assertEquals("msg", a.encrypt("now plain").type) + assertNull(a.encrypt("still plain").handshake) + } + + /** + * The bug this exists to stop: sessions that lived only in memory. A client + * that loses them re-registers a device, and every contact is shown + * "security keys changed" — the warning that is supposed to mean an attack. + */ + @Test + fun `a session survives being archived mid-conversation`() { + val alice = DeviceIdentity.create() + val bob = DeviceIdentity.create() + + var a = E2ESession.initiate(alice, bundleFor(bob)) + val first = a.encrypt("before the restart") + val b = E2ESession.respond(bob, first) + assertEquals("before the restart", b.decrypt(first)) + + // Both sides go through a restart, at different points in the chain. + a = E2ESession.fromArchive(JSONObject(a.archive().toString())) + val restoredB = E2ESession.fromArchive(JSONObject(b.archive().toString())) + + assertEquals("after the restart", restoredB.decrypt(a.encrypt("after the restart"))) + assertEquals("and back", a.decrypt(restoredB.encrypt("and back"))) + } + + /** Skipped keys are the in-flight messages; losing them at a restart loses those. */ + @Test + fun `messages in flight across a restart still open`() { + val alice = DeviceIdentity.create() + val bob = DeviceIdentity.create() + val a = E2ESession.initiate(alice, bundleFor(bob)) + val first = a.encrypt("m0") + var b = E2ESession.respond(bob, first) + b.decrypt(first) + + val later = (1..3).map { a.encrypt("m$it") } + b.decrypt(later[2]) // arrives early; 1 and 2 are now skipped keys + b = E2ESession.fromArchive(JSONObject(b.archive().toString())) + assertEquals("m1", b.decrypt(later[0])) + assertEquals("m2", b.decrypt(later[1])) + } + + @Test + fun `an unsigned prekey is refused`() { + val alice = DeviceIdentity.create() + val bob = DeviceIdentity.create() + val impostor = DeviceIdentity.create() + + // A relay substituting its own prekey, signed by a key it controls. + // Nothing else in the protocol catches this. + val tampered = bundleFor(bob).copy( + signedPreKey = X25519.rawFromPublicKey(impostor.signedPreKey.public), + signedPreKeySignature = impostor.signedPreKeySignature(), + ) + assertFailsWith { E2ESession.initiate(alice, tampered) } + } + + @Test + fun `a session opens when the one-time prekey pool is drained`() { + val alice = DeviceIdentity.create() + val bob = DeviceIdentity.create() + val a = E2ESession.initiate(alice, bundleFor(bob, withOneTime = false)) + val first = a.encrypt("no one-time prekey") + val b = E2ESession.respond(bob, first) + assertEquals("no one-time prekey", b.decrypt(first)) + } + + /** + * Rotation without retention is message loss. A sender that fetched a + * bundle before the rotation is still entitled to be understood. + */ + @Test + fun `an envelope built against a retired signed prekey still opens`() { + val alice = DeviceIdentity.create() + val bob = DeviceIdentity.create() + + val old = bundleFor(bob) + val a = E2ESession.initiate(alice, old) + val queued = a.encrypt("sent before the rotation") + + bob.rotateSignedPreKey() + assertNotEquals(old.signedPreKeyId, bob.signedPreKeyId) + + val b = E2ESession.respond(bob, queued) + assertEquals("sent before the rotation", b.decrypt(queued)) + } + + @Test + fun `a signed prekey retired past retention is refused rather than guessed`() { + val alice = DeviceIdentity.create() + val bob = DeviceIdentity.create() + val queued = E2ESession.initiate(alice, bundleFor(bob)).encrypt("far too late") + + // Rotate, then let the retention window pass. Guessing at the current + // key here would derive a garbage secret and report it as tampering. + bob.rotateSignedPreKey() + bob.pruneSignedPreKeys(System.currentTimeMillis() + DeviceIdentity.RETAIN_AFTER_MS + 1) + assertFailsWith { E2ESession.respond(bob, queued) } + } + + @Test + fun `a one-time prekey is consumed exactly once`() { + val bob = DeviceIdentity.create() + val key = bob.generateOneTimePreKeys(1).first() + assertEquals(1, bob.oneTimePreKeyCount) + assertTrue(bob.takeOneTimePreKey(key.id) != null) + assertNull(bob.takeOneTimePreKey(key.id)) + assertEquals(0, bob.oneTimePreKeyCount) + } + + @Test + fun `an envelope survives the payload round trip`() { + val alice = DeviceIdentity.create() + val bob = DeviceIdentity.create() + val a = E2ESession.initiate(alice, bundleFor(bob)) + val sent = a.encrypt("through the relay 👋") + + val received = Envelope.fromPayload(sent.toPayload()) + assertEquals("prekey", received.type) + val b = E2ESession.respond(bob, received) + assertEquals("through the relay 👋", b.decrypt(received)) + } + + @Test + fun `a device identity survives being archived`() { + val original = DeviceIdentity.create() + original.generateOneTimePreKeys(3) + original.rotateSignedPreKey() + + val restored = DeviceIdentity.fromArchive(JSONObject(original.archive().toString())) + assertEquals(original.deviceId, restored.deviceId) + assertEquals(original.signedPreKeyId, restored.signedPreKeyId) + assertEquals(3, restored.oneTimePreKeyCount) + assertContentEqualsB64(original.identityDHPublic, restored.identityDHPublic) + assertContentEqualsB64(original.identitySigningPublic, restored.identitySigningPublic) + + // The restored device must still be able to complete a handshake, which + // is the only thing that proves the PRIVATE halves came back too. + val alice = DeviceIdentity.create() + val bundle = PeerBundle.fromWire(restored.registrationWire()) + val session = E2ESession.initiate(alice, bundle) + val first = session.encrypt("after a reload") + assertEquals("after a reload", E2ESession.respond(restored, first).decrypt(first)) + } + + private fun assertContentEqualsB64(expected: ByteArray, actual: ByteArray) = + assertEquals(Bytes.b64(expected), Bytes.b64(actual)) +} diff --git a/interop/src/main/kotlin/io/flamenet/interop/Main.kt b/interop/src/main/kotlin/io/flamenet/interop/Main.kt index 326f234..620f21b 100644 --- a/interop/src/main/kotlin/io/flamenet/interop/Main.kt +++ b/interop/src/main/kotlin/io/flamenet/interop/Main.kt @@ -2,7 +2,11 @@ package io.flamenet.interop import io.flamenet.e2e.AccountKdf import io.flamenet.e2e.Aead +import io.flamenet.e2e.DeviceIdentity import io.flamenet.e2e.DoubleRatchet +import io.flamenet.e2e.E2ESession +import io.flamenet.e2e.Envelope +import io.flamenet.e2e.PeerBundle import io.flamenet.e2e.Ed25519 import io.flamenet.e2e.Hkdf import io.flamenet.e2e.KeyLog @@ -86,6 +90,13 @@ private fun handle(req: JSONObject): JSONObject = when (req.getString("cmd")) { "respond" -> respond(req) "encrypt" -> encrypt(req) "decrypt" -> decrypt(req) + // --- the session layer, which is what an app actually uses ------------ + "session_device" -> sessionDevice() + "session_initiate" -> sessionInitiate(req) + "session_respond" -> sessionRespond(req) + "session_encrypt" -> sessionEncrypt(req) + "session_decrypt" -> sessionDecrypt(req) + "session_reload" -> sessionReload(req) "safety" -> JSONObject().put( "number", SafetyNumber.compute(req.bytes("local"), req.bytes("remote"))) "account_kdf" -> accountKdf(req) @@ -237,6 +248,73 @@ private fun messageFrom(req: JSONObject): DoubleRatchet.Message { ) } +// --------------------------------------------------------------------------- +// Session layer +// --------------------------------------------------------------------------- + +private val identities = HashMap() +private val liveSessions = HashMap() + +/** A device and the bundle wire a relay would serve for it, one prekey included. */ +private fun sessionDevice(): JSONObject { + val identity = DeviceIdentity.create() + identities[identity.deviceId] = identity + val oneTime = identity.generateOneTimePreKeys(1).first() + val wire = identity.registrationWire() + .put("opk", JSONObject().put("id", oneTime.id).put("pub", b64(oneTime.publicKey))) + return JSONObject().put("device", identity.deviceId).put("bundle", wire) +} + +private fun sessionInitiate(req: JSONObject): JSONObject { + val identity = identities.getValue(req.getString("device")) + val session = E2ESession.initiate(identity, PeerBundle.fromWire(req.getJSONObject("bundle"))) + val id = next("kotlin-e2e-") + liveSessions[id] = session + return JSONObject() + .put("session", id) + .put("ad", b64(session.associatedData)) + // The full envelope, exactly as it would go on the wire. This is the + // layer where the one bug this protocol has actually shipped lives: a + // client and a relay disagreeing about a field name, caught by nothing + // until a live round trip. + .put("envelope", session.encrypt(req.getString("text")).toJson()) +} + +private fun sessionRespond(req: JSONObject): JSONObject { + val identity = identities.getValue(req.getString("device")) + val envelope = Envelope.fromJson(req.getJSONObject("envelope")) + val session = E2ESession.respond(identity, envelope) + val id = next("kotlin-e2e-") + liveSessions[id] = session + return JSONObject() + .put("session", id) + .put("ad", b64(session.associatedData)) + .put("text", session.decrypt(envelope)) +} + +private fun sessionEncrypt(req: JSONObject): JSONObject = JSONObject() + .put("envelope", liveSessions.getValue(req.getString("session")) + .encrypt(req.getString("text")).toJson()) + +private fun sessionDecrypt(req: JSONObject): JSONObject = JSONObject() + .put("text", liveSessions.getValue(req.getString("session")) + .decrypt(Envelope.fromJson(req.getJSONObject("envelope")))) + +/** + * Archive a session and restore it, in place. + * + * Exposed to the harness so the persistence path is exercised by a real + * cross-engine conversation rather than only by a Kotlin test talking to + * itself. A session that survives a restart against our own bytes but not + * against the peer's is a distinction the user experiences. + */ +private fun sessionReload(req: JSONObject): JSONObject { + val id = req.getString("session") + val archived = liveSessions.getValue(id).archive() + liveSessions[id] = E2ESession.fromArchive(JSONObject(archived.toString())) + return JSONObject().put("reloaded", true).put("bytes", archived.toString().length) +} + private fun accountKdf(req: JSONObject): JSONObject { val passphrase = req.getString("passphrase") val salt = req.bytes("salt")