obfs4 for Android, part 2: the cofactor-dirty scalar multiplication
256/256 private keys now produce lyrebird's exact public key, and the whole keypair -- private key to public key and representative -- matches end to end. 13 tests. BouncyCastle's X25519.scalarMult cannot be used for this: it clamps internally and runs 255 bits, while the dirty variant needs the COMBINED scalar fed to a 256-bit ladder with no re-clamping. The combined scalar can exceed 2^255 once lsb*L is added, so a 255-bit ladder silently drops the top bit for a fraction of keys. Only the ladder and the scalar handling are written out; the field arithmetic underneath is still BouncyCastle's. The danger here is that the wrong answer looks entirely fine. A clean X25519 public key agrees on shared secrets and round-trips; it is just confined to the prime-order subgroup and therefore distinguishable, and clamping the private key before the call is all it takes to get one. So there is a structural test beside the byte comparison: the dirty key must differ from the clean key exactly when the low three bits of the private key are non-zero, with both arms asserted to actually occur so neither passes vacuously. cswap here uses an explicit mask rather than any BouncyCastle conditional. Nothing in this module assumes those take a boolean-shaped condition, after X25519Field.cmov was found to do nothing at all when handed a plain 1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Paul Hitt
co-authored by
Claude Opus 5
Claude Opus 5
parent
db35c7ce55
commit
44e06adf92
@@ -0,0 +1,190 @@
|
||||
package io.flamenet.obfs4
|
||||
|
||||
import org.bouncycastle.math.ec.rfc7748.X25519Field
|
||||
|
||||
/**
|
||||
* X25519 scalar multiplication, including the **cofactor-dirty** variant obfs4
|
||||
* uses for key generation.
|
||||
*
|
||||
* 🔴 **The dirty variant is why obfs4 public keys differ from every normal
|
||||
* X25519 implementation's.** A conventional implementation clamps the scalar,
|
||||
* clearing the cofactor and confining the result to the prime-order subgroup —
|
||||
* and a point known to be in that subgroup is *distinguishable*, which undoes
|
||||
* the whole point of Elligator2. obfs4 keeps the cofactor component so the
|
||||
* public key ranges over the full curve.
|
||||
*
|
||||
* ⚠️ **BouncyCastle's `X25519.scalarMult` cannot be used for it.** That clamps
|
||||
* internally and runs 255 bits; the dirty variant needs the *combined* scalar
|
||||
* fed to a **256-bit** ladder with no re-clamping. The field arithmetic here is
|
||||
* still BouncyCastle's — only the ladder and the scalar handling are written
|
||||
* out.
|
||||
*
|
||||
* Follows Monocypher's `crypto_x25519_dirty_small`, which decouples the
|
||||
* cofactor from the main factor and multiplies a single combined scalar by a
|
||||
* base point of order 8·L. lyrebird instead uses the Edwards route; the two
|
||||
* agree, and the vectors check that rather than taking it on faith.
|
||||
*/
|
||||
object Ladder {
|
||||
|
||||
/**
|
||||
* A base point of order 8·L, so multiplying by it leaves the cofactor
|
||||
* intact.
|
||||
*
|
||||
* ⚠️ Its low-order component is not arbitrary — it was chosen so this route
|
||||
* lands on the same public key as the Edwards route obfs4 actually uses.
|
||||
* Any other order-8 point gives a self-consistent implementation that no
|
||||
* longer interoperates.
|
||||
*/
|
||||
private val DIRTY_BASE_POINT = byteArrayOf(
|
||||
0xd8.toByte(), 0x86.toByte(), 0x1a, 0xa2.toByte(), 0x78, 0x7a, 0xd9.toByte(), 0x26,
|
||||
0x8b.toByte(), 0x74, 0x74, 0xb6.toByte(), 0x82.toByte(), 0xe3.toByte(), 0xbe.toByte(), 0xc3.toByte(),
|
||||
0xce.toByte(), 0x36, 0x9a.toByte(), 0x1e, 0x5e, 0x31, 0x47, 0xa2.toByte(),
|
||||
0x6d, 0x37, 0x7c, 0xfd.toByte(), 0x20, 0xb5.toByte(), 0xdf.toByte(), 0x75,
|
||||
)
|
||||
|
||||
private val BASE_POINT_9 = ByteArray(32).also { it[0] = 9 }
|
||||
|
||||
/** L, the order of the prime-order subgroup, as eight 32-bit words. */
|
||||
private val GROUP_ORDER = longArrayOf(
|
||||
0x5cf5d3edL, 0x5812631aL, 0xa2f79cd6L, 0x14def9deL,
|
||||
0x00000000L, 0x00000000L, 0x00000000L, 0x10000000L,
|
||||
)
|
||||
|
||||
/** The standard X25519 clamp. */
|
||||
fun trimScalar(s: ByteArray): ByteArray = s.copyOf().also {
|
||||
it[0] = (it[0].toInt() and 248).toByte()
|
||||
it[31] = (it[31].toInt() and 127).toByte()
|
||||
it[31] = (it[31].toInt() or 64).toByte()
|
||||
}
|
||||
|
||||
/**
|
||||
* Add `(x and 7)·L` to a 256-bit little-endian scalar.
|
||||
*
|
||||
* 🔴 **This is the step that puts the cofactor back.** Trimming cleared the
|
||||
* three low bits, but those bits still carry a main-factor component that
|
||||
* has to be removed for X25519 compatibility; adding `lsb·L` (mod 8·L)
|
||||
* recombines the two into one scalar. Drop it and you get an ordinary clean
|
||||
* public key — which works, agrees on shared secrets, and is exactly the
|
||||
* distinguishable thing obfs4 avoids.
|
||||
*/
|
||||
fun addGroupOrderTimes(x: Int, s: ByteArray): ByteArray {
|
||||
val out = s.copyOf()
|
||||
val mod8 = (x and 7).toLong()
|
||||
var carry = 0L
|
||||
for (i in 0 until 8) {
|
||||
var word = 0L
|
||||
for (j in 3 downTo 0) word = (word shl 8) or (out[4 * i + j].toLong() and 0xff)
|
||||
carry += word + GROUP_ORDER[i] * mod8
|
||||
for (j in 0 until 4) out[4 * i + j] = ((carry ushr (8 * j)) and 0xff).toByte()
|
||||
carry = carry ushr 32
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
/** `bit == 1` swaps `a` and `b`, without branching on the bit. */
|
||||
private fun cswap(bit: Int, a: IntArray, b: IntArray) {
|
||||
// ⚠️ An explicit mask, deliberately. `X25519Field.cmov` was found to do
|
||||
// nothing at all when handed a plain `1`, so nothing in this file
|
||||
// assumes BouncyCastle's conditional helpers take a boolean-shaped
|
||||
// condition. See Elligator2.cselect.
|
||||
val mask = -(bit and 1)
|
||||
for (i in a.indices) {
|
||||
val t = mask and (a[i] xor b[i])
|
||||
a[i] = a[i] xor t
|
||||
b[i] = b[i] xor t
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The Montgomery ladder in projective coordinates, ref10's, via Monocypher.
|
||||
*
|
||||
* ⚠️ **`bits` is 256 for the dirty variant, not the usual 255.** The
|
||||
* combined scalar can exceed 2^255 once `lsb·L` has been added, so a ladder
|
||||
* that stops at bit 254 silently drops the top bit and returns a wrong
|
||||
* point for a fraction of keys — a bug any small sample would miss.
|
||||
*/
|
||||
fun ladder(scalar: ByteArray, base: ByteArray, bits: Int): ByteArray {
|
||||
val x1 = X25519Field.create().also { X25519Field.decode(base, 0, it) }
|
||||
val x2 = X25519Field.create().also { X25519Field.one(it) }
|
||||
val z2 = X25519Field.create()
|
||||
val x3 = x1.copyOf()
|
||||
val z3 = X25519Field.create().also { X25519Field.one(it) }
|
||||
|
||||
val t0 = X25519Field.create()
|
||||
val t1 = X25519Field.create()
|
||||
val t2 = X25519Field.create()
|
||||
|
||||
var swap = 0
|
||||
for (pos in bits - 1 downTo 0) {
|
||||
val bit = (scalar[pos shr 3].toInt() shr (pos and 7)) and 1
|
||||
swap = swap xor bit
|
||||
cswap(swap, x2, x3)
|
||||
cswap(swap, z2, z3)
|
||||
swap = bit // the xor trick: one swap deferred
|
||||
|
||||
X25519Field.sub(x3, z3, t0)
|
||||
X25519Field.sub(x2, z2, t1)
|
||||
X25519Field.add(x2, z2, x2)
|
||||
X25519Field.add(x3, z3, z2)
|
||||
X25519Field.mul(t0, x2, z3)
|
||||
X25519Field.mul(z2, t1, z2)
|
||||
X25519Field.sqr(t1, t0)
|
||||
X25519Field.sqr(x2, t1)
|
||||
X25519Field.add(z3, z2, x3)
|
||||
X25519Field.sub(z3, z2, z2)
|
||||
X25519Field.mul(t1, t0, x2)
|
||||
X25519Field.sub(t1, t0, t1)
|
||||
X25519Field.sqr(z2, z2)
|
||||
X25519Field.mul(t1, 121666, z3)
|
||||
X25519Field.sqr(x3, x3)
|
||||
X25519Field.add(t0, z3, t2)
|
||||
X25519Field.mul(x1, z2, z3)
|
||||
X25519Field.mul(t1, t2, z2)
|
||||
}
|
||||
// ⚠️ Compensates for the xor trick. Omitting it gives a result that is
|
||||
// right for exactly the scalars whose bottom bit happens to agree.
|
||||
cswap(swap, x2, x3)
|
||||
cswap(swap, z2, z3)
|
||||
|
||||
val inv = X25519Field.create().also { X25519Field.inv(z2, it) }
|
||||
val out = X25519Field.create().also { X25519Field.mul(x2, inv, it) }
|
||||
X25519Field.normalize(out)
|
||||
return ByteArray(32).also { X25519Field.encode(out, it, 0) }
|
||||
}
|
||||
|
||||
/**
|
||||
* A private key to its **cofactor-dirty** X25519 public key.
|
||||
*
|
||||
* The private key must be the full 32 bytes of entropy. Pre-clamping it
|
||||
* yields non-uniformly distributed representatives, which is the whole
|
||||
* failure this module exists to avoid.
|
||||
*/
|
||||
fun dirtyPublicKey(privateKey: ByteArray): ByteArray {
|
||||
require(privateKey.size == 32) { "a private key is 32 bytes" }
|
||||
val combined = addGroupOrderTimes(privateKey[0].toInt(), trimScalar(privateKey))
|
||||
return ladder(combined, DIRTY_BASE_POINT, 256)
|
||||
}
|
||||
|
||||
/**
|
||||
* The ordinary, cofactor-cleared public key.
|
||||
*
|
||||
* ⚠️ **A bridge's long-term identity key is CLEAN.** Only the ephemeral
|
||||
* session keys keep their cofactor, because only those travel as
|
||||
* representatives. (A dirty identity key does in fact still complete a
|
||||
* handshake — X25519 clamps, so the order-8 component cancels on both
|
||||
* sides — but every other implementation and every bridge-line tool expects
|
||||
* the clean one.)
|
||||
*/
|
||||
fun publicKey(privateKey: ByteArray): ByteArray =
|
||||
ladder(trimScalar(privateKey), BASE_POINT_9, 255)
|
||||
|
||||
/**
|
||||
* Ordinary X25519.
|
||||
*
|
||||
* 🔴 The result must be checked for all-zeros by the caller: a peer that
|
||||
* sends a low-order point drives every shared secret to zero, which a
|
||||
* handshake has to reject rather than key from.
|
||||
*/
|
||||
fun sharedSecret(privateKey: ByteArray, peerPublicKey: ByteArray): ByteArray =
|
||||
ladder(trimScalar(privateKey), peerPublicKey, 255)
|
||||
}
|
||||
@@ -0,0 +1,128 @@
|
||||
package io.flamenet.obfs4
|
||||
|
||||
import org.json.JSONObject
|
||||
import kotlin.test.Test
|
||||
import kotlin.test.assertEquals
|
||||
import kotlin.test.assertNotNull
|
||||
import kotlin.test.assertTrue
|
||||
|
||||
/**
|
||||
* Cofactor-dirty scalar base multiplication, against lyrebird.
|
||||
*
|
||||
* 🔴 **The wrong answer sits right next to the right one.** The clean X25519
|
||||
* public key — what every library computes — is a perfectly good Curve25519
|
||||
* key. It agrees on shared secrets. It round-trips. It is simply confined to
|
||||
* the prime-order subgroup and therefore *distinguishable*, which defeats the
|
||||
* entire purpose. Clamping the private key before the call is all it takes to
|
||||
* get one, so this checks structure as well as bytes.
|
||||
*/
|
||||
class LadderTest {
|
||||
|
||||
private fun vectors(): JSONObject {
|
||||
val stream = javaClass.classLoader.getResourceAsStream("elligator-vectors.json")
|
||||
assertNotNull(stream, "the vectors are missing from test resources")
|
||||
return JSONObject(stream.readBytes().decodeToString())
|
||||
}
|
||||
|
||||
private fun hex(s: String) = ByteArray(s.length / 2) {
|
||||
s.substring(it * 2, it * 2 + 2).toInt(16).toByte()
|
||||
}
|
||||
private fun hex(b: ByteArray) = b.joinToString("") { "%02x".format(it) }
|
||||
|
||||
/**
|
||||
* ⚠️ The `private` field is the 32 bytes lyrebird actually fed to
|
||||
* `ScalarBaseMult` — already past its SHA-512 pass, and *not* clamped.
|
||||
* Clamping before this call would clear the cofactor and quietly turn this
|
||||
* into the clean function.
|
||||
*/
|
||||
@Test
|
||||
fun `the dirty public key matches lyrebird`() {
|
||||
val keygen = vectors().getJSONArray("keygen")
|
||||
assertTrue(keygen.length() > 100, "too few vectors to mean anything")
|
||||
|
||||
val mismatches = mutableListOf<String>()
|
||||
for (i in 0 until keygen.length()) {
|
||||
val v = keygen.getJSONObject(i)
|
||||
val got = Ladder.dirtyPublicKey(hex(v.getString("private")))
|
||||
if (hex(got) != v.getString("public")) {
|
||||
mismatches += "private ${v.getString("private")}\n" +
|
||||
" want ${v.getString("public")}\n got ${hex(got)}"
|
||||
}
|
||||
}
|
||||
assertTrue(
|
||||
mismatches.isEmpty(),
|
||||
"${mismatches.size} of ${keygen.length()} private keys produced the wrong " +
|
||||
"public key.\n\n${mismatches.take(2).joinToString("\n")}",
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* 🔴 The structural check. If the cofactor were being cleared this would be
|
||||
* the clean key, so it must differ from the clean key exactly when the low
|
||||
* three bits of the private key are non-zero, and match it exactly when
|
||||
* they are not.
|
||||
*/
|
||||
@Test
|
||||
fun `it differs from the clean key exactly when the cofactor is set`() {
|
||||
val keygen = vectors().getJSONArray("keygen")
|
||||
val wrong = mutableListOf<String>()
|
||||
var withCofactor = 0
|
||||
var without = 0
|
||||
|
||||
for (i in 0 until keygen.length()) {
|
||||
val priv = hex(keygen.getJSONObject(i).getString("private"))
|
||||
val dirty = Ladder.dirtyPublicKey(priv)
|
||||
val clean = Ladder.publicKey(priv)
|
||||
if ((priv[0].toInt() and 7) != 0) {
|
||||
withCofactor++
|
||||
if (dirty.contentEquals(clean)) wrong += "cofactor set but the key is clean"
|
||||
} else {
|
||||
without++
|
||||
if (!dirty.contentEquals(clean)) wrong += "no cofactor but the keys differ"
|
||||
}
|
||||
}
|
||||
assertTrue(wrong.isEmpty(), "${wrong.size} keys had the wrong relationship to the clean key")
|
||||
// ⚠️ Both arms must actually occur, or the assertion above is vacuous
|
||||
// for one of them.
|
||||
assertTrue(withCofactor > 10, "no keys with a cofactor component")
|
||||
assertTrue(without > 10, "no keys without a cofactor component")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `the whole keypair matches lyrebird end to end`() {
|
||||
val keygen = vectors().getJSONArray("keygen")
|
||||
var mismatches = 0
|
||||
for (i in 0 until keygen.length()) {
|
||||
val v = keygen.getJSONObject(i)
|
||||
val pub = Ladder.dirtyPublicKey(hex(v.getString("private")))
|
||||
val want = hex(v.getString("representative"))
|
||||
.also { it[31] = (it[31].toInt() and 0x3f).toByte() }
|
||||
val matched = listOf(0, 1).any { tweak ->
|
||||
Elligator2.representativeForPublicKey(pub, tweak)
|
||||
?.also { it[31] = (it[31].toInt() and 0x3f).toByte() }
|
||||
?.contentEquals(want) == true
|
||||
}
|
||||
if (!matched || hex(pub) != v.getString("public")) mismatches++
|
||||
}
|
||||
assertEquals(0, mismatches, "$mismatches keypairs did not match lyrebird end to end")
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `trimScalar clamps the standard bits`() {
|
||||
val all = ByteArray(32) { 0xff.toByte() }
|
||||
val t = Ladder.trimScalar(all)
|
||||
assertEquals(0xf8, t[0].toInt() and 0xff, "the low three bits must be cleared")
|
||||
assertEquals(0x7f, t[31].toInt() and 0xff, "bit 255 cleared and bit 254 set")
|
||||
assertEquals(0x40, Ladder.trimScalar(ByteArray(32))[31].toInt() and 0xff)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `adding the group order is driven by the low three bits`() {
|
||||
val s = ByteArray(32) { 0x11 }
|
||||
assertTrue(Ladder.addGroupOrderTimes(0xf8, s).contentEquals(s),
|
||||
"low three bits clear must leave the scalar alone")
|
||||
for (x in 1..7) {
|
||||
assertTrue(!Ladder.addGroupOrderTimes(x, s).contentEquals(s), "x=$x must change it")
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user