Three clean rounds, and the rate limit that shapes them

Item 4. Three consecutive rounds of the Android on-device suite (18 tests each)
and three of the iOS sign-in suite (7 each), all green, against a real relay.
`Scripts/rounds.sh N` runs every layer that can run on a given machine and
prints one line per layer per round -- because "three clean rounds" should be a
command, not a ritual somebody performs from memory and reports on.

⚠️ **A skipped layer prints SKIP, never a pass.** A round that silently did
nothing looks exactly like a round that worked; that is how a suite stops
meaning anything.

The shared-account rounds needed rethinking rather than retrying. The account
service allows **three registrations per address per hour**, deliberately --
forty sign-in attempts is a person with a password manager and a bad afternoon,
forty accounts is a flood. Three sign-UPS per shape consumes an hour's budget
for one shape and fails the next, so a test written that way measures the rate
limiter rather than the product.

So sign-up is proven once per shape and the repeatable part is what a person
actually repeats: sign in, register a device, be reachable -- three rounds,
asserting every round lands on the SAME relay account. The relay keys on the
product-scoped pseudonym rather than the handle, and if it keyed on the handle
each round would quietly make a new account and nobody would notice until
messages stopped arriving.

Fresh-registration rounds stay on the local shape, which has no such limit
because a self-hosted relay is not a shared service.

⚠️ The backoff no longer pretends it can outwait an hour. It fails with the
actual constraint spelled out, because a correct defence dressed up as a flaky
test is worse than either.
This commit is contained in:
Paul Hitt
2026-08-24 06:20:32 -04:00
parent ca626b3368
commit ec03f469c7
2 changed files with 76 additions and 9 deletions
@@ -79,14 +79,24 @@ class LiveShapesTest {
*/
private fun registerPaced(accounts: AccountsClient, name: String) {
var wait = 15_000L
repeat(5) { attempt ->
repeat(3) { attempt ->
try {
accounts.register(name, PASSPHRASE)
return
} catch (e: AccountsClient.Failure.RateLimited) {
if (attempt == 4) throw e
if (attempt == 2) {
// ⚠️ **Not a product failure, and not something to retry
// through.** The service allows three registrations per
// address per HOUR, deliberately — a short backoff cannot
// outwait that, and pretending otherwise would turn a
// correct defence into a flaky test. Say what happened.
throw AssertionError(
"the account service's registration budget is spent: it allows 3 per " +
"address per hour and this run needed more. Not a defect — wait for " +
"the window to roll over, or run one shape at a time.")
}
Thread.sleep(wait)
wait = (wait * 2).coerceAtMost(120_000)
wait *= 2
}
}
}
@@ -141,16 +151,57 @@ class LiveShapesTest {
repeat(3) { round -> localRound(local, "l$round") }
}
/**
* ⚠️ **One registration, then three full rounds on it.**
*
* The account service allows **three registrations per address per hour**,
* on purpose: forty sign-in attempts is a person with a password manager
* and a bad afternoon, forty accounts is a flood. Three sign-UPS per shape
* would consume an hour's budget for one shape and fail the next — so a
* test written that way measures the rate limiter, not the product.
*
* So sign-up is proven once, and the repeatable part is the flow a person
* actually repeats: sign in, register a device, be reachable. Fresh
* registrations are covered by the local shape, which has no such limit
* because a self-hosted relay is not a shared service.
*/
@Test
fun `shared self-hosted sign-up works three times running`() {
fun `shared self-hosted works three rounds running`() {
needFleet(); needManaged()
repeat(3) { round -> sharedRound(sharedSelfHosted, "s$round") }
sharedRoundsOnOneAccount(sharedSelfHosted, "s")
}
@Test
fun `managed sign-up works three times running`() {
fun `managed works three rounds running`() {
needManaged()
repeat(3) { round -> sharedRound(managed, "m$round") }
sharedRoundsOnOneAccount(managed, "m")
}
private fun sharedRoundsOnOneAccount(base: String, tag: String) {
val name = handle(tag)
val descriptor = RelayDiscovery.describe(base)
assertTrue(descriptor?.usesSharedAccounts == true, "$base is not a shared-identity relay")
val accounts = AccountsClient(descriptor!!.accountsUrl!!)
registerPaced(accounts, name)
var firstUserId: Long? = null
repeat(3) { round ->
val session = accounts.signIn(name, PASSPHRASE)
assertTrue(session.token.isNotEmpty(), "round $round: no session token")
val account = RelayAccounts(base).sharedSession(session.token, name)
assertTrue(account.appPassword.isNotEmpty(), "round $round: no relay credential")
// ⚠️ Every round must land on the SAME relay account. The relay
// keys on the product-scoped pseudonym, not the handle; keying on
// the handle would make each round a new account and nobody would
// notice until messages stopped arriving.
if (firstUserId == null) firstUserId = account.userId
assertEquals(firstUserId, account.userId,
"round $round produced a different relay account")
registerAndProve(base, account)
}
}
// -- Messaging on each shape ---------------------------------------------
@@ -176,18 +227,34 @@ class LiveShapesTest {
* and a single combined test lets one shape's throttle hide another
* shape's result.
*/
/**
* ⚠️ A conversation on a shared-identity relay needs TWO accounts, and the
* account service allows three registrations an hour. Enabled separately
* from the round tests so a single run cannot silently consume the whole
* budget and leave the next test looking broken.
*/
@Test
fun `a conversation works on a shared self-hosted relay`() {
needFleet(); needManaged()
needFleet(); needManaged(); needBudget()
talk(sharedSelfHosted) { tag -> sharedAccount(sharedSelfHosted, tag) }
}
@Test
fun `a conversation works on the managed relay`() {
needManaged()
needManaged(); needBudget()
talk(managed) { tag -> sharedAccount(managed, tag) }
}
/**
* The account service permits three registrations per address per hour.
* These tests each need two, so they are opt-in on top of `FN_LIVE_MANAGED`
* — running everything at once cannot fit, and a suite that cannot fit its
* own limits reports failures that are really scheduling.
*/
private fun needBudget() = assumeTrue(
System.getenv("FN_LIVE_SIGNUP_BUDGET") == "1",
"set FN_LIVE_SIGNUP_BUDGET=1 — needs 2 of the 3 registrations allowed per hour")
private fun talk(base: String, make: (String) -> RelayAccounts.Account) {
val alice = ready(base, make("ta"))
val bob = ready(base, make("tb"))