Test accounts are reaching the production database #9

New Issue

2026-08-31T15:43:02Z

paulhitt commented

2026-08-31 15:43:02 +00:00

(Migrated from gitlab.com)

39 accounts were created in the production database in 30 minutes:

min 2026-08-31 01:49:18+00
max 2026-08-31 02:19:57+00
count 39

jvmt…, kt…, sl…, live…, fnsmoke…, iosm… — the handles the live
suites generate.

🔴 The tests are not the bug

They are opt-in and documented to point away from production. LiveSignUpTest:

⚠️ Opt-in via FN_LIVE_FLEET, and pointed at the disposable fleet rather than
production. It creates accounts, which is exactly what a disposable relay is
for and exactly what relay.flamenet.io is not.

Scripts/rounds.sh defaults it to https://relay1.flamenet.io, and that fleet
is genuinely disposable — relays-relay1-1 has its own SQLite at
/data/relay.sqlite and no FN_POSTGRES_URL at all.

So a correctly-invoked run cannot touch production. Something was invoked with
FN_LIVE_FLEET pointing at production, or by a path that ignores it. That path
is what this issue is for — the guard is documentation, and documentation is
not a guard.

⚠️ The backstop that would have caught it had just been removed

705faba raised the signup limit from 5/IP/hour to 50 on 2026-08-30 22:18,
about three and a half hours before this burst. 39 signups in 30 minutes from
one source fits under 50/hour and did not fit under 5.

That raise was right on its own terms — five per hour was locking out real users
behind shared NAT. But it removed an accidental backstop, and nothing deliberate
replaced it. Recording the connection rather than leaving it to be rediscovered.

Consequences

The launch metric is unusable (#8): 55 of 86 accounts are test-shaped
11 accounts have no device, so the orphan alarm is permanently pre-tripped
Retention, quotas and any future per-account job all run over this

Done when

How production was reached is established, not guessed
A live suite cannot run against production — refuse the run when the
target host resolves to the production relay, rather than documenting it
Existing test accounts removed from production, or deliberately kept and
excluded everywhere that counts them
A decision on whether prod should refuse a signup burst on its own

Invocation side spans paulhitt/flamenet-messenger-android (live/kt/sl/jvmt)
and paulhitt/flamenet-messenger-ios (iosm, Scripts/rounds.sh). Metric side is #8.

39 accounts were created in the **production** database in 30 minutes: ``` min 2026-08-31 01:49:18+00 max 2026-08-31 02:19:57+00 count 39 ``` `jvmt…`, `kt…`, `sl…`, `live…`, `fnsmoke…`, `iosm…` — the handles the live suites generate. ### 🔴 The tests are not the bug They are opt-in and documented to point away from production. `LiveSignUpTest`: > ⚠️ Opt-in via `FN_LIVE_FLEET`, and pointed at the disposable fleet rather than > production. It creates accounts, which is exactly what a disposable relay is > for and **exactly what `relay.flamenet.io` is not**. `Scripts/rounds.sh` defaults it to `https://relay1.flamenet.io`, and that fleet is genuinely disposable — `relays-relay1-1` has its own SQLite at `/data/relay.sqlite` and no `FN_POSTGRES_URL` at all. So a correctly-invoked run cannot touch production. Something was invoked with `FN_LIVE_FLEET` pointing at production, or by a path that ignores it. **That path is what this issue is for** — the guard is documentation, and documentation is not a guard. ### ⚠️ The backstop that would have caught it had just been removed `705faba` raised the signup limit from 5/IP/hour to 50 on 2026-08-30 22:18, about three and a half hours before this burst. 39 signups in 30 minutes from one source fits under 50/hour and did not fit under 5. That raise was right on its own terms — five per hour was locking out real users behind shared NAT. But it removed an accidental backstop, and nothing deliberate replaced it. Recording the connection rather than leaving it to be rediscovered. ### Consequences - The launch metric is unusable (#8): 55 of 86 accounts are test-shaped - 11 accounts have no device, so the orphan alarm is permanently pre-tripped - Retention, quotas and any future per-account job all run over this ### Done when - [ ] How production was reached is established, not guessed - [ ] A live suite **cannot** run against production — refuse the run when the target host resolves to the production relay, rather than documenting it - [ ] Existing test accounts removed from production, or deliberately kept and excluded everywhere that counts them - [ ] A decision on whether prod should refuse a signup burst on its own Invocation side spans paulhitt/flamenet-messenger-android (`live`/`kt`/`sl`/`jvmt`) and paulhitt/flamenet-messenger-ios (`iosm`, `Scripts/rounds.sh`). Metric side is #8.

paulhitt commented

2026-08-31 18:01:52 +00:00

(Migrated from gitlab.com)

mentioned in issue #8

paulhitt commented

2026-08-31 18:24:05 +00:00

(Migrated from gitlab.com)

The durable half is done — a live suite can no longer be pointed at production

LiveTarget.refuseProduction is checked by every live suite before it does
anything, and throws rather than skips. "This run was pointed at production"
is not an unmet precondition; it is a mistake that has already done damage once,
and a skip is how it goes unnoticed the next time.

Proven both directions rather than read:

FN_LIVE_FLEET=https://flamenetmessenger.com
  -> IllegalStateException: refusing to run a live suite against
     flamenetmessenger.com — that is production. These tests create real accounts.

FN_LIVE_FLEET unset
  -> tests=1 skipped=1 failures=0

🔴 The guard-s own test found a real bug in it before it shipped. The
disposable fleet lives on the same domain as production, so denying
flamenet.io and its subdomains refused relay1.flamenet.io — the exact place
these suites are meant to point. It would have blocked every legitimate run while
appearing to block only production. The fleet is now named explicitly and checked
first, which keeps the bare domain denied so a production host added later is
refused by default.

⚠️ A deny list rather than an allow list, deliberately: an allow list would also
refuse a developer-s localhost or a self-hoster-s own relay, and a guard that
gets in the way is one somebody turns off.

In paulhitt/flamenet-messenger-android!5.

Still open on this ticket

How production was reached — not established, and may not be
recoverable: the burst was 01:49–02:19 UTC and the suites are correctly gated,
so something supplied FN_LIVE_FLEET pointing at production. The guard makes
that impossible going forward regardless of the answer.
The 60 existing test accounts — still in the production database,
still distorting every count. Removing them is a destructive change to a live
database and wants your say-so, not mine.
Whether prod should refuse a signup burst on its own — related to my
raising the limit 5→50 in 705faba three hours before this happened.

### The durable half is done — a live suite can no longer be pointed at production `LiveTarget.refuseProduction` is checked by every live suite before it does anything, and **throws rather than skips**. "This run was pointed at production" is not an unmet precondition; it is a mistake that has already done damage once, and a skip is how it goes unnoticed the next time. Proven both directions rather than read: ``` FN_LIVE_FLEET=https://flamenetmessenger.com -> IllegalStateException: refusing to run a live suite against flamenetmessenger.com — that is production. These tests create real accounts. FN_LIVE_FLEET unset -> tests=1 skipped=1 failures=0 ``` 🔴 **The guard-s own test found a real bug in it before it shipped.** The disposable fleet lives on the *same domain* as production, so denying `flamenet.io` and its subdomains refused `relay1.flamenet.io` — the exact place these suites are meant to point. It would have blocked every legitimate run while appearing to block only production. The fleet is now named explicitly and checked first, which keeps the bare domain denied so a production host added later is refused by default. ⚠️ A deny list rather than an allow list, deliberately: an allow list would also refuse a developer-s localhost or a self-hoster-s own relay, and a guard that gets in the way is one somebody turns off. In paulhitt/flamenet-messenger-android!5. ### Still open on this ticket - [ ] **How production was reached** — not established, and may not be recoverable: the burst was 01:49–02:19 UTC and the suites are correctly gated, so something supplied `FN_LIVE_FLEET` pointing at production. The guard makes that impossible going forward regardless of the answer. - [ ] **The 60 existing test accounts** — still in the production database, still distorting every count. Removing them is a destructive change to a live database and wants your say-so, not mine. - [ ] Whether prod should refuse a signup burst on its own — related to my raising the limit 5→50 in `705faba` three hours before this happened.

paulhitt commented

2026-08-31 18:38:55 +00:00

(Migrated from gitlab.com)

mentioned in merge request flamenet-messenger-android!6

paulhitt commented

2026-09-05 00:21:42 +00:00

(Migrated from gitlab.com)

Done 2026-09-04, on the owner's instruction to keep fnreviewalpha and fnreviewbeta.

The cleanup

Most of the 55 had already gone; four test-shaped accounts remained:

id	handle	created
116	`iosm88377169`	2026-09-02
117	`t82691man`	2026-09-04 00:45 UTC
118	`t86674man`	2026-09-04 01:51 UTC
119	`t89471man`	2026-09-04 02:38 UTC

⚠️ The three t…man handles are RelayTypesUITests' shape and were created before today's session began, so they are not from today's runs — checked rather than assumed.

Removed in one transaction after a full pg_dump (backup-predelete-20260905001644.sql), across every table that references a user or device — enumerated from information_schema rather than remembered, because buddies has a buddy_id as well as a user_id and a kept account can list a removed one.

Two accounts remain: 114 fnreviewbeta, 115 fnreviewalpha.

🔴 One table deliberately untouched

key_log keeps all 18 leaves, including the 7 written by the removed accounts. It is append-only by design: clients pin a head on first sight and demand inclusion proofs, so deleting a leaf would make a root that was already published unverifiable. A log of key publications is not a list of live accounts.

Verified after the deletion, not assumed: head size 18 with a root present, and LiveRelayTest > an inclusion proof from the deployed log verifies passes.

⚠️ And the rows were still on disk after the delete

19 dead tuples — the removed accounts' data, gone from the live rows and still in the heap. VACUUM FULL run immediately (19 → 0) rather than waiting for Sunday's timer, since "we deleted the test accounts" should mean the bytes too. See docs/PLAINTEXT.md.

The path that created them is closed

LiveTarget.refuseProduction throws when a live suite is pointed at production, and LiveTargetTest covers it — production hosts are refused, the disposable fleet and local relays are allowed. The guard is code now, not documentation.

Split out

The sign-up backstop this issue recorded — the 5→50 raise removing an accidental one — is relay#52. It cannot be closed by flipping FN_SIGNUP_POW_BITS: enforcement is unconditional, so enabling it refuses every shipped client that lacks proof-of-work support, which presents to a new user as the app simply not working.

**Done 2026-09-04, on the owner's instruction to keep `fnreviewalpha` and `fnreviewbeta`.** ### The cleanup Most of the 55 had already gone; four test-shaped accounts remained: | id | handle | created | |---|---|---| | 116 | `iosm88377169` | 2026-09-02 | | 117 | `t82691man` | 2026-09-04 00:45 UTC | | 118 | `t86674man` | 2026-09-04 01:51 UTC | | 119 | `t89471man` | 2026-09-04 02:38 UTC | ⚠️ The three `t…man` handles are `RelayTypesUITests`' shape and were created **before** today's session began, so they are not from today's runs — checked rather than assumed. Removed in one transaction after a full `pg_dump` (`backup-predelete-20260905001644.sql`), across every table that references a user or device — enumerated from `information_schema` rather than remembered, because `buddies` has a `buddy_id` as well as a `user_id` and a kept account can list a removed one. **Two accounts remain: 114 `fnreviewbeta`, 115 `fnreviewalpha`.** ### 🔴 One table deliberately untouched `key_log` keeps all **18** leaves, including the **7** written by the removed accounts. It is append-only by design: clients pin a head on first sight and demand inclusion proofs, so deleting a leaf would make a root that was already published unverifiable. **A log of key publications is not a list of live accounts.** Verified after the deletion, not assumed: head size 18 with a root present, and `LiveRelayTest > an inclusion proof from the deployed log verifies` passes. ### ⚠️ And the rows were still on disk after the delete 19 dead tuples — the removed accounts' data, gone from the live rows and still in the heap. `VACUUM FULL` run immediately (19 → 0) rather than waiting for Sunday's timer, since "we deleted the test accounts" should mean the bytes too. See `docs/PLAINTEXT.md`. ### The path that created them is closed `LiveTarget.refuseProduction` throws when a live suite is pointed at production, and `LiveTargetTest` covers it — *production hosts are refused*, *the disposable fleet and local relays are allowed*. The guard is code now, not documentation. ### Split out The sign-up backstop this issue recorded — the 5→50 raise removing an accidental one — is **relay#52**. It cannot be closed by flipping `FN_SIGNUP_POW_BITS`: enforcement is unconditional, so enabling it refuses every shipped client that lacks proof-of-work support, which presents to a new user as the app simply not working.

paulhitt (Migrated from gitlab.com) closed this issue

2026-09-05 00:21:43 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: extant2000/flamenet-relay#9