The launch metric is mostly test accounts #8

New Issue

2026-08-31T15:42:41Z

paulhitt commented

2026-08-31 15:42:41 +00:00

(Migrated from gitlab.com)

The launch watch reports users= straight from select count(*) from users.
Most of that number is automated tests.

test-shaped | 55
other       | 31

So users=86 against a baseline of 46 reads as +40 in two days. Real,
non-test signups since that baseline: 1.

Per day, excluding test-shaped usernames:

08-29  3     08-28  2     08-27  3     08-26  2     08-24  12

Test-shaped prefixes seen so far: fnsmoke, live, kt, sl, jvmt, iosm,
gatecheck, fnreview, peer, demo, turnchk, probe, readback, both.

🔴 Two ways this misleads, not one

Real signups are invisible. On 08-31 a 30-minute test round created 39
accounts. One real signup inside that window would not have been noticeable in
any number the watch prints.

The devices-vs-users check would misfire. 11 accounts have no device —
exactly the "signup completed, key registration failed" shape the watch is meant
to catch, which would mean accounts that cannot message. All 11 are probes:

wren, juniper, nova, both6924583, turnchk85628399, turnchk85596630,
probe698173, readback141727, livecheck131818, fnsmoke1788140958a/b

No real account is stranded. But the next run of that check reads as a genuine
alarm, and the run after that gets ignored.

Done when

The watch counts only non-test-shaped accounts, and says so in its output
The orphan check excludes them too
A separate test= figure, so pollution stays visible rather than hidden

⚠️ Prefix matching is a stopgap. It is guessing from names, and a real user
called livia or ktm would be excluded silently. The durable fix is for test
accounts not to be in this database at all — #9.

The launch watch reports `users=` straight from `select count(*) from users`. Most of that number is automated tests. ``` test-shaped | 55 other | 31 ``` So `users=86` against a baseline of 46 reads as +40 in two days. Real, non-test signups since that baseline: **1**. Per day, excluding test-shaped usernames: ``` 08-29 3 08-28 2 08-27 3 08-26 2 08-24 12 ``` Test-shaped prefixes seen so far: `fnsmoke`, `live`, `kt`, `sl`, `jvmt`, `iosm`, `gatecheck`, `fnreview`, `peer`, `demo`, `turnchk`, `probe`, `readback`, `both`. ### 🔴 Two ways this misleads, not one **Real signups are invisible.** On 08-31 a 30-minute test round created 39 accounts. One real signup inside that window would not have been noticeable in any number the watch prints. **The devices-vs-users check would misfire.** 11 accounts have no device — exactly the "signup completed, key registration failed" shape the watch is meant to catch, which would mean accounts that cannot message. All 11 are probes: ``` wren, juniper, nova, both6924583, turnchk85628399, turnchk85596630, probe698173, readback141727, livecheck131818, fnsmoke1788140958a/b ``` No real account is stranded. But the next run of that check reads as a genuine alarm, and the run after that gets ignored. ### Done when - [ ] The watch counts only non-test-shaped accounts, and says so in its output - [ ] The orphan check excludes them too - [ ] A separate `test=` figure, so pollution stays visible rather than hidden ⚠️ Prefix matching is a stopgap. It is guessing from names, and a real user called `livia` or `ktm` would be excluded silently. The durable fix is for test accounts not to be in this database at all — #9.

paulhitt commented

2026-08-31 15:43:07 +00:00

(Migrated from gitlab.com)

mentioned in issue #9

paulhitt commented

2026-08-31 18:01:52 +00:00

(Migrated from gitlab.com)

Done — the watch now reports a number that means something

before:  users=86 new60m=0 devices=90 relay_errors60m=0 container=healthy
after:   users=26 (real) test_accounts=60 new60m=0 devices=90 orphans7d=0
           geo451=0 http50x=0 relay_errors60m=0 restarts=0 container=healthy

Real accounts counted separately from test-shaped ones, both shown, so
pollution stays visible rather than hidden.
new60m counts real signups only — a real person signing up is no longer
invisible inside a test round.
geo451, http50x and restarts are now in the output. They were being
asked for on every check and were not there; I was querying them by hand each
time. docker logs on the relay cannot see 451s at all — a geo-refusal never
reaches the relay, only the edge proxy.

⚠️ The orphan check needed more than a filter

Excluding test accounts took it from 11 to 3, and the 3 are real by the
prefix rule: nova, juniper, wren — all 2026-08-21, all with no email
address, none with a device. Almost certainly hand-made fixtures from before
the baseline, but not something to assert.

Left as-is, that line would read 3 on every check for ever, and an alarm that
is always on is one nobody reads. It is now scoped to the last 7 days, which
is the honest shape of the question: key registration failing after signup
happens within seconds, so a live failure is always recent. Reads 0.

🔴 Still a stopgap, as this ticket said from the start. Prefix matching guesses
from names — a real user called livia or ktm is excluded silently. The
durable fix is #9.

### Done — the watch now reports a number that means something ``` before: users=86 new60m=0 devices=90 relay_errors60m=0 container=healthy after: users=26 (real) test_accounts=60 new60m=0 devices=90 orphans7d=0 geo451=0 http50x=0 relay_errors60m=0 restarts=0 container=healthy ``` - Real accounts counted separately from test-shaped ones, both shown, so pollution stays visible rather than hidden. - `new60m` counts real signups only — a real person signing up is no longer invisible inside a test round. - `geo451`, `http50x` and `restarts` are now in the output. They were being asked for on every check and were not there; I was querying them by hand each time. `docker logs` on the relay cannot see 451s at all — a geo-refusal never reaches the relay, only the edge proxy. ### ⚠️ The orphan check needed more than a filter Excluding test accounts took it from 11 to 3, and the 3 are real by the prefix rule: `nova`, `juniper`, `wren` — all 2026-08-21, **all with no email address**, none with a device. Almost certainly hand-made fixtures from before the baseline, but not something to assert. Left as-is, that line would read `3` on every check for ever, and an alarm that is always on is one nobody reads. It is now scoped to **the last 7 days**, which is the honest shape of the question: key registration failing after signup happens within seconds, so a live failure is always recent. Reads `0`. 🔴 Still a stopgap, as this ticket said from the start. Prefix matching guesses from names — a real user called `livia` or `ktm` is excluded silently. The durable fix is #9.

paulhitt (Migrated from gitlab.com) closed this issue

2026-08-31 19:23:30 +00:00

paulhitt commented

2026-08-31 19:23:31 +00:00

(Migrated from gitlab.com)

Closing — all three boxes are ticked and verified against the live relay:

users=26 (real) test_accounts=60 new60m=0 devices=90 orphans7d=0
  geo451=0 http50x=0 relay_errors60m=0 restarts=0 container=healthy

⚠️ The prefix-matching caveat this ticket opened with has not gone away, and is
tracked where it belongs: #9. A real user called livia or ktm is still
excluded silently, and the durable answer is test accounts not being in this
database at all.

Closing — all three boxes are ticked and verified against the live relay: ``` users=26 (real) test_accounts=60 new60m=0 devices=90 orphans7d=0 geo451=0 http50x=0 relay_errors60m=0 restarts=0 container=healthy ``` ⚠️ The prefix-matching caveat this ticket opened with has not gone away, and is tracked where it belongs: #9. A real user called `livia` or `ktm` is still excluded silently, and the durable answer is test accounts not being in this database at all.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: extant2000/flamenet-relay#8