The instrumented suites look for tags the app no longer has #21

New Issue

2026-09-05T20:36:49Z

paulhitt commented

2026-09-05 20:36:49 +00:00

(Migrated from gitlab.com)

The on-device suites cannot pass, and have not been able to for some time. Found while trying to use them as evidence for what Android is proven to do.

They are not flaky and the app is not broken. They are stale — they look for things the app no longer has — and each staleness hides behind a failure message that blames something else.

What is actually wrong

1. nav-contacts and nav-account do not exist anywhere in app/src/main.

$ grep -rn 'testTag("nav-' app/src/main/kotlin/
(nothing)

Every sign-up helper ends with awaitTag("nav-contacts", 120_000), and signOutIfSignedIn() opens with awaitTagOrNull("nav-account", 8_000). The navigation was rebuilt around the pill tabs and those tags went with it, so even a completely successful sign-up cannot be detected. Two minutes per test, then "nav-contacts did not appear".

2. The age gate was never filled. submit is enabled = … && (!registering || birthYear.length == 4), and nothing typed into birth-year. 🔴 performClick() on a disabled Compose node does not throw — it does nothing. Every sign-up was tapping a dead button. Fixed on fix/relay-probe-says-when-it-fails, but fixing it alone changes nothing while (1) stands.

Two traps found on the way, both fixed

⚠️ A saved emulator snapshot restored a session signed in as fnreviewbeta against relay.flamenet.io — under a run explicitly told to use a local relay. Every test failed on "could not find the sign-up screen", and the reason was invisible until a failure's screen dump was read. scripts/instrumented.sh now always cold-boots with -no-snapshot-load.

⚠️ The suites DEFAULT to live hosts when no relay argument is given — FeatureUseTest to relay1.flamenet.io, LiveSignUpFleetTest to relay.flamenet.io. A bare ./gradlew connectedAndroidTest creates accounts on a real server and says nothing about it. The script now has no default and refuses production by name.

What this means for what we claim

🔴 These rows are not backed by currently-passing tests. PARITY.md moves a row to ✅ when "the behaviour exists and something tests it" — the something exists and does not run. That is an argument for downgrading those rows, not promoting them.

Ruled out, so nobody re-checks: the relay is fine (POST /account/signup on relay1 returns 200), the emulator is fine, and cleartext HTTP is correctly refused by Android — which is a separate bug now fixed, since the probe used to hang on "Checking what this server uses…" forever rather than saying so.

The work

Replace nav-contacts / nav-account with whatever the pill tabs actually expose — and give the shell stable tags if it has none, which is the same accessibility gap found four times on iOS today
Re-run against relay1 and see what is left
Only then decide what the ledger may say

⚠️ Worth doing as its own pass rather than squeezed into another change. Every fix so far has revealed the next staleness underneath it.

**The on-device suites cannot pass, and have not been able to for some time.** Found while trying to use them as evidence for what Android is proven to do. They are not flaky and the app is not broken. They are **stale** — they look for things the app no longer has — and each staleness hides behind a failure message that blames something else. ## What is actually wrong **1. `nav-contacts` and `nav-account` do not exist anywhere in `app/src/main`.** ``` $ grep -rn 'testTag("nav-' app/src/main/kotlin/ (nothing) ``` Every sign-up helper ends with `awaitTag("nav-contacts", 120_000)`, and `signOutIfSignedIn()` opens with `awaitTagOrNull("nav-account", 8_000)`. The navigation was rebuilt around the pill tabs and those tags went with it, so **even a completely successful sign-up cannot be detected**. Two minutes per test, then "nav-contacts did not appear". **2. The age gate was never filled.** `submit` is `enabled = … && (!registering || birthYear.length == 4)`, and nothing typed into `birth-year`. 🔴 `performClick()` on a disabled Compose node does not throw — it does nothing. Every sign-up was tapping a dead button. Fixed on `fix/relay-probe-says-when-it-fails`, but fixing it alone changes nothing while (1) stands. ## Two traps found on the way, both fixed ⚠️ **A saved emulator snapshot restored a session signed in as `fnreviewbeta` against relay.flamenet.io** — under a run explicitly told to use a local relay. Every test failed on "could not find the sign-up screen", and the reason was invisible until a failure's screen dump was read. `scripts/instrumented.sh` now always cold-boots with `-no-snapshot-load`. ⚠️ **The suites DEFAULT to live hosts** when no relay argument is given — `FeatureUseTest` to relay1.flamenet.io, `LiveSignUpFleetTest` to relay.flamenet.io. A bare `./gradlew connectedAndroidTest` creates accounts on a real server and says nothing about it. The script now has no default and refuses production by name. ## What this means for what we claim 🔴 These rows are **not** backed by currently-passing tests. `PARITY.md` moves a row to ✅ when "the behaviour exists *and* something tests it" — the something exists and does not run. That is an argument for *downgrading* those rows, not promoting them. Ruled out, so nobody re-checks: the relay is fine (`POST /account/signup` on relay1 returns 200), the emulator is fine, and cleartext HTTP is correctly refused by Android — which is a separate bug now fixed, since the probe used to hang on "Checking what this server uses…" forever rather than saying so. ## The work 1. Replace `nav-contacts` / `nav-account` with whatever the pill tabs actually expose — and give the shell stable tags if it has none, which is the same accessibility gap found four times on iOS today 2. Re-run against relay1 and see what is left 3. Only then decide what the ledger may say ⚠️ Worth doing as its own pass rather than squeezed into another change. Every fix so far has revealed the next staleness underneath it.

paulhitt commented

2026-09-06 10:51:56 +00:00

(Migrated from gitlab.com)

Correction, and the real cause

🔴 The claim in the description above is wrong. nav-contacts and nav-account DO exist. They are passed as a tag = parameter to the Kit components and applied at .testTag(tag), so grep 'testTag("nav-' finds nothing. I made that same search mistake twice — lookup-contact is in the app too, for the identical reason. Anyone reading this: search for the bare string, not the testTag("…") form.

The suite could not pass, but not for that reason. Nine defects, all test-side, none in the app — fixed in !76:

The age gate was never filled; performClick() on a disabled Compose node does not throw
submit was below the fold after that field was added. An off-screen node still reports enabled — assertIsEnabled() is a state check, not a visibility one — so the gesture reached nothing. Logging inside signIn proved it was never called across 450 seconds of waiting
The peer's name was typed into the Look-up button instead of the field
The lookup step was skipped, so confirm-contact never appeared
Home renders buddy-circle-<id>, not buddy-<id> — it reported "never became a contact" about a contact on screen
RECORD_AUDIO was never granted; the mic dialog is not in the Compose tree, so it failed as "No compose hierarchies found"
Sign-out needed scrolling to and a confirming second tap, so one tap only armed it
It used the plain sign-out, which KEEPS the credential by design — right for a person, wrong for a suite whose next test wants a different account
Groups and stories live on their own tabs; stories is the strip on Home

Ruled out, so nobody re-checks

The relay is fine — POST /account/signup on relay1 returns 200
Neither relay demands proof of work (bits: 0)
The crypto is fast: KDF 1160ms, device identity 13ms, 20 classical prekeys 1ms, 20 PQ prekeys 118ms
Cleartext HTTP is correctly refused by Android; the probe hanging on "Checking what this server uses…" was a separate real bug, fixed in !75

Where it stands

All four FeatureUseTest tests pass. Not yet all in one run: the add-contact lookup is throttled per window and four tests share it.

## Correction, and the real cause 🔴 **The claim in the description above is wrong.** `nav-contacts` and `nav-account` DO exist. They are passed as a `tag =` parameter to the Kit components and applied at `.testTag(tag)`, so `grep 'testTag("nav-'` finds nothing. I made that same search mistake twice — `lookup-contact` is in the app too, for the identical reason. Anyone reading this: search for the bare string, not the `testTag("…")` form. The suite could not pass, but not for that reason. **Nine defects, all test-side, none in the app** — fixed in !76: 1. The age gate was never filled; `performClick()` on a disabled Compose node does not throw 2. **`submit` was below the fold** after that field was added. An off-screen node still reports **enabled** — `assertIsEnabled()` is a state check, not a visibility one — so the gesture reached nothing. Logging inside `signIn` proved it was never called across 450 seconds of waiting 3. The peer's name was typed into the Look-up **button** instead of the field 4. The lookup step was skipped, so `confirm-contact` never appeared 5. Home renders `buddy-circle-<id>`, not `buddy-<id>` — it reported "never became a contact" about a contact on screen 6. `RECORD_AUDIO` was never granted; the mic dialog is not in the Compose tree, so it failed as "No compose hierarchies found" 7. Sign-out needed scrolling to **and** a confirming second tap, so one tap only armed it 8. It used the plain sign-out, which KEEPS the credential by design — right for a person, wrong for a suite whose next test wants a different account 9. Groups and stories live on their own tabs; `stories` is the strip on Home ## Ruled out, so nobody re-checks - The relay is fine — `POST /account/signup` on relay1 returns 200 - Neither relay demands proof of work (`bits: 0`) - The crypto is fast: KDF 1160ms, device identity 13ms, 20 classical prekeys 1ms, **20 PQ prekeys 118ms** - Cleartext HTTP is correctly refused by Android; the probe hanging on "Checking what this server uses…" was a separate real bug, fixed in !75 ## Where it stands All four `FeatureUseTest` tests pass. Not yet all in one run: the add-contact lookup is throttled per window and four tests share it.

paulhitt commented

2026-09-07 11:52:15 +00:00

(Migrated from gitlab.com)

Correction, 2026-09-07: cause (1) is wrong — the tags exist

nav-contacts and nav-account do not exist anywhere in app/src/main.
$ grep -rn 'testTag("nav-' app/src/main/kotlin/
(nothing)

That grep finds nothing, and the tags are there anyway. They are not written as literals — they are passed to NavButton as its last argument and applied inside it:

NavButton(model, phase, Screen.CONTACTS, KitIcon.Home, KitIcon.Home, "Home", 0, "nav-contacts")
NavButton(model, phase, Screen.ACCOUNT,  KitIcon.Account, ..., "nav-account")

private fun RowScope.NavButton(..., tag: String) {
    Box(modifier = Modifier … .testTag(tag), …)
}

All five bar tags exist this way: nav-contacts, nav-messages, nav-calls, nav-stories, nav-account. So awaitTag("nav-contacts") is waiting for something the app really does expose, and item 1 of "The work" — replace them with whatever the pill tabs expose — is not needed.

⚠️ The absence-of-evidence shape is worth naming, because it cost a diagnosis: a grep for one spelling of a thing found nothing, and nothing was read as "it does not exist". I did the same thing to myself twice today — once concluding Android had no content reporting at all (a head -8 had truncated the hits) and nearly building a duplicate of a feature that shipped weeks ago.

What that leaves

Cause (2) — the age gate never filled, and performClick() on a disabled Compose node doing nothing rather than throwing — is real, and merged on fix/relay-probe-says-when-it-fails.

So the honest status is: one real cause, already fixed; one misdiagnosed. Whether the suites pass now is unknown and cannot be settled by reading — it needs a cold-booted emulator against relay1 with scripts/instrumented.sh, which is exactly what step 2 says.

🔴 The conclusion about PARITY.md still stands and is not affected by this correction: a suite that has not been run green is not evidence, whatever the reason it was failing.

## Correction, 2026-09-07: cause (1) is wrong — the tags exist > `nav-contacts` and `nav-account` do not exist anywhere in `app/src/main`. > ``` > $ grep -rn 'testTag("nav-' app/src/main/kotlin/ > (nothing) > ``` That grep finds nothing, and the tags are there anyway. They are not written as literals — they are passed to `NavButton` as its last argument and applied inside it: ```kotlin NavButton(model, phase, Screen.CONTACTS, KitIcon.Home, KitIcon.Home, "Home", 0, "nav-contacts") NavButton(model, phase, Screen.ACCOUNT, KitIcon.Account, ..., "nav-account") private fun RowScope.NavButton(..., tag: String) { Box(modifier = Modifier … .testTag(tag), …) } ``` All five bar tags exist this way: `nav-contacts`, `nav-messages`, `nav-calls`, `nav-stories`, `nav-account`. So `awaitTag("nav-contacts")` is waiting for something the app really does expose, and item 1 of "The work" — replace them with whatever the pill tabs expose — is not needed. ⚠️ **The absence-of-evidence shape is worth naming**, because it cost a diagnosis: a grep for one spelling of a thing found nothing, and nothing was read as "it does not exist". I did the same thing to myself twice today — once concluding Android had no content reporting at all (a `head -8` had truncated the hits) and nearly building a duplicate of a feature that shipped weeks ago. ## What that leaves Cause (2) — the age gate never filled, and `performClick()` on a disabled Compose node doing nothing rather than throwing — is real, and merged on `fix/relay-probe-says-when-it-fails`. So the honest status is: **one real cause, already fixed; one misdiagnosed.** Whether the suites pass now is unknown and cannot be settled by reading — it needs a cold-booted emulator against relay1 with `scripts/instrumented.sh`, which is exactly what step 2 says. 🔴 The conclusion about `PARITY.md` still stands and is not affected by this correction: a suite that has not been *run* green is not evidence, whatever the reason it was failing.

paulhitt commented

2026-09-08 00:32:53 +00:00

(Migrated from gitlab.com)

Measured against main on 2026-09-07, to scope this rather than guess at it.

27 tags the instrumented suites wait for do not exist anywhere in app/src/main — not two. Diffed every awaitTag/awaitTagOrNull/onNodeWithTag literal against every testTag( in the app:

add-contact
attach
back
cancel-voice
contact-block
contact-call
contact-clear
contact-message
contact-remove
create-group
delete-account
lookup-contact
mark-all-read
nav-account
nav-alerts
nav-contacts
nav-messages
nav-stories
record-voice
rotate-delivery-key
save-display-name
save-password
send-voice
show-safety-number
sign-out
sign-out-erase
start-call

So the drift is not limited to the navigation rebuild that took nav-contacts and nav-account: it covers contact actions, the composer, voice notes, account management and sign-out. Roughly a third of the surface the suites drive.

⚠️ Sizing note for whoever picks this up. It is test-only and cannot break the app, but it is not a quick fix, and the fix has two possible shapes that should be chosen deliberately rather than per-file:

re-add testTags to the app for what the suites drive (inert at runtime, but it puts test scaffolding back into production composables), or
rewrite the suites against what the app exposes now — content descriptions and labels — which is more work and more durable.

Verifying either needs a device, and each sign-up helper currently burns its full two-minute timeout before failing, so a full red run is slow.

Measured against `main` on 2026-09-07, to scope this rather than guess at it. **27 tags the instrumented suites wait for do not exist anywhere in `app/src/main`** — not two. Diffed every `awaitTag`/`awaitTagOrNull`/`onNodeWithTag` literal against every `testTag(` in the app: - `add-contact` - `attach` - `back` - `cancel-voice` - `contact-block` - `contact-call` - `contact-clear` - `contact-message` - `contact-remove` - `create-group` - `delete-account` - `lookup-contact` - `mark-all-read` - `nav-account` - `nav-alerts` - `nav-contacts` - `nav-messages` - `nav-stories` - `record-voice` - `rotate-delivery-key` - `save-display-name` - `save-password` - `send-voice` - `show-safety-number` - `sign-out` - `sign-out-erase` - `start-call` So the drift is not limited to the navigation rebuild that took `nav-contacts` and `nav-account`: it covers contact actions, the composer, voice notes, account management and sign-out. Roughly a third of the surface the suites drive. ⚠️ **Sizing note for whoever picks this up.** It is test-only and cannot break the app, but it is not a quick fix, and the fix has two possible shapes that should be chosen deliberately rather than per-file: 1. re-add `testTag`s to the app for what the suites drive (inert at runtime, but it puts test scaffolding back into production composables), or 2. rewrite the suites against what the app exposes now — content descriptions and labels — which is more work and more durable. Verifying either needs a device, and each sign-up helper currently burns its full two-minute timeout before failing, so a full red run is slow.

paulhitt commented

2026-09-09 11:49:49 +00:00

(Migrated from gitlab.com)

Finding (1) is a false premise: the nav tags do exist.

The grep looked for a string literal, but the tag reaches testTag through a
variable:

Tab(Screen.CONTACTS, "Home", "nav-contacts"),   // MainActivity:529
...
NavButton(model, phase, tab.screen, icon, active, tab.label, badge, tab.tag)
    ...
    }.testTag(tag),                              // MainActivity:619

So nav-contacts and nav-account are present at runtime, and
awaitTag("nav-contacts") is not looking for something that no longer exists.
Same shape as the grep "signUp(" audit that could not match
completeSignUp(: a search that under-reports gets believed.

Finding (2), the unfilled age gate, and both traps still stand. Whatever is
failing on device, it is not the tags, so the diagnosis needs redoing before
this is worth acting on.

Finding (1) is a false premise: the nav tags do exist. The grep looked for a string literal, but the tag reaches `testTag` through a variable: ```kotlin Tab(Screen.CONTACTS, "Home", "nav-contacts"), // MainActivity:529 ... NavButton(model, phase, tab.screen, icon, active, tab.label, badge, tab.tag) ... }.testTag(tag), // MainActivity:619 ``` So `nav-contacts` and `nav-account` are present at runtime, and `awaitTag("nav-contacts")` is not looking for something that no longer exists. Same shape as the `grep "signUp("` audit that could not match `completeSignUp(`: a search that under-reports gets believed. Finding (2), the unfilled age gate, and both traps still stand. Whatever is failing on device, it is not the tags, so the diagnosis needs redoing before this is worth acting on.

paulhitt commented

2026-09-11 16:16:48 +00:00

(Migrated from gitlab.com)

Status, so this issue says what is actually true:

The diagnosis here was partly wrong: nav-contacts and nav-account do exist, in NavDestinations.tabs(). The grep looked for the literal testTag("nav-, but the tag is passed as a variable.
SignUpFlowTest now passes on an emulator against relay1 (!122). What actually blocked it was a release-signed install on the AVD, taps aimed below the fold (the form scrolls), a wait for device-id on Home when it lives on Account, and performTextInput appending into fields the last test left filled.
The script fix for the release-signed install is in !124; !122 described it but did not contain it.
Still open: FeatureUseTest, CrudSurfaceTest, ChatFlowTest, CrossPlatformTest and LiveSignUpFleetTest use the same unscrolled taps and uncleared fields and have not been run. Leaving this open for them.

Status, so this issue says what is actually true: - The diagnosis here was partly wrong: `nav-contacts` and `nav-account` **do** exist, in `NavDestinations.tabs()`. The grep looked for the literal `testTag("nav-`, but the tag is passed as a variable. - `SignUpFlowTest` now passes on an emulator against relay1 (!122). What actually blocked it was a release-signed install on the AVD, taps aimed below the fold (the form scrolls), a wait for `device-id` on Home when it lives on Account, and `performTextInput` appending into fields the last test left filled. - The script fix for the release-signed install is in !124; !122 described it but did not contain it. - **Still open:** `FeatureUseTest`, `CrudSurfaceTest`, `ChatFlowTest`, `CrossPlatformTest` and `LiveSignUpFleetTest` use the same unscrolled taps and uncleared fields and have not been run. Leaving this open for them.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: extant2000/flamenet-messenger-android#21