Eighteen tests reported PASSED while running nothing #32

2026-08-31T17:54:55Z

paulhitt commented

2026-08-31 17:54:55 +00:00

(Migrated from gitlab.com)

Closes #4

val url = base ?: return        // reports PASSED

An early return from an unmet precondition makes a test that did nothing report
as a pass. FN_LIVE_FLEET is never set in CI, so this happened on every
pipeline:

file	tests	after
`LiveKeyLogTest`	7	7
`LiveConversationTest`	6	6
`LiveRelayTest`	2	2
`LiveSealedSenderTest`	2	2
`LiveSignUpTest`	1	1

The rule was already written down here

LiveShapesTest uses assumeTrue and explains why, in its own words:

Skips are announced, not silent. return from an unmet precondition
makes a test that did nothing report as PASSED, and a suite of those is
indistinguishable from a suite that works. assumeTrue reports SKIPPED, which
is the truth.

Two files followed it, five did not. This is a consistency fix, not a new idea.

It nearly reached the specification

LiveSealedSenderTest was the evidence §15 would have cited for "Android sealed
sender, wired end to end" — tests="2" skipped="0" failures="0" on a run that
did nothing. Only reading the source reveals that, and a status table is exactly
where nobody would.

The suite now looks worse and is not

before:  client 277 tests,  0 skipped, 0 failures
after:   client 277 tests, 28 skipped, 0 failures

The 28 were always doing nothing. The number was wrong before, not now.

engine 275 tests,  0 skipped  -> 275 ran
client 277 tests, 28 skipped  -> 249 ran
app     54 tests,  0 skipped  ->  54 ran
TOTAL  606 tests, 28 skipped  -> 578 actually run

Closes #4 ```kotlin val url = base ?: return // reports PASSED ``` An early return from an unmet precondition makes a test that did nothing report as a pass. `FN_LIVE_FLEET` is never set in CI, so this happened on **every** pipeline: | file | tests | skipped before | after | |---|---|---|---| | `LiveKeyLogTest` | 7 | 0 | 7 | | `LiveConversationTest` | 6 | 0 | 6 | | `LiveRelayTest` | 2 | 0 | 2 | | `LiveSealedSenderTest` | 2 | 0 | 2 | | `LiveSignUpTest` | 1 | 0 | 1 | ### The rule was already written down here `LiveShapesTest` uses `assumeTrue` and explains why, in its own words: > **Skips are announced, not silent.** `return` from an unmet precondition > makes a test that did nothing report as PASSED, and a suite of those is > indistinguishable from a suite that works. `assumeTrue` reports SKIPPED, which > is the truth. Two files followed it, five did not. This is a consistency fix, not a new idea. ### It nearly reached the specification `LiveSealedSenderTest` was the evidence §15 would have cited for *"Android sealed sender, wired end to end"* — `tests="2" skipped="0" failures="0"` on a run that did nothing. Only reading the source reveals that, and a status table is exactly where nobody would. ### The suite now looks worse and is not ``` before: client 277 tests, 0 skipped, 0 failures after: client 277 tests, 28 skipped, 0 failures ``` The 28 were always doing nothing. The number was wrong before, not now. ``` engine 275 tests, 0 skipped -> 275 ran client 277 tests, 28 skipped -> 249 ran app 54 tests, 0 skipped -> 54 ran TOTAL 606 tests, 28 skipped -> 578 actually run ```

paulhitt commented

2026-08-31 17:56:05 +00:00

(Migrated from gitlab.com)

mentioned in commit 73d5472fae

mentioned in commit 73d5472fae768c6f990218ca5e934b63111bc6c0

paulhitt (Migrated from gitlab.com) merged commit 73d5472fae into main

2026-08-31 17:56:05 +00:00

paulhitt commented

2026-08-31 18:24:09 +00:00

(Migrated from gitlab.com)

mentioned in issue flamenet-relay#9

paulhitt commented

2026-09-09 19:01:58 +00:00

(Migrated from gitlab.com)

changed the description

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: extant2000/flamenet-messenger-android#32