Paul HittandClaude Fable 5.1
ee8465fa34
The instrumented suites had not had a green run since 2026-08-26. Since then the composer hides send behind a camera until there is text, the status chips moved to the Account screen, contacts became circles with the profile behind a long-press, the stories tab posts through new-story, alerts left the bar, and the app moved inside imePadding() so a keyboard shortens the screen. Every helper that tapped a control by its tag without scrolling to it was tapping outside the window. The suites now scroll before they tap, assert send after typing, look for the status chips and the device id on the Account screen, long-press a circle for the profile, and post a story from the stories tab. Sign-up takes a bare host in instrumented.sh; a URL became https://https:// and every test skipped. One product change. Alerts left the bar on 2026-08-28 with the note that the screen survives for the tap-through from Account, and that tap-through was never built, so the inbox had no way in. It is a row on Account now, counting unread the way the bar badge does. CrudSurfaceTest 4 of 4, FeatureUseTest 4 of 4, ChatFlowTest add-and-send green on the emulator against relay1. The reply case needs something answering as the fixture account and stays open. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
149 lines
6.7 KiB
Bash
Executable File
149 lines
6.7 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Run the on-device suites against a relay you named, on an emulator with no
|
|
# history.
|
|
#
|
|
# Two ways an instrumented run silently points at PRODUCTION, and this
|
|
# script closes both.
|
|
#
|
|
# 1. The suites DEFAULT to a live host when no relay argument is passed,
|
|
# `FeatureUseTest` to relay1.flamenet.io, `LiveSignUpFleetTest` to
|
|
# relay.flamenet.io. A plain `./gradlew connectedAndroidTest` therefore
|
|
# signs accounts up on somebody's real server, and nothing in the output
|
|
# says so.
|
|
# 2. A SAVED EMULATOR SNAPSHOT restores whatever the AVD was last doing.
|
|
# Observed 2026-09-05: a snapshot signed in as `fnreviewbeta` against
|
|
# relay.flamenet.io, restored under a test run that had been told to use a
|
|
# local relay. Every one of the ten tests failed, they could not find the
|
|
# sign-up screen, because the app was already signed in, and the reason
|
|
# was invisible until the failure's screen dump was read.
|
|
#
|
|
# Usage: scripts/instrumented.sh 10.0.2.2:18810 [ClassName ...]
|
|
set -euo pipefail
|
|
|
|
RELAY="${1:-}"
|
|
[ -n "$RELAY" ] || {
|
|
cat >&2 <<'MSG'
|
|
usage: scripts/instrumented.sh <relay-host:port> [TestClass ...]
|
|
|
|
Name the relay. There is deliberately no default: the suites have their own,
|
|
and theirs are live servers.
|
|
|
|
10.0.2.2:18810 the host machine's local relay, as seen from the emulator
|
|
MSG
|
|
exit 2
|
|
}
|
|
shift || true
|
|
|
|
# A bare host, never a URL. The suites take the host and pick the scheme
|
|
# themselves (http for the emulator's loopback, https for anything else),
|
|
# so a scheme here becomes `https://https://...` inside the reachability
|
|
# probe and every test skips. Observed 2026-09-17: nineteen tests, nine
|
|
# skipped, verdict NOT a pass, for an argument that looked right.
|
|
RELAY="${RELAY#http://}"; RELAY="${RELAY#https://}"; RELAY="${RELAY%/}"
|
|
|
|
# Production is refused, not warned about. A run that creates accounts on
|
|
# relay.flamenet.io cannot be undone by noticing afterwards.
|
|
#
|
|
# relay1/relay2 are the TEST FLEET and are allowed, that is what they are
|
|
# for, and the suites need https because Android blocks cleartext, so a local
|
|
# http relay cannot serve. Loud rather than silent: accounts really are created.
|
|
case "$RELAY" in
|
|
*//relay.flamenet.io*|relay.flamenet.io*)
|
|
echo "REFUSED: $RELAY is the PRODUCTION relay. These suites create accounts." >&2
|
|
echo "Use a test-fleet relay (relay1/relay2) or a local one." >&2
|
|
exit 1;;
|
|
*relay1.flamenet.io*|*relay2.flamenet.io*)
|
|
echo "$RELAY is a shared test relay, this run WILL create accounts on it." >&2;;
|
|
esac
|
|
|
|
CLASSES="${*:-io.flamenet.messenger.app.FeatureUseTest,io.flamenet.messenger.app.CrudSurfaceTest,io.flamenet.messenger.app.ChatFlowTest}"
|
|
CLASSES="${CLASSES// /,}"
|
|
|
|
export ANDROID_HOME="${ANDROID_HOME:-$HOME/Library/Android/sdk}"
|
|
export JAVA_HOME="${JAVA_HOME:-/opt/homebrew/opt/openjdk@21}"
|
|
ADB="$ANDROID_HOME/platform-tools/adb"
|
|
AVD="${FN_AVD:-fnmsg-android-test}"
|
|
|
|
# `-no-snapshot-load` is the point of this block, and `-no-snapshot-save`
|
|
# keeps the next run clean too. A warm boot is faster and carries the last
|
|
# session's account with it.
|
|
if ! "$ADB" shell getprop sys.boot_completed 2>/dev/null | grep -q 1; then
|
|
echo "### cold-booting $AVD (no snapshot)"
|
|
nohup "$ANDROID_HOME/emulator/emulator" -avd "$AVD" \
|
|
-no-snapshot-load -no-snapshot-save -no-boot-anim \
|
|
-netdelay none -netspeed full >/tmp/fnmsg-emu.log 2>&1 &
|
|
until "$ADB" shell getprop sys.boot_completed 2>/dev/null | grep -q 1; do sleep 5; done
|
|
fi
|
|
|
|
# Unconditionally, on BOTH paths. This uninstall used to live only in the
|
|
# "emulator already up" branch, on the reasoning that a cold boot starts clean.
|
|
# A cold boot starts with no SNAPSHOT; the disk image keeps whatever was
|
|
# installed on it. So an AVD that had been used to test a RELEASE build kept
|
|
# that build, and the debug APK could not install over it:
|
|
#
|
|
# INSTALL_FAILED_UPDATE_INCOMPATIBLE: Existing package
|
|
# io.flamenet.messenger signatures do not match newer version
|
|
#
|
|
# Release and debug are signed by different keys, which is the whole point of
|
|
# them, so this is not a rare collision: it happens every time somebody uses
|
|
# this AVD to try a real build, which is what an AVD is for.
|
|
#
|
|
# The line below already PRINTED the leftover package, and the run continued
|
|
# anyway. A check whose output nobody acts on is decoration.
|
|
"$ADB" uninstall io.flamenet.messenger >/dev/null 2>&1 || true
|
|
"$ADB" uninstall io.flamenet.messenger.test >/dev/null 2>&1 || true
|
|
|
|
echo "### installed packages after clearing:"
|
|
LEFTOVER=$("$ADB" shell pm list packages 2>/dev/null | grep -i flamenet || true)
|
|
if [ -n "$LEFTOVER" ]; then
|
|
echo "$LEFTOVER"
|
|
echo "REFUSED: could not remove the existing install above." >&2
|
|
echo "It will make the test APK fail to install, and the run would report" >&2
|
|
echo "zero tests as though nothing were wrong. Remove it by hand:" >&2
|
|
echo " $ADB uninstall io.flamenet.messenger" >&2
|
|
exit 1
|
|
else
|
|
echo " (nothing installed, good)"
|
|
fi
|
|
|
|
# To LOOK at the emulator while a suite runs, use the console path:
|
|
#
|
|
# adb emu screenrecord screenshot <dir>
|
|
#
|
|
# `adb exec-out screencap -p` returns a black frame on this emulator's GPU
|
|
# path while the display is fine. Observed 2026-09-17: the black capture read
|
|
# as a dead screen and cost an hour; the console screenshot showed the
|
|
# sign-up form, filled in, with the submit button below the fold.
|
|
|
|
echo "### relay $RELAY · classes $CLASSES"
|
|
set +e
|
|
./gradlew --no-daemon :app:connectedDebugAndroidTest \
|
|
-Pandroid.testInstrumentationRunnerArguments.relay="$RELAY" \
|
|
-Pandroid.testInstrumentationRunnerArguments.class="$CLASSES" 2>&1 \
|
|
| tee /tmp/fnmsg-instrumented.log \
|
|
| grep -E "FAILED|PASSED|SKIPPED|BUILD|Finished [0-9]+ test|Tests on"
|
|
STATUS=${PIPESTATUS[0]}
|
|
set -e
|
|
|
|
# A skip is not a pass. Every one of these suites opens with
|
|
# `assumeTrue(relayReachable())`, so an unreachable relay produces a green
|
|
# build that exercised nothing. Say which happened.
|
|
# Anchored to the TEST line, not a bare "SKIPPED". Gradle prints
|
|
# `> Task :app:preDebugAndroidTestBuild SKIPPED` for up-to-date tasks, so the
|
|
# loose grep reported four skipped tests on a run where none were skipped,
|
|
# a verdict written to stop a skip reading as a pass, inventing skips instead.
|
|
SKIPS=$(grep -cE "^io\.flamenet\..*SKIPPED" /tmp/fnmsg-instrumented.log || true)
|
|
RAN=$(grep -oE "Finished [0-9]+ test" /tmp/fnmsg-instrumented.log | tail -1 || echo "")
|
|
echo
|
|
echo "### $RAN, $SKIPS skipped"
|
|
if [ "$SKIPS" != "0" ] && [ "$STATUS" = "0" ]; then
|
|
echo "### VERDICT: SKIPPED, the relay was unreachable. This is NOT a pass."
|
|
exit 3
|
|
elif [ "$STATUS" = "0" ]; then
|
|
echo "### VERDICT: PASSED on device"
|
|
else
|
|
echo "### VERDICT: FAILED, see /tmp/fnmsg-instrumented.log"
|
|
echo "### and app/build/reports/androidTests/connected/debug/index.html"
|
|
fi
|
|
exit $STATUS
|