The deploy waits for the edge to answer, not for eight seconds #220

2026-09-22T18:41:41Z

paulhitt commented

2026-09-22 18:41:41 +00:00

(Migrated from gitlab.com)

A production deploy today passed the whole gate and then rolled itself back. The build was fine. sleep 8 was not.

Measured on the box while working out why:

container started   18:37:00.9
first healthy probe 18:37:16.7

A recreated relay needs about sixteen seconds before it answers /health. The check waited eight, read the edge's 502 as a broken deploy, and reverted a build that had just passed signup, signin, 47 rendered pages and a 25 second stability window.

This was not a flake. As written the check could not succeed. The same deploy passed on the second attempt with nothing changed but the wait, and the check itself is sound: run by hand against the running relay it answered ok three times out of three.

A deadline loop instead, leaving as soon as the edge says ok. A healthy deploy pays nothing, usually under twenty seconds. Only a deploy that is actually broken spends the budget, which is how the rollback stays correct rather than becoming something people learn to ignore.

Same bug as 45194792 in the iOS suite, where a fire-and-forget revoke was waited on for a fixed two seconds instead of until it landed.

Note on the box. The fixed script is already installed at /srv/prod/messenger/src/Scripts/staging-gate.sh, because it had to be to complete today's deploy. Merging this closes the divergence rather than opening it. Previous script backed up at /srv/prod/messenger/staging-gate.sh.bak-20260922.

Generated with Claude Code.

A production deploy today passed the whole gate and then rolled itself back. The build was fine. `sleep 8` was not. **Measured on the box while working out why:** ``` container started 18:37:00.9 first healthy probe 18:37:16.7 ``` A recreated relay needs about sixteen seconds before it answers `/health`. The check waited eight, read the edge's 502 as a broken deploy, and reverted a build that had just passed signup, signin, 47 rendered pages and a 25 second stability window. **This was not a flake.** As written the check could not succeed. The same deploy passed on the second attempt with nothing changed but the wait, and the check itself is sound: run by hand against the running relay it answered `ok` three times out of three. A deadline loop instead, leaving as soon as the edge says ok. A healthy deploy pays nothing, usually under twenty seconds. Only a deploy that is actually broken spends the budget, which is how the rollback stays correct rather than becoming something people learn to ignore. Same bug as `45194792` in the iOS suite, where a fire-and-forget revoke was waited on for a fixed two seconds instead of until it landed. **Note on the box.** The fixed script is already installed at `/srv/prod/messenger/src/Scripts/staging-gate.sh`, because it had to be to complete today's deploy. Merging this closes the divergence rather than opening it. Previous script backed up at `/srv/prod/messenger/staging-gate.sh.bak-20260922`. Generated with Claude Code.

paulhitt commented

2026-09-22 19:20:57 +00:00

(Migrated from gitlab.com)

mentioned in commit f9c45d1dc1

mentioned in commit f9c45d1dc1c250f62bf4fb77515b023bba8eee1b

paulhitt (Migrated from gitlab.com) merged commit f9c45d1dc1 into main

2026-09-22 19:20:57 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: extant2000/flamenet-relay#220