ci: the staging gate waits for Postgres over TCP, from another container, twice #255

2026-10-14T16:24:17Z

extant2000git commented

2026-10-14 16:24:17 +00:00

The image job is red on main and the deploy has not run, so production is still on the image before wave 2. The gate's Postgres readiness check is the cause, and it was wrong in both directions.

runner	what `docker exec <db> pg_isready` did	what the gate said
mac1	never succeeded inside 90 s	"staging postgres never became ready"
mac2	succeeded 2.4 s after `docker run`	"candidate never answered /health within 90s"

The diagnostics from #253 are what made the second one readable. The candidate's own log:

Opening new connection for pool failed ... Connection refused (errno: 111)
Fatal error: Error raised at top level: PSQLError(code: connectionError ...)

The postgres image runs initdb against a bootstrap server on the unix socket and then restarts, so a local-socket pg_isready can answer "accepting connections" while nothing is listening on TCP. The relay only speaks TCP and exits on a refused first connection, so that one early answer ended the run; the 90 seconds after it were spent waiting for /health from a container that had already died.

Readiness is now checked the way the relay will use it: pg_isready -h <db> -p 5432 from a separate container on the gate's network, and twice in a row, so the restart in the middle of the init cycle is not read as the end of it.

One thing deliberately not changed here. The relay exits on a single refused connection to Postgres rather than retrying. In production that is a crashloop until the database answers. Survivable, noisy, and a separate decision.

🤖 Generated with Claude Code

The image job is red on main and the deploy has not run, so production is still on the image before wave 2. The gate's Postgres readiness check is the cause, and it was wrong in both directions. | runner | what `docker exec <db> pg_isready` did | what the gate said | |---|---|---| | mac1 | never succeeded inside 90 s | "staging postgres never became ready" | | mac2 | succeeded 2.4 s after `docker run` | "candidate never answered /health within 90s" | The diagnostics from #253 are what made the second one readable. The candidate's own log: ``` Opening new connection for pool failed ... Connection refused (errno: 111) Fatal error: Error raised at top level: PSQLError(code: connectionError ...) ``` The postgres image runs initdb against a bootstrap server on the unix socket and then restarts, so a local-socket `pg_isready` can answer "accepting connections" while nothing is listening on TCP. The relay only speaks TCP and exits on a refused first connection, so that one early answer ended the run; the 90 seconds after it were spent waiting for `/health` from a container that had already died. Readiness is now checked the way the relay will use it: `pg_isready -h <db> -p 5432` from a separate container on the gate's network, and twice in a row, so the restart in the middle of the init cycle is not read as the end of it. **One thing deliberately not changed here.** The relay exits on a single refused connection to Postgres rather than retrying. In production that is a crashloop until the database answers. Survivable, noisy, and a separate decision. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

extant2000git added 1 commit 2026-10-14 16:24:18 +00:00

ci: the staging gate waits for Postgres over TCP, from another container, twice

ci / checks (pull_request) Successful in 12s

Details

ci / swift (pull_request) Successful in 8m29s

Details

ci / image (pull_request) Successful in 52s

Details

ci / deploy (pull_request) Skipped

Details

c4d50f45f7

The image job is red on main and the deploy has not run. The gate's readiness
check was wrong in both directions, and the two failures looked nothing alike.

On mac1 `docker exec <db> pg_isready` never succeeded inside 90 s, and the
gate said only "never became ready". On mac2, with the diagnostics from #253
in place, it succeeded 2.4 seconds after `docker run`, and the candidate's own
log says what happened next:

  Opening new connection for pool failed ... Connection refused (errno: 111)
  Fatal error: Error raised at top level: PSQLError(code: connectionError ...)

The postgres image runs initdb against a bootstrap server on the unix socket
and then restarts, so a local-socket `pg_isready` can answer "accepting
connections" while nothing is listening on TCP. The relay only speaks TCP, and
it exits on a refused first connection rather than retrying, so that one early
answer ended the run: the 90 seconds that followed were spent waiting for
/health from a container that had already died.

Readiness is now checked the way the relay will use it, over TCP from a
separate container on the gate's network, and twice in a row so the restart in
the middle of the init cycle is not mistaken for the end of it.

Worth a separate decision, not changed here: the relay exits on one refused
connection to Postgres. In production that is a crashloop until the database
answers, which is survivable and noisy.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

extant2000git merged commit a13e396023 into main

2026-10-14 16:33:58 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: extant2000/flamenet-relay#255