ci: the staging gate waits for Postgres over TCP, from another container, twice
ci / checks (pull_request) Successful in 12s
ci / swift (pull_request) Successful in 8m29s
ci / image (pull_request) Successful in 52s
ci / deploy (pull_request) Skipped

The image job is red on main and the deploy has not run. The gate's readiness
check was wrong in both directions, and the two failures looked nothing alike.

On mac1 `docker exec <db> pg_isready` never succeeded inside 90 s, and the
gate said only "never became ready". On mac2, with the diagnostics from #253
in place, it succeeded 2.4 seconds after `docker run`, and the candidate's own
log says what happened next:

  Opening new connection for pool failed ... Connection refused (errno: 111)
  Fatal error: Error raised at top level: PSQLError(code: connectionError ...)

The postgres image runs initdb against a bootstrap server on the unix socket
and then restarts, so a local-socket `pg_isready` can answer "accepting
connections" while nothing is listening on TCP. The relay only speaks TCP, and
it exits on a refused first connection rather than retrying, so that one early
answer ended the run: the 90 seconds that followed were spent waiting for
/health from a container that had already died.

Readiness is now checked the way the relay will use it, over TCP from a
separate container on the gate's network, and twice in a row so the restart in
the middle of the init cycle is not mistaken for the end of it.

Worth a separate decision, not changed here: the relay exits on one refused
connection to Postgres. In production that is a crashloop until the database
answers, which is survivable and noisy.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Paul Hitt
2026-10-14 12:24:06 -04:00
co-authored by Claude Opus 5
parent 6d1013dc89
commit c4d50f45f7
+37 -9
View File
@@ -37,23 +37,51 @@ docker run -d --name "${STACK}-db" --network "$NET" \
postgres:17-alpine >/dev/null
docker run -d --name "${STACK}-redis" --network "$NET" redis:7-alpine >/dev/null
echo "GATE: waiting for staging postgres"
for i in $(seq 1 45); do
docker exec "${STACK}-db" pg_isready -U staging >/dev/null 2>&1 && break
# Ready means ready OVER TCP, FROM ANOTHER CONTAINER, twice.
#
# `docker exec <db> pg_isready` was wrong in both directions, and the two
# failures looked nothing alike:
#
# - on mac1 it never succeeded inside 90 s and the gate said only "never
# became ready", so 90 seconds of waiting read as a relay bug;
# - on mac2 it succeeded 3 seconds after `docker run`, the relay dialled
# 5432, got ECONNREFUSED and died with a fatal error at top level.
#
# The second is the one that cost the deploy. The postgres image runs initdb
# against a bootstrap server on the unix socket and then restarts, so a
# local-socket `pg_isready` can answer "accepting connections" while nothing is
# listening on TCP yet. The relay only ever speaks TCP, and it exits on a
# refused first connection rather than retrying, so one early answer here is
# enough to end the run: the 90 s that followed was spent waiting on a
# container that had already died.
#
# So readiness is checked the way the relay will use it, over TCP from another
# container, and twice in a row, because the restart in the middle of the init
# cycle must not be mistaken for the end of it.
echo "GATE: waiting for staging postgres over TCP"
ready=0
for i in $(seq 1 60); do
if docker run --rm --network "$NET" postgres:17-alpine \
pg_isready -h "${STACK}-db" -p 5432 -U staging >/dev/null 2>&1; then
ready=$((ready + 1))
[ "$ready" -ge 2 ] && break
else
ready=0
fi
sleep 2
done
if ! docker exec "${STACK}-db" pg_isready -U staging >/dev/null 2>&1; then
# Say WHICH of the two it was. "never became ready" covered both a Postgres
# that was still starting and a container that never ran at all, and the two
# need different fixes. mac1 failed here with no further word, which is how
# 90 s of waiting read as a relay bug.
if [ "$ready" -lt 2 ]; then
# Which of the two it was. "Never became ready" covered both a Postgres
# still starting and a container that never ran at all, and those need
# different fixes.
echo "--- staging postgres state ---"
docker inspect -f 'state={{.State.Status}} exit={{.State.ExitCode}} error={{.State.Error}}' \
"${STACK}-db" 2>&1 || echo "the container does not exist"
echo "--- staging postgres log ---"
docker logs --tail 40 "${STACK}-db" 2>&1 || true
fail "staging postgres never became ready"
fail "staging postgres never accepted a TCP connection"
fi
echo "GATE: staging postgres is accepting TCP connections"
# Throwaway issuer keypair, minted per run and never stored.
PRIV_PEM=$(openssl genpkey -algorithm ed25519 2>/dev/null)