Drain connections on deploy (relay#33) #96

2026-09-04T12:45:23Z

paulhitt commented

2026-09-04 12:45:23 +00:00

(Migrated from gitlab.com)

The problem is the long polls, and it is subtler than "requests get
killed". A phone waiting for messages holds a connection open for up
to half a minute. When a deploy recreates the container every one of
those is severed at the socket — and from the client's side a severed
socket is indistinguishable from a network failure, so it backs off
before retrying. A deploy therefore costs every connected device a delay
it reads as the network being unreliable, and they all come back at the
same moment.

So draining COMPLETES the long polls rather than killing them. A
poll that returns 200 with nothing in it is an ordinary empty answer:
the client reconnects immediately and cleanly to whichever process is
now listening. The difference between that and a dropped socket is the
whole feature.

Nothing is refused while draining. Refusing turns a deploy into
visible errors on people's phones, which is the cost this exists to
avoid. New polls are answered at once rather than held — correct rather
than merely polite, because holding a connection on a process about to
exit guarantees the severed socket.

Health reports draining as its own state, not as unhealthy. A
proxy should stop sending NEW connections to a draining instance while
letting the ones it has finish. Reporting unhealthy would make a deploy
look like an outage, and would take the instance out before its polls
had been released.

The lifecycle handler is registered FIRST so it runs LAST on the way
down. Vapor unwinds handlers in reverse, and a drain flag set after the
pools have closed is one nobody can act on — the polls it exists to
release have already failed on a dead connection.

A second signal does not restart the clock. A deploy can send more
than one, and a drain that kept resetting would never look old enough
for an operator to act on.

567 relay tests in 50 suites, green — including all seven core paths.

**The problem is the long polls, and it is subtler than "requests get killed".** A phone waiting for messages holds a connection open for up to half a minute. When a deploy recreates the container every one of those is severed at the socket — and from the client's side a severed socket is indistinguishable from a network failure, so it backs off before retrying. A deploy therefore costs every connected device a delay it reads as the network being unreliable, and they all come back at the same moment. **So draining COMPLETES the long polls rather than killing them.** A poll that returns 200 with nothing in it is an ordinary empty answer: the client reconnects immediately and cleanly to whichever process is now listening. The difference between that and a dropped socket is the whole feature. **Nothing is refused while draining.** Refusing turns a deploy into visible errors on people's phones, which is the cost this exists to avoid. New polls are answered at once rather than held — correct rather than merely polite, because holding a connection on a process about to exit guarantees the severed socket. **Health reports `draining` as its own state, not as unhealthy.** A proxy should stop sending NEW connections to a draining instance while letting the ones it has finish. Reporting unhealthy would make a deploy look like an outage, and would take the instance out before its polls had been released. The lifecycle handler is registered FIRST so it runs LAST on the way down. Vapor unwinds handlers in reverse, and a drain flag set after the pools have closed is one nobody can act on — the polls it exists to release have already failed on a dead connection. A second signal does not restart the clock. A deploy can send more than one, and a drain that kept resetting would never look old enough for an operator to act on. 567 relay tests in 50 suites, green — including all seven core paths.

paulhitt commented

2026-09-04 12:45:23 +00:00

(Migrated from gitlab.com)

assigned to @paulhitt

paulhitt commented

2026-09-04 12:45:36 +00:00

(Migrated from gitlab.com)

restored source branch feat/drain-connections-on-deploy

restored source branch `feat/drain-connections-on-deploy`

paulhitt commented

2026-09-04 20:40:35 +00:00

(Migrated from gitlab.com)

deleted the feat/validate-attachments branch. This merge request now targets the main branch

deleted the `feat/validate-attachments` branch. This merge request now targets the `main` branch

paulhitt (Migrated from gitlab.com) merged commit 4aff413bab into main

2026-09-04 20:40:37 +00:00

paulhitt commented

2026-09-04 20:40:38 +00:00

(Migrated from gitlab.com)

mentioned in commit 4aff413bab

mentioned in commit 4aff413bab3d59f29116ff767ec017bd093858b2

paulhitt commented

2026-09-09 18:59:13 +00:00

(Migrated from gitlab.com)

changed the description

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: extant2000/flamenet-relay#96