CI cache volumes filled the disk the production database lives on #61

New Issue

2026-09-08T11:52:12Z

paulhitt commented

2026-09-08 11:52:12 +00:00

(Migrated from gitlab.com)

What happened

2026-09-08: /data hit 100% and took Postgres down (see the sibling issue on /health). The relay, its database, Redis, coturn, the media stack and the GitLab runner's build cache all share /dev/sdb.

The space was not the application's:

  41.7G  runner-294da33c…-cache-3c3f060a…-protected
   8.6G  runner-9fc4d3d1…-cache-3c3f060a…-protected
   1.5G  runner-dfafcb4d…-cache-c33bcaa1…-protected
   … 40 runner cache volumes in total

339 of 386 Docker volumes were unattached. Removing only the runner-*-cache-* ones — regenerable CI cache, nothing else — took /data from 0 bytes free to 70G.

Two compounding causes

Runner cache is unbounded and never expires. A new runner registration starts a new cache volume and the old one is never collected, so this grows with every re-registration, forever.
The reserve hid how close it was. /dev/sdb had ext4's default 5% reserved for root — 10.7G on a 200G volume. df therefore read 0 avail while 10.7G existed, and Postgres, running as non-root, got ENOSPC. Set to 1% (tune2fs -m 1 /dev/sdb), which released 8.5G immediately and is what let the database recover before the volumes were cleared. Reversible with tune2fs -m 5.

⚠️ I made this worse yesterday. Raising ovh-flamenetmessenger from 2 concurrent jobs to 3 (to clear the queue behind relay#6) means more concurrent builds and more cache churn on the same disk.

Worth fixing rather than repeating

A periodic sweep of runner-*-cache-* volumes, or [runners.docker] disable_cache / a cache size cap.
🔴 Never a blanket docker volume prune — 339 unattached volumes included application data. The sweep must match the runner cache pattern by name.
An alert on /data above ~85%. Nothing watched this; it was found by eye.
Consider moving CI off the volume the production database is on. They have no business sharing a disk.

## What happened 2026-09-08: `/data` hit 100% and took Postgres down (see the sibling issue on `/health`). The relay, its database, Redis, coturn, the media stack and **the GitLab runner's build cache all share `/dev/sdb`**. The space was not the application's: ``` 41.7G runner-294da33c…-cache-3c3f060a…-protected 8.6G runner-9fc4d3d1…-cache-3c3f060a…-protected 1.5G runner-dfafcb4d…-cache-c33bcaa1…-protected … 40 runner cache volumes in total ``` 339 of 386 Docker volumes were unattached. Removing **only** the `runner-*-cache-*` ones — regenerable CI cache, nothing else — took `/data` from 0 bytes free to 70G. ## Two compounding causes 1. **Runner cache is unbounded and never expires.** A new runner registration starts a new cache volume and the old one is never collected, so this grows with every re-registration, forever. 2. **The reserve hid how close it was.** `/dev/sdb` had ext4's default 5% reserved for root — 10.7G on a 200G volume. `df` therefore read `0 avail` while 10.7G existed, and Postgres, running as non-root, got ENOSPC. Set to 1% (`tune2fs -m 1 /dev/sdb`), which released 8.5G immediately and is what let the database recover before the volumes were cleared. Reversible with `tune2fs -m 5`. ⚠️ **I made this worse yesterday.** Raising `ovh-flamenetmessenger` from 2 concurrent jobs to 3 (to clear the queue behind relay#6) means more concurrent builds and more cache churn on the same disk. ## Worth fixing rather than repeating - A periodic sweep of `runner-*-cache-*` volumes, or `[runners.docker] disable_cache` / a cache size cap. - 🔴 **Never a blanket `docker volume prune`** — 339 unattached volumes included application data. The sweep must match the runner cache pattern by name. - An alert on `/data` above ~85%. Nothing watched this; it was found by eye. - Consider moving CI off the volume the production database is on. They have no business sharing a disk.

paulhitt commented

2026-09-08 12:20:45 +00:00

(Migrated from gitlab.com)

mentioned in merge request !94

paulhitt commented

2026-09-08 12:53:03 +00:00

(Migrated from gitlab.com)

Corrected timing. I originally described this as an overnight outage running for hours. It was 11:13–11:49 UTC (07:13–07:49 local), about 36 minutes, with 13,559 PANICs in that window. I inferred "overnight" from docker compose ps showing the relay "Up 11 hours" instead of reading the log timestamps. The 04:04 security-audit scheduled task ran three hours before it began, so it had nothing to miss.

Done since:

Monitor reads .State.Health.Status instead of .State.Running (which is true for a crash-looping container), and watches /data and / at 90%. Live on the box.
Self-heals at 93%: the five-minute check runs the runner-cache sweep itself before the wall at 100%. Drilled with the threshold lowered to 60% and the mail path stubbed — it detected 67%, swept, and logged both the trigger and the result with nobody involved.
Daily sweep at 04:30 (prune-runner-cache.timer), installed and run once: removed 8 volumes, ~1.9G, left all 350 other volumes alone.
Both scripts now live in Scripts/monitoring/ — they existed only on the box, so the one script deciding whether anybody hears about an outage was under no review. flamenet-relay!94.
A 02:13 scheduled session checks production nightly for the things a shell script cannot reason about, and may perform only the cache sweep.

Measured growth rate: 3.7G of runner cache accumulated in the few hours of CI after the disk was cleared that morning. That is why the sweep is daily rather than weekly — weekly would let ~50G build up, which is the size that filled the disk.

Still open here: /data is shared by the relay, its database, Redis, coturn, the media stack and the CI runner. The sweep and the alert make that survivable; they do not make it right. Moving CI off the production volume is the actual fix.

**Corrected timing.** I originally described this as an overnight outage running for hours. It was **11:13–11:49 UTC (07:13–07:49 local), about 36 minutes**, with 13,559 PANICs in that window. I inferred "overnight" from `docker compose ps` showing the relay "Up 11 hours" instead of reading the log timestamps. The 04:04 `security-audit` scheduled task ran three hours before it began, so it had nothing to miss. **Done since:** - Monitor reads `.State.Health.Status` instead of `.State.Running` (which is true for a crash-looping container), and watches `/data` and `/` at 90%. Live on the box. - **Self-heals at 93%**: the five-minute check runs the runner-cache sweep itself before the wall at 100%. Drilled with the threshold lowered to 60% and the mail path stubbed — it detected 67%, swept, and logged both the trigger and the result with nobody involved. - Daily sweep at 04:30 (`prune-runner-cache.timer`), installed and run once: removed 8 volumes, ~1.9G, left all 350 other volumes alone. - Both scripts now live in `Scripts/monitoring/` — they existed only on the box, so the one script deciding whether anybody hears about an outage was under no review. flamenet-relay!94. - A 02:13 scheduled session checks production nightly for the things a shell script cannot reason about, and may perform only the cache sweep. **Measured growth rate:** 3.7G of runner cache accumulated in the few hours of CI after the disk was cleared that morning. That is why the sweep is daily rather than weekly — weekly would let ~50G build up, which is the size that filled the disk. **Still open here:** `/data` is shared by the relay, its database, Redis, coturn, the media stack and the CI runner. The sweep and the alert make that survivable; they do not make it right. Moving CI off the production volume is the actual fix.

paulhitt commented

2026-09-08 17:13:16 +00:00

(Migrated from gitlab.com)

mentioned in issue #6

paulhitt commented

2026-09-09 21:37:39 +00:00

(Migrated from gitlab.com)

Re-checked against the box. The remediation is in place and running; recording the evidence rather than closing, since whether the issue is done is the reporter's call.

prune-runner-cache.timer is enabled, last fired 2026-09-09 04:31 UTC, next 2026-09-10 04:35 UTC. Daily, as the unit's own comment argues for.
/data is at 70% (135G used, 60G free), against the 100% that took Postgres down.

Worth noting the timer file's comment lost a character to an emoji sweep at some point and now opens mid-sentence; the reasoning it carries — daily rather than weekly, because CI accumulates roughly 3.7G in a few hours and a weekly sweep would let ~50G build between runs — is still correct and still worth keeping.

Re-checked against the box. **The remediation is in place and running**; recording the evidence rather than closing, since whether the issue is done is the reporter's call. - `prune-runner-cache.timer` is **enabled**, last fired 2026-09-09 04:31 UTC, next 2026-09-10 04:35 UTC. Daily, as the unit's own comment argues for. - `/data` is at **70%** (135G used, 60G free), against the 100% that took Postgres down. Worth noting the timer file's comment lost a character to an emoji sweep at some point and now opens mid-sentence; the reasoning it carries — daily rather than weekly, because CI accumulates roughly 3.7G in a few hours and a weekly sweep would let ~50G build between runs — is still correct and still worth keeping.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: extant2000/flamenet-relay#61