Sweep the runner cache daily, so the disk cannot fill again (relay#61)

The alert added alongside this tells you the disk is filling. This is what
stops it. Runner cache volumes grow without limit and share /data with the
production database; two had reached 41.7G and 8.6G when the disk hit 100%
and took Postgres down.

🔴 It is NOT a volume prune. 339 of 386 volumes were unattached that morning
and most hold real data -- a stopped service's database is unattached too.
This matches the runner cache pattern by NAME and requires the volume to be
unattached as well. Both conditions, every time.

Daily rather than weekly, measured rather than guessed: 3.7G of runner cache
accumulated in the few hours of CI after the disk was cleared this morning.
Weekly would let ~50G build up between runs, which is the size that caused
the outage.

Verified on the box: dry run listed only `runner-*-cache-*`, the real run
removed 8 volumes and ~1.9G, and all 350 other volumes were left alone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Scripts/monitoring/prune-runner-cache.sh

@@ -0,0 +1,50 @@
+#!/usr/bin/env bash
+# Reclaim GitLab-runner cache volumes, which grow without limit.
+#
+# 🔴 **This is what took the relay down on 2026-09-08.** The runner's cache
+# volumes share /data with the production database. One had reached 41.7G and
+# another 8.6G; 339 of 386 volumes were unattached. The disk hit 100%, Postgres
+# could not checkpoint, and it crash-looped in recovery for hours.
+#
+# 🔴 **NEVER `docker volume prune`.** Most of those 339 unattached volumes hold
+# real data — a stopped service's database is unattached too. This removes only
+# volumes whose NAME matches the runner cache pattern, and only when they are
+# unattached. Both conditions, every time.
+#
+# ⚠️ Cache is regenerable by definition: the next job repopulates it. The cost
+# of removing it is one slower pipeline, which is why this can run unattended
+# and a volume sweep cannot.
+set -uo pipefail
+
+DRY=0
+[ "${1:-}" = "--dry-run" ] && DRY=1
+LOG=/srv/prod/monitoring/prune-runner-cache.log
+PATTERN='^runner-.*-cache-'
+
+log() { printf '%s  %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" >> "$LOG" 2>/dev/null || true; }
+
+before=$(df --output=avail /data 2>/dev/null | tail -1 | tr -dc '0-9')
+removed=0
+freed=0
+
+# `dangling=true` is the safety rail: a volume attached to any container, even
+# a stopped one, is not listed. A job holding its cache mid-run keeps it.
+while read -r vol; do
+  [ -n "$vol" ] || continue
+  size=$(du -sm "/var/lib/docker/volumes/$vol/_data" 2>/dev/null | cut -f1)
+  [ -n "$size" ] || size=$(du -sm "/data/docker/volumes/$vol/_data" 2>/dev/null | cut -f1)
+  [ -n "$size" ] || size=0
+  if [ "$DRY" = "1" ]; then
+    printf '  would remove %6sM  %s\n' "$size" "$vol"
+  else
+    docker volume rm "$vol" >/dev/null 2>&1 && { removed=$((removed+1)); freed=$((freed+size)); }
+  fi
+done < <(docker volume ls -qf dangling=true 2>/dev/null | grep -E "$PATTERN")
+
+after=$(df --output=avail /data 2>/dev/null | tail -1 | tr -dc '0-9')
+if [ "$DRY" = "1" ]; then
+  echo "dry run — nothing removed"
+else
+  log "removed $removed runner cache volume(s), ~${freed}M; /data avail $((before/1024))M -> $((after/1024))M"
+  echo "removed $removed runner cache volume(s), ~${freed}M freed"
+fi

Scripts/monitoring/systemd/prune-runner-cache.service

@@ -0,0 +1,7 @@
+[Unit]
+Description=Reclaim GitLab-runner cache volumes (relay#61)
+Documentation=https://gitlab.com/paulhitt/flamenet-relay/-/issues/61
+
+[Service]
+Type=oneshot
+ExecStart=/srv/prod/monitoring/prune-runner-cache.sh

Scripts/monitoring/systemd/prune-runner-cache.timer

@@ -0,0 +1,16 @@
+[Unit]
+Description=Daily sweep of GitLab-runner cache volumes
+
+[Timer]
+# 🔴 Daily, not weekly. Measured 2026-09-08: 3.7G of runner cache accumulated
+# in the few hours of CI after the disk was cleared that morning. A weekly
+# sweep would let ~50G build up between runs, which is the size that filled
+# the disk and took the database down in the first place.
+OnCalendar=*-*-* 04:30:00
+# The box is not always up at 04:30, and a sweep that silently never runs is
+# the same as no sweep.
+Persistent=true
+RandomizedDelaySec=600
+
+[Install]
+WantedBy=timers.target