Sweep automatically when the disk is about to fill (relay#61)

The outage began at no particular hour and nobody was awake for it. A daily
sweep prevents the slow build-up, but a burst of CI can cross the line
between runs -- so the five-minute health check now acts when it sees the
edge coming, rather than only writing it down.

Warn at 90%, act at 93%, wall at 100%: a person hears about it before the
machine intervenes, and the alert mail says what was swept rather than
quietly hiding that anything happened.

⚠️ This is the ONLY thing here that self-heals, and only because it cannot
lose anything -- runner cache is regenerable by definition, so the cost of
acting wrongly is one slower pipeline. Nothing else in this script is safe
to automate. A monitor that starts deleting things it has not reasoned
about is worse than the outage it is trying to prevent.

Drilled on the box with the threshold lowered to 60% and the mail path
stubbed: it detected /data at 67%, ran the sweep, removed 2 volumes and
logged both the trigger and the result, with no human involved.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Scripts/monitoring/messenger-health.sh

@@ -53,6 +53,32 @@ disk_ok() {
   [ -n "$used" ] && [ "$used" -lt "${2:-90}" ]
 }
 
+# 🔴 **Self-healing, for the one failure that has actually happened.**
+# 2026-09-08: /data reached 100%, Postgres could not checkpoint, and it
+# crash-looped in recovery for hours. Nobody was awake. A daily sweep prevents
+# the slow build-up, but a burst of CI can cross the line between sweeps — so
+# this acts the moment it sees the edge coming, five minutes at worst.
+#
+# ⚠️ **Only this one remediation, and only because it cannot lose anything.**
+# Runner cache is regenerable by definition: the cost of being wrong is one
+# slower pipeline. Nothing else here is safe to automate, and a monitor that
+# starts deleting things it has not reasoned about is worse than the outage.
+#
+# ⚠️ It acts BELOW the alert threshold on purpose — warn at 90, act at 93, wall
+# at 100 — so a person hears about it before the machine has to intervene, and
+# the mail below says what it did rather than quietly hiding the problem.
+ACT_AT=93
+swept=""
+disk_used=$(df --output=pcent /data 2>/dev/null | tail -1 | tr -dc '0-9')
+if [ -n "$disk_used" ] && [ "$disk_used" -ge "$ACT_AT" ] \
+   && [ -x /srv/prod/monitoring/prune-runner-cache.sh ]; then
+  log "disk-critical: /data at ${disk_used}% — sweeping runner cache"
+  result=$(/srv/prod/monitoring/prune-runner-cache.sh 2>&1 | tail -1)
+  now=$(df --output=pcent /data 2>/dev/null | tail -1 | tr -dc '0-9')
+  log "disk-critical: $result; /data ${disk_used}% -> ${now}%"
+  swept=" (acted: swept runner cache, /data ${disk_used}% -> ${now}%)"
+fi
+
 check relay-health   curl -sfk --max-time 10 --resolve relay.flamenet.io:443:127.0.0.1 https://relay.flamenet.io/health
 check landing        curl -sfk --max-time 10 --resolve flamenetmessenger.com:443:127.0.0.1 https://flamenetmessenger.com/
 check apk            curl -sfkI --max-time 10 --resolve flamenetmessenger.com:443:127.0.0.1 https://flamenetmessenger.com/downloads/flamenet-messenger.apk
@@ -73,8 +99,8 @@ else
   printf 'FAIL %s ::%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$fails" > "$STATUS"
   log "FAIL:$fails"
   if [ "$prev" != "FAIL" ]; then
-    printf 'Subject: [flamenet] messenger FAILING:%s\n\nFailed checks:%s\nAt: %s\nSee /srv/prod/monitoring/health.log on the box.\n' \
-      "$fails" "$fails" "$(date -u)" \
+    printf 'Subject: [flamenet] messenger FAILING:%s\n\nFailed checks:%s\nAt: %s%s\nSee /srv/prod/monitoring/health.log on the box.\n' \
+      "$fails" "$fails" "$(date -u)" "$swept" \
       | docker exec -i mailserver sendmail "$ALERT_TO" || log "alert mail failed"
   fi
 fi