The monitor could not see the outage it existed to catch (relay#61) #157

Merged
paulhitt merged 3 commits from fix/monitoring-could-not-see-the-outage into main 2026-09-08 12:53:47 +00:00
paulhitt commented 2026-09-08 12:20:44 +00:00 (Migrated from gitlab.com)

Part of #61.

messenger-health.sh passed every check straight through the outage. .State.Running is true for a crash-looping container, and Docker had already marked the database (unhealthy) — the script never asked. Nothing watched free space at all.

Also brings the script into the repo; it lived only on the box, so the one script deciding whether anybody hears about an outage was under no review.

Part of #61. `messenger-health.sh` passed every check straight through the outage. `.State.Running` is true for a crash-looping container, and Docker had already marked the database `(unhealthy)` — the script never asked. Nothing watched free space at all. Also brings the script into the repo; it lived only on the box, so the one script deciding whether anybody hears about an outage was under no review.
paulhitt commented 2026-09-08 12:25:46 +00:00 (Migrated from gitlab.com)

added 1 commit

  • 4ecb762d - Sweep the runner cache daily, so the disk cannot fill again (relay#61)

Compare with previous version

added 1 commit <ul><li>4ecb762d - Sweep the runner cache daily, so the disk cannot fill again (relay#61)</li></ul> [Compare with previous version](/paulhitt/flamenet-relay/-/merge_requests/94/diffs?diff_id=2011353399&start_sha=7bf8f597906f5d03cc9b6829dcdf80280ad3b9de)
paulhitt commented 2026-09-08 12:29:40 +00:00 (Migrated from gitlab.com)

added 1 commit

  • 8e05e7ee - Sweep automatically when the disk is about to fill (relay#61)

Compare with previous version

added 1 commit <ul><li>8e05e7ee - Sweep automatically when the disk is about to fill (relay#61)</li></ul> [Compare with previous version](/paulhitt/flamenet-relay/-/merge_requests/94/diffs?diff_id=2011364863&start_sha=4ecb762d20c3e63d0109a2fa162b1ae047831eb0)
paulhitt commented 2026-09-08 12:53:04 +00:00 (Migrated from gitlab.com)

mentioned in issue #61

mentioned in issue #61
paulhitt commented 2026-09-08 12:53:47 +00:00 (Migrated from gitlab.com)

mentioned in commit f641990f63

mentioned in commit f641990f6357e0e9c4fe598f57c77fc9eb15186b
paulhitt (Migrated from gitlab.com) merged commit f641990f63 into main 2026-09-08 12:53:47 +00:00
Sign in to join this conversation.