A weekly check that the privacy properties still hold #112

Merged
paulhitt merged 1 commits from ops/privacy-drift-check into main 2026-09-04 22:11:39 +00:00
paulhitt commented 2026-09-04 22:11:27 +00:00 (Migrated from gitlab.com)

Every property measured in docs/PLAINTEXT.md was true once and could stop
being true quietly. Three already had, on the day it was written, with
nothing reporting it: /metadata promised rosters were in the clear after
the apps had fixed it, the test fleet ran an 11-day-old image while
answering every health check, and buddies held 19 dead tuples against 2
live rows because autovacuum never fires on a small table.

Ten checks, weekly, mailing ONLY on a state change — a report that arrives
every week is a report nobody reads.

The CrowdSec row is the one that most needs a machine. Nothing else
would notice IP banning going dark, and the ordinary way that happens is
somebody improving privacy by dropping the client IP from the access log,
which is the change docs/PLAINTEXT.md argues against.

A check that cannot run reports FAILURE, never a pass. The thing being
guarded against is a script that greps a file which has moved and reports
a clean sweep over nothing.

It caught real drift on its first run: the fleet had fallen behind
production again, because deploying prod does not update the fleet — an
hour after I had bumped it by hand.

Two bugs found writing it, both a monitor reporting on itself rather than
the system: cscli metrics -o json is pretty-printed so a naive grep for
"reads" matches nothing, and systemd expands % in ExecStart so an inline
date +%Y is mangled — the first version wrote "OK /bin/bash" into its
status file.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com

Every property measured in docs/PLAINTEXT.md was true once and could stop being true quietly. Three already had, on the day it was written, with nothing reporting it: /metadata promised rosters were in the clear after the apps had fixed it, the test fleet ran an 11-day-old image while answering every health check, and buddies held 19 dead tuples against 2 live rows because autovacuum never fires on a small table. Ten checks, weekly, mailing ONLY on a state change — a report that arrives every week is a report nobody reads. The CrowdSec row is the one that most needs a machine. Nothing else would notice IP banning going dark, and the ordinary way that happens is somebody improving privacy by dropping the client IP from the access log, which is the change docs/PLAINTEXT.md argues against. A check that cannot run reports FAILURE, never a pass. The thing being guarded against is a script that greps a file which has moved and reports a clean sweep over nothing. It caught real drift on its first run: the fleet had fallen behind production again, because deploying prod does not update the fleet — an hour after I had bumped it by hand. Two bugs found writing it, both a monitor reporting on itself rather than the system: cscli metrics -o json is pretty-printed so a naive grep for "reads" matches nothing, and systemd expands % in ExecStart so an inline date +%Y is mangled — the first version wrote "OK /bin/bash" into its status file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
paulhitt commented 2026-09-04 22:11:27 +00:00 (Migrated from gitlab.com)

assigned to @paulhitt

assigned to @paulhitt
paulhitt commented 2026-09-04 22:11:39 +00:00 (Migrated from gitlab.com)

mentioned in commit eb54930a57

mentioned in commit eb54930a5710c4d9a2176ff8c28e2dcfff115d2f
paulhitt (Migrated from gitlab.com) merged commit eb54930a57 into main 2026-09-04 22:11:39 +00:00
paulhitt commented 2026-09-09 18:59:00 +00:00 (Migrated from gitlab.com)

changed the description

changed the description
Sign in to join this conversation.