Cut the edge IP retention window from a week to a day (relay#19) #111

2026-09-04T22:06:35Z

paulhitt commented

2026-09-04 22:06:35 +00:00

(Migrated from gitlab.com)

Traefik logs every request with the client IP and User-Agent — app
traffic included, from people who never open the website. Measured: 590 MB
across 7 files, pruned weekly. Now pruned daily.

The obvious fix would have silently disarmed CrowdSec. Traefik can
drop ClientHost, which removes every IP and keeps the User-Agent the
install-funnel analysis needs — it looks like a complete win. But CrowdSec
acquires from this exact container log (source: docker, container_name: edge-traefik-1), so dropping the field leaves it parsing lines with no
address, banning nobody, with nothing reporting a problem.

The log serves two masters with opposite needs: the funnel wants the
User-Agent and not the IP, CrowdSec wants the IP and not the User-Agent.
One stream, so the only lever that helps both is TIME.

Truncating a file a tailer is reading can leave it seeking past EOF, so
that was verified rather than assumed: across the change with fresh
traffic, CrowdSec's reads went 1,642,488 -> 1,642,529 and parsed
1,585,311 -> 1,585,351.

And cscli metrics rounds to 1.64M, which a dozen lines cannot
move — the check only means anything with -o json.

Cleared while here: geogate mounts read-only and writes nothing
persistent (56 KB container log); CrowdSec's own alert store is
self-limiting at ~6 hours.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com

Traefik logs every request with the client IP and User-Agent — app traffic included, from people who never open the website. Measured: 590 MB across 7 files, pruned weekly. Now pruned daily. **The obvious fix would have silently disarmed CrowdSec.** Traefik can drop `ClientHost`, which removes every IP and keeps the User-Agent the install-funnel analysis needs — it looks like a complete win. But CrowdSec acquires from this exact container log (`source: docker, container_name: edge-traefik-1`), so dropping the field leaves it parsing lines with no address, banning nobody, with nothing reporting a problem. The log serves two masters with opposite needs: the funnel wants the User-Agent and not the IP, CrowdSec wants the IP and not the User-Agent. One stream, so the only lever that helps both is TIME. Truncating a file a tailer is reading can leave it seeking past EOF, so that was verified rather than assumed: across the change with fresh traffic, CrowdSec's reads went 1,642,488 -> 1,642,529 and parsed 1,585,311 -> 1,585,351. And `cscli metrics` rounds to `1.64M`, which a dozen lines cannot move — the check only means anything with `-o json`. Cleared while here: geogate mounts read-only and writes nothing persistent (56 KB container log); CrowdSec's own alert store is self-limiting at ~6 hours. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

paulhitt commented

2026-09-04 22:06:35 +00:00

(Migrated from gitlab.com)

assigned to @paulhitt

paulhitt commented

2026-09-04 22:06:56 +00:00

(Migrated from gitlab.com)

mentioned in commit 0b4782897f

mentioned in commit 0b4782897f8c53f01a2040595d91ea6ead5db17f

paulhitt (Migrated from gitlab.com) merged commit 0b4782897f into main

2026-09-04 22:06:56 +00:00

paulhitt commented

2026-09-04 22:06:57 +00:00

(Migrated from gitlab.com)

mentioned in issue #19

paulhitt commented

2026-09-09 18:59:01 +00:00

(Migrated from gitlab.com)

changed the description

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: extant2000/flamenet-relay#111