checks, swift and image all green on dca5333e. Two defects only the Linux build could show: TodayFile.Key stored a Curve25519.Signing.PublicKey in a Sendable struct, which swift-crypto does not allow and CryptoKit does, and BadgeCountTests still called Routes.placeDeposits, which wave 2 split into resolveDeposits plus Delivery.write.
14 conflict hunks across 8 files. Wave 2 landed on main while this branch sat,
and the two sides rewrote the same send path in different directions.
Routes.swift and RoutesFederation.swift: wave 2 extracted the write, the wake
and the push into `Delivery.write`, and this branch still had the inline
version plus one thing wave 2 had no idea about, the `announced` column. The
extraction is kept; `Delivery.Item` gains `announced`, defaulting false as
`Envelope.init` does, and `Delivery.insert` writes it. Without that the model,
the migration and `badgeCount` would all have merged cleanly and every badge
would have read zero, because nothing on the surviving path ever set the
column. The submit route passes `!silent`; a federated deposit passes false,
since nothing announced it.
docs/CONFIGURATION.md: the union. Main lists six outbound connections with the
setting that enables each; this branch added the Today fetch and dropped the
other five. Seven are listed now.
PostgresMigrationTests: both sides' tests, with the `LockedRows` actor left
last in the file.
.e2e-ref moves to d0274ca0, the current flamenet-e2e main. The old pin was
310c315, which was a branch head and not on main at all.
The three generated doc pages were not resolved by hand. `check-doc-pages.py`
reported spec, audit, engine and self-hosting all stale against their sources,
so `Scripts/build-doc-pages.sh` ran against the new pin and all nine pages now
match. That also fixes the spec page's section numbers: flamenet-e2e moved
anonymous reads and contact links to 8.13 and 8.14 to make room for the push
token and the Today file.
Eight more references renumbered by hand, in Wire, RoutesAnonymous,
RoutesDiscovery, ContactLinkPageTests and AnonymousReadsTests. The remaining
8.11 and 8.12 references are about push and the Today file and are correct.
`swift build` is clean on Swift 6.4.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The image job is red on main and the deploy has not run. The gate's readiness
check was wrong in both directions, and the two failures looked nothing alike.
On mac1 `docker exec <db> pg_isready` never succeeded inside 90 s, and the
gate said only "never became ready". On mac2, with the diagnostics from #253
in place, it succeeded 2.4 seconds after `docker run`, and the candidate's own
log says what happened next:
Opening new connection for pool failed ... Connection refused (errno: 111)
Fatal error: Error raised at top level: PSQLError(code: connectionError ...)
The postgres image runs initdb against a bootstrap server on the unix socket
and then restarts, so a local-socket `pg_isready` can answer "accepting
connections" while nothing is listening on TCP. The relay only speaks TCP, and
it exits on a refused first connection rather than retrying, so that one early
answer ended the run: the 90 seconds that followed were spent waiting for
/health from a container that had already died.
Readiness is now checked the way the relay will use it, over TCP from a
separate container on the gate's network, and twice in a row so the restart in
the middle of the init cycle is not mistaken for the end of it.
Worth a separate decision, not changed here: the relay exits on one refused
connection to Postgres. In production that is a crashloop until the database
answers, which is survivable and noisy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The image job failed on PR #252 and passed on main with no code difference
between them that touches CI. The runner is the difference: run 434 built on
mac2 and passed, run 441 built on mac1 and failed at "staging postgres never
became ready", on the same two commits. mac1 has the smaller OrbStack
allocation, and the gate starts three sibling containers on the host daemon.
The job now names mac2 rather than taking whichever runner is free.
The gate also prints the container's state and log when Postgres does not come
up. "Never became ready" covered both a Postgres still starting and a
container that never ran, which need different fixes, and mac1 said nothing
further, so 90 seconds of waiting read as a relay bug.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ci-staging-gate.sh spliced $ALL_PAGES into the sh -c script text. The
English page list from Entrypoint.swift is newline-separated, so the
newline ended the `for p in` list early and sh failed with "unexpected
word (expecting do)" before any page was requested (run 384, job 948).
Pass the pages as positional arguments and the relay host as an env var.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
.gitlab-ci.yml becomes .gitea/workflows/ci.yml on mac1/mac2 (amd64), with
the same checks: secret sweep, swift build and test on swift:6.1-noble,
section 8 conformance at the pinned e2e commit, doc-page hashes, the
configuration doc and package completeness (HEAD variants advisory).
The image is built with --build-arg FN_RELEASE=<sha> and the staging gate
(Scripts/ci-staging-gate.sh, same checks as staging-gate.sh) runs against
that image before it is pushed. On main, deploy/gitea-deploy.sh pulls the
gated image on the box, dumps the database, tags it flamenet-relay:prod,
checks the edge and key_log row count, rolls back to :previous on failure,
and recreates the test fleet. The self-host compose pulls from
git.extant2000.com.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ErrorReportingTests carries a JWT-shaped string whose signature decodes
to "signature-bytes-here". The secret sweep failed on it, so the fixture
count goes from 1 to 2 with the marker on that one line.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
503 is what a draining relay answers during a planned restart, so it no
longer becomes an event. Every other 5xx still does, and an error logged
while answering a 503 is still reported.
FN_RELEASE, baked into the image by `--build-arg FN_RELEASE=<sha>`
(Dockerfile, and passed through by Scripts/staging-gate.sh), is sent as
the event's release. Unset, empty or not a plain version string, and
events carry none.
The configuration, hosting and self-hosting pages, and the Spanish
self-hosting page, said SMTP was the relay's only outbound code. The
relay also dials Apple's push service, peer relays, Stripe, an S3 store
and now the error collector. Each is listed with the setting that turns
it on, ending on what a relay with all of them unset opens: nothing.
self-hosting.leaf is generated from flamenet-e2e SELF-HOSTING.md, so
.e2e-ref moves to 310c315, that change made on the pinned commit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Until now a bridge line existed in one place, the bridge container's
stdout. Doc 27 Milestone 1 needs a list: what is handed out, since when,
and what is retired.
- obfs4_bridges: host, port, fingerprint (40-hex obfs4 node ID, unique),
line, added_at, retired_at. Additive migration, appended last.
- The line is derived from the bridge's own state file, as the bridge
prints it. The private key is read from stdin, used for the X25519
public key, and dropped; no row holds it. Checked against RFC 7748 here,
and against the live bridge on the box (derived cert == printed cert).
- A retired identity is never listed again.
- No route reads the table. flamenet-relay bridges list|add|retire is
dispatched in main BEFORE configure, so it opens the database and does
not migrate, sweep, drain or boot lifecycle handlers beside the live
relay. Scripts/bridges.sh wraps it on the box, like suspend.sh.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Additive and reversible. Three new migrations appended to the boot list
(CreateIntroTokens, CreateContactLinks, AddUserDiscoverable); nothing
existing is altered or dropped, and key_log is not touched.
C1, introductions. Anonymous POST /e2e/intro {handle} or {link} returns
three single-use tokens good for ten minutes, the target's bundles (one-time
prekeys popped, as GET /e2e/keys does) and a handle certificate.
intro_tokens holds (token_hash, user_id, expires_at) and nothing about who
asked; the sweep deletes expired rows. Metered per address, per network
(sign-up's /24 and /48), per account and per (network, account), and capped
at 90 unspent tokens per account. spendDeliveryToken tries delivery_tokens
then intro_tokens, each a single DELETE ... RETURNING. Handle certificates
sign "fnhandle-v1|audience|user_id|handle|exp" with the issuer signing key
whose public half both apps pin (RelayIdentityPin); GET /e2e/handle-cert
returns the caller's own.
The sealed send path: the recipient-wide bucket now counts only refused
attempts (so a working delivery key cannot be used to lock everyone else
out), plus a per-(recipient, address) bucket (FN_SEND_RATE_RECIPIENT_IP,
default a tenth of FN_SEND_RATE). Meters about an account are keyed by a
keyed hash (LimiterKeys), never the account id.
B1/B2, Unlisted. users.discoverable, default true, set by PUT /messenger/me
{"discoverable": false}. Read and written with raw SQL, not through the User
model, so migrations that load User early in the list still build a fresh
database. lookup ?slug=, POST /messenger/buddies, GET
/e2e/account/identity/:handle, GET /messenger/users/:id and POST /e2e/intro
{handle} answer an unlisted name as a missing one: same status and body,
same meter charge, and every "not there" answer held to a 5 ms floor.
Contact links: POST /e2e/contact-links {add, revoke} by hash, at most five,
refused (409) past that, table holds only (user_id, link_hash).
D1, anonymous reads. POST /e2e/anon/presence, /e2e/anon/devices and
/e2e/anon/bundles, targets in the body, no credential read. Directory and
bundles are metered by the subject's presence token when the caller holds
one, otherwise per address. The public directory (other accounts, federated
peers, the anonymous route) no longer carries created_at or last_seen; GET
/e2e/devices/{own id} keeps them because both apps' Devices screens read
created_at there, and GET /e2e/devices/me is the new home for them.
presenceByToken is split from caller().
Also: account deletion now removes the account's delivery_tokens (left
behind before), intro_tokens and contact_links.
Tests: 845 swift-testing tests in 93 suites and 61 XCTest pass, Postgres
18 suite included (fresh and populated upgrade for wave 0b, and a
concurrent intro-spend race). 23 falsifications, each red on its own
sabotage.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each device mints its own delivery key and its own presence tokens, and
the relay kept one digest and one token set per account, so the second
device to publish replaced the first. Its contacts' sealed sends were
refused (the apps then send unsealed) and its contacts saw the account as
offline. PerDeviceDigestTests reproduces both and failed on the old code.
- New tables device_delivery_keys (unique user_id, device_id) and
device_roster_tags (unique user_id, device_id, digest). Additive:
delivery_keys and roster_tags are untouched, their rows keep working as
account-wide rows because the publishing device was never recorded.
Revert drops only the new tables.
- POST /e2e/delivery-key writes the token's device row when the device is
registered, else the account-wide row as before. The account-wide row is
dropped once every device on the account has its own: both apps publish
once and never again unless they rotate, so dropping it sooner would
refuse sealed sends to devices that will not publish again.
- The sealed check compares the presented key against every device digest
and the account-wide one, padded to FN_MAX_DEVICES + 1 comparisons, with
no early exit.
- POST /messenger/roster/tags takes an optional device_id (one of this
account's devices). With it, it replaces that device's set and removes
the account-wide set, which may be the same device's older one. Without
it, an older app replaces the account-wide set exactly as before.
- Presence looks in the device sets, then the account-wide set.
- DeviceRetirement (revoke, expiry, make room) deletes the device's rows
from both new tables. key_log only gains its usual leaf. Account
deletion now also removes roster tags, which it missed before. The
orphan sweep covers the new tables.
- The descriptor carries features.perDeviceDigests: true, so an app can
publish its digest once more and send device_id.
/metadata (en, es), API.md and its page, METADATA-PASSES.md and the vacuum
script's table list updated (roster_tags was missing there too). Upgrade
with populated legacy tables tested on SQLite and Postgres.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Groundwork for length hiding; nothing is padded or refused by default.
- Ladder {16384, 65536} as PadLadder.small/large.
- Every send is counted, without ids, as on or off the ladder by app
product and version (at most 32 builds a window, then "other"), plus a
ten-bucket length histogram. The hourly sweep line carries pad_in,
pad_out and pad_lengths. Scripts/payload-lengths.sh sums them over N days
and buckets the envelopes stored now; both read-only.
- FN_PAD_ENFORCE off|log|enforce, default off, unknown values off. enforce
refuses the whole request with code "unpadded" before any capability is
checked or spent.
- FN_PAD_ATTACHMENTS (default false) is published as
features.padAttachments in /.well-known/flamenet/relay, camelCase like the
rest of that document.
- A poll stops near 2 MiB of payload, never below one envelope, and adds
"more": true when it stopped early. The key is absent otherwise.
CONFIGURATION.md, API.md (also the conversation refusal from the previous
commit), their generated pages, /metadata (en, es) and METADATA-PASSES.md
updated. Production baseline: 415 stored envelopes, none padded, 24 in
16-24K.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FN_DELETE_ON_ACK was built, tested and off: production has it set but
empty (it was 1 from 2026-08-29 to 2026-09-01, turned off with a backup
named .env.bak-deleteonack-20260901 and no reason written down anywhere
I could find). Every delivered envelope then sat for a day, with its
recipient, sender (unsealed sends), size and times.
It is now on unless the flag says 0, false, no or off. Postgres only,
as before: on SQLite it stays off whatever the flag says, because SQLite
reuses row ids. A SQLite relay on the default is quietly off; only an
explicit request on SQLite gets the critical "REFUSED" line, so the
SQLite test fleet does not log a critical line on every boot.
Why it is safe for the two apps as they are:
- The cursor is the receipt. iOS advances lastInboxID only after an
envelope is ingested or given up on after 12 stalled polls
(E2EInbox.pollOnce); Android advances its watermark after consume,
which holds refused-identity envelopes locally (Conversations.readInbox).
Neither lowers its cursor to collect again; a reset cursor of 0
deletes nothing.
- Per device: envelopes are addressed to_device and the delete is
scoped to the polling device, so another device's inbox is untouched.
- The iOS badge counts undelivered rows; only rows already returned and
marked delivered are deleted.
- Sealed-sender rows, read receipts and call signals are ordinary
envelopes. Attachments are a separate table with their own 30 days.
- The one-day sweep stays as the backstop for a device that collects
and never polls again.
What it costs: Scripts/call-health.sh measured ringing speed from
collected envelopes, which are now gone within seconds, so that line
mostly prints 0 calls; the script says so. And a client that ever
advanced its cursor wrongly would lose the message for good instead of
having a day.
Docs: CONFIGURATION.md, README, COLDSTORAGE.md (which said it should
stay off), the prod compose comment, and the metadata and privacy pages
in both languages now say the day is the backstop. configuration.leaf
and relay.leaf regenerated against the pinned e2e ref.
Deploying this turns it on in production, because the box's .env has
FN_DELETE_ON_ACK= (empty). Set FN_DELETE_ON_ACK=0 there first to keep
it off.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A hello link is https://<relay>/hi#<address> (§6.34). The address is in
the fragment, which no browser sends, so /hi and /es/hi are one static
page for every link, and Public/hi.js reads the address on the device:
fnid: and 52 characters of the apps' base32 alphabet, then @ and a host
of letters, digits, hyphens and dots, the rule both apps use. A good
address is shown as text with "Open in Flamenet Messenger"
(flamenetmessenger://hi#<address>) and a copy button; anything else gets
a plain explanation. Both carry the install links and the iPhone
App Store or TestFlight switch from /get.
The script makes no request and builds no URL but the app link. A test
reads it for every way a page can send something, and on macOS runs it
in JavaScriptCore against stand-ins for the page's elements. Both pages
ask not to be indexed (header and meta tag), stay out of the sitemap,
load nothing from another host, and are in the staging gate's page list.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The messenger repositories move from gitlab.com/paulhitt to
gitlab.com/extant2000. Old web and git URLs redirect; the relay's container
image path does not, so docker-compose.yml names the new registry path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A deploy today passed the whole gate and then rolled itself back. The
build was fine. `sleep 8` was not.
Measured on the box while working out why: the relay container started
at 18:37:00.9 and its first healthy probe was 18:37:16.7, so a recreated
relay needs about sixteen seconds before it answers /health. The check
waited eight, read the edge's 502 as a broken deploy, and reverted a
build that had just passed signup, signin, 47 rendered pages and a 25
second stability window.
This was not a flake. As written the check could not succeed, and the
same run passed on the second attempt with nothing changed but the wait.
The check itself is sound: run by hand against the running relay it
answered ok three times out of three.
A deadline loop instead, leaving as soon as the edge says ok. A healthy
deploy pays nothing, usually under twenty seconds. Only a deploy that is
actually broken spends the budget, which is how the rollback stays
correct rather than becoming something people learn to ignore.
Same bug as 45194792 in the iOS suite, where a fire-and-forget revoke
was waited on for a fixed two seconds instead of until it landed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five refusals in the auth path were disabled one at a time to see which
the suite noticed. Four died. This one survived all 736 tests: the check
in `authenticate` that refuses a revoked device still holding a live
token, the one whose comment cites audit M2.
Two tests do cover revocation and one says it is exercising the
retirement path. Both call `poll`, and `poll` carries its own
`device_revoked` guard on the device row, so both pass whether or not
`authenticate` refuses anything. The `authenticate` check is what every
other authenticated route relies on, and nothing touched it.
With it gone a revoked phone keeps fetching bundles, sending, replenishing
prekeys and mutating groups until its token expires. Only its inbox
closes.
The test goes through `bundles`, which has no revocation check of its
own, and asserts the target's one-time prekey is still in the pool
afterwards, so the refusal is shown to happen before the route does any
work.
`Scripts/mutate.py` ships the harness. The four that died are real
coverage: proof signature, clock skew, replay admission and device key
mismatch.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The file opens by saying it lists every variable the relay reads, and
that a variable absent from it is not read. An operator sets what it
names on a machine holding other people's messages. Twenty-one were
read and undocumented: the six object-storage settings, the Redis URL
and replica count, the per-account device cap and expiry, the monthly
attachment allowance, the capacity ceiling, the SMTP password, the
Android release fields, the two connection-pool sizes, the working
directory, and FN_ROSTER_ON_SERVER, which decides whether the relay
stores the contact graph at all.
Five table rows were also stranded above the page title, with no header
row above them, so flamenetmessenger.com/configuration opened with a
paragraph of raw pipe characters before its own heading. Four of them
were the only mention of FN_REPORTS_TO, FN_CONTACT_EMAIL, the six
per-role contact addresses and FN_REPORT_RETENTION_DAYS; the fifth
duplicated a row further down. They are in the report and site-copy
tables now. A blank line inside the limits table was splitting it in two
on the rendered page.
Scripts/check-configuration-doc.py keeps the claim true, wired into CI
next to the doc-pages gate. It matches whole names, since the first
version compared substrings and a renamed variable still satisfied it.
Falsified three ways: renaming a variable, deleting its row, and
dropping the interpolated FN_CONTACT_<role> family are each caught.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`Scripts/selfhost-smoke.sh` runs the published quick start and is the
only thing that proves a self-hoster can follow the documentation. Its
own header calls it a release-time check somebody has to run, because
this project's runner cannot reach a docker daemon.
It set FN_AT_REST_SECRET, which went with the plaintext transport in
0f78fc0 on 2026-09-04, and it asserts that everything it sets exists in
.env.example. So for sixteen days it exited at its third line --
"FN_AT_REST_SECRET is not in .env.example, the documented steps have
drifted" -- before pulling a single image. Nobody knew, because the way
you find out is by running it, and the reason it exists is that nobody
does.
Dropped from there and from staging-gate.sh, which passed the same dead
variable into every candidate container.
It runs now. On current main, built from source: the documented quick
start boots and serves its landing page.
And three tests so the next one is caught in CI in milliseconds rather
than by somebody remembering: everything the smoke sets exists in
.env.example, neither script sets a variable the relay has stopped
reading, and the generated Postgres password is still hex -- base64
emits '/' about three times in four and it goes unencoded into
postgres://user:PASSWORD@db/..., where a '/' ends the authority.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The iPhone sign-in screen was drawing BrandMark, the square emblem the
icon is cut from, whose bottom edge cuts across the globe and both
bubbles. It needs the whole logo, so this generates it rather than
leaving somebody to export one by hand.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The plan for 8.9c says to wait until the old clients are gone before dropping
the contact graph, and to read that from what is still polling. The relay could
not read it. iOS sent a hardcoded 1.0 that never changed and Android sent no
User-Agent at all, so the step everything else depends on rested on a
capability that did not exist.
One nullable column on devices, overwritten, no history. The marketing version
and nothing else: no build number, no device model, no OS. Parsed narrowly,
dotted digits up to sixteen characters, because a User-Agent is
attacker-controlled and this is read on the busiest authenticated path here.
Written on the hourly liveness update, plus once out of turn when the version
changes, since waiting an hour per device to notice an upgrade makes the answer
stale exactly when somebody is watching it.
Scripts/client-versions.sh is the operator's side of it, and says plainly that
a device reporting nothing counts as old, because that is the case that cannot
be ruled out.
Disclosed on /metadata rather than added quietly. The row says what it holds,
what it deliberately does not, and why the trade is worth taking: keeping one
version string is what allows the contact graph to go.
The migration goes BEFORE BackfillKeyLog, which loads Device through the model.
Fluent names every column in its SELECT, so a column added after a
model-reading migration fails with "no such column" on a fresh database. The
tests caught it. AddUserSuspendedAt was caught by the same trap.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every icon derived from one crop of the source artwork, and that crop was
too tight on three sides: it started at x=280 when the flame arc reaches
x=38, ended at x=1040 when the arc runs to the edge of the source, and cut
the bottom at 548. So the app icon, the favicons, the social card and the
Android launcher icon all showed a flame with both ends sheared off and a
globe clipped on the left.
The crop is the full width now and stops at y=528, which is the last row
before the wordmark's F. There is no empty row between the emblem and the
word, because the flame passes behind the letters, so a rectangle cannot
take all of one without a piece of the other; 528 leaves a flat edge that
sits under the flame where nothing reads as cut.
Two other things were wrong and are fixed here. `content_fraction` shrank
the finished square rather than the artwork inside it, so the iOS icon,
asked for any margin at all, came out as a navy square inside a black
frame. And the Android foreground was sized by width against a safe zone
that is a CIRCLE, where what has to fit is the diagonal: at 0.72 the
diagonal came to 86 of 108 and a round launcher cut the flame off at both
ends. Measured at every density now, 65 to 66 of 108.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fluent's .datetime is timestamptz on Postgres, so to_timestamp() on it and
assigning extract(epoch from now()) to it both fail. now() in, the column
itself out.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The safety page said we could not suspend an account and that saying
otherwise would be untrue. It is one column and three doors now.
Scripts/suspend.sh sets or lifts it. A suspended account gets no new
credential from sign-in, a shared session or a reset; a credential it
already holds cannot become a token; and every request it makes with a
token that still verifies is refused on the heartbeat, so it takes
effect on the next request rather than when a token expires. Messages,
keys and devices are untouched: a door, not a deletion. One sentence at
every door, which both apps show verbatim.
The migration sits before AddUsernameSkeleton on purpose. That one loads
User through the model, and on a fresh database every column the model
declares has to exist by then or it fails with "no such column", which
is exactly how the first test run died. A deployed database applies the
new one on its own in that position.
The multi-homing harness nothing ran now runs nightly against
relay1/2/3 from a throwaway node container, and mails once on failure
and once on recovery, the same rule as the health check. First run:
twelve of twelve.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Templating the contact addresses put `#endif` in front of the colon the
Spanish list items already had, and Leaf reads `#endif:` as a tag
declaration with a body. /es/safety answered 500 in production through a
gate that rendered only the English page table. The colon is emitted as
a string now, the gate renders every Spanish template as well, and the
balanced-tags test covers both languages, counts `#if` against `#endif`,
and refuses a closing tag followed by a colon, which every count in the
file would otherwise still call balanced.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every check here answered "is it up". None answered "is it about to stop
being able to take another user", and the limits that decide that are
invisible until they are reached, when they arrive as connection errors
rather than as anything that says full.
Two were found at or near their ceiling while every existing check passed.
The three fleet relays were on the stock 1024 file descriptors, which the
relay itself translates to "roughly 512 held long-polls before connections
start failing" -- and it printed that warning on every boot, for months,
into a log nobody was reading. One poll is one socket, so that WAS the
concurrent-user limit: about 512 per node, on a service being prepared for a
thousand. coturn had forty relay ports for the whole service; each call that
cannot go peer-to-peer takes one, so that was forty simultaneous relayed
calls everywhere, and it would have read to users as calls that do not
connect, for some people and not others.
Both were one line of configuration. Neither was discoverable from anything
that was being watched.
So this now checks four things against their actual limits and warns at 80%,
while there is still time to act:
- held file descriptors per relay node, which is the user ceiling
- Postgres connections, the next one in line behind it
- TURN relay ports in use
- root disk, which filled once before and took Postgres into a recovery
crash-loop that every liveness check in this file passed happily
Each reads a real number from the running system rather than a constant. A
capacity check that passes because its variable came back empty is worse than
no check, so `pct_under` refuses to judge rather than silently agreeing, and
the values were confirmed against the live box before this shipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`submitReport` wrote one row to `content_reports` and stopped. No mail, no
alert, and no route or script to read the table back: a report of abuse sat on
disk until somebody thought to look, which for a report of abuse is the same
as nowhere. Meanwhile the safety page has been telling people for months that
a report "goes to the people who run this server". It went to their disk.
So `FN_REPORTS_TO` gets a mail per report -- kind, reason, who it is about,
who filed it, what they wrote and the copy they attached -- and
`Scripts/reports.sh` is the record, because mail is not one.
The mail is detached and cannot fail the report. If SMTP is down the row is
still filed and the script still finds it: refusing a report because a mail
server is unwell would lose the one copy of something somebody went out of
their way to send. With no address configured it says so in the log rather
than filing in silence, which is the state it has been in all along.
The script leads with accounts reported by more than one person. One person
reporting one thing may be a disagreement; several people reporting the same
account is the thing to look at first.
698 tests in 63 suites.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every bug that cost this week was invisible to a crash reporter, because none
of them were crashes: an answer discarded by a handler with no branch for it,
a poll holding its request open for twenty seconds, a hang-up treated as
transient until WebRTC gave up half a minute later. Things working exactly as
written and wrong. What was missing was never exceptions. It was outcomes.
Two numbers, from data this relay already receives.
WAS THE PHONE EVEN RUNG
`ring_events` records what became of each attempt: Apple took it, Apple
refused it and the token was forgotten, or there was no VoIP token to try.
Nothing new is learned about anybody -- the recipient and the time are already
columns on the envelope this belongs to -- and it is swept after seven days,
long enough to ask whether yesterday was worse and short enough not to be a
record of who was rung and when.
The third outcome is the one worth having. An entire platform has been sitting
with no push token, reachable only by its own next poll, and the only way to
find that out was for somebody to think to look.
HOW LONG THE OTHER PHONE TOOK TO FIND OUT
`delivered` said whether an envelope was collected and never when. The gap
between written and collected is the most useful thing this relay knows about
calls: a second is a phone that rings, twenty is a phone whose owner has
already put it down. Twenty is also exactly what a client holding its poll
open for twenty seconds looks like from here, which nobody could see, which is
why it lasted as long as it did.
Nullable and unbackfilled: rows written before this cannot have a truthful
value and inventing one would put a made-up number into the only report that
measures this. Written once, so a client resetting its cursor cannot rewrite
when the phone actually found out.
WHAT IT DELIBERATELY CANNOT SAY
Whether a call CONNECTED. Sealed sender means the relay does not record who
sent an envelope and cannot read what is in it, so it cannot pair an offer
with its answer. That is the design working rather than a gap to close here:
knowing would mean this relay knowing more about calls than it should.
Connection outcomes have to come from the phones, with consent, and that is a
separate decision.
`Scripts/call-health.sh` is the report. Recording can never fail a delivery:
the whole point is to watch the thing, not to become one of its failure modes.
698 tests in 63 suites.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This housekeeps the GitLab runner's build cache. It is not part of the relay
and this repository is public and about a messenger server, so it now lives
in hitthosting-infra beside the restic and nzbdav timers.
What is left under Scripts/monitoring is the messenger's own: health, the two
privacy checks, and the database vacuum.
messenger-health.sh still calls /srv/prod/monitoring/prune-runner-cache.sh
when /data crosses 93 percent. That is a path on the box, not in this tree,
so the move does not reach it, and the call is already guarded by a test that
the script is present and executable. It degrades to not sweeping if it ever
is not.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two pages went live answering 500 today and the gate reported PASSED.
It was not a gap in what the gate checks. It was that the list it
checks had quietly collapsed.
The page names are derived from the table in `Entrypoint.swift`, where
an entry is written across two lines:
(
"about", "About. Flamenet Messenger",
The pattern matched EIGHT spaces followed by an opening paren, which is
the line with no name on it. So it extracted nothing, every time, for as
long as the table has been formatted that way. The fallback hand-list
then became the entire gate: five pages, plus donate and releases named
separately, which is the seven it has been reporting.
Twenty-five pages exist. Two of them were new today, neither was in the
list of five, and both answered 500 in production while the gate said
the deploy was good.
The pattern now matches the line the name is actually on, and a floor
goes with it. Falling back is fail-safe only if somebody finds out;
degrading from twenty-five pages to five and printing PASSED is
fail-quiet, which is worse than having no gate, because it reads like
coverage. If the count collapses the derivation has broken, so the gate
now says so and stops instead of deploying on a fifth of the checks and
announcing success.
Verified against production: the fixed derivation picks 25, renders 27
with donate and releases, and every one answers 200.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The DropMessages migration removed the legacy plaintext `messages` table
outright on 2026-09-05. Section 3 counted its rows and treated an unreadable
result as drift, so the weekly check started reporting "could not read
messages" on what is the strongest possible outcome: no plaintext path left at
all, not even an empty table. It has reported that every run since, and a
check that always fails stops being read.
Absence is now measured with to_regclass instead of inferred from a count that
errored. The header's rule still holds: a probe that cannot run is a failure,
so an existence check that returns nothing still drifts, and a table that
exists with rows still drifts.
Verified against production: reports "messages is gone, not merely empty" and
all properties hold. Mutation tested both failure branches: a table with live
rows drifts, and an unreachable database drifts rather than reporting a clean
sweep over nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comments, docs, scripts and pages kept narrating their own history: what
a limit was before a date, when a repository was split, which outage a
guard was born from, what a section used to say. None of it is what a
reader came for, and a date goes stale faster than the sentence holding
it.
Dates are gone from prose across Sources, Scripts, docs and the
hand-written pages, including the release-date parentheticals in
CONFIGURATION.md, where the version already identifies the build and an
operator acts on the version.
Kept, because they are data rather than narration: the `created_at`
example in the spec, the ISO-date field comment and fixtures in
ReleaseNotes, and the observed-log tables in IDENTIFYING-DATA.md and
METADATA-PASSES.md, where the dates are the measurements.
⚠️ /privacy is the one judgement call. It carried a dated notice that
the plaintext path was removed, and a paragraph explaining that it was
being said rather than quietly deleted, because anybody who read the old
version decided something on it. The date and the "this section used to
describe" framing are gone; the disclosure stays. Removing a privacy
change notice outright is the thing that paragraph exists to prevent.
697 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The skill's own detector (conorbronsdon/avoid-ai-writing) scored every
README, doc, page and the extracted comment prose. Scores were already
0 to 3 out of 100; what it still flagged is fixed here: hollow
intensifiers (genuinely, genuine), "in order to", "additionally",
"the whole point" and "the entire point", bold sentences of seven words
or more in the docs, and bold that had wrapped across comment lines.
Comment prose now scores a maximum of 1 out of 100 across the repo.
One rule in the sweep went wrong and was caught by reading the diff:
"which is the entire point of X" lost its clause and left "of X"
dangling in four comments. Repaired by hand; the rule now keeps the
clause when it is followed by "of".
Comments and strings only. 694 tests pass.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Emoji markers, markdown bold inside code comments, and em-dashes are
gone from source comments, scripts, CI config, the markdown docs and
every served page. Counted before and after:
emoji markers 1432 -> 0
em-dashes 1938 -> 4 (table placeholders and one fenced comment)
Paragraphs that narrated their own revision history into the public
pages ("this said X until <date>", "this row used to say the opposite")
are cut back to the current fact. That history is in git.
The eight generated pages are regenerated from their sources: the spec,
audit, security, self-hosting and engine pages from flamenet-e2e, and
the relay, API, configuration and hosting pages from this repo's docs.
Two of them had been hand-edited after generation and would have
reverted on the next run; those edits now live in the sources.
The diff over Sources/ is comments and string literals only, checked by
filtering the diff for anything else. 694 tests pass before and after.
CONTRIBUTING gets a four-line rule so it does not grow back.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
When CrowdSec crashed today and Traefik failed closed, this script caught it
within five minutes -- as `FAIL: relay-health landing apk`. Three symptoms,
no cause, and the diagnosis started from nothing while every site was down.
A crashed CrowdSec is a specific thing with a specific remedy, so the alert
now says so. Checked by pointing the check at a container that does not
exist: `FAIL :: crowdsec-up`.
⚠️ `edge-traefik-1`, not `traefik`. The container has been called that since
the edge stack was split out, and `docker restart traefik` fails with "No
such container" at exactly the moment somebody is in a hurry -- which is
what happened today.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The outage began at no particular hour and nobody was awake for it. A daily
sweep prevents the slow build-up, but a burst of CI can cross the line
between runs -- so the five-minute health check now acts when it sees the
edge coming, rather than only writing it down.
Warn at 90%, act at 93%, wall at 100%: a person hears about it before the
machine intervenes, and the alert mail says what was swept rather than
quietly hiding that anything happened.
⚠️ This is the ONLY thing here that self-heals, and only because it cannot
lose anything -- runner cache is regenerable by definition, so the cost of
acting wrongly is one slower pipeline. Nothing else in this script is safe
to automate. A monitor that starts deleting things it has not reasoned
about is worse than the outage it is trying to prevent.
Drilled on the box with the threshold lowered to 60% and the mail path
stubbed: it detected /data at 67%, ran the sweep, removed 2 volumes and
logged both the trigger and the result, with no human involved.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The alert added alongside this tells you the disk is filling. This is what
stops it. Runner cache volumes grow without limit and share /data with the
production database; two had reached 41.7G and 8.6G when the disk hit 100%
and took Postgres down.
🔴 It is NOT a volume prune. 339 of 386 volumes were unattached that morning
and most hold real data -- a stopped service's database is unattached too.
This matches the runner cache pattern by NAME and requires the volume to be
unattached as well. Both conditions, every time.
Daily rather than weekly, measured rather than guessed: 3.7G of runner cache
accumulated in the few hours of CI after the disk was cleared this morning.
Weekly would let ~50G build up between runs, which is the size that caused
the outage.
Verified on the box: dry run listed only `runner-*-cache-*`, the real run
removed 8 volumes and ~1.9G, and all 350 other volumes were left alone.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
messenger-health.sh has run every five minutes for months and passed every
check straight through the 2026-09-08 outage, while Postgres crash-looped in
recovery and the disk sat at 100%.
Two blind spots, both pointing the same way:
* `docker inspect -f '{{.State.Running}}'` is TRUE for a container that is
restarting in a loop. Docker's own healthcheck had already marked the
database `(unhealthy)` -- the script never asked. It now prefers
`.State.Health.Status` where a container defines one and falls back to
Running where it does not.
* Nothing watched free space at all, and that is what caused the outage.
Now /data and / are checked at 90%, well short of full because the
failure is not gradual.
⚠️ `df` measures percent-used against the non-reserved space, so it reaches
100% while ext4's root reserve still holds gigabytes -- which is precisely
what happened, and why `df` read `0 avail` on a volume with 10.7G left. That
makes percent-used the right number to watch here, not a misleading one.
The script also moves into this repository. It lived only on the box, which
meant the one script deciding whether anybody hears about an outage was the
single thing under no review.
Checked against the running box: all three containers report `healthy`
rather than merely `true`, both disks pass at 66% and 38%, and the disk
check fails as intended when the threshold is crossed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`selfHosting` is a list of paragraphs, not a paragraph. Written as a string
it failed to decode -- and `ReleaseNotes.all` fails open, so that one field
emptied EVERY release on /releases, not just the new one. CI caught it.
The gate did not. `covers-android.py` says of itself that it is "the only
place that knows the schema", and it only ever asked whether some release
named the version. So it answered "Android 0.12 is covered by the release
dated 2026-09-07 (11 notes)" about a file that renders as a blank page --
which is precisely the outcome the gate exists to prevent.
It now validates what the renderer requires, across every release rather
than only the one being published: the failure is not local, an unrelated
older entry takes the new notes down with it. A shape the decoder would
reject exits 2 ("cannot tell"), never 0.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The nav and footer carried a purple gradient square — a placeholder that
outlived its excuse — and the site had no favicon and no og:image at all, so a
shared link rendered as a bare URL with a blank tab icon.
All of it now comes from one artwork, through `Scripts/make-brand-assets.py`.
🔴 The outputs are generated: an icon edited in place is an icon that silently
stops matching the next time the script runs, which is the same shape as the
generated doc-page mistake this repo already made once.
Two things about the source worth recording. The export carried a leftover dark
matte — RGB (33,33,33) at alpha 38 across the background — invisible on a dark
page and a grey haze on a light one; the script strips anything below alpha 48.
And the wordmark is interlocked with the emblem: "FlameNet" is drawn ON the
globe, and the speech bubbles hang down into the letter tops, so no rectangular
crop gets the whole mark without text. The crop used, (280,0,1040,548), is the
one that takes both bubbles and the flame arc and leaves the wordmark behind.
⚠️ `assetVersion` now hashes the brand images too. Without that a logo change
alone leaves the hash where it was, so every browser that has ever loaded the
site keeps serving the old mark out of cache — which looks exactly like the
deploy not having happened.
⚠️ Not changed, and worth a decision separately: the site's accent is purple
and the logo is blue with an orange flame. They do not clash badly at 30px, but
they are not one palette either.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
relay1/2/3 pin `flamenet-relay:prod`, but a container keeps running the
image it STARTED with — so every deploy silently left the fleet a build
behind. It was eleven days stale when that was first noticed, answering
every health check the whole time, while the Android live suite ran
against it and reported green about a build production had already moved
off.
That is the failure worth naming: not an outage, a suite that proves the
wrong thing and says it proved the right one.
It drifted twice more today, both times within an hour of my own deploys,
which is how I know a note in a runbook would not have held.
⚠️ Best-effort on purpose. The fleet is disposable and a production deploy
must not fail because a test relay would not come up — but it must not
quietly stop matching either, so the outcome is printed either way and
`messenger-privacy-check` asserts it independently rather than trusting
this step.
659 tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`token: "deadbeef"` — an attachment id in a truncation test — trips the
"assigned password" pattern, and has on `main` for a while. I hit it
publishing relay#50 and had to stop and prove the match was pre-existing
before I could trust my own branch.
🔴 **A gate that fails on a clean tree is worse than no gate.** It teaches you
to run it and ignore the answer, and the one time it catches something real
the habit is already to shrug.
⚠️ **The two obvious fixes are both worse.** Narrowing the pattern until
`deadbeef` slips through puts a hole in the SHAPE — the next real secret that
happens to be lowercase goes with it. Allow-listing the value silences
`deadbeef` everywhere, including a file where it is real.
So: a per-line `// secret-sweep: fixture` marker, and nothing else. It must be
typed at the site by somebody who looked at the line, and it is greppable.
🔴 **The count is pinned, because a printed number is not a gate.** Nobody
reads the output of a passing check, so "suppressions are counted" would have
been decoration — somebody could mark a real credential as a fixture and this
would say `clean` in green. `EXPECTED_FIXTURES` makes adding a marker FAIL
until the number is bumped in the same change: a second, deliberate act, in a
diff a human is already reading.
Proved all four ways rather than just the first, which is what the ticket
asked for:
clean tree → clean (190 files, 1 fixture)
planted sk_live_ with no marker → FAILED, Stripe live key
planted sk_live_ WITH a marker → FAILED, 2 markers, expected 1,
and it prints the offending line
plant removed → clean again
659 tests in 60 suites pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The count summed `$PAGES` — the list derived from `Entrypoint.swift` — while
the loop iterated `donate releases $PAGES`. So it has always understated by
one, and the commit that added `releases` made it two.
Nothing was skipped: the loop was right and the message was wrong. But a
gate whose summary disagrees with its own behaviour is one more line that
has to be checked instead of read, which is the opposite of what a gate is
for. The list is built once now, counted, and iterated.
The gate reads its page list out of `Entrypoint.swift`'s `pages` table so
that a page added tomorrow is gated tomorrow. `/releases` is deliberately
NOT in that table: its content is data read from disk per request, so a
release published on a running relay shows up without a restart.
The consequence went unnoticed until the deploy that introduced the page:
its only new page was the only page the gate would not render. `donate`
was already named explicitly for exactly this reason; `releases` joins it.
⚠️ Two pages now sit outside the derived list. A third would be worth
stopping over — the point of deriving the list was to stop maintaining
one by hand, and every explicit name here is a step back toward that.
Three problems in the /releases page, all of which shipped as "working".
🔴 **The change labels rendered as nothing.** `Change.label` was a computed
property, and Swift does not encode computed properties — so Leaf, which
renders the ENCODED release, received no `label` at all and every note lost
its "New" / "Fixed" / "Changed" prefix in public. Every unit test passed,
because each one read `.label` in Swift, where it worked perfectly. It is now
a stored property derived on decode, and the new test asserts on the encoded
JSON, which is the only thing the template can see.
🔴 **A release named a version that belonged to nothing.** One release is
several artifacts with numbers of their own: this repository called a release
"0.9" while the Android build in it was 0.10 and iOS had not shipped at all.
A reader checks the number in their app against this page, so a number that
matches no artifact is worse than no number. `platforms` now carries a version
per platform — "android 0.10", or bare "relay", which has no version a user
ever sees — and the top-level `version` is gone.
🔴 **Omitting `selfHosting` emptied the whole page.** It has a default and is
documented as optional, but Swift's synthesized decoding ignores defaults, so
a release without operator notes threw — and `all(in:)` fails open, which
turns one missing section into a blank page. Most releases have no operator
notes, so the common case was the broken one.
The shipped file is also now truthful. It announced Android and iOS changes
that nobody can install: published Android is 0.8, the repository is at 0.10
and unbuilt, and iOS is still waiting on Apple. It is a relay release, and it
says so.
Scripts/covers-android.py answers one question for the Android publish flow —
does the changelog name this APK's version — and lives here because this is
where the schema is. "Cannot tell" exits 2, never 0: the relay fails open on a
bad changelog because that costs a page, but a publish gate that failed open
would wave through exactly the build whose notes nobody wrote.
653 tests pass. The page was rendered from a running relay to confirm the
labels and tags actually appear, since that is precisely what the unit tests
could not see.
Every property measured in docs/PLAINTEXT.md was true once and could stop
being true quietly. Three already had, on the day it was written, with
nothing reporting it: /metadata promised rosters were in the clear after
the apps had fixed it, the test fleet ran an 11-day-old image while
answering every health check, and buddies held 19 dead tuples against 2
live rows because autovacuum never fires on a small table.
Ten checks, weekly, mailing ONLY on a state change — a report that arrives
every week is a report nobody reads.
🔴 The CrowdSec row is the one that most needs a machine. Nothing else
would notice IP banning going dark, and the ordinary way that happens is
somebody improving privacy by dropping the client IP from the access log,
which is the change docs/PLAINTEXT.md argues against.
⚠️ A check that cannot run reports FAILURE, never a pass. The thing being
guarded against is a script that greps a file which has moved and reports
a clean sweep over nothing.
⚠️ It caught real drift on its first run: the fleet had fallen behind
production again, because deploying prod does not update the fleet — an
hour after I had bumped it by hand.
Two bugs found writing it, both a monitor reporting on itself rather than
the system: cscli metrics -o json is pretty-printed so a naive grep for
"reads" matches nothing, and systemd expands % in ExecStart so an inline
date +%Y is mangled — the first version wrote "OK /bin/bash" into its
status file.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
🔴 Without a by-subject lookup a monitor cannot monitor. Watching an
account means noticing a device published under it that its owner did not
add — and the only lookup that existed was by device id, which requires
already knowing the device the monitor is trying to discover. The
`subject` column was there; nothing exposed it.
⚠️ A query parameter rather than a path segment. `KeyLog.logSubject`
falls back to `local:<audience>/<userID>` for an account with no fnid,
and that contains a slash — a path route would have silently 404'd for
exactly the accounts least able to notice.
⚠️ The Merkle index stays a POSITION in the whole log, not a per-subject
number. A monitor asking for an inclusion proof at the wrong index gets a
leaf that does not verify, which reads as tampering when it is
arithmetic. There is a test with an unrelated entry ahead of the subject's
so a per-subject numbering fails it.
⚠️ Unauthenticated, like the rest of §19, and the cost is written into
the docs rather than left to be discovered: anyone holding somebody's
fnid can see how many devices they have published and when. That is
inherent to key transparency — a substitution cannot be detected in a log
nobody may read — and it is not a directory, since you need the
identifier first.
`Scripts/watch-key-log.py` is the monitor: stdlib only, forty lines, so
it can be read and run by the person it protects. A hosted service
everybody was enrolled in would learn which accounts are worth watching,
which is a list nobody should be assembling.
⚠️ It says what it cannot see. A relay showing one log to one reader and
a different one to another is the fork §19.4's gossip exists to catch,
and no single watcher sees it — that is relay#23, which needs witnesses
nobody runs yet. And a relay that stops answering reports as "could not
reach", never as "no changes": the first run is a baseline rather than a
report, because treating everything as new the first time trains whoever
runs it to ignore it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The job I added failed with `env: can't execute bash`, and the reason is worse
than a missing package. This project's runner is a docker executor with
`privileged = false` and `volumes = ["/cache"]`, so the job gets an alpine
container with no bash, no docker CLI, and no path to a daemon. I wrote and
tested the script against the host shell over ssh, which is not what CI is.
Wiring it up needs a runner change — privileged, a socket mount, or a shell
executor — on a machine that also serves production. That is not a change to
make on the way past, so the job is removed rather than left failing or, worse,
made to skip. A job that always skips is the pattern I spent this afternoon
removing from the Android suite; adding one here would be indefensible.
The script stays, and says plainly where it has to run and why it is not
automatic. It has earned that much: on its first two runs it found the missing
`docker network create web` and a documented password that broke the relay
about three times in four.
⚠️ So this closes#4 only in the sense that the check now exists and works.
"Nothing boots the quick start in CI" is still true, and now has a named
blocker rather than an absence.
extant2000git
