Relay review wave 2: in-process wake, signin and tokens, scale, streaming attachments #254

Merged

extant2000git merged 19 commits from release/relay-review-wave2-2026-10-14 into main 2026-10-14 16:21:52 +00:00

Conversation 0 Commits 19 Files Changed 60

+4161 -371

extant2000git commented 2026-10-14 15:14:15 +00:00

Owner

Copy Link

Copy Source

Relay review wave 2, reopened from the branch so its image job can see the CI fix. Replaces #252, which was opened over AGit against refs/pull/252/head and could not pick up a change to main.

Same 18 commits, with main merged in (that merge is only .gitea/workflows/ci.yml and Scripts/ci-staging-gate.sh, from #253).

Why #252's image job was red: it ran on mac1 and failed at "staging postgres never became ready". The same two commits passed on mac2. #253 pins the job to mac2 and makes the gate print the container's state and log when Postgres does not come up.

30 files, 4159 insertions. In-process wake, sign-in and account tokens, scale and capacity, streaming attachments, the report digest, metrics, key-log and retention work, plus docs/ for each.

Earlier runs of this branch: swift and checks both green on be5acc5b and on a83e7a57.

🤖 Generated with Claude Code

Relay review wave 2, reopened from the branch so its image job can see the CI fix. Replaces #252, which was opened over AGit against `refs/pull/252/head` and could not pick up a change to main. Same 18 commits, with `main` merged in (that merge is only `.gitea/workflows/ci.yml` and `Scripts/ci-staging-gate.sh`, from #253). Why #252's image job was red: it ran on mac1 and failed at "staging postgres never became ready". The same two commits passed on mac2. #253 pins the job to mac2 and makes the gate print the container's state and log when Postgres does not come up. 30 files, 4159 insertions. In-process wake, sign-in and account tokens, scale and capacity, streaming attachments, the report digest, metrics, key-log and retention work, plus `docs/` for each. Earlier runs of this branch: `swift` and `checks` both green on `be5acc5b` and on `a83e7a57`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

extant2000git added 19 commits 2026-10-14 15:14:16 +00:00

Wake held polls in process and read poll sizes before payloads ... 2e5f0e3d39

One relay keeps a waiter per held poll in memory (WakeHub). The poll
registers once, before its first cursor query, and Delivery.write signals
it after the commit, so a send landing between the query and the wait is
kept and the poll returns at once. A held poll holds no Redis connection.
With FN_RELAY_REPLICAS above 1 wake-ups still go through BRPOP on a Redis
list, behind the same Waiter interface; registration there deletes tokens
left by earlier sends instead of popping them later. A failed wait sleeps
one second before the next cursor query. The Redis pool default is 32 per
loop on one relay and stays 512 with replicas.

The cursor query reads id and payload length first, cuts at the 2 MiB
budget, then loads payloads for the kept rows only. Response shape is
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Document in-process wake-ups and the Redis pool defaults ... 42be0d9116

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Throttle sign-in per address, move bcrypt off the event loop, check account tokens ... 44b3ddc01b

Relay review 2026-10-14, three owner-approved changes, each safe to deploy
before anything else moves.

Sign-in: a second failure bucket per address (FN_SIGNIN_MAX_PER_IP, 30 per
15 minutes, 0 = off) sits beside the address+username pair, so one address
can no longer try a password against every username. A success clears only
the pair. Every bcrypt hash and verify in AccountRoutes and PasswordReset now
runs on application.threadPool through offLoop.

App credentials: the relay mints every one as 32 bytes from the CSPRNG
(randomHexToken). New rows are stored as sha256:<hex>; a bcrypt row is
verified once and rewritten as SHA-256 on success, only when the secret has
the 64-hex shape the relay mints.

Account-service tokens: /messenger/session reads iss, aud (string or list)
and jti. FN_ACCOUNT_TOKEN_STRICT (off by default) logs and counts tokens that
lack them, carry the wrong ones, or replay a jti, and accepts them; on, it
refuses them. jti is spent through the replay cache with a TTL to the token's
exp, after the name checks so a refused handle does not burn the token. Boot
fails with strict on and FN_ACCOUNT_TOKEN_ISSUER unset. SECURITY.md has the
rollout order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Pin federation peer keys, digest reports hourly, serve /metrics on its own port ... 9473d304b4

- FN_FEDERATION_PEERS takes host, host=key or host=key1|key2. A pinned peer
  whose descriptor publishes another key is refused (keyNotPinned, 401).
  Boot warns for each bare host and for each malformed pin.
- Content reports: FN_REPORT_RATE (default 10/hr) per reporter through the
  existing limiter, 429 over it. One mail per report replaced by
  ReportDigest: at most one mail an hour listing reports since the last,
  none when there are none. Watermark in a new report_digests table, seeded
  at the newest existing report; content_reports is unchanged.
- RelayMetrics counters and a NIO listener on FN_METRICS_PORT (off by
  default, 127.0.0.1 by default) answering only GET /metrics. Boot refuses
  FN_METRICS_PORT equal to FN_PORT.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Docs: Postgres for self-hosting, key pinning and rotation, report digest, metrics ... 966c747881

SQLite is documented as local development only, with no new features.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Relay review 2026-10-14: bounded presence, cached log head, set-based retention ... ec6755b15a

- Presence by token: three queries per request (device tags, account
  tags, users) with digest IN (...), down from two or three per token.
  New index roster_tags_digest; device_roster_tags_digest already existed.
- Key log head: KeyLogHeadCache serves (size, root, signature) after one
  MAX(seq) read, and loads only leaves past the cached seq. It refreshes
  on the first read after an append commits, never inside the append's
  transaction, and stays correct across replicas. GET /e2e/log/head is
  metered at 3600 per address per hour.
- Retention: expired devices picked by one SQL query (ids only), orphaned
  push tokens deleted with NOT EXISTS, used prekeys deleted with a bare
  `used` predicate matching the new partial index prekeys_used_created.
- noteAliveOrRefuse reads only suspended_at per request. Last-seen is
  written at most once per heartbeat and once per 60 s per account,
  tracked in memory per process.
- AddScanIndexes builds both indexes CONCURRENTLY on Postgres: FluentKit
  runs migrations outside a transaction. An INVALID leftover is rebuilt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Keep held-poll tests off the shared capacity gauge ... b82c6dfc8e

RelayCapacity.shared is process-global. WakeTests holds 21 polls at once,
which moved the exact peakHeld that CapacityTests asserts. Both now take
one lock (ProcessEnvironmentLock.withExclusiveHeldPolls). The wake-race
test allows 5 s against a 30 s hold, so a loaded runner does not fail it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merge feat/inprocess-wake d5b319b233

Merge feat/signin-and-tokens f1c19902b7

Merge feat/scans-and-scale ... 7064c5c3f4

# Conflicts:
#	Sources/FlamenetRelay/Migrations.swift
#	Sources/FlamenetRelay/Routes.swift

Merge feat/fed-reports-metrics ... f95cd947e5

# Conflicts:
#	Sources/FlamenetRelay/Config.swift
#	Sources/FlamenetRelay/Migrations.swift

Count Redis wake-up failures in /metrics ... eef3b65003

Adds flamenet_wake_errors_total and counts both catch blocks in
RoutesWake (send and wait). Only the replica (Redis) path can fail;
the in-process waiter cannot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Stream attachment uploads and downloads through storage ... ea1b2f228e

Uploads are authenticated before the body is read: token, proof headers,
device and account are checked first, then Content-Length (required, 411
without it) against attachmentMaxBytes and the rate limits. The body then
streams to storage while it is counted and hashed. The proof signature,
which covers the body's SHA-256, is checked after the last byte, and a
blob whose proof fails, or whose body ends short of Content-Length, is
deleted.

S3 PUTs are signed UNSIGNED-PAYLOAD and sent with a known length through
AsyncHTTPClient. The digest is unknown until the last byte, so
x-amz-checksum-sha256 cannot go in a header, and B2 refuses aws-chunked.
GETs stream the bucket's body to the client, Range still asked of the
bucket. The disk store writes to a .partial file and renames it.

Vapor runs a streamed body's closure a second time when writing the
response fails; the bucket body is handed out once, because a second read
traps the process.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Stop test fixtures starving the cooperative pool ... 6ea4f001e0

WakeTests "a send that commits between the cursor query and the wait"
failed in full runs (14 s, 36 s, 42 s against a 5 s bound) and passed
alone. The wake was never lost: with timestamps, the send committed and
signalled at +13.41 s and the poll returned at +14.12 s. The 13 s went
before the send, in a process-wide stall.

A `sample` taken during a stall showed all 8 cooperative threads busy:
7 in Bcrypt.hash at the default cost 12, called synchronously from
fixtures (RosterTagTests, MobileRoutesTests, MobileExtrasTests,
PasswordResetTests account helpers), and 1 in blockingShutdown's
semaphore. One cost-12 hash is 0.31 s in a debug build; cost 4 is
1.3 ms. Every fixture now hashes at cost 4, as WaveZeroBSupport already
did, and a source scan fails if one goes back to the default. The full
run drops from about 60 s to 45 s.

WakeTests now times the poll from the moment the send committed, so a
stall before the send cannot read as a lost wake. With the product's
pending flag disabled it still fails, at 31.8 s.

DeliveryOrderPostgresTests counted every ungranted advisory lock on the
server, including other suites' key-log appends in their own databases,
so the second send could start before the first had inserted ("free"
got id 1). The wait now counts only this database's lock on holdKey.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merge feat/attachment-streaming 2ea4cbe153

Attachments refuse only listed deleted accounts ... 38f2af7276

The streaming upload path repeats the account check before reading the
body, and its copy still refused any subject without a row on an
issuing relay. Same rule as Routes.noteAliveOrRefuse now: refuse only
ids in deleted_accounts. Conformance 266/0/2 (was failing 6 in
sealed attachments); full suite 971/971 twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Document the sign-in and account-token settings; rebuild doc pages ... a83e7a57a8

CI's configuration-doc check found FN_ACCOUNT_TOKEN_ISSUER,
FN_ACCOUNT_TOKEN_STRICT and FN_SIGNIN_MAX_PER_IP read but not
documented. configuration.leaf and relay.leaf regenerated with
Scripts/build-doc-pages.sh; 9 generated pages match their sources.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ci: re-run; the staging gate's postgres never became ready on the runner (run 440) ... be5acc5b64

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merge remote-tracking branch 'gitea/main' into HEAD 404468d983

extant2000git merged commit 6d1013dc89 into main 2026-10-14 16:21:52 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

accuracy

A public page claims something the code does not do

architecture

blocked:apple

Waiting on App Store review

bug

ci

deploy

docs

federation

hardening

idea

investigation

ops

Deploy, CI, infrastructure

parity

One platform has it and the other does not

planning

privacy

Changes what is stored, or what we say is stored

reliability

scalability

security

self-hosting

signup

stability

tech-debt

tooling

unwired

Built in the engine, not reachable from the UI

verification

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

extant2000git

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: extant2000/flamenet-relay#254