extant2000/flamenet-relay
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request 'Relay review wave 2: in-process wake, signin and tokens, scale, streaming attachments' (#254) from release/relay-review-wave2-2026-10-14 into main
ci / checks (push) Successful in 10s
Details
ci / swift (push) Successful in 9m17s
Details
ci / image (push) Successful in 1m1s
Details
ci / deploy (push) Successful in 21s
Details

Browse Source 
swift, checks and image all green on 404468d9. The image job passed on mac2 with the staging gate, which is what #253 fixed; on mac1 it had failed twice at the same step.
This commit was merged in pull request #254.
This commit is contained in:
extant2000git
2026-10-14 16:21:52 +00:00
parent ebdfc87eb9 404468d983
commit 6d1013dc89
60 changed files with 4161 additions and 371 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.env.example
+11 -8
View File
@@ -54,7 +54,7 @@ FN_CONTACT_EMAIL=
# FN_CONTACT_LEGAL=
# FN_CONTACT_POSTMASTER=
# Where a report from inside the app is mailed. Needs the SMTP settings below.
# Where the hourly digest of reports is mailed. Needs the SMTP settings below.
# Blank, reports are still kept in the database and read with
# Scripts/reports.sh, and the log says nobody was told. This is also the
# fallback for FN_CONTACT_EMAIL when that is blank.
@@ -184,7 +184,9 @@ FN_PUBLIC_HOST=
FN_ROSTER_ON_SERVER=true
FN_FEDERATION=closed
# Comma-separated peer hostnames, e.g.: relay.flamenet.io,relay.example.org
# Comma-separated peers. Pin each one's federation key: host=key, or
# host=key1|key2 during a rotation. A bare host is unpinned and logs a warning.
# relay.flamenet.io=<base64 key>,relay.example.org=<base64 key>
FN_FEDERATION_PEERS=
# This relay's federation signing identity, a 32-byte Ed25519 seed:
@@ -246,16 +248,17 @@ FN_S3_PREFIX=attachments/
# Rolling 30-day per-account attachment budget. Totals only, never per-blob.
FN_ATTACHMENT_BYTES_PER_MONTH=524288000
# Redis. docker-compose.yml runs without it: a single relay holds rate limits
# and replay protection in memory, and a held poll wakes on its own timer
# rather than the moment something arrives. docker-compose.prod.yml and
# Redis. docker-compose.yml runs without it: a single relay holds rate limits,
# replay protection and held-poll wake-ups in memory. docker-compose.prod.yml and
# .federated.yml run a Redis and set this themselves. Set it here only if you
# point the base file at a Redis of your own.
# FN_REDIS_URL=redis://redis:6379
# Concurrency knobs, measured in docs/CAPACITY.md. Held long-polls each pin a
# Redis connection; capacity for open apps is roughly event-loops x this.
FN_REDIS_MAX_CONNECTIONS_PER_LOOP=512
# Concurrency knobs, measured in docs/CAPACITY.md. Redis connections per event
# loop: 32 by default for one relay, which waits in memory. With
# FN_RELAY_REPLICAS above 1 each held poll pins one for its BRPOP, the default
# is 512, and capacity for open apps is roughly event-loops x this.
# FN_REDIS_MAX_CONNECTIONS_PER_LOOP=512
#
# Postgres connections are different, and a held poll does NOT pin one. It
# borrows one per cursor query for a few milliseconds. What draws on this is
Package.swift
+6
View File
@@ -19,6 +19,10 @@ let package = Package(
.package(url: "https://github.com/apple/swift-crypto.git", from: "3.0.0"),
// Wake-up fan-out for long-polling.
.package(url: "https://github.com/vapor/redis.git", from: "4.11.0"),
// Already in the graph through Vapor. Named here because the
// attachment store streams through its request and response types,
// which Vapor does not re-export.
.package(url: "https://github.com/swift-server/async-http-client.git", from: "1.21.0"),
],
targets: [
.executableTarget(
@@ -31,6 +35,7 @@ let package = Package(
.product(name: "Redis", package: "redis"),
.product(name: "Leaf", package: "leaf"),
.product(name: "Crypto", package: "swift-crypto"),
.product(name: "AsyncHTTPClient", package: "async-http-client"),
]
),
.testTarget(
@@ -38,6 +43,7 @@ let package = Package(
dependencies: [
"FlamenetRelay",
.product(name: "VaporTesting", package: "vapor"),
.product(name: "AsyncHTTPClient", package: "async-http-client"),
// Only PostgresMigrationTests uses it, and only when
// FN_TEST_POSTGRES names a reachable server: CI has no
// Postgres service, so the dialect production runs on is
README.md
+8 -2
View File
@@ -27,10 +27,13 @@ see `.env.example`.
swift build -c release
FN_ISSUER_PUBLIC_KEY=<base64 Ed25519 public key> \
FN_RELAY_AUDIENCE=https://relay.flamenet.io \
FN_DB=/var/lib/flamenet/relay.sqlite \
FN_POSTGRES_URL=postgres://flamenet:<password>@localhost:5432/relay \
.build/release/flamenet-relay
```
Self-hosting requires Postgres. Without `FN_POSTGRES_URL` the relay uses a
SQLite file, which is for local development only and receives no new features.
It refuses to start without an issuer key. There is no safe default for that
value, and a relay that accepts unsigned tokens is worse than one that is down.
@@ -38,11 +41,14 @@ value, and a relay that accepts unsigned tokens is worse than one that is down.
|---|---|---|
| `FN_ISSUER_PUBLIC_KEY` | n/a | Required. Base64 Ed25519 public key of the token issuer. |
| `FN_RELAY_AUDIENCE` | n/a | Required. The `aud` this relay accepts. |
| `FN_DB` | `relay.sqlite` | SQLite file. |
| `FN_POSTGRES_URL` | unset | Postgres. Required for self-hosting. |
| `FN_DB` | `relay.sqlite` | SQLite file, used without `FN_POSTGRES_URL`. Local development only. |
| `FN_HOST` / `FN_PORT` | `127.0.0.1` / `8080` | Bind address. |
| `FN_WORKDIR` | inferred | Directory containing `Resources/` and `Public/`. Set it in a unit file. |
| `FN_CLOCK_SKEW` | `60` | Seconds of tolerance on a proof timestamp. |
| `FN_FEDERATION` | `closed` | `closed`, `allowlist` or `open`. Closed is a complete relay, not a limited one, and is what a self-hoster gets without configuring anything. An unrecognised value falls back to `closed` rather than refusing to boot, a typo in an optional variable should not stop a relay serving its own users. |
| `FN_FEDERATION_PEERS` | empty | `host`, `host=key` or `host=key1\|key2`. A pinned peer publishing another key is refused. See `docs/CONFIGURATION.md` for rotation. |
| `FN_METRICS_PORT` | unset | Prometheus `/metrics` on a second port, loopback by default. Off unless set. |
| `FN_BUNDLE_RATE` / `FN_BUNDLE_RATE_TARGET` | `600` / `60` | Bundle fetches per hour, per requester and per pair. |
| `FN_SEND_RATE` | `3000` | Envelopes per hour. |
| `FN_DELIVERED_RETENTION_DAYS` | `1` | The backstop for a collected envelope the app never confirms. |
Resources/Views/configuration.leaf
+170 -20
View File
@@ -3,7 +3,7 @@
<!-- GENERATED by Scripts/build-doc-pages.sh from docs/CONFIGURATION.md.
Edits made HERE are deleted the next time that script runs.
Change the markdown source and re-run it.
source-sha256: 6d54d991e7466090 -->
source-sha256: 9135be46efeade65 -->
<article class="band doc doc-generated">
<div class="wrap wrap-narrow">
<h1 id="configuring-a-relay">Configuring a relay</h1>
@@ -113,6 +113,9 @@ only thing that stops a leaked token.</td>
compromise: the signing key is in the same address space. That trade is
for self-hosters, who are running one box either way.</p>
<h2 id="storage">Storage</h2>
<p><strong>A self-hosted relay needs Postgres.</strong> SQLite is for
local development only and receives no new features. Delete-on-ack
already runs only on Postgres, because SQLite reuses row ids.</p>
<table>
<thead>
<tr>
@@ -125,14 +128,14 @@ for self-hosters, who are running one box either way.</p>
<tr>
<td><code>FN_POSTGRES_URL</code></td>
<td>unset</td>
<td>Set it and Postgres is used.</td>
<td>Set it and Postgres is used. Required for any relay with real
accounts.</td>
</tr>
<tr>
<td><code>FN_DB</code></td>
<td><code>relay.sqlite</code></td>
<td>SQLite file, used when the above is unset. <strong>The schema is
identical on both</strong>: SQLite is not a degraded mode, and is the
right choice for a relay serving a household or a company.</td>
<td>SQLite file, used when the above is unset. Local development
only.</td>
</tr>
<tr>
<td><code>FN_ATTACHMENTS_DIR</code></td>
@@ -190,14 +193,17 @@ instance without a shared store keeps its own copy of each.</p>
<tr>
<td><code>FN_REDIS_URL</code></td>
<td>unset</td>
<td>Shared store for the state above. Set but unusable, held polls fall
back to polling and the log says so.</td>
<td>Shared store for the state above. Set but unusable, the log says so.
With <code>FN_RELAY_REPLICAS</code> above 1 it also carries held-poll
wake-ups between relays.</td>
</tr>
<tr>
<td><code>FN_REDIS_MAX_CONNECTIONS_PER_LOOP</code></td>
<td><code>512</code></td>
<td>Redis connections per event loop, which is also the ceiling on held
polls.</td>
<td><code>32</code>, or <code>512</code> with
<code>FN_RELAY_REPLICAS</code> above 1</td>
<td>Redis connections per event loop. With several replicas each held
poll pins one, so it is also the ceiling on held polls. One relay wakes
held polls in memory.</td>
</tr>
<tr>
<td><code>FN_RELAY_REPLICAS</code></td>
@@ -317,14 +323,21 @@ serving its own users.</td>
<tr>
<td><code>FN_FEDERATION_PEERS</code></td>
<td>empty</td>
<td>Comma-separated hostnames, used only under <code>allowlist</code>.
Enforced <strong>outbound as well as inbound</strong>, from the same
rule. Peers named here are fetched wherever they resolve: the outbound
address guard, which refuses private and loopback addresses, is for
names an attacker can choose under <code>open</code> federation, not for
a name you typed and pinned by key. Three relays on one host resolve
each other to private addresses, and would otherwise refuse every
deposit between them as unreachable.</td>
<td>Comma-separated peers, each <code>host</code>, <code>host=key</code>
or <code>host=key1|key2</code>. The key is the peer's federation public
key, base64, as its <code>/.well-known/flamenet/relay</code> publishes
it in <code>federationKey</code>. A pinned peer whose descriptor
publishes any other key is refused with 401. A bare <code>host</code>
accepts whatever key the descriptor publishes, and boot logs a warning
naming it. The host list is used only under <code>allowlist</code>; pins
apply under <code>allowlist</code> and <code>open</code>. Enforced
<strong>outbound as well as inbound</strong>, from the same rule. Peers
named here are fetched wherever they resolve: the outbound address
guard, which refuses private and loopback addresses, is for names an
attacker can choose under <code>open</code> federation, not for a name
you typed and pinned by key. Three relays on one host resolve each other
to private addresses, and would otherwise refuse every deposit between
them as unreachable.</td>
</tr>
<tr>
<td><code>FN_FEDERATION_KEY</code></td>
@@ -336,6 +349,26 @@ stranger discover it.</td>
</tr>
</tbody>
</table>
<h3 id="pinning-and-rotating-a-peers-key">Pinning and rotating a peer's
key</h3>
<p>Read the peer's key and pin it:</p>
<pre class="bash"><code>curl -s https://peer.example/.well-known/flamenet/relay | jq -r .federationKey
FN_FEDERATION_PEERS=peer.example=&lt;that key&gt;</code></pre>
<p>A peer rotating its key tells you the new public key first. Then:</p>
<ol type="1">
<li>Pin both, old first:
<code>peer.example=&lt;old&gt;|&lt;new&gt;</code>. Restart. Both are
accepted.</li>
<li>The peer switches <code>FN_FEDERATION_KEY</code> to the new seed and
restarts.</li>
<li>Wait an hour: the relay caches a peer's descriptor for 3600
seconds.</li>
<li>Drop the old key: <code>peer.example=&lt;new&gt;</code>.
Restart.</li>
</ol>
<p>Rotating your own key is the same steps from the other side: send
each peer the new public key, wait until all of them pin both, then
switch.</p>
<p><code>FN_FEDERATION_KEY</code> is deliberately <strong>not</strong>
the issuer key. A key that only ever needs to say "this relay said this"
should not also be able to mint a token for any user on it.</p>
@@ -402,6 +435,22 @@ anything holding it can impersonate any account.</td>
<td>Which account service to advertise. Point it at your own if you run
one.</td>
</tr>
<tr>
<td><code>FN_ACCOUNT_TOKEN_ISSUER</code></td>
<td>unset</td>
<td>The <code>iss</code> an account-service token must carry. Required
before strict mode can be switched on; the relay refuses to boot with
strict on and this unset.</td>
</tr>
<tr>
<td><code>FN_ACCOUNT_TOKEN_STRICT</code></td>
<td><code>false</code></td>
<td>Off, a token without <code>iss</code>, <code>aud</code> and
<code>jti</code>, or with the wrong ones, is logged, counted and
accepted. On, it is refused with 401, and each <code>jti</code> is
accepted once until the token expires. Turn it on only after the account
service stamps all three (steps in SECURITY.md).</td>
</tr>
</tbody>
</table>
<p><strong>Configured, not fetched.</strong> This decides who may assert
@@ -440,6 +489,14 @@ address and per address-target pair, or this one per presence token when
the caller presents one.</td>
</tr>
<tr>
<td><code>FN_SIGNIN_MAX_PER_IP</code></td>
<td><code>30</code> per 15 min</td>
<td>Failed sign-ins from one address across every username, beside the
per-address-and-username throttle. A success clears only the
per-username count. <strong><code>0</code> switches it
off.</strong></td>
</tr>
<tr>
<td><code>FN_SIGNUP_MAX_PER_HOUR</code></td>
<td><code>50</code>/hr</td>
<td>New accounts per IP. An IP is not a person, carrier CGNAT, offices
@@ -652,11 +709,23 @@ typo, leaves them on.</td>
<tr>
<td><code>FN_REPORTS_TO</code></td>
<td>unset</td>
<td>Where a filed report is mailed, through the SMTP settings. Unset,
reports are kept in <code>content_reports</code> and read with
<td>Where the report digest is mailed, through the SMTP settings. One
message lists every report filed since the last one, at most one an
hour, checked every 5 minutes; an hour with no reports sends nothing. At
most 200 reports per message, each note and copy cut at 1,000
characters. A digest that fails to send is retried with the same
reports. The watermark is the <code>report_digests</code> table, so a
restart neither loses nor repeats a report. Unset, reports are kept in
<code>content_reports</code> and read with
<code>Scripts/reports.sh</code>, and the log says nobody was told.</td>
</tr>
<tr>
<td><code>FN_REPORT_RATE</code></td>
<td><code>10</code>/hr</td>
<td>Reports one account may file per hour. Over it, the route answers
429 and nothing is stored.</td>
</tr>
<tr>
<td><code>FN_REPORT_RETENTION_DAYS</code></td>
<td><code>365</code></td>
<td>How long a content report, and the plaintext copy the reporter chose
@@ -1051,6 +1120,87 @@ bridge beside the relay, deriving its line from the state volume;
<code>Scripts/bridges.sh retire &lt;fingerprint&gt;</code> takes one out
of service. A retired identity is never listed again. The script header
shows how to list a bridge on another machine.</p>
<h2 id="metrics">Metrics</h2>
<p>Off. Set <code>FN_METRICS_PORT</code> and the relay opens a second
listener that answers <code>GET /metrics</code> in the Prometheus text
format and nothing else. The public port never serves
<code>/metrics</code>, and the relay refuses to boot when the two ports
are the same.</p>
<table>
<thead>
<tr>
<th>Variable</th>
<th>Default</th>
<th>Notes</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>FN_METRICS_PORT</code></td>
<td>unset (off)</td>
<td>Port for the metrics listener.</td>
</tr>
<tr>
<td><code>FN_METRICS_HOST</code></td>
<td><code>127.0.0.1</code></td>
<td>Bind address. In Docker, set <code>0.0.0.0</code> so a scraper on
the same network can reach it, and do not publish the port.</td>
</tr>
</tbody>
</table>
<table>
<thead>
<tr>
<th>Metric</th>
<th>Type</th>
<th>Counts</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>flamenet_held_polls</code></td>
<td>gauge</td>
<td>Long-polls held open now.</td>
</tr>
<tr>
<td><code>flamenet_push_sent_total</code></td>
<td>counter</td>
<td>APNs pushes Apple accepted.</td>
</tr>
<tr>
<td><code>flamenet_push_failed_total</code></td>
<td>counter</td>
<td>APNs pushes refused, unreachable or not signed.</td>
</tr>
<tr>
<td><code>flamenet_push_dropped_total</code></td>
<td>counter</td>
<td>Pushes dropped because the push queue was full.</td>
</tr>
<tr>
<td><code>flamenet_limiter_fail_open_total</code></td>
<td>counter</td>
<td>Rate-limit checks allowed because Redis did not answer.</td>
</tr>
<tr>
<td><code>flamenet_envelopes_written_total</code></td>
<td>counter</td>
<td>Envelopes committed.</td>
</tr>
<tr>
<td><code>flamenet_federation_deposits_in_total</code></td>
<td>counter</td>
<td>Deposits accepted from peers.</td>
</tr>
<tr>
<td><code>flamenet_federation_deposits_out_total</code></td>
<td>counter</td>
<td>Deposits a peer accepted from this relay.</td>
</tr>
</tbody>
</table>
<p>Counters start at zero with the process. No metric carries an id, a
name or an address.</p>
<h2 id="site-copy">Site copy</h2>
<table>
<thead>
Resources/Views/relay.leaf
+25 -3
View File
@@ -3,7 +3,7 @@
<!-- GENERATED by Scripts/build-doc-pages.sh from README.md.
Edits made HERE are deleted the next time that script runs.
Change the markdown source and re-run it.
source-sha256: d24ba7ffde0aae1c -->
source-sha256: 202b05628c6a4b2d -->
<article class="band doc doc-generated">
<div class="wrap wrap-narrow">
<h1 id="flamenet-relay">flamenet-relay</h1>
@@ -34,8 +34,11 @@ and off is the right answer for almost everyone see
<pre class="bash"><code>swift build -c release
FN_ISSUER_PUBLIC_KEY=&lt;base64 Ed25519 public key&gt; \
FN_RELAY_AUDIENCE=https://relay.flamenet.io \
FN_DB=/var/lib/flamenet/relay.sqlite \
FN_POSTGRES_URL=postgres://flamenet:&lt;password&gt;@localhost:5432/relay \
.build/release/flamenet-relay</code></pre>
<p>Self-hosting requires Postgres. Without <code>FN_POSTGRES_URL</code>
the relay uses a SQLite file, which is for local development only and
receives no new features.</p>
<p>It refuses to start without an issuer key. There is no safe default
for that value, and a relay that accepts unsigned tokens is worse than
one that is down.</p>
@@ -59,9 +62,15 @@ one that is down.</p>
<td>Required. The <code>aud</code> this relay accepts.</td>
</tr>
<tr>
<td><code>FN_POSTGRES_URL</code></td>
<td>unset</td>
<td>Postgres. Required for self-hosting.</td>
</tr>
<tr>
<td><code>FN_DB</code></td>
<td><code>relay.sqlite</code></td>
<td>SQLite file.</td>
<td>SQLite file, used without <code>FN_POSTGRES_URL</code>. Local
development only.</td>
</tr>
<tr>
<td><code>FN_HOST</code> / <code>FN_PORT</code></td>
@@ -89,6 +98,19 @@ gets without configuring anything. An unrecognised value falls back to
variable should not stop a relay serving its own users.</td>
</tr>
<tr>
<td><code>FN_FEDERATION_PEERS</code></td>
<td>empty</td>
<td><code>host</code>, <code>host=key</code> or
<code>host=key1|key2</code>. A pinned peer publishing another key is
refused. See <code>docs/CONFIGURATION.md</code> for rotation.</td>
</tr>
<tr>
<td><code>FN_METRICS_PORT</code></td>
<td>unset</td>
<td>Prometheus <code>/metrics</code> on a second port, loopback by
default. Off unless set.</td>
</tr>
<tr>
<td><code>FN_BUNDLE_RATE</code> /
<code>FN_BUNDLE_RATE_TARGET</code></td>
<td><code>600</code> / <code>60</code></td>
SECURITY.md
+50
View File
@@ -44,7 +44,57 @@ long, rendered from that instance's own configuration.
throttling and sign-up limits apply to everyone at once.
- Keep `FN_ISSUER_PRIVATE_KEY` and `FN_AT_REST_SECRET` out of the image and off
shared disks. A leaked issuer key lets someone mint tokens for any account.
- Pin every federation peer's key in `FN_FEDERATION_PEERS` (`host=key`).
An unpinned peer is believed with whatever key its descriptor serves, and
boot logs a warning for each one.
- Keep `FN_METRICS_PORT` off the public network. The counters carry no ids,
and they still describe your traffic.
- Raise the file-descriptor limit (`ulimits: nofile: 65536`). Each held
long-poll costs two; the default 1024 caps you near 450 open apps.
- Back up the database *and* rehearse the restore. `docs/CAPACITY.md` has the
measured limits; a backup nobody has restored is a hypothesis.
## Account-service tokens (`FN_ACCOUNT_TOKEN_STRICT`)
`POST /api/flamenet/v1/messenger/session` trades an account-service token for a
relay credential. Today the relay checks the signature, `sub` and `exp`. The
account service stamps no `iss`, `aud` or `jti`, so the relay cannot tell a
token minted for it from one minted for the file server, and the same token
can be traded twice.
Two settings close that:
- `FN_ACCOUNT_TOKEN_ISSUER`: the `iss` a token must carry.
- `FN_ACCOUNT_TOKEN_STRICT`: off by default. Off, a token without `iss`, `aud`
(equal to `FN_RELAY_AUDIENCE`, as a string or inside a list) or `jti`, or
with the wrong value, or with a `jti` already spent, is accepted and logged
as `account-service token accepted with <faults>; <n> such since start`.
On, the same token gets a 401. A `jti` is remembered in the replay cache
(Redis when configured) until the token's `exp`. The relay refuses to boot
with strict on and no issuer set.
Order, so nobody is locked out:
1. Change the account service to stamp `iss`, `aud` (this relay's
`FN_RELAY_AUDIENCE`) and a fresh random `jti` on every token it hands the
app for this relay. A token sent here must be minted for this one exchange:
the second trade of the same token is refused once strict is on.
2. Deploy the relay with `FN_ACCOUNT_TOKEN_ISSUER` set and strict still off.
3. Watch the log for `account-service token accepted with` until it stops
appearing for live sessions (older app builds holding cached tokens drain
within the token lifetime).
4. Set `FN_ACCOUNT_TOKEN_STRICT=true` and redeploy. To back out, set it to
`false`.
## Sign-in limits and app credentials
- Failed sign-ins are counted per address and username (5 per 15 minutes) and
per address across all usernames (`FN_SIGNIN_MAX_PER_IP`, default 30 per 15
minutes, 0 turns it off). Behind an unpinned proxy every client shares one
address, so the second bucket locks out sign-in for everyone after 30
failures; set `FN_TRUSTED_PROXY_ADDRS` first.
- Every bcrypt hash and verify runs on the application thread pool, off the
event loops that carry polls and sends.
- App credentials are 32 random bytes minted by the relay. New ones are stored
as `sha256:<hex>`. An older bcrypt row is verified with bcrypt once and
rewritten as SHA-256 on that success. Passwords stay on bcrypt.
Sources/FlamenetRelay/APNs.swift
+10
View File
@@ -109,6 +109,7 @@ struct APNs: Sendable {
let jwt: String
do { jwt = try await Self.cache.current { try mintJWT() } } catch {
logger.error("APNs token could not be minted: \(error)")
RelayMetrics.shared.add(.pushFailed)
return false
}
@@ -134,11 +135,14 @@ struct APNs: Sendable {
switch response.status.code {
case 200:
await Self.health.succeeded()
RelayMetrics.shared.add(.pushSent)
return false
case 410:
RelayMetrics.shared.add(.pushFailed)
logger.notice("APNs reports a device is gone; forgetting its token")
return true
default:
RelayMetrics.shared.add(.pushFailed)
let reason = response.body.map { String(buffer: $0) } ?? ""
let proven = await Self.health.everWorked
logger.error(
@@ -151,6 +155,7 @@ struct APNs: Sendable {
}
} catch {
// Never propagates.
RelayMetrics.shared.add(.pushFailed)
logger.error("APNs unreachable: \(error)")
return false
}
@@ -162,6 +167,7 @@ struct APNs: Sendable {
let jwt: String
do { jwt = try await Self.cache.current { try mintJWT() } } catch {
logger.error("APNs token could not be minted: \(error)")
RelayMetrics.shared.add(.pushFailed)
return false
}
@@ -184,8 +190,10 @@ struct APNs: Sendable {
switch response.status.code {
case 200:
await Self.health.succeeded()
RelayMetrics.shared.add(.pushSent)
return false
case 410:
RelayMetrics.shared.add(.pushFailed)
logger.notice("APNs reports a VoIP token is gone; forgetting it")
return true
default:
@@ -193,6 +201,7 @@ struct APNs: Sendable {
// the environment said only "failed: 400", the one path where
// the diagnosis matters most, because a call that never rings
// is what somebody notices first.
RelayMetrics.shared.add(.pushFailed)
let reason = response.body.map { String(buffer: $0) } ?? ""
let proven = await Self.health.everWorked
logger.error(
@@ -207,6 +216,7 @@ struct APNs: Sendable {
return proven && reason.contains("BadDeviceToken")
}
} catch {
RelayMetrics.shared.add(.pushFailed)
logger.error("APNs VoIP push could not be sent: \(error)")
return false
}
Sources/FlamenetRelay/AttachmentStore.swift
+174 -27
View File
@@ -1,10 +1,16 @@
import AsyncHTTPClient
import Crypto
import Foundation
import NIOCore
import Vapor
/// Where sealed attachment blobs live.
protocol AttachmentStore: Sendable {
func write(_ token: String, data: Data) async throws
/// Store exactly `length` bytes from `body`, one chunk at a time. When
/// `body` throws or ends at any other length, nothing is left stored.
func write<Body: AsyncSequence & Sendable>(
_ token: String, length: Int, body: Body
) async throws where Body.Element == ByteBuffer
/// The stored size, or nil when the backend cannot say cheaply.
func storedSize(_ token: String) async -> Int?
/// The blob as a Response, or nil when the bytes are gone.
@@ -18,6 +24,24 @@ protocol AttachmentStore: Sendable {
func reachability() async -> StorageReachability
}
extension AttachmentStore {
/// The whole blob at once, for callers that already hold it in memory.
func write(_ token: String, data: Data) async throws {
let buffer = ByteBuffer(data: data)
let body = AsyncStream<ByteBuffer> {
$0.yield(buffer)
$0.finish()
}
try await write(token, length: data.count, body: body)
}
}
/// A body that ended at a length other than the one declared for it.
struct AttachmentLengthMismatch: Error {
let declared: Int
let received: Int
}
/// What a status probe of the blob backend can conclude.
enum StorageReachability: Sendable {
case reachable
@@ -32,12 +56,45 @@ enum StorageReachability: Sendable {
struct DirectoryAttachmentStore: AttachmentStore {
let directory: String
func write(_ token: String, data: Data) async throws {
/// Written to a `.partial` file and renamed into place after the last
/// byte, so a reader never sees half a blob.
func write<Body: AsyncSequence & Sendable>(
_ token: String, length: Int, body: Body
) async throws where Body.Element == ByteBuffer {
try FileManager.default.createDirectory(
atPath: directory, withIntermediateDirectories: true)
try data.write(
to: URL(fileURLWithPath: attachmentPath(directory, token)),
options: .atomic)
let path = attachmentPath(directory, token)
let partial = path + ".partial"
guard FileManager.default.createFile(atPath: partial, contents: nil) else {
throw Abort(.internalServerError, reason: "cannot create \(partial)")
}
do {
let handle = try FileHandle(forWritingTo: URL(fileURLWithPath: partial))
var written = 0
do {
for try await chunk in body {
written += chunk.readableBytes
guard written <= length else { break }
try handle.write(contentsOf: chunk.readableBytesView)
}
try handle.close()
} catch {
try? handle.close()
throw error
}
guard written == length else {
throw AttachmentLengthMismatch(declared: length, received: written)
}
// rename(2) replaces an existing file in one step; `moveItem`
// refuses to.
guard rename(partial, path) == 0 else {
throw Abort(
.internalServerError, reason: "cannot rename \(partial): errno \(errno)")
}
} catch {
try? FileManager.default.removeItem(atPath: partial)
throw error
}
}
func storedSize(_ token: String) async -> Int? {
@@ -75,8 +132,28 @@ struct DirectoryAttachmentStore: AttachmentStore {
// MARK: - S3
/// The HTTP hop to the bucket. Production is AsyncHTTPClient; tests swap in a
/// bucket held in memory.
protocol S3Transport: Sendable {
/// The response body may still be arriving when this returns.
func execute(_ request: HTTPClientRequest) async throws -> HTTPClientResponse
}
/// AsyncHTTPClient streams a request body and a response body. Vapor's
/// `Client` collects both into memory.
struct HTTPClientS3Transport: S3Transport {
let client: HTTPClient
/// The whole exchange, body included. A download is paced by the phone
/// reading it, and 10 MB at 20 KB/s takes about 8 minutes.
var timeout: TimeAmount = .minutes(15)
func execute(_ request: HTTPClientRequest) async throws -> HTTPClientResponse {
try await client.execute(request, timeout: timeout)
}
}
/// An S3-compatible bucket, spoken to directly with SigV4, no SDK, because
/// three verbs against one bucket do not need one.
/// four verbs against one bucket do not need one.
struct S3AttachmentStore: AttachmentStore {
struct S3Config: Sendable {
let endpoint: String // e.g. https://s3.us-west-004.backblazeb2.com
@@ -88,13 +165,13 @@ struct S3AttachmentStore: AttachmentStore {
}
let s3: S3Config
let client: any Client
let transport: any S3Transport
/// The migration fallback, the directory blobs used to live in.
let localFallback: DirectoryAttachmentStore
private func key(_ token: String) -> String { "\(s3.prefix)\(token).bin" }
private func url(_ token: String) -> URI {
URI(string: "\(s3.endpoint)/\(s3.bucket)/\(key(token))")
private func url(_ token: String) -> String {
"\(s3.endpoint)/\(s3.bucket)/\(key(token))"
}
/// Nil, deliberately.
@@ -118,8 +195,7 @@ struct S3AttachmentStore: AttachmentStore {
/// nothing, reads nothing, and cannot return anybody's data.
func reachability() async -> StorageReachability {
do {
let response = try await signedRequest(
.HEAD, token: "status-probe-does-not-exist", body: Data())
let response = try await send(.HEAD, token: "status-probe-does-not-exist")
switch response.status.code {
case 200, 204, 404: return .reachable
default: return .unreachable
@@ -129,8 +205,32 @@ struct S3AttachmentStore: AttachmentStore {
}
}
func write(_ token: String, data: Data) async throws {
let response = try await signedRequest(.PUT, token: token, body: data)
/// One PUT, the body passed through as it arrives.
///
/// The payload is signed as `UNSIGNED-PAYLOAD`. A SigV4 payload hash has
/// to be in the headers before the first byte, and the hash is known only
/// after the last. B2 refuses the `aws-chunked` signing that signs chunk
/// by chunk. `x-amz-checksum-sha256` has the same problem as a header, and
/// as a trailer it needs `aws-chunked` too. Three things protect the bytes
/// instead: TLS to the bucket, the `Content-Length` the bucket holds the
/// upload to, and the client's proof, which signs the SHA-256 of the body
/// and is checked against the bytes that went past.
func write<Body: AsyncSequence & Sendable>(
_ token: String, length: Int, body: Body
) async throws where Body.Element == ByteBuffer {
var request = try signedRequest(.PUT, token: token, payloadHash: "UNSIGNED-PAYLOAD")
request.body = .stream(body, length: .known(Int64(length)))
let response: HTTPClientResponse
do {
response = try await transport.execute(request)
} catch {
// The body ended short, failed, or the connection dropped. A
// bucket does not keep a PUT it did not receive in full; the
// DELETE makes sure of it.
try? await delete(token)
throw error
}
await Self.discardBody(response)
guard response.status == .ok else {
throw Abort(
.internalServerError,
@@ -138,7 +238,8 @@ struct S3AttachmentStore: AttachmentStore {
}
}
/// The range is asked of the bucket, not applied after the fact.
/// The range is asked of the bucket, not applied after the fact, and the
/// bytes go to the client as they arrive from the bucket.
func respond(
_ token: String, request: Request,
range: ClosedRange<Int>?
@@ -150,9 +251,7 @@ struct S3AttachmentStore: AttachmentStore {
value: AttachmentRange.upstreamHeader(
first: range.lowerBound, last: range.upperBound))
}
let response = try await signedRequest(
.GET, token: token, body: Data(),
extraHeaders: extra)
let response = try await send(.GET, token: token, extraHeaders: extra)
// 206 when a range was asked for, 200 when it was not.
if response.status == .ok || response.status == .partialContent {
var headers = HTTPHeaders()
@@ -160,9 +259,20 @@ struct S3AttachmentStore: AttachmentStore {
if let contentRange = response.headers.first(name: .contentRange) {
headers.replaceOrAdd(name: .contentRange, value: contentRange)
}
let body = response.body.map { Response.Body(buffer: $0) } ?? .empty
// -1 sends it chunked when the bucket gave no length.
let count = response.headers.first(name: .contentLength).flatMap { Int($0) } ?? -1
let upstream = TakeOnce(response.body)
let body = Response.Body(
managedAsyncStream: { writer in
// Vapor runs this closure a second time, with a writer
// that only fails, when writing the response to the
// client fails. A bucket body traps on a second read.
guard let upstream = upstream.take() else { return }
for try await chunk in upstream { try await writer.writeBuffer(chunk) }
}, count: count)
return Response(status: response.status, headers: headers, body: body)
}
await Self.discardBody(response)
if response.status == .notFound {
// Pre-switch blobs still on disk.
return try await localFallback.respond(token, request: request, range: range)
@@ -171,7 +281,7 @@ struct S3AttachmentStore: AttachmentStore {
}
func delete(_ token: String) async throws {
let response = try await signedRequest(.DELETE, token: token, body: Data())
let response = try await send(.DELETE, token: token)
// 204 = deleted; 404 = already gone.
guard
response.status == .noContent || response.status == .ok
@@ -186,21 +296,43 @@ struct S3AttachmentStore: AttachmentStore {
// MARK: SigV4
/// A request with no body. The reply body is left unread for a GET and
/// read and dropped for anything else.
private func send(
_ method: HTTPMethod, token: String,
extraHeaders: HTTPHeaders = [:]
) async throws -> HTTPClientResponse {
let response = try await transport.execute(
signedRequest(
method, token: token, payloadHash: Self.emptyPayloadHash,
extraHeaders: extraHeaders))
if method != .GET { await Self.discardBody(response) }
return response
}
/// An error document or nothing. Reading it returns the connection to the
/// pool; dropping it unread closes the connection.
private static func discardBody(_ response: HTTPClientResponse) async {
_ = try? await response.body.collect(upTo: 64 * 1024)
}
/// SHA-256 of zero bytes.
private static let emptyPayloadHash = SHA256.hash(data: Data()).hexString
/// `extraHeaders` are sent but NOT signed, and that is correct rather than
/// a shortcut: SigV4 covers whatever `SignedHeaders` names, and `Range` is
/// deliberately not among them.
private func signedRequest(
_ method: HTTPMethod, token: String,
body: Data,
payloadHash: String,
extraHeaders: HTTPHeaders = [:]
) async throws -> ClientResponse {
) throws -> HTTPClientRequest {
guard let host = URL(string: s3.endpoint)?.host else {
throw Abort(.internalServerError, reason: "S3 endpoint has no host")
}
let now = Date()
let amzDate = Self.amzDateFormatter.string(from: now)
let dateStamp = String(amzDate.prefix(8))
let payloadHash = SHA256.hash(data: body).hexString
let canonicalURI = "/\(s3.bucket)/\(key(token))"
let canonicalHeaders =
@@ -242,11 +374,10 @@ struct S3AttachmentStore: AttachmentStore {
+ "SignedHeaders=\(signedHeaders), Signature=\(signature)")
for header in extraHeaders { headers.add(name: header.name, value: header.value) }
return try await client.send(method, headers: headers, to: url(token)) { req in
if !body.isEmpty {
req.body = ByteBuffer(data: body)
}
}
var request = HTTPClientRequest(url: url(token))
request.method = method
request.headers = headers
return request
}
/// 20260829T012345Z.
@@ -259,6 +390,22 @@ struct S3AttachmentStore: AttachmentStore {
}()
}
/// Hands its value out once.
private final class TakeOnce<Value: Sendable>: @unchecked Sendable {
private let lock = NSLock()
private var value: Value?
init(_ value: Value) { self.value = value }
func take() -> Value? {
lock.lock()
defer { lock.unlock() }
let taken = value
value = nil
return taken
}
}
extension SHA256.Digest {
fileprivate var hexString: String { map { String(format: "%02x", $0) }.joined() }
}
Sources/FlamenetRelay/Config.swift
+81 -4
View File
@@ -76,6 +76,26 @@ struct Config: Sendable {
/// and lock everybody else out; this is what bounds that somebody instead.
var sendRatePerRecipientIP: Int = 300
/// Failed sign-ins per address per fifteen minutes, across every username
/// (`FN_SIGNIN_MAX_PER_IP`, 0 = off).
var signinMaxPerIP: Int = AccountRoutes.signinMaxPerIPDefault
/// Account-service tokens at `/messenger/session`: when true, a token
/// without `iss`, `aud` and `jti`, or with the wrong ones, or a `jti`
/// already spent, is refused. Off, the same tokens are logged, counted and
/// accepted (`FN_ACCOUNT_TOKEN_STRICT`, see SECURITY.md).
var accountTokenStrict: Bool = false
/// The `iss` an account-service token must carry (`FN_ACCOUNT_TOKEN_ISSUER`).
var accountTokenIssuer: String = ""
/// Content reports one account may file per hour (`FN_REPORT_RATE`).
var reportRatePerHour: Int = 10
/// The Prometheus listener: a second port, never the public one. 0, the
/// default, starts no listener. `FN_METRICS_HOST` defaults to loopback.
var metricsHost: String = "127.0.0.1"
var metricsPort: Int = 0
/// Whether this relay will exchange traffic with other relays.
let federationPolicy: String
@@ -84,6 +104,11 @@ struct Config: Sendable {
/// Peers this relay will exchange traffic with under `allowlist`.
let federationPeers: [String]
/// The federation keys pinned for each peer, from `host=key` or
/// `host=key1|key2` in `FN_FEDERATION_PEERS`. A peer with an entry here is
/// refused when its descriptor publishes any other key. A peer without one
/// is unpinned, and boot logs a warning for it.
var federationPeerKeys: [String: [String]] = [:]
/// Whether to screen the allowlist at boot and log what was seen
/// (relay#58).
let iosLive: Bool
@@ -236,6 +261,7 @@ struct Config: Sendable {
return raw == "1" || raw == "true" || raw == "yes" || raw == "on"
}
let peers = parseFederationPeers(env["FN_FEDERATION_PEERS"] ?? "")
var config = Config(
host: env["FN_HOST"] ?? "127.0.0.1",
port: int("FN_PORT", 8080),
@@ -262,10 +288,7 @@ struct Config: Sendable {
}(),
federationKeyRaw: env["FN_FEDERATION_KEY"]
.flatMap { $0.isEmpty ? nil : Data(base64Encoded: $0) },
federationPeers: (env["FN_FEDERATION_PEERS"] ?? "")
.split(separator: ",")
.map { $0.trimmingCharacters(in: .whitespaces) }
.filter { !$0.isEmpty },
federationPeers: peers.hosts,
iosLive: bool("FN_IOS_LIVE", false),
iosStoreURL: env["FN_IOS_STORE_URL"]?.trimmingCharacters(in: .whitespaces) ?? "",
federationScreening: {
@@ -356,6 +379,10 @@ struct Config: Sendable {
.flatMap { $0.isEmpty ? nil : Data(base64Encoded: $0) },
tokenLifetime: int("FN_TOKEN_LIFETIME", Issuer.defaultLifetime)
)
config.federationPeerKeys = peers.keys
config.reportRatePerHour = int("FN_REPORT_RATE", 10)
config.metricsHost = env["FN_METRICS_HOST"].flatMap { $0.isEmpty ? nil : $0 } ?? "127.0.0.1"
config.metricsPort = max(0, int("FN_METRICS_PORT", 0))
config.padEnforce = PadEnforcement(setting: env["FN_PAD_ENFORCE"])
config.padAttachments = bool("FN_PAD_ATTACHMENTS", false)
config.introRatePerIP = int("FN_INTRO_RATE", 30)
@@ -365,9 +392,59 @@ struct Config: Sendable {
// A tenth of the recipient-wide rate unless set.
config.sendRatePerRecipientIP = int(
"FN_SEND_RATE_RECIPIENT_IP", max(1, config.sendRatePerHour / 10))
config.signinMaxPerIP = int("FN_SIGNIN_MAX_PER_IP", AccountRoutes.signinMaxPerIPDefault)
config.accountTokenStrict = bool("FN_ACCOUNT_TOKEN_STRICT", false)
config.accountTokenIssuer =
env["FN_ACCOUNT_TOKEN_ISSUER"]?.trimmingCharacters(in: .whitespaces) ?? ""
return config
}
/// `host`, `host=key` or `host=key1|key2`, comma separated. A key is
/// base64 and may end in `=`, so only the first `=` splits.
static func parseFederationPeers(_ raw: String) -> (hosts: [String], keys: [String: [String]]) {
var hosts: [String] = []
var keys: [String: [String]] = [:]
for entry in raw.split(separator: ",") {
let parts = entry.split(separator: "=", maxSplits: 1)
let host = parts.first.map { $0.trimmingCharacters(in: .whitespaces) } ?? ""
guard !host.isEmpty else { continue }
if !hosts.contains(host) { hosts.append(host) }
guard parts.count == 2 else { continue }
let pins = parts[1].split(separator: "|")
.map { $0.trimmingCharacters(in: .whitespaces) }
.filter { !$0.isEmpty }
// `host=` with nothing after it pins an empty set, which refuses
// every key: a pin the operator started typing fails closed.
keys[host, default: []].append(contentsOf: pins)
}
return (hosts, keys)
}
/// What boot logs about the pins: one line per unpinned peer, and one per
/// pinned value that is not a 32-byte Ed25519 public key (it matches no
/// descriptor, so that peer is refused until it is fixed).
func federationPinWarnings() -> [String] {
guard federationPolicy != "closed" else { return [] }
var lines: [String] = []
for host in federationPeers {
guard let pins = federationPeerKeys[host] else {
lines.append(
"federation peer \(host) is unpinned: any key its descriptor publishes is "
+ "accepted. Pin it as \(host)=<base64 key> in FN_FEDERATION_PEERS.")
continue
}
if pins.isEmpty {
lines.append("federation peer \(host) has an empty pin and is refused")
}
for pin in pins where Data(base64Encoded: pin)?.count != 32 {
lines.append(
"federation peer \(host) has a pinned key that is not a base64 32-byte "
+ "Ed25519 public key; it matches nothing")
}
}
return lines
}
/// Where attachments actually live: the configured value if it is
/// absolute, otherwise relative to the working directory.
func attachmentsPath(relativeTo workingDirectory: String) -> String {
Sources/FlamenetRelay/Delivery.swift
+1
View File
@@ -44,6 +44,7 @@ enum Delivery {
let ids = try await insert(items, on: tx)
return (ids, Set(items.map(\.toDevice)), result)
}
RelayMetrics.shared.add(.envelopesWritten, ids.count)
// After the commit, never inside the transaction.
if !ids.isEmpty { await notify(devices, push: push, on: app) }
return (ids, result)
Sources/FlamenetRelay/Entrypoint.swift
+60 -19
View File
@@ -153,10 +153,12 @@ func configure(_ app: Application) async throws {
// survive a restart and can be shared by more than one replica.
let redisURL = ProcessInfo.processInfo.environment["FN_REDIS_URL"] ?? ""
if !redisURL.isEmpty, let parsed = URL(string: redisURL), let host = parsed.host {
// This number is the held-poll ceiling.
// With several replicas each held poll pins one connection for its
// BRPOP, so this number is the held-poll ceiling. One relay waits in
// memory and uses Redis only for short commands.
let perLoop =
Int(ProcessInfo.processInfo.environment["FN_REDIS_MAX_CONNECTIONS_PER_LOOP"] ?? "")
?? 512
?? (Horizontal.declaredReplicas > 1 ? 512 : 32)
app.redis.configuration = try? RedisConfiguration(
hostname: host,
port: parsed.port ?? 6379,
@@ -212,7 +214,7 @@ func configure(_ app: Application) async throws {
endpoint: endpoint, bucket: config.s3Bucket,
region: config.s3Region, accessKey: config.s3AccessKey,
secretKey: config.s3SecretKey, prefix: config.s3Prefix),
client: app.client,
transport: HTTPClientS3Transport(client: app.http.client.shared),
localFallback: directoryStore)
app.logger.notice(
"attachments in S3, bucket \(config.s3Bucket), reads fall back to disk for pre-switch blobs"
@@ -235,7 +237,8 @@ func configure(_ app: Application) async throws {
audience: config.audience,
skew: config.clockSkew,
directory: peerDirectory,
replay: replayCache),
replay: replayCache,
peerKeys: config.federationPeerKeys),
federationSender: {
guard config.federationPolicy != "closed",
let seed = config.federationKeyRaw
@@ -280,6 +283,7 @@ func configure(_ app: Application) async throws {
"FN_FEDERATION=allowlist with no FN_FEDERATION_PEERS: this relay federates with nobody"
)
}
for line in config.federationPinWarnings() { app.logger.warning("\(line)") }
// relay#58, screen the allowlist and say what was seen.
if config.federationScreening {
let peers = config.federationPeers
@@ -323,7 +327,14 @@ func configure(_ app: Application) async throws {
Mailer(config: $0, group: app.eventLoopGroup, logger: app.logger)
},
siteURL: config.audience,
disposableEmailDomains: config.disposableEmailDomains)
disposableEmailDomains: config.disposableEmailDomains,
signinMaxPerIP: config.signinMaxPerIP,
threadPool: app.threadPool,
accountTokens: AccountTokenPolicy(
strict: config.accountTokenStrict,
issuer: config.accountTokenIssuer,
audience: config.audience,
replay: replayCache))
try register(account: accounts, config: config, on: app)
// The shipped iOS client's surface, mounted where it already looks.
@@ -357,6 +368,13 @@ func configure(_ app: Application) async throws {
let enclaveSecret = Environment.get("FN_ENCLAVE_JWT_SECRET") ?? ""
// Either credential enables the route.
if enclaveSecret.count >= 32 || enclavePublicKey != nil {
// Strict with no issuer would refuse every token; failing the boot
// says why.
if config.accountTokenStrict && config.accountTokenIssuer.isEmpty {
app.logger.critical(
"FN_ACCOUNT_TOKEN_STRICT is on but FN_ACCOUNT_TOKEN_ISSUER is not set")
throw Config.ConfigError.missing("FN_ACCOUNT_TOKEN_ISSUER")
}
compat.post("session") { req in
try await accounts.enclaveSession(
req, secret: enclaveSecret, publicKey: enclavePublicKey, now: unixNow())
@@ -408,6 +426,26 @@ func configure(_ app: Application) async throws {
app.lifecycle.use(BootTask { await retention.run() })
// One mail per hour at most, listing the reports filed since the last.
if config.contentReportsEnabled, let to = config.reportsTo, let mailer = routes.mailer {
let digest = ReportDigest(
db: app.db, to: to,
send: { try await mailer.send(to: $0, subject: $1, body: $2) },
logger: app.logger)
app.lifecycle.use(BootTask { await digest.run() })
}
// Prometheus, on its own listener. Off unless FN_METRICS_PORT is set, and
// never the public port: one socket serving both would put the counters
// on the internet.
if config.metricsPort > 0 {
guard config.metricsPort != config.port else {
app.logger.critical("FN_METRICS_PORT must differ from FN_PORT")
throw Config.ConfigError.missing("an FN_METRICS_PORT other than FN_PORT")
}
app.lifecycle.use(MetricsLifecycle(host: config.metricsHost, port: config.metricsPort))
}
app.logger.info("relay listening on \(config.host):\(config.port) for \(config.audience)")
}
@@ -1330,12 +1368,18 @@ func register(routes r: Routes, on app: Application) throws {
try await r.send(req, body: body(of: req), clientIP: clientIP(req), now: unixNow())
}
// Opt-in, because it deletes ciphertext irrecoverably.
if app.redis.configuration != nil {
app.storage[Wake.EnabledKey.self] = true
app.logger.notice("wake-ups via Redis, held polls block instead of re-querying")
} else if !(ProcessInfo.processInfo.environment["FN_REDIS_URL"] ?? "").isEmpty {
app.logger.critical(
"FN_REDIS_URL is set but unusable; held polls will fall back to polling.")
// One relay wakes held polls in memory. Several relays sharing a
// database wake through Redis, which `Horizontal.enforce` requires.
if Horizontal.declaredReplicas > 1 && app.redis.configuration != nil {
app.storage[Wake.ChannelKey.self] = .redis
app.logger.notice("wake-ups via Redis, shared with the other replicas")
} else {
app.storage[Wake.ChannelKey.self] = .inProcess(WakeHub())
}
if app.redis.configuration == nil
&& !(ProcessInfo.processInfo.environment["FN_REDIS_URL"] ?? "").isEmpty
{
app.logger.critical("FN_REDIS_URL is set but unusable.")
}
// iOS wake-ups.
@@ -1425,14 +1469,11 @@ func register(routes r: Routes, on app: Application) throws {
try await r.setDeliveryTokens(req, body: body(of: req), now: unixNow())
}
// §8.7, the body is a raw ciphertext blob, not JSON, and needs its own
// collection limit: the global default is deliberately small so that an
// unauthenticated sealed submission cannot ask for much memory.
e2e.on(
.POST, "attachments",
body: .collect(maxSize: ByteCount(value: r.config.attachmentMaxBytes))
) { req in
try await r.uploadAttachment(req, body: body(of: req), now: unixNow())
// §8.7, the body is a raw ciphertext blob, not JSON, and is streamed to
// storage rather than collected. `uploadAttachment` authenticates and
// checks Content-Length against the size limit before reading a byte.
e2e.on(.POST, "attachments", body: .stream) { req in
try await r.uploadAttachment(req, now: unixNow())
}
e2e.get("attachments", ":token") { req in
guard let token = req.parameters.get("token") else {
Sources/FlamenetRelay/Federation.swift
+12
View File
@@ -39,6 +39,7 @@ enum Federation {
case unknownPeer(String)
case badSignature
case replayed
case keyNotPinned(String)
var description: String {
switch self {
@@ -49,6 +50,8 @@ enum Federation {
case .unknownPeer(let p): return "could not learn the signing key for \(p)"
case .badSignature: return "federation signature does not verify"
case .replayed: return "this federation request has already been seen"
case .keyNotPinned(let p):
return "the federation key \(p) publishes is not a key this relay pins for it"
}
}
@@ -180,6 +183,8 @@ struct FederationVerifier: Sendable {
let skew: Int
let directory: PeerDirectory?
let replay: any ReplayChecking
/// Pinned keys per peer, base64. See `Config.federationPeerKeys`.
var peerKeys: [String: [String]] = [:]
var enabled: Bool { policy != "closed" }
@@ -212,6 +217,13 @@ struct FederationVerifier: Sendable {
let keyRaw = Data(base64Encoded: keyB64),
let key = try? Curve25519.Signing.PublicKey(rawRepresentation: keyRaw)
else { throw Federation.AuthError.unknownPeer(origin) }
// A pinned peer is believed only with a pinned key. Two pins carry a
// rotation: the old key until the peer switches, the new one after.
if let pins = peerKeys[origin],
!pins.contains(where: { Data(base64Encoded: $0) == keyRaw })
{
throw Federation.AuthError.keyNotPinned(origin)
}
guard let sig = Data(base64Encoded: signature) else {
throw Federation.AuthError.badSignature
Sources/FlamenetRelay/KeyLog.swift
+55
View File
@@ -245,3 +245,58 @@ enum KeyLog {
.compactMap { Data(base64Encoded: $0.leafHash) }
}
}
/// The log head, held in memory (relay review 2026-10-14).
///
/// A head request reads one value, the newest `seq`, off the primary key. When
/// it matches the cached one the cached head is served. When it is higher, only
/// the leaves after the cached `seq` are loaded and the root is recomputed.
///
/// The refresh therefore happens on the first head read after an append
/// commits, and never inside the append's transaction: rows an open
/// transaction inserted are invisible to this read. It is also correct with
/// several replicas, because a replica that did not make the append sees the
/// new `seq` all the same. On Postgres, `KeyLog.append` holds an advisory lock
/// from before the row takes its `seq` until commit, so a committed `seq` has
/// every smaller `seq` that will ever commit already committed. SQLite has one
/// writer. A rolled-back append leaves a gap, which costs nothing here.
actor KeyLogHeadCache {
struct Head: Equatable, Sendable {
let size: Int
let root: Data
/// Nil when this relay has no signing key.
let signature: Data?
}
private var lastSeq = 0
private var leaves: [Data] = []
private var head: Head?
func current(
on db: any Database, sign: @Sendable (Int, Data) -> Data?
) async throws -> Head {
let newest = try await LogEntry.query(on: db).max(\.$id) ?? 0
if newest < lastSeq {
// A smaller log than the one cached: a restore or a different
// database. Start again rather than serve a head it does not have.
lastSeq = 0
leaves = []
head = nil
}
if let head, newest == lastSeq { return head }
let from = lastSeq
let rows = try await LogEntry.query(on: db)
.filter(\.$id > from).sort(\.$id, .ascending)
.field(\.$id).field(\.$leafHash).all()
// Another request extended the cache while this one waited; its head
// is at least as new as the one these rows would give.
if from != lastSeq, let head { return head }
// Same rule as `KeyLog.leaves`: a leaf that does not decode is skipped.
leaves += rows.compactMap { Data(base64Encoded: $0.leafHash) }
lastSeq = max(from, rows.last?.id ?? from)
let root = MerkleLog.root(leaves)
let fresh = Head(size: leaves.count, root: root, signature: sign(leaves.count, root))
head = fresh
return fresh
}
}
Sources/FlamenetRelay/Metrics.swift
+155
View File
@@ -0,0 +1,155 @@
import Foundation
import NIOConcurrencyHelpers
import NIOCore
import NIOHTTP1
import NIOPosix
import Vapor
/// Process-wide counters for the Prometheus endpoint.
///
/// Counts only, never ids or addresses. They start at zero with the process,
/// which is what Prometheus expects of a counter.
final class RelayMetrics: Sendable {
static let shared = RelayMetrics()
enum Counter: String, CaseIterable, Sendable {
case pushSent = "flamenet_push_sent_total"
case pushFailed = "flamenet_push_failed_total"
case limiterFailOpen = "flamenet_limiter_fail_open_total"
case envelopesWritten = "flamenet_envelopes_written_total"
case federationDepositsIn = "flamenet_federation_deposits_in_total"
case federationDepositsOut = "flamenet_federation_deposits_out_total"
case wakeErrors = "flamenet_wake_errors_total"
var help: String {
switch self {
case .pushSent: return "APNs pushes Apple accepted."
case .pushFailed: return "APNs pushes refused, unreachable, or not signed."
case .limiterFailOpen:
return "Rate-limit checks allowed because Redis did not answer."
case .envelopesWritten: return "Envelopes committed to the database."
case .federationDepositsIn: return "Deposits accepted from peer relays."
case .federationDepositsOut: return "Deposits a peer relay accepted from this one."
case .wakeErrors: return "Redis wake-up sends or waits that failed (replica mode only)."
}
}
}
private let counts = NIOLockedValueBox([Counter: Int]())
func add(_ counter: Counter, _ amount: Int = 1) {
guard amount > 0 else { return }
counts.withLockedValue { $0[counter, default: 0] += amount }
}
func value(_ counter: Counter) -> Int {
counts.withLockedValue { $0[counter] ?? 0 }
}
/// The Prometheus text format, version 0.0.4.
func render(heldPolls: Int, pushDropped: Int) -> String {
var out = ""
func metric(_ name: String, _ type: String, _ help: String, _ value: Int) {
out += "# HELP \(name) \(help)\n# TYPE \(name) \(type)\n\(name) \(value)\n"
}
metric("flamenet_held_polls", "gauge", "Long-polls held open right now.", heldPolls)
metric(
"flamenet_push_dropped_total", "counter",
"Pushes dropped because the push queue was full.", pushDropped)
for counter in Counter.allCases {
metric(counter.rawValue, "counter", counter.help, value(counter))
}
return out
}
}
/// A plain HTTP listener that answers `GET /metrics` and nothing else.
///
/// Its own socket on `FN_METRICS_PORT`, so the public port has no `/metrics`
/// route at all.
enum MetricsServer {
static func start(
host: String, port: Int, group: any EventLoopGroup,
render: @escaping @Sendable () -> String
) async throws -> any Channel {
try await ServerBootstrap(group: group)
.serverChannelOption(ChannelOptions.backlog, value: 16)
.serverChannelOption(ChannelOptions.socketOption(.so_reuseaddr), value: 1)
.childChannelInitializer { channel in
channel.pipeline.configureHTTPServerPipeline().flatMap {
channel.pipeline.addHandler(MetricsHandler(render: render))
}
}
.bind(host: host, port: port)
.get()
}
}
private final class MetricsHandler: ChannelInboundHandler, @unchecked Sendable {
typealias InboundIn = HTTPServerRequestPart
typealias OutboundOut = HTTPServerResponsePart
private let render: @Sendable () -> String
private var head: HTTPRequestHead?
init(render: @escaping @Sendable () -> String) { self.render = render }
func channelRead(context: ChannelHandlerContext, data: NIOAny) {
switch unwrapInboundIn(data) {
case .head(let h): head = h
case .body: break
case .end:
guard let head else { return }
let path = head.uri.split(separator: "?", maxSplits: 1).first.map(String.init)
let found = head.method == .GET && path == "/metrics"
let body = found ? render() : "not found\n"
var headers = HTTPHeaders()
headers.add(
name: "Content-Type",
value: found ? "text/plain; version=0.0.4; charset=utf-8" : "text/plain")
headers.add(name: "Content-Length", value: String(body.utf8.count))
headers.add(name: "Connection", value: "close")
let response = HTTPResponseHead(
version: head.version, status: found ? .ok : .notFound, headers: headers)
context.write(wrapOutboundOut(.head(response)), promise: nil)
var buffer = context.channel.allocator.buffer(capacity: body.utf8.count)
buffer.writeString(body)
context.write(wrapOutboundOut(.body(.byteBuffer(buffer))), promise: nil)
let channel = context.channel
context.writeAndFlush(wrapOutboundOut(.end(nil))).whenComplete { _ in
channel.close(promise: nil)
}
self.head = nil
}
}
}
/// Starts the metrics listener after boot and closes it at shutdown.
final class MetricsLifecycle: LifecycleHandler, @unchecked Sendable {
private let host: String
private let port: Int
private var channel: (any Channel)?
init(host: String, port: Int) {
self.host = host
self.port = port
}
func didBootAsync(_ application: Application) async throws {
// Read once: the push worker is installed before boot, and a NIO
// thread should not reach into application storage.
let pushes = application.storage[PushWorker.Key.self]
channel = try await MetricsServer.start(
host: host, port: port, group: application.eventLoopGroup
) {
RelayMetrics.shared.render(
heldPolls: RelayCapacity.shared.snapshot().heldNow,
pushDropped: pushes?.dropped ?? 0)
}
application.logger.notice("metrics on http://\(host):\(port)/metrics")
}
func shutdownAsync(_ application: Application) async {
try? await channel?.close()
}
}
Sources/FlamenetRelay/Migrations.swift
+55
View File
@@ -748,6 +748,57 @@ struct TuneUserDataAutovacuum: AsyncMigration {
}
}
/// Indexes for two lookups that scanned (relay review 2026-10-14).
///
/// - `roster_tags_digest`: presence looks a token up by digest alone, and
/// `roster_tags_user` leads with `user_id`, so it could not serve that.
/// `device_roster_tags` already has `device_roster_tags_digest`.
/// - `prekeys_used_created`: the retention sweep deletes used prekeys by age.
///
/// Postgres builds them CONCURRENTLY, so writes to either table carry on while
/// the index builds. That statement refuses to run inside a transaction, and
/// none is open here: FluentKit's migrator calls `prepare` on the pooled
/// database directly, with no transaction around it, and each raw statement
/// runs in autocommit. A build that failed halfway leaves an INVALID index
/// that `IF NOT EXISTS` would keep, so one of those is dropped first.
struct AddScanIndexes: AsyncMigration {
static let indexes: [(name: String, definition: String)] = [
("roster_tags_digest", "roster_tags (digest)"),
("prekeys_used_created", "prekeys (created_at) WHERE used"),
]
func prepare(on database: any Database) async throws {
guard let sql = database as? any SQLDatabase else { return }
let postgres = sql.dialect.name == "postgresql"
for (name, definition) in Self.indexes {
if postgres {
struct Row: Decodable { let invalid: Bool }
let row = try await sql.raw(
"""
SELECT NOT i.indisvalid AS invalid FROM pg_index i
JOIN pg_class c ON c.oid = i.indexrelid
WHERE c.relname = \(bind: name)
"""
).first(decoding: Row.self)
if row?.invalid == true {
try await sql.raw("DROP INDEX CONCURRENTLY IF EXISTS \(unsafeRaw: name)").run()
}
}
let concurrently = postgres ? "CONCURRENTLY " : ""
try await sql.raw(
"CREATE INDEX \(unsafeRaw: concurrently)IF NOT EXISTS \(unsafeRaw: name) ON \(unsafeRaw: definition)"
).run()
}
}
func revert(on database: any Database) async throws {
guard let sql = database as? any SQLDatabase else { return }
for (name, _) in Self.indexes {
try await sql.raw("DROP INDEX IF EXISTS \(unsafeRaw: name)").run()
}
}
}
/// The schema, in the order Fluent must apply it.
enum RelayMigrations {
/// `pairsCleanup` adds the two irreversible pairs stage 2-A clean-ups.
@@ -814,6 +865,10 @@ enum RelayMigrations {
CreateObfs4Bridges(),
// Additive: ids of deleted accounts, read by the account check.
CreateDeletedAccounts(),
// Additive: two indexes, CONCURRENTLY on Postgres.
AddScanIndexes(),
// Additive: the report digest's watermark. Reports are untouched.
CreateReportDigests(),
]
return list
}
Sources/FlamenetRelay/Padding.swift
+67 -6
View File
@@ -1,4 +1,6 @@
import Fluent
import Foundation
import SQLKit
import Vapor
/// Length hiding, the relay's half (padding stage 1.1, 2026-10-07).
@@ -90,15 +92,74 @@ extension Routes {
static func withinPollBudget(_ rows: [Envelope], budget: Int = pollPayloadBudget)
-> (rows: [Envelope], more: Bool)
{
var kept: [Envelope] = []
let (count, more) = withinPollBudget(sizes: rows.map(\.payload.utf8.count), budget: budget)
return (Array(rows.prefix(count)), more)
}
/// How many of `sizes`, in order, fit the budget (never fewer than one),
/// and whether it cut.
static func withinPollBudget(sizes: [Int], budget: Int = pollPayloadBudget)
-> (count: Int, more: Bool)
{
var bytes = 0
for row in rows {
let size = row.payload.utf8.count
if !kept.isEmpty && bytes + size > budget { return (kept, true) }
kept.append(row)
for (i, size) in sizes.enumerated() {
if i > 0 && bytes + size > budget { return (i, true) }
bytes += size
}
return (kept, false)
return (sizes.count, false)
}
/// Tests read how many payload bytes each cursor query loaded, 0 for an
/// empty one.
struct PollLoadedKey: StorageKey { typealias Value = @Sendable (Int) -> Void }
/// One cursor query, sizes first.
///
/// Reads id and payload length for up to `limit` rows, cuts at the budget
/// (never below one row), then loads payloads for the kept rows only. A
/// poll with 500 padded envelopes waiting loads about 2 MiB, not 32 MB.
/// `more` tells a new client to come straight back; an old one ignores
/// the key and picks the rest up from the cursor on its next poll.
static func pollRows(
toDevice device: String, after: Int, limit: Int, on db: any Database,
app: Application, budget: Int = pollPayloadBudget
) async throws -> (rows: [Envelope], more: Bool) {
guard let sql = db as? any SQLDatabase else {
let all = try await Envelope.query(on: db)
.filter(\.$toDevice == device).filter(\.$id > after)
.sort(\.$id, .ascending).limit(limit).all()
return withinPollBudget(all, budget: budget)
}
// Bytes, as `withinPollBudget` counts them.
let length =
sql.dialect.name == "postgresql"
? "octet_length(payload)" : "length(CAST(payload AS BLOB))"
struct Size: Decodable {
let id: Int
let size: Int
}
let sizes = try await sql.raw(
"""
SELECT id, \(unsafeRaw: length) AS size FROM envelopes
WHERE to_device = \(bind: device) AND id > \(bind: after)
ORDER BY id ASC LIMIT \(bind: limit)
"""
).all(decoding: Size.self)
guard !sizes.isEmpty else {
app.storage[PollLoadedKey.self]?(0)
return ([], false)
}
let (count, more) = withinPollBudget(sizes: sizes.map(\.size), budget: budget)
let last = sizes[count - 1].id
let rows = try await Envelope.query(on: db)
.filter(\.$toDevice == device)
.filter(\.$id > after)
.filter(\.$id <= last)
.sort(\.$id, .ascending)
.limit(count)
.all()
app.storage[PollLoadedKey.self]?(rows.reduce(0) { $0 + $1.payload.utf8.count })
return (rows, more)
}
/// The app product and version from a User-Agent, for counting only.
Sources/FlamenetRelay/PasswordReset.swift
+2 -1
View File
@@ -159,7 +159,8 @@ extension AccountRoutes {
throw WireError("bad_token", "That reset code is not valid.", 400)
}
user.passwordHash = try Bcrypt.hash(input.password)
let password = input.password
user.passwordHash = try await offLoop { try Bcrypt.hash(password) }
try await user.update(on: request.db)
row.usedAt = Date(timeIntervalSince1970: TimeInterval(now))
Sources/FlamenetRelay/Presence.swift
+35
View File
@@ -20,3 +20,38 @@ enum Presence {
return now.timeIntervalSince(lastSeen) < onlineWindow * 2
}
}
/// Which heartbeat this process last wrote for each account.
///
/// Lets the authenticated path skip reading `last_seen_at` and write it at
/// most once per `minimumGap` per account. Heartbeats are already coarsened
/// to `Presence.onlineWindow`, so in steady running this is one write per
/// account per window. Per process: two replicas may each write the same
/// heartbeat once, which is harmless.
actor LastSeenWrites {
static let minimumGap: TimeInterval = 60
/// Above this many accounts, entries older than the online window go.
static let pruneAbove = 100_000
private var written: [Int: (beat: Date, at: Date)] = [:]
/// Whether to write `beat` for this account now. Claims the write when
/// it answers yes, so concurrent requests write once.
func claim(_ userID: Int, beat: Date, now: Date = Date()) -> Bool {
if let last = written[userID],
last.beat >= beat || now.timeIntervalSince(last.at) < Self.minimumGap
{
return false
}
if written.count >= Self.pruneAbove {
written = written.filter { now.timeIntervalSince($0.value.at) < Presence.onlineWindow }
}
written[userID] = (beat, now)
return true
}
/// Forget a claimed write that failed, so the next request retries it.
func release(_ userID: Int) {
written[userID] = nil
}
}
Sources/FlamenetRelay/RateLimiting.swift
+11 -2
View File
@@ -75,6 +75,7 @@ final class RedisRateLimiter: RateLimiting, @unchecked Sendable {
limit: limit, now: now)
}
logger.error("rate limiter (redis) unavailable, allowing: \(error)")
RelayMetrics.shared.add(.limiterFailOpen)
return true
}
}
@@ -103,6 +104,7 @@ final class RedisRateLimiter: RateLimiting, @unchecked Sendable {
return await Self.localFallback.peek(key, limit: limit, now: now)
}
logger.error("rate limiter (redis) unavailable, allowing: \(error)")
RelayMetrics.shared.add(.limiterFailOpen)
return true
}
}
@@ -160,6 +162,9 @@ actor SendMetrics {
protocol ReplayChecking: Sendable {
/// True if this is the first time `signature` has been seen.
func admit(_ signature: String, now: Int) async -> Bool
/// The same, remembered for `ttl` seconds: a one-time token id stays
/// spent until the token itself expires.
func admit(_ signature: String, now: Int, ttl: Int) async -> Bool
}
extension ReplayCache: ReplayChecking {}
@@ -179,17 +184,21 @@ final class RedisReplayCache: ReplayChecking, @unchecked Sendable {
}
func admit(_ signature: String, now: Int) async -> Bool {
await admit(signature, now: now, ttl: window)
}
func admit(_ signature: String, now: Int, ttl: Int) async -> Bool {
do {
let result = try await redis.set(
RedisKey("replay:\(signature)"),
to: String(now),
onCondition: .keyDoesNotExist,
expiration: .seconds(window)
expiration: .seconds(max(ttl, 1))
).get()
return result == .ok
} catch {
logger.error("replay cache (redis) unavailable, falling back to in-process: \(error)")
return await local.admit(signature, now: now)
return await local.admit(signature, now: now, ttl: ttl)
}
}
}
Sources/FlamenetRelay/ReportDigest.swift
+142
View File
@@ -0,0 +1,142 @@
import Fluent
import Foundation
import Logging
import SQLKit
/// One digest that went out: every report with `id <= upToID` has been mailed.
///
/// The watermark lives in the database so a restart neither loses nor repeats
/// reports, and `up_to_id` is the primary key so two replicas that pick the
/// same batch cannot both claim it.
final class ReportDigestMark: Model, @unchecked Sendable {
static let schema = "report_digests"
@ID(custom: "up_to_id", generatedBy: .user) var id: Int?
@Field(key: "sent_at") var sentAt: Int
init() {}
init(upToID: Int, sentAt: Int) {
self.id = upToID
self.sentAt = sentAt
}
}
/// Seeded with the newest report already on disk, so the first digest after a
/// deploy lists only reports filed after it. The ones before were mailed one
/// at a time.
struct CreateReportDigests: AsyncMigration {
func prepare(on database: any Database) async throws {
try await database.schema(ReportDigestMark.schema)
.field("up_to_id", .int, .identifier(auto: false))
.field("sent_at", .int, .required)
.create()
let newest = try await ContentReport.query(on: database).max(\.$id) ?? 0
try await ReportDigestMark(upToID: newest, sentAt: 0).create(on: database)
}
func revert(on database: any Database) async throws {
try await database.schema(ReportDigestMark.schema).delete()
}
}
/// Mails the operator one message listing the reports filed since the last
/// one, at most once an hour, and nothing when there are none.
struct ReportDigest: Sendable {
typealias Send =
@Sendable (_ to: String, _ subject: String, _ body: String) async throws -> Void
let db: any Database
let to: String
let send: Send
let logger: Logger
/// Shortest gap between two digests.
static let interval = 3600
/// How often the loop asks whether one is due.
static let checkEvery: Duration = .seconds(300)
/// Reports per message; the rest wait for the next one.
static let maxPerDigest = 200
/// Characters of each note and copy quoted in the message.
static let excerpt = 1_000
func run() async {
while !Task.isCancelled {
do {
_ = try await sendIfDue(now: Int(Date().timeIntervalSince1970))
} catch {
logger.error("report digest failed: \(error)")
}
try? await Task.sleep(for: Self.checkEvery)
}
}
/// Send one digest if an hour has passed since the last and there is
/// something new. Returns how many reports it listed.
@discardableResult
func sendIfDue(now: Int) async throws -> Int {
let last = try await ReportDigestMark.query(on: db).sort(\.$id, .descending).first()
let watermark = last?.id ?? 0
if let last, now - last.sentAt < Self.interval { return 0 }
let reports = try await ContentReport.query(on: db)
.filter(\.$id > watermark)
.sort(\.$id, .ascending)
.limit(Self.maxPerDigest)
.all()
guard let newest = reports.last?.id else { return 0 }
// Claim the batch before mailing it. A second replica inserting the
// same `up_to_id` fails on the primary key and sends nothing.
let mark = ReportDigestMark(upToID: newest, sentAt: now)
do {
try await mark.create(on: db)
} catch {
logger.notice("report digest up to \(newest) already claimed")
return 0
}
do {
try await send(
to,
"Flamenet reports: \(reports.count) new",
Self.body(reports))
} catch {
// Released, so the next check retries the same reports. Loud,
// because until then nobody has been told they are here.
try? await mark.delete(on: db)
logger.error(
"report digest of \(reports.count) could not be mailed, retrying: \(error)")
throw error
}
return reports.count
}
static func body(_ reports: [ContentReport]) -> String {
func clip(_ text: String) -> String {
text.count > excerpt ? String(text.prefix(excerpt)) + " [cut]" : text
}
var lines = [
"\(reports.count) report\(reports.count == 1 ? " was" : "s were") filed on this relay "
+ "since the last digest.",
"",
]
for r in reports {
lines += [
"id \(r.id.map(String.init) ?? "?")",
"kind \(r.kind)",
"reason \(r.reason)",
"about user \(r.aboutID)",
"filed by user \(r.reporterID)",
"filed at \(ISO8601DateFormatter().string(from: Date(timeIntervalSince1970: TimeInterval(r.createdAt))))",
"What they wrote:",
r.note.isEmpty ? "(nothing)" : clip(r.note),
"The copy they attached:",
r.content.isEmpty ? "(none)" : clip(r.content),
"",
]
}
lines.append("Read the full text with Scripts/reports.sh 1 full.")
return lines.joined(separator: "\n")
}
}
Sources/FlamenetRelay/Retention.swift
+67 -57
View File
@@ -73,9 +73,17 @@ struct Retention: @unchecked Sendable {
.filter(\.$delivered == false)
.filter(\.$createdAt < undeliveredCutoff).delete()
// Consumed one-time prekeys, once no sender can still be holding a
// bundle that references them.
try await PreKey.query(on: db)
.filter(\.$used == true).filter(\.$createdAt < cutoff(7)).delete()
// bundle that references them. `used` is written as the bare column,
// the predicate of the partial index `prekeys_used_created`, so the
// planner can match it.
if let sql = db as? any SQLDatabase {
try await sql.raw(
"DELETE FROM prekeys WHERE used AND created_at < \(bind: cutoff(7))"
).run()
} else {
try await PreKey.query(on: db)
.filter(\.$used == true).filter(\.$createdAt < cutoff(7)).delete()
}
try await PQPreKey.query(on: db)
.filter(\.$used == true).filter(\.$createdAt < cutoff(7)).delete()
@@ -114,19 +122,8 @@ struct Retention: @unchecked Sendable {
// Dead devices.
var expiredDevices = 0
if config.deviceExpiryDays > 0 {
let deviceCutoff = cutoff(config.deviceExpiryDays)
let all = try await Device.query(on: db).all()
var newestPerUser: [Int: Date] = [:]
for d in all {
newestPerUser[d.userID] = max(
newestPerUser[d.userID] ?? .distantPast,
DeviceRetirement.activity(d))
}
for device in all {
let seen = DeviceRetirement.activity(device)
guard seen < deviceCutoff,
seen < newestPerUser[device.userID] ?? .distantPast
else { continue }
for id in try await expiredDeviceIDs(before: cutoff(config.deviceExpiryDays)) {
guard let device = try await Device.find(id, on: db) else { continue }
try await db.retryingTransaction { tx in
try await DeviceRetirement.retire(
device, because: .expired, audience: config.audience, on: tx)
@@ -135,14 +132,9 @@ struct Retention: @unchecked Sendable {
}
}
// Push tokens whose device is gone.
let liveDeviceIDs = Set(try await Device.query(on: db).all().compactMap(\.id))
var orphanedTokens = 0
for row in try await PushToken.query(on: db).all() {
guard let id = row.id, !liveDeviceIDs.contains(id) else { continue }
try await row.delete(on: db)
orphanedTokens += 1
}
// Push tokens whose device is gone. Unlike the pass below, an empty
// devices table still clears every token: a token belongs to a device.
let orphanedTokens = try await deleteOrphans("push_tokens", "device_id")
// Prekeys and envelopes whose device is gone.
//
@@ -232,9 +224,54 @@ struct Retention: @unchecked Sendable {
return readOld + anyOld
}
/// Devices unused since `cutoff` on an account with a device used more
/// recently. Ids only: the rows are loaded one at a time to retire.
///
/// Activity is `DeviceRetirement.activity` in SQL: last seen, else
/// registered. A device with neither counts as used now, so it is never
/// expired and always counts as the newer device.
func expiredDeviceIDs(before cutoff: Date) async throws -> [String] {
guard let sql = db as? any SQLDatabase else { return [] }
struct Row: Decodable { let id: String }
return try await sql.raw(
"""
SELECT d.device_id AS id FROM devices d
WHERE COALESCE(d.last_seen, d.created_at) < \(bind: cutoff)
AND EXISTS (
SELECT 1 FROM devices o WHERE o.user_id = d.user_id
AND COALESCE(o.last_seen, o.created_at, \(bind: Date()))
> COALESCE(d.last_seen, d.created_at)
)
ORDER BY d.device_id
"""
).all(decoding: Row.self).map(\.id)
}
/// Delete the rows of `table` whose `column` names no device, and count
/// them first.
///
/// NOT EXISTS rather than NOT IN: `NOT IN` against a subquery that can
/// yield NULL is never true, and the whole sweep would silently do
/// nothing. Both dialects plan this the same way off the device_id
/// indexes.
func deleteOrphans(_ table: String, _ column: String) async throws -> Int {
guard let sql = db as? any SQLDatabase else { return 0 }
let orphaned: SQLQueryString = """
FROM \(unsafeRaw: table) WHERE NOT EXISTS (
SELECT 1 FROM devices d WHERE d.device_id = \(unsafeRaw: table).\(unsafeRaw: column)
)
"""
struct Row: Decodable { let n: Int }
let before =
try await sql.raw("SELECT count(*) AS n " + orphaned).first(decoding: Row.self)?.n ?? 0
guard before > 0 else { return 0 }
try await sql.raw("DELETE " + orphaned).run()
return before
}
/// Prekey pools and mailboxes belonging to devices that are gone.
private func sweepOrphanedDeviceRows() async throws -> (prekeys: Int, envelopes: Int) {
guard let sql = db as? any SQLDatabase else { return (0, 0) }
guard db is any SQLDatabase else { return (0, 0) }
// Nothing is an orphan when there are no parents to be an orphan of.
//
@@ -245,42 +282,15 @@ struct Retention: @unchecked Sendable {
// it is a state worth refusing to act on.
guard try await Device.query(on: db).count() > 0 else { return (0, 0) }
// NOT EXISTS rather than NOT IN: `NOT IN` against a subquery that can
// yield NULL is never true, and the whole sweep would silently do
// nothing. Both dialects plan this the same way off the existing
// device_id indexes.
func count(_ table: String, _ column: String) async throws -> Int {
struct Row: Decodable { let n: Int }
return try await sql.raw(
"""
SELECT count(*) AS n FROM \(unsafeRaw: table) WHERE NOT EXISTS (
SELECT 1 FROM devices d WHERE d.device_id = \(unsafeRaw: table).\(unsafeRaw: column)
)
"""
).first(decoding: Row.self)?.n ?? 0
}
func purge(_ table: String, _ column: String) async throws -> Int {
let before = try await count(table, column)
guard before > 0 else { return 0 }
try await sql.raw(
"""
DELETE FROM \(unsafeRaw: table) WHERE NOT EXISTS (
SELECT 1 FROM devices d WHERE d.device_id = \(unsafeRaw: table).\(unsafeRaw: column)
)
"""
).run()
return before
}
let prekeys =
try await purge("prekeys", "device_id")
+ purge("pq_prekeys", "device_id")
let envelopes = try await purge("envelopes", "to_device")
try await deleteOrphans("prekeys", "device_id")
+ deleteOrphans("pq_prekeys", "device_id")
let envelopes = try await deleteOrphans("envelopes", "to_device")
// Digests of a device that is gone open nothing useful, but they are
// still a count of somebody's contacts.
let digests =
try await purge("device_delivery_keys", "device_id")
+ purge("device_roster_tags", "device_id")
try await deleteOrphans("device_delivery_keys", "device_id")
+ deleteOrphans("device_roster_tags", "device_id")
if prekeys > 0 || envelopes > 0 || digests > 0 {
// At notice, not debug: in ordinary running this is always zero,
// so any number at all means a device row went out of band.
Sources/FlamenetRelay/Routes.swift
+38 -30
View File
@@ -27,6 +27,10 @@ struct Routes: Sendable {
var mailer: Mailer? = nil
/// Where those go. See `Config.reportsTo`.
var reportsTo: String? = nil
/// Heartbeats this process has written, see `noteAliveOrRefuse`.
let lastSeenWrites = LastSeenWrites()
/// The signed log head, see `KeyLogHeadCache`.
let keyLogHead = KeyLogHeadCache()
static let maxBodyBytes = 4 * 1024 * 1024
@@ -168,8 +172,15 @@ struct Routes: Sendable {
/// A missing row is refused only when the id is listed in
/// deleted_accounts: absence alone does not mean deletion, because a
/// relay may verify tokens for subjects it holds no row for.
///
/// Each request reads the suspension column and nothing else. The write is
/// skipped when this process already wrote the same heartbeat, or wrote
/// any within `LastSeenWrites.minimumGap`.
private func noteAliveOrRefuse(_ userID: Int, on db: any Database) async throws {
guard let user = try await User.find(userID, on: db) else {
guard
let user = try await User.query(on: db).filter(\.$id == userID)
.field(\.$id).field(\.$suspendedAt).first()
else {
if try await DeletedAccount.find(userID, on: db) != nil {
throw WireError.unauthorized("This account no longer exists.")
}
@@ -177,9 +188,14 @@ struct Routes: Sendable {
}
if user.suspendedAt != nil { throw User.suspendedRefusal }
let beat = Presence.coarsen(Date())
guard user.lastSeenAt != beat else { return }
user.lastSeenAt = beat
try? await user.update(on: db)
guard await lastSeenWrites.claim(userID, beat: beat) else { return }
do {
try await User.query(on: db).filter(\.$id == userID)
.set(\.$lastSeenAt, to: beat).update()
} catch {
// Released, so the next request tries the write again.
await lastSeenWrites.release(userID)
}
}
// MARK: - §8.1 POST /e2e/devices
@@ -933,12 +949,16 @@ struct Routes: Sendable {
// How long to hold this request open when there is nothing yet.
let wait = min(max(request.query[Int.self, at: "wait"] ?? 0, 0), 30)
var rows = try await Envelope.query(on: request.db)
.filter(\.$toDevice == deviceID)
.filter(\.$id > after)
.sort(\.$id, .ascending)
.limit(limit)
.all()
// Registered before the first cursor query, so a send that commits
// between that query and the wait still wakes this poll.
let waiter: any Wake.Waiter =
wait > 0
? await Wake.register(device: deviceID, on: request.application) : Wake.NoWaiter()
defer { waiter.close() }
var (rows, more) = try await Self.pollRows(
toDevice: deviceID, after: after, limit: limit, on: request.db,
app: request.application)
// Nothing yet, and the caller is willing to wait: hold the request
// rather than make them come back in five seconds.
@@ -952,16 +972,12 @@ struct Routes: Sendable {
// common case, where the phone loses signal and the task is
// cancelled out from under the loop below.
defer { RelayCapacity.shared.leave() }
await request.application.storage[Wake.HeldKey.self]?(deviceID)
let deadline = Date().addingTimeInterval(Double(hold))
while rows.isEmpty && Date() < deadline {
// Block on a wake-up when there is a Redis to block on, and
// fall back to the interval when there is not.
let remaining = max(1, Int(deadline.timeIntervalSinceNow.rounded(.down)))
if await Wake.wait(
device: deviceID, timeout: remaining,
on: request.application) == false
&& request.application.storage[Wake.EnabledKey.self] != true
{
// A failed wait (Redis down, or no wake channel) sleeps a
// second, so the loop never re-queries back to back.
if await waiter.wait(seconds: deadline.timeIntervalSinceNow) == .failed {
try? await Task.sleep(nanoseconds: 1_000_000_000)
}
if Task.isCancelled { break }
@@ -971,22 +987,14 @@ struct Routes: Sendable {
{
break
}
rows = try await Envelope.query(on: request.db)
.filter(\.$toDevice == deviceID)
.filter(\.$id > after)
.sort(\.$id, .ascending)
.limit(limit)
.all()
(rows, more) = try await Self.pollRows(
toDevice: deviceID, after: after, limit: limit, on: request.db,
app: request.application)
}
}
// Cut at the payload budget, never below one row. `more` tells a new
// client to come straight back; an old one ignores the key and picks
// the rest up on its next poll from the cursor, which only covers
// what was returned.
let (returned, more) = Self.withinPollBudget(rows)
var cursor = after
let messages = returned.map { e -> PollResponse.Message in
let messages = rows.map { e -> PollResponse.Message in
cursor = max(cursor, e.id ?? cursor)
return .init(
id: e.id ?? 0, from_user: e.fromUser, from_device: e.fromDevice,
Sources/FlamenetRelay/RoutesAccount.swift
+176 -8
View File
@@ -1,6 +1,7 @@
import Crypto
import Fluent
import Foundation
import NIOPosix
import Vapor
/// `/account/*`, the built-in identity provider.
@@ -51,9 +52,25 @@ struct AccountRoutes: Sendable {
/// built-in list.
var disposableEmailDomains: Set<String> = []
/// Failed sign-ins per address per window, whatever username each names.
/// 0 turns the bucket off.
var signinMaxPerIP: Int = AccountRoutes.signinMaxPerIPDefault
/// Where bcrypt runs. One verify is about a quarter second of CPU, and on
/// an event loop that quarter second stalls every poll and send sharing
/// the loop. Entrypoint passes `application.threadPool`.
var threadPool: NIOThreadPool = .singleton
/// What `/messenger/session` asks of an account-service token beyond its
/// signature, `sub` and `exp`.
var accountTokens = AccountTokenPolicy()
/// Five failures per IP+username per fifteen minutes, matching the plugin.
static let signinMaxAttempts = 5
static let signinLockout = 900
/// Thirty failures per address per fifteen minutes. The pair bucket above
/// allows one address five guesses at EACH username; this caps the total.
static let signinMaxPerIPDefault = 30
/// Member lookups per caller per window.
static let lookupMaxPerWindow = 20
@@ -183,7 +200,8 @@ struct AccountRoutes: Sendable {
user.displayName =
(input.display_name?.trimmingCharacters(in: .whitespacesAndNewlines))
.flatMap { $0.isEmpty ? nil : $0 } ?? input.username
user.passwordHash = try Bcrypt.hash(input.password)
let password = input.password
user.passwordHash = try await offLoop { try Bcrypt.hash(password) }
user.chosenStatus = ChosenStatus.online.rawValue
try await user.create(on: request.db)
@@ -216,10 +234,18 @@ struct AccountRoutes: Sendable {
let input = try request.content.decode(SigninRequest.self)
let username = User.normalize(input.username)
let throttleKey = "signin:\(clientIP)|\(username)"
// The pair key lets one address try a password against every username
// in turn. This one counts that address's failures across all of them.
let addressKey = "signin:ip:\(clientIP)"
guard await limits.peek(throttleKey, limit: Self.signinMaxAttempts, now: now) else {
throw WireError.rateLimited("Too many sign-in attempts. Try again shortly.")
}
if signinMaxPerIP > 0 {
guard await limits.peek(addressKey, limit: signinMaxPerIP, now: now) else {
throw WireError.rateLimited("Too many sign-in attempts. Try again shortly.")
}
}
// The client's field is "username or email" and always has been, so a
// server that only matched usernames would reject half the people who
@@ -232,13 +258,20 @@ struct AccountRoutes: Sendable {
// Verify against a dummy hash when the user does not exist, so that a
// missing account and a wrong password take the same time.
let hash = Self.hashToVerify(user?.passwordHash)
let ok = (try? Bcrypt.verify(input.password, created: hash)) ?? false
let password = input.password
let ok = (try? await offLoop { try Bcrypt.verify(password, created: hash) }) ?? false
guard ok, let user, let userID = user.id else {
_ = await limits.consume(throttleKey, limit: Self.signinMaxAttempts, now: now)
if signinMaxPerIP > 0 {
_ = await limits.consume(addressKey, limit: signinMaxPerIP, now: now)
}
throw WireError.unauthorized("Incorrect username or password.")
}
// Only the pair is forgiven. Clearing the address count too would let
// somebody holding one real account sign into it between guesses and
// reset their own budget.
await limits.clear(throttleKey)
let secret = try await mintCredential(userID: userID, label: "signin", on: request.db)
return try .json(
@@ -258,6 +291,7 @@ struct AccountRoutes: Sendable {
let claims = try Self.verifyEnclaveToken(
input.token, secret: secret, publicKey: publicKey, now: now)
try await accountTokens.check(claims, logger: request.logger)
// Look up by pseudonym first: after the first sign-in the handle is
// irrelevant, and honouring it would let somebody rename themselves
@@ -266,6 +300,7 @@ struct AccountRoutes: Sendable {
.filter(\.$enclavePseudonym == claims.subject).first(),
let userID = existing.id
{
try await accountTokens.spend(claims, now: now, logger: request.logger)
let credential = try await mintCredential(
userID: userID, label: "flamenet", on: request.db)
return try .json(
@@ -298,6 +333,10 @@ struct AccountRoutes: Sendable {
+ "this server. Pick another.", 409)
}
// Spent after the name checks, so a person told "taken" can try another
// name with the same token.
try await accountTokens.spend(claims, now: now, logger: request.logger)
let user = User()
user.username = username
user.usernameSkeleton = skeleton
@@ -325,7 +364,7 @@ struct AccountRoutes: Sendable {
_ token: String, secret: String,
publicKey: Data? = nil,
now: Int
) throws -> (subject: String, expires: Double) {
) throws -> AccountTokenClaims {
let parts = token.split(separator: ".", omittingEmptySubsequences: false)
guard parts.count == 3 else { throw WireError.unauthorized("Malformed token.") }
@@ -375,7 +414,14 @@ struct AccountRoutes: Sendable {
throw WireError.refused(.tokenExpired, "Token has expired.")
}
return (subject, exp)
// Read here, judged by `AccountTokenPolicy`: `aud` may be one string
// or a list, as JWT allows.
let audiences =
(payload["aud"] as? [String]) ?? (payload["aud"] as? String).map { [$0] } ?? []
return AccountTokenClaims(
subject: subject, expires: exp,
issuer: payload["iss"] as? String, audiences: audiences,
id: payload["jti"] as? String)
}
// MARK: - GET /messenger/me
@@ -622,7 +668,7 @@ struct AccountRoutes: Sendable {
let row = AppCredential()
row.userID = userID
row.selector = selector
row.secretHash = try Bcrypt.hash(secret)
row.secretHash = Self.sha256CredentialHash(secret)
row.label = label
try await row.create(on: db)
return "\(selector).\(secret)"
@@ -641,14 +687,54 @@ struct AccountRoutes: Sendable {
let row = try await AppCredential.query(on: db)
.filter(\.$selector == String(parts[0])).first()
else { return nil }
guard (try? Bcrypt.verify(String(parts[1]), created: row.secretHash)) == true else {
return nil
let secret = String(parts[1])
let stored = row.secretHash
if stored.hasPrefix(Self.sha256CredentialPrefix) {
guard Self.isMintedSecret(secret),
constantTimeEquals(
Data(stored.utf8), Data(Self.sha256CredentialHash(secret).utf8))
else { return nil }
} else {
guard (try? await offLoop { try Bcrypt.verify(secret, created: stored) }) == true
else { return nil }
// A bcrypt row from before 2026-10-14 moves to SHA-256 on its first
// success.
if Self.isMintedSecret(secret) {
row.secretHash = Self.sha256CredentialHash(secret)
}
}
row.lastUsedAt = Date()
try? await row.update(on: db)
return row.userID
}
/// Marks a credential hash stored as `sha256:<64 hex>`.
///
/// Every credential secret is minted here by `randomHexToken`: 32 bytes
/// from the system CSPRNG, hex encoded. Clients never choose one. A secret
/// with 256 bits of entropy cannot be guessed whatever the hash costs, so
/// bcrypt's cost bought nothing and charged a quarter second of CPU on
/// every Basic-auth request. Passwords people type stay on bcrypt.
static let sha256CredentialPrefix = "sha256:"
static func sha256CredentialHash(_ secret: String) -> String {
sha256CredentialPrefix
+ SHA256.hash(data: Data(secret.utf8)).map { String(format: "%02x", $0) }.joined()
}
/// The exact shape `randomHexToken` produces: 64 lowercase hex digits.
/// Only a secret of this shape is moved to, or accepted under, SHA-256.
static func isMintedSecret(_ secret: String) -> Bool {
secret.utf8.count == 64
&& secret.utf8.allSatisfy { (48...57).contains($0) || (97...102).contains($0) }
}
/// Runs `work` on `threadPool`, off the event loop. Every bcrypt hash and
/// verify in this type goes through here.
func offLoop<T: Sendable>(_ work: @escaping @Sendable () throws -> T) async throws -> T {
try await threadPool.runIfActive(work)
}
/// A real bcrypt hash of a value nothing will ever present, used to keep
/// the timing of "no such user" identical to "wrong password".
/// The hash sign-in spends its time on, whatever the account is.
@@ -735,7 +821,7 @@ extension AccountRoutes {
guard password.count >= 8 else {
throw WireError("weak_password", "Passwords must be at least 8 characters.", 400)
}
user.passwordHash = try Bcrypt.hash(password)
user.passwordHash = try await offLoop { try Bcrypt.hash(password) }
}
try await user.update(on: request.db)
@@ -838,3 +924,85 @@ struct UpdateMeRequest: Content {
/// Listed (true) or Unlisted (false), design B.
var discoverable: Bool? = nil
}
// MARK: - Account-service tokens
/// What `verifyEnclaveToken` read from a token whose signature verified.
struct AccountTokenClaims: Sendable {
let subject: String
let expires: Double
let issuer: String?
let audiences: [String]
let id: String?
}
/// `iss`, `aud` and single-use `jti` for account-service tokens.
///
/// Two steps. With `strict` off (the default) a token missing any of the
/// three, or carrying the wrong one, or a spent `jti`, is logged, counted and
/// accepted. With it on, the same token is refused. The account service
/// stamps none of the three today, so it changes first; SECURITY.md has the
/// order.
struct AccountTokenPolicy: Sendable {
var strict = false
/// The `iss` to require. Boot refuses `strict` with this empty.
var issuer = ""
/// The relay's own audience, `FN_RELAY_AUDIENCE`.
var audience = ""
/// Shared with request proofs, so a `jti` spent on one replica is spent on
/// all of them when Redis is configured.
var replay: (any ReplayChecking)? = nil
let lenient = LenientAccountTokens()
/// `iss` and `aud`, checked before anything is looked up.
func check(_ claims: AccountTokenClaims, logger: Logger) async throws {
var faults: [String] = []
if let iss = claims.issuer {
if iss != issuer { faults.append("wrong iss") }
} else {
faults.append("no iss")
}
if claims.audiences.isEmpty {
faults.append("no aud")
} else if !claims.audiences.contains(audience) {
faults.append("wrong aud")
}
try await judge(faults, logger: logger)
}
/// `jti`, spent once, remembered until the token expires.
func spend(_ claims: AccountTokenClaims, now: Int, logger: Logger) async throws {
guard let id = claims.id, !id.isEmpty else {
return try await judge(["no jti"], logger: logger)
}
let ttl = Int(claims.expires.rounded(.up)) - now
if let replay, !(await replay.admit("acct-jti:\(id)", now: now, ttl: ttl)) {
try await judge(["replayed jti"], logger: logger)
}
}
private func judge(_ faults: [String], logger: Logger) async throws {
guard !faults.isEmpty else { return }
let why = faults.joined(separator: ", ")
if strict {
logger.warning("account-service token refused: \(why)")
throw WireError.unauthorized("This sign-in token is not accepted here.")
}
let total = await lenient.record(faults)
logger.warning(
"account-service token accepted with \(why); \(total) such since start; FN_ACCOUNT_TOKEN_STRICT is off"
)
}
}
/// Counts the faults strict mode would have refused, by name.
actor LenientAccountTokens {
private(set) var byFault: [String: Int] = [:]
private(set) var total = 0
func record(_ faults: [String]) -> Int {
for f in faults { byFault[f, default: 0] += 1 }
total += 1
return total
}
}
Sources/FlamenetRelay/RoutesAttachments.swift
+189 -11
View File
@@ -8,16 +8,41 @@ extension Routes {
// MARK: - POST /e2e/attachments
func uploadAttachment(_ request: Request, body: Data, now: Int) async throws -> Response {
// The proof signs a digest of the body, which here is the whole blob.
let caller = try await authenticate(
request, body: body, path: "/e2e/attachments", now: now)
/// The route's entry: the body is the request's own stream, never
/// collected.
func uploadAttachment(_ request: Request, now: Int) async throws -> Response {
try await uploadAttachment(
request, body: request.body, length: Self.declaredLength(of: request), now: now)
}
/// `Content-Length`, or the size of a body that is already in memory.
static func declaredLength(of request: Request) -> Int? {
if let header = request.headers.first(name: .contentLength) { return Int(header) }
return request.body.data?.readableBytes
}
/// Up to 10 MB, streamed to storage without being held.
///
/// Everything that does not need the body is checked before its first
/// byte is read: the token, the proof headers, the device, the account,
/// the size, and the rate limits. The proof's signature covers the
/// SHA-256 of the body, so it is checked after the last byte, against a
/// hash taken as the bytes went past, and a blob whose proof fails is
/// deleted again.
func uploadAttachment<Body: AsyncSequence & Sendable>(
_ request: Request, body: Body, length: Int?, now: Int
) async throws -> Response where Body.Element == ByteBuffer {
let path = "/e2e/attachments"
let (claims, caller) = try await authenticateBeforeBody(request, path: path, now: now)
guard let length else {
throw WireError("length_required", "Content-Length is required.", 411)
}
// An AES-GCM sealed box is at least nonce(12) + tag(16).
guard body.count >= 28 else {
guard length >= 28 else {
throw WireError.badRequest("Attachment body required.")
}
guard body.count <= config.attachmentMaxBytes else {
guard length <= config.attachmentMaxBytes else {
throw WireError("too_large", "Attachment exceeds the size limit.", 413)
}
@@ -25,7 +50,7 @@ extension Routes {
[
(key: "att:count:\(caller.userID)", amount: 1, limit: config.attachmentRatePerHour),
(
key: "att:bytes:\(caller.userID)", amount: body.count,
key: "att:bytes:\(caller.userID)", amount: length,
limit: config.attachmentBytesPerHour
),
], now: now)
@@ -35,7 +60,7 @@ extension Routes {
}
// The month-scale budget (C3).
let withinBudget = await attachmentQuota.consume(
"att:month:\(caller.userID)", amount: body.count,
"att:month:\(caller.userID)", amount: length,
limit: config.attachmentBytesPerMonth, now: now)
guard withinBudget else {
throw WireError(
@@ -48,17 +73,34 @@ extension Routes {
// travels inside the ratchet, so it has to be unguessable, not merely
// unique.
let token = randomHexToken()
let metered = MeteredBody(base: body, declared: length)
do {
try await attachmentStore.write(token, data: body)
try await attachmentStore.write(token, length: length, body: metered)
} catch {
if let mismatch = metered.mismatch {
// Usually the phone dropped the connection part way.
request.logger.notice(
"attachment upload ended at \(mismatch.received) of \(mismatch.declared) bytes")
throw WireError(
"incomplete_body",
"The attachment body did not match its Content-Length.", 400)
}
request.logger.error("attachment write failed: \(error)")
throw WireError("storage_error", "Could not store attachment.", 500)
}
do {
try await authenticateAfterBody(
request, claims: claims, path: path, digest: metered.digest, now: now)
} catch {
try? await attachmentStore.delete(token)
throw error
}
do {
let row = Attachment()
row.id = token
row.bytes = body.count
row.bytes = length
try await row.create(on: request.db)
} catch {
// The row is the only thing retention can see.
@@ -67,7 +109,70 @@ extension Routes {
throw WireError("storage_error", "Could not index attachment.", 500)
}
return try .json(AttachmentUploadResponse(ok: true, id: token, bytes: body.count))
return try .json(AttachmentUploadResponse(ok: true, id: token, bytes: length))
}
/// `authenticated` without the proof's signature, which needs the body.
/// The same checks in the same order, and the same errors.
private func authenticateBeforeBody(
_ request: Request, path: String, now: Int
) async throws -> (TokenClaims, Caller) {
guard let token = request.headers.bearerAuthorization?.token else {
throw WireError.unauthorized()
}
do {
let claims = try verifier.verify(token: token, now: now)
let caller = try verifier.checkProofHeaders(
claims: claims,
timestamp: request.headers.first(name: Headers.timestamp),
nonce: request.headers.first(name: Headers.nonce),
signature: request.headers.first(name: Headers.signature),
now: now)
if let known = try await Device.find(caller.deviceID, on: request.db) {
guard known.ikSig == caller.identityKey.base64EncodedString() else {
throw AuthError.deviceMismatch
}
} else if try await KeyLog.isRetired(deviceID: caller.deviceID, on: request.db) {
throw AuthError.deviceRetired
}
if let user = try await User.find(caller.userID, on: request.db) {
if user.suspendedAt != nil { throw User.suspendedRefusal }
} else if try await DeletedAccount.find(caller.userID, on: request.db) != nil {
// Same rule as Routes.noteAliveOrRefuse: only a listed
// deletion is refused, absence alone is not.
throw WireError.unauthorized("This account no longer exists.")
}
return (claims, caller)
} catch AuthError.deviceRetired {
throw WireError.deviceRevoked
} catch let e as AuthError {
throw WireError.unauthorized(e.description)
}
}
/// The signature over the streamed body's digest, then the replay check.
private func authenticateAfterBody(
_ request: Request, claims: TokenClaims, path: String, digest: String, now: Int
) async throws {
let proofSig = request.headers.first(name: Headers.signature)
do {
_ = try verifier.verifyProof(
claims: claims, method: request.method.string, path: path,
bodyDigest: digest,
timestamp: request.headers.first(name: Headers.timestamp),
nonce: request.headers.first(name: Headers.nonce),
signature: proofSig, now: now)
guard let proofSig, await replay.admit(proofSig, now: now) else {
throw AuthError.replayed
}
} catch AuthError.replayed {
throw WireError(
"proof_replayed",
AuthError.replayed.description + "; sign the request again, the token is fine",
401)
} catch let e as AuthError {
throw WireError.unauthorized(e.description)
}
}
// MARK: - GET /e2e/attachments/{id}
@@ -144,3 +249,76 @@ func randomHexToken() -> String {
raw.map { String(format: "%02x", $0) }.joined()
}
}
/// An upload body on its way to storage, counted and hashed as it passes.
///
/// It throws when the body runs past `declared`, and when it ends short of
/// it. A dropped connection ends a Vapor request stream the same way a
/// finished one does, so the count is the only thing that tells them apart.
struct MeteredBody<Base: AsyncSequence & Sendable>: AsyncSequence, Sendable
where Base.Element == ByteBuffer {
typealias Element = ByteBuffer
let base: Base
let declared: Int
private let state = State()
init(base: Base, declared: Int) {
self.base = base
self.declared = declared
}
/// Hex SHA-256 of every byte that went past.
var digest: String {
state.current.hash.finalize().map { String(format: "%02x", $0) }.joined()
}
/// Set when the body ended at a length other than `declared`.
var mismatch: AttachmentLengthMismatch? { state.current.mismatch }
func makeAsyncIterator() -> Iterator {
Iterator(base: base.makeAsyncIterator(), declared: declared, state: state)
}
struct Iterator: AsyncIteratorProtocol {
var base: Base.AsyncIterator
let declared: Int
let state: State
mutating func next() async throws -> ByteBuffer? {
let chunk = try await base.next()
try state.record(chunk, declared: declared)
return chunk
}
}
final class State: @unchecked Sendable {
struct Values {
var received = 0
var hash = SHA256()
var mismatch: AttachmentLengthMismatch?
}
private let lock = NSLock()
private var values = Values()
var current: Values {
lock.lock()
defer { lock.unlock() }
return values
}
/// Nil is the end of the body.
func record(_ chunk: ByteBuffer?, declared: Int) throws {
lock.lock()
defer { lock.unlock() }
values.received += chunk?.readableBytes ?? 0
if values.received > declared || (chunk == nil && values.received < declared) {
let mismatch = AttachmentLengthMismatch(
declared: declared, received: values.received)
values.mismatch = mismatch
throw mismatch
}
if let chunk { values.hash.update(data: chunk.readableBytesView) }
}
}
}
Sources/FlamenetRelay/RoutesFederation.swift
+2
View File
@@ -72,6 +72,7 @@ extension Routes {
let resolved = try await Self.resolveDeposits(input.envelopes, on: db)
return (resolved.items, resolved.unknown)
}
RelayMetrics.shared.add(.federationDepositsIn)
return try .json(
FederationDepositResponse(
@@ -263,6 +264,7 @@ extension Routes {
accepted += result.accepted
unknown.append(contentsOf: result.unknown)
reached.append(relay)
RelayMetrics.shared.add(.federationDepositsOut)
} catch FederationSender.SendError.notPermitted(let peer) {
// Refusal is NOT unreachability.
refused.append(peer)
Sources/FlamenetRelay/RoutesKeyLog.swift
+23 -9
View File
@@ -6,20 +6,34 @@ import Vapor
/// §19, key transparency.
extension Routes {
/// The signed head.
/// Head reads per address per hour. A client reads the head when it
/// checks a key, so this is far above any one person and below a loop.
static let logHeadPerHour = 3600
/// The signed head, from `keyLogHead`, which loads no leaves while the log
/// is unchanged.
func logHead(_ request: Request) async throws -> Response {
let leaves = try await KeyLog.leaves(on: request.db)
let root = MerkleLog.root(leaves)
let size = leaves.count
let signed = Data("fnkt-v1|\(size)|\(root.base64EncodedString())".utf8)
let now = unixNow()
let meter = "log:head:ip:\(clientIP(request))"
guard await limits.consume(meter, limit: Self.logHeadPerHour, now: now) else {
throw WireError.rateLimited(
"Too many log head requests from here recently.",
retryAfter: await limits.retryAfter([meter], now: now))
}
let seed = config.issuerSigningKeyRaw
let head = try await keyLogHead.current(on: request.db) { size, root in
guard let seed, let key = try? Curve25519.Signing.PrivateKey(rawRepresentation: seed)
else { return nil }
return try? key.signature(
for: Data("fnkt-v1|\(size)|\(root.base64EncodedString())".utf8))
}
let size = head.size
let root = head.root
// A relay running without a signing key (verification only, issuance
// delegated elsewhere) can still serve the log, it just cannot make its
// head into evidence.
guard let seed = config.issuerSigningKeyRaw,
let key = try? Curve25519.Signing.PrivateKey(rawRepresentation: seed),
let signature = try? key.signature(for: signed)
else {
guard let signature = head.signature else {
throw WireError(
"log_unsigned",
"This relay has no signing key, so its log head is unsigned.", 503)
Sources/FlamenetRelay/RoutesMobile.swift
+60 -29
View File
@@ -177,38 +177,65 @@ struct MobileRoutes: Sendable {
static func presence(
for tokens: [String], viewer: Int, on db: any Database
) async throws -> PresenceByTokenResponse {
var rows: [PresenceByTokenResponse.Row] = []
// At most three queries for the whole request: device tags, account
// tags for the digests still unmatched, then the subjects' rows.
var digestOf: [String: String] = [:]
for raw in tokens {
guard let token = Self.validToken(raw),
let subject = try await subject(ofToken: token, on: db)
else {
rows.append(.init(token: raw, presence: EffectiveStatus.offline.rawValue))
continue
}
// Holding the token IS the permission, so this does not go back
// through the buddy table. The subject issued it and can revoke it
// by publishing a set without it.
rows.append(
.init(
token: raw,
presence: try await Self.statusIgnoringRoster(
of: subject, to: viewer, on: db
).rawValue))
guard let token = Self.validToken(raw) else { continue }
digestOf[raw] = Data(SHA256.hash(data: token)).base64EncodedString()
}
return PresenceByTokenResponse(presence: rows)
let subjects = try await subjects(ofDigests: Set(digestOf.values), on: db)
var users: [Int: User] = [:]
if !subjects.isEmpty {
for user in try await User.query(on: db)
.filter(\.$id ~~ Set(subjects.values))
.field(\.$id).field(\.$chosenStatus).field(\.$lastSeenAt).all()
{
if let id = user.id { users[id] = user }
}
}
// Holding the token IS the permission, so this does not go back
// through the buddy table. The subject issued it and can revoke it by
// publishing a set without it.
return PresenceByTokenResponse(
presence: tokens.map { raw in
let subject = digestOf[raw].flatMap { subjects[$0] }
return .init(
token: raw,
presence: Self.status(of: subject.flatMap { users[$0] }).rawValue)
})
}
/// Whose presence a token shows, or nil. A device's own set first, then
/// the account-wide one an older app publishes.
/// Whose presence each digest shows. A device's own set first, then the
/// account-wide one an older app publishes. A digest that matches nothing
/// is absent from the answer.
static func subjects(
ofDigests digests: Set<String>, on db: any Database
) async throws -> [String: Int] {
guard !digests.isEmpty else { return [:] }
var found: [String: Int] = [:]
// Lowest id first, so a digest on two rows resolves as `first()` did.
for tag in try await DeviceRosterTag.query(on: db)
.filter(\.$digest ~~ digests).sort(\.$id, .ascending)
.field(\.$id).field(\.$digest).field(\.$userID).all()
where found[tag.digest] == nil {
found[tag.digest] = tag.userID
}
let rest = digests.subtracting(found.keys)
guard !rest.isEmpty else { return found }
for tag in try await RosterTag.query(on: db)
.filter(\.$digest ~~ rest).sort(\.$id, .ascending)
.field(\.$id).field(\.$digest).field(\.$userID).all()
where found[tag.digest] == nil {
found[tag.digest] = tag.userID
}
return found
}
/// Whose presence one token shows, or nil.
static func subject(ofToken token: Data, on db: any Database) async throws -> Int? {
let digest = Data(SHA256.hash(data: token)).base64EncodedString()
if let tag = try await DeviceRosterTag.query(on: db)
.filter(\.$digest == digest).first()
{
return tag.userID
}
return try await RosterTag.query(on: db)
.filter(\.$digest == digest).first()?.userID
return try await subjects(ofDigests: [digest], on: db)[digest]
}
// MARK: - GET /messenger/users/:id
@@ -515,9 +542,13 @@ struct MobileRoutes: Sendable {
static func statusIgnoringRoster(
of subject: Int, to viewer: Int, on db: any Database
) async throws -> EffectiveStatus {
guard let user = try await User.find(subject, on: db) else { return .offline }
guard let chosen = ChosenStatus(rawValue: user.chosenStatus), chosen != .invisible,
Presence.isFresh(user.lastSeenAt)
status(of: try await User.find(subject, on: db))
}
/// The status a loaded account row shows, offline when there is none.
static func status(of user: User?) -> EffectiveStatus {
guard let user, let chosen = ChosenStatus(rawValue: user.chosenStatus),
chosen != .invisible, Presence.isFresh(user.lastSeenAt)
else { return .offline }
return chosen.effective
}
Sources/FlamenetRelay/RoutesModeration.swift
+11 -40
View File
@@ -68,6 +68,13 @@ extension Routes {
guard submission.about != caller.userID else {
throw WireError("bad_subject", "You cannot report yourself.", 400)
}
// Per reporter, per hour, so one account cannot bury the others.
let admitted = await limits.consumeAll(
[(key: "report:\(caller.userID)", amount: 1, limit: config.reportRatePerHour)],
now: now)
guard admitted else {
throw WireError.rateLimited("Too many reports recently. Try again later.")
}
let content = String((submission.content ?? "").prefix(ContentReport.maximumContent))
let note = String((submission.note ?? "").prefix(ContentReport.maximumContent))
@@ -81,46 +88,10 @@ extension Routes {
report.createdAt = now
try await report.save(on: request.db)
// Tell somebody. A row nobody is told about is a report nobody reads,
// and the person who filed it has been told it reached the operator.
//
// Detached and never able to fail the report: if the mail cannot go
// out, the row is still filed and `Scripts/reports.sh` still finds it.
// Refusing the report because a mail server is down would lose the one
// copy of something somebody went out of their way to send.
if let to = reportsTo, let mailer = self.mailer {
let id = report.id.map(String.init) ?? "unsaved"
let summary = """
A report was filed on this relay.
id \(id)
kind \(report.kind)
reason \(report.reason)
about user \(report.aboutID)
filed by user \(report.reporterID)
What they wrote:
\(report.note.isEmpty ? "(nothing)" : report.note)
The copy they attached:
\(report.content.isEmpty ? "(none)" : report.content)
Read the rest, and mark it seen, with Scripts/reports.sh.
"""
let logger = request.logger
Task.detached {
do {
try await mailer.send(
to: to,
subject: "Flamenet report \(id): \(report.reason) (\(report.kind))",
body: summary)
} catch {
// Loud, because the report is now only in the table and
// nobody has been told it is there.
logger.error("report \(id) filed but could not be mailed: \(error)")
}
}
} else {
// Told in the hourly digest (`ReportDigest`), one message for every
// report since the last, read from this table. One mail per report let
// one account fill the operator's inbox.
if reportsTo == nil || mailer == nil {
let id = report.id.map(String.init) ?? "?"
request.logger.warning(
"report \(id) filed with nowhere to send it; set FN_REPORTS_TO or nobody is told")
Sources/FlamenetRelay/RoutesWake.swift
+198 -9
View File
@@ -1,10 +1,188 @@
import Foundation
import NIOConcurrencyHelpers
import Redis
import Vapor
/// Wake-ups for long-polling.
///
/// A held poll registers a waiter for its device once, before its first
/// cursor query, and `Delivery.write` signals the device after the commit.
/// A signal that lands between the query and the wait is kept on the waiter,
/// so the next wait returns at once.
///
/// One relay process keeps the waiters in memory (`WakeHub`): no Redis
/// connection and no database connection is held while a poll waits. When
/// several relays share one database (`FN_RELAY_REPLICAS` above 1) a send can
/// land on another process, so wake-ups go through a Redis list per device
/// and the waiter blocks on `BRPOP`.
enum Wake {
/// How a held poll's wait ended.
enum Outcome: Equatable, Sendable {
case woken
case timedOut
/// No wake-up channel, or Redis failed. The caller sleeps a second
/// before querying again.
case failed
}
/// One held poll's registration.
protocol Waiter: Sendable {
func wait(seconds: Double) async -> Outcome
/// Unregister. Safe to call more than once.
func close()
}
/// Where wake-ups travel.
enum Channel: Sendable {
case inProcess(WakeHub)
case redis
}
struct ChannelKey: StorageKey { typealias Value = Channel }
/// Tests run code between a poll's first cursor query and its first wait.
struct HeldKey: StorageKey { typealias Value = @Sendable (String) async -> Void }
/// Register a held poll for `device`. Call before the first cursor query.
static func register(device: String, on app: Application) async -> any Waiter {
switch app.storage[ChannelKey.self] {
case .inProcess(let hub):
return hub.register(device: device)
case .redis:
return await RedisWaiter(device: device, app: app)
case nil:
return NoWaiter()
}
}
/// Tell a device something arrived.
static func signal(device: String, on app: Application) async {
switch app.storage[ChannelKey.self] {
case .inProcess(let hub):
hub.signal(device: device)
case .redis:
await RedisWaiter.signal(device: device, on: app)
case nil:
return
}
}
/// No channel: every wait fails, so the poll re-queries once a second.
struct NoWaiter: Waiter {
func wait(seconds: Double) async -> Outcome { .failed }
func close() {}
}
}
/// The in-process waiter map: device id to the polls held for it.
final class WakeHub: Sendable {
private let waiters = NIOLockedValueBox<[String: [WakeHubWaiter]]>([:])
func register(device: String) -> WakeHubWaiter {
let waiter = WakeHubWaiter(device: device, hub: self)
waiters.withLockedValue { $0[device, default: []].append(waiter) }
return waiter
}
func signal(device: String) {
let targets = waiters.withLockedValue { $0[device] ?? [] }
for waiter in targets { waiter.fire(.woken) }
}
func remove(_ waiter: WakeHubWaiter) {
waiters.withLockedValue { map in
guard var list = map[waiter.device] else { return }
list.removeAll { $0 === waiter }
map[waiter.device] = list.isEmpty ? nil : list
}
}
/// Waiters registered now, across all devices.
var count: Int { waiters.withLockedValue { $0.values.reduce(0) { $0 + $1.count } } }
}
final class WakeHubWaiter: Wake.Waiter {
let device: String
private let hub: WakeHub
private struct State {
/// A signal that arrived while nothing was waiting.
var pending = false
var closed = false
var continuation: CheckedContinuation<Wake.Outcome, Never>?
/// Which wait the timer belongs to, so a late timer cannot end the
/// next wait early.
var generation = 0
var timer: Task<Void, Never>?
}
private let state = NIOLockedValueBox(State())
init(device: String, hub: WakeHub) {
self.device = device
self.hub = hub
}
func wait(seconds: Double) async -> Wake.Outcome {
await withTaskCancellationHandler {
await withCheckedContinuation { (c: CheckedContinuation<Wake.Outcome, Never>) in
let now: Wake.Outcome? = state.withLockedValue { s in
if s.pending {
s.pending = false
return .woken
}
if s.closed { return .timedOut }
s.generation += 1
s.continuation = c
let generation = s.generation
let nanos = UInt64(max(0, seconds) * 1_000_000_000)
s.timer = Task { [weak self] in
try? await Task.sleep(nanoseconds: nanos)
self?.fire(.timedOut, generation: generation)
}
return nil
}
if let now { c.resume(returning: now) }
}
} onCancel: {
close()
}
}
/// Resume the wait in progress, or keep a wake for the next one.
func fire(_ outcome: Wake.Outcome, generation: Int? = nil) {
let resume: CheckedContinuation<Wake.Outcome, Never>? = state.withLockedValue { s in
if let generation, generation != s.generation { return nil }
guard let c = s.continuation else {
if outcome == .woken && !s.closed { s.pending = true }
return nil
}
s.continuation = nil
s.timer?.cancel()
s.timer = nil
return c
}
resume?.resume(returning: outcome)
}
func close() {
let resume = state.withLockedValue { s -> CheckedContinuation<Wake.Outcome, Never>? in
s.closed = true
s.pending = false
let c = s.continuation
s.continuation = nil
s.timer?.cancel()
s.timer = nil
return c
}
resume?.resume(returning: .timedOut)
hub.remove(self)
}
}
/// Wake-ups through Redis, for several relays sharing one database.
struct RedisWaiter: Wake.Waiter {
/// Per device, because messages are sealed per device.
static func key(device: String) -> RedisKey { RedisKey("wake:\(device)") }
@@ -14,32 +192,43 @@ enum Wake {
/// Capped.
static let maxDepth = 63
/// Tell a device something arrived.
let device: String
let app: Application
/// Drops tokens left by earlier sends. The first cursor query, which runs
/// after this, sees those rows, so a stale token would only cost a wasted
/// query.
init(device: String, app: Application) async {
self.device = device
self.app = app
_ = try? await app.redis.delete(Self.key(device: device)).get()
}
static func signal(device: String, on app: Application) async {
guard app.storage[EnabledKey.self] == true else { return }
do {
let k = key(device: device)
_ = try await app.redis.lpush("1", into: k).get()
_ = try await app.redis.ltrim(k, before: 0, after: maxDepth).get()
_ = try await app.redis.expire(k, after: .seconds(Int64(ttlSeconds))).get()
} catch {
RelayMetrics.shared.add(.wakeErrors)
app.logger.debug("wake-up for \(device) not sent: \(error)")
}
}
/// Block until a token arrives or `timeout` elapses.
static func wait(device: String, timeout: Int, on app: Application) async -> Bool {
guard app.storage[EnabledKey.self] == true else { return false }
func wait(seconds: Double) async -> Wake.Outcome {
do {
let timeout = max(1, Int64(seconds.rounded(.down)))
let popped = try await app.redis
.brpop(from: key(device: device), timeout: .seconds(Int64(timeout)))
.brpop(from: Self.key(device: device), timeout: .seconds(timeout))
.get()
return popped != nil
return popped == nil ? .timedOut : .woken
} catch {
RelayMetrics.shared.add(.wakeErrors)
app.logger.debug("wake-up wait for \(device) failed: \(error)")
return false
return .failed
}
}
struct EnabledKey: StorageKey { typealias Value = Bool }
func close() {}
}
Sources/FlamenetRelay/Token.swift
+67 -10
View File
@@ -67,6 +67,7 @@ enum AuthError: Error, CustomStringConvertible {
/// Remembers proof signatures for as long as one could still be inside the
/// clock-skew window.
actor ReplayCache {
/// Each entry holds the second it may be admitted again.
private var seen: [String: Int] = [:]
private let window: Int
@@ -74,9 +75,14 @@ actor ReplayCache {
/// True if this is the first time we have seen `signature`.
func admit(_ signature: String, now: Int) -> Bool {
if seen.count > 4096 { seen = seen.filter { now - $0.value < window } }
if let at = seen[signature], now - at < window { return false }
seen[signature] = now
admit(signature, now: now, ttl: window)
}
/// The same, remembered for `ttl` seconds instead of the skew window.
func admit(_ signature: String, now: Int, ttl: Int) -> Bool {
if seen.count > 4096 { seen = seen.filter { $0.value > now } }
if let until = seen[signature], until > now { return false }
seen[signature] = now + max(ttl, 1)
return true
}
}
@@ -131,6 +137,62 @@ struct TokenVerifier: Sendable {
signature: String?,
now: Int
) throws -> Caller {
let digest = SHA256.hash(data: body).map { String(format: "%02x", $0) }.joined()
return try verifyProof(
claims: claims, method: method, path: path, bodyDigest: digest,
timestamp: timestamp, nonce: nonce, signature: signature, now: now)
}
/// The same check over a digest the caller computed, for a body that was
/// streamed to storage instead of held in memory (attachment uploads).
func verifyProof(
claims: TokenClaims,
method: String,
path: String,
bodyDigest digest: String,
timestamp: String?,
nonce: String?,
signature: String?,
now: Int
) throws -> Caller {
let proof = try proofParts(
claims: claims, timestamp: timestamp, nonce: nonce,
signature: signature, now: now)
let canonical = Data(
"\(method)\n\(path)\n\(proof.timestamp)\n\(proof.nonce)\n\(digest)".utf8)
guard proof.deviceKey.isValidSignature(proof.signature, for: canonical) else {
throw AuthError.badProof
}
return proof.caller
}
/// Every part of the proof that can be checked before the body arrives:
/// the headers are present, the timestamp is inside the skew window, and
/// the key and signature decode. The signature covers the body's SHA-256,
/// so verifying it has to wait for the last byte.
func checkProofHeaders(
claims: TokenClaims,
timestamp: String?,
nonce: String?,
signature: String?,
now: Int
) throws -> Caller {
try proofParts(
claims: claims, timestamp: timestamp, nonce: nonce,
signature: signature, now: now
).caller
}
private func proofParts(
claims: TokenClaims,
timestamp: String?,
nonce: String?,
signature: String?,
now: Int
) throws -> (
caller: Caller, deviceKey: Curve25519.Signing.PublicKey,
signature: Data, timestamp: String, nonce: String
) {
guard let timestamp, let signature, let nonce, !nonce.isEmpty else {
throw AuthError.missingProof
}
@@ -141,13 +203,8 @@ struct TokenVerifier: Sendable {
let sigBytes = Data(base64Encoded: signature)
else { throw AuthError.badProof }
let digest = SHA256.hash(data: body).map { String(format: "%02x", $0) }.joined()
let canonical = Data("\(method)\n\(path)\n\(timestamp)\n\(nonce)\n\(digest)".utf8)
guard deviceKey.isValidSignature(sigBytes, for: canonical) else {
throw AuthError.badProof
}
return Caller(userID: claims.sub, deviceID: claims.dev, identityKey: keyBytes)
let caller = Caller(userID: claims.sub, deviceID: claims.dev, identityKey: keyBytes)
return (caller, deviceKey, sigBytes, timestamp, nonce)
}
}
Tests/FlamenetRelayTests/AccountLifecycleTests.swift
+2 -2
View File
@@ -35,7 +35,7 @@ struct AccountLifecycleTests {
u.id = id
u.username = name
u.email = "\(name)@example.com"
u.passwordHash = try Bcrypt.hash("originalpassword")
u.passwordHash = try Bcrypt.hash("originalpassword", cost: 4)
u.displayName = name.capitalized
u.chosenStatus = ChosenStatus.online.rawValue
u.enclavePseudonym = enclave
@@ -46,7 +46,7 @@ struct AccountLifecycleTests {
let c = AppCredential()
c.userID = id
c.selector = selector
c.secretHash = try Bcrypt.hash(secret)
c.secretHash = try Bcrypt.hash(secret, cost: 4)
c.label = "phone"
try await c.create(on: h.db)
return "\(selector).\(secret)"
Tests/FlamenetRelayTests/AttachmentRangeTests.swift
+1 -1
View File
@@ -131,7 +131,7 @@ struct AttachmentRangeTests {
let body = Data(contents.utf8)
let response = try await h.routes.uploadAttachment(
h.request(.POST, "/e2e/attachments", as: d, body: body, now: now),
body: body, now: now)
now: now)
struct Up: Decodable { let id: String }
return try JSONDecoder().decode(
Up.self, from: Data((response.body.string ?? "").utf8)
Tests/FlamenetRelayTests/AttachmentRoutesTests.swift
+2 -2
View File
@@ -22,7 +22,7 @@ struct AttachmentRoutesTests {
) async throws -> Response {
try await h.routes.uploadAttachment(
h.request(.POST, "/e2e/attachments", as: d, body: body, now: now),
body: body, now: now)
now: now)
}
private func token(of res: Response) throws -> String {
@@ -158,7 +158,7 @@ struct AttachmentRoutesTests {
let body = blob(64)
let req = h.anonymousRequest(.POST, "/e2e/attachments", body: body)
await #expect(throws: WireError.self) {
_ = try await h.routes.uploadAttachment(req, body: body, now: now)
_ = try await h.routes.uploadAttachment(req, now: now)
}
}
}
Tests/FlamenetRelayTests/AttachmentStreamingTests.swift
+452
View File
@@ -0,0 +1,452 @@
import AsyncHTTPClient
import Crypto
import Fluent
import Foundation
import NIOCore
import Testing
import Vapor
@testable import FlamenetRelay
/// Uploads streamed to storage and downloads streamed from it (relay review
/// 2026-10-14): nothing read before auth, nothing stored that did not arrive
/// whole, and no blob held in memory on the way out.
@Suite("attachment streaming")
struct AttachmentStreamingTests {
private let now = 1_700_000_000
// MARK: - Helpers
/// Records whether anybody asked it for a byte.
struct UntouchedBody: AsyncSequence, Sendable {
typealias Element = ByteBuffer
let touched = Flag()
final class Flag: @unchecked Sendable {
private let lock = NSLock()
private var _value = false
var value: Bool { lock.withLock { _value } }
func set() { lock.withLock { _value = true } }
}
struct AsyncIterator: AsyncIteratorProtocol {
let touched: Flag
mutating func next() async throws -> ByteBuffer? {
touched.set()
return nil
}
}
func makeAsyncIterator() -> AsyncIterator { AsyncIterator(touched: touched) }
}
/// `data` in pieces of `size`, as a socket delivers it.
private func pieces(_ data: Data, size: Int) -> AsyncStream<ByteBuffer> {
AsyncStream { continuation in
var offset = 0
while offset < data.count {
let end = min(offset + size, data.count)
continuation.yield(ByteBuffer(data: data.subdata(in: offset..<end)))
offset = end
}
continuation.finish()
}
}
private func s3(_ bucket: FakeBucket, fallback: String) -> S3AttachmentStore {
S3AttachmentStore(
s3: .init(
endpoint: "https://s3.example.com", bucket: "blobs",
region: "us-west-004", accessKey: "AKIA", secretKey: "secret",
prefix: "attachments/"),
transport: bucket,
localFallback: DirectoryAttachmentStore(directory: fallback))
}
/// The harness's routes over a different store.
private func routes(_ h: RelayHarness, store: any AttachmentStore) -> FlamenetRelay.Routes {
let r = h.routes
return FlamenetRelay.Routes(
config: r.config, verifier: r.verifier, limits: r.limits,
sendMetrics: r.sendMetrics, federation: r.federation,
federationSender: r.federationSender, replay: r.replay,
attachmentStore: store, attachmentQuota: r.attachmentQuota)
}
private func device(_ h: RelayHarness) async throws -> RelayHarness.TestDevice {
let d = h.device()
try await h.register(d, on: h.db)
return d
}
private func blob(_ n: Int) -> Data {
Data((0..<n).map { UInt8(truncatingIfNeeded: $0 &* 31) })
}
private func code(_ error: (any Error)?) -> String? { (error as? WireError)?.code }
// MARK: - Before the body
@Test("an unauthenticated upload is refused without reading the body")
func unauthenticatedReadsNothing() async throws {
try await withRelay { h in
let d = try await device(h)
// No credentials at all.
let anonymous = UntouchedBody()
await #expect(throws: WireError.self) {
_ = try await h.routes.uploadAttachment(
h.anonymousRequest(.POST, "/e2e/attachments"),
body: anonymous, length: 64, now: now)
}
#expect(!anonymous.touched.value)
// A good token with the proof headers stripped.
let request = h.request(.POST, "/e2e/attachments", as: d, body: blob(64), now: now)
request.headers.remove(name: FlamenetRelay.Routes.Headers.signature)
let stripped = UntouchedBody()
await #expect(throws: WireError.self) {
_ = try await h.routes.uploadAttachment(
request, body: stripped, length: 64, now: now)
}
#expect(!stripped.touched.value)
// An expired token.
let expired = h.request(
.POST, "/e2e/attachments", as: d, body: blob(64), now: now,
token: h.token(for: d, now: now - 7200, expiresIn: 60))
let late = UntouchedBody()
let err = await #expect(throws: WireError.self) {
_ = try await h.routes.uploadAttachment(expired, body: late, length: 64, now: now)
}
#expect((err as? WireError)?.data.status == 401)
#expect(!late.touched.value)
}
}
@Test("an oversize or missing Content-Length is refused without reading the body")
func oversizeReadsNothing() async throws {
try await withRelay(env: ["FN_ATTACHMENT_MAX_BYTES": "1024"]) { h in
let d = try await device(h)
let big = UntouchedBody()
let err = await #expect(throws: WireError.self) {
_ = try await h.routes.uploadAttachment(
h.request(.POST, "/e2e/attachments", as: d, now: now),
body: big, length: 1025, now: now)
}
#expect(code(err) == "too_large")
#expect(!big.touched.value)
let unsized = UntouchedBody()
let missing = await #expect(throws: WireError.self) {
_ = try await h.routes.uploadAttachment(
h.request(.POST, "/e2e/attachments", as: d, now: now),
body: unsized, length: nil, now: now)
}
#expect(code(missing) == "length_required")
#expect(!unsized.touched.value)
#expect(try await Attachment.query(on: h.db).count() == 0)
}
}
/// The route reads the header, and a streamed request has no other
/// length.
@Test("the declared length comes from Content-Length")
func declaredLength() async throws {
try await withRelay { h in
let request = h.anonymousRequest(.POST, "/e2e/attachments", body: blob(10))
#expect(FlamenetRelay.Routes.declaredLength(of: request) == 10)
request.headers.replaceOrAdd(name: .contentLength, value: "4096")
#expect(FlamenetRelay.Routes.declaredLength(of: request) == 4096)
}
}
// MARK: - The body
@Test("a streamed upload lands whole in the bucket, signed UNSIGNED-PAYLOAD")
func streamedUploadToBucket() async throws {
try await withRelay { h in
let d = try await device(h)
let bucket = FakeBucket()
let r = routes(h, store: s3(bucket, fallback: h.attachmentDir))
let contents = blob(100_000)
let response = try await r.uploadAttachment(
h.request(.POST, "/e2e/attachments", as: d, body: contents, now: now),
body: pieces(contents, size: 4096), length: contents.count, now: now)
#expect(response.status == .ok)
#expect(bucket.objects.count == 1)
#expect(bucket.objects.values.first == contents)
let put = try #require(bucket.requests.first { $0.method == .PUT })
#expect(put.headers.first(name: "x-amz-content-sha256") == "UNSIGNED-PAYLOAD")
#expect(
put.headers.first(name: "authorization")?.contains("x-amz-content-sha256") == true)
let row = try #require(try await Attachment.query(on: h.db).first())
#expect(row.bytes == contents.count)
}
}
@Test("an upload that ends short leaves no object, no file and no row")
func shortUploadLeavesNothing() async throws {
try await withRelay { h in
let d = try await device(h)
let contents = blob(2000)
let bucket = FakeBucket()
let stores: [any AttachmentStore] = [
s3(bucket, fallback: h.attachmentDir),
DirectoryAttachmentStore(directory: h.attachmentDir),
]
for store in stores {
let err = await #expect(throws: WireError.self) {
_ = try await routes(h, store: store).uploadAttachment(
h.request(.POST, "/e2e/attachments", as: d, body: contents, now: now),
body: pieces(contents.prefix(1500), size: 500),
length: contents.count, now: now)
}
#expect(code(err) == "incomplete_body")
}
#expect(bucket.objects.isEmpty)
#expect(bucket.requests.contains { $0.method == .DELETE })
let left = (try? FileManager.default.contentsOfDirectory(atPath: h.attachmentDir)) ?? []
#expect(left.isEmpty, "left on disk: \(left)")
#expect(try await Attachment.query(on: h.db).count() == 0)
}
}
@Test("a body that runs past its Content-Length is refused and not kept")
func longUploadLeavesNothing() async throws {
try await withRelay { h in
let d = try await device(h)
let contents = blob(2000)
let err = await #expect(throws: WireError.self) {
_ = try await h.routes.uploadAttachment(
h.request(.POST, "/e2e/attachments", as: d, body: contents, now: now),
body: pieces(contents, size: 500), length: 1500, now: now)
}
#expect(code(err) == "incomplete_body")
let left = (try? FileManager.default.contentsOfDirectory(atPath: h.attachmentDir)) ?? []
#expect(left.isEmpty, "left on disk: \(left)")
}
}
/// The proof signs the body's digest, which exists only after the last
/// byte. A body other than the one signed is stored, then refused and
/// deleted.
@Test("a body that does not match the proof is deleted and refused")
func proofMismatchDeletes() async throws {
try await withRelay { h in
let d = try await device(h)
let bucket = FakeBucket()
let r = routes(h, store: s3(bucket, fallback: h.attachmentDir))
let signed = blob(4096)
var sent = signed
sent[100] ^= 0xFF
let err = await #expect(throws: WireError.self) {
_ = try await r.uploadAttachment(
h.request(.POST, "/e2e/attachments", as: d, body: signed, now: now),
body: pieces(sent, size: 1024), length: sent.count, now: now)
}
#expect((err as? WireError)?.data.status == 401)
#expect(bucket.requests.contains { $0.method == .PUT })
#expect(bucket.objects.isEmpty)
#expect(try await Attachment.query(on: h.db).count() == 0)
}
}
@Test("a replayed upload is refused after the body and not kept")
func replayedUploadDeletes() async throws {
try await withRelay { h in
let d = try await device(h)
let contents = blob(256)
let request = h.request(.POST, "/e2e/attachments", as: d, body: contents, now: now)
_ = try await h.routes.uploadAttachment(
request, body: pieces(contents, size: 64), length: contents.count, now: now)
let err = await #expect(throws: WireError.self) {
_ = try await h.routes.uploadAttachment(
request, body: pieces(contents, size: 64), length: contents.count, now: now)
}
#expect(code(err) == "proof_replayed")
#expect(try await Attachment.query(on: h.db).count() == 1)
let files = try FileManager.default.contentsOfDirectory(atPath: h.attachmentDir)
#expect(files.count == 1)
}
}
/// The production route over a real socket: `body: .stream`, the length
/// from the header, and the bytes on disk exactly as sent.
@Test("an upload over a real connection streams to disk, and a refused one stores nothing")
func uploadOverTheWire() async throws {
try await withRelay { h in
let d = try await device(h)
// Production's error shape; the harness does not run `configure`.
h.app.middleware.use(WireErrorMiddleware())
try register(routes: h.routes, on: h.app)
try await h.app.server.start(address: .hostname("127.0.0.1", port: 0))
let port = try #require(h.app.http.server.shared.localAddress?.port)
let contents = blob(3 * 1024 * 1024)
func post(_ headers: HTTPHeaders) async throws -> (HTTPResponseStatus, Data) {
var request = HTTPClientRequest(url: "http://127.0.0.1:\(port)/e2e/attachments")
request.method = .POST
request.headers = headers
request.body = .bytes(ByteBuffer(data: contents))
let response = try await HTTPClient.shared.execute(request, timeout: .seconds(30))
let body = try await response.body.collect(upTo: 1 << 16)
return (response.status, Data(buffer: body))
}
let (refused, _) = try await post([:])
#expect(refused == .unauthorized, "\(refused)")
let signed = h.request(
.POST, "/e2e/attachments", as: d, body: contents, now: unixNow())
let (status, body) = try await post(signed.headers)
await h.app.server.shutdown()
#expect(status == .ok, "\(String(decoding: body, as: UTF8.self))")
struct Made: Decodable { let id: String }
let id = try JSONDecoder().decode(Made.self, from: body).id
#expect(
FileManager.default.contents(atPath: attachmentPath(h.attachmentDir, id))
== contents)
#expect(try await Attachment.query(on: h.db).count() == 1)
#expect(
try FileManager.default.contentsOfDirectory(atPath: h.attachmentDir) == [
"\(id).bin"
])
}
}
// MARK: - Downloads
@Test("a range is asked of the bucket and comes back as a 206 slice")
func rangeThroughBucket() async throws {
try await withRelay { h in
let d = try await device(h)
let bucket = FakeBucket()
let r = routes(h, store: s3(bucket, fallback: h.attachmentDir))
let contents = blob(5000)
let up = try await r.uploadAttachment(
h.request(.POST, "/e2e/attachments", as: d, body: contents, now: now),
now: now)
struct Made: Decodable { let id: String }
let id = try JSONDecoder().decode(Made.self, from: Data(up.body.string!.utf8)).id
let request = h.request(.GET, "/e2e/attachments/\(id)", as: d, now: now)
request.headers.add(name: .range, value: "bytes=1000-1999")
let response = try await r.downloadAttachment(request, token: id, now: now)
#expect(response.status == .partialContent)
#expect(response.headers.first(name: .contentRange) == "bytes 1000-1999/5000")
#expect(response.body.count == 1000)
let body = try await response.body.collect(on: h.app.eventLoopGroup.next()).get()
#expect(body.map { Data(buffer: $0) } == contents.subdata(in: 1000..<2000))
let get = try #require(bucket.requests.last { $0.method == .GET })
#expect(get.headers.first(name: "range") == "bytes=1000-1999")
// Vapor runs a body's stream a second time when writing it to the
// client fails. A bucket body read twice traps the process.
_ = try? await response.body.collect(on: h.app.eventLoopGroup.next()).get()
}
}
/// The bucket's body is held at chunk 4 of 256 until the client has
/// received bytes. A relay that collected the object first would never
/// send any, and the test would time out.
///
/// The timeout is long because the full suite can stall the cooperative
/// thread pool for 30 seconds at a time (`blockingShutdown` waits on a
/// semaphore). Alone, this test takes under a second.
@Test("a download reaches the client while the bucket is still sending")
func downloadStreams() async throws {
try await withRelay { h in
let chunkSize = 64 * 1024
let chunks = 256 // 16 MB
let hold = 4
let (gate, open) = AsyncStream<Void>.makeStream()
let produced = Counter()
let token = randomHexToken()
let bucket = FakeBucket(onGet: { _ in
HTTPClientResponse(
status: .ok,
headers: ["content-length": "\(chunkSize * chunks)"],
body: .stream(
Held(
chunk: ByteBuffer(repeating: 0x5A, count: chunkSize),
count: chunks, holdAt: hold, gate: gate, produced: produced)))
})
let store = s3(bucket, fallback: h.attachmentDir)
h.app.get("blob") { req in
try await store.respond(token, request: req, range: nil)!
}
try await h.app.server.start(address: .hostname("127.0.0.1", port: 0))
let port = try #require(h.app.http.server.shared.localAddress?.port)
let outcome = try await withThrowingTaskGroup(of: (Int, Int)?.self) { group in
group.addTask {
let response = try await HTTPClient.shared.execute(
HTTPClientRequest(url: "http://127.0.0.1:\(port)/blob"),
timeout: .seconds(180))
var received = 0
var heldWhenFirstBytesArrived = -1
for try await piece in response.body {
if heldWhenFirstBytesArrived < 0 {
heldWhenFirstBytesArrived = produced.value
open.yield(())
}
received += piece.readableBytes
}
return (received, heldWhenFirstBytesArrived)
}
group.addTask {
try await Task.sleep(for: .seconds(120))
return nil
}
let first = try await group.next()!
group.cancelAll()
return first
}
await h.app.server.shutdown()
let (received, producedBeforeFirstByte) = try #require(
outcome, "no bytes reached the client while the bucket was held")
#expect(producedBeforeFirstByte <= hold)
#expect(received == chunkSize * chunks)
#expect(produced.value == chunks)
}
}
final class Counter: @unchecked Sendable {
private let lock = NSLock()
private var n = 0
var value: Int { lock.withLock { n } }
func increment() { lock.withLock { n += 1 } }
}
/// `count` copies of `chunk`, stopping before index `holdAt` until `gate`
/// yields.
struct Held: AsyncSequence, Sendable {
typealias Element = ByteBuffer
let chunk: ByteBuffer
let count: Int
let holdAt: Int
let gate: AsyncStream<Void>
let produced: Counter
struct AsyncIterator: AsyncIteratorProtocol {
let source: Held
var index = 0
mutating func next() async -> ByteBuffer? {
guard index < source.count else { return nil }
if index == source.holdAt {
var waiting = source.gate.makeAsyncIterator()
_ = await waiting.next()
}
index += 1
source.produced.increment()
return source.chunk
}
}
func makeAsyncIterator() -> AsyncIterator { AsyncIterator(source: self) }
}
}
Tests/FlamenetRelayTests/CapacityTests.swift
+12 -5
View File
@@ -169,7 +169,7 @@ struct CapacityTests {
endpoint: "https://example.invalid", bucket: "b",
region: "r", accessKey: "k", secretKey: "s",
prefix: "attachments/"),
client: h.app.client,
transport: HTTPClientS3Transport(client: h.app.http.client.shared),
localFallback: DirectoryAttachmentStore(directory: h.attachmentDir))
let check = await StatusChecks.storage(s3)
#expect(check.state != .answering)
@@ -243,6 +243,13 @@ extension EndToEndTests {
private let now = 1_700_000_000
/// A relay, with no other suite's held polls moving the shared gauge.
private func withGaugeRelay(_ body: (RelayHarness) async throws -> Void) async throws {
try await ProcessEnvironmentLock.withExclusiveHeldPolls {
try await withRelay(body)
}
}
private func device(_ h: RelayHarness) async throws -> RelayHarness.TestDevice {
let d = h.device(user: 1, id: "aaaa-0001")
try await h.register(d, on: h.db)
@@ -291,7 +298,7 @@ extension EndToEndTests {
/// concurrency stays flat.
@Test("a poll cancelled mid-hold gives its slot back")
func cancelledPollDoesNotLeakTheGauge() async throws {
try await withRelay { h in
try await withGaugeRelay { h in
await h.app.drain.reset()
RelayCapacity.shared.reset()
let alice = try await device(h)
@@ -323,7 +330,7 @@ extension EndToEndTests {
/// A poll that returns because mail arrived also has to balance.
@Test("a poll released by an arriving message gives its slot back")
func deliveredPollDoesNotLeakTheGauge() async throws {
try await withRelay { h in
try await withGaugeRelay { h in
await h.app.drain.reset()
RelayCapacity.shared.reset()
let alice = try await device(h)
@@ -356,7 +363,7 @@ extension EndToEndTests {
/// The exit path a trailing `leave()` at the end of the loop misses.
@Test("a poll that throws mid-hold gives its slot back")
func throwingPollDoesNotLeakTheGauge() async throws {
try await withRelay { h in
try await withGaugeRelay { h in
await h.app.drain.reset()
RelayCapacity.shared.reset()
let alice = try await device(h)
@@ -385,7 +392,7 @@ extension EndToEndTests {
/// Only the waiting is counted.
@Test("a poll that never waits is never counted as a held connection")
func instantPollIsNotCounted() async throws {
try await withRelay { h in
try await withGaugeRelay { h in
await h.app.drain.reset()
RelayCapacity.shared.reset()
let alice = try await device(h)
Tests/FlamenetRelayTests/ContactGraphRemovalTests.swift
+1 -1
View File
@@ -52,7 +52,7 @@ struct ContactGraphRemovalTests {
let c = AppCredential()
c.userID = id
c.selector = selector
c.secretHash = try Bcrypt.hash(secret)
c.secretHash = try Bcrypt.hash(secret, cost: 4)
c.label = "phone"
try await c.create(on: h.db)
return "\(selector).\(secret)"
Tests/FlamenetRelayTests/DeliveryOrderPostgresTests.swift
+11 -1
View File
@@ -121,11 +121,21 @@ struct DeliveryOrderPostgresTests {
let first = Task { try await send("held") }
// Wait until the first send is inside its transaction, waiting.
// pg_locks lists every database on the server, and the other
// Postgres suites running alongside this one wait on the key-log
// advisory lock in their own databases. Counting those let the
// second send start before the first had inserted (`free` got id
// 1), so only this database's wait on `holdKey` counts.
var waiting = false
for _ in 0..<200 where !waiting {
let n =
try await sql.raw(
"SELECT count(*) AS n FROM pg_locks WHERE locktype = 'advisory' AND NOT granted"
"""
SELECT count(*) AS n FROM pg_locks
WHERE locktype = 'advisory' AND NOT granted
AND database = (SELECT oid FROM pg_database WHERE datname = current_database())
AND classid = 0 AND objid = \(unsafeRaw: String(holdKey)) AND objsubid = 1
"""
).first()?.decode(column: "n", as: Int.self) ?? 0
waiting = n > 0
if !waiting { try await Task.sleep(nanoseconds: 50_000_000) }
Tests/FlamenetRelayTests/FakeBucket.swift
+88
View File
@@ -0,0 +1,88 @@
import AsyncHTTPClient
import Foundation
import NIOCore
import Vapor
@testable import FlamenetRelay
/// An S3 bucket in memory, behind the same transport seam production uses.
///
/// A PUT is kept only once its whole body has arrived: a body that throws part
/// way leaves nothing, which is what a real bucket does with a PUT whose
/// connection drops. `status` answers every request with that status instead,
/// and `failing` makes every request fail to connect.
final class FakeBucket: S3Transport, @unchecked Sendable {
struct Sent {
let method: HTTPMethod
let url: String
let headers: HTTPHeaders
}
struct Unreachable: Error {}
private let lock = NSLock()
private var _objects: [String: Data] = [:]
private var _requests: [Sent] = []
private let status: HTTPResponseStatus?
private let failing: Bool
/// Answers a GET in place of the stored object, when it returns non-nil.
let onGet: (@Sendable (String) -> HTTPClientResponse?)?
init(
status: HTTPResponseStatus? = nil, failing: Bool = false,
onGet: (@Sendable (String) -> HTTPClientResponse?)? = nil
) {
self.status = status
self.failing = failing
self.onGet = onGet
}
var requests: [Sent] { lock.withLock { _requests } }
var objects: [String: Data] { lock.withLock { _objects } }
func execute(_ request: HTTPClientRequest) async throws -> HTTPClientResponse {
lock.withLock {
_requests.append(
.init(method: request.method, url: request.url, headers: request.headers))
}
if failing { throw Unreachable() }
if let status { return HTTPClientResponse(status: status) }
let key = request.url
switch request.method {
case .PUT:
var data = Data()
if let body = request.body {
for try await chunk in body { data.append(contentsOf: chunk.readableBytesView) }
}
lock.withLock { _objects[key] = data }
return HTTPClientResponse(status: .ok)
case .GET:
if let answer = onGet?(key) { return answer }
guard let data = objects[key] else { return HTTPClientResponse(status: .notFound) }
guard let range = request.headers.first(name: "range"),
case .partial(let first, let last) = AttachmentRange.decide(
header: range, total: data.count)
else {
return HTTPClientResponse(
status: .ok, headers: ["content-length": "\(data.count)"],
body: .bytes(ByteBuffer(data: data)))
}
let slice = data.subdata(in: first..<(last + 1))
return HTTPClientResponse(
status: .partialContent,
headers: [
"content-length": "\(slice.count)",
"content-range": AttachmentRange.contentRange(
first: first, last: last, total: data.count),
],
body: .bytes(ByteBuffer(data: slice)))
case .HEAD:
return HTTPClientResponse(status: objects[key] == nil ? .notFound : .ok)
case .DELETE:
lock.withLock { _objects[key] = nil }
return HTTPClientResponse(status: .noContent)
default:
return HTTPClientResponse(status: .methodNotAllowed)
}
}
}
Tests/FlamenetRelayTests/FederationPeerTests.swift
+93
View File
@@ -371,3 +371,96 @@ struct FederationPeerTests {
}
}
}
// MARK: - Pinned federation keys
extension FederationPeerTests {
private func pinned(
_ pins: [String], publishing key: Curve25519.Signing.PrivateKey
) -> FederationVerifier {
var (v, _) = verifier(descriptorBody: descriptor(audience: "peer.example", key: key))
v.peerKeys = ["peer.example": pins]
return v
}
private func verifySigned(
by key: Curve25519.Signing.PrivateKey, with v: FederationVerifier
) async throws -> Federation.Peer {
let s = signed(key)
return try await v.verify(
method: "POST", path: "/e2e/federation/deposit", body: Data(),
origin: "peer.example", timestamp: s.timestamp, nonce: s.nonce,
signature: s.signature, now: now)
}
private func b64(_ key: Curve25519.Signing.PrivateKey) -> String {
key.publicKey.rawRepresentation.base64EncodedString()
}
@Test("a peer publishing its pinned key is accepted")
func pinnedKeyAccepted() async throws {
let key = Curve25519.Signing.PrivateKey()
let peer = try await verifySigned(by: key, with: pinned([b64(key)], publishing: key))
#expect(peer.audience == "peer.example")
}
/// The descriptor and the signature agree with each other, so only the
/// pin catches a peer whose descriptor was swapped.
@Test("a peer publishing a key other than its pin is refused")
func wrongKeyRefused() async throws {
let key = Curve25519.Signing.PrivateKey()
let other = Curve25519.Signing.PrivateKey()
let v = pinned([b64(other)], publishing: key)
var refusal: Federation.AuthError?
do { _ = try await verifySigned(by: key, with: v) } catch let e as Federation.AuthError {
refusal = e
}
guard case .keyNotPinned("peer.example") = refusal else {
Issue.record("expected keyNotPinned, got \(String(describing: refusal))")
return
}
#expect(refusal?.status == .unauthorized)
}
@Test("during a rotation both pinned keys are accepted")
func rotationAcceptsBoth() async throws {
let old = Curve25519.Signing.PrivateKey()
let new = Curve25519.Signing.PrivateKey()
for key in [old, new] {
let v = pinned([b64(old), b64(new)], publishing: key)
let peer = try await verifySigned(by: key, with: v)
#expect(peer.audience == "peer.example")
}
}
@Test("FN_FEDERATION_PEERS reads host=key1|key2 and keeps base64 padding")
func peersParse() {
let a = b64(Curve25519.Signing.PrivateKey())
let b = b64(Curve25519.Signing.PrivateKey())
#expect(a.hasSuffix("="))
let parsed = Config.parseFederationPeers(
" one.example , two.example=\(a)|\(b),three.example=\(a)")
#expect(parsed.hosts == ["one.example", "two.example", "three.example"])
#expect(parsed.keys["one.example"] == nil)
#expect(parsed.keys["two.example"] == [a, b])
#expect(parsed.keys["three.example"] == [a])
}
@Test("a bare host boots with a warning that it is unpinned")
func bareHostWarns() throws {
let key = b64(Curve25519.Signing.PrivateKey())
let config = try Config.fromEnvironment([
"FN_ISSUER_PUBLIC_KEY": key,
"FN_RELAY_AUDIENCE": "us.example",
"FN_FEDERATION": "allowlist",
"FN_FEDERATION_PEERS": "bare.example,pinned.example=\(key),bad.example=nope",
])
#expect(config.federationPeers == ["bare.example", "pinned.example", "bad.example"])
let warnings = config.federationPinWarnings()
#expect(warnings.count == 2)
#expect(warnings.contains { $0.contains("bare.example is unpinned") })
#expect(warnings.contains { $0.contains("bad.example has a pinned key that is not") })
#expect(!warnings.contains { $0.contains("pinned.example") })
}
}
Tests/FlamenetRelayTests/GroupFanOutTests.swift
+1 -1
View File
@@ -40,7 +40,7 @@ extension EndToEndTests {
let picture = Data(String(repeating: "p", count: 64 * 1024).utf8)
let upload = try await h.routes.uploadAttachment(
h.request(.POST, "/e2e/attachments", as: ada, body: picture, now: now),
body: picture, now: now)
now: now)
struct Up: Decodable { let id: String }
let token = try JSONDecoder().decode(
Up.self, from: Data(text(upload).utf8)
Tests/FlamenetRelayTests/MetricsTests.swift
+128
View File
@@ -0,0 +1,128 @@
import Crypto
import Foundation
import NIOCore
import NIOPosix
import Testing
import Vapor
import VaporTesting
@testable import FlamenetRelay
/// The Prometheus endpoint: on its own port, never the public one.
@Suite("metrics")
struct MetricsTests {
/// A port nothing is listening on, found by binding 0 and letting go.
private func freePort() async throws -> Int {
let channel = try await ServerBootstrap(group: MultiThreadedEventLoopGroup.singleton)
.bind(host: "127.0.0.1", port: 0).get()
let port = channel.localAddress?.port ?? 0
try await channel.close()
return port
}
private func withRelayApp(
metricsPort: Int, _ body: (Application) async throws -> Void
) async throws {
try await ProcessEnvironmentLock.withExclusiveEnvironment {
let issuer = Curve25519.Signing.PrivateKey()
let dbPath = NSTemporaryDirectory() + "fn-metrics-\(UUID().uuidString).sqlite"
let attachments = NSTemporaryDirectory() + "fn-metrics-att-\(UUID().uuidString)"
let env: [String: String] = [
"FN_ISSUER_PUBLIC_KEY": issuer.publicKey.rawRepresentation.base64EncodedString(),
"FN_RELAY_AUDIENCE": "test.relay",
"FN_DB": dbPath,
"FN_ATTACHMENTS_DIR": attachments,
"FN_LOG_LEVEL": "critical",
"FN_METRICS_PORT": String(metricsPort),
]
let previous = env.keys.reduce(into: [String: String?]()) {
$0[$1] = ProcessInfo.processInfo.environment[$1]
}
for (k, v) in env { setenv(k, v, 1) }
defer {
for (k, v) in previous { if let v { setenv(k, v, 1) } else { unsetenv(k) } }
try? FileManager.default.removeItem(atPath: dbPath)
try? FileManager.default.removeItem(atPath: attachments)
}
let app = try await Application.make(.testing)
do {
try await configure(app)
// What `serve` does; the metrics listener starts here.
try await app.asyncBoot()
try await body(app)
} catch {
try? await app.asyncShutdown()
throw error
}
try? await app.asyncShutdown()
}
}
@Test("the counters render in the Prometheus text format")
func renders() {
RelayMetrics.shared.add(.envelopesWritten, 2)
RelayMetrics.shared.add(.federationDepositsIn)
let text = RelayMetrics.shared.render(heldPolls: 7, pushDropped: 3)
#expect(text.contains("# TYPE flamenet_held_polls gauge\nflamenet_held_polls 7\n"))
#expect(text.contains("flamenet_push_dropped_total 3\n"))
for name in [
"flamenet_push_sent_total", "flamenet_push_failed_total",
"flamenet_limiter_fail_open_total", "flamenet_envelopes_written_total",
"flamenet_federation_deposits_in_total", "flamenet_federation_deposits_out_total",
] {
#expect(text.contains("# TYPE \(name) counter\n\(name) "), "missing \(name)")
}
#expect(RelayMetrics.shared.value(.envelopesWritten) >= 2)
}
@Test("/metrics is absent on the public port and served on the metrics port")
func separatePort() async throws {
let port = try await freePort()
try await withRelayApp(metricsPort: port) { app in
try await app.testing().test(.GET, "/metrics") { res in
#expect(res.status == .notFound)
#expect(!res.body.string.contains("flamenet_held_polls"))
}
let res = try await app.client.get(URI(string: "http://127.0.0.1:\(port)/metrics"))
#expect(res.status == .ok)
#expect(res.headers.first(name: .contentType)?.hasPrefix("text/plain") == true)
let text = res.body.map { String(buffer: $0) } ?? ""
for name in [
"flamenet_held_polls", "flamenet_push_sent_total", "flamenet_push_failed_total",
"flamenet_push_dropped_total", "flamenet_limiter_fail_open_total",
"flamenet_envelopes_written_total", "flamenet_federation_deposits_in_total",
] {
#expect(text.contains("\n\(name) "), "missing \(name)")
}
// Only /metrics, on that port too.
let other = try await app.client.get(URI(string: "http://127.0.0.1:\(port)/health"))
#expect(other.status == .notFound)
}
}
@Test("metrics are off unless FN_METRICS_PORT is set, and bind loopback by default")
func offByDefault() throws {
func config(_ extra: [String: String]) throws -> Config {
var env = [
"FN_ISSUER_PUBLIC_KEY": Curve25519.Signing.PrivateKey()
.publicKey.rawRepresentation.base64EncodedString(),
"FN_RELAY_AUDIENCE": "test.relay",
]
for (k, v) in extra { env[k] = v }
return try Config.fromEnvironment(env)
}
#expect(try config([:]).metricsPort == 0)
#expect(try config([:]).metricsHost == "127.0.0.1")
#expect(try config(["FN_METRICS_PORT": "9464"]).metricsPort == 9464)
}
@Test("a metrics port equal to the public port refuses to boot")
func samePortRefused() async throws {
await #expect(throws: (any Error).self) {
try await withRelayApp(metricsPort: 8080) { _ in }
}
}
}
Tests/FlamenetRelayTests/MobileExtrasTests.swift
+1 -1
View File
@@ -37,7 +37,7 @@ struct MobileExtrasTests {
let c = AppCredential()
c.userID = id
c.selector = selector
c.secretHash = try Bcrypt.hash(secret)
c.secretHash = try Bcrypt.hash(secret, cost: 4)
c.label = "phone"
try await c.create(on: h.db)
return "\(selector).\(secret)"
Tests/FlamenetRelayTests/MobileRoutesTests.swift
+1 -1
View File
@@ -41,7 +41,7 @@ struct MobileRoutesTests {
let c = AppCredential()
c.userID = id
c.selector = selector
c.secretHash = try Bcrypt.hash(secret)
c.secretHash = try Bcrypt.hash(secret, cost: 4)
c.label = "phone"
try await c.create(on: h.db)
return "\(selector).\(secret)"
Tests/FlamenetRelayTests/PairsStageTwoATests.swift
+1 -1
View File
@@ -167,7 +167,7 @@ struct PairsStageTwoATests {
let c = AppCredential()
c.userID = id
c.selector = selector
c.secretHash = try Bcrypt.hash(secret)
c.secretHash = try Bcrypt.hash(secret, cost: 4)
c.label = "phone"
try await c.create(on: h.db)
return "\(selector).\(secret)"
Tests/FlamenetRelayTests/PasswordResetTests.swift
+1 -1
View File
@@ -30,7 +30,7 @@ struct PasswordResetTests {
let u = User()
u.username = username
u.email = email
u.passwordHash = try Bcrypt.hash("originalpassword")
u.passwordHash = try Bcrypt.hash("originalpassword", cost: 4)
u.displayName = username
u.enclavePseudonym = enclave
try await u.create(on: h.db)
Tests/FlamenetRelayTests/PerDeviceDigestTests.swift
+1 -1
View File
@@ -76,7 +76,7 @@ struct PerDeviceDigestTests {
let c = AppCredential()
c.userID = id
c.selector = selector
c.secretHash = try Bcrypt.hash(secret)
c.secretHash = try Bcrypt.hash(secret, cost: 4)
c.label = "phone"
try await c.create(on: h.db)
return "\(selector).\(secret)"
Tests/FlamenetRelayTests/ProcessEnvironmentLock.swift
+21 -3
View File
@@ -52,13 +52,31 @@ actor ProcessEnvironmentLock {
static func withExclusiveEnvironment<T>(
_ body: () async throws -> T
) async rethrows -> T {
await shared.acquire()
try await exclusive(shared, body)
}
/// `RelayCapacity.shared` is process-global as well. A poll held by one
/// suite moves the gauge that `CapacityTests` asserts exact values on, so
/// suites that hold polls by the dozen take this lock, and so do those
/// assertions.
static let heldPolls = ProcessEnvironmentLock()
static func withExclusiveHeldPolls<T>(
_ body: () async throws -> T
) async rethrows -> T {
try await exclusive(heldPolls, body)
}
private static func exclusive<T>(
_ lock: ProcessEnvironmentLock, _ body: () async throws -> T
) async rethrows -> T {
await lock.acquire()
do {
let result = try await body()
await shared.release()
await lock.release()
return result
} catch {
await shared.release()
await lock.release()
throw error
}
}
Tests/FlamenetRelayTests/PushAndS3Tests.swift
+22 -31
View File
@@ -146,13 +146,13 @@ struct PushAndS3Tests {
// MARK: - Object storage
private func s3(_ client: RecordingClient, fallback: String) -> S3AttachmentStore {
private func s3(_ bucket: FakeBucket, fallback: String) -> S3AttachmentStore {
S3AttachmentStore(
s3: .init(
endpoint: "https://s3.example.com", bucket: "blobs",
region: "us-west-004", accessKey: "AKIA", secretKey: "secret",
prefix: "attachments/"),
client: client,
transport: bucket,
localFallback: DirectoryAttachmentStore(directory: fallback))
}
@@ -163,16 +163,15 @@ struct PushAndS3Tests {
@Test("object storage reports reachable when the bucket answers as a bucket")
func s3ReachableOn404() async throws {
try await withRelay { h in
let loop = EmbeddedEventLoop()
for status in [HTTPStatus.notFound, .ok, .noContent] {
let client = RecordingClient(eventLoop: loop, status: status)
let state = await s3(client, fallback: h.attachmentDir).reachability()
let bucket = FakeBucket(status: status)
let state = await s3(bucket, fallback: h.attachmentDir).reachability()
guard case .reachable = state else {
Issue.record("\(status.code) should read as reachable, got \(state)")
continue
}
// Signed, and a HEAD: the probe must not read or store a thing.
let req = try #require(client.requests.first)
let req = try #require(bucket.requests.first)
#expect(req.method == .HEAD)
#expect(req.headers.first(name: "Authorization")?.contains("AWS4-HMAC-SHA256") == true)
}
@@ -184,10 +183,9 @@ struct PushAndS3Tests {
@Test("a bucket that refuses or does not answer reads as down")
func s3UnreachableOnRefusal() async throws {
try await withRelay { h in
let loop = EmbeddedEventLoop()
for status in [HTTPStatus.forbidden, .unauthorized, .internalServerError] {
let state = await s3(
RecordingClient(eventLoop: loop, status: status),
FakeBucket(status: status),
fallback: h.attachmentDir
).reachability()
guard case .unreachable = state else {
@@ -196,7 +194,7 @@ struct PushAndS3Tests {
}
}
let dead = await s3(
RecordingClient(eventLoop: loop, failing: true),
FakeBucket(failing: true),
fallback: h.attachmentDir
).reachability()
guard case .unreachable = dead else {
@@ -209,15 +207,14 @@ struct PushAndS3Tests {
@Test("a write is a signed PUT to the bucket path")
func s3Write() async throws {
try await withRelay { h in
let loop = EmbeddedEventLoop()
let client = RecordingClient(eventLoop: loop)
let bucket = FakeBucket()
let token = randomHexToken()
try await s3(client, fallback: h.attachmentDir)
try await s3(bucket, fallback: h.attachmentDir)
.write(token, data: Data(repeating: 1, count: 64))
let req = try #require(client.requests.first)
let req = try #require(bucket.requests.first)
#expect(req.method == .PUT)
#expect(req.url.string == "https://s3.example.com/blobs/attachments/\(token).bin")
#expect(req.url == "https://s3.example.com/blobs/attachments/\(token).bin")
// SigV4: the request must actually carry a signature and a payload
// hash, or the bucket answers 403 and blobs silently never store.
#expect(req.headers.first(name: "Authorization")?.contains("AWS4-HMAC-SHA256") == true)
@@ -230,10 +227,9 @@ struct PushAndS3Tests {
@Test("a refused write is an error, not a silent success")
func s3WriteFailure() async throws {
try await withRelay { h in
let loop = EmbeddedEventLoop()
let client = RecordingClient(eventLoop: loop, status: .forbidden)
let bucket = FakeBucket(status: .forbidden)
await #expect(throws: (any Error).self) {
try await s3(client, fallback: h.attachmentDir)
try await s3(bucket, fallback: h.attachmentDir)
.write(randomHexToken(), data: Data(repeating: 1, count: 8))
}
}
@@ -244,18 +240,17 @@ struct PushAndS3Tests {
@Test("a blob missing from the bucket falls back to local disk")
func s3FallsBackToDisk() async throws {
try await withRelay { h in
let loop = EmbeddedEventLoop()
let token = randomHexToken()
// The pre-switch blob, still on disk.
try await DirectoryAttachmentStore(directory: h.attachmentDir)
.write(token, data: Data(repeating: 7, count: 32))
let client = RecordingClient(eventLoop: loop, status: .notFound)
let bucket = FakeBucket(status: .notFound)
let req = Request(
application: h.app, method: .GET,
url: URI(path: "/e2e/attachments/\(token)"),
on: h.app.eventLoopGroup.next())
let response = try await s3(client, fallback: h.attachmentDir)
let response = try await s3(bucket, fallback: h.attachmentDir)
.respond(token, request: req, range: nil)
#expect(response?.status == .ok)
}
@@ -264,14 +259,13 @@ struct PushAndS3Tests {
@Test("a bucket error on read is surfaced rather than treated as absent")
func s3ReadFailure() async throws {
try await withRelay { h in
let loop = EmbeddedEventLoop()
let client = RecordingClient(eventLoop: loop, status: .internalServerError)
let bucket = FakeBucket(status: .internalServerError)
let req = Request(
application: h.app, method: .GET,
url: URI(path: "/e2e/attachments/x"),
on: h.app.eventLoopGroup.next())
await #expect(throws: (any Error).self) {
_ = try await s3(client, fallback: h.attachmentDir)
_ = try await s3(bucket, fallback: h.attachmentDir)
.respond(randomHexToken(), request: req, range: nil)
}
}
@@ -283,10 +277,9 @@ struct PushAndS3Tests {
@Test("delete accepts both deleted and already-gone")
func s3DeleteAcceptsBoth() async throws {
try await withRelay { h in
let loop = EmbeddedEventLoop()
for status in [HTTPStatus.noContent, .ok, .notFound] {
let client = RecordingClient(eventLoop: loop, status: status)
try await s3(client, fallback: h.attachmentDir).delete(randomHexToken())
let bucket = FakeBucket(status: status)
try await s3(bucket, fallback: h.attachmentDir).delete(randomHexToken())
}
}
}
@@ -294,10 +287,9 @@ struct PushAndS3Tests {
@Test("a bucket error on delete is surfaced")
func s3DeleteFailure() async throws {
try await withRelay { h in
let loop = EmbeddedEventLoop()
let client = RecordingClient(eventLoop: loop, status: .forbidden)
let bucket = FakeBucket(status: .forbidden)
await #expect(throws: (any Error).self) {
try await s3(client, fallback: h.attachmentDir).delete(randomHexToken())
try await s3(bucket, fallback: h.attachmentDir).delete(randomHexToken())
}
}
}
@@ -308,7 +300,6 @@ struct PushAndS3Tests {
@Test("deleting removes the local fallback copy as well")
func s3DeleteClearsDisk() async throws {
try await withRelay { h in
let loop = EmbeddedEventLoop()
let token = randomHexToken()
try await DirectoryAttachmentStore(directory: h.attachmentDir)
.write(token, data: Data(repeating: 3, count: 16))
@@ -317,7 +308,7 @@ struct PushAndS3Tests {
atPath: attachmentPath(h.attachmentDir, token)))
try await s3(
RecordingClient(eventLoop: loop, status: .noContent),
FakeBucket(status: .noContent),
fallback: h.attachmentDir
).delete(token)
#expect(
Tests/FlamenetRelayTests/ReportDigestTests.swift
+153
View File
@@ -0,0 +1,153 @@
import Crypto
import Fluent
import Foundation
import NIOConcurrencyHelpers
import Testing
import Vapor
@testable import FlamenetRelay
/// The per-reporter limit, and the hourly digest that replaced one mail per
/// report.
@Suite("report digest")
struct ReportDigestTests {
private let now = 1_700_000_000
/// Every message the digest would have sent.
private final class Outbox: @unchecked Sendable {
private let box = NIOLockedValueBox<[(to: String, subject: String, body: String)]>([])
var failNext = false
var sent: [(to: String, subject: String, body: String)] { box.withLockedValue { $0 } }
func send(_ to: String, _ subject: String, _ body: String) throws {
if failNext {
failNext = false
throw Abort(.serviceUnavailable)
}
box.withLockedValue { $0.append((to, subject, body)) }
}
}
private func digest(_ h: RelayHarness, _ outbox: Outbox) -> ReportDigest {
ReportDigest(
db: h.db, to: "ops@example.org",
send: { try outbox.send($0, $1, $2) },
logger: h.app.logger)
}
@discardableResult
private func fileRow(_ h: RelayHarness, about: Int, note: String) async throws -> Int {
let r = ContentReport()
r.reporterID = 1
r.aboutID = about
r.kind = "message"
r.reason = "spam"
r.content = "copy \(about)"
r.note = note
r.createdAt = now
try await r.save(on: h.db)
return r.id!
}
@Test("one reporter over the hourly limit gets a 429")
func reportLimit() async throws {
try await withRelay(env: ["FN_REPORT_RATE": "2"]) { h in
let d = h.device(user: 1, id: "aaaa-0001")
_ = try await h.register(d, on: h.db)
func file() async throws -> Response {
let body = Data(#"{"about":2,"kind":"story","reason":"spam"}"#.utf8)
let req = h.request(.POST, "/e2e/moderation/report", as: d, body: body, now: now)
return try await h.routes.submitReport(req, now: now)
}
#expect(try await file().status == .ok)
#expect(try await file().status == .ok)
var status = 0
do { _ = try await file() } catch let e as WireError { status = e.data.status }
#expect(status == 429)
// The refused one was not stored.
#expect(try await ContentReport.query(on: h.db).count() == 2)
}
}
@Test("the default limit is 10 an hour")
func reportLimitDefault() throws {
let config = try Config.fromEnvironment([
"FN_ISSUER_PUBLIC_KEY": Curve25519.Signing.PrivateKey()
.publicKey.rawRepresentation.base64EncodedString(),
"FN_RELAY_AUDIENCE": "test.relay",
])
#expect(config.reportRatePerHour == 10)
}
@Test("N reports go out as one email, and an empty hour sends none")
func batchesIntoOne() async throws {
try await withRelay { h in
let outbox = Outbox()
let digest = digest(h, outbox)
// Nothing filed: nothing sent.
#expect(try await digest.sendIfDue(now: now) == 0)
#expect(outbox.sent.isEmpty)
var ids: [Int] = []
for about in 2...4 { ids.append(try await fileRow(h, about: about, note: "n\(about)")) }
#expect(try await digest.sendIfDue(now: now) == 3)
#expect(outbox.sent.count == 1)
let mail = try #require(outbox.sent.first)
#expect(mail.to == "ops@example.org")
#expect(mail.subject == "Flamenet reports: 3 new")
for (id, about) in zip(ids, 2...4) {
#expect(mail.body.contains("id \(id)"))
#expect(mail.body.contains("about user \(about)"))
#expect(mail.body.contains("n\(about)"))
}
// The same reports are not mailed twice, and an hour with nothing
// new sends nothing.
#expect(try await digest.sendIfDue(now: now + 4000) == 0)
#expect(outbox.sent.count == 1)
// A report inside the hour waits for the hour.
try await fileRow(h, about: 5, note: "later")
#expect(try await digest.sendIfDue(now: now + 600) == 0)
#expect(outbox.sent.count == 1)
#expect(try await digest.sendIfDue(now: now + 3600) == 1)
#expect(outbox.sent.count == 2)
#expect(outbox.sent[1].body.contains("later"))
#expect(!outbox.sent[1].body.contains("n2"))
// The stored reports are untouched.
#expect(try await ContentReport.query(on: h.db).count() == 4)
}
}
@Test("a digest that could not be mailed is retried with the same reports")
func failedSendRetries() async throws {
try await withRelay { h in
let outbox = Outbox()
let digest = digest(h, outbox)
try await fileRow(h, about: 2, note: "first")
outbox.failNext = true
await #expect(throws: (any Error).self) { try await digest.sendIfDue(now: now) }
#expect(outbox.sent.isEmpty)
#expect(try await digest.sendIfDue(now: now + 300) == 1)
#expect(outbox.sent.first?.body.contains("first") == true)
}
}
/// The migration seeds the watermark with the newest report, so a deploy
/// does not mail every report ever filed again.
@Test("reports filed before the digest existed are not mailed again")
func seededAtTheNewestReport() async throws {
try await withRelay { h in
try await fileRow(h, about: 2, note: "old")
// What the migration does on a database that already has reports.
try await CreateReportDigests().revert(on: h.db)
try await CreateReportDigests().prepare(on: h.db)
let outbox = Outbox()
#expect(try await digest(h, outbox).sendIfDue(now: now) == 0)
#expect(outbox.sent.isEmpty)
}
}
}
Tests/FlamenetRelayTests/RosterTagTests.swift
+1 -1
View File
@@ -48,7 +48,7 @@ struct RosterTagTests {
let c = AppCredential()
c.userID = id
c.selector = selector
c.secretHash = try Bcrypt.hash(secret)
c.secretHash = try Bcrypt.hash(secret, cost: 4)
c.label = "phone"
try await c.create(on: h.db)
return "\(selector).\(secret)"
Tests/FlamenetRelayTests/ScanAndScaleTests.swift
+446
View File
@@ -0,0 +1,446 @@
import Crypto
import Fluent
import FluentPostgresDriver
import Foundation
import Logging
import SQLKit
import Testing
import Vapor
@testable import FlamenetRelay
/// Relay review 2026-10-14: presence in bounded queries, an O(1) log head,
/// set-based retention, and last-seen writes held in memory.
@Suite("scans and scale")
struct ScanAndScaleTests {
// MARK: - Presence
private func user(_ h: RelayHarness, id: Int, status: ChosenStatus, fresh: Bool) async throws {
let u = User()
u.id = id
u.username = "u\(id)"
u.passwordHash = "x"
u.displayName = "u\(id)"
u.chosenStatus = status.rawValue
u.lastSeenAt = fresh ? Presence.coarsen(Date()) : Date(timeIntervalSince1970: 0)
try await u.create(on: h.db)
}
private func sha(_ token: Data) -> String {
Data(SHA256.hash(data: token)).base64EncodedString()
}
/// The answer one token got before batching: device set, then account
/// set, then the account row.
private func oneAtATime(_ raw: String, on db: any Database) async throws -> String {
guard let token = MobileRoutes.validToken(raw) else { return "offline" }
let digest = sha(token)
let subject: Int?
if let tag = try await DeviceRosterTag.query(on: db)
.filter(\.$digest == digest).sort(\.$id).first()
{
subject = tag.userID
} else {
subject = try await RosterTag.query(on: db)
.filter(\.$digest == digest).sort(\.$id).first()?.userID
}
guard let subject, let u = try await User.find(subject, on: db),
let chosen = ChosenStatus(rawValue: u.chosenStatus), chosen != .invisible,
Presence.isFresh(u.lastSeenAt)
else { return "offline" }
return chosen.effective.rawValue
}
@Test("presence for 500 tokens takes at most three queries and answers as before")
func presenceIsBatched() async throws {
try await withRelay { h in
let statuses: [ChosenStatus] = [.online, .away, .busy, .invisible]
for id in 1...40 {
try await user(h, id: id, status: statuses[id % 4], fresh: id % 5 != 0)
}
var tokens: [String] = []
for i in 0..<500 {
let token = Data((0..<32).map { _ in UInt8.random(in: 0...255) })
let raw = token.base64EncodedString()
switch i {
case 0..<200:
let tag = DeviceRosterTag()
tag.userID = i % 40 + 1
tag.deviceID = "aaaa-0001"
tag.digest = sha(token)
try await tag.create(on: h.db)
// The same digest on an account set too: the device row wins.
if i < 10 {
let shadow = RosterTag()
shadow.userID = (i + 1) % 40 + 1
shadow.digest = sha(token)
try await shadow.create(on: h.db)
}
tokens.append(raw)
case 200..<400:
let tag = RosterTag()
tag.userID = i % 40 + 1
tag.digest = sha(token)
try await tag.create(on: h.db)
tokens.append(raw)
case 400..<480:
tokens.append(raw) // matches nothing
default:
tokens.append("not-a-token-\(i)")
}
}
h.app.fluent.history.start()
let answer = try await MobileRoutes.presence(for: tokens, viewer: 0, on: h.app.db)
let queries = h.app.fluent.history.queries.count
h.app.fluent.history.stop()
h.app.fluent.history.clear()
#expect(queries <= 3, "\(queries) queries for 500 tokens")
#expect(answer.presence.map(\.token) == tokens)
var expected: [String] = []
for raw in tokens { expected.append(try await oneAtATime(raw, on: h.db)) }
#expect(answer.presence.map(\.presence) == expected)
#expect(Set(expected).count > 1, "the seed should give more than one answer")
}
}
// MARK: - Key log head
private func append(_ db: any Database, _ range: Range<Int>) async throws {
for i in range {
try await KeyLog.append(
userID: i + 1, subject: "s\(i)", deviceID: String(format: "aaaa-%04x", i),
ikSig: "sig\(i)", ikDH: "dh\(i)", kind: "publish", on: db)
}
}
@Test("the head is served from memory and matches a full recompute after appends")
func headIsCached() async throws {
try await withRelay { h in
let cache = KeyLogHeadCache()
let unsigned: @Sendable (Int, Data) -> Data? = { _, _ in nil }
let empty = try await cache.current(on: h.db, sign: unsigned)
#expect(empty.size == 0 && empty.root == MerkleLog.root([]))
try await append(h.db, 0..<5)
var head = try await cache.current(on: h.db, sign: unsigned)
var leaves = try await KeyLog.leaves(on: h.db)
#expect(head.size == 5 && head.root == MerkleLog.root(leaves))
// Unchanged log: one query, an aggregate, and no leaf read.
h.app.fluent.history.start()
let again = try await cache.current(on: h.app.db, sign: unsigned)
let seen = h.app.fluent.history.queries
h.app.fluent.history.stop()
h.app.fluent.history.clear()
#expect(again == head)
#expect(seen.count == 1)
#expect(
seen.allSatisfy {
if case .aggregate = $0.action { return true } else { return false }
}, "\(seen.map(\.action))")
// Appended: only the new leaves are read, and the root agrees.
try await append(h.db, 5..<12)
head = try await cache.current(on: h.db, sign: unsigned)
leaves = try await KeyLog.leaves(on: h.db)
#expect(head.size == 12 && head.root == MerkleLog.root(leaves))
let fresh = try await KeyLogHeadCache().current(on: h.db, sign: unsigned)
#expect(fresh == head)
}
}
@Test("the head route signs the cached head and is metered per address")
func headRouteLimit() async throws {
let key = Curve25519.Signing.PrivateKey()
let limits = RateLimiter()
try await withRelay(
env: ["FN_ISSUER_PRIVATE_KEY": key.rawRepresentation.base64EncodedString()],
limits: limits
) { h in
try await append(h.db, 0..<3)
func get() -> Request {
Request(
application: h.app, method: .GET, url: URI(string: "/e2e/log/head"),
on: h.app.eventLoopGroup.next())
}
struct Head: Decodable {
let size: Int
let root: String
let signature: String
}
let res = try await h.routes.logHead(get())
let head = try JSONDecoder().decode(
Head.self, from: Data(buffer: res.body.buffer ?? ByteBuffer()))
let root = MerkleLog.root(try await KeyLog.leaves(on: h.db)).base64EncodedString()
#expect(head.size == 3 && head.root == root)
#expect(
key.publicKey.isValidSignature(
try #require(Data(base64Encoded: head.signature)),
for: Data("fnkt-v1|3|\(root)".utf8)))
// Spend the rest of this address's hour, then one more.
let meter = "log:head:ip:\(clientIP(get()))"
_ = await limits.consume(
meter, amount: Routes.logHeadPerHour - 1,
limit: Routes.logHeadPerHour, now: unixNow())
do {
_ = try await h.routes.logHead(get())
Issue.record("a head read past the hourly limit was served")
} catch let e as WireError {
#expect(e.httpStatus == .tooManyRequests)
}
}
}
// MARK: - Last seen
@Test("last-seen is written once per heartbeat and suspension still refuses at once")
func lastSeenWrites() async throws {
try await withRelay { h in
try await user(h, id: 3, status: .online, fresh: false)
let d = h.device(user: 3, id: "aaaa-0003")
try await h.register(d, on: h.db)
let now = 1_700_000_000
func poll() async throws -> Response {
try await h.routes.poll(
h.request(
.GET, "/e2e/messages?wait=0", as: d, now: now,
signingPath: "/e2e/messages"),
now: now)
}
let beat = Presence.coarsen(Date())
#expect(try await poll().status == .ok)
let seen = try await User.find(3, on: h.db)?.lastSeenAt
#expect(seen != nil && seen! >= beat, "the first request writes the heartbeat")
// Cleared behind the relay's back: a second request in the same
// heartbeat does not read or write it.
try await User.query(on: h.db).filter(\.$id == 3).set(\.$lastSeenAt, to: nil).update()
#expect(try await poll().status == .ok)
#expect(try await User.find(3, on: h.db)?.lastSeenAt == nil)
// Suspension is read on every request.
try await User.query(on: h.db).filter(\.$id == 3).set(\.$suspendedAt, to: Date())
.update()
do {
_ = try await poll()
Issue.record("a suspended account polled")
} catch {
#expect((error as? WireError)?.code == "account_suspended", "\(error)")
}
}
}
@Test("a later heartbeat is written once the gap has passed")
func lastSeenAdvances() async {
let writes = LastSeenWrites()
let t0 = Date(timeIntervalSince1970: 1_700_000_000)
let b0 = Presence.coarsen(t0)
let b1 = b0.addingTimeInterval(Presence.onlineWindow)
#expect(await writes.claim(1, beat: b0, now: t0))
#expect(await !writes.claim(1, beat: b0, now: t0 + 120), "same heartbeat")
#expect(await !writes.claim(1, beat: b1, now: t0 + 30), "inside the gap")
#expect(await writes.claim(1, beat: b1, now: t0 + 61))
#expect(await writes.claim(2, beat: b1, now: t0 + 61), "per account")
await writes.release(2)
#expect(await writes.claim(2, beat: b1, now: t0 + 62), "a failed write is retried")
}
// MARK: - Retention
private struct Snapshot: Equatable {
var devices: Set<String> = []
var pushTokens: Set<String> = []
var prekeys: Set<Int> = []
var expired: Set<String> = []
}
private func snapshot(_ db: any Database) async throws -> Snapshot {
Snapshot(
devices: Set(try await Device.query(on: db).all().compactMap(\.id)),
pushTokens: Set(try await PushToken.query(on: db).all().compactMap(\.id)),
prekeys: Set(try await PreKey.query(on: db).all().compactMap(\.id)),
expired: Set(
try await LogEntry.query(on: db).filter(\.$kind == "expire").all().map(\.deviceID)))
}
/// What the sweep removed before it was set-based, worked out from whole
/// tables the way it used to.
private func expectedAfterSweep(_ db: any Database, expiryDays: Int) async throws -> Snapshot {
var s = try await snapshot(db)
let all = try await Device.query(on: db).all()
let deviceCutoff = Date().addingTimeInterval(-Double(expiryDays) * 86_400)
var newestPerUser: [Int: Date] = [:]
for d in all {
newestPerUser[d.userID] = max(
newestPerUser[d.userID] ?? .distantPast, DeviceRetirement.activity(d))
}
for d in all {
let seen = DeviceRetirement.activity(d)
guard let id = d.id, seen < deviceCutoff,
seen < newestPerUser[d.userID] ?? .distantPast
else { continue }
s.devices.remove(id)
s.expired.insert(id)
}
s.pushTokens.formIntersection(s.devices)
let prekeyCutoff = Date().addingTimeInterval(-7 * 86_400)
for p in try await PreKey.query(on: db).all() {
guard let id = p.id else { continue }
if (p.used && (p.createdAt ?? Date()) < prekeyCutoff) || !s.devices.contains(p.deviceID)
{
s.prekeys.remove(id)
}
}
return s
}
private func seedRetention(_ h: RelayHarness) async throws {
let db = h.db
let ago = { (days: Double) in Date().addingTimeInterval(-days * 86_400) }
let tie = ago(200)
// (user, device, last seen, registered)
let rows: [(Int, String, Date?, Date?)] = [
(1, "aaaa-0001", ago(200), nil), (1, "aaaa-0002", ago(1), nil),
(2, "aaaa-0003", ago(500), nil),
(3, "aaaa-0004", nil, nil), (3, "aaaa-0005", ago(200), nil),
(4, "aaaa-0006", tie, nil), (4, "aaaa-0007", tie, nil),
(5, "aaaa-0008", nil, ago(300)), (5, "aaaa-0009", ago(100), nil),
(6, "aaaa-000a", nil, ago(300)), (6, "aaaa-000b", nil, ago(20)),
]
for (user, id, seen, registered) in rows {
try await h.register(h.device(user: user, id: id), on: db)
let d = try #require(try await Device.find(id, on: db))
d.lastSeen = seen
if let registered { d.createdAt = registered }
try await d.save(on: db)
}
for (id, user) in [
("aaaa-0001", 1), ("aaaa-0002", 1), ("aaaa-0003", 2), ("dead-0001", 9),
] {
let t = PushToken()
t.id = id
t.userID = user
t.platform = "apns"
t.token = "t-\(id)"
try await t.create(on: db)
}
for (n, (device, used, days)) in [
("aaaa-0002", true, 10.0), ("aaaa-0002", true, 1), ("aaaa-0002", false, 10),
("aaaa-0003", true, 30), ("aaaa-0003", false, 30), ("dead-0002", false, 1),
].enumerated() {
let p = PreKey()
p.deviceID = device
p.preKeyID = n
p.pub = "pub\(n)"
p.used = used
try await p.create(on: db)
p.createdAt = ago(days)
try await p.update(on: db)
}
}
private func checkRetention(_ h: RelayHarness) async throws {
try await seedRetention(h)
let expected = try await expectedAfterSweep(h.db, expiryDays: 90)
try await Retention(
db: h.db, config: h.routes.config, logger: Logger(label: "test"),
attachmentStore: h.routes.attachmentStore,
sendMetrics: SendMetrics(), redis: nil
).sweepOnce()
let after = try await snapshot(h.db)
#expect(after == expected)
// The seed has to exercise each rule, or matching proves nothing.
#expect(after.expired == ["aaaa-0001", "aaaa-0005", "aaaa-0008", "aaaa-000a"])
#expect(after.pushTokens == ["aaaa-0002", "aaaa-0003"])
#expect(after.prekeys.count == 3)
}
@Test("set-based retention deletes the rows the whole-table sweep did, on SQLite")
func retentionMatchesOnSQLite() async throws {
try await withRelay(env: ["FN_DEVICE_EXPIRY_DAYS": "90"]) { h in
try await checkRetention(h)
}
}
@Test(
"set-based retention deletes the rows the whole-table sweep did, on Postgres",
.enabled(if: ProcessInfo.processInfo.environment["FN_TEST_POSTGRES"]?.isEmpty == false))
func retentionMatchesOnPostgres() async throws {
try await withPostgresRelay(env: ["FN_DEVICE_EXPIRY_DAYS": "90"]) { h in
try await checkRetention(h)
}
}
@Test(
"the scan indexes are built and valid on Postgres",
.enabled(if: ProcessInfo.processInfo.environment["FN_TEST_POSTGRES"]?.isEmpty == false))
func indexesOnPostgres() async throws {
try await withPostgresRelay { h in
let sql = try #require(h.db as? any SQLDatabase)
struct Row: Decodable {
let relname: String
let indisvalid: Bool
}
let rows = try await sql.raw(
"""
SELECT c.relname, i.indisvalid FROM pg_index i
JOIN pg_class c ON c.oid = i.indexrelid
WHERE c.relname IN ('roster_tags_digest', 'prekeys_used_created',
'device_roster_tags_digest')
ORDER BY c.relname
"""
).all(decoding: Row.self)
#expect(
rows.map(\.relname) == [
"device_roster_tags_digest", "prekeys_used_created", "roster_tags_digest",
])
#expect(rows.allSatisfy { $0.indisvalid })
// An invalid index left by a failed build is dropped and rebuilt.
struct Valid: Decodable { let indisvalid: Bool }
try await sql.raw(
"UPDATE pg_index SET indisvalid = false WHERE indexrelid = 'roster_tags_digest'::regclass"
).run()
try await AddScanIndexes().prepare(on: h.db)
let valid = try await sql.raw(
"SELECT indisvalid FROM pg_index WHERE indexrelid = 'roster_tags_digest'::regclass"
).first(decoding: Valid.self)
#expect(valid?.indisvalid == true)
}
}
private func withPostgresRelay(
env: [String: String] = [:], _ body: (RelayHarness) async throws -> Void
) async throws {
guard let url = ProcessInfo.processInfo.environment["FN_TEST_POSTGRES"],
var components = URLComponents(string: url)
else { return }
let name =
"fn_scale_test_"
+ UUID().uuidString.replacingOccurrences(of: "-", with: "").lowercased()
let admin = try await Application.make(.testing)
admin.logger.logLevel = .critical
try admin.databases.use(.postgres(url: url), as: .psql)
try await (admin.db as? any SQLDatabase)?
.raw(SQLQueryString("CREATE DATABASE \(unsafeRaw: name)")).run()
components.path = "/" + name
var failure: (any Error)?
do {
let h = try await RelayHarness.make(env: env, postgres: components.string ?? url)
do { try await body(h) } catch { failure = error }
await h.shutdown()
} catch {
failure = error
}
try? await (admin.db as? any SQLDatabase)?
.raw(SQLQueryString("DROP DATABASE IF EXISTS \(unsafeRaw: name) WITH (FORCE)")).run()
try await admin.asyncShutdown()
if let failure { throw failure }
}
}
Tests/FlamenetRelayTests/SigninAndAccountTokenTests.swift
+404
View File
@@ -0,0 +1,404 @@
import Crypto
import Fluent
import Foundation
import NIOPosix
import Testing
import Vapor
@testable import FlamenetRelay
/// Relay review 2026-10-14: the per-address sign-in bucket, bcrypt off the
/// event loop, SHA-256 app credentials, and account-service token claims.
@Suite("sign-in and account tokens")
struct SigninAndAccountTokenTests {
private let now = 1_700_000_000
private let ip = "203.0.113.7"
private let enclaveSecret = String(repeating: "k", count: 40)
private func routes(
_ h: RelayHarness, signinMaxPerIP: Int = AccountRoutes.signinMaxPerIPDefault,
threadPool: NIOThreadPool = .singleton,
accountTokens: AccountTokenPolicy = AccountTokenPolicy()
) -> AccountRoutes {
var r = AccountRoutes(
issuer: Issuer(
signingKeyRaw: Data(repeating: 1, count: 32),
audience: "test.relay", lifetime: 300),
limits: h.routes.limits,
signupMaxPerHour: 100)
r.signinMaxPerIP = signinMaxPerIP
r.threadPool = threadPool
r.accountTokens = accountTokens
return r
}
private func post(_ h: RelayHarness, _ path: String, _ body: String) -> Request {
var headers = HTTPHeaders()
headers.contentType = .json
var buf = ByteBufferAllocator().buffer(capacity: body.utf8.count)
buf.writeString(body)
return Request(
application: h.app, method: .POST, url: URI(path: path),
headers: headers, collectedBody: buf,
on: h.app.eventLoopGroup.next())
}
private func signup(_ h: RelayHarness, _ user: String) async throws {
_ = try await routes(h).signup(
post(
h, "/account/signup",
#"{"username":"\#(user)","email":"\#(user)@example.com","password":"hunter2hunter2"}"#
),
clientIP: "198.51.100.1", now: now)
}
private func signin(
_ r: AccountRoutes, _ h: RelayHarness, _ user: String, _ password: String,
from address: String? = nil
) async throws -> Response {
try await r.signin(
post(
h, "/account/signin",
#"{"username":"\#(user)","password":"\#(password)"}"#),
clientIP: address ?? ip, now: now)
}
private func code(_ error: (any Error)?) -> String {
(error as? WireError)?.code ?? "not-a-wire-error"
}
// MARK: - Per-address bucket
@Test("one address rotating usernames is stopped by the per-address bucket")
func addressBucketStopsRotation() async throws {
try await withRelay { h in
try await signup(h, "ada")
let r = routes(h, signinMaxPerIP: 3)
// One wrong guess each at three different names: the pair bucket
// (five per name) never fills.
for name in ["bob", "cyd", "dee"] {
let err = await #expect(throws: WireError.self) {
_ = try await signin(r, h, name, "wrongwrongwrong")
}
#expect(code(err) == "unauthorized")
}
// The fourth, even the right password for a real account, is
// refused as rate-limited.
let err = await #expect(throws: WireError.self) {
_ = try await signin(r, h, "ada", "hunter2hunter2")
}
#expect(code(err) == "rate_limited")
// Another address is unaffected.
let res = try await signin(r, h, "ada", "hunter2hunter2", from: "203.0.113.99")
#expect(res.status == .ok)
}
}
@Test("a successful sign-in does not reset the address count")
func successKeepsAddressCount() async throws {
try await withRelay { h in
try await signup(h, "ada")
let r = routes(h, signinMaxPerIP: 2)
_ = await #expect(throws: WireError.self) {
_ = try await signin(r, h, "bob", "wrongwrongwrong")
}
_ = try await signin(r, h, "ada", "hunter2hunter2")
_ = await #expect(throws: WireError.self) {
_ = try await signin(r, h, "cyd", "wrongwrongwrong")
}
let err = await #expect(throws: WireError.self) {
_ = try await signin(r, h, "dee", "wrongwrongwrong")
}
#expect(code(err) == "rate_limited")
}
}
// MARK: - bcrypt off the event loop
@Test("offLoop runs its work on a pool thread, never an event loop")
func offLoopLeavesTheEventLoop() async throws {
try await withRelay { h in
let r = routes(h, threadPool: h.app.threadPool)
let onLoop = try await r.offLoop {
MultiThreadedEventLoopGroup.currentEventLoop != nil
}
#expect(onLoop == false)
// The same probe from inside an event loop does see one, so the
// check above can fail.
let probe = try await h.app.eventLoopGroup.next().submit {
MultiThreadedEventLoopGroup.currentEventLoop != nil
}.get()
#expect(probe)
}
}
/// A pool that is not running refuses work. If sign-in and credential
/// checks still passed with one, bcrypt would be running somewhere else.
@Test("sign-in and credential checks fail when their pool is stopped")
func bcryptGoesThroughThePool() async throws {
try await withRelay { h in
try await signup(h, "ada")
let stopped = NIOThreadPool(numberOfThreads: 1)
let r = routes(h, threadPool: stopped)
await #expect(throws: WireError.self) {
_ = try await signin(r, h, "ada", "hunter2hunter2")
}
let user = try #require(
try await User.query(on: h.db).filter(\.$username == "ada").first())
let presented = "sel000000001.legacy-secret"
let c = AppCredential()
c.userID = user.id!
c.selector = "sel000000001"
c.secretHash = try Bcrypt.hash("legacy-secret", cost: 4)
c.label = "old"
try await c.create(on: h.db)
#expect(try await r.verifyAppCredential(presented, on: h.db) == nil)
#expect(try await routes(h).verifyAppCredential(presented, on: h.db) == user.id)
}
}
@Test("every bcrypt call in the sources runs through offLoop")
func noBcryptOnTheLoop() throws {
let sources = URL(fileURLWithPath: #filePath)
.deletingLastPathComponent().deletingLastPathComponent().deletingLastPathComponent()
.appendingPathComponent("Sources/FlamenetRelay")
let files = try FileManager.default.contentsOfDirectory(atPath: sources.path)
.filter { $0.hasSuffix(".swift") }
var offenders: [String] = []
for file in files {
let text = try String(
contentsOf: sources.appendingPathComponent(file), encoding: .utf8)
for (n, line) in text.split(separator: "\n").enumerated()
where line.contains("Bcrypt.hash(") || line.contains("Bcrypt.verify(") {
if !line.contains("offLoop") { offenders.append("\(file):\(n + 1)") }
}
}
#expect(offenders.isEmpty, "bcrypt on the calling thread at \(offenders)")
}
/// A fixture hash at the default cost 12 takes 0.3 s of CPU in a debug
/// build, on the test's own cooperative thread. Seven fixtures hashing at
/// once held seven of the eight cooperative threads, and every other test
/// in the process stalled for seconds (WakeTests measured 14 s and 42 s
/// for a poll that takes 10 ms). Cost 4 takes about 1 ms.
@Test("every bcrypt hash in the test fixtures uses cost 4")
func fixturesHashCheaply() throws {
let tests = URL(fileURLWithPath: #filePath).deletingLastPathComponent()
let files = try FileManager.default.contentsOfDirectory(atPath: tests.path)
.filter { $0.hasSuffix(".swift") }
// Split, so this line is not a match itself.
let call = "Bcrypt" + ".hash("
var offenders: [String] = []
for file in files {
let text = try String(contentsOf: tests.appendingPathComponent(file), encoding: .utf8)
for (n, line) in text.split(separator: "\n", omittingEmptySubsequences: false)
.enumerated()
where line.contains(call) && !line.contains("cost: 4") && !line.contains("contains(") {
offenders.append("\(file):\(n + 1)")
}
}
#expect(offenders.isEmpty, "bcrypt at the default cost in fixtures at \(offenders)")
}
// MARK: - App credentials
@Test("a new credential is stored as SHA-256")
func newCredentialIsSHA256() async throws {
try await withRelay { h in
try await signup(h, "ada")
let row = try #require(try await AppCredential.query(on: h.db).first())
#expect(row.secretHash.hasPrefix(AccountRoutes.sha256CredentialPrefix))
#expect(row.secretHash.count == AccountRoutes.sha256CredentialPrefix.count + 64)
}
}
@Test("an old bcrypt credential still works and is rehashed to SHA-256")
func bcryptCredentialIsRehashed() async throws {
try await withRelay { h in
let r = routes(h)
let selector = String(randomHexToken().prefix(12))
let secret = randomHexToken()
let c = AppCredential()
c.userID = 42
c.selector = selector
c.secretHash = try Bcrypt.hash(secret, cost: 4)
c.label = "phone"
try await c.create(on: h.db)
#expect(try await r.verifyAppCredential("\(selector).\(secret)", on: h.db) == 42)
let after = try #require(try await AppCredential.find(c.id, on: h.db))
#expect(after.secretHash == AccountRoutes.sha256CredentialHash(secret))
// Still works on the new format, and a wrong secret still fails.
#expect(try await r.verifyAppCredential("\(selector).\(secret)", on: h.db) == 42)
let wrong = String(secret.dropLast()) + (secret.hasSuffix("0") ? "1" : "0")
#expect(try await r.verifyAppCredential("\(selector).\(wrong)", on: h.db) == nil)
}
}
@Test("a bcrypt secret that is not 256-bit hex stays on bcrypt")
func oddSecretStaysOnBcrypt() async throws {
try await withRelay { h in
let c = AppCredential()
c.userID = 7
c.selector = "abcabcabcabc"
c.secretHash = try Bcrypt.hash("short-secret", cost: 4)
c.label = "odd"
try await c.create(on: h.db)
let id = try await routes(h).verifyAppCredential(
"abcabcabcabc.short-secret", on: h.db)
#expect(id == 7)
let after = try #require(try await AppCredential.find(c.id, on: h.db))
#expect(after.secretHash.hasPrefix("$2"))
}
}
// MARK: - Account-service tokens
private func token(_ claims: [String: Any]) throws -> String {
let header = base64URLEncode(Data(#"{"alg":"HS256","typ":"JWT"}"#.utf8))
let body = base64URLEncode(try JSONSerialization.data(withJSONObject: claims))
let mac = HMAC<SHA256>.authenticationCode(
for: Data("\(header).\(body)".utf8),
using: SymmetricKey(data: Data(enclaveSecret.utf8)))
return "\(header).\(body).\(base64URLEncode(Data(mac)))"
}
private func claims(
iss: String? = "accounts.test", aud: Any? = "test.relay", jti: String? = nil,
sub: String = "pseudo-1"
) -> [String: Any] {
var c: [String: Any] = ["sub": sub, "exp": Double(now + 600)]
if let iss { c["iss"] = iss }
if let aud { c["aud"] = aud }
c["jti"] = jti ?? UUID().uuidString
return c
}
private func session(
_ r: AccountRoutes, _ h: RelayHarness, _ token: String, handle: String = "ada"
) async throws -> Response {
try await r.enclaveSession(
post(h, "/messenger/session", #"{"token":"\#(token)","handle":"\#(handle)"}"#),
secret: enclaveSecret, publicKey: nil, now: now)
}
private func strict() -> AccountTokenPolicy {
AccountTokenPolicy(
strict: true, issuer: "accounts.test", audience: "test.relay",
replay: ReplayCache(window: 120))
}
@Test("strict off accepts a token without iss, aud or jti, and counts it")
func lenientAcceptsAndCounts() async throws {
try await withRelay { h in
let policy = AccountTokenPolicy(
strict: false, issuer: "accounts.test", audience: "test.relay",
replay: ReplayCache(window: 120))
let r = routes(h, accountTokens: policy)
let legacy = try token(["sub": "pseudo-1", "exp": Double(now + 600)])
#expect(try await session(r, h, legacy).status == .ok)
// The same token again: a replay strict mode would refuse.
#expect(try await session(r, h, legacy).status == .ok)
let counts = await policy.lenient.byFault
#expect(counts["no iss"] == 2)
#expect(counts["no aud"] == 2)
#expect(counts["no jti"] == 2)
#expect(await policy.lenient.total == 4)
}
}
@Test("strict on accepts a token with the right iss, aud and a fresh jti")
func strictAcceptsGoodToken() async throws {
try await withRelay { h in
let r = routes(h, accountTokens: strict())
#expect(try await session(r, h, try token(claims())).status == .ok)
// `aud` as a list, as JWT allows.
let listed = try token(claims(aud: ["files.test", "test.relay"]))
#expect(try await session(r, h, listed).status == .ok)
}
}
@Test("strict on refuses missing iss, missing aud, wrong aud, wrong iss and no jti")
func strictRefusesBadClaims() async throws {
try await withRelay { h in
let r = routes(h, accountTokens: strict())
var noJTI = claims()
noJTI.removeValue(forKey: "jti")
for bad in [
claims(iss: nil), claims(aud: nil), claims(aud: "other.relay"),
claims(iss: "someone.else"), noJTI,
] {
let err = await #expect(throws: WireError.self) {
_ = try await session(r, h, try token(bad))
}
#expect(code(err) == "unauthorized")
}
// Nothing was created by any of them.
let users = try await User.query(on: h.db).count()
#expect(users == 0)
}
}
@Test("strict on refuses a replayed jti")
func strictRefusesReplay() async throws {
try await withRelay { h in
let r = routes(h, accountTokens: strict())
let once = try token(claims(jti: "one-time"))
#expect(try await session(r, h, once).status == .ok)
let err = await #expect(throws: WireError.self) {
_ = try await session(r, h, once)
}
#expect(code(err) == "unauthorized")
}
}
@Test("a jti is not spent by a name that was refused")
func refusedNameKeepsTheJTI() async throws {
try await withRelay { h in
try await signup(h, "ada")
let r = routes(h, accountTokens: strict())
let t = try token(claims(jti: "retry-me"))
let err = await #expect(throws: WireError.self) {
_ = try await session(r, h, t, handle: "ada")
}
#expect(code(err) == "username_taken")
#expect(try await session(r, h, t, handle: "ada2").status == .ok)
}
}
@Test("the replay cache holds an entry for the ttl it was given")
func replayTTL() async {
let c = ReplayCache(window: 120)
#expect(await c.admit("x", now: now, ttl: 600))
#expect(await c.admit("x", now: now + 599, ttl: 600) == false)
#expect(await c.admit("x", now: now + 600, ttl: 600))
}
@Test("FN_ACCOUNT_TOKEN_STRICT and FN_ACCOUNT_TOKEN_ISSUER are read; strict is off by default")
func configReadsTheSwitch() throws {
let base = [
"FN_ISSUER_PUBLIC_KEY": Curve25519.Signing.PrivateKey().publicKey
.rawRepresentation.base64EncodedString(),
"FN_RELAY_AUDIENCE": "test.relay",
]
let off = try Config.fromEnvironment(base)
#expect(off.accountTokenStrict == false)
#expect(off.signinMaxPerIP == AccountRoutes.signinMaxPerIPDefault)
var on = base
on["FN_ACCOUNT_TOKEN_STRICT"] = "true"
on["FN_ACCOUNT_TOKEN_ISSUER"] = "accounts.test"
on["FN_SIGNIN_MAX_PER_IP"] = "12"
let c = try Config.fromEnvironment(on)
#expect(c.accountTokenStrict)
#expect(c.accountTokenIssuer == "accounts.test")
#expect(c.signinMaxPerIP == 12)
}
}
Tests/FlamenetRelayTests/SuspensionTests.swift
+1 -1
View File
@@ -39,7 +39,7 @@ struct SuspensionTests {
let c = AppCredential()
c.userID = id
c.selector = selector
c.secretHash = try Bcrypt.hash(secret)
c.secretHash = try Bcrypt.hash(secret, cost: 4)
c.label = "phone"
try await c.create(on: h.db)
return "\(selector).\(secret)"
Tests/FlamenetRelayTests/WakeTests.swift
+236
View File
@@ -0,0 +1,236 @@
import Fluent
import FluentPostgresDriver
import Foundation
import NIOConcurrencyHelpers
import NIOCore
import Redis
import SQLKit
import Testing
import Vapor
@testable import FlamenetRelay
/// Held polls wake from the in-process waiter map, and a poll loads payloads
/// only within its budget.
@Suite("wake-ups and the poll budget")
struct WakeTests {
private let now = 1_700_000_000
/// Counts what each cursor query loaded.
final class Loads: @unchecked Sendable {
private let lock = NIOLock()
private var _bytes: [Int] = []
var bytes: [Int] { lock.withLock { _bytes } }
init(_ h: RelayHarness) {
h.app.storage[Routes.PollLoadedKey.self] = { [lock] n in
lock.withLock { self._bytes.append(n) }
}
}
}
/// Held polls move `RelayCapacity.shared`, which `CapacityTests` asserts on.
private func withHeldPollRelay(_ body: (RelayHarness) async throws -> Void) async throws {
try await ProcessEnvironmentLock.withExclusiveHeldPolls {
try await withRelay(body)
}
}
private func held(
_ h: RelayHarness, as d: RelayHarness.TestDevice, wait: Int
) -> Task<PollResponse, any Error> {
Task {
let req = h.request(
.GET, "/e2e/messages?wait=\(wait)", as: d, now: now,
signingPath: "/e2e/messages")
let res = try await h.routes.poll(req, now: now)
return try JSONDecoder().decode(
PollResponse.self, from: Data((res.body.string ?? "").utf8))
}
}
private func item(to d: RelayHarness.TestDevice, _ payload: String) -> Delivery.Item {
.init(
toUser: d.userID, toDevice: d.deviceID, fromUser: 1, fromDevice: "aaaa-0001",
payload: payload, envelopeID: nil)
}
@Test("a send that commits between the cursor query and the wait still wakes the poll")
func wakeBetweenQueryAndWait() async throws {
try await withHeldPollRelay { h in
let hub = WakeHub()
h.app.storage[Wake.ChannelKey.self] = .inProcess(hub)
let d = h.device(user: 2, id: "bbbb-0002")
try await h.register(d, on: h.db)
// Runs after the first (empty) query, before the first wait.
let (app, db) = (h.app, h.db)
let item = item(to: d, "slipped in")
// When the send committed and signalled. Timed from here, so a
// stall elsewhere in the process before the send (the full suite
// runs a hundred suites at once) is not read as a lost wake.
let wrote = NIOLockedValueBox<Date?>(nil)
h.app.storage[Wake.HeldKey.self] = { _ in
if (try? await Delivery.write([item], db: db, app: app)) != nil {
wrote.withLockedValue { $0 = Date() }
}
}
// A lost wake holds the poll for all 30 s.
let out = try await held(h, as: d, wait: 30).value
#expect(out.messages.map(\.payload) == ["slipped in"])
let sent = try #require(wrote.withLockedValue { $0 }, "the send did not commit")
#expect(Date().timeIntervalSince(sent) < 5, "the wake was lost and the poll waited")
#expect(hub.count == 0, "the poll left its waiter registered")
}
}
@Test("one send to many devices wakes each held poll with its own device's wake")
func manyHeldPollsOneSend() async throws {
try await withHeldPollRelay { h in
let hub = WakeHub()
h.app.storage[Wake.ChannelKey.self] = .inProcess(hub)
let devices = (0..<20).map {
h.device(user: 10 + $0, id: String(format: "cccc-%04x", $0))
}
let bystander = h.device(user: 99, id: "dddd-0099")
for d in devices + [bystander] { try await h.register(d, on: h.db) }
let polls = devices.map { held(h, as: $0, wait: 20) }
let waiting = held(h, as: bystander, wait: 20)
for _ in 0..<200 where hub.count < devices.count + 1 {
try await Task.sleep(nanoseconds: 10_000_000)
}
#expect(hub.count == devices.count + 1)
let began = Date()
_ = try await Delivery.write(
devices.map { item(to: $0, "for \($0.deviceID)") }, db: h.db, app: h.app)
for (d, poll) in zip(devices, polls) {
let out = try await poll.value
#expect(out.messages.map(\.payload) == ["for \(d.deviceID)"])
}
#expect(Date().timeIntervalSince(began) < 2)
// The bystander was not woken and is still held.
#expect(hub.count == 1)
waiting.cancel()
#expect(try await waiting.value.messages.isEmpty)
#expect(hub.count == 0)
}
}
@Test("a failing Redis sleeps a second between queries instead of looping")
func redisDownSleeps() async throws {
try await withHeldPollRelay { h in
// Port 1 refuses, and the pool gives up after 50 ms.
h.app.redis.configuration = try RedisConfiguration(
hostname: "127.0.0.1", port: 1,
pool: .init(connectionRetryTimeout: .milliseconds(50)))
// The Redis pools are made at boot.
try await h.app.asyncBoot()
h.app.storage[Wake.ChannelKey.self] = .redis
let loads = Loads(h)
let d = h.device(user: 2, id: "bbbb-0002")
try await h.register(d, on: h.db)
let out = try await held(h, as: d, wait: 3).value
#expect(out.messages.isEmpty)
// One query, then about one per second for three seconds.
#expect(loads.bytes.count <= 5, "\(loads.bytes.count) cursor queries in 3 s")
}
}
@Test("a poll over large envelopes loads only what fits the budget")
func budgetLoadsOnlyWhatFits() async throws {
try await withRelay { h in try await budget(h) }
}
@Test(
"a poll over large envelopes loads only what fits the budget, on Postgres",
.enabled(if: ProcessInfo.processInfo.environment["FN_TEST_POSTGRES"]?.isEmpty == false))
func budgetLoadsOnlyWhatFitsPostgres() async throws {
try await withPostgresRelay { h in try await budget(h) }
}
private func budget(_ h: RelayHarness) async throws {
let d = h.device(user: 2, id: "bbbb-0002")
try await h.register(d, on: h.db)
let row = 900 * 1024
_ = try await Delivery.write(
(0..<5).map { item(to: d, String(repeating: Character(String($0)), count: row)) },
db: h.db, app: h.app)
let loads = Loads(h)
let first = try await held(h, as: d, wait: 0).value
#expect(first.messages.count == 2)
#expect(first.more == true)
#expect(loads.bytes == [2 * row])
#expect(loads.bytes.allSatisfy { $0 <= Routes.pollPayloadBudget + row })
let second = try await pollAfter(h, d, first.cursor)
#expect(second.messages.count == 2)
let third = try await pollAfter(h, d, second.cursor)
#expect(third.messages.count == 1)
#expect(third.more == nil)
#expect(loads.bytes == [2 * row, 2 * row, row])
}
private func pollAfter(_ h: RelayHarness, _ d: RelayHarness.TestDevice, _ after: Int)
async throws -> PollResponse
{
let req = h.request(
.GET, "/e2e/messages?after=\(after)", as: d, now: now, signingPath: "/e2e/messages")
let res = try await h.routes.poll(req, now: now)
return try JSONDecoder().decode(PollResponse.self, from: Data((res.body.string ?? "").utf8))
}
/// A database of its own on `FN_TEST_POSTGRES`, dropped after.
private func withPostgresRelay(_ body: (RelayHarness) async throws -> Void) async throws {
guard let url = ProcessInfo.processInfo.environment["FN_TEST_POSTGRES"],
var components = URLComponents(string: url)
else { return }
let name =
"fn_wake_test_"
+ UUID().uuidString.replacingOccurrences(of: "-", with: "").lowercased()
let admin = try await Application.make(.testing)
admin.logger.logLevel = .critical
try admin.databases.use(.postgres(url: url), as: .psql)
try await (admin.db as? any SQLDatabase)?
.raw(SQLQueryString("CREATE DATABASE \(unsafeRaw: name)")).run()
components.path = "/" + name
var failure: (any Error)?
do {
let h = try await RelayHarness.make(postgres: components.string ?? url)
do { try await body(h) } catch { failure = error }
await h.shutdown()
} catch {
failure = error
}
try? await (admin.db as? any SQLDatabase)?
.raw(SQLQueryString("DROP DATABASE IF EXISTS \(unsafeRaw: name) WITH (FORCE)")).run()
try await admin.asyncShutdown()
if let failure { throw failure }
}
}
@Suite("wake hub")
struct WakeHubTests {
@Test("a signal before the wait is kept, and other devices' signals are not")
func pendingSignal() async {
let hub = WakeHub()
let a = hub.register(device: "a")
let b = hub.register(device: "b")
hub.signal(device: "a")
#expect(await a.wait(seconds: 5) == .woken)
#expect(await b.wait(seconds: 0.05) == .timedOut)
// Consumed once.
#expect(await a.wait(seconds: 0.05) == .timedOut)
a.close()
a.close()
b.close()
#expect(hub.count == 0)
// A signal with nobody waiting goes nowhere.
hub.signal(device: "a")
#expect(hub.count == 0)
}
}
docker-compose.federated.yml
+3 -3
View File
@@ -13,9 +13,9 @@
# caddy , TLS on 443, automatic certificates. Federation peers dial your
# public hostname over HTTPS; a relay on 127.0.0.1:8080 federates
# with nobody.
# redis , wake-ups for held polls. Optional in principle; at federated
# scale the interval fallback turns delivery latency into a coin
# toss, so it ships on in this profile.
# redis , rate limits and replay protection that survive a restart,
# and wake-ups for held polls when FN_RELAY_REPLICAS is above 1.
# One relay wakes held polls in memory.
# coturn , voice/video call relaying for peers behind NAT. Without it,
# calls connect only when both phones can reach each other
# directly, which across the public internet is "sometimes".
docs/CAPACITY.md
+30 -8
View File
@@ -16,19 +16,41 @@ Delivery was 100% of accepted sends in every run. The p95 growth at 2000
held polls is the next knee approaching; the p99 tails at every scale are
wake-ups that landed between two polls and waited out a cycle.
The table above predates in-process wake-ups (2026-10-14) and has not been
re-run since. Re-run it before quoting these numbers for the current image.
## The two ceilings this measurement found, and their fixes
1. **The Redis client pool.** Every held long-poll pins one Redis connection
for its whole `BRPOP` wait, and the pool is per event loop. At the old 32
per loop, ~250 held polls saturated the pool and p50 went from 18 ms to
seconds while the CPU burned three cores queueing. Now 512 per loop,
settable with `FN_REDIS_MAX_CONNECTIONS_PER_LOOP`.
1. **The Redis client pool.** Each held long-poll used to pin one Redis
connection for its whole `BRPOP` wait, and the pool is per event loop. At
32 per loop, ~250 held polls saturated the pool and p50 went from 18 ms to
seconds while the CPU burned three cores queueing. The first fix raised the
pool to 512 per loop.
2. **File descriptors.** A held poll costs two fds (HTTP socket + Redis
connection). Docker's default soft limit of 1024 caps the relay at ~450
open apps, and past it connection setup fails hard. The compose files set
Since 2026-10-14 one relay keeps its waiters in memory (`WakeHub` in
`RoutesWake.swift`): a poll registers once per request, before its first
cursor query, and `Delivery.write` signals it after the commit. A held poll
holds no Redis connection and no database connection, so the pool is back
to 32 per loop, for short commands only. A wake that lands between the
query and the wait is kept on the waiter, which removes the p99 tail of
wake-ups that waited out a cycle.
With `FN_RELAY_REPLICAS` above 1 a send can land on another relay, so
wake-ups still go through `BRPOP` on a Redis list per device, the pool
default stays 512, and 512 per loop is the held-poll ceiling. Either
default is settable with `FN_REDIS_MAX_CONNECTIONS_PER_LOOP`. A failed
Redis wait sleeps one second before the next cursor query.
2. **File descriptors.** A held poll costs one fd (the HTTP socket) on one
relay, and two (plus its Redis connection) with several replicas. Docker's
default soft limit of 1024 capped the relay at ~450 open apps at two fds
each, and past it connection setup fails hard. The compose files set
`ulimits: nofile: 65536`, and the relay warns at boot under a low limit.
A poll reads id and payload length first, then loads payloads only for rows
inside the 2 MiB budget (`Routes.pollRows`). 500 padded envelopes waiting cost
one 2 MiB read, not 32 MB.
Also worth knowing: `FN_PG_MAX_CONNECTIONS_PER_LOOP` (default 4). Fluent's
own default is ONE connection per event loop. And the SQLite tier locks
under parallel registration: one file is for one household, not concurrency.
docs/CONFIGURATION.md
+59 -6
View File
@@ -51,10 +51,14 @@ running one box either way.
## Storage
**A self-hosted relay needs Postgres.** SQLite is for local development only
and receives no new features. Delete-on-ack already runs only on Postgres,
because SQLite reuses row ids.
| Variable | Default | Notes |
|---|---|---|
| `FN_POSTGRES_URL` | unset | Set it and Postgres is used. |
| `FN_DB` | `relay.sqlite` | SQLite file, used when the above is unset. **The schema is identical on both**: SQLite is not a degraded mode, and is the right choice for a relay serving a household or a company. |
| `FN_POSTGRES_URL` | unset | Set it and Postgres is used. Required for any relay with real accounts. |
| `FN_DB` | `relay.sqlite` | SQLite file, used when the above is unset. Local development only. |
| `FN_ATTACHMENTS_DIR` | `attachments` | Blobs are files, not rows, in both modes. |
| `FN_WORKDIR` | the process's own directory | Where every relative path above is resolved from. |
| `FN_S3_ENDPOINT` | unset | Set it and attachments go to object storage instead of `FN_ATTACHMENTS_DIR`. |
@@ -70,8 +74,8 @@ store keeps its own copy of each.
| Variable | Default | Notes |
|---|---|---|
| `FN_REDIS_URL` | unset | Shared store for the state above. Set but unusable, held polls fall back to polling and the log says so. |
| `FN_REDIS_MAX_CONNECTIONS_PER_LOOP` | `512` | Redis connections per event loop, which is also the ceiling on held polls. |
| `FN_REDIS_URL` | unset | Shared store for the state above. Set but unusable, the log says so. With `FN_RELAY_REPLICAS` above 1 it also carries held-poll wake-ups between relays. |
| `FN_REDIS_MAX_CONNECTIONS_PER_LOOP` | `32`, or `512` with `FN_RELAY_REPLICAS` above 1 | Redis connections per event loop. With several replicas each held poll pins one, so it is also the ceiling on held polls. One relay wakes held polls in memory. |
| `FN_RELAY_REPLICAS` | `1` | How many instances share this database. More than one with no `FN_REDIS_URL` is refused at boot: a single-use proof signature would be accepted once per instance, and an attacker would get that many times the sign-in attempts. |
## The contact graph
@@ -103,9 +107,28 @@ should stand alone without configuring anything.
| Variable | Default | Notes |
|---|---|---|
| `FN_FEDERATION` | `closed` | `closed`, `allowlist`, or `open`. An unrecognised value falls back to `closed` rather than refusing to boot, a typo in an optional variable should not stop a relay serving its own users. |
| `FN_FEDERATION_PEERS` | empty | Comma-separated hostnames, used only under `allowlist`. Enforced **outbound as well as inbound**, from the same rule. Peers named here are fetched wherever they resolve: the outbound address guard, which refuses private and loopback addresses, is for names an attacker can choose under `open` federation, not for a name you typed and pinned by key. Three relays on one host resolve each other to private addresses, and would otherwise refuse every deposit between them as unreachable. |
| `FN_FEDERATION_PEERS` | empty | Comma-separated peers, each `host`, `host=key` or `host=key1\|key2`. The key is the peer's federation public key, base64, as its `/.well-known/flamenet/relay` publishes it in `federationKey`. A pinned peer whose descriptor publishes any other key is refused with 401. A bare `host` accepts whatever key the descriptor publishes, and boot logs a warning naming it. The host list is used only under `allowlist`; pins apply under `allowlist` and `open`. Enforced **outbound as well as inbound**, from the same rule. Peers named here are fetched wherever they resolve: the outbound address guard, which refuses private and loopback addresses, is for names an attacker can choose under `open` federation, not for a name you typed and pinned by key. Three relays on one host resolve each other to private addresses, and would otherwise refuse every deposit between them as unreachable. |
| `FN_FEDERATION_KEY` | unset | Base64 Ed25519 seed. **Required** when `FN_FEDERATION` is not `closed`: the relay refuses to boot otherwise, rather than starting unreachable and letting a stranger discover it. |
### Pinning and rotating a peer's key
Read the peer's key and pin it:
```bash
curl -s https://peer.example/.well-known/flamenet/relay | jq -r .federationKey
FN_FEDERATION_PEERS=peer.example=<that key>
```
A peer rotating its key tells you the new public key first. Then:
1. Pin both, old first: `peer.example=<old>|<new>`. Restart. Both are accepted.
2. The peer switches `FN_FEDERATION_KEY` to the new seed and restarts.
3. Wait an hour: the relay caches a peer's descriptor for 3600 seconds.
4. Drop the old key: `peer.example=<new>`. Restart.
Rotating your own key is the same steps from the other side: send each peer
the new public key, wait until all of them pin both, then switch.
`FN_FEDERATION_KEY` is deliberately **not** the issuer key. A key that only ever
needs to say "this relay said this" should not also be able to mint a token for
any user on it.
@@ -146,6 +169,8 @@ Leaving all of these unset gives you a relay with its own user table.
| `FN_ENCLAVE_PUBLIC_KEY` | unset | Base64 Ed25519 public key from the account service's `/.well-known/jwks.json` (the `x` member, base64url-decoded). **This is the one to use.** |
| `FN_ENCLAVE_JWT_SECRET` | unset | The older shared HMAC secret. Works, and is why third-party relays were never possible: the secret that checks a token also mints one, so anything holding it can impersonate any account. |
| `FN_ACCOUNTS_URL` | `https://accounts.flamenet.io` | Which account service to advertise. Point it at your own if you run one. |
| `FN_ACCOUNT_TOKEN_ISSUER` | unset | The `iss` an account-service token must carry. Required before strict mode can be switched on; the relay refuses to boot with strict on and this unset. |
| `FN_ACCOUNT_TOKEN_STRICT` | `false` | Off, a token without `iss`, `aud` and `jti`, or with the wrong ones, is logged, counted and accepted. On, it is refused with 401, and each `jti` is accepted once until the token expires. Turn it on only after the account service stamps all three (steps in SECURITY.md). |
**Configured, not fetched.** This decides who may assert an identity at your
relay; a relay that learned it over the network could be pointed at a different
@@ -163,6 +188,7 @@ Defaults are sane; raise them only with a reason.
|---|---|---|
| `FN_BUNDLE_RATE` | `600`/hr | Prekey-bundle fetches per requester. Each pops a one-time prekey per device, so an unmetered version is a pool drain. |
| `FN_BUNDLE_RATE_TARGET` | `60`/hr | Per requester-target pair. Deliberately **not** metered per target alone, that would let one abuser make a popular account unstartable for everyone. The anonymous `POST /e2e/anon/bundles` uses the same two numbers per address and per address-target pair, or this one per presence token when the caller presents one. |
| `FN_SIGNIN_MAX_PER_IP` | `30` per 15 min | Failed sign-ins from one address across every username, beside the per-address-and-username throttle. A success clears only the per-username count. **`0` switches it off.** |
| `FN_SIGNUP_MAX_PER_HOUR` | `50`/hr | New accounts per IP. An IP is not a person, carrier CGNAT, offices and campuses all share one. It was `5`, which refused the sixth user behind any shared connection. Per-IP is trivially rotated, so treat this as a speed bump against lazy bulk registration rather than a defence. |
| `FN_SIGNUP_MAX_PER_NETWORK` | `400`/hr | New accounts per network, a /24 for IPv4, a /48 for IPv6. The per-address limit counts a rented subnet as separate strangers, and more addresses is the cheapest thing an abuser buys. Deliberately generous: a campus or a carrier behind CGNAT is thousands of real people on one network, and refusing them is the worse failure. **`0` switches network counting off**, leaving only the per-address limit. |
| `FN_SIGNUP_POW_BITS` | `0` (off) | Leading zero bits of SHA-256 work a new account must do before it is created (relay#52). **Not a CAPTCHA on purpose**: a CAPTCHA hands a third party every visitor's connection, which is a strange thing for a product whose argument is that it collects almost nothing. This collects nothing at all. Capped at `24` however it is set: higher does not read as spam protection to the person it hits, it reads as an app that hangs on the sign-up button. |
@@ -201,7 +227,8 @@ On. A person can report something to whoever runs the server, so blocking is not
| Variable | Default | What it does |
|---|---|---|
| `FN_CONTENT_REPORTS` | on | Set to `false`, `0`, `no` or `off` to stop taking reports. Anything else, including a typo, leaves them on. |
| `FN_REPORTS_TO` | unset | Where a filed report is mailed, through the SMTP settings. Unset, reports are kept in `content_reports` and read with `Scripts/reports.sh`, and the log says nobody was told. |
| `FN_REPORTS_TO` | unset | Where the report digest is mailed, through the SMTP settings. One message lists every report filed since the last one, at most one an hour, checked every 5 minutes; an hour with no reports sends nothing. At most 200 reports per message, each note and copy cut at 1,000 characters. A digest that fails to send is retried with the same reports. The watermark is the `report_digests` table, so a restart neither loses nor repeats a report. Unset, reports are kept in `content_reports` and read with `Scripts/reports.sh`, and the log says nobody was told. |
| `FN_REPORT_RATE` | `10`/hr | Reports one account may file per hour. Over it, the route answers 429 and nothing is stored. |
| `FN_REPORT_RETENTION_DAYS` | `365` | How long a content report, and the plaintext copy the reporter chose to attach, is kept. Long enough to show a pattern, bounded so it is not forever. Shown on /metadata. |
Off is a legitimate answer, not a broken relay. If nobody is going to read them, collecting reports is a promise the server cannot keep, and saying so plainly is better than a queue nobody opens.
@@ -327,6 +354,32 @@ service (`--all` adds retired ones) and `Scripts/bridges.sh retire <fingerprint>
takes one out of service. A retired identity is never listed again. The script
header shows how to list a bridge on another machine.
## Metrics
Off. Set `FN_METRICS_PORT` and the relay opens a second listener that answers
`GET /metrics` in the Prometheus text format and nothing else. The public port
never serves `/metrics`, and the relay refuses to boot when the two ports are
the same.
| Variable | Default | Notes |
|---|---|---|
| `FN_METRICS_PORT` | unset (off) | Port for the metrics listener. |
| `FN_METRICS_HOST` | `127.0.0.1` | Bind address. In Docker, set `0.0.0.0` so a scraper on the same network can reach it, and do not publish the port. |
| Metric | Type | Counts |
|---|---|---|
| `flamenet_held_polls` | gauge | Long-polls held open now. |
| `flamenet_push_sent_total` | counter | APNs pushes Apple accepted. |
| `flamenet_push_failed_total` | counter | APNs pushes refused, unreachable or not signed. |
| `flamenet_push_dropped_total` | counter | Pushes dropped because the push queue was full. |
| `flamenet_limiter_fail_open_total` | counter | Rate-limit checks allowed because Redis did not answer. |
| `flamenet_envelopes_written_total` | counter | Envelopes committed. |
| `flamenet_federation_deposits_in_total` | counter | Deposits accepted from peers. |
| `flamenet_federation_deposits_out_total` | counter | Deposits a peer accepted from this relay. |
Counters start at zero with the process. No metric carries an id, a name or an
address.
## Site copy
| Variable | Default | Notes |
docs/DATABASE.md
+22
View File
@@ -94,3 +94,25 @@ looks like a permissions problem and is not: `pg_dump --schema-only` carries the
`_fluent_migrations` table but none of its rows, so Fluent believed nothing had
been migrated and tried to create everything again. A rehearsal that omits the
bookkeeping is not a rehearsal.
## Indexes built CONCURRENTLY
`AddScanIndexes` (relay review 2026-10-14) adds `roster_tags_digest` on
`roster_tags (digest)` and `prekeys_used_created` on
`prekeys (created_at) WHERE used`. On Postgres it runs `CREATE INDEX
CONCURRENTLY`, which refuses to run inside a transaction. FluentKit's migrator
opens none around `prepare`, so the migration builds them at boot, and
`ScanAndScaleTests` proves it against Postgres 17.
A concurrent build waits for every open transaction on the table to finish, so
boot waits with it. Both tables are small today. To keep the build out of the
boot path on a large table, run these as `relayapp` before deploying; the
migration then finds them and does nothing:
CREATE INDEX CONCURRENTLY IF NOT EXISTS roster_tags_digest
ON roster_tags (digest);
CREATE INDEX CONCURRENTLY IF NOT EXISTS prekeys_used_created
ON prekeys (created_at) WHERE used;
A build that fails halfway leaves an index marked INVALID. The migration drops
and rebuilds one of those; by hand, `DROP INDEX CONCURRENTLY` it first.
docs/FANOUT.md
+2 -1
View File
@@ -133,7 +133,8 @@ global envelope id, so clients see no change. A group fan-out locks one
device set per request, in sorted order, so two group sends cannot deadlock.
SQLite has one writer and skips the lock.
After the commit it wakes held polls and queues one push job on the push
After the commit it wakes held polls (in memory on one relay, through Redis
with several, see `RoutesWake.swift`) and queues one push job on the push
worker: 8 jobs run at a time and 1024 wait. A full queue drops the push and
counts it; the envelope is still collected on the next poll. Shutdown flushes
the queue for up to 10 seconds. Federated deposits now wake and push the same
docs/POOLING.md
+4 -4
View File
@@ -12,10 +12,10 @@ because it changes what the fix is for.
A held long-poll does **not** pin a database connection. `Routes.poll` borrows
one from the pool for each cursor query, a few milliseconds, and gives it
back between iterations. That was a deliberate choice, and `RoutesWake.swift`
says why: Postgres `LISTEN` *would* have pinned a backend per waiter, which is
what caps simultaneous waiters at about 86, and it was rejected for exactly that
reason.
back between iterations, and waits in memory between them (`RoutesWake.swift`).
That was a deliberate choice: Postgres `LISTEN` *would* have pinned a backend
per waiter, which is what caps simultaneous waiters at about 86, and it was
rejected for exactly that reason.
So the pressure on `max_connections` is not proportional to **held polls**. It
is proportional to the **query rate**, and `docs/COST.md` measures that at nine