Files
flamenet-relay/docker-compose.prod.yml
Paul HittandClaude Opus 5.5 af56f810a8
ci / checks (pull_request) Successful in 12s
ci / swift (pull_request) Successful in 8m50s
ci / image (pull_request) Successful in 50s
ci / deploy (pull_request) Skipped
Prod compose: record the hardening running on the box since 2026-10-15
The box's compose got these during the security audit and the repo copy
did not, so a rebuild from the repo would have undone them:
- Redis requires a password (FN_REDIS_PASSWORD, written to a config file
  inside the container; healthcheck via REDISCLI_AUTH); FN_REDIS_URL
  carries it.
- The internal network is internal: true, so the database and Redis have
  no route off the box. The relay reaches S3 and APNs through web.
- relay, redis and obfs4bridge drop every capability; db keeps the five its
  entrypoint needs. All four set no-new-privileges.
- obfs4bridge is on web only.

.env.example documents FN_REDIS_PASSWORD. The deploy itself uses the
box's compose, so this changes nothing that is running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 06:38:04 -04:00

474 lines
25 KiB
YAML

# Flamenet relay, production.
#
# The relay the iOS app talks to, at relay.flamenet.io. The project's
# public pages are the same service (it serves its own site), but the app is
# pinned to this hostname and nothing else: relay tokens are audience-bound, so
# a token minted here is not valid at another name.
#
# Differences from the development compose file next to it:
# - no ports published on the host; Traefik on the `web` network reaches it
# - FN_RELAY_AUDIENCE is the real hostname, not the local placeholder
# - the database volume is named for this stack rather than the directory
#
# Two hostnames, one audience. `relay.flamenet.io` is the canonical name and the
# token audience; `messenger.flamenet.io` is what the service was called until
# the rename and still routes here, because builds already in the field dial it.
# The client decodes the audience it is handed and never compares it to the host
# it asked, so the old name reaching a relay with the new audience is fine.
# Retire the messenger router only once a client pinned to the new name has
# actually shipped.
#
# docker compose -f docker-compose.prod.yml up -d
#
# `web` is Traefik's edge network and already exists; the database sits on an
# internal network with no route off the box.
services:
db:
image: postgres:17-alpine
restart: unless-stopped
environment:
POSTGRES_DB: flamenet
POSTGRES_USER: flamenet
POSTGRES_PASSWORD: "${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}"
volumes:
- relay-db:/var/lib/postgresql/data
security_opt: ["no-new-privileges:true"]
cap_drop: [ALL]
# The entrypoint fixes data-dir ownership and drops to the postgres user.
cap_add: [CHOWN, DAC_OVERRIDE, FOWNER, SETGID, SETUID]
networks: [internal]
healthcheck:
test: ["CMD-SHELL", "pg_isready -U flamenet -d flamenet"]
interval: 10s
timeout: 5s
retries: 10
# Its own instance, not another product's: coupling two means one restart
# takes out both. Not persisted on purpose, see the note in RoutesWake.swift.
redis:
image: redis:7-alpine
# A password since 2026-10-15. It goes into a config file inside the
# container so it never appears in the host's process list; redis-cli
# reads it from REDISCLI_AUTH for the same reason. FN_REDIS_URL carries it
# as redis://:<password>@redis:6379.
user: redis
environment:
REDIS_PASSWORD: "${FN_REDIS_PASSWORD:?set FN_REDIS_PASSWORD in .env}"
command: ["sh", "-c", "printf 'requirepass %s\\n' \"$$REDIS_PASSWORD\" > /tmp/auth.conf && exec redis-server /tmp/auth.conf --save '' --appendonly no --maxmemory 64mb --maxmemory-policy allkeys-lru"]
restart: unless-stopped
# No Linux capabilities beyond what the process needs, and no privilege
# gain through setuid binaries (security audit 2026-10-15).
security_opt: ["no-new-privileges:true"]
cap_drop: [ALL]
networks: [internal]
healthcheck:
test: ["CMD-SHELL", "REDISCLI_AUTH=\"$$REDIS_PASSWORD\" redis-cli ping | grep -q PONG"]
interval: 10s
timeout: 3s
retries: 5
relay:
build: .
image: flamenet-relay:prod
# Bound to loopback, and it was missing entirely. This file published
# no port at all, so the relay was reachable only from the `web` network,
# fine behind a reverse proxy, and wrong for the guide, which tells a new
# operator to open http://localhost:8080 and see the landing page. They got
# nothing, after a build and a start that both looked successful.
#
# `127.0.0.1:` on purpose. Publishing `8080:8080` would put an
# unproxied, TLS-less relay on every interface the host has, including the
# public one, for somebody who has not reached the TLS section yet.
ports:
- "127.0.0.1:8080:8080"
restart: unless-stopped
depends_on:
db:
condition: service_healthy
environment:
FN_HOST: "0.0.0.0"
# Errors and criticals only. `debug` logs request URLs and SQL, and a
# protocol URL contains a device id, never raise this to `debug` on a
# machine carrying real accounts.
FN_LOG_LEVEL: "${FN_LOG_LEVEL:-error}"
FN_POSTGRES_URL: "postgres://flamenet:${POSTGRES_PASSWORD}@db:5432/flamenet"
# Must match what the app is pinned to. Tokens carry this as their
# audience and the relay checks it, so a mismatch rejects every token
# with no other symptom.
FN_RELAY_AUDIENCE: "relay.flamenet.io"
FN_ISSUER_PUBLIC_KEY: "${FN_ISSUER_PUBLIC_KEY:?generate an issuer keypair - see .env.example}"
FN_ISSUER_PRIVATE_KEY: "${FN_ISSUER_PRIVATE_KEY:?generate an issuer keypair - see .env.example}"
# The shared FlameNet account service's signing key, when this relay
# accepts FlameNet sessions. Empty on a self-hosted relay, which is its
# own identity provider and registers no such route.
#
# It has to be listed here to exist inside the container: `.env` only
# substitutes variables *into this file*, it does not pass them through.
# A value present in .env and absent from this block reaches nothing, and
# the route it enables simply never appears, a 404 with no error
# anywhere, which is a poor way to find out.
FN_ENCLAVE_JWT_SECRET: "${FN_ENCLAVE_JWT_SECRET:-}"
# Voice calls (§8.10). `FN_TURN_SECRET` is coturn's `--static-auth-secret`:
# the relay HMACs an expiry into a credential the TURN server verifies
# without either side storing a password, so it must match EXACTLY or
# every call needing a relay fails authentication.
#
# Required, because this stack now RUNS the coturn these configure.
# They were optional while TURN was somebody else's problem; with the
# service in the same file, an empty secret here and a real one there
# means every relayed call fails authentication, and the two must not be
# able to drift. Both sides read the same variable and both refuse to
# start without it.
#
# Absent, `/e2e/turn` answers 503 and calls fall back to peer-to-peer,
# still a supported deployment, and the way to choose it is to remove the
# coturn service rather than to leave these blank.
#
# `FN_TURN_HOST` is a name the CLIENT resolves for STUN/TURN over UDP,
# and coturn takes it as its realm. It must stay unproxied in DNS:
# Cloudflare does not proxy UDP, and a proxied record makes every relayed
# call fail while the record looks perfectly healthy.
# Optional, and `:-` not `:?`. These were required, which made the
# documented quick start fail on its first command: SELF-HOSTING says
# voice calls are optional, `.env.example` ships both of these empty, and
# Compose's `:?` errors on empty as well as unset. Copying the example and
# running `docker compose up -d`, exactly what the guide says to do,
# died before anything started.
#
# Absent, `/e2e/turn` answers 503 and calls are peer-to-peer or nothing,
# which is a supported deployment rather than a fault.
FN_TURN_SECRET: "${FN_TURN_SECRET:-}"
# Wake-ups for held polls. Unset falls back to an interval.
FN_REDIS_URL: "${FN_REDIS_URL:-}"
# New accounts per IP per hour. See .env.example, the short version is
# that an IP is not a person, and the old default of 5 locked out the
# sixth user behind any shared connection.
FN_SIGNUP_MAX_PER_HOUR: "${FN_SIGNUP_MAX_PER_HOUR:-50}"
# The edge proxy whose X-Forwarded-For the relay believes: its pinned
# address on `web`, and nothing wider. Every container on `web` can reach
# the relay directly and write the header. Empty, the relay meters every
# request by the proxy's own address, so per-IP limits hit everyone at
# once. See docs/CONFIGURATION.md.
FN_TRUSTED_PROXY_ADDRS: "${FN_TRUSTED_PROXY_ADDRS:-}"
# Empty means ON: a collected envelope is deleted when the app's next
# poll confirms it, instead of sitting for FN_DELIVERED_RETENTION_DAYS.
# Set 0 to turn it off. Postgres only whatever this says: SQLite reuses
# row ids, so the cursor would delete the next message unread. See
# DeleteOnAckTests.
FN_DELETE_ON_ACK: "${FN_DELETE_ON_ACK:-}"
# Blob storage, and it was unreachable. Compose's `environment:` is
# an allowlist: a variable set in `.env` and not named here never reaches
# the container, so a self-hoster following the docs to put attachments in
# S3 got a relay that silently kept using local disk.
FN_S3_ENDPOINT: "${FN_S3_ENDPOINT:-}"
FN_S3_BUCKET: "${FN_S3_BUCKET:-}"
FN_S3_REGION: "${FN_S3_REGION:-}"
FN_S3_ACCESS_KEY_ID: "${FN_S3_ACCESS_KEY_ID:-}"
FN_S3_SECRET_KEY: "${FN_S3_SECRET_KEY:-}"
FN_S3_PREFIX: "${FN_S3_PREFIX:-attachments/}"
# Same trap. SELF-HOSTING tells an operator to "set four variables
# in .env" to publish an Android build to their own users. Without these
# lines that instruction did nothing at all.
FN_ANDROID_VERSION_CODE: "${FN_ANDROID_VERSION_CODE:-}"
FN_ANDROID_VERSION_NAME: "${FN_ANDROID_VERSION_NAME:-}"
FN_ANDROID_SHA256: "${FN_ANDROID_SHA256:-}"
FN_ANDROID_NOTES: "${FN_ANDROID_NOTES:-}"
# iOS wake-ups. Unset and iOS receives when the app is opened.
FN_APNS_KEY_P8: "${FN_APNS_KEY_P8:-}"
FN_APNS_KEY_ID: "${FN_APNS_KEY_ID:-}"
FN_APNS_TEAM_ID: "${FN_APNS_TEAM_ID:-}"
FN_APNS_TOPIC: "${FN_APNS_TOPIC:-net.flamenet.messenger}"
# Sandbox by default. A production push to a debug token fails with
# BadDeviceToken and the error does not say which way round it is.
FN_APNS_ENV: "${FN_APNS_ENV:-sandbox}"
# Where a content report is mailed, as soon as it is filed.
#
# Unset, a report is still filed and still readable with
# Scripts/reports.sh, and the log says plainly that nobody was told --
# which is the state this ran in for months while the safety page
# promised that reports reach the people who run the relay.
#
# Mailing it also needs FN_SMTP_HOST and FN_SMTP_FROM. Without those
# this relay cannot send mail at all, and neither can password reset.
FN_REPORTS_TO: "${FN_REPORTS_TO:-}"
FN_FUNDING_MANIFESTS: "${FN_FUNDING_MANIFESTS:-}"
FN_TURN_HOST: "${FN_TURN_HOST:-}"
FN_DONATE_PROVIDER: "${FN_DONATE_PROVIDER:-}"
FN_DONATE_SECRET: "${FN_DONATE_SECRET:-}"
# The connected account the donation is charged on, and the publishable
# key the browser mounts the form with. Without the first, the same key
# charges the platform account and Stripe reports no error.
FN_DONATE_ACCOUNT: "${FN_DONATE_ACCOUNT:-}"
FN_DONATE_PUBLISHABLE_KEY: "${FN_DONATE_PUBLISHABLE_KEY:-}"
# Everything else `.env` documents reaches the relay through the file
# itself. What is listed above wins over it.
env_file:
- .env
volumes:
- relay-attachments:/data/attachments
# No Linux capabilities beyond what the process needs, and no privilege
# gain through setuid binaries (security audit 2026-10-15).
security_opt: ["no-new-privileges:true"]
cap_drop: [ALL]
networks: [web, internal]
labels:
traefik.enable: "true"
traefik.docker.network: "web"
# Router names here match `/srv/prod/messenger/docker-compose.yaml`
# on the box, which is the file that actually runs. This one does not
# deploy anything, it is the reference for what production looks like,
# so the names have to line up or the two describe the same hostnames
# under different labels and neither reads as authoritative.
#
# `flamenet-relay` is the SERVICE name in both. It is not a router name in
# either any more; every router below points at it.
traefik.http.services.flamenet-relay.loadbalancer.server.port: "8080"
# The canonical name, and the token audience.
traefik.http.routers.relay-subdomain.rule: "Host(`relay.flamenet.io`)"
traefik.http.routers.relay-subdomain.entrypoints: "websecure"
traefik.http.routers.relay-subdomain.tls.certresolver: "le"
traefik.http.routers.relay-subdomain.service: "flamenet-relay"
# The pre-rename name, kept reachable for builds already in the field.
# Its own router rather than a second Host on the rule above, so retiring
# it later is a deletion rather than an edit to a working rule.
traefik.http.routers.messenger-subdomain.rule: "Host(`messenger.flamenet.io`)"
traefik.http.routers.messenger-subdomain.entrypoints: "websecure"
traefik.http.routers.messenger-subdomain.tls.certresolver: "le"
traefik.http.routers.messenger-subdomain.service: "flamenet-relay"
# api-docs.relay.flamenet.io, the same relay, opening on its API
# reference instead of its landing page.
#
# The documents already exist and are already served: /api, /configuration
# and /hosting are rendered by this container from the repository's own
# markdown, so an operator reading them is reading what their relay
# actually ships. What was missing was only a name that lands on them, and
# a name is worth having, "read the API docs at relay.flamenet.io/api" is
# a sentence people mistype.
#
# A router plus a path rewrite rather than a second service: there is only
# one relay, and pointing a second container at the same documents would
# be two things to keep in step for no gain.
#
# The A record exists (added 2026-08-25, 135.148.121.177, unproxied) and
# the certificate has issued.
#
# Traefik does not retry ACME for a host that had no DNS when it
# first saw the router. This router existed for a while before the
# record did, and it stayed certificate-less afterwards: port 80 answered
# 301, so routing looked healthy, while HTTPS refused the connection and
# nothing logged an error. Recreating the container re-emits the labels
# and issuance follows within a minute, `docker compose up -d --no-deps
# --no-build relay`, which is narrower than restarting Traefik and every
# other site on the box with it.
traefik.http.routers.relay-apidocs.rule: "Host(`api-docs.relay.flamenet.io`)"
traefik.http.routers.relay-apidocs.entrypoints: "websecure"
traefik.http.routers.relay-apidocs.tls.certresolver: "le"
traefik.http.routers.relay-apidocs.service: "flamenet-relay"
traefik.http.routers.relay-apidocs.middlewares: "apidocs-root"
# Only the bare root is rewritten. Every other path is left alone, so the
# cross-links between /api, /configuration and /hosting keep working under
# this name instead of dead-ending.
# `$$`, not `$`. Compose interpolates this file, and a lone `$` is the
# start of a variable reference, it warns and the anchor can be eaten.
# `$$` is the documented escape and reaches Traefik as a single `$`.
traefik.http.middlewares.apidocs-root.replacepathregex.regex: "^/$$"
traefik.http.middlewares.apidocs-root.replacepathregex.replacement: "/api"
# The public pages.
#
# These were commented out here while flamenetmessenger.com was still
# claimed by the provisioner container, because two routers matching one
# Host rule is a coin toss rather than a takeover. That has since been
# resolved on the box: the relay is the only container claiming the
# domain, and it serves the landing page there today. Enabled to match.
traefik.http.routers.flamenet-site.rule: "Host(`flamenetmessenger.com`) || Host(`www.flamenetmessenger.com`)"
traefik.http.routers.flamenet-site.entrypoints: "websecure"
traefik.http.routers.flamenet-site.tls.certresolver: "le"
traefik.http.routers.flamenet-site.service: "flamenet-relay"
# One difference from the box, kept deliberately. There, a router still
# called `flamenet-relay` ALSO matches `Host(messenger.flamenet.io)`,
# the same rule `messenger-subdomain` already carries. Two routers on one
# rule is redundant rather than harmful when both point at the same
# service, but it is a leftover, and reproducing it here would spread it.
# `messenger-subdomain` covers that hostname on its own.
# Voice/video relaying for peers behind NAT (§8.10).
#
# Not optional in practice, whatever the name suggests. STUN alone
# connects two peers only when at least one NAT is permissive; symmetric NAT
# and CGNAT, which is to say most mobile networks, need a relay in the
# middle. Without this service `/e2e/turn` answers 503 and calls fail for a
# large share of real users while messaging keeps working, which is a
# miserable thing to diagnose from a bug report, so it ships here rather
# than only in `docker-compose.federated.yml`.
#
# Host networking, deliberately. TURN allocates relay ports
# dynamically across the 49160-49200 range, and publishing 40 UDP ports
# through the Docker proxy costs a userland process per port. That also means
# it does NOT sit on the `internal` network and does not need to: it never
# talks to the relay. The two are bound only by the shared secret.
#
# Open 3478/udp, 3478/tcp and 49160-49200/udp on the host firewall. A
# closed range fails exactly like a wrong secret, the call connects, then
# nothing.
coturn:
image: coturn/coturn:4.6-alpine
# Opt in with `--profile calls`. It used to start unconditionally and
# hard-require the two TURN variables, so a relay that wanted no calls at
# all still could not come up without configuring them. A service nobody
# asked for should not be able to block the whole stack.
profiles: ["calls"]
restart: unless-stopped
network_mode: host
# coturn logs client IP:port (a reset TCP connection is logged with the
# peer's address at the default level). Left alone it writes every line
# twice: to stdout, and to a dated file under /var/tmp inside the
# container that is never rotated or deleted. `--log-file=stdout` below
# drops the files, and this caps what Docker keeps. The cap is by size,
# not by age, and a quiet relay takes a long time to fill 2 MB, so an
# operator who wants a time limit should ship the log somewhere that
# expires it (deploy/retention/README.md shows one way).
logging:
driver: json-file
options:
max-size: "1m"
max-file: "2"
command:
- "--no-cli"
# One stream, no files inside the container. Do not add `--verbose`
# outside a debugging session: it logs every allocation with its
# username and both addresses.
- "--log-file=stdout"
# `:-`, not `:?`, even though coturn needs these. Compose
# interpolates the WHOLE file before it filters by profile, so a `:?` here
# failed `up -d` for operators who had not enabled calls at all, the
# profile gates whether the service RUNS, not whether its variables are
# read. Left empty, coturn refuses to start and only somebody who asked
# for `--profile calls` ever sees it, which is where that failure belongs.
#
# Empty rather than a placeholder secret. A default value here would be
# a shared credential every install could mint TURN allocations with.
- "--realm=${FN_TURN_HOST:-}"
- "--fingerprint"
# The REST scheme: the relay HMACs an expiry into a credential coturn
# verifies from this same secret, so neither side stores a password.
# It must match FN_TURN_SECRET above EXACTLY or every relayed call fails
# authentication while STUN-only calls keep working, which reads as an
# intermittent bug rather than a configuration error.
- "--use-auth-secret"
- "--static-auth-secret=${FN_TURN_SECRET:-}"
- "--listening-port=3478"
- "--listening-ip=0.0.0.0"
- "--min-port=49160"
- "--max-port=49200"
# The machine's PUBLIC address. Behind NAT, coturn must know the
# address the outside world sees, or every allocation hands out a
# candidate nobody can reach and the failure looks like packet loss.
# Same reasoning as the two above: interpolation happens before profile
# filtering, so a hard requirement here breaks `up -d` for people who
# never asked for calls.
- "--external-ip=${FN_TURN_EXTERNAL_IP:-}"
- "--no-tls"
- "--no-dtls"
- "--no-multicast-peers"
- "--no-rfc5780"
# ---------------------------------------------------------------------------
# obfs4 bridge, an obfuscated way in for clients on networks that block or
# fingerprint TLS to this relay.
#
# A separate container, not part of the relay, and that is deliberate.
# obfs4 traffic is indistinguishable from random bytes: it carries no SNI and
# no headers, so Traefik cannot route it, that is the point. It
# therefore needs its own published port rather than sitting behind the
# existing ingress.
#
# Blocking the bridge must not block the relay. Bridges are meant to be
# burnable and replaceable; the relay is not. Running them in one process, or
# at one address, means a censor who identifies the bridge has taken out
# messaging for everyone in that country, including people who never used it.
# On a host with a single public IP that separation is only partial, so treat
# a bridge on the relay's own address as a stopgap, not the destination.
#
# Opt-in. Not started by default: `docker compose --profile obfs4 up -d`.
# A bridge nobody asked for is an open port nobody is watching.
obfs4bridge:
profiles: ["obfs4"]
# Built from a sibling checkout of flamenet-elligator2 (public, MIT).
# Clone it next to this one, or set FN_OBFS4_IMAGE to an image you can
# pull.
build:
context: ${FN_OBFS4_CONTEXT:-../flamenet-elligator2}
image: ${FN_OBFS4_IMAGE:-flamenet-obfs4bridge:prod}
restart: unless-stopped
# Published on every interface, unlike the relay. It has to be: a bridge
# that only loopback can reach helps nobody. What makes that safe is that
# the protocol answers no one who cannot complete a handshake.
ports:
- "${FN_OBFS4_PORT:-8443}:8443"
environment:
# 🔴 **The TLS front door, not the relay's HTTP port.** The apps send TLS
# for relay.flamenet.io through the tunnel, so the bytes coming out of
# the bridge must reach something that terminates that TLS: the edge
# proxy on 443. Pointed at `relay:8080` (the setting until October 2026)
# the obfs4 handshake succeeds, a plain-HTTP check through it returns
# 200, and no app session ever completes. Reference/bridgecheck in
# flamenet-elligator2 has a `-tls` mode that shows the difference.
FN_OBFS4_UPSTREAM: "${FN_OBFS4_UPSTREAM:-traefik:443}"
# 🔴 **Each client's own address, passed on.** Without this line the
# edge proxy sees the bridge's container address for every client, so
# the country rule, the rate limits and any ban apply to all bridge
# users at once or to none. With it, they apply per client, as for a
# direct connection. It REQUIRES the edge proxy to trust exactly this
# container's address for the PROXY protocol, and nothing else on the
# network (Traefik: entryPoints.websecure.proxyProtocol.trustedIPs set
# to FN_OBFS4_WEB_IP/32). Trust the whole network instead and any
# container on it can claim any address. Set it to 0 if your proxy is
# not configured for it, or it will drop every bridged connection.
FN_OBFS4_PROXY_PROTOCOL: "${FN_OBFS4_PROXY_PROTOCOL:-1}"
# Containers cannot know their own public address, so the bridge line
# says `<this-host>` until this is set. Set it, or what you hand clients
# will not be dialable.
FN_OBFS4_ADVERTISE: "${FN_OBFS4_ADVERTISE:-}"
volumes:
# Escrow-critical. This holds the bridge's identity key. Clients pin
# it through the `cert=` in their bridge line, so losing this volume
# orphans every client that ever received one. The process refuses to
# start rather than silently re-key, so an unmounted volume fails loudly
# instead of quietly breaking reconnects.
- obfs4-state:/var/lib/obfs4
# No Linux capabilities beyond what the process needs, and no privilege
# gain through setuid binaries (security audit 2026-10-15).
security_opt: ["no-new-privileges:true"]
cap_drop: [ALL]
# Only on the web network: the bridge forwards to traefik:443 and needs
# nothing on internal, where the database and Redis live.
networks:
# On the edge proxy's network to reach 443, at a fixed address because
# the proxy's PROXY-protocol trust is pinned to it. An unpinned address
# can be handed to another container after a restart, and that
# container would inherit the trust.
web:
ipv4_address: "${FN_OBFS4_WEB_IP:-172.18.255.240}"
depends_on:
- relay
networks:
web:
external: true
# No route off the box (2026-10-15). The database and Redis need none; the
# relay reaches S3 and APNs through web.
internal:
driver: bridge
internal: true
volumes:
relay-db:
relay-attachments:
obfs4-state: