From 250a3bb63f1f52a3039141aa0907f6f97cba573d Mon Sep 17 00:00:00 2001 From: Paul Hitt Date: Mon, 31 Aug 2026 13:48:31 -0400 Subject: [PATCH 1/3] Run the quick start we publish, and it did not work MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Nothing in this repository ever ran the instructions it hands to self-hosters. A job that does found two faults on its first real run, on top of the one it was written for. 🔴 The documented password broke the relay about three times in four. SELF-HOSTING.md said `openssl rand -base64 32` for POSTGRES_PASSWORD, and that password is interpolated into `postgres://flamenet:@db:5432/flamenet` unencoded. Base64 emits `/` and `+` 72.9% of the time, measured; a `/` ends the URL authority and the relay dies with Fatal error: NSURLErrorDomain Code=-1000 postgres://flamenet:ycP3Ecow1FjhKoOszG9X10u/xjqayh/yPAg5Z/Rku5Y=@db:5432/flamenet which reads like the database is unreachable rather than like the password needs escaping. Production has never hit it because its password is hex — generated some way other than the documented one. ⚠️ The project already knew. `.env.example` carries a paragraph explaining exactly this and saying to use `-hex`. The published quick start disagreed with the file it tells people to copy, and the published one is the one people follow. 🔴 And `.env.example` itself was corrupted, in 8791321. A block of the FN_HOST_PORT comment had been spliced into the middle of the line that pairs `openssl rand -hex 32` with POSTGRES_PASSWORD, destroying the only correct instruction in the tree, and leaving a bare `POSTGRES_PASSWORD` line with no `=`. Restored from d019034. The check boots rather than validating YAML, because both of these failures happen at `up`: `docker compose config` renders that file happily with no `web` network and with a password that will not parse. It asserts the landing page as well as /health — the compose file's own header promises a working install explains itself rather than answering 404, and a missing Resources copy would leave health green. ⚠️ Its own network name and compose project, never `web` — the deployment box has a real `web` network and this script deletes what it creates. Verified on the box, which is the CI runner's architecture: base64 reproduces the crash, hex boots and serves the page. Closes #4 Co-Authored-By: Claude Opus 4.8 --- .env.example | 14 ++--- .gitlab-ci.yml | 17 +++++ Scripts/selfhost-smoke.sh | 128 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 152 insertions(+), 7 deletions(-) create mode 100755 Scripts/selfhost-smoke.sh diff --git a/.env.example b/.env.example index 44e5ea8..9a2e64b 100644 --- a/.env.example +++ b/.env.example @@ -1,12 +1,6 @@ # Copy to .env and fill in. Generate the three secrets with: # -# openssl rand -hex 32 # # Host port the relay is published on, on 127.0.0.1 only. Change this rather -# than editing docker-compose.yml — 8080 is frequently already taken on a -# machine that self-hosts other things, and editing the compose file makes the -# next `git pull` conflict. -FN_HOST_PORT=8080 - -POSTGRES_PASSWORD +# openssl rand -hex 32 # POSTGRES_PASSWORD # openssl rand -base64 32 # FN_AT_REST_SECRET # # POSTGRES_PASSWORD is hex, not base64, and that is not a style choice: it is @@ -22,6 +16,12 @@ POSTGRES_PASSWORD # Keep issuer.pem. Losing it invalidates every token you have issued; leaking it # lets anyone mint tokens this relay will accept. +# Host port the relay is published on, on 127.0.0.1 only. Change this rather +# than editing docker-compose.yml — 8080 is frequently already taken on a +# machine that self-hosts other things, and editing the compose file makes the +# next `git pull` conflict. +FN_HOST_PORT=8080 + POSTGRES_PASSWORD= FN_AT_REST_SECRET= FN_ISSUER_PRIVATE_KEY= diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index d6d334e..1a306af 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -170,6 +170,23 @@ conformance: timeout: 40 minutes # What gets a package wrong is missing files, not failing logic. +# 🔴 Nothing in this repository ran the instructions it publishes, so when the +# quick start broke it stayed broken — three faults at once, and the last one +# survived a day of nobody noticing. The people it fails are self-hosters, at +# the moment they are deciding whether this project works. +# +# ⚠️ It BOOTS rather than validating YAML. `docker compose config` renders that +# file happily with no `web` network in existence — the external-network check +# happens at `up`, and so does the database URL the password goes into. A job +# that only linted would have passed throughout both outages it exists to catch. +selfhost: + <<: *rules + stage: test + tags: [vps] + script: + - Scripts/selfhost-smoke.sh + timeout: 15m + # 🔴 The generated pages are what people actually read. A source can be fixed # and the page left stale, and nothing says so — /self-hosting was served for a # day telling self-hosters to run `docker compose up -d` without the diff --git a/Scripts/selfhost-smoke.sh b/Scripts/selfhost-smoke.sh new file mode 100755 index 0000000..c33e218 --- /dev/null +++ b/Scripts/selfhost-smoke.sh @@ -0,0 +1,128 @@ +#!/usr/bin/env bash +# Run the published self-hosting quick start, exactly as SELF-HOSTING.md gives it. +# +# 🔴 **The quick start broke, silently, and stayed broken for a day.** Three +# separate faults at once — and the one that survived longest was a missing +# `docker network create web`, which fails only at `up` time, after the images +# have been pulled. Nothing in this repository ran those instructions, so +# nothing said so. The people it failed for were self-hosters, at the exact +# moment they were deciding whether this project works. +# +# ⚠️ **It has to actually BOOT.** `docker compose config` renders that compose +# file happily with no `web` network in existence: the external-network check +# happens at `up`. A job that only validated the YAML would have passed +# throughout the outage it exists to prevent. +# +# Scripts/selfhost-smoke.sh # published image, the documented path +# Scripts/selfhost-smoke.sh --build # build from source, the ARM path +# +# Leaves nothing behind: a scratch directory, its own compose project name, and +# a trap that tears down volumes as well as containers. +set -euo pipefail + +here="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +mode="${1:-}" +project="selfhost-smoke-$$" +# ⚠️ Its own network name, not `web`. A shared name would collide with a real +# proxy network on a machine that has one — and this script deletes it at the +# end, which on the production box would be a very bad afternoon. +network="${project}-web" +work="$(mktemp -d)" + +cleanup() { + docker compose -p "$project" --project-directory "$work" \ + -f "$work/docker-compose.yml" -f "$work/docker-compose.override.yml" \ + down -v --remove-orphans >/dev/null 2>&1 || true + docker network rm "$network" >/dev/null 2>&1 || true + rm -rf "$work" +} +trap cleanup EXIT + +echo "SELFHOST: laying out a fresh checkout the way a self-hoster would" +cp "$here/docker-compose.yml" "$work/docker-compose.yml" +cp "$here/docker-compose.build.yml" "$work/docker-compose.build.yml" 2>/dev/null || true +cp "$here/.env.example" "$work/.env" +# The build path needs the sources the Dockerfile copies. +if [ "$mode" = "--build" ]; then + cp -R "$here/Sources" "$here/Tests" "$here/Resources" "$here/Public" \ + "$here/Package.swift" "$here/Package.resolved" "$here/Dockerfile" "$work/" 2>/dev/null || true +fi + +echo "SELFHOST: generating the secrets .env asks for" +# The same four commands the documentation prints. Generated rather than +# hardcoded, so a change to what the compose file demands shows up here. +pem="$work/issuer.pem" +openssl genpkey -algorithm ed25519 -out "$pem" 2>/dev/null +set_env() { + # ⚠️ In-place edit that tolerates an empty value on the right of `=`, which is + # what .env.example ships. Anchored to the line start so a variable whose name + # is a suffix of another one cannot be hit. + python3 - "$work/.env" "$1" "$2" <<'PY' +import sys, re +path, key, val = sys.argv[1], sys.argv[2], sys.argv[3] +text = open(path, encoding='utf-8').read() +new, n = re.subn(rf'^{re.escape(key)}=.*$', f'{key}={val}', text, count=1, flags=re.M) +if n != 1: + sys.exit(f'{key} is not in .env.example — the documented steps have drifted') +open(path, 'w', encoding='utf-8').write(new) +PY +} +# 🔴 **hex, not base64, and this is the documented command for a reason.** +# `openssl rand -base64 32` emits `/` and `+` about 73% of the time, and the +# password goes into `postgres://user:PASSWORD@db/...` unencoded — a `/` ends +# the authority and the relay dies with `NSURLErrorDomain Code=-1000`, which +# names the URL and says nothing about passwords. This script generates what +# SELF-HOSTING.md tells people to generate; if that drifts back to base64, this +# fails again and says why. +set_env POSTGRES_PASSWORD "$(openssl rand -hex 32)" +set_env FN_AT_REST_SECRET "$(openssl rand -base64 32 | tr -d '\n')" +set_env FN_ISSUER_PRIVATE_KEY "$(openssl pkey -in "$pem" -outform DER 2>/dev/null | tail -c 32 | base64 | tr -d '\n')" +set_env FN_ISSUER_PUBLIC_KEY "$(openssl pkey -in "$pem" -pubout -outform DER 2>/dev/null | tail -c 32 | base64 | tr -d '\n')" + +# A port nobody else is on, so this cannot collide with a relay already running +# on the machine — which on the deployment box is the production one. +port=18973 +set_env FN_HOST_PORT "$port" + +echo "SELFHOST: docker network create (the step that was missing)" +docker network create "$network" >/dev/null +# The compose file names the external network `web`; point it at ours for the +# duration without editing the file a self-hoster would actually use. +cat > "$work/docker-compose.override.yml" < Date: Mon, 31 Aug 2026 14:47:26 -0400 Subject: [PATCH 2/3] The self-hosting check cannot run on this runner MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The job I added failed with `env: can't execute bash`, and the reason is worse than a missing package. This project's runner is a docker executor with `privileged = false` and `volumes = ["/cache"]`, so the job gets an alpine container with no bash, no docker CLI, and no path to a daemon. I wrote and tested the script against the host shell over ssh, which is not what CI is. Wiring it up needs a runner change — privileged, a socket mount, or a shell executor — on a machine that also serves production. That is not a change to make on the way past, so the job is removed rather than left failing or, worse, made to skip. A job that always skips is the pattern I spent this afternoon removing from the Android suite; adding one here would be indefensible. The script stays, and says plainly where it has to run and why it is not automatic. It has earned that much: on its first two runs it found the missing `docker network create web` and a documented password that broke the relay about three times in four. ⚠️ So this closes #4 only in the sense that the check now exists and works. "Nothing boots the quick start in CI" is still true, and now has a named blocker rather than an absence. --- .gitlab-ci.yml | 17 ----------------- Scripts/selfhost-smoke.sh | 17 +++++++++++++++++ 2 files changed, 17 insertions(+), 17 deletions(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 1a306af..d6d334e 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -170,23 +170,6 @@ conformance: timeout: 40 minutes # What gets a package wrong is missing files, not failing logic. -# 🔴 Nothing in this repository ran the instructions it publishes, so when the -# quick start broke it stayed broken — three faults at once, and the last one -# survived a day of nobody noticing. The people it fails are self-hosters, at -# the moment they are deciding whether this project works. -# -# ⚠️ It BOOTS rather than validating YAML. `docker compose config` renders that -# file happily with no `web` network in existence — the external-network check -# happens at `up`, and so does the database URL the password goes into. A job -# that only linted would have passed throughout both outages it exists to catch. -selfhost: - <<: *rules - stage: test - tags: [vps] - script: - - Scripts/selfhost-smoke.sh - timeout: 15m - # 🔴 The generated pages are what people actually read. A source can be fixed # and the page left stale, and nothing says so — /self-hosting was served for a # day telling self-hosters to run `docker compose up -d` without the diff --git a/Scripts/selfhost-smoke.sh b/Scripts/selfhost-smoke.sh index c33e218..bf73a15 100755 --- a/Scripts/selfhost-smoke.sh +++ b/Scripts/selfhost-smoke.sh @@ -16,6 +16,23 @@ # Scripts/selfhost-smoke.sh # published image, the documented path # Scripts/selfhost-smoke.sh --build # build from source, the ARM path # +# 🔴 **Not wired into CI, and not for want of trying.** This project's runner is +# a docker executor with `privileged = false` and no socket mounted, so a job +# here gets an alpine container with no bash, no docker CLI and no way to reach +# a daemon — `env: can't execute bash` is as far as it got. Running it in CI +# needs a runner change (privileged, a socket mount, or a shell executor), which +# is a decision about a machine that also serves production. +# +# ⚠️ Until then this is a RELEASE-TIME check somebody has to run, on any machine +# with docker: +# +# ssh "cd /srv/prod/messenger/src && ./Scripts/selfhost-smoke.sh" +# +# A check that only runs when remembered is weaker than one that cannot be +# forgotten. It is still worth having: on its first two runs it found the +# missing `docker network create web` AND a documented password that broke the +# relay three times in four. +# # Leaves nothing behind: a scratch directory, its own compose project name, and # a trap that tears down volumes as well as containers. set -euo pipefail -- 2.54.0 From 10ef95199ef820273ab86493b3cef7cd3320e08a Mon Sep 17 00:00:00 2001 From: Paul Hitt Date: Mon, 31 Aug 2026 14:48:19 -0400 Subject: [PATCH 3/3] Regenerate /self-hosting: the password command people follow MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follows flamenet-e2e 40fd2f5. The published page still tells self-hosters to generate POSTGRES_PASSWORD with `openssl rand -base64 32`, which breaks the relay about three times in four, so the correction is not live until this ships. The drift guard caught this unprompted, for the second time today: self-hosting.leaf from flamenet-e2e/SELF-HOSTING.md (a4ed8fd6… != 0f36788a…) Which is the whole point of it. The source was fixed in another repository and the page it feeds would otherwise have stayed stale and served — the exact failure that left the missing `docker network create web` live for a day. --- Resources/Views/self-hosting.leaf | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/Resources/Views/self-hosting.leaf b/Resources/Views/self-hosting.leaf index 8e52112..261d299 100644 --- a/Resources/Views/self-hosting.leaf +++ b/Resources/Views/self-hosting.leaf @@ -3,7 +3,7 @@ + source-sha256: 0f36788aea97abd7 -->

Self-hosting

@@ -52,12 +52,24 @@ told.

cd flamenet-relay cp .env.example .env

Generate the three secrets .env asks for:

-
openssl rand -base64 32                    #("#") POSTGRES_PASSWORD
+
openssl rand -hex 32                       #("#") POSTGRES_PASSWORD
 openssl rand -base64 32                    #("#") FN_AT_REST_SECRET
 
 openssl genpkey -algorithm ed25519 -out issuer.pem
 openssl pkey -in issuer.pem -outform DER | tail -c 32 | base64          #("#") FN_ISSUER_PRIVATE_KEY
 openssl pkey -in issuer.pem -pubout -outform DER | tail -c 32 | base64  #("#") FN_ISSUER_PUBLIC_KEY

+
🔴 -hex for the database password, not
+-base64, and this one is not cosmetic. It is
+interpolated into
+postgres://flamenet:<password>@db:5432/flamenet,
+where base64's /, + and = are all
+meaningful. A password containing / makes that URL invalid
+and the relay dies at start-up with a bare Code=-1000 and
+the connection string echoed back at you — which reads like the database
+is unreachable rather than like the password needs escaping.
+openssl rand -base64 32 produces one of those roughly
+three times in four, so following the old instructions
+usually failed.

 
Then create the proxy network the compose file attaches to, and
 start:

 
docker network create web
-- 
2.54.0