The relay connects to Postgres as a superuser — drop it to a scoped role #15

New Issue

2026-09-01T22:59:05Z

paulhitt commented

2026-09-01 22:59:05 +00:00

(Migrated from gitlab.com)

The relay connects to Postgres as a superuser. Measured on production
2026-09-01:

 rolname  | rolsuper | rolbypassrls
----------+----------+--------------
 flamenet | t        | t

25 tables in public, 2 databases on that cluster. Zero RLS, zero policies, zero
triggers on key_log.

Why RLS is not the answer here

Worth stating so nobody adds it and thinks the job is done: a superuser bypasses RLS
unconditionally, and rolbypassrls is set as well. More fundamentally, the relay is the
only client of this database — there is no second tenant for a policy to separate it from.
The key log's integrity is cryptographic and enforced by CLIENTS: each leaf is a hash of
fixed fields, the head is a signed Merkle root, clients pin that head on first sight and
demand inclusion proofs. Two validly-signed heads of the same size with different roots
are evidence the operator cannot retract. Tampering is designed to be detected, not
prevented at the server — the threat model already assumes a hostile operator.

What the actual exposure is

Not the operator — them owning the box is assumed. It is blast radius from a flaw in the
relay itself. Any SQL injection, deserialization bug or path that reaches a query with
attacker-influenced input becomes, today, full cluster control:

read and forge key_log rows (breaking the one guarantee clients verify)
read every row of users, devices, app_credentials, push_tokens
DROP anything, including the other database on the cluster
COPY … FROM PROGRAM — superuser can execute shell commands on the database host

That last one turns a SQL flaw into remote code execution, which is the reason this is
worth doing regardless of how confident anyone is in the query layer.

Suggested change

A dedicated non-superuser role for the relay:

NOSUPERUSER NOCREATEDB NOCREATEROLE NOBYPASSRLS
GRANT SELECT, INSERT, UPDATE, DELETE on the tables it actually writes, and USAGE, SELECT on sequences
No DROP, no TRUNCATE, and no rights on the other database
Consider key_log as INSERT-only — no UPDATE or DELETE grant at all. It is
append-only by design, and a grant that cannot express that is a grant worth removing.
This is defence in depth, not a replacement for the Merkle proof.

⚠️ This is a production credential change, and getting it wrong takes the relay down
with permission errors that will read as application bugs. Do it behind
Scripts/staging-gate.sh like any other deploy, and check the migration path — Fluent
runs DDL at boot, so the new role needs whatever CREATE/ALTER rights the migrations
require, or a separate owner role must run them.

⚠️ The same question applies to the other services on this cluster; this ticket covers the
messenger relay only.

Filed during the 2026-09-01 code freeze. Not actioned.

**The relay connects to Postgres as a superuser.** Measured on production 2026-09-01: ``` rolname | rolsuper | rolbypassrls ----------+----------+-------------- flamenet | t | t ``` 25 tables in `public`, 2 databases on that cluster. Zero RLS, zero policies, zero triggers on `key_log`. ## Why RLS is not the answer here Worth stating so nobody adds it and thinks the job is done: **a superuser bypasses RLS unconditionally**, and `rolbypassrls` is set as well. More fundamentally, the relay is the only client of this database — there is no second tenant for a policy to separate it from. The key log's integrity is cryptographic and enforced by CLIENTS: each leaf is a hash of fixed fields, the head is a signed Merkle root, clients pin that head on first sight and demand inclusion proofs. Two validly-signed heads of the same size with different roots are evidence the operator cannot retract. Tampering is designed to be **detected**, not prevented at the server — the threat model already assumes a hostile operator. ## What the actual exposure is Not the operator — them owning the box is assumed. It is **blast radius from a flaw in the relay itself**. Any SQL injection, deserialization bug or path that reaches a query with attacker-influenced input becomes, today, full cluster control: - read and forge `key_log` rows (breaking the one guarantee clients verify) - read every row of `users`, `devices`, `app_credentials`, `push_tokens` - `DROP` anything, including the other database on the cluster - `COPY … FROM PROGRAM` — superuser can execute shell commands on the database host That last one turns a SQL flaw into remote code execution, which is the reason this is worth doing regardless of how confident anyone is in the query layer. ## Suggested change A dedicated non-superuser role for the relay: - `NOSUPERUSER NOCREATEDB NOCREATEROLE NOBYPASSRLS` - `GRANT SELECT, INSERT, UPDATE, DELETE` on the tables it actually writes, and `USAGE, SELECT` on sequences - **No `DROP`, no `TRUNCATE`**, and no rights on the other database - Consider `key_log` as `INSERT`-only — no `UPDATE` or `DELETE` grant at all. It is append-only by design, and a grant that cannot express that is a grant worth removing. This is defence in depth, not a replacement for the Merkle proof. ⚠️ **This is a production credential change**, and getting it wrong takes the relay down with permission errors that will read as application bugs. Do it behind `Scripts/staging-gate.sh` like any other deploy, and check the migration path — Fluent runs DDL at boot, so the new role needs whatever `CREATE`/`ALTER` rights the migrations require, or a separate owner role must run them. ⚠️ The same question applies to the other services on this cluster; this ticket covers the messenger relay only. Filed during the 2026-09-01 code freeze. Not actioned.

paulhitt commented

2026-09-01 23:00:11 +00:00

(Migrated from gitlab.com)

mentioned in issue #16

paulhitt commented

2026-09-01 23:01:53 +00:00

(Migrated from gitlab.com)

mentioned in issue #17

paulhitt commented

2026-09-01 23:11:46 +00:00

(Migrated from gitlab.com)

mentioned in issue #21

paulhitt commented

2026-09-04 21:46:18 +00:00

(Migrated from gitlab.com)

Done and applied to production 2026-09-04. Written up in docs/DATABASE.md (!45).

🔴 One thing the issue could not have known: the bootstrap superuser cannot be demoted. Postgres refuses outright —

ERROR:  permission denied to alter role
DETAIL:  The bootstrap superuser must have the SUPERUSER attribute.

So this is a new scoped role (relayapp) and a switchover, not a demotion. flamenet still exists and is still the superuser; the relay stopped being it.

⚠️ The role OWNS its tables rather than holding only the DML grants the issue suggested. Fluent migrates at boot, so a role that cannot create a table or an index would fail the next deploy carrying a migration — and that failure presents as an application bug, not a missing grant. Ownership gives DDL on its own tables and nothing else, which still closes everything that mattered.

⚠️ Cross-database access needed a separate step. Postgres grants CONNECT on every database to PUBLIC, so the new role could still open postgres and read pg_catalog until REVOKE CONNECT ON DATABASE postgres FROM PUBLIC. Verified with has_database_privilege, not by reading datacl — a revoke from PUBLIC leaves any direct role grant in place.

Measured on production after the change:

	before	after
`COPY … TO PROGRAM` (RCE on the DB host)	allowed	refused
`CREATE ROLE`	allowed	refused
Connect to the other database	allowed	refused
Bypass RLS	yes	no
DDL on its own tables (migrations)	yes	preserved

Rehearsed first against a throwaway holding the real schema and the real _fluent_migrations rows — no user data. The first rehearsal failed with relation "devices" already exists, which looks like a permissions problem and is not: --schema-only carries that table but none of its rows, so Fluent tried to create everything again.

Not done, deliberately: key_log is not INSERT-only. Ownership carries UPDATE/DELETE implicitly, and splitting into a migration role plus a restricted runtime role is a second change with its own failure mode. Recorded in docs/DATABASE.md as worth doing.

**Done and applied to production 2026-09-04.** Written up in `docs/DATABASE.md` (!45). 🔴 **One thing the issue could not have known: the bootstrap superuser cannot be demoted.** Postgres refuses outright — ERROR: permission denied to alter role DETAIL: The bootstrap superuser must have the SUPERUSER attribute. So this is a new scoped role (`relayapp`) and a switchover, not a demotion. `flamenet` still exists and is still the superuser; the relay stopped being it. ⚠️ **The role OWNS its tables rather than holding only the DML grants the issue suggested.** Fluent migrates at boot, so a role that cannot create a table or an index would fail the next deploy carrying a migration — and that failure presents as an application bug, not a missing grant. Ownership gives DDL on its own tables and nothing else, which still closes everything that mattered. ⚠️ **Cross-database access needed a separate step.** Postgres grants `CONNECT` on every database to `PUBLIC`, so the new role could still open `postgres` and read `pg_catalog` until `REVOKE CONNECT ON DATABASE postgres FROM PUBLIC`. Verified with `has_database_privilege`, not by reading `datacl` — a revoke from PUBLIC leaves any direct role grant in place. Measured on production after the change: | | before | after | |---|---|---| | `COPY … TO PROGRAM` (RCE on the DB host) | allowed | **refused** | | `CREATE ROLE` | allowed | **refused** | | Connect to the other database | allowed | **refused** | | Bypass RLS | yes | no | | DDL on its own tables (migrations) | yes | **preserved** | Rehearsed first against a throwaway holding the real schema **and the real `_fluent_migrations` rows** — no user data. The first rehearsal failed with `relation "devices" already exists`, which looks like a permissions problem and is not: `--schema-only` carries that table but none of its rows, so Fluent tried to create everything again. **Not done, deliberately:** `key_log` is not INSERT-only. Ownership carries UPDATE/DELETE implicitly, and splitting into a migration role plus a restricted runtime role is a second change with its own failure mode. Recorded in `docs/DATABASE.md` as worth doing.

paulhitt (Migrated from gitlab.com) closed this issue

2026-09-04 21:46:19 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: extant2000/flamenet-relay#15