Pipelines queue over an hour behind the shared OVH runner #6

New Issue

2026-08-31T15:10:16Z

paulhitt commented

2026-08-31 15:10:16 +00:00

(Migrated from gitlab.com)

The main pipeline for c09637eb was created at 13:49 and had not started by
15:05 — over an hour queued, with every job pending and no runner assigned:

image-build    created   tags=['vps'] queued=0s
complete       pending   tags=['vps'] queued=400s
conformance    pending   tags=['vps'] queued=400s
test           pending   tags=['vps'] queued=400s
build          pending   tags=['vps'] queued=400s

All jobs are tagged vps, so they need the OVH runner, which was at load
19-28 running CI for other projects alongside production services.

Consequences seen today

Two merge requests were merged on local test evidence rather than on CI,
because waiting was not viable. That is a real weakening of the merge gate.
The staging gate failed a good build on a 30s postgres timeout under the same
load (fixed in 2e1fd8e by waiting 90s).

Worth considering

More runner capacity, or a second runner for this project
Moving the cheap jobs (complete is test -f checks on alpine) off the
vps tag so they are not queued behind Swift builds
interruptible: true so superseded pipelines free their slot — several
pipelines today were cancelled by newer pushes only after occupying the queue

Done when

A pipeline on main reliably starts within a few minutes
Or: the constraint is accepted and written down, so merging on local
evidence is a recorded decision rather than an improvisation

The `main` pipeline for `c09637eb` was created at 13:49 and had not started by 15:05 — over an hour queued, with every job `pending` and no runner assigned: ``` image-build created tags=['vps'] queued=0s complete pending tags=['vps'] queued=400s conformance pending tags=['vps'] queued=400s test pending tags=['vps'] queued=400s build pending tags=['vps'] queued=400s ``` All jobs are tagged `vps`, so they need the OVH runner, which was at load 19-28 running CI for other projects alongside production services. ### Consequences seen today - Two merge requests were merged on local test evidence rather than on CI, because waiting was not viable. That is a real weakening of the merge gate. - The staging gate failed a good build on a 30s postgres timeout under the same load (fixed in `2e1fd8e` by waiting 90s). ### Worth considering - More runner capacity, or a second runner for this project - Moving the cheap jobs (`complete` is `test -f` checks on alpine) off the `vps` tag so they are not queued behind Swift builds - `interruptible: true` so superseded pipelines free their slot — several pipelines today were cancelled by newer pushes only after occupying the queue ### Done when - [ ] A pipeline on `main` reliably starts within a few minutes - [ ] Or: the constraint is accepted and written down, so merging on local evidence is a recorded decision rather than an improvisation

paulhitt commented

2026-08-31 17:57:27 +00:00

(Migrated from gitlab.com)

Diagnosed. It is not the box being busy — it is a `limit = 1`.

/etc/gitlab-runner/config.toml
concurrent = 5                      # global, across all 13 runners
  name = "ovh-flamenet"   limit = 1     # ← this project
  name = "ovh-flamenetmessenger"  limit = 1

So this pipeline runs strictly one job at a time. build, test and
conformance are Swift; complete and doc-pages are alpine containers that
finish in seconds. The alpine ones queue behind a ten-minute Swift build for no
reason other than the limit.

That is the whole of the hour: not contention for the machine, but self-imposed
serialisation.

⚠️ My earlier suggestion on this ticket was wrong

I wrote that the cheap jobs should move off the vps tag onto shared runners.
They cannot. The default: block says why, and I should have read it first:

🔴 vps, not the shared runners. Every job in this pipeline failed with
ci_quota_exceeded on first run: GitLab.com-s shared minutes are spent […]
⚠️ It is registered with run_untagged = false, so a job WITHOUT this tag is
not picked up at all — it sits pending forever rather than failing.

So untagging would have made it worse, silently.

The actual change

   name = "ovh-flamenet"
-  limit = 1
+  limit = 3

🔴 The global concurrent = 5 is unchanged, so the worst-case load on the box
is unchanged. This does not let more jobs run overall — it stops this project
serialising itself inside the budget that already exists.

⚠️ But it does reallocate shared capacity. Thirteen runners share those five
slots, and letting flamenet hold three means the hitthosting projects can hold at
most two while a flamenet pipeline is running. That is a scheduling decision
across systems, on the machine that also runs production, so it is not mine to
make unilaterally — say the word and it is a one-line edit plus a runner reload.

limit = 2 is the cautious version: it halves wall-clock here and leaves three
slots for everything else.

### Diagnosed. It is not the box being busy — it is a `limit = 1`. ``` /etc/gitlab-runner/config.toml concurrent = 5 # global, across all 13 runners name = "ovh-flamenet" limit = 1 # ← this project name = "ovh-flamenetmessenger" limit = 1 ``` So this pipeline runs **strictly one job at a time**. `build`, `test` and `conformance` are Swift; `complete` and `doc-pages` are alpine containers that finish in seconds. The alpine ones queue behind a ten-minute Swift build for no reason other than the limit. That is the whole of the hour: not contention for the machine, but self-imposed serialisation. ### ⚠️ My earlier suggestion on this ticket was wrong I wrote that the cheap jobs should move off the `vps` tag onto shared runners. They cannot. The `default:` block says why, and I should have read it first: > 🔴 **`vps`, not the shared runners.** Every job in this pipeline failed with > `ci_quota_exceeded` on first run: GitLab.com-s shared minutes are spent […] > ⚠️ It is registered with `run_untagged = false`, so a job WITHOUT this tag is > not picked up at all — it sits pending forever rather than failing. So untagging would have made it worse, silently. ### The actual change ```diff name = "ovh-flamenet" - limit = 1 + limit = 3 ``` 🔴 **The global `concurrent = 5` is unchanged, so the worst-case load on the box is unchanged.** This does not let more jobs run overall — it stops this project serialising itself inside the budget that already exists. ⚠️ **But it does reallocate shared capacity.** Thirteen runners share those five slots, and letting flamenet hold three means the hitthosting projects can hold at most two while a flamenet pipeline is running. That is a scheduling decision across systems, on the machine that also runs production, so it is not mine to make unilaterally — say the word and it is a one-line edit plus a runner reload. `limit = 2` is the cautious version: it halves wall-clock here and leaves three slots for everything else.

paulhitt commented

2026-08-31 18:47:41 +00:00

(Migrated from gitlab.com)

mentioned in merge request !8

paulhitt commented

2026-08-31 19:23:53 +00:00

(Migrated from gitlab.com)

mentioned in issue #10

paulhitt commented

2026-08-31 19:32:11 +00:00

(Migrated from gitlab.com)

mentioned in merge request !9

paulhitt commented

2026-09-04 23:12:56 +00:00

(Migrated from gitlab.com)

Live evidence, 2026-09-04. !52 sat with all six jobs pending — not slow, not queued behind its own stages, simply unclaimed by any runner for over ten minutes.

⚠️ And this is not just a delay; it cost correctness today. Because pipelines never finish, I merged roughly twenty MRs with immediate merge rather than merge-when-pipeline-succeeds — and the pipelines that did run were reporting two real failures the whole time:

doc-pages was failing on the api.leaf drift I later found by hand;
test was failing because the suite did not compile on Linux at all.

The guard worked and was overridden, and the reason it was overridden is this ticket. A check that takes longer than the work it gates is a check people learn to merge past.

Also visible in the same window: main's pipelines are cancelled one after another by merge churn, so even a green MR pipeline is followed by a main pipeline that never completes.

!52 is now set to merge-when-pipeline-succeeds rather than merged by hand, and will land when a runner picks it up. Its fix was verified directly instead — built and run in the actual CI image (swift:6.1-noble, as root): 659 tests pass.

**Live evidence, 2026-09-04.** !52 sat with **all six jobs `pending`** — not slow, not queued behind its own stages, simply unclaimed by any runner for over ten minutes. ⚠️ **And this is not just a delay; it cost correctness today.** Because pipelines never finish, I merged roughly twenty MRs with immediate merge rather than merge-when-pipeline-succeeds — and the pipelines that *did* run were reporting two real failures the whole time: - `doc-pages` was failing on the `api.leaf` drift I later found by hand; - `test` was failing because the suite did not compile on Linux at all. **The guard worked and was overridden**, and the reason it was overridden is this ticket. A check that takes longer than the work it gates is a check people learn to merge past. Also visible in the same window: `main`'s pipelines are **cancelled** one after another by merge churn, so even a green MR pipeline is followed by a main pipeline that never completes. !52 is now set to merge-when-pipeline-succeeds rather than merged by hand, and will land when a runner picks it up. Its fix was verified directly instead — built and run in the actual CI image (`swift:6.1-noble`, as root): 659 tests pass.

paulhitt commented

2026-09-05 09:21:45 +00:00

(Migrated from gitlab.com)

Correction: last night's stall was NOT this ticket

I attributed six stuck MRs to runner starvation and told the owner so. That was wrong, and the correction matters because it points somewhere else.

Measured 2026-09-05: 0 pipelines pending, 0 running. The runner was idle. What was actually happening is that main had been red since 00:20 — every pipeline since !52 failing the doc-pages job — and a red main blocks every MR queued behind it. Six sat on merge_when_pipeline_succeeds waiting for a green that could not arrive.

Root cause is a cross-repo coupling, now filed as relay#56. Fixed for the moment by relay!59.

⚠️ What made this easy to misdiagnose is worth writing down. The symptom of "starved runner" and the symptom of "red main" are identical from the MR list: can_be_merged, auto-merge armed, nothing happening. I had a ready explanation from earlier in the day and stopped looking. The tell was one API call away — pipelines?status=pending returning zero — and I did not make it until the next morning.

This ticket is still real. relay!52 did queue behind a genuinely busy shared runner earlier on 2026-09-04, and the oversubscription is unchanged. But it is not the thing that cost last night, and it should not absorb the credit for relay#56's failure.

## Correction: last night's stall was NOT this ticket I attributed six stuck MRs to runner starvation and told the owner so. That was wrong, and the correction matters because it points somewhere else. Measured 2026-09-05: **0 pipelines pending, 0 running.** The runner was idle. What was actually happening is that `main` had been red since 00:20 — every pipeline since !52 failing the `doc-pages` job — and a red main blocks every MR queued behind it. Six sat on `merge_when_pipeline_succeeds` waiting for a green that could not arrive. Root cause is a cross-repo coupling, now filed as relay#56. Fixed for the moment by relay!59. ⚠️ **What made this easy to misdiagnose is worth writing down.** The symptom of "starved runner" and the symptom of "red main" are identical from the MR list: `can_be_merged`, auto-merge armed, nothing happening. I had a ready explanation from earlier in the day and stopped looking. The tell was one API call away — `pipelines?status=pending` returning zero — and I did not make it until the next morning. **This ticket is still real.** relay!52 did queue behind a genuinely busy shared runner earlier on 2026-09-04, and the oversubscription is unchanged. But it is not the thing that cost last night, and it should not absorb the credit for relay#56's failure.

paulhitt commented

2026-09-08 12:50:39 +00:00

(Migrated from gitlab.com)

mentioned in merge request !95

paulhitt commented

2026-09-08 17:13:15 +00:00

(Migrated from gitlab.com)

2026-09-08: this now produces FALSE FAILURES, not just slow ones

!96 changes fifteen lines of shell and nothing else — the Swift tree is byte-identical to main. Its test job failed:

status: failed   failure_reason: script_failure   duration: 891s

The same commit runs green locally in 84 seconds. 891s is more than ten times that, and the job's log blew through GitLab's 4 MB cap and was truncated, so the actual failure was not even visible. The box was at load 22–46 at the time, with the staging gate compiling Swift alongside three CI jobs.

That is the part worth escalating: a queue that is merely slow costs patience. A queue that turns green code red costs trust in the gate, and the natural response — merging on local evidence — is exactly the weakening this issue was opened about.

Measured today

Peak load 46 on 8 cores; ; jobs waiting 47 minutes.
Pipelines for already-merged MRs keep running. !85, !89, !90, !91 and !92 were all merged with their pipelines still queued or executing, holding the runner against work that mattered. GitLab does not cancel these; I cancelled five by hand and load fell 36 → 13.

Done since

!95 — image-build no longer runs on every merge request. On a branch it was kaniko with --no-push: a ten-minute image build, discarded, on the box that serves production. It now runs only when the Dockerfile, Package.swift, Package.resolved or .gitlab-ci.yml change, and always on main. Confirmed working: image-build is absent from !96's pipeline.
ovh-flamenetmessenger raised 2 → 3 concurrent (2026-09-07, while load was low).

Still the root cause

CI shares a box — and a disk — with the production relay, its Postgres, Redis, coturn and the media stack. See #61, where that shared disk filled and took the database down. The load spike that produced today's false failure came from the staging gate, which is the deploy path, so the two contend by design.

Neither the runner limit nor the image-build rule changes that. Moving CI off this box is the fix.

## 2026-09-08: this now produces FALSE FAILURES, not just slow ones !96 changes **fifteen lines of shell** and nothing else — the Swift tree is byte-identical to main. Its `test` job failed: ``` status: failed failure_reason: script_failure duration: 891s ``` The same commit runs green locally in **84 seconds**. 891s is more than ten times that, and the job's log blew through GitLab's 4 MB cap and was truncated, so the actual failure was not even visible. The box was at load 22–46 at the time, with the staging gate compiling Swift alongside three CI jobs. That is the part worth escalating: a queue that is merely slow costs patience. A queue that turns green code red costs trust in the gate, and the natural response — merging on local evidence — is exactly the weakening this issue was opened about. ## Measured today - Peak load **46** on 8 cores; ; jobs waiting **47 minutes**. - **Pipelines for already-merged MRs keep running.** !85, !89, !90, !91 and !92 were all merged with their pipelines still queued or executing, holding the runner against work that mattered. GitLab does not cancel these; I cancelled five by hand and load fell 36 → 13. ## Done since - **!95** — `image-build` no longer runs on every merge request. On a branch it was kaniko with `--no-push`: a ten-minute image build, discarded, on the box that serves production. It now runs only when the Dockerfile, `Package.swift`, `Package.resolved` or `.gitlab-ci.yml` change, and always on main. **Confirmed working:** `image-build` is absent from !96's pipeline. - `ovh-flamenetmessenger` raised 2 → 3 concurrent (2026-09-07, while load was low). ## Still the root cause CI shares a box — and a disk — with the production relay, its Postgres, Redis, coturn and the media stack. See #61, where that shared disk filled and took the database down. The load spike that produced today's false failure came from the staging gate, which is the deploy path, so the two contend by design. Neither the runner limit nor the image-build rule changes that. Moving CI off this box is the fix.

extant2000git commented

2026-10-16 12:19:39 +00:00

Closing as obsolete in the 2026-10-16 issue cleanup. The relay's last GitLab runner job was 2026-10-09; CI now runs as Gitea Actions on mac1 and mac2, off the OVH box. The last 12 runs on main started 0 to 10 seconds after they were created, which meets the starts-within-minutes criterion.

extant2000git closed this issue

2026-10-16 12:19:39 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: extant2000/flamenet-relay#6

Pipelines queue over an hour behind the shared OVH runner #6

Consequences seen today

Worth considering

Done when

Diagnosed. It is not the box being busy — it is a limit = 1.

⚠️ My earlier suggestion on this ticket was wrong

The actual change

Correction: last night's stall was NOT this ticket

2026-09-08: this now produces FALSE FAILURES, not just slow ones

Measured today

Done since

Still the root cause

Diagnosed. It is not the box being busy — it is a `limit = 1`.