CI: a pull request that can no longer be merged should not hold a runner (80% of slot time) #1585

Merged
jaylouisw merged 1 commit from ci/1116-0220 into main 2026-11-16 04:56:14 +00:00
Owner

What was slow, with the numbers

The runner has two slots and a backlog measured in hours, so a queued job is dispatched long after the decision it was queued for. Measured through /api/v1/repos/jaylouisw/packetforge/actions/tasks on 2026-11-16, over the 200 most recent jobs (task ids 6422-6621):

  • 156 of 199 pull-request jobs, 18,529 of 23,097 runner seconds (80.2%), were dispatched for a pull request that was already merged or closed when the job started.
  • Per job: Test (Node 18) 22, Test (Node 20) 22, Test (Node 22) 22, Server smoke test 22, Docker build 22, Classify 23, Rebase 23.
  • Clearest single case: #1545 merged 01:00:11; its three test legs — tasks 6586, 6587, 6588 (run 2996) — started 02:10:53, 02:13:00, 02:14:20 and spent 206 s / 193 s / 187 s on a branch nobody could merge.
  • Same for #1560 (merged 01:35:12, legs 6593-6595 from 02:17:01) and #1553 (merged 23:40:17, legs 6609-6610 from 02:23:53). The rebase job kept running too: task 6606 for #1560 at 02:23:12, task 6592 for #1545 at 02:16:39.

The change

ci.yml: the changes job asks the API once, before anything expensive, whether the pull request can still be merged, and exports live — written as false only on a positive "merged":true or "state":"closed" in the reply. test is gated on needs.changes.outputs.live != 'false'; smoke and docker add the same clause to their existing data-only condition.

pr-rebase.yml: the same check, in the step rather than in the job if: — #1172 already measured that gating this job on pull_request.* stops it being dispatched for healthy pull requests too. A merged/closed PR is no longer rebased or armed, which removes the push → synchronize → full-matrix path against a dead branch.

Fail-open by construction: no output is written unless the answer is positively closed. A timed-out API, a missing token, an unparseable body, a push event (the step is if: github.event_name == 'pull_request') or a runner that predates this change all leave live unset, and != 'false' keeps the full five jobs. That is the only direction that matters: a green check on an untested pull request is the failure mode this must never introduce.

Coverage

Unchanged for every pull request that can still merge: three Node versions, all nine steps, smoke, docker — running live on this PR below. The only runs that skip are those whose pull request the API reports as already merged/closed, where a check can no longer gate anything.

Measured before shipping

  • Guard proof against the live API, using the exact script the workflow runs:
    • pull 1560 (merged): pull #1560 is already merged ... live=false, no matrix.
    • pull 1584 (open): no output written → full matrix, unchanged behaviour.
    • API unreachable: could not ask whether pull #1560 is still open ... running the full matrix → fail-open.
  • scripts/check-control-bytes.js, scripts/check-ci-disk-hygiene.js, scripts/syntax-check.js (333 files) pass; both workflow YAMLs parse; both embedded scripts pass bash -n.

Red on main, and not in this stream

main was red at dd6a0a0f: test/spec-vs-description.test.js → no device promises more ports of a kind than it has — 4 devices claiming 2x 20G QSFP+ with no port able to do it (edgecore/AS4610-30P, -30T, -54P, -54T). Seen in task 6586 (run 2996): 257 suites / 9370 assertions, 1 failed; the same failure is in tasks 6587, 6588, 6593, 6594, 6609, 6610.

That is catalogue data (devices/), which this stream does not touch. Current main is green: npm test on origin/main = 5a4fcd54 locally gives 263 suites / 9548 assertions / 0 failed, so the AS4610 claim was fixed by a later catalogue commit. Nothing in this PR depends on that.

## What was slow, with the numbers The runner has two slots and a backlog measured in hours, so a queued job is dispatched long after the decision it was queued for. Measured through `/api/v1/repos/jaylouisw/packetforge/actions/tasks` on 2026-11-16, over the 200 most recent jobs (task ids 6422-6621): - **156 of 199 pull-request jobs**, **18,529 of 23,097 runner seconds (80.2%)**, were dispatched for a pull request that was already **merged or closed when the job started**. - Per job: Test (Node 18) 22, Test (Node 20) 22, Test (Node 22) 22, Server smoke test 22, Docker build 22, Classify 23, Rebase 23. - Clearest single case: **#1545 merged 01:00:11**; its three test legs — tasks **6586, 6587, 6588** (run 2996) — started **02:10:53, 02:13:00, 02:14:20** and spent 206 s / 193 s / 187 s on a branch nobody could merge. - Same for #1560 (merged 01:35:12, legs 6593-6595 from 02:17:01) and #1553 (merged 23:40:17, legs 6609-6610 from 02:23:53). The rebase job kept running too: task **6606** for #1560 at 02:23:12, task **6592** for #1545 at 02:16:39. ## The change `ci.yml`: the `changes` job asks the API once, **before** anything expensive, whether the pull request can still be merged, and exports `live` — written as `false` **only** on a positive `"merged":true` or `"state":"closed"` in the reply. `test` is gated on `needs.changes.outputs.live != 'false'`; `smoke` and `docker` add the same clause to their existing data-only condition. `pr-rebase.yml`: the same check, in the step rather than in the job `if:` — #1172 already measured that gating this job on `pull_request.*` stops it being dispatched for healthy pull requests too. A merged/closed PR is no longer rebased or armed, which removes the push → `synchronize` → full-matrix path against a dead branch. **Fail-open by construction:** no output is written unless the answer is positively closed. A timed-out API, a missing token, an unparseable body, a `push` event (the step is `if: github.event_name == 'pull_request'`) or a runner that predates this change all leave `live` unset, and `!= 'false'` keeps the full five jobs. That is the only direction that matters: a green check on an untested pull request is the failure mode this must never introduce. ## Coverage Unchanged for every pull request that can still merge: three Node versions, all nine steps, smoke, docker — running live on this PR below. The only runs that skip are those whose pull request the API reports as already merged/closed, where a check can no longer gate anything. ## Measured before shipping - Guard proof against the live API, using the exact script the workflow runs: - pull 1560 (merged): `pull #1560 is already merged ... live=false`, no matrix. - pull 1584 (open): no output written → full matrix, unchanged behaviour. - API unreachable: `could not ask whether pull #1560 is still open ... running the full matrix` → fail-open. - `scripts/check-control-bytes.js`, `scripts/check-ci-disk-hygiene.js`, `scripts/syntax-check.js` (333 files) pass; both workflow YAMLs parse; both embedded scripts pass `bash -n`. ## Red on main, and not in this stream `main` was red at `dd6a0a0f`: `test/spec-vs-description.test.js` → `no device promises more ports of a kind than it has` — 4 devices claiming 2x 20G QSFP+ with no port able to do it (`edgecore/AS4610-30P`, `-30T`, `-54P`, `-54T`). Seen in task 6586 (run 2996): 257 suites / 9370 assertions, 1 failed; the same failure is in tasks 6587, 6588, 6593, 6594, 6609, 6610. That is catalogue data (`devices/`), which this stream does not touch. **Current main is green**: `npm test` on `origin/main` = `5a4fcd54` locally gives 263 suites / 9548 assertions / 0 failed, so the AS4610 claim was fixed by a later catalogue commit. Nothing in this PR depends on that.
CI: skip the matrix for a pull request that can no longer be merged
Some checks failed
CI / Classify the change (pull_request) Successful in 33s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Failing after 11m13s
CI / Test (Node 20) (pull_request) Successful in 3m32s
CI / Docker build (pull_request) Failing after 14m26s
CI / Server smoke test (pull_request) Failing after 14m37s
CI / Test (Node 22) (pull_request) Failing after 14m39s
CI / Test (Node 18) (pull_request) Failing after 14m46s
9a75c0f86a
80% of this runner's time on 2026-11-16 went to pull requests that were
already merged when their jobs started: 156 of the 199 pull-request jobs
among tasks 6422-6621, 18,529 of 23,097 runner seconds. #1545 merged at
01:00:11 and its three test legs (tasks 6586-6588, run 2996) started at
02:10:53.

The queue is hours deep on two slots, so a job arrives long after the
decision it was queued for. `changes` now asks the API once whether the
pull request is still open and writes live=false only on a positive
"merged":true or "state":"closed"; every job's condition is
!= 'false', so a timeout, a missing token, an unparseable body or a push
event leaves the full five jobs running. pr-rebase.yml gets the same
guard in the step rather than the job `if:`, because #1172 measured that
gating the job on pull_request fields stops healthy pull requests
dispatching too.
Author
Owner

Measured on the instance tonight (read-only queries against forgejo.db, Forgejo 11.0.16+gitea-1.22.0). This PR's premise — that slots are being burned on already-merged PRs — does not match the data, so please check this before merging.

Pull-request runs are not slow, they are not running at all. action_run, last 6 hours, by event and status:

event waiting running cancelled success
pull_request 50 9 30 0
pull_request_sync 54 28 10 0
push 0 2 5 147

The last successful pull_request run is 2977 (refs/pull/1504/head), created 2026-11-12 21:13. There has been none since.

The six open PRs (#1585–#1590, now seven) were never queued. Each of their runs (3504, 3505, 3511–3514, 3516–3519, 3522, 3523) has created == updated to the second: cancelled in the same second it was created. The root job is left status=5, task_id=0, its dependents status=7 (skipped, needs unmet), and no action_task row is ever written — so there is nothing for the runner to pick up and nothing that a queue-depth fix can reach. The 32 pr-rebase.yml runs still waiting are for closed PRs 1560–1580, not for these.

pr-rebase.yml has never succeeded, in its whole history: 297 waiting, 95 running, 14 cancelled, 1 failure, 0 success. That is the check that arms auto-merge, so on this instance auto-merge has never been armed by it — which is why nothing merges without a hand.

The runner itself is healthy: capacity 2, a task accepted every 10–20 s, 147 push-run successes in six hours.

Prime suspect: cancel-in-progress with a concurrency group that does not evaluate per-PR. Both ci.yml (cancel-in-progress: ${{ github.event_name == 'pull_request' }}) and pr-rebase.yml (cancel-in-progress: true) cancel at creation, while closed-audit.yml (false) and auto-rebase.yml (no block) dispatch normally — that pattern fits a group that collapses to one constant string repo-wide and then cancels the run it just created. Not proven, and it is a workflow file, not mine to change: the cheapest test is cancel-in-progress: false on pr-rebase.yml alone and one fresh PR push.

Needs the CI loop's / Jay's decision, not mine.

Measured on the instance tonight (read-only queries against `forgejo.db`, Forgejo 11.0.16+gitea-1.22.0). This PR's premise — that slots are being burned on already-merged PRs — does not match the data, so please check this before merging. **Pull-request runs are not slow, they are not running at all.** `action_run`, last 6 hours, by event and status: | event | waiting | running | cancelled | success | |---|---|---|---|---| | `pull_request` | 50 | 9 | 30 | **0** | | `pull_request_sync` | 54 | 28 | 10 | **0** | | `push` | 0 | 2 | 5 | **147** | The last successful `pull_request` run is 2977 (`refs/pull/1504/head`), created 2026-11-12 21:13. There has been none since. **The six open PRs (#1585–#1590, now seven) were never queued.** Each of their runs (3504, 3505, 3511–3514, 3516–3519, 3522, 3523) has `created == updated` to the second: cancelled in the same second it was created. The root job is left `status=5, task_id=0`, its dependents `status=7 (skipped, needs unmet)`, and **no `action_task` row is ever written** — so there is nothing for the runner to pick up and nothing that a queue-depth fix can reach. The 32 `pr-rebase.yml` runs still `waiting` are for closed PRs 1560–1580, not for these. **`pr-rebase.yml` has never succeeded, in its whole history:** 297 waiting, 95 running, 14 cancelled, 1 failure, 0 success. That is the check that arms auto-merge, so on this instance auto-merge has never been armed by it — which is why nothing merges without a hand. The runner itself is healthy: capacity 2, a task accepted every 10–20 s, 147 push-run successes in six hours. Prime suspect: `cancel-in-progress` with a concurrency group that does not evaluate per-PR. Both `ci.yml` (`cancel-in-progress: ${{ github.event_name == 'pull_request' }}`) and `pr-rebase.yml` (`cancel-in-progress: true`) cancel at creation, while `closed-audit.yml` (`false`) and `auto-rebase.yml` (no block) dispatch normally — that pattern fits a group that collapses to one constant string repo-wide and then cancels the run it just created. Not proven, and it is a workflow file, not mine to change: the cheapest test is `cancel-in-progress: false` on `pr-rebase.yml` alone and one fresh PR push. Needs the CI loop's / Jay's decision, not mine.
Author
Owner

Correction to last night's measurement, and one piece of evidence for this PR.

The earlier table said pull_request runs were never enqueued. That is no longer true, and it was already becoming untrue when I wrote it: runs are created for every PR push, and their jobs do reach the runner. What the runner is doing with them is the problem.

Measured 2026-11-16 03:29, read-only (forgejo.db, 11.0.16):

  • action_task 6699, 6700 (Test (Node 20), Test (Node 22)) — running, ref = refs/pull/1561/head. PR #1561 is closed (since 00:27).
  • action_task 6704 (Rebase onto main and arm auto-merge) — waiting, ref = refs/pull/1583/head. PR #1583 is closed (since 02:00).
  • Runs 3527/3528 (ci + pr-rebase for open PR #1592, sha c6987798) — created and cancelled, started = 0, every job task_id = 0.

So a single runner is spending three-minute test legs on pull requests that can no longer be merged, while the open ones are cancelled before they start. That is precisely what this PR's first-check skip is meant to stop — the premise holds, even though my earlier evidence for it was wrong. Lifetime for event = pull_request: 6 success, 32 cancelled, 440 waiting, 653 rows still marked running (unfinalised, not live work).

Also worth knowing: the one leg that did run to completion on a closed PR (#1561, Node 22) came back red — 257 suites, 9,388 assertions, 1 failed in test/spec-vs-description.test.js. Not blocking any merge, just noise on a dead branch.

This PR still cannot validate itself: its own runs are among the cancelled ones, head 9a75c0f8 unchanged since 02:35. Needs Jay's decision, not another push.

Correction to last night's measurement, and one piece of evidence **for** this PR. The earlier table said pull_request runs were never enqueued. That is no longer true, and it was already becoming untrue when I wrote it: runs are created for every PR push, and their jobs *do* reach the runner. What the runner is doing with them is the problem. Measured 2026-11-16 03:29, read-only (`forgejo.db`, 11.0.16): - `action_task` 6699, 6700 (`Test (Node 20)`, `Test (Node 22)`) — running, `ref = refs/pull/1561/head`. PR **#1561 is closed** (since 00:27). - `action_task` 6704 (`Rebase onto main and arm auto-merge`) — waiting, `ref = refs/pull/1583/head`. PR **#1583 is closed** (since 02:00). - Runs 3527/3528 (ci + pr-rebase for open PR #1592, sha `c6987798`) — created and cancelled, `started = 0`, every job `task_id = 0`. So a single runner is spending three-minute test legs on pull requests that can no longer be merged, while the open ones are cancelled before they start. That is precisely what this PR's first-check skip is meant to stop — the premise holds, even though my earlier evidence for it was wrong. Lifetime for `event = pull_request`: 6 success, 32 cancelled, 440 waiting, 653 rows still marked running (unfinalised, not live work). Also worth knowing: the one leg that did run to completion on a closed PR (#1561, Node 22) came back **red** — 257 suites, 9,388 assertions, 1 failed in `test/spec-vs-description.test.js`. Not blocking any merge, just noise on a dead branch. This PR still cannot validate itself: its own runs are among the cancelled ones, `head` `9a75c0f8` unchanged since 02:35. Needs Jay's decision, not another push.
Author
Owner

Third measurement, 2026-11-16 03:55, read-only (forgejo.db 11.0.16). Not a correction this time - a different failure than either earlier theory.

Open pull-request runs are cancelled before dispatch; closed ones still execute.

  • 12 open PRs (#1585-#1596), 12/12 mergeable, 7/7 pending on every head SHA, zero success, zero failure.
  • Each has runs with full job sets: #1585 = runs 3504 (6 jobs) + 3505 (1 job), status=5 (cancelled). Across #1584-#1596 the pattern is identical: 2 runs, 6+1 jobs, status cancelled.
  • None of those runs ever produced an action_task row - the runner never asked for the jobs. A PR cannot go green while its run is cancelled before pickup, and #1585's own run is one of them, so waiting for it to be green is circular.

Meanwhile the runner is busy, with the closed PRs. Tasks 6705-6707 (refs/pull/1559/head, closed 23:15) and 6710-6714 (refs/pull/1572/head, closed) are running right now.

Capacity, measured: 81 tasks finished in the last hour (repo 12) = 1.35/min. Live waiting tasks (id>=6000) = 576, so ~7 h of backlog; new PRs arrive at ~11 runs x 7 jobs per hour, i.e. demand is at or above supply. A separate 4,263 waiting tasks with id<6000 (oldest id 14) are never picked. One pick task: database is locked in the runner log at 03:32:54.

Needs Jay's decision: merge this ungreen (the run-level waste it targets is real but it will never be green here), and/or find what cancels a pull_request run before dispatch - that, not the mergeable filter, is what holds the queue.

Third measurement, 2026-11-16 03:55, read-only (`forgejo.db` 11.0.16). Not a correction this time - a different failure than either earlier theory. **Open pull-request runs are cancelled before dispatch; closed ones still execute.** - 12 open PRs (#1585-#1596), 12/12 mergeable, 7/7 pending on every head SHA, zero success, zero failure. - Each has runs with full job sets: #1585 = runs 3504 (6 jobs) + 3505 (1 job), `status=5` (cancelled). Across #1584-#1596 the pattern is identical: 2 runs, 6+1 jobs, status cancelled. - None of those runs ever produced an `action_task` row - the runner never asked for the jobs. A PR cannot go green while its run is cancelled before pickup, and #1585's own run is one of them, so waiting for it to be green is circular. **Meanwhile the runner is busy, with the closed PRs.** Tasks 6705-6707 (`refs/pull/1559/head`, closed 23:15) and 6710-6714 (`refs/pull/1572/head`, closed) are running right now. **Capacity, measured:** 81 tasks finished in the last hour (repo 12) = 1.35/min. Live waiting tasks (`id>=6000`) = 576, so ~7 h of backlog; new PRs arrive at ~11 runs x 7 jobs per hour, i.e. demand is at or above supply. A separate 4,263 waiting tasks with `id<6000` (oldest id 14) are never picked. One `pick task: database is locked` in the runner log at 03:32:54. Needs Jay's decision: merge this ungreen (the run-level waste it targets is real but it will never be green here), and/or find what cancels a pull_request run before dispatch - that, not the mergeable filter, is what holds the queue.
Author
Owner

Correction to comment 7887 — the runs are not being cancelled before dispatch. action_run.status=5 is running, not cancelled (enum: 1 success, 2 failure, 3 cancelled, 4 waiting, 5 running, 6 skipped). Evidence: runs 3355 (refs/pull/1574/head) and 3357 (#1563) are status 5 and executed jobs between 03:43 and 03:54 today (commit_status went pending -> failure); run 3352 shows 1 after completing. No action_task rows means the runner has not reached the run yet — task rows are created at dispatch (run 3355: run row 00:57:43, first task 03:43:51).

Queue at 03:55, measured read-only from forgejo.db:

  • 67 runs in flight (ids 3368-3544), 302 jobs, 29 runs partly executed.
  • 32 runs / 182 jobs (60%) are on refs/pull/<N>/head for PRs that are already closed (#1545-#1574). Only 28 runs / 98 jobs belong to the 14 open PRs.
  • Dispatch is FIFO by run id, so closed-PR work runs ahead of every open PR.
  • Throughput 75-90 jobs/hour. 302 jobs left => an open PR's run starts ~3.5-4h after it is opened; #1598's run (03:53) should start around 06:30-07:00.

So the mergeable filter is not what holds the queue: 60% of it is work for PRs nobody can merge any more, and nothing cancels a run when its PR closes. Cancelling runs on the pull_request closed event fixes this without touching the mergeable logic.

Side note for whoever owns test/: the closed-PR runs are failing Node legs for a real reason — #1574's Node 18 failed test/spec-vs-description.test.js (4 Edgecore AS4610 specs claim 2x 20G QSFP+ with no port that can do it). That suite passes on current main (060a3608), so it is that branch, not a main regression.

Correction to comment 7887 — the runs are **not** being cancelled before dispatch. `action_run.status=5` is `running`, not `cancelled` (enum: 1 success, 2 failure, 3 cancelled, 4 waiting, 5 running, 6 skipped). Evidence: runs 3355 (refs/pull/1574/head) and 3357 (#1563) are status 5 and executed jobs between 03:43 and 03:54 today (commit_status went pending -> failure); run 3352 shows 1 after completing. No `action_task` rows means the runner has not reached the run yet — task rows are created at dispatch (run 3355: run row 00:57:43, first task 03:43:51). Queue at 03:55, measured read-only from forgejo.db: - 67 runs in flight (ids 3368-3544), 302 jobs, 29 runs partly executed. - 32 runs / 182 jobs (60%) are on `refs/pull/<N>/head` for PRs that are already **closed** (#1545-#1574). Only 28 runs / 98 jobs belong to the 14 open PRs. - Dispatch is FIFO by run id, so closed-PR work runs ahead of every open PR. - Throughput 75-90 jobs/hour. 302 jobs left => an open PR's run starts ~3.5-4h after it is opened; #1598's run (03:53) should start around 06:30-07:00. So the mergeable filter is not what holds the queue: 60% of it is work for PRs nobody can merge any more, and nothing cancels a run when its PR closes. Cancelling runs on the `pull_request` closed event fixes this without touching the mergeable logic. Side note for whoever owns test/: the closed-PR runs are failing Node legs for a real reason — #1574's Node 18 failed `test/spec-vs-description.test.js` (4 Edgecore AS4610 specs claim 2x 20G QSFP+ with no port that can do it). That suite passes on current main (060a3608), so it is that branch, not a main regression.
jaylouisw scheduled this pull request to auto merge when all checks succeed 2026-11-16 04:55:57 +00:00
jaylouisw force-pushed ci/1116-0220 from 9a75c0f86a
Some checks failed
CI / Classify the change (pull_request) Successful in 33s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Failing after 11m13s
CI / Test (Node 20) (pull_request) Successful in 3m32s
CI / Docker build (pull_request) Failing after 14m26s
CI / Server smoke test (pull_request) Failing after 14m37s
CI / Test (Node 22) (pull_request) Failing after 14m39s
CI / Test (Node 18) (pull_request) Failing after 14m46s
to 5eb1a2c7d9
Some checks failed
CI / Classify the change (pull_request) Successful in 48s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 42s
CI / Test (Node 20) (pull_request) Has been skipped
CI / Test (Node 22) (pull_request) Has been skipped
CI / Docker build (pull_request) Has been skipped
CI / Server smoke test (pull_request) Failing after 10m41s
CI / Test (Node 18) (pull_request) Failing after 11m6s
2026-11-16 04:55:58 +00:00
Compare
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jaylouisw/packetforge!1585
No description provided.