CI: a pull request that can no longer be merged should not hold a runner (80% of slot time) #1585
No reviewers
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
jaylouisw/packetforge!1585
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "ci/1116-0220"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What was slow, with the numbers
The runner has two slots and a backlog measured in hours, so a queued job is dispatched long after the decision it was queued for. Measured through
/api/v1/repos/jaylouisw/packetforge/actions/taskson 2026-11-16, over the 200 most recent jobs (task ids 6422-6621):The change
ci.yml: thechangesjob asks the API once, before anything expensive, whether the pull request can still be merged, and exportslive— written asfalseonly on a positive"merged":trueor"state":"closed"in the reply.testis gated onneeds.changes.outputs.live != 'false';smokeanddockeradd the same clause to their existing data-only condition.pr-rebase.yml: the same check, in the step rather than in the jobif:— #1172 already measured that gating this job onpull_request.*stops it being dispatched for healthy pull requests too. A merged/closed PR is no longer rebased or armed, which removes the push →synchronize→ full-matrix path against a dead branch.Fail-open by construction: no output is written unless the answer is positively closed. A timed-out API, a missing token, an unparseable body, a
pushevent (the step isif: github.event_name == 'pull_request') or a runner that predates this change all leaveliveunset, and!= 'false'keeps the full five jobs. That is the only direction that matters: a green check on an untested pull request is the failure mode this must never introduce.Coverage
Unchanged for every pull request that can still merge: three Node versions, all nine steps, smoke, docker — running live on this PR below. The only runs that skip are those whose pull request the API reports as already merged/closed, where a check can no longer gate anything.
Measured before shipping
pull #1560 is already merged ... live=false, no matrix.could not ask whether pull #1560 is still open ... running the full matrix→ fail-open.scripts/check-control-bytes.js,scripts/check-ci-disk-hygiene.js,scripts/syntax-check.js(333 files) pass; both workflow YAMLs parse; both embedded scripts passbash -n.Red on main, and not in this stream
mainwas red atdd6a0a0f:test/spec-vs-description.test.js→no device promises more ports of a kind than it has— 4 devices claiming 2x 20G QSFP+ with no port able to do it (edgecore/AS4610-30P,-30T,-54P,-54T). Seen in task 6586 (run 2996): 257 suites / 9370 assertions, 1 failed; the same failure is in tasks 6587, 6588, 6593, 6594, 6609, 6610.That is catalogue data (
devices/), which this stream does not touch. Current main is green:npm testonorigin/main=5a4fcd54locally gives 263 suites / 9548 assertions / 0 failed, so the AS4610 claim was fixed by a later catalogue commit. Nothing in this PR depends on that.Measured on the instance tonight (read-only queries against
forgejo.db, Forgejo 11.0.16+gitea-1.22.0). This PR's premise — that slots are being burned on already-merged PRs — does not match the data, so please check this before merging.Pull-request runs are not slow, they are not running at all.
action_run, last 6 hours, by event and status:pull_requestpull_request_syncpushThe last successful
pull_requestrun is 2977 (refs/pull/1504/head), created 2026-11-12 21:13. There has been none since.The six open PRs (#1585–#1590, now seven) were never queued. Each of their runs (3504, 3505, 3511–3514, 3516–3519, 3522, 3523) has
created == updatedto the second: cancelled in the same second it was created. The root job is leftstatus=5, task_id=0, its dependentsstatus=7 (skipped, needs unmet), and noaction_taskrow is ever written — so there is nothing for the runner to pick up and nothing that a queue-depth fix can reach. The 32pr-rebase.ymlruns stillwaitingare for closed PRs 1560–1580, not for these.pr-rebase.ymlhas never succeeded, in its whole history: 297 waiting, 95 running, 14 cancelled, 1 failure, 0 success. That is the check that arms auto-merge, so on this instance auto-merge has never been armed by it — which is why nothing merges without a hand.The runner itself is healthy: capacity 2, a task accepted every 10–20 s, 147 push-run successes in six hours.
Prime suspect:
cancel-in-progresswith a concurrency group that does not evaluate per-PR. Bothci.yml(cancel-in-progress: ${{ github.event_name == 'pull_request' }}) andpr-rebase.yml(cancel-in-progress: true) cancel at creation, whileclosed-audit.yml(false) andauto-rebase.yml(no block) dispatch normally — that pattern fits a group that collapses to one constant string repo-wide and then cancels the run it just created. Not proven, and it is a workflow file, not mine to change: the cheapest test iscancel-in-progress: falseonpr-rebase.ymlalone and one fresh PR push.Needs the CI loop's / Jay's decision, not mine.
Correction to last night's measurement, and one piece of evidence for this PR.
The earlier table said pull_request runs were never enqueued. That is no longer true, and it was already becoming untrue when I wrote it: runs are created for every PR push, and their jobs do reach the runner. What the runner is doing with them is the problem.
Measured 2026-11-16 03:29, read-only (
forgejo.db, 11.0.16):action_task6699, 6700 (Test (Node 20),Test (Node 22)) — running,ref = refs/pull/1561/head. PR #1561 is closed (since 00:27).action_task6704 (Rebase onto main and arm auto-merge) — waiting,ref = refs/pull/1583/head. PR #1583 is closed (since 02:00).c6987798) — created and cancelled,started = 0, every jobtask_id = 0.So a single runner is spending three-minute test legs on pull requests that can no longer be merged, while the open ones are cancelled before they start. That is precisely what this PR's first-check skip is meant to stop — the premise holds, even though my earlier evidence for it was wrong. Lifetime for
event = pull_request: 6 success, 32 cancelled, 440 waiting, 653 rows still marked running (unfinalised, not live work).Also worth knowing: the one leg that did run to completion on a closed PR (#1561, Node 22) came back red — 257 suites, 9,388 assertions, 1 failed in
test/spec-vs-description.test.js. Not blocking any merge, just noise on a dead branch.This PR still cannot validate itself: its own runs are among the cancelled ones,
head9a75c0f8unchanged since 02:35. Needs Jay's decision, not another push.Third measurement, 2026-11-16 03:55, read-only (
forgejo.db11.0.16). Not a correction this time - a different failure than either earlier theory.Open pull-request runs are cancelled before dispatch; closed ones still execute.
status=5(cancelled). Across #1584-#1596 the pattern is identical: 2 runs, 6+1 jobs, status cancelled.action_taskrow - the runner never asked for the jobs. A PR cannot go green while its run is cancelled before pickup, and #1585's own run is one of them, so waiting for it to be green is circular.Meanwhile the runner is busy, with the closed PRs. Tasks 6705-6707 (
refs/pull/1559/head, closed 23:15) and 6710-6714 (refs/pull/1572/head, closed) are running right now.Capacity, measured: 81 tasks finished in the last hour (repo 12) = 1.35/min. Live waiting tasks (
id>=6000) = 576, so ~7 h of backlog; new PRs arrive at ~11 runs x 7 jobs per hour, i.e. demand is at or above supply. A separate 4,263 waiting tasks withid<6000(oldest id 14) are never picked. Onepick task: database is lockedin the runner log at 03:32:54.Needs Jay's decision: merge this ungreen (the run-level waste it targets is real but it will never be green here), and/or find what cancels a pull_request run before dispatch - that, not the mergeable filter, is what holds the queue.
Correction to comment 7887 — the runs are not being cancelled before dispatch.
action_run.status=5isrunning, notcancelled(enum: 1 success, 2 failure, 3 cancelled, 4 waiting, 5 running, 6 skipped). Evidence: runs 3355 (refs/pull/1574/head) and 3357 (#1563) are status 5 and executed jobs between 03:43 and 03:54 today (commit_status went pending -> failure); run 3352 shows 1 after completing. Noaction_taskrows means the runner has not reached the run yet — task rows are created at dispatch (run 3355: run row 00:57:43, first task 03:43:51).Queue at 03:55, measured read-only from forgejo.db:
refs/pull/<N>/headfor PRs that are already closed (#1545-#1574). Only 28 runs / 98 jobs belong to the 14 open PRs.So the mergeable filter is not what holds the queue: 60% of it is work for PRs nobody can merge any more, and nothing cancels a run when its PR closes. Cancelling runs on the
pull_requestclosed event fixes this without touching the mergeable logic.Side note for whoever owns test/: the closed-PR runs are failing Node legs for a real reason — #1574's Node 18 failed
test/spec-vs-description.test.js(4 Edgecore AS4610 specs claim 2x 20G QSFP+ with no port that can do it). That suite passes on current main (060a3608), so it is that branch, not a main regression.9a75c0f86a5eb1a2c7d9