Bound every CI job, so a hung one cannot hold the only runner #1066

Merged
jaylouisw merged 1 commit from ci/bound-stuck-jobs into main 2026-11-08 14:51:12 +00:00
Owner

Partial fix for #930 — the half that is ours to fix.

That issue documents a task stuck in running for over forty minutes while every other pull request waited behind it. With one runner, a hung job is not a slow job: it is a queue that has stopped, and it is indistinguishable from a queue that is merely busy. That is why #930 first looked like "pull_request events never create tasks" — at the moment of looking, nothing was moving.

Every job now carries a bound:

job bound healthy runtime
test (×3 matrix) 15 min ~75 s
smoke 10 min ~25 s
docker 10 min ~20 s

A hang now costs ten minutes and returns the runner, instead of blocking the whole queue until someone notices. It does not make CI faster; it makes a failure cost minutes instead of an afternoon.

This does not close #930. The other half is throughput and main verification on merge, which needs either a second runner or a scheduler that works (see the note below) — neither of which is a code change in this repository.

YAML validated: test → 15, smoke → 10, docker → 10; the concurrency block from #1060 is intact.

Refs #930.

Partial fix for #930 — the half that is ours to fix. That issue documents a task stuck in `running` for **over forty minutes** while every other pull request waited behind it. With one runner, a hung job is not a slow job: it is a queue that has stopped, and it is indistinguishable from a queue that is merely busy. That is why #930 first looked like "`pull_request` events never create tasks" — at the moment of looking, nothing was moving. Every job now carries a bound: | job | bound | healthy runtime | |---|---|---| | `test` (×3 matrix) | 15 min | ~75 s | | `smoke` | 10 min | ~25 s | | `docker` | 10 min | ~20 s | A hang now costs ten minutes and returns the runner, instead of blocking the whole queue until someone notices. It does not make CI faster; it makes a failure cost minutes instead of an afternoon. This does **not** close #930. The other half is throughput and `main` verification on merge, which needs either a second runner or a scheduler that works (see the note below) — neither of which is a code change in this repository. YAML validated: `test → 15`, `smoke → 10`, `docker → 10`; the `concurrency` block from #1060 is intact. Refs #930.
Bound every CI job, so a hung one cannot hold the only runner
Some checks failed
CI / Test (Node 18) (pull_request) Failing after 1m34s
CI / Test (Node 20) (pull_request) Failing after 1m29s
CI / Server smoke test (pull_request) Successful in 26s
CI / Docker build (pull_request) Successful in 9s
CI / Test (Node 22) (pull_request) Failing after 1m26s
c27e352b0a
Issue #930 records a task stuck in 'running' for over forty minutes while every
other pull request waited behind it. With one runner, a hung job is not a slow
job: it is a queue that has stopped, and it is indistinguishable from a queue
that is merely busy - which is why that issue first looked like 'pull_request
events create no tasks'.

Every job now carries a timeout: 15 minutes for the test matrix (a healthy run is
about 75 seconds), 10 minutes for the smoke and docker jobs. A hang now costs ten
minutes and frees the runner, instead of blocking everything until someone
notices.

Refs #930.
jaylouisw scheduled this pull request to auto merge when all checks succeed 2026-11-08 14:49:50 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jaylouisw/packetforge!1066
No description provided.