ci: bound every network fetch, and answer the classify job instead of hanging #1856

Merged
jaylouisw merged 1 commit from ci/1117-1420 into main 2026-11-17 14:33:27 +00:00
Owner

What was slow/broken

Classify the change is the gate on all five other jobs, and its timeout-minutes: 5 is not enforced by this runner. From /actions/tasks, classify tasks that held a slot and then failed silently (no log written at all, so no step ever returned):

task run branch outcome duration
12980 5238 #1851 failure 828 s (13m48s)
10860 4596 #1739 failure 765 s
7139 3346 #1611 failure 2616 s
7127 3339 #1610 failure 2741 s
7100 3326 #1604 failure 3331 s
2694 1342 #1184 failure 27,702 s

For the whole of each, the PR's five jobs read Blocked by required conditions (commit status for 1f647259 on #1851: CI / Classify the change failure - Failing after 13m48s, everything else pending - Blocked by required conditions). Duration is run_started_at -> updated_at, so this is execution, not queueing; jobs/*/logs returns file does not exist for every one of them.

Change (workflow only)

  • every git fetch in changes, test, smoke, docker is now timeout N + -c http.lowSpeedLimit=1000 -c http.lowSpeedTime=30 (120 s classify, 60 s base, 300 s legs) - the same pair that fixed the stuck fetch in the demo deploy script;
  • the classify fetch no longer fails the job when it cannot fetch: no tree means no diff, so it publishes the answer classify-change.js gives for an unreadable base - data_only=false, code=true - and the five jobs run;
  • the classifier itself is bounded at 120 s and falls back the same way.

Coverage unchanged

data_only=true (the only output that lets smoke and Docker stand down) is never written on a fallback path, and live is untouched. Fallbacks only ever add jobs, never remove one.

Measured

  • before: classify failure, 828 s, run 5238 / task 12980, PR #1851 blocked (5 jobs Blocked by required conditions).
  • after: same pair of commits as a green reference - #1854's a5e90d17 over 3866046d, run 5259 answered data_only=true / code=false / changed files (1): data docs/CLI.md; the new step body answers byte-identically (checked out against odin in a scratch clone), and a dead server answers data_only=false / code=true and exits 0 in ~2 s.
  • all 22 run: blocks pass bash -n; check-ci-disk-hygiene, check-node-surface, check-canonical-urls pass.
## What was slow/broken `Classify the change` is the gate on all five other jobs, and its `timeout-minutes: 5` is not enforced by this runner. From `/actions/tasks`, classify tasks that held a slot and then failed silently (no log written at all, so no step ever returned): | task | run | branch | outcome | duration | |---|---|---|---|---| | 12980 | 5238 | #1851 | failure | 828 s (13m48s) | | 10860 | 4596 | #1739 | failure | 765 s | | 7139 | 3346 | #1611 | failure | 2616 s | | 7127 | 3339 | #1610 | failure | 2741 s | | 7100 | 3326 | #1604 | failure | 3331 s | | 2694 | 1342 | #1184 | failure | 27,702 s | For the whole of each, the PR's five jobs read *Blocked by required conditions* (commit status for `1f647259` on #1851: `CI / Classify the change failure - Failing after 13m48s`, everything else `pending - Blocked by required conditions`). Duration is `run_started_at -> updated_at`, so this is execution, not queueing; `jobs/*/logs` returns `file does not exist` for every one of them. ## Change (workflow only) - every `git fetch` in `changes`, `test`, `smoke`, `docker` is now `timeout N` + `-c http.lowSpeedLimit=1000 -c http.lowSpeedTime=30` (120 s classify, 60 s base, 300 s legs) - the same pair that fixed the stuck fetch in the demo deploy script; - the classify fetch no longer fails the job when it cannot fetch: no tree means no diff, so it publishes the answer `classify-change.js` gives for an unreadable base - `data_only=false`, `code=true` - and the five jobs run; - the classifier itself is bounded at 120 s and falls back the same way. ## Coverage unchanged `data_only=true` (the only output that lets smoke and Docker stand down) is never written on a fallback path, and `live` is untouched. Fallbacks only ever add jobs, never remove one. ## Measured - before: classify failure, 828 s, run 5238 / task 12980, PR #1851 blocked (5 jobs `Blocked by required conditions`). - after: same pair of commits as a green reference - #1854's `a5e90d17` over `3866046d`, run 5259 answered `data_only=true / code=false / changed files (1): data docs/CLI.md`; the new step body answers byte-identically (checked out against odin in a scratch clone), and a dead server answers `data_only=false / code=true` and exits 0 in ~2 s. - all 22 `run:` blocks pass `bash -n`; `check-ci-disk-hygiene`, `check-node-surface`, `check-canonical-urls` pass.
ci: bound every network fetch, and answer the classify job instead of hanging
All checks were successful
CI / Classify the change (pull_request) Successful in 9s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 14s
CI / Server smoke test (pull_request) Successful in 7s
CI / Docker build (pull_request) Successful in 23s
CI / Test (Node 18) (pull_request) Successful in 1m1s
CI / Test (Node 20) (pull_request) Successful in 49s
CI / Test (Node 22) (pull_request) Successful in 51s
CI / Arm auto-merge (pull_request) Successful in 7s
5abd3a2861
The changes job is the gate on all five others, and its declared
timeout-minutes: 5 was not enforced by this runner. Measured through
/actions/tasks: task 12980 (run 5238, pull #1851) reported "Failing after
13m48s", task 10860 (run 4596) 12m45s, task 7139 (run 3346) 43m36s, task 2694
(run 1342) 27,702 s - none of them wrote a log, so no step ever returned, and
every one of those pull requests read "Blocked by required conditions" on all
five jobs for the whole time.

So the bound moves into the step: timeout plus git's own low-speed abort on the
fetch in all four jobs, and on the classifier itself. The 120 s classify fetch
no longer fails the job: with no tree there is no diff, and the honest answer
for a diff nobody can read is the one classify-change.js already prints for an
unreadable base - run everything. data_only=true, the only output that lets
smoke and Docker stand down, is never written on that path.

Verified locally against odin: PR #1854's pair (a5e90d17 over 3866046d) still
answers data_only=true / code=false / changed files (1): data docs/CLI.md,
byte-identical to run 5259; a dead server answers data_only=false / code=true
and exits 0. All 22 run blocks pass bash -n; check-ci-disk-hygiene,
check-node-surface and check-canonical-urls pass.
jaylouisw force-pushed ci/1117-1420 from 5abd3a2861
All checks were successful
CI / Classify the change (pull_request) Successful in 9s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 14s
CI / Server smoke test (pull_request) Successful in 7s
CI / Docker build (pull_request) Successful in 23s
CI / Test (Node 18) (pull_request) Successful in 1m1s
CI / Test (Node 20) (pull_request) Successful in 49s
CI / Test (Node 22) (pull_request) Successful in 51s
CI / Arm auto-merge (pull_request) Successful in 7s
to 61dca67c5f
All checks were successful
CI / Classify the change (pull_request) Successful in 9s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 10s
CI / Server smoke test (pull_request) Successful in 9s
CI / Test (Node 20) (pull_request) Successful in 48s
CI / Test (Node 22) (pull_request) Successful in 52s
CI / Test (Node 18) (pull_request) Successful in 57s
CI / Docker build (pull_request) Successful in 19s
CI / Arm auto-merge (pull_request) Successful in 18s
2026-11-17 14:30:22 +00:00
Compare
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jaylouisw/packetforge!1856
No description provided.