CI: a hung smoke test costs 3 minutes, not 15, and keeps its log #1870

Merged
jaylouisw merged 1 commit from ci/1117-1621 into main 2026-11-17 16:34:55 +00:00
Owner

What was slow/broken

Server smoke test is the only job in ci.yml whose work step has no bound: run: node scripts/smoke-http.js. The script's client settles on the response's end or on error, so a connection that is accepted and never answered leaves the promise pending forever and Node waits until something outside it kills the process.

Three failures today, each on a pull request whose three test legs passed the same commit in 52-67 s:

run task job duration log
4832 (pull #1798) 11669 Server smoke test 676 s dbfs open "actions_log/.../11669.log.zst": file does not exist
4688 (pull #1784) 11193 Server smoke test 872 s same, no log
4373 (pull #1755) 10082 Server smoke test 792 s same, no log

2,340 s of a two-slot runner in one day, and not one of them left a diagnosable log: the process was killed instead of exiting, so the runner has no actions_log object for the task.

timeout-minutes: 10 on the job is not the bound: this runner does not enforce it - all three durations exceed 600 s, and the same non-enforcement is already documented in the changes job (task 12980, 13m48s).

The change (workflow only)

.forgejo/workflows/ci.yml, the smoke job's last step: timeout --signal=TERM --kill-after=15 180 node scripts/smoke-http.js, with the step's own status kept and a 124/137 annotated as a hung request rather than a slow server.

Baseline / after

  • Baseline: task 11669 (run 4832) - Server smoke test, 676 s, failure, no log; task 11193 (run 4688) 872 s; task 10082 (run 4373) 792 s.
  • After: the smoke step on this PR's run - task id cited below once the run reports - expected ~11 s (the 24 h median over 469 smoke task records), i.e. unchanged; the bound only applies to the hang path.
  • Verified locally: the exact step body under a 2 s bound - hanging child exits 124 with the annotation and its output intact; a fast child exits 0.

Coverage unchanged

No assertion, step or job is removed, skipped or retried. A slow-but-working smoke test still passes (180 s is 6x the slowest green job in the last 24 h, 9x the median); a genuinely hung one is still red - it just fails in 3 minutes and keeps its log instead of holding a slot for 15 and leaving nothing.

Still red / not addressed here: the test legs have the same class of hang (task 11437 808 s, 11555 668 s, 11556 661 s, 11667 683 s - all with no log), and one genuine flake on main at 15:28 today: run 5320, task 13248, test/api-config-door.test.js, Error: read ECONNRESET at 6124 ms while Node 18 and 22 passed the same commit. Both are follow-ups.

## What was slow/broken `Server smoke test` is the only job in `ci.yml` whose work step has no bound: `run: node scripts/smoke-http.js`. The script's client settles on the response's `end` or on `error`, so a connection that is accepted and never answered leaves the promise pending forever and Node waits until something outside it kills the process. Three failures today, each on a pull request whose three test legs passed the same commit in 52-67 s: | run | task | job | duration | log | |---|---|---|---|---| | 4832 (pull #1798) | 11669 | Server smoke test | 676 s | `dbfs open "actions_log/.../11669.log.zst": file does not exist` | | 4688 (pull #1784) | 11193 | Server smoke test | 872 s | same, no log | | 4373 (pull #1755) | 10082 | Server smoke test | 792 s | same, no log | 2,340 s of a two-slot runner in one day, and not one of them left a diagnosable log: the process was killed instead of exiting, so the runner has no `actions_log` object for the task. `timeout-minutes: 10` on the job is not the bound: this runner does not enforce it - all three durations exceed 600 s, and the same non-enforcement is already documented in the `changes` job (task 12980, 13m48s). ## The change (workflow only) `.forgejo/workflows/ci.yml`, the `smoke` job's last step: `timeout --signal=TERM --kill-after=15 180 node scripts/smoke-http.js`, with the step's own status kept and a 124/137 annotated as a hung request rather than a slow server. ## Baseline / after - Baseline: task 11669 (run 4832) - `Server smoke test`, 676 s, failure, no log; task 11193 (run 4688) 872 s; task 10082 (run 4373) 792 s. - After: the smoke step on this PR's run - task id cited below once the run reports - expected ~11 s (the 24 h median over 469 smoke task records), i.e. unchanged; the bound only applies to the hang path. - Verified locally: the exact step body under a 2 s bound - hanging child exits 124 with the annotation and its output intact; a fast child exits 0. ## Coverage unchanged No assertion, step or job is removed, skipped or retried. A slow-but-working smoke test still passes (180 s is 6x the slowest green job in the last 24 h, 9x the median); a genuinely hung one is still red - it just fails in 3 minutes and keeps its log instead of holding a slot for 15 and leaving nothing. Still red / not addressed here: the test legs have the same class of hang (task 11437 808 s, 11555 668 s, 11556 661 s, 11667 683 s - all with no log), and one genuine flake on main at 15:28 today: run 5320, task 13248, `test/api-config-door.test.js`, `Error: read ECONNRESET` at 6124 ms while Node 18 and 22 passed the same commit. Both are follow-ups.
CI: a hung smoke test costs 3 minutes, not 15, and keeps its log
All checks were successful
CI / Classify the change (pull_request) Successful in 8s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 14s
CI / Server smoke test (pull_request) Successful in 10s
CI / Test (Node 20) (pull_request) Successful in 51s
CI / Test (Node 18) (pull_request) Successful in 59s
CI / Test (Node 22) (pull_request) Successful in 56s
CI / Docker build (pull_request) Successful in 20s
CI / Arm auto-merge (pull_request) Successful in 5s
0313849c8a
`scripts/smoke-http.js` is the only step in ci.yml with no bound. Its HTTP
client settles on `end` or on `error`; a connection accepted and never
answered is neither, so the process waits until something outside it kills
it. Three smoke failures on 2026-11-17 did exactly that, on pull requests
whose three test legs passed the same commit in 52-67 s:

  run 4832 / task 11669  failed after 676 s
  run 4688 / task 11193  failed after 872 s
  run 4373 / task 10082  failed after 792 s

All three left no log: the runner answers `dbfs open
"actions_log/.../11669.log.zst": file does not exist`, so the red check
could not be diagnosed. 2,340 s of a two-slot runner on one day, with
nothing to show for it.

`timeout-minutes: 10` on the job is not the bound - this runner does not
enforce it, and all three durations exceed it. So the bound is in the
step: 180 s under `timeout --signal=TERM --kill-after=15`, against a job
that is 11 s median and 30 s at its slowest green run in the last 24 h
(469 task records). The step keeps its own exit status, annotates a 124 or
137 as a hung request rather than a slow server, and keeps its log because
Node is signalled instead of the container being killed.

No assertion changes: a slow-but-working smoke test still passes, a hung
one is still red.
jaylouisw force-pushed ci/1117-1621 from 0313849c8a
All checks were successful
CI / Classify the change (pull_request) Successful in 8s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 14s
CI / Server smoke test (pull_request) Successful in 10s
CI / Test (Node 20) (pull_request) Successful in 51s
CI / Test (Node 18) (pull_request) Successful in 59s
CI / Test (Node 22) (pull_request) Successful in 56s
CI / Docker build (pull_request) Successful in 20s
CI / Arm auto-merge (pull_request) Successful in 5s
to 555dd76529
All checks were successful
CI / Classify the change (pull_request) Successful in 13s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 7s
CI / Server smoke test (pull_request) Successful in 8s
CI / Docker build (pull_request) Successful in 22s
CI / Test (Node 20) (pull_request) Successful in 47s
CI / Test (Node 22) (pull_request) Successful in 51s
CI / Test (Node 18) (pull_request) Successful in 56s
CI / Arm auto-merge (pull_request) Successful in 18s
2026-11-17 16:31:40 +00:00
Compare
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jaylouisw/packetforge!1870
No description provided.