CI: a hung smoke test costs 3 minutes, not 15, and keeps its log #1870
No reviewers
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
jaylouisw/packetforge!1870
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "ci/1117-1621"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What was slow/broken
Server smoke testis the only job inci.ymlwhose work step has no bound:run: node scripts/smoke-http.js. The script's client settles on the response'sendor onerror, so a connection that is accepted and never answered leaves the promise pending forever and Node waits until something outside it kills the process.Three failures today, each on a pull request whose three test legs passed the same commit in 52-67 s:
dbfs open "actions_log/.../11669.log.zst": file does not exist2,340 s of a two-slot runner in one day, and not one of them left a diagnosable log: the process was killed instead of exiting, so the runner has no
actions_logobject for the task.timeout-minutes: 10on the job is not the bound: this runner does not enforce it - all three durations exceed 600 s, and the same non-enforcement is already documented in thechangesjob (task 12980, 13m48s).The change (workflow only)
.forgejo/workflows/ci.yml, thesmokejob's last step:timeout --signal=TERM --kill-after=15 180 node scripts/smoke-http.js, with the step's own status kept and a 124/137 annotated as a hung request rather than a slow server.Baseline / after
Server smoke test, 676 s, failure, no log; task 11193 (run 4688) 872 s; task 10082 (run 4373) 792 s.Coverage unchanged
No assertion, step or job is removed, skipped or retried. A slow-but-working smoke test still passes (180 s is 6x the slowest green job in the last 24 h, 9x the median); a genuinely hung one is still red - it just fails in 3 minutes and keeps its log instead of holding a slot for 15 and leaving nothing.
Still red / not addressed here: the test legs have the same class of hang (task 11437 808 s, 11555 668 s, 11556 661 s, 11667 683 s - all with no log), and one genuine flake on main at 15:28 today: run 5320, task 13248,
test/api-config-door.test.js,Error: read ECONNRESETat 6124 ms while Node 18 and 22 passed the same commit. Both are follow-ups.0313849c8a555dd76529