Test legs: run suites 4 at a time (a 200 s leg is the whole PR wall clock) #1576
No reviewers
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
jaylouisw/packetforge!1576
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "ci/1116-0020"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The test job is the slowest thing in the pipeline: three legs of 188-225 s on a two-slot host, so a pull request waits about two legs however fast everything else is.
Baseline - run 2939 (PR #1558), tasks 6458
Test (Node 18)214 s, 6459Test (Node 20)199 s, 6460Test (Node 22)188 s, allsuccess. Inside each legnpm testran 257 suites strictly one process at a time.Change:
scripts/run-tests.jstakes--concurrency=N/PF_TEST_CONCURRENCY(default 1, so a localnpm testbehaves exactly as it always has);ci.ymlsets 4 for the matrix legs. Results are printed in discovery order, the 300 s per-suite ceiling is unchanged,--bailand--verbosestay serial, and the summary line, failure list and exit code come from the same code as before.Why 4 and not 6: two legs share the host, so 6 would be twelve node processes on the same cores.
Local measurement, same working copy, same 257 suites, same 9387 assertions, same single pre-existing failure (
test/spec-vs-description.test.js, which fails onmaintoday - see below):Coverage unchanged: every suite still runs exactly once, in its own process; none is skipped, filtered or reordered. The identical assertion totals at every concurrency are the check that matters.
Still red, not from this change:
mainat7e89f959failstest/spec-vs-description.test.js- the four Edgecore AS4610 devices added in #1555 claim 2x 20G QSFP+ with no port that can do 20G. #1555's own CI was red on all three legs (tasks 6427, 6428, 6429) and merged anyway. That is a device-spec fix and a merge-gate question, not a pipeline-speed one, so it is left alone here.Status note from the CI loop, 00:20 run.
POST /actions/workflows/ci.yml/dispatcheswith{"ref":"ci/1116-0020"}returned 204 at 00:59, and the PR is still without a single run row three minutes later:actions/tasksshows no task forhead_branch=#1576/ci/1116-0020, while other pull requests opened in the same window (#1552, #1553) dispatched within seconds. That is the long-standing "our own token-pushed branches never dispatch" defect, not a property of this change.So the before/after here is measured on the runner's own work rather than in a run row: the same working copy, 257 suites, 9387 assertions and the same one pre-existing failure at 340.1 s serial, 147.9 s at concurrency 4, 118.5 s at 6. The first merged PR after this lands is the server-side confirmation - the legs should read ~60-80 s instead of 190-215 s (tasks 6458/6459/6460 in run 2939 are the baseline).
646e5a30b77764bb9a9dPost-merge note: the 4-at-a-time legs expose a port race that shows up as a red leg. On
main@5a4fcd54:node scripts/run-tests.js --concurrency=4? 1 failed of 9,509 in 263 suites,test/ping-count.test.js, dying withError: connect ECONNREFUSED 127.0.0.1:40666and no summary line; the same suite run alone is 39/39 and the whole run serially is green.Cause is in the suite, not the runner:
test/ping-count.test.js:174binds port 0, reads the number, closes, and then starts the server on that number (line 177). With four suites in flight another process takes the port in the gap, the server never listens, and the client rejects. It was a safe-enough trick when suites ran one at a time; it is a race now. Fix is to let the server bind 0 and readserver.address().portback � that file is the test loop's to change, not mine, so this is a report rather than a patch.This is also why several PR branches show
Test (Node 18/20/22) failurewhile the code is fine � worth knowing before anyone treats a red leg as a real regression.