ci: run the suite 10-way, not 6-way (measured 60.4 s -> 38.3 s per leg) #1706

Merged
jaylouisw merged 2 commits from ci/1116-1820 into main 2026-11-16 18:33:18 +00:00
Owner

The suite is the whole wall clock of every Test (Node *) leg, and the number of processes it is allowed to overlap was chosen for 281 suites: 6. There are 285 now, and the same host has 40 cores.

Baseline (measured, not assumed)

Same commit eb172083, node:20-bookworm in the runner's own image on odin, node scripts/run-tests.js:

concurrency one leg alone three legs started at the same second
6 (current) 31.9 s 60.5 / 60.1 / 60.6 s
10 21.9 s 38.3 / 38.2 / 38.3 s
12 20.6 s -

Every one of those runs: 285 suites, 10234 assertions, 0 failed. The three-leg row is the one that matters - it is the shape the matrix actually takes on a three-slot runner - and it is 37% off the leg.

Caveat kept honest: the only failure seen in the benchmark was canonical-urls' "the tree is clean again once the probe is removed", which fails identically at 6 and at 10 and only in the benchmark copies (they are cp -a of a shallow clone, not a runner checkout). Real legs on this commit are green.

After

PF_TEST_CONCURRENCY: 6 -> 10 in the one step that runs the suite. 10 rather than 12: 12 buys 1.3 s on an idle host and allocates a fifth more processes under the contention where the cost is actually paid. The harness caps at 16 and nothing else in the workflow changes.

Coverage unchanged: all three Node legs still run all 285 suites on 18, 20 and 22, plus the syntax check in parallel, the example labs and the missing-devices scan. Nothing is skipped, nothing is filtered, no timeout is touched.

Attribution: a further +1-2 s per leg goes to the duplicated check-control-bytes.js (it is both a step and the second command of npm test) - left alone because it also runs inside npm test on purpose, as a gate that does not depend on the step graph.

The suite is the whole wall clock of every `Test (Node *)` leg, and the number of processes it is allowed to overlap was chosen for 281 suites: 6. There are 285 now, and the same host has 40 cores. ## Baseline (measured, not assumed) Same commit `eb172083`, node:20-bookworm in the runner's own image on odin, `node scripts/run-tests.js`: | concurrency | one leg alone | three legs started at the same second | |---|---|---| | 6 (current) | 31.9 s | 60.5 / 60.1 / 60.6 s | | 10 | 21.9 s | 38.3 / 38.2 / 38.3 s | | 12 | 20.6 s | - | Every one of those runs: 285 suites, 10234 assertions, 0 failed. The three-leg row is the one that matters - it is the shape the matrix actually takes on a three-slot runner - and it is 37% off the leg. Caveat kept honest: the only failure seen in the benchmark was `canonical-urls`' "the tree is clean again once the probe is removed", which fails identically at 6 and at 10 and only in the benchmark copies (they are `cp -a` of a shallow clone, not a runner checkout). Real legs on this commit are green. ## After `PF_TEST_CONCURRENCY: 6` -> `10` in the one step that runs the suite. 10 rather than 12: 12 buys 1.3 s on an idle host and allocates a fifth more processes under the contention where the cost is actually paid. The harness caps at 16 and nothing else in the workflow changes. Coverage unchanged: all three Node legs still run all 285 suites on 18, 20 and 22, plus the syntax check in parallel, the example labs and the missing-devices scan. Nothing is skipped, nothing is filtered, no timeout is touched. Attribution: a further +1-2 s per leg goes to the duplicated `check-control-bytes.js` (it is both a step and the second command of `npm test`) - left alone because it also runs inside `npm test` on purpose, as a gate that does not depend on the step graph.
ci: run the suite 10-way, not 6-way (measured: 60.4 s -> 38.3 s per leg under matrix contention)
All checks were successful
CI / Classify the change (pull_request) Successful in 18s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 20s
CI / Server smoke test (pull_request) Successful in 17s
CI / Test (Node 20) (pull_request) Successful in 51s
CI / Test (Node 18) (pull_request) Successful in 1m0s
CI / Test (Node 22) (pull_request) Successful in 54s
CI / Docker build (pull_request) Successful in 26s
ead50d61f7
Same 285 suites and the same 10234 assertions in every leg; only the number of
concurrent processes changes. Benchmarked in the runner's own node:20-bookworm
image on odin against commit eb172083, one leg alone: 31.9 s at 6, 21.9 s at 10,
20.6 s at 12. Three legs started at the same second with private workdirs, which
is what the matrix does: 60.5/60.1/60.6 s at 6, 38.3/38.2/38.3 s at 10.
jaylouisw scheduled this pull request to auto merge when all checks succeed 2026-11-16 18:33:10 +00:00
jaylouisw force-pushed ci/1116-1820 from ead50d61f7
All checks were successful
CI / Classify the change (pull_request) Successful in 18s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 20s
CI / Server smoke test (pull_request) Successful in 17s
CI / Test (Node 20) (pull_request) Successful in 51s
CI / Test (Node 18) (pull_request) Successful in 1m0s
CI / Test (Node 22) (pull_request) Successful in 54s
CI / Docker build (pull_request) Successful in 26s
to 04b4558e77
All checks were successful
CI / Classify the change (pull_request) Successful in 13s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 14s
CI / Test (Node 18) (pull_request) Has been skipped
CI / Test (Node 20) (pull_request) Has been skipped
CI / Test (Node 22) (pull_request) Has been skipped
CI / Server smoke test (pull_request) Has been skipped
CI / Docker build (pull_request) Has been skipped
2026-11-16 18:33:14 +00:00
Compare
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jaylouisw/packetforge!1706
No description provided.