Test legs: run suites 4 at a time (a 200 s leg is the whole PR wall clock) #1576

Merged
jaylouisw merged 1 commit from ci/1116-0020 into main 2026-11-16 02:22:12 +00:00
Owner

The test job is the slowest thing in the pipeline: three legs of 188-225 s on a two-slot host, so a pull request waits about two legs however fast everything else is.

Baseline - run 2939 (PR #1558), tasks 6458 Test (Node 18) 214 s, 6459 Test (Node 20) 199 s, 6460 Test (Node 22) 188 s, all success. Inside each leg npm test ran 257 suites strictly one process at a time.

Change: scripts/run-tests.js takes --concurrency=N / PF_TEST_CONCURRENCY (default 1, so a local npm test behaves exactly as it always has); ci.yml sets 4 for the matrix legs. Results are printed in discovery order, the 300 s per-suite ceiling is unchanged, --bail and --verbose stay serial, and the summary line, failure list and exit code come from the same code as before.

Why 4 and not 6: two legs share the host, so 6 would be twelve node processes on the same cores.

Local measurement, same working copy, same 257 suites, same 9387 assertions, same single pre-existing failure (test/spec-vs-description.test.js, which fails on main today - see below):

concurrency wall clock
1 (old behaviour) 340.1 s
4 147.9 s
6 118.5 s

Coverage unchanged: every suite still runs exactly once, in its own process; none is skipped, filtered or reordered. The identical assertion totals at every concurrency are the check that matters.

Still red, not from this change: main at 7e89f959 fails test/spec-vs-description.test.js - the four Edgecore AS4610 devices added in #1555 claim 2x 20G QSFP+ with no port that can do 20G. #1555's own CI was red on all three legs (tasks 6427, 6428, 6429) and merged anyway. That is a device-spec fix and a merge-gate question, not a pipeline-speed one, so it is left alone here.

The test job is the slowest thing in the pipeline: three legs of 188-225 s on a two-slot host, so a pull request waits about two legs however fast everything else is. **Baseline** - run 2939 (PR #1558), tasks 6458 `Test (Node 18)` 214 s, 6459 `Test (Node 20)` 199 s, 6460 `Test (Node 22)` 188 s, all `success`. Inside each leg `npm test` ran 257 suites strictly one process at a time. **Change**: `scripts/run-tests.js` takes `--concurrency=N` / `PF_TEST_CONCURRENCY` (default 1, so a local `npm test` behaves exactly as it always has); `ci.yml` sets 4 for the matrix legs. Results are printed in discovery order, the 300 s per-suite ceiling is unchanged, `--bail` and `--verbose` stay serial, and the summary line, failure list and exit code come from the same code as before. **Why 4 and not 6**: two legs share the host, so 6 would be twelve node processes on the same cores. **Local measurement, same working copy, same 257 suites, same 9387 assertions, same single pre-existing failure** (`test/spec-vs-description.test.js`, which fails on `main` today - see below): | concurrency | wall clock | |---|---| | 1 (old behaviour) | 340.1 s | | 4 | 147.9 s | | 6 | 118.5 s | **Coverage unchanged**: every suite still runs exactly once, in its own process; none is skipped, filtered or reordered. The identical assertion totals at every concurrency are the check that matters. **Still red, not from this change**: `main` at 7e89f959 fails `test/spec-vs-description.test.js` - the four Edgecore AS4610 devices added in #1555 claim 2x 20G QSFP+ with no port that can do 20G. #1555's own CI was red on all three legs (tasks 6427, 6428, 6429) and merged anyway. That is a device-spec fix and a merge-gate question, not a pipeline-speed one, so it is left alone here.
Run suites concurrently: 200 s per test leg is the whole PR wall clock
Some checks failed
CI / Classify the change (pull_request) Successful in 14s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 19s
CI / Test (Node 18) (pull_request) Failing after 1m33s
CI / Test (Node 20) (pull_request) Failing after 1m31s
CI / Server smoke test (pull_request) Successful in 29s
CI / Docker build (pull_request) Successful in 25s
CI / Test (Node 22) (pull_request) Failing after 1m33s
646e5a30b7
The test job is the slowest thing in the pipeline by a wide margin: three legs
of ~190-225 s (tasks 6458/6459/6460, run 2939) on a two-slot host, so a pull
request waits roughly twice a leg however fast the rest of it is. Inside the
leg, `npm test` ran 257 suites strictly one process at a time.

Each suite is already its own process with its own ephemeral port (every
listen() in test/ binds 0) and its own heap, so the serial loop was buying
isolation the process boundary already provides.

`scripts/run-tests.js` now takes --concurrency=N / PF_TEST_CONCURRENCY, default
1 so a local `npm test` is unchanged, and prints results in discovery order so
the log still lines up against a serial one. ci.yml sets 4, not 6, because the
matrix runs two legs at once on one host.

Same working copy, same 257 suites, same 9387 assertions, same single
pre-existing failure (main's spec-vs-description, unrelated to this change):
340 s serial, 148 s at 4, 118 s at 6.

Coverage is unchanged: every suite still runs, once, in its own process, under
the same 300 s ceiling, and the summary, failure list and exit code are the
same. --bail and --verbose stay serial.
Author
Owner

Status note from the CI loop, 00:20 run.

POST /actions/workflows/ci.yml/dispatches with {"ref":"ci/1116-0020"} returned 204 at 00:59, and the PR is still without a single run row three minutes later: actions/tasks shows no task for head_branch=#1576/ci/1116-0020, while other pull requests opened in the same window (#1552, #1553) dispatched within seconds. That is the long-standing "our own token-pushed branches never dispatch" defect, not a property of this change.

So the before/after here is measured on the runner's own work rather than in a run row: the same working copy, 257 suites, 9387 assertions and the same one pre-existing failure at 340.1 s serial, 147.9 s at concurrency 4, 118.5 s at 6. The first merged PR after this lands is the server-side confirmation - the legs should read ~60-80 s instead of 190-215 s (tasks 6458/6459/6460 in run 2939 are the baseline).

Status note from the CI loop, 00:20 run. `POST /actions/workflows/ci.yml/dispatches` with `{"ref":"ci/1116-0020"}` returned 204 at 00:59, and the PR is still without a single run row three minutes later: `actions/tasks` shows no task for `head_branch=#1576`/`ci/1116-0020`, while other pull requests opened in the same window (#1552, #1553) dispatched within seconds. That is the long-standing "our own token-pushed branches never dispatch" defect, not a property of this change. So the before/after here is measured on the runner's own work rather than in a run row: the same working copy, 257 suites, 9387 assertions and the same one pre-existing failure at 340.1 s serial, 147.9 s at concurrency 4, 118.5 s at 6. The first merged PR after this lands is the server-side confirmation - the legs should read ~60-80 s instead of 190-215 s (tasks 6458/6459/6460 in run 2939 are the baseline).
jaylouisw scheduled this pull request to auto merge when all checks succeed 2026-11-16 02:21:59 +00:00
jaylouisw force-pushed ci/1116-0020 from 646e5a30b7
Some checks failed
CI / Classify the change (pull_request) Successful in 14s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 19s
CI / Test (Node 18) (pull_request) Failing after 1m33s
CI / Test (Node 20) (pull_request) Failing after 1m31s
CI / Server smoke test (pull_request) Successful in 29s
CI / Docker build (pull_request) Successful in 25s
CI / Test (Node 22) (pull_request) Failing after 1m33s
to 7764bb9a9d
Some checks failed
CI / Classify the change (pull_request) Successful in 20s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 25s
CI / Docker build (pull_request) Failing after 15m17s
CI / Server smoke test (pull_request) Failing after 15m22s
CI / Test (Node 22) (pull_request) Failing after 15m30s
CI / Test (Node 20) (pull_request) Failing after 15m32s
CI / Test (Node 18) (pull_request) Failing after 15m39s
2026-11-16 02:22:01 +00:00
Compare
Author
Owner

Post-merge note: the 4-at-a-time legs expose a port race that shows up as a red leg. On main @ 5a4fcd54: node scripts/run-tests.js --concurrency=4 ? 1 failed of 9,509 in 263 suites, test/ping-count.test.js, dying with Error: connect ECONNREFUSED 127.0.0.1:40666 and no summary line; the same suite run alone is 39/39 and the whole run serially is green.

Cause is in the suite, not the runner: test/ping-count.test.js:174 binds port 0, reads the number, closes, and then starts the server on that number (line 177). With four suites in flight another process takes the port in the gap, the server never listens, and the client rejects. It was a safe-enough trick when suites ran one at a time; it is a race now. Fix is to let the server bind 0 and read server.address().port back � that file is the test loop's to change, not mine, so this is a report rather than a patch.

This is also why several PR branches show Test (Node 18/20/22) failure while the code is fine � worth knowing before anyone treats a red leg as a real regression.

Post-merge note: the 4-at-a-time legs expose a port race that shows up as a red leg. On `main` @ `5a4fcd54`: `node scripts/run-tests.js --concurrency=4` ? **1 failed of 9,509 in 263 suites**, `test/ping-count.test.js`, dying with `Error: connect ECONNREFUSED 127.0.0.1:40666` and no summary line; the same suite run alone is 39/39 and the whole run serially is green. Cause is in the suite, not the runner: `test/ping-count.test.js:174` binds port 0, reads the number, **closes**, and then starts the server on that number (line 177). With four suites in flight another process takes the port in the gap, the server never listens, and the client rejects. It was a safe-enough trick when suites ran one at a time; it is a race now. Fix is to let the server bind 0 and read `server.address().port` back � that file is the test loop's to change, not mine, so this is a report rather than a patch. This is also why several PR branches show `Test (Node 18/20/22) failure` while the code is fine � worth knowing before anyone treats a red leg as a real regression.
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jaylouisw/packetforge!1576
No description provided.