Twenty suites still cap a server start at 20 s, and one of them failed a 10-way run #1720

Closed
opened 2026-11-16 20:14:39 +00:00 by jaylouisw · 0 comments
Owner

The measurement

PF_TEST_CONCURRENCY=10 node scripts/run-tests.js on f4e22080, 2026-11-16 20:05 UTC:

287 suite(s) in 135.9 s — 10227 assertions passed, 1 failed

Failed:
  ✗ test/board-id-is-a-name.test.js

Not an assertion — a crash, in the restart half of the suite (line 83), after the deadline expired:

test crashed: Error: server did not answer: connect ECONNREFUSED 127.0.0.1:16131 |
    at test/board-id-is-a-name.test.js:83:50

The | is the child's captured output, and it is empty: the second server never printed its banner
in 20 s, so it had not reached listen.

Why 20 s is the wrong number

the same suite alone 15.2 s, 65 assertions, pass
the same suite, 10 copies at once 15.2 s each, 10/10 pass
the same suite inside the 10-way run 26.8 s (next run, passed), 27.8 s (the failing run)
10 servers started at once, /api/catalog 200 3.5-4.8 s, max 4.8 s

So a start is 4.8 s idle-load and this suite spends ~11 s more than that just being slow inside a
10-way run, which leaves the restart's 20 s cap a couple of seconds of headroom. It missed once in
10227 assertions, which is exactly how a red CI leg looks: nothing is broken, the bound is too tight.

#1714 measured this and #1715 set the bound to 60 s for 13 suites. These 20 still say 20 s — the
class was fixed where it had already failed, not where it was about to.

The fix

  1. Every server-start wait in test/*.test.js gets one minute, so a slow machine fails for being
    broken and not for being busy.
  2. A guard, so the next new suite cannot pick 20 s again: no Date.now() + <literal> under a minute
    anywhere in test/.
  3. board-id-is-a-name gains the #1715 assertion — a port nothing is listening on fails at the
    deadline it was given
    , not after a fixed number of tries — because a bound nothing exercises is
    how this class came back.

Found in the issues loop; no code under test is at fault.

## The measurement `PF_TEST_CONCURRENCY=10 node scripts/run-tests.js` on `f4e22080`, 2026-11-16 20:05 UTC: ``` 287 suite(s) in 135.9 s — 10227 assertions passed, 1 failed Failed: ✗ test/board-id-is-a-name.test.js ``` Not an assertion — a crash, in the *restart* half of the suite (line 83), after the deadline expired: ``` test crashed: Error: server did not answer: connect ECONNREFUSED 127.0.0.1:16131 | at test/board-id-is-a-name.test.js:83:50 ``` The `|` is the child's captured output, and it is **empty**: the second server never printed its banner in 20 s, so it had not reached `listen`. ## Why 20 s is the wrong number | | | |---|---| | the same suite alone | 15.2 s, 65 assertions, pass | | the same suite, 10 copies at once | 15.2 s each, 10/10 pass | | the same suite inside the 10-way run | **26.8 s** (next run, passed), 27.8 s (the failing run) | | 10 servers started at once, `/api/catalog` 200 | 3.5-4.8 s, max **4.8 s** | So a start is 4.8 s idle-load and this suite spends ~11 s more than that just being slow inside a 10-way run, which leaves the restart's 20 s cap a couple of seconds of headroom. It missed once in 10227 assertions, which is exactly how a red CI leg looks: nothing is broken, the bound is too tight. #1714 measured this and #1715 set the bound to **60 s** for 13 suites. These 20 still say 20 s — the class was fixed where it had already failed, not where it was about to. ## The fix 1. Every server-start wait in `test/*.test.js` gets one minute, so a slow machine fails for being broken and not for being busy. 2. A guard, so the next new suite cannot pick 20 s again: no `Date.now() + <literal>` under a minute anywhere in `test/`. 3. `board-id-is-a-name` gains the #1715 assertion — a port nothing is listening on fails at *the deadline it was given*, not after a fixed number of tries — because a bound nothing exercises is how this class came back. Found in the issues loop; no code under test is at fault.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jaylouisw/packetforge#1720
No description provided.