Thirteen test suites give up on a server start after a fixed number of tries, so a 10-way CI leg goes red on a slow start #1714

Closed
opened 2026-11-16 19:37:32 +00:00 by jaylouisw · 0 comments
Owner

The measurement

server/index.js needs time to answer /api/version: 2757, 2955, 3084, 2980, 2952 ms over five starts on an idle host. Start ten at once - which is what PF_TEST_CONCURRENCY=10 does, the value .forgejo/workflows/ci.yml took today in f6fd08ba - and every one of them takes 6976-7410 ms.

Thirteen suites decide the server is broken by counting attempts rather than by how much time has passed:

pattern cap suites
for (let i = 0; i < 60; i++) { try ... break } catch { sleep 100 } 6 s, and then falls through silently link-instant, ping-count, ping-instant, ping-surface
waitReady: let left = 60; if (--left <= 0) reject('server did not start') 6 s server-compression, server-static-cache
inline if (i > 150) throw new Error('server did not start') 15 s clock, api-change-clock, console-over-http, counter-payload, layout-roundtrip, link-roundtrip, runtime-agreement

So a loaded host turns a slow start into a red leg, and the four silent ones fail as ECONNREFUSED 127.0.0.1:6904 on the next request, which reads like a missing endpoint rather than a server that was not up.

Reproduced

PF_TEST_CONCURRENCY=10 node scripts/run-tests.js on origin/main @ 53daa3a4:

  • run 1: 285 suite(s) in 202.7 s - 10155 assertions passed, 3 failed - board-id-is-a-name, server-compression, server-static-cache
  • run 2: 285 suite(s) in ... s - 6 failed - link-instant, ping-count, ping-instant, ping-surface, server-compression, server-static-cache

All thirteen pass individually, serially. The red legs on main today are the same defect seen from the outside: Test (Node 20) at 18:54 on 53daa3a4, which is a docs-only diff, with Node 18 and 22 green on the same sha; all three legs at 11:23 on d8164131; Test (Node 18) at 16:11 on 394e1389.

The fix

The pattern #1242 already established in test/server-immutable-json.test.js: a wall-clock deadline (60 s) instead of an attempt count, and a failure that names what the last attempt saw - server did not start within 60000 ms (last error: ECONNREFUSED). Applied to the thirteen suites above, same pass condition, same coverage, and the two waitReady files gain an assertion that a closed port still fails at the deadline it was given (400 ms) so the counting behaviour cannot come back unnoticed.

Filed from a worker run; the pull request follows.

## The measurement `server/index.js` needs time to answer `/api/version`: **2757, 2955, 3084, 2980, 2952 ms** over five starts on an idle host. Start ten at once - which is what `PF_TEST_CONCURRENCY=10` does, the value `.forgejo/workflows/ci.yml` took today in f6fd08ba - and every one of them takes **6976-7410 ms**. Thirteen suites decide the server is broken by *counting attempts* rather than by how much time has passed: | pattern | cap | suites | |---|---|---| | `for (let i = 0; i < 60; i++) { try ... break } catch { sleep 100 }` | 6 s, **and then falls through silently** | link-instant, ping-count, ping-instant, ping-surface | | `waitReady`: `let left = 60; if (--left <= 0) reject('server did not start')` | 6 s | server-compression, server-static-cache | | inline `if (i > 150) throw new Error('server did not start')` | 15 s | clock, api-change-clock, console-over-http, counter-payload, layout-roundtrip, link-roundtrip, runtime-agreement | So a loaded host turns a slow start into a red leg, and the four silent ones fail as `ECONNREFUSED 127.0.0.1:6904` on the *next* request, which reads like a missing endpoint rather than a server that was not up. ## Reproduced `PF_TEST_CONCURRENCY=10 node scripts/run-tests.js` on `origin/main` @ `53daa3a4`: * run 1: `285 suite(s) in 202.7 s - 10155 assertions passed, 3 failed` - board-id-is-a-name, server-compression, server-static-cache * run 2: `285 suite(s) in ... s - 6 failed` - link-instant, ping-count, ping-instant, ping-surface, server-compression, server-static-cache All thirteen pass individually, serially. The red legs on `main` today are the same defect seen from the outside: `Test (Node 20)` at 18:54 on `53daa3a4`, which is a docs-only diff, with Node 18 and 22 green on the same sha; all three legs at 11:23 on `d8164131`; `Test (Node 18)` at 16:11 on `394e1389`. ## The fix The pattern #1242 already established in `test/server-immutable-json.test.js`: a **wall-clock deadline** (60 s) instead of an attempt count, and a failure that names what the last attempt saw - `server did not start within 60000 ms (last error: ECONNREFUSED)`. Applied to the thirteen suites above, same pass condition, same coverage, and the two `waitReady` files gain an assertion that a closed port still fails at the deadline it was given (400 ms) so the counting behaviour cannot come back unnoticed. Filed from a worker run; the pull request follows.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jaylouisw/packetforge#1714
No description provided.