A test that waits says what it waited for (#1415) #1416

Merged
jaylouisw merged 1 commit from fix/ws-flake into main 2026-11-11 02:59:18 +00:00
Owner

Closes #1415.

CI's Test (Node 22) went red on ebc81a9e and again on the #1414 pull request while Node 18 and Node 20 stayed green, and the entire diagnosis available from the run was:

Error: no matching frame within 8000ms

Reproduced here under Node 22.14: one full-suite run failed state-payload-shape (0 passed, 1 failed, no summary line); the next two runs of the same suite on the same build passed 21/21. A flake - but the message names neither the frames the client received nor anything the server said, so no run could ever prove that.

Two changes, both in the harness:

  • a timed-out wait reports what it saw - the frame types received on that connection, plus the tail of the child server's own log;
  • the connect frame is retried once on a fresh socket - a cold socket that missed the first frame and a server that never answers look identical from the client, and a second attempt tells them apart. A retry that was needed prints note: the connect frame needed a second socket (...), because a flake silently absorbed is a flake nobody fixes.

Verified: state-payload-shape 21/21 on Node 24 and on Node 22.14; full suite 208 suites, 7,989 assertions, 0 failed on both; syntax 276 files OK.

Closes #1415. CI's `Test (Node 22)` went red on `ebc81a9e` and again on the #1414 pull request while Node 18 and Node 20 stayed green, and the entire diagnosis available from the run was: ``` Error: no matching frame within 8000ms ``` Reproduced here under Node 22.14: one full-suite run failed `state-payload-shape` (0 passed, 1 failed, no summary line); the next two runs of the same suite on the same build passed 21/21. A flake - but the message names neither the frames the client received nor anything the server said, so no run could ever prove that. Two changes, both in the harness: - **a timed-out wait reports what it saw** - the frame types received on that connection, plus the tail of the child server's own log; - **the connect frame is retried once on a fresh socket** - a cold socket that missed the first frame and a server that never answers look identical from the client, and a second attempt tells them apart. A retry that was needed prints `note: the connect frame needed a second socket (...)`, because a flake silently absorbed is a flake nobody fixes. **Verified:** `state-payload-shape` 21/21 on Node 24 and on Node 22.14; full suite **208 suites, 7,989 assertions, 0 failed** on both; syntax 276 files OK.
A test that waits says what it waited for (#1415)
All checks were successful
CI / Classify the change (pull_request) Successful in 9s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 11s
CI / Test (Node 20) (pull_request) Successful in 2m39s
CI / Test (Node 18) (pull_request) Successful in 2m48s
CI / Server smoke test (pull_request) Successful in 25s
CI / Docker build (pull_request) Successful in 42s
CI / Test (Node 22) (pull_request) Successful in 2m40s
6e3928f4f2
CI's Node 22 container went red on ebc81a9e and again on the #1414 pull request, and both times the only thing the log said was a screenshot of a stack trace: 'no matching frame within 8000ms'. The suite that failed (state-payload-shape) waits for the server's first WebSocket frame, then for a broadcast after a mutation; when the wait expires it rejected with a sentence that named neither the frames it had received nor what the server was doing.

Reproduced here: a full run under Node 22.14 failed state-payload-shape once, and the next two runs of the same suite, the same build, passed - a flake, not a regression, and not something the failure message could ever have proved.

Two changes, both to the harness:

- a timed-out wait now reports the frames it did receive (by type) and the tail of the server's own log, so the next occurrence explains itself;
- the connect frame is retried once on a fresh socket - a cold socket that missed the first frame and a server that never answers look identical from the client, and a second attempt tells them apart. A retry that was needed prints a note, because a flake silently absorbed is a flake nobody fixes.

Verified: state-payload-shape 21/21 on Node 24 and on Node 22.14; full suite 208 suites, 7,989 assertions, 0 failed, on both.
jaylouisw scheduled this pull request to auto merge when all checks succeed 2026-11-11 02:59:14 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jaylouisw/packetforge!1416
No description provided.