server-immutable-json fails under load: a 6-second readiness wait and no diagnosis #1242

Closed
opened 2026-11-09 17:50:55 +00:00 by jaylouisw · 0 comments
Owner

What happens

test/server-immutable-json.test.js fails intermittently on a loaded machine:

✗ server-immutable-json   0 passed, 1 failed   7304 ms (no summary line)
    Error: server did not start
      at test/server-immutable-json.test.js:75

It passes when run alone (21 checks). The suite is not flaky in what it asserts — it is flaky in how long
it is willing to wait:

function waitReady(port) {
  let left = 60;                       // 60 attempts x 100 ms = ~6 s, on a machine already running
  ...if (--left <= 0) return reject(new Error('server did not start'));
}

The child it starts is server/index.js, which loads and serialises a 4.4 MB catalogue before it listens.
Six seconds is enough for that on an idle machine and not enough when the test runner is executing other
suites. The failure also reports nothing about why: the child's stderr is discarded, so a genuine startup
failure and a slow startup look identical.

Why it matters

A red suite is what tells everyone the last change broke something. A test that goes red under load
trains people to re-run until green, and the next real failure gets re-run too. It also costs the
automation: the loop this repo is developed under treats a red suite as a stop signal.

Expected

  • wait against a wall-clock deadline (generous: a minute), not an attempt count;
  • on failure, report the deadline and the child's last output, so "did not start" becomes a diagnosis;
  • a regression check that a deliberately slow server is still awaited, and that a port that never opens
    fails with a message naming the deadline.

Scope

The readiness wait in one test file. Nothing about the endpoints or their caching changes.

## What happens `test/server-immutable-json.test.js` fails intermittently on a loaded machine: ``` ✗ server-immutable-json 0 passed, 1 failed 7304 ms (no summary line) Error: server did not start at test/server-immutable-json.test.js:75 ``` It passes when run alone (21 checks). The suite is not flaky in what it asserts — it is flaky in how long it is willing to wait: ```js function waitReady(port) { let left = 60; // 60 attempts x 100 ms = ~6 s, on a machine already running ...if (--left <= 0) return reject(new Error('server did not start')); } ``` The child it starts is `server/index.js`, which loads and serialises a 4.4 MB catalogue before it listens. Six seconds is enough for that on an idle machine and not enough when the test runner is executing other suites. The failure also reports nothing about *why*: the child's stderr is discarded, so a genuine startup failure and a slow startup look identical. ## Why it matters A red suite is what tells everyone the last change broke something. A test that goes red under load trains people to re-run until green, and the next real failure gets re-run too. It also costs the automation: the loop this repo is developed under treats a red suite as a stop signal. ## Expected * wait against a **wall-clock deadline** (generous: a minute), not an attempt count; * on failure, report the deadline and the child's last output, so "did not start" becomes a diagnosis; * a regression check that a deliberately slow server is still awaited, and that a port that never opens fails with a message naming the deadline. ## Scope The readiness wait in one test file. Nothing about the endpoints or their caching changes.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jaylouisw/packetforge#1242
No description provided.