main is red: property-validator fails at seed 65 because 58 of the engine's 75 error codes are not in validation.RULES #1654

Closed
opened 2026-11-16 11:21:02 +00:00 by jaylouisw · 1 comment
Owner

main is red: all three CI / Test legs fail on test/property-validator.test.js, deterministically, at the same seed. This has been true since #1647 and is still true on 969c999c (after #1649, #1650).

Repro

git checkout --detach 969c999c
node --test test/property-validator.test.js
# Fixtures: 2202 loadable specs, 90 media, 24 rule codes
# not ok - 1 property violation(s) across 720 random link attempts
#   [seed 65 d0:SFP56-3 -- d2:sfp1 media="console-rs232": unknown code E_BIDI_MISMATCH]
# 18 passed, 1 failed

Green at 30049dc6, red from 677cfb0e (#1647).

Cause

Not a behaviour change — a fixture draw shift. test/property-validator.test.js:125 asserts every code connect() returns is in validation.RULES:

const RULE_CODES = new Set(validation.RULES.map(r => (r && r.code) || r));  // line 48
if (!RULE_CODES.has(e.code) && e.code !== 'E_PORT_IN_USE') problems.push(...);  // line 125
  • validation.RULES is the connection rule table: 24 entries.
  • The engine emits 75 distinct codes under code: '...' across engine/*.js — 58 are outside that table, because config/link/optical validators each own their codes.
  • E_BIDI_MISMATCH is raised at engine/optical.js:225 (one side single-strand BiDi, the other needs two strands). It is a connect()-time error and it is not in RULES.
  • The E_PORT_IN_USE exemption on line 125 is the same bug papered over once already.

The five device specs added in #1647 changed the seeded draw so that seed 65 now lands on the BiDi pairing; removing any one of them makes the test pass. Nothing about the engine got worse — the assertion was always too narrow, and every new device spec re-rolls it.

Fix — needs a decision (both files are outside my remit)

  1. test/: assert what was meant — that e.code is a well-formed non-empty code (or the union of every code the validators can emit), and delete the E_PORT_IN_USE special case. Cheapest, no engine change.
  2. engine/: publish the media/optical family (E_BIDI_MISMATCH, E_BIDI_SAME_WAVELENGTH, E_UNKNOWN_PART, E_OPTICAL_BUDGET, …) into validation.RULES so connect()'s code surface is fully documented, and keep the strict assertion.

Until one lands, main stays red and every PR rebased onto it inherits a failing suite.

`main` is red: all three `CI / Test` legs fail on `test/property-validator.test.js`, deterministically, at the same seed. This has been true since **#1647** and is still true on `969c999c` (after #1649, #1650). ## Repro ``` git checkout --detach 969c999c node --test test/property-validator.test.js # Fixtures: 2202 loadable specs, 90 media, 24 rule codes # not ok - 1 property violation(s) across 720 random link attempts # [seed 65 d0:SFP56-3 -- d2:sfp1 media="console-rs232": unknown code E_BIDI_MISMATCH] # 18 passed, 1 failed ``` Green at `30049dc6`, red from `677cfb0e` (#1647). ## Cause Not a behaviour change — a fixture draw shift. `test/property-validator.test.js:125` asserts every code `connect()` returns is in `validation.RULES`: ```js const RULE_CODES = new Set(validation.RULES.map(r => (r && r.code) || r)); // line 48 if (!RULE_CODES.has(e.code) && e.code !== 'E_PORT_IN_USE') problems.push(...); // line 125 ``` * `validation.RULES` is the **connection** rule table: **24** entries. * The engine emits **75** distinct codes under `code: '...'` across `engine/*.js` — **58** are outside that table, because config/link/optical validators each own their codes. * `E_BIDI_MISMATCH` is raised at `engine/optical.js:225` (one side single-strand BiDi, the other needs two strands). It is a `connect()`-time error and it is not in `RULES`. * The `E_PORT_IN_USE` exemption on line 125 is the same bug papered over once already. The five device specs added in #1647 changed the seeded draw so that seed 65 now lands on the BiDi pairing; removing any one of them makes the test pass. Nothing about the engine got worse — the assertion was always too narrow, and every new device spec re-rolls it. ## Fix — needs a decision (both files are outside my remit) 1. **test/**: assert what was meant — that `e.code` is a well-formed non-empty code (or the union of every code the validators can emit), and delete the `E_PORT_IN_USE` special case. Cheapest, no engine change. 2. **engine/**: publish the media/optical family (`E_BIDI_MISMATCH`, `E_BIDI_SAME_WAVELENGTH`, `E_UNKNOWN_PART`, `E_OPTICAL_BUDGET`, …) into `validation.RULES` so `connect()`'s code surface is fully documented, and keep the strict assertion. Until one lands, `main` stays red and every PR rebased onto it inherits a failing suite.
Author
Owner

Main is green again — but not fixed. This is a mask, not a repair.

a9034db1 (#1655) is the first green tip in three: CI / Test (Node 18/20/22) all success there, while 903a03f2 and d8164131 fail. Nothing touched the validator or its test in between, so the test did not get better — the sample moved.

Verified locally, same machine, same Node:

  • d8164131: node --test test/property-validator.test.js → 19 assertions, 1 failed: [seed 65 d0:SFP56-3 -- d2:sfp1 media="console-rs232": unknown code E_BIDI_MISMATCH] (720 random link attempts).
  • a9034db1: same command → 19 passed, 0 failed.

The mechanism: the property test builds topologies with specs[Math.floor(rand() * specs.length)] over the real catalogue, so every catalogue change re-rolls what seed 65 means. #1655 added one device (2202 → 2203 specs); the sampling shifted, seed 65 no longer lands on that pair, and the unknown-code failure disappeared with it. Any future device or engine change can bring it straight back, on someone else's PR, with no obvious relationship to their work.

Scope is wider than the one code. validation.RULES carries 24 codes; the validator path (validation.js, topology.js, media.js, optical.js) emits 51, so 34 codes are outside the test's RULE_CODES gate and would each be reported as "unknown":

E_BAD_LENGTH E_BAUD_MISMATCH E_BIDI_MISMATCH E_BIDI_SAME_WAVELENGTH E_COMBO_OCCUPIED E_FLOW_BAD_RECORD E_FLOW_GHOST_ENDPOINT E_IFACE_BAD_ADDRESS E_IFACE_BAD_MAC E_IFACE_BAD_MASK E_IFACE_DUPLICATE_MAC E_IFACE_NO_MASK E_LAYOUT_BAD_ENTRY E_LAYOUT_MALFORMED E_LAYOUT_UNKNOWN_DEVICE E_LINK E_LINK_BAD_LABEL E_LINK_BAD_STATE E_LINK_DUPLICATE_ID E_LINK_ENDPOINT_MISSING E_LINK_MALFORMED E_LINK_NOT_RESTORED E_LINK_SELF_LOOP E_LINK_UNCHECKABLE E_NOT_A_TOPOLOGY E_NO_DEVICE E_NO_FREE_PORTS E_NO_LINK E_OPTICAL_BUDGET E_SAME_DEVICE E_UNKNOWN E_UNKNOWN_PART E_UNKNOWN_PON_CLASS E_UNKNOWN_SPLITTER

Some of those (E_LINK_*, E_IFACE_*, E_LAYOUT_*, E_FLOW_*) belong to import/restore and are probably never reachable from validateLink, so they may be legitimately absent. But at minimum E_BIDI_MISMATCH, E_BIDI_SAME_WAVELENGTH, E_BAUD_MISMATCH, E_OPTICAL_BUDGET, E_COMBO_OCCUPIED, E_NO_FREE_PORTS and E_SAME_DEVICE are validateLink results — E_BIDI_MISMATCH is proven reachable by the repro above.

Decision still open (unchanged from when this issue was filed — 0 comments so far):

  1. Publish the reachable codes into validation.RULES (they get severities and descriptions, and the test's gate becomes meaningful), or
  2. decide the gate should be an explicit KNOWN_CODES list separate from RULES in the test, and add the 34, or
  3. make the gate derive the code set from the engine rather than from RULES, so it can never drift.

Until one is picked, this test passes by coincidence and any catalogue PR can turn main red for reasons unrelated to its diff. Filing this, not fixing it: engine/ and test/ are not mine to edit from the surgery loop.

Surgery loop, 2026-11-16 11:53 — verified by checkout of both tips in C:/Users/jay/pf-wt/surgery; no worktrees or branches left behind.

**Main is green again — but not fixed. This is a mask, not a repair.** `a9034db1` (#1655) is the first green tip in three: `CI / Test (Node 18/20/22)` all success there, while `903a03f2` and `d8164131` fail. Nothing touched the validator or its test in between, so the test did not get better — the *sample* moved. Verified locally, same machine, same Node: - `d8164131`: `node --test test/property-validator.test.js` → **19 assertions, 1 failed**: `[seed 65 d0:SFP56-3 -- d2:sfp1 media="console-rs232": unknown code E_BIDI_MISMATCH]` (720 random link attempts). - `a9034db1`: same command → **19 passed, 0 failed**. The mechanism: the property test builds topologies with `specs[Math.floor(rand() * specs.length)]` over the real catalogue, so **every catalogue change re-rolls what seed 65 means**. #1655 added one device (2202 → 2203 specs); the sampling shifted, seed 65 no longer lands on that pair, and the unknown-code failure disappeared with it. Any future device or engine change can bring it straight back, on someone else's PR, with no obvious relationship to their work. Scope is wider than the one code. `validation.RULES` carries **24** codes; the validator path (`validation.js`, `topology.js`, `media.js`, `optical.js`) emits **51**, so **34 codes are outside the test's `RULE_CODES` gate** and would each be reported as "unknown": `E_BAD_LENGTH E_BAUD_MISMATCH E_BIDI_MISMATCH E_BIDI_SAME_WAVELENGTH E_COMBO_OCCUPIED E_FLOW_BAD_RECORD E_FLOW_GHOST_ENDPOINT E_IFACE_BAD_ADDRESS E_IFACE_BAD_MAC E_IFACE_BAD_MASK E_IFACE_DUPLICATE_MAC E_IFACE_NO_MASK E_LAYOUT_BAD_ENTRY E_LAYOUT_MALFORMED E_LAYOUT_UNKNOWN_DEVICE E_LINK E_LINK_BAD_LABEL E_LINK_BAD_STATE E_LINK_DUPLICATE_ID E_LINK_ENDPOINT_MISSING E_LINK_MALFORMED E_LINK_NOT_RESTORED E_LINK_SELF_LOOP E_LINK_UNCHECKABLE E_NOT_A_TOPOLOGY E_NO_DEVICE E_NO_FREE_PORTS E_NO_LINK E_OPTICAL_BUDGET E_SAME_DEVICE E_UNKNOWN E_UNKNOWN_PART E_UNKNOWN_PON_CLASS E_UNKNOWN_SPLITTER` Some of those (`E_LINK_*`, `E_IFACE_*`, `E_LAYOUT_*`, `E_FLOW_*`) belong to import/restore and are probably never reachable from `validateLink`, so they may be legitimately absent. But at minimum `E_BIDI_MISMATCH`, `E_BIDI_SAME_WAVELENGTH`, `E_BAUD_MISMATCH`, `E_OPTICAL_BUDGET`, `E_COMBO_OCCUPIED`, `E_NO_FREE_PORTS` and `E_SAME_DEVICE` are `validateLink` results — `E_BIDI_MISMATCH` is *proven* reachable by the repro above. Decision still open (unchanged from when this issue was filed — 0 comments so far): 1. **Publish the reachable codes into `validation.RULES`** (they get severities and descriptions, and the test's gate becomes meaningful), or 2. **decide the gate should be an explicit `KNOWN_CODES` list separate from `RULES`** in the test, and add the 34, or 3. **make the gate derive the code set from the engine** rather than from `RULES`, so it can never drift. Until one is picked, this test passes by coincidence and any catalogue PR can turn main red for reasons unrelated to its diff. Filing this, not fixing it: `engine/` and `test/` are not mine to edit from the surgery loop. *Surgery loop, 2026-11-16 11:53 — verified by checkout of both tips in `C:/Users/jay/pf-wt/surgery`; no worktrees or branches left behind.*
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jaylouisw/packetforge#1654
No description provided.