Audit closed requests against the catalogue: the six wrong closes were invisible by design #1210

Merged
jaylouisw merged 1 commit from feat/closed-issue-audit into main 2026-11-09 13:45:43 +00:00
Owner

Six requests were closed as "impossible to model - no Ethernet port" while every one of those devices was already in devices/: #54, #167, #360, #362, #374, #387. The sweeper could not catch it, and the reason is structural: it only ever looks at open issues. Closing a request removes it from every automated view, so a wrong close is permanent, invisible, and looks exactly like work that was done.

scripts/check-closed-issues.js walks the other side of the ledger:

verdict meaning
answered the catalogue has the device and a closing comment names the file
silent the catalogue has the device, no comment names it - the shape a wrong close has
unmatched nothing in the catalogue answers the subject; most closed issues, not a finding

Against the live tracker (120 most recent closed device requests): 52 answered with evidence, 0 silent, 68 unmatched.

Three decisions worth naming, because each is a way the audit could become the problem it looks for:

  • Exact matches only. A fuzzy match claiming "already answered" by a device the issue did not name is the same error in the other direction.
  • Device requests only ([Device]: ...) unless --all. A bug report mentioning an OLT - "OLTs promise 10G SFP+ uplinks they do not model" - matched an OLT and produced a finding that meant nothing.
  • A missing token is a skip, not a failure. Runs on push to main (schedules never reach a runner here) and never fails the build: a report nobody can act on because the pipeline went red is worse than no report.

It never reopens or closes anything - the action stays with a person; this only makes sure the person is looking. Documented in docs/RESOLUTIONS.md.

17 checks in test/closed-issues-audit.test.js, including all six real cases against the live catalogue. Suite: 116 suites, 5,378 assertions, 0 failed.

Six requests were closed as "impossible to model - no Ethernet port" while **every one of those devices was already in `devices/`**: #54, #167, #360, #362, #374, #387. The sweeper could not catch it, and the reason is structural: it only ever looks at **open** issues. Closing a request removes it from every automated view, so a wrong close is permanent, invisible, and looks exactly like work that was done. `scripts/check-closed-issues.js` walks the other side of the ledger: | verdict | meaning | |---|---| | **answered** | the catalogue has the device *and* a closing comment names the file | | **silent** | the catalogue has the device, no comment names it - the shape a wrong close has | | **unmatched** | nothing in the catalogue answers the subject; most closed issues, not a finding | Against the live tracker (120 most recent closed device requests): **52 answered with evidence, 0 silent, 68 unmatched**. Three decisions worth naming, because each is a way the audit could become the problem it looks for: - **Exact matches only.** A fuzzy match claiming "already answered" by a device the issue did not name is the same error in the other direction. - **Device requests only** (`[Device]: ...`) unless `--all`. A bug report mentioning an OLT - "OLTs promise 10G SFP+ uplinks they do not model" - matched an OLT and produced a finding that meant nothing. - **A missing token is a skip, not a failure.** Runs on push to `main` (schedules never reach a runner here) and never fails the build: a report nobody can act on because the pipeline went red is worse than no report. It never reopens or closes anything - the action stays with a person; this only makes sure the person is looking. Documented in `docs/RESOLUTIONS.md`. 17 checks in `test/closed-issues-audit.test.js`, including all six real cases against the live catalogue. Suite: 116 suites, 5,378 assertions, 0 failed.
Audit closed requests against the catalogue: the six wrong closes were invisible by design
All checks were successful
CI / Classify the change (pull_request) Successful in 11s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 12s
CI / Test (Node 18) (pull_request) Successful in 1m49s
CI / Test (Node 20) (pull_request) Successful in 1m42s
CI / Test (Node 22) (pull_request) Successful in 1m40s
CI / Server smoke test (pull_request) Successful in 26s
CI / Docker build (pull_request) Successful in 43s
00ffbc65c0
Six requests were closed as "impossible to model - no Ethernet port" while **every one of those
devices was already in `devices/`**: #54, #167, #360, #362, #374, #387. The sweeper could not catch
it, and the reason is structural rather than a bug: it only ever looks at **open** issues. Closing a
request removes it from every automated view, so a wrong close is permanent, invisible, and looks
exactly like work that was done.

`scripts/check-closed-issues.js` walks the other side of the ledger. For closed requests it reports:

| verdict | meaning |
|---|---|
| **answered** | the catalogue has the device *and* a closing comment names the file |
| **silent** | the catalogue has the device, no comment names it - the shape a wrong close has |
| **unmatched** | nothing in the catalogue answers the subject; most closed issues, not a finding |

Against the live tracker (120 most recent closed device requests): **52 answered with evidence, 0
silent, 68 unmatched.**

Three decisions worth naming, because each one is a way the audit could itself become the problem it
looks for:

- **Exact matches only.** A fuzzy match that claims "this was already answered" by a device the
  issue did not name is the same error in the other direction.
- **Device requests only** (`[Device]: ...`) unless `--all` is passed. A bug report that mentions an
  OLT - "OLTs promise 10G SFP+ uplinks they do not model" - matched an OLT and produced a finding
  that meant nothing.
- **A missing token is a skip, not a failure.** The workflow runs on push to `main` (schedules never
  reach a runner on this instance) and never fails the build: a report nobody can act on because
  the pipeline went red is worse than no report.

It never reopens or closes anything. The action stays with a person; this only makes sure the
person is looking.

17 checks in `test/closed-issues-audit.test.js`, including all six real cases against the live
catalogue. Suite: 116 suites, 5,378 assertions, 0 failed.
jaylouisw scheduled this pull request to auto merge when all checks succeed 2026-11-09 13:45:27 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jaylouisw/packetforge!1210
No description provided.