ci(main): a red main is opened as an issue, not left as a red X on a maintenance tick #2326

Merged
jaylouisw merged 1 commit from ci/1120-2227 into main 2026-11-20 22:44:29 +00:00
Owner

What was broken

main has no reporter. A red push run is a red X on a run nobody opens, on a workflow nobody watches, and it means "main is broken, and every pull request and every rebase onto it inherits the break".

Measured on 2026-11-20:

  • 0ff45aa4 (#2291) landed 10:09:39Z and broke four checks on its own push run 8473: Test (Node 18) (task 22381), Test (Node 22) (task 22383), Server smoke and Docker build (task 22389) - and the auto-rebase pass on the same head (8472, task 22379). Test (Node 20) (task 22382) and the three maintenance jobs were green on that head, so this was that commit, not the runner.
  • No merge landed between 10:09:39Z and 4c29aa46 at 17:51:48Z: 7 h 42 min with main broken and the merge queue frozen.
  • The only signal in that window was 15 scheduled Auto-rebase ticks, runs 8476-8490, every one failure in 1-3 s. Each red X was attached to a maintenance workflow and none of them named main's head.
  • Recovery came from a human workflow_dispatch (run 8493, 17:52:10Z), not from the pipeline.

The change

.forgejo/workflows/main-health.yml - every 15 minutes, and on dispatch, read /repos/{repo}/commits/{main's head}/status.

  • any failure/error -> open one issue titled main is red listing every failing context with its job URL. The head sha is written into the body, so a later tick on the same break adds nothing; a newer red head updates the same issue (comment + body).
  • every check success -> one closing comment, then close the issue.
  • pending, or a head with no statuses yet -> says so and does nothing (a run in flight is not a verdict).
  • unreadable API, unlistable issues, missing token -> ::warning naming it. It never reports green from a read it could not make, and it never opens a second issue for a break the first one already reports.
  • No checkout, no matrix, no uses: action; run token only, permissions: contents: read, issues: write.

The whole lifecycle was exercised against real heads before this PR (issue #2325, left closed on purpose): opened for 0ff45aa4 with the four failing checks (HTTP 201), asked again for the same head and added nothing, updated for a second red head 9a70a2fb (comment + body marker), then closed on the green head 98676312 (comment + close). 3 comments, state closed.

Measured

before after
detection of a red main 7 h 42 min (10:09:39Z -> 17:52:13Z), found by a human <= 15 min, next tick
what the reader gets 15 red X's on auto-rebase, none naming main one issue, the failing contexts, the job links
cost 0 run 8701, 4.0 s, Check main's head, dry run of the red path against 0ff45aa4 (run 8701 is on this branch, sha 89170760)

A quiet tick is two API reads and one issue listing, ~1.4 s measured locally; 96 ticks a day is under 3 minutes of runner time against the ~1100 the pipeline spends.

Coverage

Unchanged. ci.yml, auto-rebase.yml and pr-rebase.yml are untouched: nothing is edited, skipped, retried or relaxed, and this workflow adds no gate - it cannot merge, arm or rebase anything, and Arm auto-merge remains the only thing that merges. What it adds is a report of a state that was already true and already unread.

Still red: nothing. The last 12 h of runs are green (0 failures in 237 runs, 22:00-22:28 the busiest of them). For the record: the 15 red auto-rebase ticks were caused by 0ff45aa4 itself and stopped the moment main healed; their 1-3 s failure remains undiagnosed because a run log needs a login this worker does not have, and this workflow would have reported the actual break on the first tick instead of leaving 15 unexplained red X's.

## What was broken `main` has no reporter. A red push run is a red X on a run nobody opens, on a workflow nobody watches, and it means "main is broken, and every pull request and every rebase onto it inherits the break". Measured on 2026-11-20: - `0ff45aa4` (#2291) landed 10:09:39Z and broke four checks on its own push run **8473**: `Test (Node 18)` (task 22381), `Test (Node 22)` (task 22383), `Server smoke and Docker build` (task 22389) - and the auto-rebase pass on the same head (**8472**, task 22379). `Test (Node 20)` (task 22382) and the three maintenance jobs were green on that head, so this was that commit, not the runner. - No merge landed between 10:09:39Z and `4c29aa46` at 17:51:48Z: **7 h 42 min** with main broken and the merge queue frozen. - The only signal in that window was 15 scheduled Auto-rebase ticks, runs **8476-8490**, every one `failure` in 1-3 s. Each red X was attached to a maintenance workflow and none of them named main's head. - Recovery came from a human `workflow_dispatch` (run **8493**, 17:52:10Z), not from the pipeline. ## The change `.forgejo/workflows/main-health.yml` - every 15 minutes, and on dispatch, read `/repos/{repo}/commits/{main's head}/status`. - any `failure`/`error` -> open **one** issue titled `main is red` listing every failing context with its job URL. The head sha is written into the body, so a later tick on the same break adds nothing; a newer red head updates the same issue (comment + body). - every check `success` -> one closing comment, then close the issue. - `pending`, or a head with no statuses yet -> says so and does nothing (a run in flight is not a verdict). - unreadable API, unlistable issues, missing token -> `::warning` naming it. It never reports green from a read it could not make, and it never opens a second issue for a break the first one already reports. - No checkout, no matrix, no `uses:` action; run token only, `permissions: contents: read, issues: write`. The whole lifecycle was exercised against real heads before this PR (issue **#2325**, left closed on purpose): opened for `0ff45aa4` with the four failing checks (HTTP 201), asked again for the same head and added nothing, updated for a second red head `9a70a2fb` (comment + body marker), then closed on the green head `98676312` (comment + close). 3 comments, state `closed`. ## Measured | | before | after | |---|---|---| | detection of a red main | 7 h 42 min (10:09:39Z -> 17:52:13Z), found by a human | <= 15 min, next tick | | what the reader gets | 15 red X's on auto-rebase, none naming main | one issue, the failing contexts, the job links | | cost | 0 | run **8701**, **4.0 s**, `Check main's head`, dry run of the red path against `0ff45aa4` (run 8701 is on this branch, sha 89170760) | A quiet tick is two API reads and one issue listing, ~1.4 s measured locally; 96 ticks a day is under 3 minutes of runner time against the ~1100 the pipeline spends. ## Coverage Unchanged. `ci.yml`, `auto-rebase.yml` and `pr-rebase.yml` are untouched: nothing is edited, skipped, retried or relaxed, and this workflow adds no gate - it cannot merge, arm or rebase anything, and `Arm auto-merge` remains the only thing that merges. What it adds is a report of a state that was already true and already unread. Still red: nothing. The last 12 h of runs are green (0 failures in 237 runs, 22:00-22:28 the busiest of them). For the record: the 15 red auto-rebase ticks were caused by `0ff45aa4` itself and stopped the moment main healed; their 1-3 s failure remains undiagnosed because a run log needs a login this worker does not have, and this workflow would have reported the actual break on the first tick instead of leaving 15 unexplained red X's.
ci(main): a red main is opened as an issue, not left as a red X on a maintenance tick
All checks were successful
CI / Classify the change (pull_request) Successful in 14s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 9s
CI / Test (Node 18) (pull_request) Successful in 59s
CI / Test (Node 22) (pull_request) Successful in 55s
CI / Test (Node 20) (pull_request) Successful in 58s
CI / Server smoke and Docker build (pull_request) Successful in 20s
CI / Arm auto-merge (pull_request) Successful in 5s
89170760b2
Measured on 2026-11-20: 0ff45aa4 broke four checks on its own push run (8473, tasks 22381/22383/22389) at 10:09:39Z, no merge landed until 17:51:48Z, and the only signals were 15 scheduled Auto-rebase ticks (runs 8476-8490) red in 1-3 s each. Nothing named main. This reads /commits/{sha}/status on a 15-minute tick and opens one issue titled 'main is red', updated once per head, closed on the first green tick.
forgejo-actions scheduled this pull request to auto merge when all checks succeed 2026-11-20 22:42:18 +00:00
jaylouisw deleted branch ci/1120-2227 2026-11-20 22:44:30 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jaylouisw/packetforge!2326
No description provided.