ci: stop rebasing pull requests that are already red (#1749) #1789

Merged
jaylouisw merged 1 commit from ci/1117-0420 into main 2026-11-17 04:38:24 +00:00
Owner

Baseline

Job: Auto-rebase / Rebase open PRs and arm auto-merge (.forgejo/workflows/auto-rebase.yml), 24 h to 2026-11-17 04:27 UTC: n=290, median 18 s, p90 56 s, 276 success — the job itself is cheap. What it causes is not.

It takes the five lowest-numbered mergeable PRs and force-pushes each onto main, and every push starts a fresh five-job matrix. When the head of the queue is stuck red, that is a matrix every few minutes for nothing. From /actions/tasks (11,007 entries), six hours 22:27 → 04:28 UTC:

PR force-pushes distinct heads runs matrix result
#1749 33 33 4321, 4323, 4342 … 4719 99/99 test legs failure
#1739 30 30 4256, 4270 … 4717 90/90 failure
#1743 14 14 4254, 4258 … 4415 42/42 failure

Their Test (Node 18/20/22) legs alone are 14,041 s = 3.9 runner-hours in six hours. It is still live: #1739 moved ccf73f92 → 3f6076e7 while I was measuring it, ~24 min after the previous head.

Both branches are settled red on every head (/commits/<sha>/status: three test legs failure, classify/smoke/docker/arm success) and the failure is in each branch's own test file — nothing a rebase touches.

Change

Read the head's statuses before the rebase. A definite failure (failure/error) skips the push and prints PR #N: checks already failing on <sha> - not rebasing it. pending does not count, so a head whose checks are still reporting is rebased exactly as before — skipping those is how a PR would sit behind main for no reason.

A skipped branch no longer consumes one of the five slots (candidate window goes to four per slot; the per-run ceiling is still batch actual rebases), so red branches at the head of the queue cannot starve the rest of it. Summary line gains red=N.

After

Same decision function, run against the live queue at 04:47 UTC:

PR #1739: checks already failing on 3f6076e7 - not rebasing it
PR #1749: would rebase (slot 1 of 5)
PR #1787: would rebase (slot 2 of 5)
looked=2 red=1

The job runs only on push: main / schedule, and pull-request events are not dispatched for ci/* branches on this instance, so there is no new run id to quote yet: the first post-merge tick will carry red=N on its summary line. Verification done here: bash -n on the executed step, YAML parse, and the live-API run above against the real heads.

Coverage unchanged

Every head still runs the full matrix, including the rebased one; pr-rebase.yml still rebases on every push (which is the only event that can turn a red branch green); no check is skipped, weakened or removed. The only push not made is the one that provably cannot help.

Refs: #1749, #1739, #1743.

## Baseline **Job:** `Auto-rebase / Rebase open PRs and arm auto-merge` (`.forgejo/workflows/auto-rebase.yml`), 24 h to 2026-11-17 04:27 UTC: **n=290, median 18 s, p90 56 s, 276 success** — the job itself is cheap. What it *causes* is not. It takes the five lowest-numbered mergeable PRs and force-pushes each onto main, and every push starts a fresh five-job matrix. When the head of the queue is stuck red, that is a matrix every few minutes for nothing. From `/actions/tasks` (11,007 entries), six hours 22:27 → 04:28 UTC: | PR | force-pushes | distinct heads | runs | matrix result | |----|----|----|----|----| | #1749 | 33 | 33 | 4321, 4323, 4342 … 4719 | 99/99 test legs failure | | #1739 | 30 | 30 | 4256, 4270 … 4717 | 90/90 failure | | #1743 | 14 | 14 | 4254, 4258 … 4415 | 42/42 failure | Their `Test (Node 18/20/22)` legs alone are **14,041 s = 3.9 runner-hours in six hours**. It is still live: #1739 moved `ccf73f92` → `3f6076e7` while I was measuring it, ~24 min after the previous head. Both branches are settled red on every head (`/commits/<sha>/status`: three test legs `failure`, classify/smoke/docker/arm `success`) and the failure is in each branch's own test file — nothing a rebase touches. ## Change Read the head's statuses before the rebase. A **definite failure** (`failure`/`error`) skips the push and prints `PR #N: checks already failing on <sha> - not rebasing it`. `pending` does **not** count, so a head whose checks are still reporting is rebased exactly as before — skipping those is how a PR would sit behind main for no reason. A skipped branch no longer consumes one of the five slots (candidate window goes to four per slot; the per-run ceiling is still `batch` actual rebases), so red branches at the head of the queue cannot starve the rest of it. Summary line gains `red=N`. ## After Same decision function, run against the live queue at 04:47 UTC: ``` PR #1739: checks already failing on 3f6076e7 - not rebasing it PR #1749: would rebase (slot 1 of 5) PR #1787: would rebase (slot 2 of 5) looked=2 red=1 ``` The job runs only on `push: main` / schedule, and pull-request events are not dispatched for `ci/*` branches on this instance, so there is no new run id to quote yet: the first post-merge tick will carry `red=N` on its summary line. Verification done here: `bash -n` on the executed step, YAML parse, and the live-API run above against the real heads. ## Coverage unchanged Every head still runs the full matrix, including the rebased one; `pr-rebase.yml` still rebases on every push (which is the only event that can turn a red branch green); no check is skipped, weakened or removed. The only push not made is the one that provably cannot help. Refs: #1749, #1739, #1743.
ci: stop rebasing pull requests that are already red (#1749)
All checks were successful
CI / Classify the change (pull_request) Successful in 13s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 20s
CI / Server smoke test (pull_request) Successful in 8s
CI / Docker build (pull_request) Successful in 22s
CI / Test (Node 18) (pull_request) Successful in 1m3s
CI / Test (Node 20) (pull_request) Successful in 53s
CI / Test (Node 22) (pull_request) Successful in 57s
CI / Arm auto-merge (pull_request) Successful in 6s
8afab98e1e
The scheduled rebase takes the five lowest-numbered mergeable pull requests and
force-pushes each one onto main. When those five are stuck red - which, at the
head of a queue ordered oldest first, they usually are - every one of those
pushes starts a fresh five-job matrix that fails the same way, and the branch
cannot merge whatever the rebase produced.

Measured over the six hours 2026-11-16 22:27 to 2026-11-17 04:28 UTC from
/actions/tasks (11,007 entries): #1749 was force-pushed 33 times (runs 4321,
4323, 4342 ... 4719 - 33 distinct heads, 99 test legs, all failing), #1739 30
times (runs 4256 ... 4717), #1743 14 times. Their test legs alone are 14,041 s,
3.9 runner-hours, in six hours.

So: read the head's commit statuses before the fetch. If any check has reported
a definite failure, skip the rebase and say so. `pending` does not count - a
head whose checks are still reporting is rebased exactly as before, because
skipping a branch that might go green is how a pull request sits behind main
for no reason.

A skipped branch must not consume one of the five slots either, or the red
branches at the head of the queue starve the rest of it forever; the candidate
window is now four per slot and the per-run ceiling is still `batch` actual
rebases, so the number of matrices a tick can enqueue is unchanged.

Coverage is unchanged: every head still runs the full matrix including the
rebased one, pr-rebase.yml still rebases on every push, and no check is
skipped, weakened or removed. The only push not made is one that provably
cannot help.
jaylouisw force-pushed ci/1117-0420 from 8afab98e1e
All checks were successful
CI / Classify the change (pull_request) Successful in 13s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 20s
CI / Server smoke test (pull_request) Successful in 8s
CI / Docker build (pull_request) Successful in 22s
CI / Test (Node 18) (pull_request) Successful in 1m3s
CI / Test (Node 20) (pull_request) Successful in 53s
CI / Test (Node 22) (pull_request) Successful in 57s
CI / Arm auto-merge (pull_request) Successful in 6s
to 7bf7012e85
All checks were successful
CI / Classify the change (pull_request) Successful in 11s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 7s
CI / Server smoke test (pull_request) Successful in 17s
CI / Test (Node 20) (pull_request) Successful in 53s
CI / Test (Node 22) (pull_request) Successful in 55s
CI / Test (Node 18) (pull_request) Successful in 1m0s
CI / Docker build (pull_request) Successful in 21s
CI / Arm auto-merge (pull_request) Successful in 22s
2026-11-17 04:34:54 +00:00
Compare
Author
Owner

After — a real run, not a prediction

The job triggers only on push: main and schedule, and pull-request events are not dispatched for ci/* branches here, so I fired the modified workflow against this branch with workflow_dispatch and read its log:

Run 4747 (job id 11391, success, 04:38:05 → 04:38:44): batch=5 rebased=1 current=3 conflicts=0 duplicates=1 armed=0 rearmed=0 red=0 — the new counters are live and the step completes; #1787 rebased, nothing else disturbed.

Run 4767 (job id 11446, 04:45:28 → 04:46:37), dispatched ~2 min later once both stuck branches had a failure recorded:

PR #1739: checks already failing on 4531cc33... - not rebasing it
PR #1749: checks already failing on bc747907... - not rebasing it
PR #1790: already contains main
PR #1791: already contains main
batch=5 rebased=0 current=2 conflicts=0 duplicates=0 armed=0 rearmed=0 red=2

Two force-pushes and two five-job matrices not enqueued, on one tick. Before the change that same tick pushed both branches — that is what produced 33 heads in six hours on #1749 alone.

Still red and unaffected by this: #1749 fails one wall-clock assertion in the test file it adds (test/trunk-allowed-remove-from-all.test.js:167, ok(ms < 3000)), #1739 fails three stale hardcodes in existing test/ files. Both are test/ edits, which this stream will not make.

### After — a real run, not a prediction The job triggers only on `push: main` and schedule, and pull-request events are not dispatched for `ci/*` branches here, so I fired the modified workflow against this branch with `workflow_dispatch` and read its log: **Run 4747** (job id 11391, success, 04:38:05 → 04:38:44): `batch=5 rebased=1 current=3 conflicts=0 duplicates=1 armed=0 rearmed=0 red=0` — the new counters are live and the step completes; `#1787` rebased, nothing else disturbed. **Run 4767** (job id 11446, 04:45:28 → 04:46:37), dispatched ~2 min later once both stuck branches had a failure recorded: ``` PR #1739: checks already failing on 4531cc33... - not rebasing it PR #1749: checks already failing on bc747907... - not rebasing it PR #1790: already contains main PR #1791: already contains main batch=5 rebased=0 current=2 conflicts=0 duplicates=0 armed=0 rearmed=0 red=2 ``` Two force-pushes and two five-job matrices not enqueued, on one tick. Before the change that same tick pushed both branches — that is what produced 33 heads in six hours on #1749 alone. Still red and unaffected by this: #1749 fails one wall-clock assertion in the test file it adds (`test/trunk-allowed-remove-from-all.test.js:167`, `ok(ms < 3000)`), #1739 fails three stale hardcodes in existing `test/` files. Both are `test/` edits, which this stream will not make.
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jaylouisw/packetforge!1789
No description provided.