auto-rebase dies in job setup when data.forgejo.org is unreachable - 16 consecutive ticks, 7.5 h with no rebase or arming #2366

Closed
opened 2026-11-21 04:35:04 +00:00 by jaylouisw · 1 comment
Owner

What happened

On 2026-11-20 the runner lost egress to data.forgejo.org. Every auto-rebase.yml tick in that window failed before its first step:

  • last green tick: run 8460, task 22358, 10:08:44Z
  • 16 consecutive red: runs 8472 (task 22379, 10:11:53Z) then 8476-8490 (tasks 22392-22413, 10:30:03Z - 17:30:03Z)
  • first green again: run 8493, task 22421, 17:52:11Z

Each one: Set up job 2-3 s, then every real step cancelled, job failure. The whole log is 12 lines (run 8490, jobs/0/attempt/1/logs):

Runner 01871f2b... received task 22413 of job rebase, triggered by event: schedule
workflow prepared
Start image=node:20-bookworm
docker create / docker run ...
git fetch 'https://data.forgejo.org/actions/checkout' # ref=v4
could not fetch remote 'origin': fatal: unable to access
  'https://data.forgejo.org/actions/checkout/': Failed to connect to
  data.forgejo.org:443 after 15 ms: Could not connect to server: exit status 128
Job failed

Root cause

The runner pre-clones every uses: action of a job at dispatch, before step one. auto-rebase.yml is the only workflow left in this repository with a uses: (actions/checkout@v4, line ~305), so it is the only one that cannot start when the internet is down. ci.yml removed that dependency deliberately (its checkout is git init + git fetch --depth=1, see the note at the top of ci.yml), which is why the test legs, smoke, docker and arm kept running through the same outage - and why the docker-CLI fallout of the same outage was already fixed in ci.yml (task 22378, run 8465).

Cost: 7 h 20 m in which no stale pull request was rebased and the scheduled arming backstop did not run, plus 16 red runs that read like code failures.

Why the workflow cannot fix it itself

The job has to keep a checkout step: test/ci-disk-hygiene.test.js (case #1508, lines ~186-190) asserts that the rebase job contains a step matching uses:\s*actions/checkout@, positioned after the df -Pk room guard and gated on its room output. Any uses: - even one behind if: false - is pre-fetched at dispatch, so the setup failure is unavoidable while that line exists.

Proposed patch (needs the test/ owner)

  1. auto-rebase.yml: replace the uses: actions/checkout@v4 step with the blobless git init + git fetch --depth=1 <sha> the rest of the repository already uses (pr-rebase.yml, close-merged-issues.yml). Same commit, same tree, same git clean -qfdx, still gated on the room guard; removes ~3.8 s of action clone + ~9.2 s of ref fetch per tick as well.
  2. test/ci-disk-hygiene.test.js: make the invariant "the room guard precedes whatever checks out", not "the job uses the checkout action" - accept a step whose text has uses: actions/checkout@ or git + fetch --depth=1, keep the guardAt < checkoutAt and gated() assertions unchanged.

No coverage changes: the tick still rebases the same branches onto the same base with real git, and still arms through the same API calls.

Not affected

pr-rebase.yml (event-triggered rebase) has no uses: and kept working; ci.yml's per-PR arm job kept arming after each matrix.

## What happened On 2026-11-20 the runner lost egress to `data.forgejo.org`. Every `auto-rebase.yml` tick in that window failed **before its first step**: - last green tick: run **8460**, task **22358**, 10:08:44Z - 16 consecutive red: runs **8472** (task 22379, 10:11:53Z) then **8476-8490** (tasks **22392-22413**, 10:30:03Z - 17:30:03Z) - first green again: run **8493**, task **22421**, 17:52:11Z Each one: `Set up job` 2-3 s, then every real step `cancelled`, job `failure`. The whole log is 12 lines (run 8490, jobs/0/attempt/1/logs): ``` Runner 01871f2b... received task 22413 of job rebase, triggered by event: schedule workflow prepared Start image=node:20-bookworm docker create / docker run ... git fetch 'https://data.forgejo.org/actions/checkout' # ref=v4 could not fetch remote 'origin': fatal: unable to access 'https://data.forgejo.org/actions/checkout/': Failed to connect to data.forgejo.org:443 after 15 ms: Could not connect to server: exit status 128 Job failed ``` ## Root cause The runner pre-clones every `uses:` action of a job at dispatch, before step one. `auto-rebase.yml` is the **only** workflow left in this repository with a `uses:` (`actions/checkout@v4`, line ~305), so it is the only one that cannot start when the internet is down. `ci.yml` removed that dependency deliberately (its checkout is `git init` + `git fetch --depth=1`, see the note at the top of `ci.yml`), which is why the test legs, smoke, docker and arm kept running through the same outage - and why the docker-CLI fallout of the same outage was already fixed in `ci.yml` (task 22378, run 8465). Cost: 7 h 20 m in which no stale pull request was rebased and the scheduled arming backstop did not run, plus 16 red runs that read like code failures. ## Why the workflow cannot fix it itself The job has to keep a checkout step: `test/ci-disk-hygiene.test.js` (case #1508, lines ~186-190) asserts that the `rebase` job contains a step matching `uses:\s*actions/checkout@`, positioned after the `df -Pk` room guard and gated on its `room` output. Any `uses:` - even one behind `if: false` - is pre-fetched at dispatch, so the setup failure is unavoidable while that line exists. ## Proposed patch (needs the test/ owner) 1. `auto-rebase.yml`: replace the `uses: actions/checkout@v4` step with the blobless `git init` + `git fetch --depth=1 <sha>` the rest of the repository already uses (`pr-rebase.yml`, `close-merged-issues.yml`). Same commit, same tree, same `git clean -qfdx`, still gated on the room guard; removes ~3.8 s of action clone + ~9.2 s of ref fetch per tick as well. 2. `test/ci-disk-hygiene.test.js`: make the invariant "the room guard precedes whatever checks out", not "the job uses the checkout action" - accept a step whose text has `uses: actions/checkout@` **or** `git` + `fetch --depth=1`, keep the `guardAt < checkoutAt` and `gated()` assertions unchanged. No coverage changes: the tick still rebases the same branches onto the same base with real git, and still arms through the same API calls. ## Not affected `pr-rebase.yml` (event-triggered rebase) has no `uses:` and kept working; `ci.yml`'s per-PR arm job kept arming after each matrix.
Author
Owner

Fixed in #2374 (branch fix/1121-0532). Both halves of the proposal landed: the uses: actions/checkout@v4 step is now git init + one blobless fetch of main, gated exactly as before, with credentials added to origin only for the push; and test/ci-disk-hygiene.test.js now reads the invariant as "the room guard precedes whatever materialises the tree", plus a new case that fails on main and asserts the rebase job runs no uses: at all, so this cannot come back. npm test: 435 suites, 14624 assertions, 0 failed. The loop was replayed against a local bare repo from an object-less clone - fetch, checkout -B, merge-base --is-ancestor, rebase, diff --quiet and --force-with-lease push all succeed, blobs fetched on demand.

Fixed in #2374 (branch `fix/1121-0532`). Both halves of the proposal landed: the `uses: actions/checkout@v4` step is now `git init` + one blobless fetch of `main`, gated exactly as before, with credentials added to `origin` only for the push; and `test/ci-disk-hygiene.test.js` now reads the invariant as "the room guard precedes whatever materialises the tree", plus a new case that fails on `main` and asserts the `rebase` job runs no `uses:` at all, so this cannot come back. `npm test`: 435 suites, 14624 assertions, 0 failed. The loop was replayed against a local bare repo from an object-less clone - fetch, `checkout -B`, `merge-base --is-ancestor`, `rebase`, `diff --quiet` and `--force-with-lease` push all succeed, blobs fetched on demand.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jaylouisw/packetforge#2366
No description provided.