ci(pr-rebase): bound the fetch and the rebase - a stalled host held a slot for 106 minutes #1947

Merged
jaylouisw merged 1 commit from ci/1118-0625 into main 2026-11-18 06:42:10 +00:00
Owner

What was broken

Rebase onto main and arm auto-merge (pr-rebase.yml) had two network operations with no ceiling, and this runner does not enforce timeout-minutes (ci.yml documents that separately).

  • task 14673 / run 5763 / pull #1915: started 2026-11-18T00:07:30Z, finished 01:53:33Z = 6363 s (106 min) against a 10-minute bound. Status failure - a red X on a pull request with nothing wrong with it, on a host with two runner slots.
  • task 10861 / run 4597 / pull #1739: 762 s (12.7 min), failure.
  • Baseline for the job over the 24 h to 2026-11-18T02:21Z (/actions/tasks, 14437 runs): 329 runs, median 9 s, mean 32.8 s, 180.0 runner-minutes. Those two tasks are 118.8 of the 180 minutes - 66%% of the job's cost in two stuck runs.

The two unbounded calls:

  1. Step 1's fetch. git fetch with neither a timeout nor a low-speed abort. Step 2 exports GIT_HTTP_LOW_SPEED_LIMIT/_TIME; step 1 did not, and git waits forever for a peer that accepts the connection and then stops answering. This host did exactly that twice this morning (40 s of silence, then a refused connection on port 3000).
  2. git rebase. Bare, and the clone is --filter=blob:none, so the replay lazily fetches the blobs it needs from the same host - an unbounded network read inside a command nobody thought of as network I/O.

The change

  • timeout 90 on both fetches in step 1, plus GIT_TERMINAL_PROMPT=0 and the low-speed abort the deploy script uses. The existing fail-open path is untouched: a fetch that trips this lands on the same ::warning:: and exit 0 that a failed fetch already used, and the branch is left untouched.
  • bounded 180 git rebase, the same ceiling git push and git fetch already got, with rc 124/137 handled separately so a timeout is not reported to the author as a conflict.

Coverage unchanged

Same trigger, same two commits, same rebase, same push, same arm. Nothing is skipped or retried differently; the only behavioural difference is that an operation that would have hung now stops and says so.

After

Run ids on this branch below (pr-rebase step 1 is what the fetch bound lives in, so its own run exercises it). A 106-minute stall cannot be reproduced on demand; what is measured is that the job still passes and still takes its usual time with the bounds in place.

**What was broken** `Rebase onto main and arm auto-merge` (pr-rebase.yml) had two network operations with no ceiling, and this runner does not enforce `timeout-minutes` (ci.yml documents that separately). - task 14673 / run 5763 / pull #1915: started 2026-11-18T00:07:30Z, finished 01:53:33Z = **6363 s (106 min)** against a 10-minute bound. Status `failure` - a red X on a pull request with nothing wrong with it, on a host with two runner slots. - task 10861 / run 4597 / pull #1739: 762 s (12.7 min), `failure`. - Baseline for the job over the 24 h to 2026-11-18T02:21Z (/actions/tasks, 14437 runs): 329 runs, median 9 s, mean 32.8 s, 180.0 runner-minutes. Those two tasks are 118.8 of the 180 minutes - 66%% of the job's cost in two stuck runs. The two unbounded calls: 1. **Step 1's fetch.** `git fetch` with neither a `timeout` nor a low-speed abort. Step 2 exports `GIT_HTTP_LOW_SPEED_LIMIT`/`_TIME`; step 1 did not, and git waits forever for a peer that accepts the connection and then stops answering. This host did exactly that twice this morning (40 s of silence, then a refused connection on port 3000). 2. **`git rebase`.** Bare, and the clone is `--filter=blob:none`, so the replay lazily fetches the blobs it needs from the same host - an unbounded network read inside a command nobody thought of as network I/O. **The change** - `timeout 90` on both fetches in step 1, plus `GIT_TERMINAL_PROMPT=0` and the low-speed abort the deploy script uses. The existing fail-open path is untouched: a fetch that trips this lands on the same `::warning::` and `exit 0` that a failed fetch already used, and the branch is left untouched. - `bounded 180 git rebase`, the same ceiling `git push` and `git fetch` already got, with `rc 124/137` handled separately so a timeout is **not** reported to the author as a conflict. **Coverage unchanged** Same trigger, same two commits, same rebase, same push, same arm. Nothing is skipped or retried differently; the only behavioural difference is that an operation that would have hung now stops and says so. **After** Run ids on this branch below (pr-rebase step 1 is what the fetch bound lives in, so its own run exercises it). A 106-minute stall cannot be reproduced on demand; what is measured is that the job still passes and still takes its usual time with the bounds in place.
ci(pr-rebase): bound the fetch and the rebase, so a stalled host cannot hold a slot for 106 minutes
All checks were successful
CI / Classify the change (pull_request) Successful in 5s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 8s
CI / Server smoke test (pull_request) Successful in 6s
CI / Docker build (pull_request) Successful in 19s
CI / Test (Node 20) (pull_request) Successful in 56s
CI / Test (Node 22) (pull_request) Successful in 59s
CI / Test (Node 18) (pull_request) Successful in 1m3s
CI / Arm auto-merge (pull_request) Successful in 17s
8da6b819ab
jaylouisw force-pushed ci/1118-0625 from 8da6b819ab
All checks were successful
CI / Classify the change (pull_request) Successful in 5s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 8s
CI / Server smoke test (pull_request) Successful in 6s
CI / Docker build (pull_request) Successful in 19s
CI / Test (Node 20) (pull_request) Successful in 56s
CI / Test (Node 22) (pull_request) Successful in 59s
CI / Test (Node 18) (pull_request) Successful in 1m3s
CI / Arm auto-merge (pull_request) Successful in 17s
to 3142ad5ec9
All checks were successful
CI / Classify the change (pull_request) Successful in 7s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 8s
CI / Server smoke test (pull_request) Successful in 7s
CI / Docker build (pull_request) Successful in 19s
CI / Test (Node 20) (pull_request) Successful in 48s
CI / Test (Node 18) (pull_request) Successful in 57s
CI / Test (Node 22) (pull_request) Successful in 52s
CI / Arm auto-merge (pull_request) Successful in 16s
2026-11-18 06:38:08 +00:00
Compare
jaylouisw force-pushed ci/1118-0625 from 3142ad5ec9
All checks were successful
CI / Classify the change (pull_request) Successful in 7s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 8s
CI / Server smoke test (pull_request) Successful in 7s
CI / Docker build (pull_request) Successful in 19s
CI / Test (Node 20) (pull_request) Successful in 48s
CI / Test (Node 18) (pull_request) Successful in 57s
CI / Test (Node 22) (pull_request) Successful in 52s
CI / Arm auto-merge (pull_request) Successful in 16s
to b3b2b4708c
All checks were successful
CI / Classify the change (pull_request) Successful in 7s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 4s
CI / Server smoke test (pull_request) Successful in 5s
CI / Docker build (pull_request) Successful in 19s
CI / Test (Node 20) (pull_request) Successful in 45s
CI / Test (Node 18) (pull_request) Successful in 52s
CI / Test (Node 22) (pull_request) Successful in 46s
CI / Arm auto-merge (pull_request) Successful in 8s
2026-11-18 06:39:07 +00:00
Compare
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jaylouisw/packetforge!1947
No description provided.