ci(pr-rebase): bound the fetch and the rebase - a stalled host held a slot for 106 minutes #1947
No reviewers
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
jaylouisw/packetforge!1947
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "ci/1118-0625"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What was broken
Rebase onto main and arm auto-merge(pr-rebase.yml) had two network operations with no ceiling, and this runner does not enforcetimeout-minutes(ci.yml documents that separately).failure- a red X on a pull request with nothing wrong with it, on a host with two runner slots.failure.The two unbounded calls:
git fetchwith neither atimeoutnor a low-speed abort. Step 2 exportsGIT_HTTP_LOW_SPEED_LIMIT/_TIME; step 1 did not, and git waits forever for a peer that accepts the connection and then stops answering. This host did exactly that twice this morning (40 s of silence, then a refused connection on port 3000).git rebase. Bare, and the clone is--filter=blob:none, so the replay lazily fetches the blobs it needs from the same host - an unbounded network read inside a command nobody thought of as network I/O.The change
timeout 90on both fetches in step 1, plusGIT_TERMINAL_PROMPT=0and the low-speed abort the deploy script uses. The existing fail-open path is untouched: a fetch that trips this lands on the same::warning::andexit 0that a failed fetch already used, and the branch is left untouched.bounded 180 git rebase, the same ceilinggit pushandgit fetchalready got, withrc 124/137handled separately so a timeout is not reported to the author as a conflict.Coverage unchanged
Same trigger, same two commits, same rebase, same push, same arm. Nothing is skipped or retried differently; the only behavioural difference is that an operation that would have hung now stops and says so.
After
Run ids on this branch below (pr-rebase step 1 is what the fetch bound lives in, so its own run exercises it). A 106-minute stall cannot be reproduced on demand; what is measured is that the job still passes and still takes its usual time with the bounds in place.
8da6b819ab3142ad5ec93142ad5ec9b3b2b4708c