CI: bound every network call the rebase maintenance jobs make #1556
No reviewers
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
jaylouisw/packetforge!1556
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "ci/1115-2221"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Baseline
Rebase onto main and arm auto-merge(pr-rebase.yml) — task 6299 / run 2890, pull request #1549,22:07:18Z -> 22:21:25Z= 847 s, outcome failure. Median for the same job over the last 80 tasks is 22 s (task 6295/run 2886 = 24 s, task 6311/run 2894 = 22 s), so this is a 38x outlier.What it cost, from the same API snapshot:
PR rebase / Rebase onto main and arm auto-merge (pull_request)=failureonf3e9118c.if: failure()reporter is a later step and a timeout kill skips it; #1549's last comment is the conflict note from the earlier, successful run 2886.Test (Node 18/20/22),Server smoke test,Docker build) were stillpendingat 22:26, and the job held one of the host's two runner slots for fourteen minutes.Why
Nothing in the job was bounded. The mergeable poll was bounded by iteration count only — 12 passes of a bare
curlwithsleep 10between — and a bare curl inherits no timeout:curl -sSto an unresponsive peer (10.255.255.1) gives up after 21029 ms here, 130 s on a Linux runner with default SYN retries.git fetchandgit pushwere unbounded on top of that./actions/tasksanswered in 0.45 s at 22:26 and returned 0 bytes in 60 s earlier the same evening.auto-rebase.ymlhad the identical shape and notimeout-minutesat all — an unbounded scheduled job that force-pushes pull-request branches.Change
http():curl --connect-timeout 5 --max-time 20 --retry 2for every call in both files.bounded(): coreutilstimeoutaroundgit fetch(120 s) andgit push(180 s).GIT_HTTP_LOW_SPEED_LIMIT=1000/..._TIME=30,GIT_TERMINAL_PROMPT=0.409on the arm call now reads as armed (it is "already armed").auto-rebase.ymlgetstimeout-minutes: 15.After (measured)
Same unreachable peer, poll loop only: before 372 s -> after 134 s (deadline 120 s, 18 bounded calls).
bounded 5 git ls-remotereturns 124 instead of hanging.bash -non all threerun:blocks,yaml.safe_load,check-control-bytes.js,check-ci-disk-hygiene.js,syntax-check.js(325 files) andtest/ci-disk-hygiene.test.js(8/8) +test/classify-change.test.js(6/6) all pass.Worst case for the whole pr-rebase job is now ~5 min instead of unbounded-then-killed at 10, and the typical case is unchanged (22 s, nothing to rebase).
Coverage unchanged
Every check still runs, every failure path still reports a
::warning::and exits 0, the rebase/force-with-lease/conflict-comment behaviour is untouched. The only difference is that a stalled host costs a log line after 20 s instead of a red X after ten minutes.Still red: PR #1549's rebase check cannot be re-run by this PR — it needs a fresh
pull_requestevent on that branch. Its conflict is a human decision (it is reported as such by run 2886).Measured pair, both runs against the same unreachable peer (10.255.255.1) with the poll loop extracted verbatim from the workflow:
i=0; while [ $i -lt 12 ]with barecurl+sleep 10deadline=$((SECONDS+120))withhttp()+sleep 5Retry arithmetic, for accuracy:
--max-time 20is the ceiling for a peer that accepts the connection and then goes quiet, and--retry 2can triple that to 60 s for one call. The loop is bounded by the wall clock rather than by the call count, so that still lands inside 120 s plus one call - which is what the 134 s above is. Connection timeouts are not retried past--connect-timeout 5in practice: 18 attempts, 5.0-5.1 s each, in the after run.bounded 5 git ls-remote http://10.255.255.1/x.gitexits 124 (killed) instead of hanging, so the fetch and push ceilings work the same way.PR checks for this branch were dispatched at 22:33:53 (
CI / Classify the changeetc., all 7pending) - the after-state rebase run will show its own duration when the queue reaches it.04aa79deb8d932f618a6