CI: bound every network call the rebase maintenance jobs make #1556

Merged
jaylouisw merged 4 commits from ci/1115-2221 into main 2026-11-15 23:09:05 +00:00
Owner

Baseline

Rebase onto main and arm auto-merge (pr-rebase.yml) — task 6299 / run 2890, pull request #1549, 22:07:18Z -> 22:21:25Z = 847 s, outcome failure. Median for the same job over the last 80 tasks is 22 s (task 6295/run 2886 = 24 s, task 6311/run 2894 = 22 s), so this is a 38x outlier.

What it cost, from the same API snapshot:

  • PR #1549 wore a red X for a maintenance job, PR rebase / Rebase onto main and arm auto-merge (pull_request) = failure on f3e9118c.
  • No comment explained it. The if: failure() reporter is a later step and a timeout kill skips it; #1549's last comment is the conflict note from the earlier, successful run 2886.
  • Its CI checks (Test (Node 18/20/22), Server smoke test, Docker build) were still pending at 22:26, and the job held one of the host's two runner slots for fourteen minutes.

Why

Nothing in the job was bounded. The mergeable poll was bounded by iteration count only — 12 passes of a bare curl with sleep 10 between — and a bare curl inherits no timeout:

  • curl -sS to an unresponsive peer (10.255.255.1) gives up after 21029 ms here, 130 s on a Linux runner with default SYN retries.
  • 12 x (21 + 10) = 372 s of pure waiting before the loop had done anything.
  • The checkout, git fetch and git push were unbounded on top of that.
  • The API stalls rather than refuses on this host: /actions/tasks answered in 0.45 s at 22:26 and returned 0 bytes in 60 s earlier the same evening.

auto-rebase.yml had the identical shape and no timeout-minutes at all — an unbounded scheduled job that force-pushes pull-request branches.

Change

  • http(): curl --connect-timeout 5 --max-time 20 --retry 2 for every call in both files.
  • bounded(): coreutils timeout around git fetch (120 s) and git push (180 s).
  • GIT_HTTP_LOW_SPEED_LIMIT=1000 / ..._TIME=30, GIT_TERMINAL_PROMPT=0.
  • Poll against a 120 s wall clock instead of an iteration count; 5 s between passes so a healthy PR still arms in ~10 s.
  • 409 on the arm call now reads as armed (it is "already armed").
  • auto-rebase.yml gets timeout-minutes: 15.

After (measured)

Same unreachable peer, poll loop only: before 372 s -> after 134 s (deadline 120 s, 18 bounded calls). bounded 5 git ls-remote returns 124 instead of hanging. bash -n on all three run: blocks, yaml.safe_load, check-control-bytes.js, check-ci-disk-hygiene.js, syntax-check.js (325 files) and test/ci-disk-hygiene.test.js (8/8) + test/classify-change.test.js (6/6) all pass.

Worst case for the whole pr-rebase job is now ~5 min instead of unbounded-then-killed at 10, and the typical case is unchanged (22 s, nothing to rebase).

Coverage unchanged

Every check still runs, every failure path still reports a ::warning:: and exits 0, the rebase/force-with-lease/conflict-comment behaviour is untouched. The only difference is that a stalled host costs a log line after 20 s instead of a red X after ten minutes.

Still red: PR #1549's rebase check cannot be re-run by this PR — it needs a fresh pull_request event on that branch. Its conflict is a human decision (it is reported as such by run 2886).

## Baseline `Rebase onto main and arm auto-merge` (pr-rebase.yml) — **task 6299 / run 2890**, pull request #1549, `22:07:18Z -> 22:21:25Z` = **847 s**, outcome **failure**. Median for the same job over the last 80 tasks is **22 s** (task 6295/run 2886 = 24 s, task 6311/run 2894 = 22 s), so this is a 38x outlier. What it cost, from the same API snapshot: - PR #1549 wore a **red X** for a maintenance job, `PR rebase / Rebase onto main and arm auto-merge (pull_request)` = `failure` on `f3e9118c`. - **No comment explained it.** The `if: failure()` reporter is a later step and a timeout kill skips it; #1549's last comment is the *conflict* note from the earlier, successful run 2886. - Its CI checks (`Test (Node 18/20/22)`, `Server smoke test`, `Docker build`) were still `pending` at 22:26, and the job held one of the host's two runner slots for fourteen minutes. ## Why Nothing in the job was bounded. The mergeable poll was bounded by iteration count only — 12 passes of a bare `curl` with `sleep 10` between — and a bare curl inherits no timeout: - `curl -sS` to an unresponsive peer (10.255.255.1) gives up after **21029 ms** here, 130 s on a Linux runner with default SYN retries. - 12 x (21 + 10) = **372 s** of pure waiting before the loop had done anything. - The checkout, `git fetch` and `git push` were unbounded on top of that. - The API stalls rather than refuses on this host: `/actions/tasks` answered in **0.45 s** at 22:26 and returned **0 bytes in 60 s** earlier the same evening. `auto-rebase.yml` had the identical shape **and no `timeout-minutes` at all** — an unbounded scheduled job that force-pushes pull-request branches. ## Change - `http()`: `curl --connect-timeout 5 --max-time 20 --retry 2` for every call in both files. - `bounded()`: coreutils `timeout` around `git fetch` (120 s) and `git push` (180 s). - `GIT_HTTP_LOW_SPEED_LIMIT=1000` / `..._TIME=30`, `GIT_TERMINAL_PROMPT=0`. - Poll against a 120 s wall clock instead of an iteration count; 5 s between passes so a healthy PR still arms in ~10 s. - `409` on the arm call now reads as armed (it is "already armed"). - `auto-rebase.yml` gets `timeout-minutes: 15`. ## After (measured) Same unreachable peer, poll loop only: **before 372 s -> after 134 s** (deadline 120 s, 18 bounded calls). `bounded 5 git ls-remote` returns **124** instead of hanging. `bash -n` on all three `run:` blocks, `yaml.safe_load`, `check-control-bytes.js`, `check-ci-disk-hygiene.js`, `syntax-check.js` (325 files) and `test/ci-disk-hygiene.test.js` (8/8) + `test/classify-change.test.js` (6/6) all pass. Worst case for the whole pr-rebase job is now ~5 min instead of unbounded-then-killed at 10, and the typical case is unchanged (22 s, nothing to rebase). ## Coverage unchanged Every check still runs, every failure path still reports a `::warning::` and exits 0, the rebase/force-with-lease/conflict-comment behaviour is untouched. The only difference is that a stalled host costs a log line after 20 s instead of a red X after ten minutes. Still red: PR #1549's rebase check cannot be re-run by this PR — it needs a fresh `pull_request` event on that branch. Its conflict is a human decision (it is reported as such by run 2886).
CI: bound every network call the rebase maintenance jobs make
All checks were successful
CI / Classify the change (pull_request) Successful in 18s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 21s
CI / Test (Node 18) (pull_request) Successful in 3m42s
CI / Test (Node 20) (pull_request) Successful in 3m26s
CI / Server smoke test (pull_request) Successful in 33s
CI / Docker build (pull_request) Successful in 24s
CI / Test (Node 22) (pull_request) Successful in 3m20s
04aa79deb8
Task 6299 (run 2890, pull request #1549) ran the "Rebase onto main and arm
auto-merge" job for 847 s against a 22 s median for that job, spent its whole
10-minute budget, came back `failure` on a pull request whose branch was never
touched, and left no comment: the `if: failure()` reporter is a later step, and a
timeout kill skips it. While it did, #1549's own CI checks sat pending and one of
the host's two runner slots was held for fourteen minutes.

The cause is that nothing in these jobs was bounded. The mergeable poll was
bounded by iteration count only - twelve passes of a bare `curl` with a 10 s
sleep between - and a bare curl inherits no timeout, so an unresponsive peer
costs it 21.03 s per attempt on this host (measured: `curl -sS` to 10.255.255.1
fails after 21029 ms) and 130 s on a Linux runner: 12 x (21 + 10) = 372 s of
waiting before it had done anything. The checkout, the fetch and the push were
unbounded on top of that. The API does stall here rather than refuse - the same
/actions/tasks query answered in 0.45 s at 22:26 and returned nothing inside 60 s
earlier the same evening.

So both rebase paths now use `http()` (5 s connect, 20 s per call) and
`bounded()` (coreutils timeout) for git fetch/push, git gets a low-speed abort,
and the poll runs against a 120 s wall clock instead of a count. auto-rebase.yml
had the same shape and no timeout-minutes at all; it now has 15.

Measured on the same unreachable peer, poll loop only: before 372 s, after 134 s
(deadline 120 s), and `bounded 5 git ls-remote` returns 124 instead of hanging.

Coverage unchanged: every check in both jobs still runs, every failure path still
reports a :⚠️: and exits 0, and the only behaviour that differs is that a
stalled host produces a log line in 20 s instead of a red X in ten minutes.
Author
Owner

Measured pair, both runs against the same unreachable peer (10.255.255.1) with the poll loop extracted verbatim from the workflow:

loop wall clock
before: i=0; while [ $i -lt 12 ] with bare curl + sleep 10 376 s (12 x (21 + 10), measured)
after: deadline=$((SECONDS+120)) with http() + sleep 5 134 s (measured)

Retry arithmetic, for accuracy: --max-time 20 is the ceiling for a peer that accepts the connection and then goes quiet, and --retry 2 can triple that to 60 s for one call. The loop is bounded by the wall clock rather than by the call count, so that still lands inside 120 s plus one call - which is what the 134 s above is. Connection timeouts are not retried past --connect-timeout 5 in practice: 18 attempts, 5.0-5.1 s each, in the after run.

bounded 5 git ls-remote http://10.255.255.1/x.git exits 124 (killed) instead of hanging, so the fetch and push ceilings work the same way.

PR checks for this branch were dispatched at 22:33:53 (CI / Classify the change etc., all 7 pending) - the after-state rebase run will show its own duration when the queue reaches it.

Measured pair, both runs against the same unreachable peer (10.255.255.1) with the poll loop extracted verbatim from the workflow: | loop | wall clock | |---|---| | before: `i=0; while [ $i -lt 12 ]` with bare `curl` + `sleep 10` | **376 s** (12 x (21 + 10), measured) | | after: `deadline=$((SECONDS+120))` with `http()` + `sleep 5` | **134 s** (measured) | Retry arithmetic, for accuracy: `--max-time 20` is the ceiling for a peer that accepts the connection and then goes quiet, and `--retry 2` can triple that to 60 s for one call. The loop is bounded by the wall clock rather than by the call count, so that still lands inside 120 s plus one call - which is what the 134 s above is. Connection timeouts are not retried past `--connect-timeout 5` in practice: 18 attempts, 5.0-5.1 s each, in the after run. `bounded 5 git ls-remote http://10.255.255.1/x.git` exits **124** (killed) instead of hanging, so the fetch and push ceilings work the same way. PR checks for this branch were dispatched at 22:33:53 (`CI / Classify the change` etc., all 7 `pending`) - the after-state rebase run will show its own duration when the queue reaches it.
jaylouisw scheduled this pull request to auto merge when all checks succeed 2026-11-15 23:08:55 +00:00
jaylouisw force-pushed ci/1115-2221 from 04aa79deb8
All checks were successful
CI / Classify the change (pull_request) Successful in 18s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 21s
CI / Test (Node 18) (pull_request) Successful in 3m42s
CI / Test (Node 20) (pull_request) Successful in 3m26s
CI / Server smoke test (pull_request) Successful in 33s
CI / Docker build (pull_request) Successful in 24s
CI / Test (Node 22) (pull_request) Successful in 3m20s
to d932f618a6
All checks were successful
CI / Classify the change (pull_request) Successful in 19s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 2m24s
CI / Test (Node 18) (pull_request) Successful in 3m34s
CI / Test (Node 20) (pull_request) Successful in 3m16s
CI / Server smoke test (pull_request) Successful in 35s
CI / Docker build (pull_request) Successful in 25s
CI / Test (Node 22) (pull_request) Successful in 3m23s
2026-11-15 23:08:59 +00:00
Compare
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jaylouisw/packetforge!1556
No description provided.