ci(docker): retry the image build once when the shared builder cache loses the export race #2424

Merged
jaylouisw merged 1 commit from ci/1121-1227 into main 2026-11-21 13:09:11 +00:00
Owner

What was red

Server smoke and Docker build failed twice in the last 36 h on main with the same 5-second signature, on commits whose three test legs were green:

run task when error
8834 23298 2026-11-21T01:12:53Z failed to export image: failed to set parent sha256:572e12ed7a9a...: unknown parent image ID sha256:572e12ed7a9a...
8287 21914 2026-11-20T05:12:58Z failed to export image: ... unknown parent image ID sha256:3aeec9dfa5b0...

arm has needs: [test, docker], so each one blocked the merge of a green commit until it was re-run by hand.

Why the existing fix did not cover it

The per-run tag removed the shared-tag deleter, and the prune guard is working: the concurrent run 8830 printed not pruning this time: 2 Docker build(s) are running on this daemon at 01:12:49.5, 4 s before 8834 died. Both failures happened while 2-3 builds shared the daemon (8834 started 01:12:37 beside 8830 at 01:12:31; 8287 started 05:12:44 beside 8285 at 05:12:40 and 8281 at 05:12:34). The legacy builder's cache is content-addressed and shared, so two builds of the same early layers race on the same intermediate and the loser cannot export - no image of the wrong name is being deleted any more; the builder itself is the loser.

The change

.forgejo/workflows/ci.yml, Build image step only:

  • the build runs through one build_once function (same command, same tag, same timeout ... 180);
  • if - and only if - the log carries unknown parent image ID / failed to export image, it is retried once, after 3 s, with a ::warning that names the race;
  • a Dockerfile error is not retried (stub COPY failed: -> one attempt, exit 1), a 180 s stop is not retried and still prints its ::error (stub -> one attempt, exit 124), and a second failing attempt reddens the job with ::error.

Nothing else changes: same Dockerfile, same tree, same daemon, same 180 s bound per attempt (363 s worst case, inside the job's 10 min), and the retry can only pass by actually building the image. Both attempts are printed in full.

Measured

  • baseline, green Build image step: median 11 s, max 13 s over runs 9348, 9334, 9338, 9342, 9322, 9315, 9307, 9281; docker job median 22 s.
  • baseline, red: runs 8834 and 8287 above - 2 of 389 docker jobs in 36 h, both blocking arm.
  • after: run 9373 (this PR) - Build image 10 s, docker job 30 s, Successfully tagged packetforge:ci-run-9844, one build in the log (no retry fired). Legs 60/58/57 s, unchanged.

Coverage is unchanged: same three Node legs, same smoke script, same 8 Dockerfile static checks, same image build, and this tree's own check-ci-disk-hygiene passed in 9373.

## What was red `Server smoke and Docker build` failed twice in the last 36 h on **main** with the same 5-second signature, on commits whose three test legs were green: | run | task | when | error | |---|---|---|---| | 8834 | 23298 | 2026-11-21T01:12:53Z | `failed to export image: failed to set parent sha256:572e12ed7a9a...: unknown parent image ID sha256:572e12ed7a9a...` | | 8287 | 21914 | 2026-11-20T05:12:58Z | `failed to export image: ... unknown parent image ID sha256:3aeec9dfa5b0...` | `arm` has `needs: [test, docker]`, so each one blocked the merge of a green commit until it was re-run by hand. ## Why the existing fix did not cover it The per-run tag removed the shared-tag deleter, and the prune guard is working: the concurrent run 8830 printed `not pruning this time: 2 Docker build(s) are running on this daemon` at 01:12:49.5, 4 s before 8834 died. Both failures happened while 2-3 builds shared the daemon (8834 started 01:12:37 beside 8830 at 01:12:31; 8287 started 05:12:44 beside 8285 at 05:12:40 and 8281 at 05:12:34). The legacy builder's cache is content-addressed and shared, so two builds of the same early layers race on the same intermediate and the loser cannot export - no image of the wrong name is being deleted any more; the builder itself is the loser. ## The change `.forgejo/workflows/ci.yml`, `Build image` step only: - the build runs through one `build_once` function (same command, same tag, same `timeout ... 180`); - if - and only if - the log carries `unknown parent image ID` / `failed to export image`, it is retried **once**, after 3 s, with a `::warning` that names the race; - a Dockerfile error is not retried (stub `COPY failed:` -> one attempt, exit 1), a 180 s stop is not retried and still prints its `::error` (stub -> one attempt, exit 124), and a second failing attempt reddens the job with `::error`. Nothing else changes: same Dockerfile, same tree, same daemon, same 180 s bound per attempt (363 s worst case, inside the job's 10 min), and the retry can only pass by actually building the image. Both attempts are printed in full. ## Measured - baseline, green `Build image` step: median 11 s, max 13 s over runs 9348, 9334, 9338, 9342, 9322, 9315, 9307, 9281; docker job median 22 s. - baseline, red: runs 8834 and 8287 above - 2 of 389 docker jobs in 36 h, both blocking `arm`. - after: **run 9373** (this PR) - `Build image` 10 s, docker job 30 s, `Successfully tagged packetforge:ci-run-9844`, one build in the log (no retry fired). Legs 60/58/57 s, unchanged. Coverage is unchanged: same three Node legs, same smoke script, same 8 Dockerfile static checks, same image build, and this tree's own `check-ci-disk-hygiene` passed in 9373.
ci(docker): retry the image build once when the shared builder cache loses the export race
All checks were successful
CI / Server smoke and Docker build (pull_request) Successful in 30s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 7s
CI / Test (Node 22) (pull_request) Successful in 57s
CI / Test (Node 20) (pull_request) Successful in 58s
CI / Test (Node 18) (pull_request) Successful in 1m0s
CI / Arm auto-merge (pull_request) Successful in 3s
f4bc6fb2c9
Two red 'Server smoke and Docker build' jobs in the last 36 h died 5 s into
'docker build' with 'failed to export image: failed to set parent
sha256:...: unknown parent image ID sha256:...' - run 8834 (task 23298,
2026-11-21T01:12:53Z) and run 8287 (task 21914, 2026-11-20T05:12:58Z).
Both commits had three green test legs; 'arm' needs this job, so each one
held a merge until it was re-run by hand. The prune guard of the
concurrent run was doing its job (run 8830 at 01:12:49.5: 'not pruning this
time: 2 Docker build(s) are running'), so the loser of the race is the
legacy builder's content-addressed cache itself.

The build is now run through a single function and, only when the log
carries that export signature, retried exactly once with the same tag,
the same command and the same 180 s bound. A Dockerfile error, a timeout
and a failing second attempt all still fail the step and are named.
forgejo-actions scheduled this pull request to auto merge when all checks succeed 2026-11-21 12:53:03 +00:00
jaylouisw deleted branch ci/1121-1227 2026-11-21 13:09:13 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jaylouisw/packetforge!2424
No description provided.