ci(docker): retry the image build once when the shared builder cache loses the export race #2424
No reviewers
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
jaylouisw/packetforge!2424
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "ci/1121-1227"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What was red
Server smoke and Docker buildfailed twice in the last 36 h on main with the same 5-second signature, on commits whose three test legs were green:failed to export image: failed to set parent sha256:572e12ed7a9a...: unknown parent image ID sha256:572e12ed7a9a...failed to export image: ... unknown parent image ID sha256:3aeec9dfa5b0...armhasneeds: [test, docker], so each one blocked the merge of a green commit until it was re-run by hand.Why the existing fix did not cover it
The per-run tag removed the shared-tag deleter, and the prune guard is working: the concurrent run 8830 printed
not pruning this time: 2 Docker build(s) are running on this daemonat 01:12:49.5, 4 s before 8834 died. Both failures happened while 2-3 builds shared the daemon (8834 started 01:12:37 beside 8830 at 01:12:31; 8287 started 05:12:44 beside 8285 at 05:12:40 and 8281 at 05:12:34). The legacy builder's cache is content-addressed and shared, so two builds of the same early layers race on the same intermediate and the loser cannot export - no image of the wrong name is being deleted any more; the builder itself is the loser.The change
.forgejo/workflows/ci.yml,Build imagestep only:build_oncefunction (same command, same tag, sametimeout ... 180);unknown parent image ID/failed to export image, it is retried once, after 3 s, with a::warningthat names the race;COPY failed:-> one attempt, exit 1), a 180 s stop is not retried and still prints its::error(stub -> one attempt, exit 124), and a second failing attempt reddens the job with::error.Nothing else changes: same Dockerfile, same tree, same daemon, same 180 s bound per attempt (363 s worst case, inside the job's 10 min), and the retry can only pass by actually building the image. Both attempts are printed in full.
Measured
Build imagestep: median 11 s, max 13 s over runs 9348, 9334, 9338, 9342, 9322, 9315, 9307, 9281; docker job median 22 s.arm.Build image10 s, docker job 30 s,Successfully tagged packetforge:ci-run-9844, one build in the log (no retry fired). Legs 60/58/57 s, unchanged.Coverage is unchanged: same three Node legs, same smoke script, same 8 Dockerfile static checks, same image build, and this tree's own
check-ci-disk-hygienepassed in 9373.