ci(docker): stop pruning the intermediates the build cache still names #2193
No reviewers
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
jaylouisw/packetforge!2193
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "ci/1119-1827"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What was red, and why
Docker buildreddened main twice today with the same signature, while the other nine checks on those commits were green:05686c6), 2026-11-19T09:44:57Z,Failing after 14sdf8e41d), 2026-11-19T17:10:54Z,Failing after 15sCause: the always() cleanup step ran
docker image prune -f --filter "until=24h", which deletes old dangling images — and with the legacy builder those are exactly the parents its cache names.WORKDIR /apphas not changed in weeks, so the intermediate behind it is older than a day while the cache entry that says---> Using cacheis not. The prune removes the image, leaves the entry, and the next build dies at the first step it cannot reuse.docker system dfin run 7762 reported 6.7 GB (21%) of images reclaimable, i.e. the prune had plenty to delete each run. Becausearmneedsdocker, this also held the merge gate behind a cache left over from an unrelated earlier push.Baseline
unknown parent image IDChange
Workflow only —
.forgejo/workflows/ci.yml,dockerjob, cleanup step: thedocker image prune -f --filter "until=24h"line is removed. Everything else is untouched:if: always(),docker image rm -f packetforge:ci(the only image this job creates),docker builder prune -f --keep-storage 2GB(which bounds the cache and its intermediate images coherently), anddocker system df.(I first shipped a retry that cleared the cache from inside the build step. That was the wrong shape:
test/ci-disk-hygiene.test.jsrequires cleanup to live after the build in the always() step, and it failed the matrix — correctly. Reverted; this is the root fix.)After
CI / Docker build (pull_request)— Successful in 19s.Test (Node 18)— Successful in 1m16s (contended run).Coverage
Unchanged. The image is still built on every code change, from the same Dockerfile, against the same host daemon; the disk bound of this job is unchanged and its rule checker is green:
node scripts/check-ci-disk-hygiene.js→ "Every job that builds an image bounds what it leaves behind. (6 workflow file(s))"node --test test/ci-disk-hygiene.test.js→ 12/12No check was weakened, skipped or deleted; the only thing that changed is that a stale local cache can no longer decide the colour of a check about a commit.
Second half of the fix. The retry I first shipped here was the wrong shape: it cleared the cache with a prune inside the build step, which the repository's own disk rule forbids (cleanup after the build, in the always() step), and it would have paid a failed attempt on every later run. The cause is one line in that always() step: 'docker image prune -f --filter until=24h' deletes old dangling images, and with the legacy builder those are exactly the parents its cache names - WORKDIR /app has not changed in weeks, so the intermediate behind it is older than a day while the cache entry is not. main went red at COPY twice today because of it: run 7378 (task 19408, 09:44:57Z) Failing after 14s run 7762 (task 20470, 17:10:54Z) Failing after 15s failed to export image: failed to set parent sha256:39e26cbad... unknown parent image ID sha256:39e26cbad... The image this job creates is already removed by name; the layers behind it are bounded by the builder prune on the next line (--keep-storage 2GB, unchanged), so this line only ever removed state the next build needed.ci(docker): survive a builder cache whose parent image was trimmed awayto ci(docker): stop pruning the intermediates the build cache still namesjaylouisw referenced this pull request2026-11-21 12:51:55 +00:00