ci(docker): stop pruning the intermediates the build cache still names #2193

Merged
jaylouisw merged 2 commits from ci/1119-1827 into main 2026-11-19 18:39:46 +00:00
Owner

What was red, and why

Docker build reddened main twice today with the same signature, while the other nine checks on those commits were green:

Step 2/8 : WORKDIR /app  ---> Using cache
Step 3/8 : COPY . .
failed to export image: failed to set parent sha256:39e26cbad...
unknown parent image ID sha256:39e26cbad...
  • run 7378 — task 19408, push to main (05686c6), 2026-11-19T09:44:57Z, Failing after 14s
  • run 7762 — task 20470, push to main (df8e41d), 2026-11-19T17:10:54Z, Failing after 15s

Cause: the always() cleanup step ran docker image prune -f --filter "until=24h", which deletes old dangling images — and with the legacy builder those are exactly the parents its cache names. WORKDIR /app has not changed in weeks, so the intermediate behind it is older than a day while the cache entry that says ---> Using cache is not. The prune removes the image, leaves the entry, and the next build dies at the first step it cannot reuse. docker system df in run 7762 reported 6.7 GB (21%) of images reclaimable, i.e. the prune had plenty to delete each run. Because arm needs docker, this also held the merge gate behind a cache left over from an unrelated earlier push.

Baseline

job duration outcome run / task
Docker build (main) 14 s failure, unknown parent image ID run 7378 / task 19408
Docker build (main) 15 s failure, same run 7762 / task 20470
Docker build (green, unchanged code path) 15-25 s success runs 7803, 7800

Change

Workflow only — .forgejo/workflows/ci.yml, docker job, cleanup step: the docker image prune -f --filter "until=24h" line is removed. Everything else is untouched: if: always(), docker image rm -f packetforge:ci (the only image this job creates), docker builder prune -f --keep-storage 2GB (which bounds the cache and its intermediate images coherently), and docker system df.

(I first shipped a retry that cleared the cache from inside the build step. That was the wrong shape: test/ci-disk-hygiene.test.js requires cleanup to live after the build in the always() step, and it failed the matrix — correctly. Reverted; this is the root fix.)

After

  • run 7815, job 5: CI / Docker build (pull_request) — Successful in 19s.
  • run 7815, job 1: Test (Node 18) — Successful in 1m16s (contended run).

Coverage

Unchanged. The image is still built on every code change, from the same Dockerfile, against the same host daemon; the disk bound of this job is unchanged and its rule checker is green:

  • node scripts/check-ci-disk-hygiene.js → "Every job that builds an image bounds what it leaves behind. (6 workflow file(s))"
  • node --test test/ci-disk-hygiene.test.js → 12/12
  • full suite locally: 382 suites, 12890 assertions, 0 failed

No check was weakened, skipped or deleted; the only thing that changed is that a stale local cache can no longer decide the colour of a check about a commit.

## What was red, and why `Docker build` reddened **main** twice today with the same signature, while the other nine checks on those commits were green: ``` Step 2/8 : WORKDIR /app ---> Using cache Step 3/8 : COPY . . failed to export image: failed to set parent sha256:39e26cbad... unknown parent image ID sha256:39e26cbad... ``` - **run 7378** — task 19408, push to main (`05686c6`), 2026-11-19T09:44:57Z, `Failing after 14s` - **run 7762** — task 20470, push to main (`df8e41d`), 2026-11-19T17:10:54Z, `Failing after 15s` Cause: the always() cleanup step ran `docker image prune -f --filter "until=24h"`, which deletes *old dangling* images — and with the legacy builder those are exactly the parents its cache names. `WORKDIR /app` has not changed in weeks, so the intermediate behind it is older than a day while the cache entry that says `---> Using cache` is not. The prune removes the image, leaves the entry, and the next build dies at the first step it cannot reuse. `docker system df` in run 7762 reported 6.7 GB (21%) of images reclaimable, i.e. the prune had plenty to delete each run. Because `arm` needs `docker`, this also held the merge gate behind a cache left over from an unrelated earlier push. ## Baseline | job | duration | outcome | run / task | |---|---|---|---| | Docker build (main) | 14 s | failure, `unknown parent image ID` | run 7378 / task 19408 | | Docker build (main) | 15 s | failure, same | run 7762 / task 20470 | | Docker build (green, unchanged code path) | 15-25 s | success | runs 7803, 7800 | ## Change Workflow only — `.forgejo/workflows/ci.yml`, `docker` job, cleanup step: the `docker image prune -f --filter "until=24h"` line is removed. Everything else is untouched: `if: always()`, `docker image rm -f packetforge:ci` (the only image this job creates), `docker builder prune -f --keep-storage 2GB` (which bounds the cache *and* its intermediate images coherently), and `docker system df`. (I first shipped a retry that cleared the cache from inside the build step. That was the wrong shape: `test/ci-disk-hygiene.test.js` requires cleanup to live after the build in the always() step, and it failed the matrix — correctly. Reverted; this is the root fix.) ## After - **run 7815**, job 5: `CI / Docker build (pull_request)` — **Successful in 19s**. - **run 7815**, job 1: `Test (Node 18)` — Successful in 1m16s (contended run). ## Coverage Unchanged. The image is still built on every code change, from the same Dockerfile, against the same host daemon; the disk bound of this job is unchanged and its rule checker is green: - `node scripts/check-ci-disk-hygiene.js` → "Every job that builds an image bounds what it leaves behind. (6 workflow file(s))" - `node --test test/ci-disk-hygiene.test.js` → 12/12 - full suite locally: **382 suites, 12890 assertions, 0 failed** No check was weakened, skipped or deleted; the only thing that changed is that a stale local cache can no longer decide the colour of a check about a commit.
ci(docker): survive a builder cache whose parent image was trimmed away
Some checks failed
CI / Classify the change (pull_request) Successful in 14s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 10s
CI / Server smoke test (pull_request) Successful in 10s
CI / Test (Node 20) (pull_request) Failing after 56s
CI / Test (Node 22) (pull_request) Failing after 55s
CI / Test (Node 18) (pull_request) Failing after 59s
CI / Docker build (pull_request) Successful in 22s
CI / Arm auto-merge (pull_request) Successful in 3s
0740bf182a
main went red twice today on a build that had nothing to do with the commit:
run 7378 (task 19408, 09:44:57Z) failed after 14 s and run 7762 (task 20470,
17:10:54Z) after 15 s, both with

  Step 2/8 : WORKDIR /app  ---> Using cache
  Step 3/8 : COPY . .
  failed to export image: failed to set parent sha256:39e26cbad...
  unknown parent image ID sha256:39e26cbad...

The legacy builder reused a cached layer whose parent intermediate image the
post-build '-keep-storage 2GB' trim had already dropped. The build step now
recognises that one signature, clears the builder cache (bounded) and builds
the same Dockerfile once more; every other failure, including a real Dockerfile
error, still fails on the first attempt with the same exit status. The 2 GB
bound and the always() cleanup step are unchanged.
ci(docker): stop pruning the intermediates the build cache still names
All checks were successful
CI / Classify the change (pull_request) Successful in 28s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 19s
CI / Server smoke test (pull_request) Successful in 14s
CI / Test (Node 22) (pull_request) Successful in 1m8s
CI / Test (Node 20) (pull_request) Successful in 1m11s
CI / Test (Node 18) (pull_request) Successful in 1m16s
CI / Docker build (pull_request) Successful in 19s
CI / Arm auto-merge (pull_request) Successful in 3s
e198f84fed
Second half of the fix. The retry I first shipped here was the wrong shape: it
cleared the cache with a prune inside the build step, which the repository's own
disk rule forbids (cleanup after the build, in the always() step), and it would
have paid a failed attempt on every later run.

The cause is one line in that always() step: 'docker image prune -f --filter
until=24h' deletes old dangling images, and with the legacy builder those are
exactly the parents its cache names - WORKDIR /app has not changed in weeks, so
the intermediate behind it is older than a day while the cache entry is not.
main went red at COPY twice today because of it:

  run 7378 (task 19408, 09:44:57Z)  Failing after 14s
  run 7762 (task 20470, 17:10:54Z)  Failing after 15s
    failed to export image: failed to set parent sha256:39e26cbad...
    unknown parent image ID sha256:39e26cbad...

The image this job creates is already removed by name; the layers behind it are
bounded by the builder prune on the next line (--keep-storage 2GB, unchanged),
so this line only ever removed state the next build needed.
forgejo-actions scheduled this pull request to auto merge when all checks succeed 2026-11-19 18:37:27 +00:00
jaylouisw changed title from ci(docker): survive a builder cache whose parent image was trimmed away to ci(docker): stop pruning the intermediates the build cache still names 2026-11-19 18:38:01 +00:00
jaylouisw deleted branch ci/1119-1827 2026-11-19 18:39:48 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jaylouisw/packetforge!2193
No description provided.