ci(docker): a per-run image tag, and a cache prune that waits for the daemon #2205

Merged
jaylouisw merged 2 commits from ci/1119-2027 into main 2026-11-19 20:43:36 +00:00
Owner

Baseline. Docker build failed 5 times in the 72 h to 2026-11-19T20:27Z (757 tasks, median 21 s). Every failure died 14-19 s into the job, and in 4 of 5 another Docker build on the same host daemon finished 13-16 s into it - the second that job's cleanup ran:

failing task run started died concurrent build finished
20834 7887 20:11:18Z 14 s run 7883 at 20:11:31
20470 7762 17:10:54Z 15 s run 7758 at 17:11:10
19408 7378 09:44:57Z 14 s run 7374 at 09:45:10
19273 7330 08:45:26Z 19 s run 7326 at 08:45:42

Every build tagged packetforge:ci, one shared name, and cleanup ran docker image rm -f packetforge:ci - which removes the image, because the tag is its only reference, and that image was the cache parent the concurrent build was exporting against. Same signature #2193 chased (failed to export image: ... unknown parent image ID); this was the other half of it, and arm has needs: docker, so each one held a fully green pull request's merge behind a cache collision.

Change.

  • The image is tagged packetforge:ci-run-<run_id> and cleanup removes exactly that tag, so no run can reach another run's image. Keyed on the run, not the sha: two runs of the same commit would share a sha tag.
  • The two calls that touch the shared cache (removal of the pre-change packetforge:ci leftovers, and docker builder prune --keep-storage 2GB) run only when the daemon is this job's alone. It is one bounded call (actions/tasks?status=running&limit=50, ~0.2 s) and the cleanup runs inside its own running task, so a count of one means nothing else is building. If the API cannot be read the prune runs as today - an unreadable API must not become an unbounded cache.

After. Run 7895 (commit ea60729c) and runs 7898, 7901, 7904: Docker build green in 21 s, 22 s, 22 s, 19 s - baseline median 21 s, no cost added. Final commit 5a875bee: 8/8 green (Node 18 1m1s, Node 20 56s, Node 22 57s, smoke 11s, Docker 22s, arm 4s).

Coverage unchanged. Same Dockerfile, same scripts/check-dockerfile.js, same docker build (180 s bound), same 2 GB cache bound, same image removed under if: always(), same docker system df. scripts/check-ci-disk-hygiene.js and test/ci-disk-hygiene.test.js pass unmodified (12/12). Nothing skipped, softened or retried.

Workflows only; engine/, test/ and docs/ untouched.

**Baseline.** `Docker build` failed 5 times in the 72 h to 2026-11-19T20:27Z (757 tasks, median 21 s). Every failure died 14-19 s into the job, and in 4 of 5 another `Docker build` on the same host daemon finished 13-16 s into it - the second that job's cleanup ran: | failing task | run | started | died | concurrent build finished | |---|---|---|---|---| | 20834 | 7887 | 20:11:18Z | 14 s | run 7883 at 20:11:31 | | 20470 | 7762 | 17:10:54Z | 15 s | run 7758 at 17:11:10 | | 19408 | 7378 | 09:44:57Z | 14 s | run 7374 at 09:45:10 | | 19273 | 7330 | 08:45:26Z | 19 s | run 7326 at 08:45:42 | Every build tagged `packetforge:ci`, one shared name, and cleanup ran `docker image rm -f packetforge:ci` - which removes the *image*, because the tag is its only reference, and that image was the cache parent the concurrent build was exporting against. Same signature #2193 chased (`failed to export image: ... unknown parent image ID`); this was the other half of it, and `arm` has `needs: docker`, so each one held a fully green pull request's merge behind a cache collision. **Change.** - The image is tagged `packetforge:ci-run-<run_id>` and cleanup removes exactly that tag, so no run can reach another run's image. Keyed on the run, not the sha: two runs of the same commit would share a sha tag. - The two calls that touch the *shared* cache (removal of the pre-change `packetforge:ci` leftovers, and `docker builder prune --keep-storage 2GB`) run only when the daemon is this job's alone. It is one bounded call (`actions/tasks?status=running&limit=50`, ~0.2 s) and the cleanup runs inside its own running task, so a count of one means nothing else is building. If the API cannot be read the prune runs as today - an unreadable API must not become an unbounded cache. **After.** Run 7895 (commit ea60729c) and runs 7898, 7901, 7904: `Docker build` green in 21 s, 22 s, 22 s, 19 s - baseline median 21 s, no cost added. Final commit 5a875bee: 8/8 green (Node 18 1m1s, Node 20 56s, Node 22 57s, smoke 11s, Docker 22s, arm 4s). **Coverage unchanged.** Same Dockerfile, same `scripts/check-dockerfile.js`, same `docker build` (180 s bound), same 2 GB cache bound, same image removed under `if: always()`, same `docker system df`. `scripts/check-ci-disk-hygiene.js` and `test/ci-disk-hygiene.test.js` pass unmodified (12/12). Nothing skipped, softened or retried. Workflows only; `engine/`, `test/` and `docs/` untouched.
ci(docker): tag the image per commit, and prune only when no other build is running
All checks were successful
CI / Classify the change (pull_request) Successful in 14s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 12s
CI / Server smoke test (pull_request) Successful in 10s
CI / Test (Node 22) (pull_request) Successful in 55s
CI / Test (Node 20) (pull_request) Successful in 56s
CI / Test (Node 18) (pull_request) Successful in 1m1s
CI / Docker build (pull_request) Successful in 21s
CI / Arm auto-merge (pull_request) Successful in 3s
ea60729cc9
Five `Docker build` tasks failed in the 72 hours to 2026-11-19T20:27Z, each
14-19 s into a job whose green median is 21 s, and each while another
`Docker build` on the same host daemon was finishing - run 7887/task 20834
(20:11:18Z) against run 7883 finishing at 20:11:31, 7762 against 7758, 7378
against 7374, 7330 against 7326. The finishing job's cleanup removed the one
tag every build shared (`packetforge:ci`), which removes the image, which is
the cache parent the other build was exporting against: 'failed to export
image: ... unknown parent image ID', the signature #2193 already chased.

Builds now tag `packetforge:ci-<sha>` and remove exactly that tag, and the
builder prune - which drops least-recently-used intermediates that a
concurrent build may be naming as parents - stands down while another
`Docker build` task is running, which is one bounded call to the actions
API. Coverage is unchanged: same Dockerfile, same static checks, same
`docker build`, same 2 GB cache bound, same image removed afterwards.
forgejo-actions scheduled this pull request to auto merge when all checks succeed 2026-11-19 20:37:14 +00:00
ci(docker): key the tag on the run, and count rather than identify
All checks were successful
CI / Classify the change (pull_request) Successful in 8s
PR rebase / Rebase onto main and arm auto-merge (pull_request) Successful in 7s
CI / Server smoke test (pull_request) Successful in 11s
CI / Test (Node 20) (pull_request) Successful in 56s
CI / Test (Node 18) (pull_request) Successful in 1m1s
CI / Test (Node 22) (pull_request) Successful in 57s
CI / Docker build (pull_request) Successful in 22s
CI / Arm auto-merge (pull_request) Successful in 4s
5a875bee54
Two corrections to the previous commit, both about an expression that can
silently return nothing:

- the tag is `packetforge:ci-run-<run_id>`, not the commit sha. Two runs of
  the same commit (a re-run, two branches on one head) would share a sha tag
  and could still remove each other's image, which is the failure the tag
  exists to stop. `github.run_id` is already resolved elsewhere in these
  workflows (pr-rebase.yml) so it is known to work here.
- the prune guard counts running `Docker build` tasks instead of filtering
  out this run's own by run number. The cleanup runs inside the running task,
  so one means the daemon is this job; a count needs no identity to resolve,
  and an identity that resolves empty would have said 'someone else is
  building' forever and switched the 2 GB bound off.

Same coverage: same build command, same 2 GB bound, same image removed.
jaylouisw deleted branch ci/1119-2027 2026-11-19 20:43:37 +00:00
jaylouisw changed title from ci(docker): a per-commit image tag, and a prune that waits for the daemon to ci(docker): a per-run image tag, and a cache prune that waits for the daemon 2026-11-19 20:47:09 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jaylouisw/packetforge!2205
No description provided.