A CI job that writes to the host's disk has to bound what it writes #1499
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
jaylouisw/packetforge#1499
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
What happened
On 2026-11-12 the forge ran out of disk in the middle of a working day. It never said
"disk full". It said three things that look unrelated:
git push→remote: error: couldn't set 'refs/heads/fix/short-line'/! [remote rejected] (failed to update ref).500withcould not write config file /data/gitea/data/tmp/local-repo/upload.git…/config: No space left on device.testjobs were never created. PR #1498had
Classify the changeandRebase onto mainand nothing else. Two force-pushesproduced no new run at all; a
workflow_dispatchon the same branch created onlyClassify the change— the one job with noneeds:. Everything downstream of aneeds:was silently absent, so the PR sat armed against checks that could neverexist.
main's own runs had been failing since ~06:32 for the same reason.What caused it
The
dockerjob in.forgejo/workflows/ci.ymlbuilds against the host daemon —node:20-bookwormhas no docker binary, so the job installs the CLI and talks to themounted
/var/run/docker.sock. That means the image layer cache it creates lives on thesame volume as Forgejo's database and repositories, and nothing ever removed it. Every
code push left a little more behind, forever.
What this changes
.forgejo/workflows/ci.yml,dockerjob:docker system dfanddf -h /are printed before thebuild and again after cleanup, so a growing volume reads as a trend in a log instead of
being discovered by a rejected push.
packetforge:ciexists to prove the Dockerfilebuilds; nothing consumes it, so it is removed. Dangling images older than a day are
pruned (
--filter until=24h, so no other service's fresh work is touched).docker builder prune -f --keep-storage 2GB,falling back to a plain prune when the daemon is docker 20.10 and has no such flag.
if: always()on the cleanup step: a build that dies half way is exactly the casethat leaves the most behind, and a cleanup written without it would be skipped then.
The rule is now enforced rather than remembered:
scripts/check-ci-disk-hygiene.js— reads every workflow file and requires that a jobrunning
docker buildalso prunes the builder cache, in a step carryingif: always(),and reports
docker system df. Full-line comments are stripped first, so prosedocumenting the rule cannot trip it.
npm run check:ci-disk.testjob runs it on every pull request.test/ci-disk-hygiene.test.js— 8 tests: the real workflows pass; a build with nocleanup is reported twice (
E_UNBOUNDED_DOCKER_BUILD,E_NO_DISK_REPORT); cleanupwithout
if: always()isE_PRUNE_SKIPPED_ON_FAILURE; cleanup in a different jobdoes not excuse the job that builds; a comment mentioning
docker buildis notevidence of one; the real cleanup step comes after the build, removes the image, bounds
the cache and reports the disk.
docs/CI.md§ The disk the jobs share — the three symptoms and what each oneactually is, since none of them looks like a disk.
Verification
check-ci-disk-hygieneexits 1 withE_UNBOUNDED_DOCKER_BUILDandE_NO_DISK_REPORT;test/ci-disk-hygiene.test.jsfails 2 of 8.
Every job that builds an image bounds what it leaves behind. (5 workflow file(s)),test suite 8/8.
node --checkon both new files; the workflow still parses (yaml.safe_load): jobschanges,test,smoke,docker; the cleanup step carriesif: always()and sitsafter the build.
Not claimed
This does not free the space that is already gone — that needs the host
(
docker system df, thendocker builder prune). It stops the next fill, and makes thenext one visible while it is happening.
Landed on
mainas5bc43ccf. Verified on the merge commit:check-ci-disk-hygienereports 5 workflow files bounded,ci-disk-hygiene8/8,skip-audit15/15, syntax 317 files, tree clean (the full suite was 247 suites / 9128 assertions / 0 failed on the branch before the merge).The dispatcher bug reproduced on this PR, which is the useful part. Runs created for #1500:
Classify the changeTest (Node 18/20/22),Server smoke test,Docker buildRebase onto main and arm auto-mergeSo it is not about pushes and not about disk space at dispatch time: the job with no
needs:is enqueued and executed, and every job with aneeds:never gets a task row. That is why the PR sat armed for five minutes with nothing running, and why I merged it with the checks unable to exist — the same decision, and the same recorded verification, as #1498.This does not free the space already gone from the host (
docker system df, thendocker builder prune); it stops the next fill and makes it visible while it happens.