CI: bound what the docker job writes to the host's disk #1500
Closed
jaylouisw
wants to merge 1 commit from
ci/bound-the-build-cache into main
pull from: ci/bound-the-build-cache
merge into: jaylouisw:main
jaylouisw:main
jaylouisw:rev/1121-1812
jaylouisw:catalogue/1121-1822
jaylouisw:docs/1121-1817
jaylouisw:catalogue/1121-1752
jaylouisw:docs/1121-1747
jaylouisw:rev/1121-1742
jaylouisw:catalogue/1121-1722
jaylouisw:rev/1121-1712
jaylouisw:docs/1121-1717
jaylouisw:catalogue/1121-1652
jaylouisw:rev/1121-1642
jaylouisw:docs/1121-1647
jaylouisw:ci/1121-1646
jaylouisw:ci/1121-1427
jaylouisw:rev/1121-1412
jaylouisw:catalogue/1121-1422
jaylouisw:docs/1121-1417
jaylouisw:catalogue/1121-1352
jaylouisw:rev/1121-1342
jaylouisw:docs/1121-1347
jaylouisw:rev/1121-1212
jaylouisw:catalogue/1121-1222
jaylouisw:docs/1121-1217
jaylouisw:ci/1121-1027
jaylouisw:catalogue/1121-1022
jaylouisw:rev/1121-1013
jaylouisw:docs/1121-1017
jaylouisw:catalogue/1121-0852
jaylouisw:rev/1121-0842
jaylouisw:docs/1121-0847
jaylouisw:ci/1121-0827
jaylouisw:catalogue/1121-0822
jaylouisw:rev/1121-0812
jaylouisw:catalogue/1121-0752
jaylouisw:rev/1121-0742
jaylouisw:docs/1121-0747
jaylouisw:catalogue/1121-0722
jaylouisw:docs/1121-0717
jaylouisw:rev/1121-0712
jaylouisw:rev/1121-0642
jaylouisw:plan/1121-0637
jaylouisw:ci/1121-0627
jaylouisw:catalogue/1121-0622
jaylouisw:rev/1121-0612
jaylouisw:docs/1121-0617
jaylouisw:plan/1120-0337
jaylouisw:docs/1120-0347
jaylouisw:rev/1119-1942
jaylouisw:rev/1119-1912
jaylouisw:catalogue/1119-1752
jaylouisw:docs/1119-1347
jaylouisw:catalogue/1119-1352
jaylouisw:fix/1119-1232
jaylouisw:fix/1119-0602
jaylouisw:catalogue/1119-0452
jaylouisw:catalogue/1119-0422
jaylouisw:docs/1119-0417
jaylouisw:fix/1119-0202
jaylouisw:catalogue/1119-0222
jaylouisw:rev/1119-0042
jaylouisw:catalogue/1118-2252
jaylouisw:docs/1118-2047
jaylouisw:catalogue/1118-2152
jaylouisw:fix/1118-2037
jaylouisw:ci/1118-2027
jaylouisw:docs/1118-1416
jaylouisw:fix/1118-1006
jaylouisw:catalogue/1118-0326
jaylouisw:catalogue/1117-2326
jaylouisw:catalogue/1117-0025
jaylouisw:catalogue/1116-1730
jaylouisw:fix/1523-speed-downgrade-one-end
jaylouisw:fix/enable-required-and-empty-pipe-help
jaylouisw:land-all-v5
jaylouisw:land-all-v4
jaylouisw:land-all-v3
jaylouisw:land-all-v2
jaylouisw:land-all
jaylouisw:fix/board-id
jaylouisw:fix/stated-range
jaylouisw:fix/no-form-names-v2
jaylouisw:fix/voice-vlan
jaylouisw:devices/catalogue-batch
jaylouisw:devices/powerline-moca
jaylouisw:devices/firewalls-enterprise
jaylouisw:devices/pon-splitters
jaylouisw:devices/nvr-and-taps
jaylouisw:devices/four-nvr-models
jaylouisw:fix/contributing-exemption
jaylouisw:fix/stale-github-urls
jaylouisw:ci/cancel-superseded-runs
jaylouisw:gh/1438
jaylouisw:device/uisp-console-lbe-xr
jaylouisw:gh/1442
jaylouisw:docs/readme-current-counts
jaylouisw:feat/device-ubiquiti-wave-fiber-onu
jaylouisw:fix/address-ownership
jaylouisw:feat/vrrp-in-path
jaylouisw:fix/merge-research-duplicate
jaylouisw:docs/hybrid-patch-current
jaylouisw:fix/audit-port-template
jaylouisw:fix/2930f-jl256a-uplinks
jaylouisw:feat/dhcp-in-path
jaylouisw:feat/hackgpon-ubiquiti-uisp-fiber-xgs
jaylouisw:feat/hackgpon-adtran-sdx-621
jaylouisw:tool/issue-duplicate-sweep
jaylouisw:feat/device-mikrotik-rb433ah
jaylouisw:feat/hackgpon-ma5671a
jaylouisw:issues/categories-and-labels
jaylouisw:ci/run-every-suite-locally
jaylouisw:chore/http-smoke
No reviewers
Labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set.
Reference
jaylouisw/packetforge!1500
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "ci/bound-the-build-cache"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #1499
What happened
On 2026-11-12 the forge ran out of disk in the middle of a working day. It never said
"disk full". It said three things that look unrelated:
git push→remote: error: couldn't set 'refs/heads/fix/short-line'/! [remote rejected] (failed to update ref).500withcould not write config file /data/gitea/data/tmp/local-repo/upload.git…/config: No space left on device.testjobs were never created. PR #1498had
Classify the changeandRebase onto mainand nothing else. Two force-pushesproduced no new run at all; a
workflow_dispatchon the same branch created onlyClassify the change— the one job with noneeds:. Everything downstream of aneeds:was silently absent, so the PR sat armed against checks that could neverexist.
main's own runs had been failing since ~06:32 for the same reason.What caused it
The
dockerjob in.forgejo/workflows/ci.ymlbuilds against the host daemon —node:20-bookwormhas no docker binary, so the job installs the CLI and talks to themounted
/var/run/docker.sock. That means the image layer cache it creates lives on thesame volume as Forgejo's database and repositories, and nothing ever removed it. Every
code push left a little more behind, forever.
What this changes
.forgejo/workflows/ci.yml,dockerjob:docker system dfanddf -h /are printed before thebuild and again after cleanup, so a growing volume reads as a trend in a log instead of
being discovered by a rejected push.
packetforge:ciexists to prove the Dockerfilebuilds; nothing consumes it, so it is removed. Dangling images older than a day are
pruned (
--filter until=24h, so no other service's fresh work is touched).docker builder prune -f --keep-storage 2GB,falling back to a plain prune when the daemon is docker 20.10 and has no such flag.
if: always()on the cleanup step: a build that dies half way is exactly the casethat leaves the most behind, and a cleanup written without it would be skipped then.
The rule is now enforced rather than remembered:
scripts/check-ci-disk-hygiene.js— reads every workflow file and requires that a jobrunning
docker buildalso prunes the builder cache, in a step carryingif: always(),and reports
docker system df. Full-line comments are stripped first, so prosedocumenting the rule cannot trip it.
npm run check:ci-disk.testjob runs it on every pull request.test/ci-disk-hygiene.test.js— 8 tests: the real workflows pass; a build with nocleanup is reported twice (
E_UNBOUNDED_DOCKER_BUILD,E_NO_DISK_REPORT); cleanupwithout
if: always()isE_PRUNE_SKIPPED_ON_FAILURE; cleanup in a different jobdoes not excuse the job that builds; a comment mentioning
docker buildis notevidence of one; the real cleanup step comes after the build, removes the image, bounds
the cache and reports the disk.
docs/CI.md§ The disk the jobs share — the three symptoms and what each oneactually is, since none of them looks like a disk.
Verification
check-ci-disk-hygieneexits 1 withE_UNBOUNDED_DOCKER_BUILDandE_NO_DISK_REPORT;test/ci-disk-hygiene.test.jsfails 2 of 8.
Every job that builds an image bounds what it leaves behind. (5 workflow file(s)),test suite 8/8.
node --checkon both new files; the workflow still parses (yaml.safe_load): jobschanges,test,smoke,docker; the cleanup step carriesif: always()and sitsafter the build.
Not claimed
This does not free the space that is already gone — that needs the host
(
docker system df, thendocker builder prune). It stops the next fill, and makes thenext one visible while it is happening.
Landed on
mainas5bc43ccf. Verified on the merge commit:check-ci-disk-hygienereports 5 workflow files bounded,ci-disk-hygiene8/8,skip-audit15/15, syntax 317 files, tree clean (the full suite was 247 suites / 9128 assertions / 0 failed on the branch before the merge).The dispatcher bug reproduced on this PR, which is the useful part. Runs created for #1500:
Classify the changeTest (Node 18/20/22),Server smoke test,Docker buildRebase onto main and arm auto-mergeSo it is not about pushes and not about disk space at dispatch time: the job with no
needs:is enqueued and executed, and every job with aneeds:never gets a task row. That is why the PR sat armed for five minutes with nothing running, and why I merged it with the checks unable to exist — the same decision, and the same recorded verification, as #1498.This does not free the space already gone from the host (
docker system df, thendocker builder prune); it stops the next fill and makes it visible while it happens.nothat names one thing must not remove another #1502no spanning-tree portfastcleared the bridge priority, plus two smaller untruths in the console #1501Pull request closed