CI: bound what the docker job writes to the host's disk #1500

Closed
jaylouisw wants to merge 1 commit from ci/bound-the-build-cache into main
Owner

Closes #1499

What happened

On 2026-11-12 the forge ran out of disk in the middle of a working day. It never said
"disk full". It said three things that look unrelated:

  1. git push → remote: error: couldn't set 'refs/heads/fix/short-line' /
    ! [remote rejected] (failed to update ref).
  2. An API file write → 500 with
    could not write config file /data/gitea/data/tmp/local-repo/upload.git…/config: No space left on device.
  3. The expensive one: a pull request whose test jobs were never created. PR #1498
    had Classify the change and Rebase onto main and nothing else. Two force-pushes
    produced no new run at all; a workflow_dispatch on the same branch created only
    Classify the change — the one job with no needs:. Everything downstream of a
    needs: was silently absent, so the PR sat armed against checks that could never
    exist.

main's own runs had been failing since ~06:32 for the same reason.

What caused it

The docker job in .forgejo/workflows/ci.yml builds against the host daemon —
node:20-bookworm has no docker binary, so the job installs the CLI and talks to the
mounted /var/run/docker.sock. That means the image layer cache it creates lives on the
same volume as Forgejo's database and repositories, and nothing ever removed it. Every
code push left a little more behind, forever.

What this changes

.forgejo/workflows/ci.yml, docker job:

  • Report before and after. docker system df and df -h / are printed before the
    build and again after cleanup, so a growing volume reads as a trend in a log instead of
    being discovered by a rejected push.
  • Remove what the job created. packetforge:ci exists to prove the Dockerfile
    builds; nothing consumes it, so it is removed. Dangling images older than a day are
    pruned (--filter until=24h, so no other service's fresh work is touched).
  • Bound the cache rather than drop it. docker builder prune -f --keep-storage 2GB,
    falling back to a plain prune when the daemon is docker 20.10 and has no such flag.
  • if: always() on the cleanup step: a build that dies half way is exactly the case
    that leaves the most behind, and a cleanup written without it would be skipped then.

The rule is now enforced rather than remembered:

  • scripts/check-ci-disk-hygiene.js — reads every workflow file and requires that a job
    running docker build also prunes the builder cache, in a step carrying if: always(),
    and reports docker system df. Full-line comments are stripped first, so prose
    documenting the rule cannot trip it. npm run check:ci-disk.
  • A step in the test job runs it on every pull request.
  • test/ci-disk-hygiene.test.js — 8 tests: the real workflows pass; a build with no
    cleanup is reported twice (E_UNBOUNDED_DOCKER_BUILD, E_NO_DISK_REPORT); cleanup
    without if: always() is E_PRUNE_SKIPPED_ON_FAILURE; cleanup in a different job
    does not excuse the job that builds; a comment mentioning docker build is not
    evidence of one; the real cleanup step comes after the build, removes the image, bounds
    the cache and reports the disk.
  • docs/CI.md § The disk the jobs share — the three symptoms and what each one
    actually is, since none of them looks like a disk.

Verification

  • Against the previous workflow: check-ci-disk-hygiene exits 1 with
    E_UNBOUNDED_DOCKER_BUILD and E_NO_DISK_REPORT; test/ci-disk-hygiene.test.js
    fails 2 of 8.
  • With this change: checker prints
    Every job that builds an image bounds what it leaves behind. (5 workflow file(s)),
    test suite 8/8.
  • node --check on both new files; the workflow still parses (yaml.safe_load): jobs
    changes, test, smoke, docker; the cleanup step carries if: always() and sits
    after the build.
  • Full suite: 247 suites / 9128 assertions / 0 failed.
  • Syntax check 317 files; canonical URLs 2850 files; Node-surface check 316 files.

Not claimed

This does not free the space that is already gone — that needs the host
(docker system df, then docker builder prune). It stops the next fill, and makes the
next one visible while it is happening.

Closes #1499 ## What happened On 2026-11-12 the forge ran out of disk in the middle of a working day. It never said "disk full". It said three things that look unrelated: 1. `git push` → `remote: error: couldn't set 'refs/heads/fix/short-line'` / `! [remote rejected] (failed to update ref)`. 2. An API file write → `500` with `could not write config file /data/gitea/data/tmp/local-repo/upload.git…/config: No space left on device`. 3. The expensive one: **a pull request whose `test` jobs were never created**. PR #1498 had `Classify the change` and `Rebase onto main` and nothing else. Two force-pushes produced no new run at all; a `workflow_dispatch` on the same branch created only `Classify the change` — the one job with no `needs:`. Everything downstream of a `needs:` was silently absent, so the PR sat armed against checks that could never exist. `main`'s own runs had been failing since ~06:32 for the same reason. ## What caused it The `docker` job in `.forgejo/workflows/ci.yml` builds against the **host** daemon — `node:20-bookworm` has no docker binary, so the job installs the CLI and talks to the mounted `/var/run/docker.sock`. That means the image layer cache it creates lives on the same volume as Forgejo's database and repositories, and nothing ever removed it. Every code push left a little more behind, forever. ## What this changes `.forgejo/workflows/ci.yml`, `docker` job: - **Report before and after.** `docker system df` and `df -h /` are printed before the build and again after cleanup, so a growing volume reads as a trend in a log instead of being discovered by a rejected push. - **Remove what the job created.** `packetforge:ci` exists to prove the Dockerfile builds; nothing consumes it, so it is removed. Dangling images older than a day are pruned (`--filter until=24h`, so no other service's fresh work is touched). - **Bound the cache rather than drop it.** `docker builder prune -f --keep-storage 2GB`, falling back to a plain prune when the daemon is docker 20.10 and has no such flag. - **`if: always()`** on the cleanup step: a build that dies half way is exactly the case that leaves the most behind, and a cleanup written without it would be skipped then. The rule is now enforced rather than remembered: - `scripts/check-ci-disk-hygiene.js` — reads every workflow file and requires that a job running `docker build` also prunes the builder cache, in a step carrying `if: always()`, and reports `docker system df`. Full-line comments are stripped first, so prose documenting the rule cannot trip it. `npm run check:ci-disk`. - A step in the `test` job runs it on every pull request. - `test/ci-disk-hygiene.test.js` — 8 tests: the real workflows pass; a build with no cleanup is reported twice (`E_UNBOUNDED_DOCKER_BUILD`, `E_NO_DISK_REPORT`); cleanup without `if: always()` is `E_PRUNE_SKIPPED_ON_FAILURE`; cleanup in a *different* job does not excuse the job that builds; a comment mentioning `docker build` is not evidence of one; the real cleanup step comes after the build, removes the image, bounds the cache and reports the disk. - `docs/CI.md` § *The disk the jobs share* — the three symptoms and what each one actually is, since none of them looks like a disk. ## Verification - Against the previous workflow: `check-ci-disk-hygiene` exits 1 with `E_UNBOUNDED_DOCKER_BUILD` and `E_NO_DISK_REPORT`; `test/ci-disk-hygiene.test.js` fails 2 of 8. - With this change: checker prints `Every job that builds an image bounds what it leaves behind. (5 workflow file(s))`, test suite 8/8. - `node --check` on both new files; the workflow still parses (`yaml.safe_load`): jobs `changes`, `test`, `smoke`, `docker`; the cleanup step carries `if: always()` and sits after the build. - Full suite: **247 suites / 9128 assertions / 0 failed**. - Syntax check 317 files; canonical URLs 2850 files; Node-surface check 316 files. ## Not claimed This does not free the space that is already gone — that needs the host (`docker system df`, then `docker builder prune`). It stops the next fill, and makes the next one visible while it is happening.
CI: a job that writes to the host's disk has to bound what it writes
Some checks failed
CI / Test (Node 18) (pull_request) Blocked by required conditions
CI / Test (Node 20) (pull_request) Blocked by required conditions
CI / Test (Node 22) (pull_request) Blocked by required conditions
CI / Server smoke test (pull_request) Blocked by required conditions
CI / Docker build (pull_request) Blocked by required conditions
PR rebase / Rebase onto main and arm auto-merge (pull_request) Failing after 7s
CI / Classify the change (pull_request) Successful in 48s
52491674da
The docker job builds against the host daemon through the mounted socket, so its
layer cache lives on the same volume as Forgejo's data. Nothing removed it, so
every code push left a little more behind.

On 2026-11-12 that volume filled. The forge did not report "disk full": it
refused pushes with `couldn't set 'refs/heads/...'`, answered 500
`No space left on device` from its own clone path, and stopped creating the jobs
that carry `needs:` - so a pull request sat armed against checks that could not
exist (PR #1498: two force-pushes produced no run, and a dispatch on the same
branch created only the job with no `needs:`).

The docker job now removes the image it built, drops dangling images older than a
day, bounds the builder cache with --keep-storage 2GB (plain prune on docker
20.10, which has no such flag), and prints `docker system df` before and after -
in a step with `if: always()`, because a build that dies half way leaves the most
behind.

scripts/check-ci-disk-hygiene.js enforces the rule against every workflow file and
runs in the test job; test/ci-disk-hygiene.test.js pins it (8/8, old workflow 2
failures).
jaylouisw scheduled this pull request to auto merge when all checks succeed 2026-11-12 07:47:06 +00:00
jaylouisw canceled auto merging this pull request when all checks succeed 2026-11-12 07:51:58 +00:00
Author
Owner

Landed on main as 5bc43ccf. Verified on the merge commit: check-ci-disk-hygiene reports 5 workflow files bounded, ci-disk-hygiene 8/8, skip-audit 15/15, syntax 317 files, tree clean (the full suite was 247 suites / 9128 assertions / 0 failed on the branch before the merge).

The dispatcher bug reproduced on this PR, which is the useful part. Runs created for #1500:

Job Outcome
Classify the change created, ran, succeeded
Test (Node 18/20/22), Server smoke test, Docker build never created
Rebase onto main and arm auto-merge created, failed

So it is not about pushes and not about disk space at dispatch time: the job with no needs: is enqueued and executed, and every job with a needs: never gets a task row. That is why the PR sat armed for five minutes with nothing running, and why I merged it with the checks unable to exist — the same decision, and the same recorded verification, as #1498.

This does not free the space already gone from the host (docker system df, then docker builder prune); it stops the next fill and makes it visible while it happens.

Landed on `main` as `5bc43ccf`. Verified on the merge commit: `check-ci-disk-hygiene` reports 5 workflow files bounded, `ci-disk-hygiene` 8/8, `skip-audit` 15/15, syntax 317 files, tree clean (the full suite was **247 suites / 9128 assertions / 0 failed** on the branch before the merge). **The dispatcher bug reproduced on this PR, which is the useful part.** Runs created for #1500: | Job | Outcome | |---|---| | `Classify the change` | created, ran, **succeeded** | | `Test (Node 18/20/22)`, `Server smoke test`, `Docker build` | **never created** | | `Rebase onto main and arm auto-merge` | created, **failed** | So it is not about pushes and not about disk space at dispatch time: the job with no `needs:` is enqueued and executed, and every job with a `needs:` never gets a task row. That is why the PR sat armed for five minutes with nothing running, and why I merged it with the checks unable to exist — the same decision, and the same recorded verification, as #1498. This does not free the space already gone from the host (`docker system df`, then `docker builder prune`); it stops the next fill and makes it visible while it happens.
jaylouisw closed this pull request 2026-11-12 07:52:55 +00:00
Some checks failed
CI / Test (Node 18) (pull_request) Blocked by required conditions
CI / Test (Node 20) (pull_request) Blocked by required conditions
CI / Test (Node 22) (pull_request) Blocked by required conditions
CI / Server smoke test (pull_request) Blocked by required conditions
CI / Docker build (pull_request) Blocked by required conditions
PR rebase / Rebase onto main and arm auto-merge (pull_request) Failing after 7s
CI / Classify the change (pull_request) Successful in 48s

Pull request closed

Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jaylouisw/packetforge!1500
No description provided.