A CI job that writes to the host's disk has to bound what it writes #1499

Closed
opened 2026-11-12 07:46:52 +00:00 by jaylouisw · 1 comment
Owner

What happened

On 2026-11-12 the forge ran out of disk in the middle of a working day. It never said
"disk full". It said three things that look unrelated:

  1. git push → remote: error: couldn't set 'refs/heads/fix/short-line' /
    ! [remote rejected] (failed to update ref).
  2. An API file write → 500 with
    could not write config file /data/gitea/data/tmp/local-repo/upload.git…/config: No space left on device.
  3. The expensive one: a pull request whose test jobs were never created. PR #1498
    had Classify the change and Rebase onto main and nothing else. Two force-pushes
    produced no new run at all; a workflow_dispatch on the same branch created only
    Classify the change — the one job with no needs:. Everything downstream of a
    needs: was silently absent, so the PR sat armed against checks that could never
    exist.

main's own runs had been failing since ~06:32 for the same reason.

What caused it

The docker job in .forgejo/workflows/ci.yml builds against the host daemon —
node:20-bookworm has no docker binary, so the job installs the CLI and talks to the
mounted /var/run/docker.sock. That means the image layer cache it creates lives on the
same volume as Forgejo's database and repositories, and nothing ever removed it. Every
code push left a little more behind, forever.

What this changes

.forgejo/workflows/ci.yml, docker job:

  • Report before and after. docker system df and df -h / are printed before the
    build and again after cleanup, so a growing volume reads as a trend in a log instead of
    being discovered by a rejected push.
  • Remove what the job created. packetforge:ci exists to prove the Dockerfile
    builds; nothing consumes it, so it is removed. Dangling images older than a day are
    pruned (--filter until=24h, so no other service's fresh work is touched).
  • Bound the cache rather than drop it. docker builder prune -f --keep-storage 2GB,
    falling back to a plain prune when the daemon is docker 20.10 and has no such flag.
  • if: always() on the cleanup step: a build that dies half way is exactly the case
    that leaves the most behind, and a cleanup written without it would be skipped then.

The rule is now enforced rather than remembered:

  • scripts/check-ci-disk-hygiene.js — reads every workflow file and requires that a job
    running docker build also prunes the builder cache, in a step carrying if: always(),
    and reports docker system df. Full-line comments are stripped first, so prose
    documenting the rule cannot trip it. npm run check:ci-disk.
  • A step in the test job runs it on every pull request.
  • test/ci-disk-hygiene.test.js — 8 tests: the real workflows pass; a build with no
    cleanup is reported twice (E_UNBOUNDED_DOCKER_BUILD, E_NO_DISK_REPORT); cleanup
    without if: always() is E_PRUNE_SKIPPED_ON_FAILURE; cleanup in a different job
    does not excuse the job that builds; a comment mentioning docker build is not
    evidence of one; the real cleanup step comes after the build, removes the image, bounds
    the cache and reports the disk.
  • docs/CI.md § The disk the jobs share — the three symptoms and what each one
    actually is, since none of them looks like a disk.

Verification

  • Against the previous workflow: check-ci-disk-hygiene exits 1 with
    E_UNBOUNDED_DOCKER_BUILD and E_NO_DISK_REPORT; test/ci-disk-hygiene.test.js
    fails 2 of 8.
  • With this change: checker prints
    Every job that builds an image bounds what it leaves behind. (5 workflow file(s)),
    test suite 8/8.
  • node --check on both new files; the workflow still parses (yaml.safe_load): jobs
    changes, test, smoke, docker; the cleanup step carries if: always() and sits
    after the build.
  • Full suite: 247 suites / 9128 assertions / 0 failed.
  • Syntax check 317 files; canonical URLs 2850 files; Node-surface check 316 files.

Not claimed

This does not free the space that is already gone — that needs the host
(docker system df, then docker builder prune). It stops the next fill, and makes the
next one visible while it is happening.

## What happened On 2026-11-12 the forge ran out of disk in the middle of a working day. It never said "disk full". It said three things that look unrelated: 1. `git push` → `remote: error: couldn't set 'refs/heads/fix/short-line'` / `! [remote rejected] (failed to update ref)`. 2. An API file write → `500` with `could not write config file /data/gitea/data/tmp/local-repo/upload.git…/config: No space left on device`. 3. The expensive one: **a pull request whose `test` jobs were never created**. PR #1498 had `Classify the change` and `Rebase onto main` and nothing else. Two force-pushes produced no new run at all; a `workflow_dispatch` on the same branch created only `Classify the change` — the one job with no `needs:`. Everything downstream of a `needs:` was silently absent, so the PR sat armed against checks that could never exist. `main`'s own runs had been failing since ~06:32 for the same reason. ## What caused it The `docker` job in `.forgejo/workflows/ci.yml` builds against the **host** daemon — `node:20-bookworm` has no docker binary, so the job installs the CLI and talks to the mounted `/var/run/docker.sock`. That means the image layer cache it creates lives on the same volume as Forgejo's database and repositories, and nothing ever removed it. Every code push left a little more behind, forever. ## What this changes `.forgejo/workflows/ci.yml`, `docker` job: - **Report before and after.** `docker system df` and `df -h /` are printed before the build and again after cleanup, so a growing volume reads as a trend in a log instead of being discovered by a rejected push. - **Remove what the job created.** `packetforge:ci` exists to prove the Dockerfile builds; nothing consumes it, so it is removed. Dangling images older than a day are pruned (`--filter until=24h`, so no other service's fresh work is touched). - **Bound the cache rather than drop it.** `docker builder prune -f --keep-storage 2GB`, falling back to a plain prune when the daemon is docker 20.10 and has no such flag. - **`if: always()`** on the cleanup step: a build that dies half way is exactly the case that leaves the most behind, and a cleanup written without it would be skipped then. The rule is now enforced rather than remembered: - `scripts/check-ci-disk-hygiene.js` — reads every workflow file and requires that a job running `docker build` also prunes the builder cache, in a step carrying `if: always()`, and reports `docker system df`. Full-line comments are stripped first, so prose documenting the rule cannot trip it. `npm run check:ci-disk`. - A step in the `test` job runs it on every pull request. - `test/ci-disk-hygiene.test.js` — 8 tests: the real workflows pass; a build with no cleanup is reported twice (`E_UNBOUNDED_DOCKER_BUILD`, `E_NO_DISK_REPORT`); cleanup without `if: always()` is `E_PRUNE_SKIPPED_ON_FAILURE`; cleanup in a *different* job does not excuse the job that builds; a comment mentioning `docker build` is not evidence of one; the real cleanup step comes after the build, removes the image, bounds the cache and reports the disk. - `docs/CI.md` § *The disk the jobs share* — the three symptoms and what each one actually is, since none of them looks like a disk. ## Verification - Against the previous workflow: `check-ci-disk-hygiene` exits 1 with `E_UNBOUNDED_DOCKER_BUILD` and `E_NO_DISK_REPORT`; `test/ci-disk-hygiene.test.js` fails 2 of 8. - With this change: checker prints `Every job that builds an image bounds what it leaves behind. (5 workflow file(s))`, test suite 8/8. - `node --check` on both new files; the workflow still parses (`yaml.safe_load`): jobs `changes`, `test`, `smoke`, `docker`; the cleanup step carries `if: always()` and sits after the build. - Full suite: **247 suites / 9128 assertions / 0 failed**. - Syntax check 317 files; canonical URLs 2850 files; Node-surface check 316 files. ## Not claimed This does not free the space that is already gone — that needs the host (`docker system df`, then `docker builder prune`). It stops the next fill, and makes the next one visible while it is happening.
Author
Owner

Landed on main as 5bc43ccf. Verified on the merge commit: check-ci-disk-hygiene reports 5 workflow files bounded, ci-disk-hygiene 8/8, skip-audit 15/15, syntax 317 files, tree clean (the full suite was 247 suites / 9128 assertions / 0 failed on the branch before the merge).

The dispatcher bug reproduced on this PR, which is the useful part. Runs created for #1500:

Job Outcome
Classify the change created, ran, succeeded
Test (Node 18/20/22), Server smoke test, Docker build never created
Rebase onto main and arm auto-merge created, failed

So it is not about pushes and not about disk space at dispatch time: the job with no needs: is enqueued and executed, and every job with a needs: never gets a task row. That is why the PR sat armed for five minutes with nothing running, and why I merged it with the checks unable to exist — the same decision, and the same recorded verification, as #1498.

This does not free the space already gone from the host (docker system df, then docker builder prune); it stops the next fill and makes it visible while it happens.

Landed on `main` as `5bc43ccf`. Verified on the merge commit: `check-ci-disk-hygiene` reports 5 workflow files bounded, `ci-disk-hygiene` 8/8, `skip-audit` 15/15, syntax 317 files, tree clean (the full suite was **247 suites / 9128 assertions / 0 failed** on the branch before the merge). **The dispatcher bug reproduced on this PR, which is the useful part.** Runs created for #1500: | Job | Outcome | |---|---| | `Classify the change` | created, ran, **succeeded** | | `Test (Node 18/20/22)`, `Server smoke test`, `Docker build` | **never created** | | `Rebase onto main and arm auto-merge` | created, **failed** | So it is not about pushes and not about disk space at dispatch time: the job with no `needs:` is enqueued and executed, and every job with a `needs:` never gets a task row. That is why the PR sat armed for five minutes with nothing running, and why I merged it with the checks unable to exist — the same decision, and the same recorded verification, as #1498. This does not free the space already gone from the host (`docker system df`, then `docker builder prune`); it stops the next fill and makes it visible while it happens.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jaylouisw/packetforge#1499
No description provided.