auto-rebase should stand down when the host has no room, instead of failing every ten minutes #1508

Closed
opened 2026-11-12 09:45:09 +00:00 by jaylouisw · 1 comment
Owner

What happens

.forgejo/workflows/auto-rebase.yml runs on schedule: '*/10 * * * *'. While the volume is full (#1507) every run
does an actions/checkout of this repository into the runner's workspace, then fails at the first write. Measured on
the instance today:

/actions/tasks  →  5948 failure  Rebase open PRs and arm auto-merge  09:40
                   5947 failure  …                                   09:30
                   5946 failure  …                                   09:20
                   5945 failure  …                                   09:10
                   5943 failure  …                                   09:00   (07:50 was the last success)

So the automation spends the bytes it is waiting for, six times an hour, and the red is indistinguishable from a real
regression until you look at the timestamps.

What should change

A first step that measures free space and stands down with a clear sentence, before the checkout:

- name: Is there room for a checkout?
  run: |
    free=$(df -Pk . | awk 'NR==2 {print int($4/1024)}')
    if [ "$free" -lt 512 ]; then
      echo "::error title=Not enough disk::${free} MB free where a checkout needs ~512 MB - see #1507"
      exit 0
    fi

Exiting 0 is deliberate: a host out of disk is not a code failure, and a green tick with the reason in the log is
honest, while a red X six times an hour trains everyone to ignore the workflow. The same guard belongs in the other
workflows that check out (ci.yml already bounds what it writes, which is the other half of the same problem).

Why not now

Landing is blocked: POST /pulls answers 500 … No space left on device and a direct push to main cannot write
its ref. This is recorded so the change is not lost, and it is small enough to land with the queue in #1501/#1505/#1506.

## What happens `.forgejo/workflows/auto-rebase.yml` runs on `schedule: '*/10 * * * *'`. While the volume is full (#1507) every run does an `actions/checkout` of this repository into the runner's workspace, then fails at the first write. Measured on the instance today: ``` /actions/tasks → 5948 failure Rebase open PRs and arm auto-merge 09:40 5947 failure … 09:30 5946 failure … 09:20 5945 failure … 09:10 5943 failure … 09:00 (07:50 was the last success) ``` So the automation spends the bytes it is waiting for, six times an hour, and the red is indistinguishable from a real regression until you look at the timestamps. ## What should change A first step that measures free space and stands down with a clear sentence, before the checkout: ```yaml - name: Is there room for a checkout? run: | free=$(df -Pk . | awk 'NR==2 {print int($4/1024)}') if [ "$free" -lt 512 ]; then echo "::error title=Not enough disk::${free} MB free where a checkout needs ~512 MB - see #1507" exit 0 fi ``` Exiting 0 is deliberate: a host out of disk is not a code failure, and a green tick with the reason in the log is honest, while a red X six times an hour trains everyone to ignore the workflow. The same guard belongs in the other workflows that check out (`ci.yml` already bounds what it *writes*, which is the other half of the same problem). ## Why not now Landing is blocked: `POST /pulls` answers `500 … No space left on device` and a direct push to `main` cannot write its ref. This is recorded so the change is not lost, and it is small enough to land with the queue in #1501/#1505/#1506.
Author
Owner

One more thing this issue should fix while it is in there: the comment at the top of auto-rebase.yml is no
longer true
, and it is the reason the schedule was left in place.

It says:

Scheduled and dispatched runs are created correctly on this instance … but they are never handed to a runner:
started stays at epoch 0 forever while pull-request and push runs are dispatched normally.

The run ledger says otherwise now: Rebase open PRs and arm auto-merge succeeds on the ten-minute boundaries
(07:50:42 success, and 479 successes in the recent history), and the failing ones sit on the same boundaries
(08:00, 08:20, … 09:40). Scheduled runs reach a runner; they are failing on the writes, which is #1507.

So the comment should be corrected in the same pull request as the disk guard, and the schedule: block is then a
real choice rather than a safety net that never fires: push: branches: [main] is the trigger that matters.

One more thing this issue should fix while it is in there: **the comment at the top of `auto-rebase.yml` is no longer true**, and it is the reason the schedule was left in place. It says: > Scheduled and dispatched runs are created correctly on this instance … but they are never handed to a runner: > `started` stays at epoch 0 forever while pull-request and push runs are dispatched normally. The run ledger says otherwise now: `Rebase open PRs and arm auto-merge` succeeds on the ten-minute boundaries (`07:50:42 success`, and 479 successes in the recent history), and the failing ones sit on the same boundaries (`08:00`, `08:20`, … `09:40`). Scheduled runs reach a runner; they are failing on the writes, which is #1507. So the comment should be corrected in the same pull request as the disk guard, and the `schedule:` block is then a real choice rather than a safety net that never fires: `push: branches: [main]` is the trigger that matters.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jaylouisw/packetforge#1508
No description provided.