Stop the scheduled rebase deadlocking behind itself #1071

Merged
jaylouisw merged 1 commit from fix/auto-rebase-deadlock into main 2026-11-08 15:25:18 +00:00
Owner

Now that I can read the database on odin, the diagnosis changes completely: the schedule was never broken.

action_schedule_spec for this repo shows the */10 * * * * cron advancing its next-run timestamp on every tick, exactly on time. action_run holds seventeen scheduled runs and two dispatched ones. The runs exist; so do their action_run_job rows.

What they never got was a runner. concurrency: {group: auto-rebase, cancel-in-progress: false} deadlocks once the runner is saturated: the first run enters the group, waits, never starts; the next tick queues behind it; and so on. Today's evidence — 13 scheduled runs cancelled with started = epoch 0, 2 waiting for hours — while ordinary pull-request runs were being picked up normally. That asymmetry is exactly why this read as "scheduled workflows do not fire" in #930.

For periodic maintenance the right semantics are that a missed tick is superseded, not queued. cancel-in-progress: true keeps the mutual exclusion the group exists for — two rebase passes still never overlap — and lets the next tick take the place of one that never started.

Also fixed this session, on the host

  • The runner's capacity was 1 on a 40-core machine: the whole instance ran one CI job at a time. Raised to 2 — two tasks now start in the same second, doubling how fast the 189-PR backlog drains and shortening the window a scheduled run has to survive in.
  • Stuck runs from earlier ticks cancelled to clear the group.

Verification: the next ticks after this lands should start rather than be cancelled.

Now that I can read the database on `odin`, the diagnosis changes completely: **the schedule was never broken.** `action_schedule_spec` for this repo shows the `*/10 * * * *` cron advancing its next-run timestamp on every tick, exactly on time. `action_run` holds seventeen scheduled runs and two dispatched ones. The runs exist; so do their `action_run_job` rows. What they never got was a runner. `concurrency: {group: auto-rebase, cancel-in-progress: false}` deadlocks once the runner is saturated: the first run enters the group, waits, never starts; the next tick queues behind it; and so on. Today's evidence — **13 scheduled runs cancelled with `started = epoch 0`**, 2 waiting for hours — while ordinary pull-request runs were being picked up normally. That asymmetry is exactly why this read as "scheduled workflows do not fire" in #930. For periodic maintenance the right semantics are that a missed tick is superseded, not queued. `cancel-in-progress: true` keeps the mutual exclusion the group exists for — two rebase passes still never overlap — and lets the next tick take the place of one that never started. ### Also fixed this session, on the host - The runner's `capacity` was **1** on a 40-core machine: the whole instance ran one CI job at a time. Raised to 2 — two tasks now start in the same second, doubling how fast the 189-PR backlog drains and shortening the window a scheduled run has to survive in. - Stuck runs from earlier ticks cancelled to clear the group. Verification: the next ticks after this lands should start rather than be cancelled.
Stop the scheduled rebase deadlocking behind itself
Some checks failed
CI / Test (Node 18) (pull_request) Failing after 1m34s
CI / Test (Node 20) (pull_request) Failing after 1m27s
CI / Server smoke test (pull_request) Successful in 30s
CI / Docker build (pull_request) Successful in 11s
CI / Test (Node 22) (pull_request) Failing after 1m25s
f2e34fb94d
The schedule was never broken. The cron fired every ten minutes, exactly on time,
and the database proves it: action_schedule_spec advances its next-run timestamp on
every tick.

What was broken is what happened next. auto-rebase had concurrency group
'auto-rebase' with cancel-in-progress: false. A run that entered the group and did
not get a runner stayed waiting; the next tick queued behind it; and so on. The run
table today:

  thirteen scheduled runs cancelled without ever starting (started = epoch 0)
  two still waiting from hours earlier
  correct-looking cron advancement the whole time

Meanwhile ordinary pull-request runs were being picked up normally, which is why
this looked like 'scheduled workflows do not fire' rather than 'scheduled runs
never get to start'.

For a periodic maintenance job the correct semantics are that a missed tick is
superseded, not queued. cancel-in-progress: true keeps the mutual exclusion the
group exists for - two rebase passes still never overlap - while letting the next
tick take the place of one that never started.
jaylouisw scheduled this pull request to auto merge when all checks succeed 2026-11-08 15:11:02 +00:00
jaylouisw scheduled this pull request to auto merge when all checks succeed 2026-11-08 15:19:42 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jaylouisw/packetforge!1071
No description provided.