Stop the scheduled rebase deadlocking behind itself #1071
No reviewers
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
jaylouisw/packetforge!1071
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "fix/auto-rebase-deadlock"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Now that I can read the database on
odin, the diagnosis changes completely: the schedule was never broken.action_schedule_specfor this repo shows the*/10 * * * *cron advancing its next-run timestamp on every tick, exactly on time.action_runholds seventeen scheduled runs and two dispatched ones. The runs exist; so do theiraction_run_jobrows.What they never got was a runner.
concurrency: {group: auto-rebase, cancel-in-progress: false}deadlocks once the runner is saturated: the first run enters the group, waits, never starts; the next tick queues behind it; and so on. Today's evidence — 13 scheduled runs cancelled withstarted = epoch 0, 2 waiting for hours — while ordinary pull-request runs were being picked up normally. That asymmetry is exactly why this read as "scheduled workflows do not fire" in #930.For periodic maintenance the right semantics are that a missed tick is superseded, not queued.
cancel-in-progress: truekeeps the mutual exclusion the group exists for — two rebase passes still never overlap — and lets the next tick take the place of one that never started.Also fixed this session, on the host
capacitywas 1 on a 40-core machine: the whole instance ran one CI job at a time. Raised to 2 — two tasks now start in the same second, doubling how fast the 189-PR backlog drains and shortening the window a scheduled run has to survive in.Verification: the next ticks after this lands should start rather than be cancelled.