PRs are not getting CI runs: pull_request events never create tasks #930
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
jaylouisw/packetforge#930
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
pull_requestevents are not creating CI runs for most pull requests, so the only gate on achange is whatever the author ran locally.
Evidence (this repo, today):
ffcbbb356405ff4c0b07224f4644c3078150a10690ef1522None of those shas has a single row in
/api/v1/repos/jaylouisw/packetforge/actions/tasks. Thingstried that did not start a run:
pull_request: synchronize)pull_request: reopened)The workflow's
pushtrigger isbranches: [main], so pushing to a feature branch cannot startit either -
pull_requestis the only path, and it is the one that is not firing.Related: one task has been in
runningfor over 40 minutes (fd23d56e, run 48, "Test (Node 20)").With runner capacity 1 a task that never finishes would block the queue, but it does not explain
runs that were never created at all.
Why it matters: the repo's own rule is one issue, one PR. If PRs cannot get a CI run, the rule
degrades into "merge whatever you ran locally", which is exactly the habit the workflow exists to
prevent.
To reproduce: push a branch, open a PR against
mainvia the API, then checkactions/tasksforthat head sha. Expected: 5 queued tasks. Actual: none.
Follow-up worth checking server-side: the runner's stuck task, and whether the actions event
queue dropped these events.
Status update, with evidence, because the original claim was too strong.
The workflow configuration is correct and the pipeline can be fully green. Commit
6674201(the branch that fixed the Docker job) now has all five jobs passing:
So the two defects this issue was opened for are fixed: the Docker job no longer tries to reach a
daemon that is not there, and CodeQL no longer runs on a server that has no CodeQL.
The remaining problem is latency, not configuration
pull_requestand branch-push events do create tasks - they are simply picked up hours later.The runs on
6674201,7350b83,b9b899b,305a653and others appeared well after their pushes,which is why the early observation in this issue ("PR events never create CI tasks") looked true
from the outside: at the time of looking, there were none.
The practical consequence is unchanged and worth keeping this open:
main(including API merges, which produce a new commit onmainwithout a pushevent the runner sees promptly) currently get no task at all, so
mainitself is the leastverified branch in the repository.
Workaround in use
ci/main-verifyis a throwaway branch force-pushed to the currentmainHEAD whenever a mergelands. A branch push does eventually fire, so pushing that ref schedules a full matrix run of the
exact merged state - which is the only way to get 18/20/22 verification of
mainuntil the queuebehaves.
What would actually fix it
mainverification that does not depend on a push event: either a scheduled run againstmain,or an API merge that emits a push event the runner treats like any other.
Until then, every merge in this repository states in its PR body that it was verified locally, and
ci/main-verifyexists to make that check retroactive rather than absent.Progress on the two halves of this issue
1. Stuck tasks — fixed (#1066). A job that hangs holds the only runner, and that is indistinguishable from a queue that is merely busy — which is why this issue first read as "events create no tasks". Every job now has a
timeout-minutesbound (15 for the matrix, 10 for smoke and docker). A hang costs ten minutes and returns the runner.2. Waste in the queue — fixed (#1060). Every rebase or amend used to leave its superseded run in the queue. Pull-request runs now share a concurrency group and cancel the run they replace.
3. Configuration — fixed (#1059).
.github/workflows/was still running on Forgejo: the docker job failed withdocker: command not found(exit 127) on every PR whose head predated.forgejo/, and CodeQL cannot run without GitHub. Those are gone frommain.Still open, and not fixable from this repository
mainverification on merge. API merges do not reach the runner as a push event, somainremains the least-verified branch. Theci/main-verifyworkaround described above also does not currently fire:push:is scoped tobranches: [main], so a push to that scratch branch matches no trigger. Either widen the trigger or verifymainon a schedule.schedule:has never fired on this instance, andPOST /actions/workflows/<file>/dispatchesreturns 204 while creating no run — forci.ymlas well as forauto-rebase.yml. That needs a look at the Forgejo config on the host, not at the workflow files.A worked example of why this issue matters, found today
mainwent red this morning and stayed red until I fixed it in #1068. The sequence is worth recording here because it is exactly the failure mode this issue predicts, and it is easy to mistake for a bad device file.25eb699landedtest/mikrotik-naming.test.jsonmainat 23:51 on 2026-11-07. It asserts, among other things, that a port name must not advertise a cage it cannot deliver.#776("Add ROSE Data server (RDS)") merged at 14:49 on 2026-11-08 ase9e9fb8.SFP+281..284on ports carryingconnector: sfp28, speed: 25G, andqsfp281/qsfp282for the 100G cages.mainwas red from that moment. Nobody was told.The reason is not that the check was missing. It is that the check never ran against the tree that shipped. The PR's last CI run was against a base that predated
25eb699; the new test only met the new device file when the squash was applied tomain, and nothing re-testsmainafter a merge. Green head, red default branch, no signal.This is the concrete cost of the "
mainis unverified on merge" gap below — not theoretical rot, a broken default branch that sat for hours and would have surfaced as "CI is flaky" the next time someone rebased.Two consequences:
mainand re-run. Rebasing before arming is not queue hygiene, it is the only thing that makes the green check mean anything. This is what #1064 (realgit rebase+ wait formergeable) exists to do.mainneeds verification after merge, or the next one of these waits for someone to notice by accident.Fixed in #1068 (
SFP28-1..4,QSFP28-1..2) — verified 27/27, and it is a data-only change.Correction: the scheduler was never broken
My earlier comments blamed the host configuration. That was wrong, and I could have known it earlier — I had shell access to
odinthe whole time and did not use it. Reading the actual database settles it.What the database says (
/data/gitea/forgejo.dbon odin):action_schedule_specadvances its next-run timestamp on every tick, exactly on time. 17 scheduled runs exist. Theiraction_run_jobrows exist. The cron fired correctly the entire time — every ten minutes, without missing one.What was actually wrong, in two parts:
cancel-in-progress: falseon a saturated runner is a deadlock. The first run enters the concurrency group, waits for a runner, never gets one; the next tick queues behind it; repeat. Every scheduled run today hasstarted = epoch 0— it was never handed to a runner at all. Fixed in #1071 by making a missed tick superseded rather than queued.The runner's
capacitywas1on a 40-core machine. One CI job at a time, instance-wide. Measured: 177 runs in 8 h, median 239 s between starts. That is the "latency" this issue has been circling, and it is a one-line config change, not infrastructure. Raised to2; two tasks now start in the same second.So the three things I previously described as needing host intervention were: one repo change, one config line, and nothing.
What remains genuinely true from the earlier analysis
mainis the proof (see the earlier comment).ci.ymlhas noworkflow_dispatch, so it cannot be run on demand. Worth adding.codeql.yml) is fixed by #1059.Thanks for pushing back on the "blocked" framing — it was wrong on every count.