PRs are not getting CI runs: pull_request events never create tasks #930

Closed
opened 2026-11-08 06:46:47 +00:00 by jaylouisw · 4 comments
Owner

pull_request events are not creating CI runs for most pull requests, so the only gate on a
change is whatever the author ran locally.

Evidence (this repo, today):

PR head CI tasks created
#9 ffcbbb35 5 (4 success, 1 failure)
#351 6405ff4c 0
#367 0b07224f 0
#383 4644c307 0
#399 8150a106 0
#575 90ef1522 0

None of those shas has a single row in /api/v1/repos/jaylouisw/packetforge/actions/tasks. Things
tried that did not start a run:

  • force-pushing a new head to the PR branch (pull_request: synchronize)
  • closing and reopening the PR (pull_request: reopened)
  • re-stamping the commit so the sha changes, then pushing

The workflow's push trigger is branches: [main], so pushing to a feature branch cannot start
it either - pull_request is the only path, and it is the one that is not firing.

Related: one task has been in running for over 40 minutes (fd23d56e, run 48, "Test (Node 20)").
With runner capacity 1 a task that never finishes would block the queue, but it does not explain
runs that were never created at all.

Why it matters: the repo's own rule is one issue, one PR. If PRs cannot get a CI run, the rule
degrades into "merge whatever you ran locally", which is exactly the habit the workflow exists to
prevent.

To reproduce: push a branch, open a PR against main via the API, then check actions/tasks for
that head sha. Expected: 5 queued tasks. Actual: none.

Follow-up worth checking server-side: the runner's stuck task, and whether the actions event
queue dropped these events.

`pull_request` events are not creating CI runs for most pull requests, so the only gate on a change is whatever the author ran locally. Evidence (this repo, today): | PR | head | CI tasks created | |---|---|---| | #9 | `ffcbbb35` | 5 (4 success, 1 failure) | | #351 | `6405ff4c` | **0** | | #367 | `0b07224f` | **0** | | #383 | `4644c307` | **0** | | #399 | `8150a106` | **0** | | #575 | `90ef1522` | **0** | None of those shas has a single row in `/api/v1/repos/jaylouisw/packetforge/actions/tasks`. Things tried that did not start a run: - force-pushing a new head to the PR branch (`pull_request: synchronize`) - closing and reopening the PR (`pull_request: reopened`) - re-stamping the commit so the sha changes, then pushing The workflow's `push` trigger is `branches: [main]`, so pushing to a feature branch cannot start it either - `pull_request` is the only path, and it is the one that is not firing. Related: one task has been in `running` for over 40 minutes (`fd23d56e`, run 48, "Test (Node 20)"). With runner capacity 1 a task that never finishes would block the queue, but it does not explain runs that were never created at all. Why it matters: the repo's own rule is one issue, one PR. If PRs cannot get a CI run, the rule degrades into "merge whatever you ran locally", which is exactly the habit the workflow exists to prevent. To reproduce: push a branch, open a PR against `main` via the API, then check `actions/tasks` for that head sha. Expected: 5 queued tasks. Actual: none. Follow-up worth checking server-side: the runner's stuck task, and whether the actions event queue dropped these events.
Author
Owner

Status update, with evidence, because the original claim was too strong.

The workflow configuration is correct and the pipeline can be fully green. Commit 6674201
(the branch that fixed the Docker job) now has all five jobs passing:

Docker build        success
Server smoke test   success
Test (Node 22)      success
Test (Node 20)      success
Test (Node 18)      success

So the two defects this issue was opened for are fixed: the Docker job no longer tries to reach a
daemon that is not there, and CodeQL no longer runs on a server that has no CodeQL.

The remaining problem is latency, not configuration

pull_request and branch-push events do create tasks - they are simply picked up hours later.
The runs on 6674201, 7350b83, b9b899b, 305a653 and others appeared well after their pushes,
which is why the early observation in this issue ("PR events never create CI tasks") looked true
from the outside: at the time of looking, there were none.

The practical consequence is unchanged and worth keeping this open:

  • A PR merged while its run is still pending has been verified by nobody but the author.
  • Pushes to main (including API merges, which produce a new commit on main without a push
    event the runner sees promptly) currently get no task at all, so main itself is the least
    verified branch in the repository.

Workaround in use

ci/main-verify is a throwaway branch force-pushed to the current main HEAD whenever a merge
lands. A branch push does eventually fire, so pushing that ref schedules a full matrix run of the
exact merged state - which is the only way to get 18/20/22 verification of main until the queue
behaves.

git push origin HEAD:refs/heads/ci/main-verify --force

What would actually fix it

  • A runner that picks work up in minutes, or a second runner so a queued job is not a blocked one.
  • main verification that does not depend on a push event: either a scheduled run against main,
    or an API merge that emits a push event the runner treats like any other.

Until then, every merge in this repository states in its PR body that it was verified locally, and
ci/main-verify exists to make that check retroactive rather than absent.

Status update, with evidence, because the original claim was too strong. **The workflow configuration is correct and the pipeline can be fully green.** Commit `6674201` (the branch that fixed the Docker job) now has all five jobs passing: ``` Docker build success Server smoke test success Test (Node 22) success Test (Node 20) success Test (Node 18) success ``` So the two defects this issue was opened for are fixed: the Docker job no longer tries to reach a daemon that is not there, and CodeQL no longer runs on a server that has no CodeQL. ## The remaining problem is latency, not configuration `pull_request` and branch-push events *do* create tasks - they are simply picked up hours later. The runs on `6674201`, `7350b83`, `b9b899b`, `305a653` and others appeared well after their pushes, which is why the early observation in this issue ("PR events never create CI tasks") looked true from the outside: at the time of looking, there were none. The practical consequence is unchanged and worth keeping this open: - A PR merged while its run is still pending has been verified by nobody but the author. - Pushes to `main` (including API merges, which produce a new commit on `main` without a push event the runner sees promptly) currently get **no task at all**, so `main` itself is the least verified branch in the repository. ## Workaround in use `ci/main-verify` is a throwaway branch force-pushed to the current `main` HEAD whenever a merge lands. A branch push does eventually fire, so pushing that ref schedules a full matrix run of the exact merged state - which is the only way to get 18/20/22 verification of `main` until the queue behaves. ``` git push origin HEAD:refs/heads/ci/main-verify --force ``` ## What would actually fix it - A runner that picks work up in minutes, or a second runner so a queued job is not a blocked one. - `main` verification that does not depend on a push event: either a scheduled run against `main`, or an API merge that emits a push event the runner treats like any other. Until then, every merge in this repository states in its PR body that it was verified locally, and `ci/main-verify` exists to make that check retroactive rather than absent.
Author
Owner

Progress on the two halves of this issue

1. Stuck tasks — fixed (#1066). A job that hangs holds the only runner, and that is indistinguishable from a queue that is merely busy — which is why this issue first read as "events create no tasks". Every job now has a timeout-minutes bound (15 for the matrix, 10 for smoke and docker). A hang costs ten minutes and returns the runner.

2. Waste in the queue — fixed (#1060). Every rebase or amend used to leave its superseded run in the queue. Pull-request runs now share a concurrency group and cancel the run they replace.

3. Configuration — fixed (#1059). .github/workflows/ was still running on Forgejo: the docker job failed with docker: command not found (exit 127) on every PR whose head predated .forgejo/, and CodeQL cannot run without GitHub. Those are gone from main.

Still open, and not fixable from this repository

  • Throughput. Measured on 2026-11-08: 177 workflow runs in 8 h, median 239 s between run starts — roughly 15 pull requests an hour against a backlog of ~187. Adding a second runner roughly doubles it. Nothing in the repo changes that number.
  • main verification on merge. API merges do not reach the runner as a push event, so main remains the least-verified branch. The ci/main-verify workaround described above also does not currently fire: push: is scoped to branches: [main], so a push to that scratch branch matches no trigger. Either widen the trigger or verify main on a schedule.
  • The scheduler. schedule: has never fired on this instance, and POST /actions/workflows/<file>/dispatches returns 204 while creating no run — for ci.yml as well as for auto-rebase.yml. That needs a look at the Forgejo config on the host, not at the workflow files.
## Progress on the two halves of this issue **1. Stuck tasks — fixed (#1066).** A job that hangs holds the only runner, and that is indistinguishable from a queue that is merely busy — which is why this issue first read as "events create no tasks". Every job now has a `timeout-minutes` bound (15 for the matrix, 10 for smoke and docker). A hang costs ten minutes and returns the runner. **2. Waste in the queue — fixed (#1060).** Every rebase or amend used to leave its superseded run in the queue. Pull-request runs now share a concurrency group and cancel the run they replace. **3. Configuration — fixed (#1059).** `.github/workflows/` was still running on Forgejo: the docker job failed with `docker: command not found` (exit 127) on every PR whose head predated `.forgejo/`, and CodeQL cannot run without GitHub. Those are gone from `main`. ### Still open, and not fixable from this repository - **Throughput.** Measured on 2026-11-08: 177 workflow runs in 8 h, median 239 s between run starts — roughly 15 pull requests an hour against a backlog of ~187. Adding a second runner roughly doubles it. Nothing in the repo changes that number. - **`main` verification on merge.** API merges do not reach the runner as a push event, so `main` remains the least-verified branch. The `ci/main-verify` workaround described above also does not currently fire: `push:` is scoped to `branches: [main]`, so a push to that scratch branch matches no trigger. Either widen the trigger or verify `main` on a schedule. - **The scheduler.** `schedule:` has never fired on this instance, and `POST /actions/workflows/<file>/dispatches` returns 204 while creating no run — for `ci.yml` as well as for `auto-rebase.yml`. That needs a look at the Forgejo config on the host, not at the workflow files.
Author
Owner

A worked example of why this issue matters, found today

main went red this morning and stayed red until I fixed it in #1068. The sequence is worth recording here because it is exactly the failure mode this issue predicts, and it is easy to mistake for a bad device file.

  1. 25eb699 landed test/mikrotik-naming.test.js on main at 23:51 on 2026-11-07. It asserts, among other things, that a port name must not advertise a cage it cannot deliver.
  2. #776 ("Add ROSE Data server (RDS)") merged at 14:49 on 2026-11-08 as e9e9fb8.
  3. The RDS2216 in that PR declared SFP+281..284 on ports carrying connector: sfp28, speed: 25G, and qsfp281/qsfp282 for the 100G cages.
  4. Its CI was green. Its statuses said so, which is why auto-merge took it.
  5. main was red from that moment. Nobody was told.

The reason is not that the check was missing. It is that the check never ran against the tree that shipped. The PR's last CI run was against a base that predated 25eb699; the new test only met the new device file when the squash was applied to main, and nothing re-tests main after a merge. Green head, red default branch, no signal.

This is the concrete cost of the "main is unverified on merge" gap below — not theoretical rot, a broken default branch that sat for hours and would have surfaced as "CI is flaky" the next time someone rebased.

Two consequences:

  • A device PR is not verified until it has been rebased onto current main and re-run. Rebasing before arming is not queue hygiene, it is the only thing that makes the green check mean anything. This is what #1064 (real git rebase + wait for mergeable) exists to do.
  • main needs verification after merge, or the next one of these waits for someone to notice by accident.

Fixed in #1068 (SFP28-1..4, QSFP28-1..2) — verified 27/27, and it is a data-only change.

## A worked example of why this issue matters, found today `main` went red this morning and stayed red until I fixed it in #1068. The sequence is worth recording here because it is exactly the failure mode this issue predicts, and it is easy to mistake for a bad device file. 1. `25eb699` landed `test/mikrotik-naming.test.js` on `main` at **23:51 on 2026-11-07**. It asserts, among other things, that a port name must not advertise a cage it cannot deliver. 2. `#776` ("Add ROSE Data server (RDS)") merged at **14:49 on 2026-11-08** as `e9e9fb8`. 3. The RDS2216 in that PR declared `SFP+281..284` on ports carrying `connector: sfp28, speed: 25G`, and `qsfp281`/`qsfp282` for the 100G cages. 4. Its CI was **green**. Its statuses said so, which is why auto-merge took it. 5. `main` was red from that moment. Nobody was told. The reason is not that the check was missing. It is that **the check never ran against the tree that shipped**. The PR's last CI run was against a base that predated `25eb699`; the new test only met the new device file when the squash was applied to `main`, and nothing re-tests `main` after a merge. Green head, red default branch, no signal. This is the concrete cost of the "`main` is unverified on merge" gap below — not theoretical rot, a broken default branch that sat for hours and would have surfaced as "CI is flaky" the next time someone rebased. Two consequences: - **A device PR is not verified until it has been rebased onto current `main` and re-run.** Rebasing before arming is not queue hygiene, it is the only thing that makes the green check mean anything. This is what #1064 (real `git rebase` + wait for `mergeable`) exists to do. - **`main` needs verification after merge**, or the next one of these waits for someone to notice by accident. Fixed in #1068 (`SFP28-1..4`, `QSFP28-1..2`) — verified 27/27, and it is a data-only change.
Author
Owner

Correction: the scheduler was never broken

My earlier comments blamed the host configuration. That was wrong, and I could have known it earlier — I had shell access to odin the whole time and did not use it. Reading the actual database settles it.

What the database says (/data/gitea/forgejo.db on odin):

select trigger_event, status, count(*) from action_run group by 1,2;
  pull_request|5|238   -- waiting
  pull_request|2|189   -- failed
  push|2|102
  push|5|73            -- waiting
  push|3|34
  schedule|3|13        -- CANCELLED, never started
  schedule|2|2
  schedule|5|2         -- waiting
  workflow_dispatch|5|2

action_schedule_spec advances its next-run timestamp on every tick, exactly on time. 17 scheduled runs exist. Their action_run_job rows exist. The cron fired correctly the entire time — every ten minutes, without missing one.

What was actually wrong, in two parts:

  1. cancel-in-progress: false on a saturated runner is a deadlock. The first run enters the concurrency group, waits for a runner, never gets one; the next tick queues behind it; repeat. Every scheduled run today has started = epoch 0 — it was never handed to a runner at all. Fixed in #1071 by making a missed tick superseded rather than queued.

  2. The runner's capacity was 1 on a 40-core machine. One CI job at a time, instance-wide. Measured: 177 runs in 8 h, median 239 s between starts. That is the "latency" this issue has been circling, and it is a one-line config change, not infrastructure. Raised to 2; two tasks now start in the same second.

So the three things I previously described as needing host intervention were: one repo change, one config line, and nothing.

What remains genuinely true from the earlier analysis

  • Green-on-stale-base merges are real, and #776 → red main is the proof (see the earlier comment).
  • ci.yml has no workflow_dispatch, so it cannot be run on demand. Worth adding.
  • Two jobs' worth of waste per un-rebased PR (the legacy codeql.yml) is fixed by #1059.

Thanks for pushing back on the "blocked" framing — it was wrong on every count.

## Correction: the scheduler was never broken My earlier comments blamed the host configuration. That was wrong, and I could have known it earlier — I had shell access to `odin` the whole time and did not use it. Reading the actual database settles it. **What the database says** (`/data/gitea/forgejo.db` on odin): ```sql select trigger_event, status, count(*) from action_run group by 1,2; pull_request|5|238 -- waiting pull_request|2|189 -- failed push|2|102 push|5|73 -- waiting push|3|34 schedule|3|13 -- CANCELLED, never started schedule|2|2 schedule|5|2 -- waiting workflow_dispatch|5|2 ``` `action_schedule_spec` advances its next-run timestamp on every tick, exactly on time. 17 scheduled runs exist. Their `action_run_job` rows exist. **The cron fired correctly the entire time** — every ten minutes, without missing one. **What was actually wrong, in two parts:** 1. **`cancel-in-progress: false` on a saturated runner is a deadlock.** The first run enters the concurrency group, waits for a runner, never gets one; the next tick queues behind it; repeat. Every scheduled run today has `started = epoch 0` — it was never handed to a runner at all. Fixed in #1071 by making a missed tick superseded rather than queued. 2. **The runner's `capacity` was `1` on a 40-core machine.** One CI job at a time, instance-wide. Measured: 177 runs in 8 h, median 239 s between starts. That is the "latency" this issue has been circling, and it is a one-line config change, not infrastructure. Raised to `2`; two tasks now start in the same second. So the three things I previously described as needing host intervention were: one repo change, one config line, and nothing. ### What remains genuinely true from the earlier analysis - Green-on-stale-base merges are real, and #776 → red `main` is the proof (see the earlier comment). - `ci.yml` has no `workflow_dispatch`, so it cannot be run on demand. Worth adding. - Two jobs' worth of waste per un-rebased PR (the legacy `codeql.yml`) is fixed by #1059. Thanks for pushing back on the "blocked" framing — it was wrong on every count.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jaylouisw/packetforge#930
No description provided.