[Bug][CircleCI] collectJobs and collectWorkflows collect on_hold workflows again on every run, so the raw tables grow without a limit

Aperta
  • Qualcosa non va?
#9,304 0 commenti 1 reazione 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

@zahorniak ci sta già lavorando.

Dal 2/11/2026.

  • #9309 di @zahorniak — aperta

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

Search before asking
  • I had searched in the issues and found no similar issues.
What happened

The CircleCI plugin collects the same jobs and workflows again on every incremental run, without a limit. The raw tables _raw_circleci_api_jobs and _raw_circleci_api_workflows grow every day, and the circleci task gets slower every day.

Both collectors have a "collect unfinished details" step:

Subtask Input query Statuses
collectJobs UnfinishedJobsInputClauses (backend/plugins/circleci/tasks/job_collector.go) running, not_running, queued, on_hold
collectWorkflows CollectUnfinishedDetails.BuildInputIterator (backend/plugins/circleci/tasks/workflow_collector.go) running, on_hold, failing

These queries have no age limit. A CircleCI approval job (type: approval) that nobody approves stays on_hold for ever, and its workflow stays on_hold too. So DevLake collects all jobs of such a workflow again on every run, for ever. The set of these workflows only grows.

The step runs with Incremental: true (helpers/pluginhelper/api/api_collector_stateful.go), so the collector only adds rows to the raw table. The old rows for the same jobs stay.

Environment
DevLake version v1.0.3-beta14 (the code is the same on main at 351d86738)
Plugin circleci
Database MySQL 8.4
CircleCI CircleCI Cloud (circleci.com API v2)
Trigger Nightly blueprint (incremental, skipCollectors=false), 2 CircleCI projects

Data from our instance

  • 14,356 workflows are on_hold and 810 are failing. They were created from February to November 2026. The jobs that stay on_hold are approval jobs that nobody approved (for example hold-deploy, hold-tests, start-development).
  • One nightly run added 166,536 rows to _raw_circleci_api_jobs:
Source of the raw rows Rows Workflows
"unfinished details" step 166,508 14,866
new workflows 28 4
  • _raw_circleci_api_jobs has about 9.3M rows (10.3 GB) for about 248k jobs. It grows by about 160k rows per day, also on weekends.
  • extractJobs reads the full raw table on every run (see #9305), so the circleci task takes 3–8 hours per run.
What do you expect to happen

DevLake should collect an unfinished workflow again only for a limited time after it was created. The raw tables should grow only with new CI activity.

How to reproduce
  1. Use a CircleCI project with a workflow that has an approval job (type: approval).
  2. Trigger the workflow and do not approve the job.
  3. Add the project to a blueprint and run the blueprint (incremental) several times.
  4. Each run adds all jobs of that workflow to the raw table again:
SELECT url, COUNT(*) FROM _raw_circleci_api_jobs GROUP BY url ORDER BY 2 DESC LIMIT 10;
-- the same /v2/workflow/<id>/job URL appears once per run
Anything else

Proposed fix (plan of attack)

In UnfinishedJobsInputClauses and in the workflow collector's unfinished-details iterator, select only workflows created in a lookback window:

dal.Where("... AND created_date >= ?", time.Now().AddDate(0, 0, -lookbackDays))
  • Compute the date in Go, so the query works on MySQL and PostgreSQL.
  • Use a default of 30 days. It can be an option if maintainers prefer.
  • For the job query, join the workflow table (or use the job's workflow created_date) to apply the window.

Data that supports a 30-day window: in our instance, 99.6% of approved workflows got their approval within 14 days. All approvals except 1 of about 1,860 came within 30 days.

Rejected alternatives

  • Do not treat on_hold as a final status. A workflow that gets its approval later would then never be collected again, so its deploy jobs after the approval would be missing from DORA deployments.
  • Delete the previous raw rows of the same input before collecting it again. This needs a change in the framework collector and is more complex.

I can start work on a PR for this if the DevLake maintainers approve this approach.

Version

v1.0.3-beta14

Are you willing to submit PR?
  • Yes I am willing to submit a PR!
Code of Conduct
Lingua principale
Go
Stelle
3.2k
Fork
832
Merge medio
1g 22h
PR unite (30g)
75

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/devlake

Tutte le issue di apache/devlake

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.