Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

[Bug][GitHub] Closed (unmarked) Pull Requests are re-fetched on every run

Aperta
  • Qualcosa non va?
#9,240 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 3 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
64/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
github, go
Ambito
data-engineering

Direzione di ricerca

Start with plugins/github/tasks/pr_collector.go, where the issue's stack trace points to pull request collection, and trace how incremental collection selects closed, unmerged PRs for individual fetches. Compare the selection with the reported github_updated_at and the API's updated_at behavior. Done means unchanged closed PRs are skipped, while PRs that change or reopen are collected; the issue names no test file, so locate the relevant collector tests first.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

type/bug
Search before asking
  • I had searched in the issues and found no similar issues.
What happened

One of my repo failed last night pipeline on the Collect Pull Requests task on a PR that had been closed (not merged) for more than a month and had not been updated since. For the story GitHub returned a 422 on it.

That made me take a deeper look as there was no reason to re-fetch that PR.
it looks like every closed but not merged PR is being re-fetched on every incremental run. On my instance that's about ~15k closed PRs.

For example, one of the oldest PR in my DB was closed in August 2019, has not been updated since, however at every run:

SELECT number, state, merged, github_created_at, closed_at, github_updated_at
FROM _tool_github_pull_requests
WHERE state = 'closed' AND merged = 0 AND merged_at IS NULL
ORDER BY github_updated_at ASC LIMIT 1;

number,state,merged,github_created_at,closed_at,github_updated_at
29,closed,0,2019-08-02 05:39:12,2019-08-08 08:31:19,2019-08-08 08:31:19

SELECT id, created_at, url FROM _raw_github_api_pull_requests
WHERE url LIKE '%/[REPOSITORY]/pulls/29'
ORDER BY created_at DESC LIMIT 7;

id,created_at,url
5014408,2026-10-19 00:27:23,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4992046,2026-10-18 00:27:07,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4969613,2026-10-17 00:27:19,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4947192,2026-10-16 00:27:38,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4924806,2026-10-15 00:27:32,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4902464,2026-10-14 00:27:21,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4880192,2026-10-13 00:27:17,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29

183 identical fetches over the last 182 days of raw data we retain, for a PR untouched for 7 years.

Sanitized error:

subtask Collect Pull Requests ended unexpectedly
caused by: Retry exceeded 3 times calling repos/[ORGANIZATION]/[REPOSITORY]/pulls/29.
The last error was: Http DoAsync error calling [method:GET path:repos/[ORGANIZATION]/[REPOSITORY]/pulls/29 query:map[]].
Response: {"message":"The request could not be processed because too many files changed","documentation_url":"https://docs.github.com/rest/pulls/pulls#get-a-pull-request","status":"422"} (422)

attached stack trace
  -- stack trace:
  | github.com/apache/devlake/core/runner.RunPluginSubTasks
  |     /app/core/runner/run_task.go:341
  | [...repeated from below...]
Wraps: (2) subtask Collect Pull Requests ended unexpectedly
Wraps: (3) attached stack trace
  -- stack trace:
  | github.com/apache/devlake/helpers/pluginhelper/api.(*WorkerScheduler).WaitAsync
  |     /app/helpers/pluginhelper/api/worker_scheduler.go:173
  | github.com/apache/devlake/helpers/pluginhelper/api.(*ApiCollector).Execute
  |     /app/helpers/pluginhelper/api/api_collector.go:206
  | github.com/apache/devlake/helpers/pluginhelper/api.(*StatefulApiCollector).Execute
  |     /app/helpers/pluginhelper/api/api_collector_stateful.go:97
  | github.com/apache/devlake/plugins/github/tasks.CollectApiPullRequests
  |     /app/plugins/github/tasks/pr_collector.go:141
  | github.com/apache/devlake/core/runner.runSubtask
  |     /app/core/runner/run_task.go:433
  | github.com/apache/devlake/core/runner.RunPluginSubTasks
  |     /app/core/runner/run_task.go:338
  | github.com/apache/devlake/core/runner.RunPluginTask
  |     /app/core/runner/run_task.go:173
  | github.com/apache/devlake/core/runner.RunTask
  |     /app/core/runner/run_task.go:147
  | github.com/apache/devlake/server/services.runTaskStandalone
  |     /app/server/services/task_runner.go:114
  | github.com/apache/devlake/server/services.RunTasksStandalone.func1
  |     /app/server/services/task.go:187
  | runtime.goexit
  |     /usr/local/go/src/runtime/asm_amd64.s:1771
Wraps: (4) Retry exceeded 3 times calling repos/[ORGANIZATION]/[REPOSITORY]/pulls/29. ...
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError

### What do you expect to happen

A PR should only be re-fetched individually when it has actually changed since it was last collected, else in a few years time will be fetching way more closed than opened PR.

Reopening, pushing, commenting, reviewing, merging or closing a PR all bump `updated_at`, so if a closed PR gets re-opened Devlake should pick it up.


### How to reproduce

1. Collect a repository that has closed-unmerged PRs.
2. Run incremental collection again.
3. Observe one `GET /repos/{org}/{repo}/pulls/{number}` per open **and** per closed-unmerged
   PR in the database, on every run, including PRs untouched for years:
   ```sql
   SELECT COUNT(*) FROM _tool_github_pull_requests
   WHERE state != 'closed' OR merged = 0;
Anything else

Happens at every collection, and I assume collection time will increase the more unmerged closed PR there are.
Also DB size will become bloated with duplicated data seeing that each non-updated run stores the same payload in _raw_github_api_pull_requests.

Hope I didn't make obvious mistakes in that analysis.

Version

v1.0.3-beta18

Are you willing to submit PR?
  • Yes I am willing to submit a PR!
Code of Conduct
Lingua principale
Go
Stelle
3.2k
Fork
824
Merge medio
2g 22h
PR unite (30g)
46

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/devlake

Tutte le issue di apache/devlake

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.