Hacktoberfest 2026 : les issues que les mainteneurs ont marquées pour octobre, ouvertes et accessibles aux débutants. Parcourir les issues Hacktoberfest

[Bug][GitHub] Closed (unmarked) Pull Requests are re-fetched on every run

Ouverte
  • Quelque chose ne va pas ?
#9,240 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 3 jours

@dvecy y travaille déjà.

Depuis le 20/10/2026.

  • #9244 par @dvecy — ouverte

Évaluation

Difficulté
3/5
Temps estimé
1-2 jours
Accessibilité débutants
64/100
Type d'issue
Bug
Clarté
Plutôt claire
Activité
Active
Stack technique
github, go
Domaine
data-engineering

Piste de recherche

Start with plugins/github/tasks/pr_collector.go, where the issue's stack trace points to pull request collection, and trace how incremental collection selects closed, unmerged PRs for individual fetches. Compare the selection with the reported github_updated_at and the API's updated_at behavior. Done means unchanged closed PRs are skipped, while PRs that change or reopen are collected; the issue names no test file, so locate the relevant collector tests first.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

type/bug
Search before asking
  • I had searched in the issues and found no similar issues.
What happened

One of my repo failed last night pipeline on the Collect Pull Requests task on a PR that had been closed (not merged) for more than a month and had not been updated since. For the story GitHub returned a 422 on it.

That made me take a deeper look as there was no reason to re-fetch that PR.
it looks like every closed but not merged PR is being re-fetched on every incremental run. On my instance that's about ~15k closed PRs.

For example, one of the oldest PR in my DB was closed in August 2019, has not been updated since, however at every run:

SELECT number, state, merged, github_created_at, closed_at, github_updated_at
FROM _tool_github_pull_requests
WHERE state = 'closed' AND merged = 0 AND merged_at IS NULL
ORDER BY github_updated_at ASC LIMIT 1;

number,state,merged,github_created_at,closed_at,github_updated_at
29,closed,0,2019-08-02 05:39:12,2019-08-08 08:31:19,2019-08-08 08:31:19

SELECT id, created_at, url FROM _raw_github_api_pull_requests
WHERE url LIKE '%/[REPOSITORY]/pulls/29'
ORDER BY created_at DESC LIMIT 7;

id,created_at,url
5014408,2026-10-19 00:27:23,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4992046,2026-10-18 00:27:07,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4969613,2026-10-17 00:27:19,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4947192,2026-10-16 00:27:38,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4924806,2026-10-15 00:27:32,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4902464,2026-10-14 00:27:21,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4880192,2026-10-13 00:27:17,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29

183 identical fetches over the last 182 days of raw data we retain, for a PR untouched for 7 years.

Sanitized error:

subtask Collect Pull Requests ended unexpectedly
caused by: Retry exceeded 3 times calling repos/[ORGANIZATION]/[REPOSITORY]/pulls/29.
The last error was: Http DoAsync error calling [method:GET path:repos/[ORGANIZATION]/[REPOSITORY]/pulls/29 query:map[]].
Response: {"message":"The request could not be processed because too many files changed","documentation_url":"https://docs.github.com/rest/pulls/pulls#get-a-pull-request","status":"422"} (422)

attached stack trace
  -- stack trace:
  | github.com/apache/devlake/core/runner.RunPluginSubTasks
  |     /app/core/runner/run_task.go:341
  | [...repeated from below...]
Wraps: (2) subtask Collect Pull Requests ended unexpectedly
Wraps: (3) attached stack trace
  -- stack trace:
  | github.com/apache/devlake/helpers/pluginhelper/api.(*WorkerScheduler).WaitAsync
  |     /app/helpers/pluginhelper/api/worker_scheduler.go:173
  | github.com/apache/devlake/helpers/pluginhelper/api.(*ApiCollector).Execute
  |     /app/helpers/pluginhelper/api/api_collector.go:206
  | github.com/apache/devlake/helpers/pluginhelper/api.(*StatefulApiCollector).Execute
  |     /app/helpers/pluginhelper/api/api_collector_stateful.go:97
  | github.com/apache/devlake/plugins/github/tasks.CollectApiPullRequests
  |     /app/plugins/github/tasks/pr_collector.go:141
  | github.com/apache/devlake/core/runner.runSubtask
  |     /app/core/runner/run_task.go:433
  | github.com/apache/devlake/core/runner.RunPluginSubTasks
  |     /app/core/runner/run_task.go:338
  | github.com/apache/devlake/core/runner.RunPluginTask
  |     /app/core/runner/run_task.go:173
  | github.com/apache/devlake/core/runner.RunTask
  |     /app/core/runner/run_task.go:147
  | github.com/apache/devlake/server/services.runTaskStandalone
  |     /app/server/services/task_runner.go:114
  | github.com/apache/devlake/server/services.RunTasksStandalone.func1
  |     /app/server/services/task.go:187
  | runtime.goexit
  |     /usr/local/go/src/runtime/asm_amd64.s:1771
Wraps: (4) Retry exceeded 3 times calling repos/[ORGANIZATION]/[REPOSITORY]/pulls/29. ...
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError

### What do you expect to happen

A PR should only be re-fetched individually when it has actually changed since it was last collected, else in a few years time will be fetching way more closed than opened PR.

Reopening, pushing, commenting, reviewing, merging or closing a PR all bump `updated_at`, so if a closed PR gets re-opened Devlake should pick it up.


### How to reproduce

1. Collect a repository that has closed-unmerged PRs.
2. Run incremental collection again.
3. Observe one `GET /repos/{org}/{repo}/pulls/{number}` per open **and** per closed-unmerged
   PR in the database, on every run, including PRs untouched for years:
   ```sql
   SELECT COUNT(*) FROM _tool_github_pull_requests
   WHERE state != 'closed' OR merged = 0;
Anything else

Happens at every collection, and I assume collection time will increase the more unmerged closed PR there are.
Also DB size will become bloated with duplicated data seeing that each non-updated run stores the same payload in _raw_github_api_pull_requests.

Hope I didn't make obvious mistakes in that analysis.

Version

v1.0.3-beta18

Are you willing to submit PR?
  • Yes I am willing to submit a PR!
Code of Conduct
Langage dominant
Go
Étoiles
3.2k
Forks
824
Merge moyen
2 j 22 h
PR mergées (30 j)
45

Préparer son environnement

Ouvrir dans Codespaces

Lance le conteneur de développement du projet dans votre navigateur, avec votre propre compte GitHub.

  • Aucun Dockerfile ni fichier Docker Compose
  • Propose un modèle de pull request
  • Aucun guide de contribution

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de apache/devlake

Toutes les issues de apache/devlake

Issues similaires

Plus d'issues Go

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.