Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Bug][GitHub] Closed (unmarked) Pull Requests are re-fetched on every run

Abierto
  • ¿Algo no está bien?
#9,240 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 3 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
64/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
github, go
Área
data-engineering

Línea de trabajo

Start with plugins/github/tasks/pr_collector.go, where the issue's stack trace points to pull request collection, and trace how incremental collection selects closed, unmerged PRs for individual fetches. Compare the selection with the reported github_updated_at and the API's updated_at behavior. Done means unchanged closed PRs are skipped, while PRs that change or reopen are collected; the issue names no test file, so locate the relevant collector tests first.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

type/bug
Search before asking
  • I had searched in the issues and found no similar issues.
What happened

One of my repo failed last night pipeline on the Collect Pull Requests task on a PR that had been closed (not merged) for more than a month and had not been updated since. For the story GitHub returned a 422 on it.

That made me take a deeper look as there was no reason to re-fetch that PR.
it looks like every closed but not merged PR is being re-fetched on every incremental run. On my instance that's about ~15k closed PRs.

For example, one of the oldest PR in my DB was closed in August 2019, has not been updated since, however at every run:

SELECT number, state, merged, github_created_at, closed_at, github_updated_at
FROM _tool_github_pull_requests
WHERE state = 'closed' AND merged = 0 AND merged_at IS NULL
ORDER BY github_updated_at ASC LIMIT 1;

number,state,merged,github_created_at,closed_at,github_updated_at
29,closed,0,2019-08-02 05:39:12,2019-08-08 08:31:19,2019-08-08 08:31:19

SELECT id, created_at, url FROM _raw_github_api_pull_requests
WHERE url LIKE '%/[REPOSITORY]/pulls/29'
ORDER BY created_at DESC LIMIT 7;

id,created_at,url
5014408,2026-10-19 00:27:23,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4992046,2026-10-18 00:27:07,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4969613,2026-10-17 00:27:19,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4947192,2026-10-16 00:27:38,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4924806,2026-10-15 00:27:32,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4902464,2026-10-14 00:27:21,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4880192,2026-10-13 00:27:17,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29

183 identical fetches over the last 182 days of raw data we retain, for a PR untouched for 7 years.

Sanitized error:

subtask Collect Pull Requests ended unexpectedly
caused by: Retry exceeded 3 times calling repos/[ORGANIZATION]/[REPOSITORY]/pulls/29.
The last error was: Http DoAsync error calling [method:GET path:repos/[ORGANIZATION]/[REPOSITORY]/pulls/29 query:map[]].
Response: {"message":"The request could not be processed because too many files changed","documentation_url":"https://docs.github.com/rest/pulls/pulls#get-a-pull-request","status":"422"} (422)

attached stack trace
  -- stack trace:
  | github.com/apache/devlake/core/runner.RunPluginSubTasks
  |     /app/core/runner/run_task.go:341
  | [...repeated from below...]
Wraps: (2) subtask Collect Pull Requests ended unexpectedly
Wraps: (3) attached stack trace
  -- stack trace:
  | github.com/apache/devlake/helpers/pluginhelper/api.(*WorkerScheduler).WaitAsync
  |     /app/helpers/pluginhelper/api/worker_scheduler.go:173
  | github.com/apache/devlake/helpers/pluginhelper/api.(*ApiCollector).Execute
  |     /app/helpers/pluginhelper/api/api_collector.go:206
  | github.com/apache/devlake/helpers/pluginhelper/api.(*StatefulApiCollector).Execute
  |     /app/helpers/pluginhelper/api/api_collector_stateful.go:97
  | github.com/apache/devlake/plugins/github/tasks.CollectApiPullRequests
  |     /app/plugins/github/tasks/pr_collector.go:141
  | github.com/apache/devlake/core/runner.runSubtask
  |     /app/core/runner/run_task.go:433
  | github.com/apache/devlake/core/runner.RunPluginSubTasks
  |     /app/core/runner/run_task.go:338
  | github.com/apache/devlake/core/runner.RunPluginTask
  |     /app/core/runner/run_task.go:173
  | github.com/apache/devlake/core/runner.RunTask
  |     /app/core/runner/run_task.go:147
  | github.com/apache/devlake/server/services.runTaskStandalone
  |     /app/server/services/task_runner.go:114
  | github.com/apache/devlake/server/services.RunTasksStandalone.func1
  |     /app/server/services/task.go:187
  | runtime.goexit
  |     /usr/local/go/src/runtime/asm_amd64.s:1771
Wraps: (4) Retry exceeded 3 times calling repos/[ORGANIZATION]/[REPOSITORY]/pulls/29. ...
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError

### What do you expect to happen

A PR should only be re-fetched individually when it has actually changed since it was last collected, else in a few years time will be fetching way more closed than opened PR.

Reopening, pushing, commenting, reviewing, merging or closing a PR all bump `updated_at`, so if a closed PR gets re-opened Devlake should pick it up.


### How to reproduce

1. Collect a repository that has closed-unmerged PRs.
2. Run incremental collection again.
3. Observe one `GET /repos/{org}/{repo}/pulls/{number}` per open **and** per closed-unmerged
   PR in the database, on every run, including PRs untouched for years:
   ```sql
   SELECT COUNT(*) FROM _tool_github_pull_requests
   WHERE state != 'closed' OR merged = 0;
Anything else

Happens at every collection, and I assume collection time will increase the more unmerged closed PR there are.
Also DB size will become bloated with duplicated data seeing that each non-updated run stores the same payload in _raw_github_api_pull_requests.

Hope I didn't make obvious mistakes in that analysis.

Version

v1.0.3-beta18

Are you willing to submit PR?
  • Yes I am willing to submit a PR!
Code of Conduct
Lenguaje dominante
Go
Estrellas
3.2k
Forks
824
Merge medio
2 d 22 h
PR fusionados (30 d)
46

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

  • Sin Dockerfile ni archivo de Docker Compose
  • Tiene una plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/devlake

Todos los issues de apache/devlake

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.