Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

[Bug][GitHub] Closed (unmarked) Pull Requests are re-fetched on every run

Đang mở
  • Có gì không ổn?
#9,240 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 3 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
3/5
Thời gian dự kiến
1-2 ngày
Mức phù hợp với người mới
64/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
github, go
Lĩnh vực
data-engineering

Hướng nghiên cứu

Start with plugins/github/tasks/pr_collector.go, where the issue's stack trace points to pull request collection, and trace how incremental collection selects closed, unmerged PRs for individual fetches. Compare the selection with the reported github_updated_at and the API's updated_at behavior. Done means unchanged closed PRs are skipped, while PRs that change or reopen are collected; the issue names no test file, so locate the relevant collector tests first.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

type/bug
Search before asking
  • I had searched in the issues and found no similar issues.
What happened

One of my repo failed last night pipeline on the Collect Pull Requests task on a PR that had been closed (not merged) for more than a month and had not been updated since. For the story GitHub returned a 422 on it.

That made me take a deeper look as there was no reason to re-fetch that PR.
it looks like every closed but not merged PR is being re-fetched on every incremental run. On my instance that's about ~15k closed PRs.

For example, one of the oldest PR in my DB was closed in August 2019, has not been updated since, however at every run:

SELECT number, state, merged, github_created_at, closed_at, github_updated_at
FROM _tool_github_pull_requests
WHERE state = 'closed' AND merged = 0 AND merged_at IS NULL
ORDER BY github_updated_at ASC LIMIT 1;

number,state,merged,github_created_at,closed_at,github_updated_at
29,closed,0,2019-08-02 05:39:12,2019-08-08 08:31:19,2019-08-08 08:31:19

SELECT id, created_at, url FROM _raw_github_api_pull_requests
WHERE url LIKE '%/[REPOSITORY]/pulls/29'
ORDER BY created_at DESC LIMIT 7;

id,created_at,url
5014408,2026-10-19 00:27:23,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4992046,2026-10-18 00:27:07,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4969613,2026-10-17 00:27:19,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4947192,2026-10-16 00:27:38,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4924806,2026-10-15 00:27:32,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4902464,2026-10-14 00:27:21,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29
4880192,2026-10-13 00:27:17,https://api.github.com/repos/[ORGANIZATION]/[REPOSITORY]/pulls/29

183 identical fetches over the last 182 days of raw data we retain, for a PR untouched for 7 years.

Sanitized error:

subtask Collect Pull Requests ended unexpectedly
caused by: Retry exceeded 3 times calling repos/[ORGANIZATION]/[REPOSITORY]/pulls/29.
The last error was: Http DoAsync error calling [method:GET path:repos/[ORGANIZATION]/[REPOSITORY]/pulls/29 query:map[]].
Response: {"message":"The request could not be processed because too many files changed","documentation_url":"https://docs.github.com/rest/pulls/pulls#get-a-pull-request","status":"422"} (422)

attached stack trace
  -- stack trace:
  | github.com/apache/devlake/core/runner.RunPluginSubTasks
  |     /app/core/runner/run_task.go:341
  | [...repeated from below...]
Wraps: (2) subtask Collect Pull Requests ended unexpectedly
Wraps: (3) attached stack trace
  -- stack trace:
  | github.com/apache/devlake/helpers/pluginhelper/api.(*WorkerScheduler).WaitAsync
  |     /app/helpers/pluginhelper/api/worker_scheduler.go:173
  | github.com/apache/devlake/helpers/pluginhelper/api.(*ApiCollector).Execute
  |     /app/helpers/pluginhelper/api/api_collector.go:206
  | github.com/apache/devlake/helpers/pluginhelper/api.(*StatefulApiCollector).Execute
  |     /app/helpers/pluginhelper/api/api_collector_stateful.go:97
  | github.com/apache/devlake/plugins/github/tasks.CollectApiPullRequests
  |     /app/plugins/github/tasks/pr_collector.go:141
  | github.com/apache/devlake/core/runner.runSubtask
  |     /app/core/runner/run_task.go:433
  | github.com/apache/devlake/core/runner.RunPluginSubTasks
  |     /app/core/runner/run_task.go:338
  | github.com/apache/devlake/core/runner.RunPluginTask
  |     /app/core/runner/run_task.go:173
  | github.com/apache/devlake/core/runner.RunTask
  |     /app/core/runner/run_task.go:147
  | github.com/apache/devlake/server/services.runTaskStandalone
  |     /app/server/services/task_runner.go:114
  | github.com/apache/devlake/server/services.RunTasksStandalone.func1
  |     /app/server/services/task.go:187
  | runtime.goexit
  |     /usr/local/go/src/runtime/asm_amd64.s:1771
Wraps: (4) Retry exceeded 3 times calling repos/[ORGANIZATION]/[REPOSITORY]/pulls/29. ...
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.leafError

### What do you expect to happen

A PR should only be re-fetched individually when it has actually changed since it was last collected, else in a few years time will be fetching way more closed than opened PR.

Reopening, pushing, commenting, reviewing, merging or closing a PR all bump `updated_at`, so if a closed PR gets re-opened Devlake should pick it up.


### How to reproduce

1. Collect a repository that has closed-unmerged PRs.
2. Run incremental collection again.
3. Observe one `GET /repos/{org}/{repo}/pulls/{number}` per open **and** per closed-unmerged
   PR in the database, on every run, including PRs untouched for years:
   ```sql
   SELECT COUNT(*) FROM _tool_github_pull_requests
   WHERE state != 'closed' OR merged = 0;
Anything else

Happens at every collection, and I assume collection time will increase the more unmerged closed PR there are.
Also DB size will become bloated with duplicated data seeing that each non-updated run stores the same payload in _raw_github_api_pull_requests.

Hope I didn't make obvious mistakes in that analysis.

Version

v1.0.3-beta18

Are you willing to submit PR?
  • Yes I am willing to submit a PR!
Code of Conduct
Ngôn ngữ chính
Go
Star
3.2k
Fork
824
Merge trung bình
2 ngày 22 giờ
Pull request đã merge (30 ngày)
46

Chuẩn bị môi trường

Mở trong Codespaces

Khởi chạy dev container của dự án ngay trên trình duyệt, bằng tài khoản GitHub của bạn.

  • Không có Dockerfile hay tệp Docker Compose
  • Có mẫu pull request
  • Không có hướng dẫn đóng góp

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của apache/devlake

Tất cả issue của apache/devlake

Issue tương tự

Thêm issue về Go

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.