[Bug][GitHub] In-progress workflow runs are skipped and not collected after completion

Start with the GitHub “Collect Workflow Runs” task and the workflow collector, focusing on its handling of the GitHub /actions/runs results and incremental syncs. Reproduce the reported sequence with an in_progress run, then a completed run, and verify that the completed workflow is retained or revisited and its final status is available after a later incremental sync.

Search before asking

• I had searched in the issues and found no similar issues.

What happened

Hi DevLake team,

DevLake is a great platform, and it has already helped us solve several issues with our engineering data and reporting. We really appreciate all the work and effort from the DevLake team and community. Thank you for your continued support and for making such a valuable platform available to the community.

We are using Apache DevLake v1.0.2 with the GitHub plugin to collect GitHub Actions workflow runs.

Environment

• DevLake version: v1.0.2
• Plugin: GitHub
• Deployment: AWS ECS
• Docker image: Official Apache DevLake image
• Sync mode: Incremental sync
• Workflow Run sync frequency: Every 2 hours (0 */2 * * *)

Our production GitHub Actions workflows can sometimes run for 3–4 hours or longer. Because of this, we configured the GitHub Workflow Runs sync policy to run every 2 hours so that completed workflow runs can be collected regularly.

Problem

During the "Collect Workflow Runs" task, we see log messages such as:

Skipping run{id: 3372***, number: 1158***} with status in_progress
Skipping run{id: 3372***, number: 303***} with status in_progress

The workflow runs are therefore skipped while they are still in_progress.

Later, the GitHub Actions workflow finishes and GitHub changes the workflow run status to completed.

However, during subsequent incremental syncs, the previously skipped workflow run does not appear to be collected or updated after it becomes completed.

This means that a long-running workflow can be missed by the normal incremental sync if it is still in_progress when the collector checks it.

Expected behavior

We would expect DevLake to:

1. Detect the workflow run while it is in_progress.
2. Retain/track the workflow run as unfinished.
3. Re-check the workflow run during subsequent incremental syncs.
4. Once GitHub reports the run as completed, update the existing workflow run record with the final status and information.

Current behavior

Currently, the workflow run is skipped while it is in_progress.

If the workflow finishes after that collection cycle, the subsequent incremental sync does not appear to revisit the previously skipped run.

Investigation so far

We also checked whether this behavior has been addressed in newer DevLake versions.

Based on our investigation, the same completed-only behavior appears to exist in v1.0.3 and v1.0.4-alpha. We also understand that the GitHub workflow collector was rewritten in newer development versions to address GitHub /actions/runs pagination limits, but it still appears to keep only runs with status == "completed".

Therefore, upgrading the official image may improve GitHub collection reliability for large repositories, but it does not appear to address this specific long-running workflow scenario.

We also could not find an existing DevLake issue or PR specifically covering:

"Store in_progress GitHub Actions workflow runs and update them when they become completed."

The related pagination work appears to address a different problem.

Questions

Could you please confirm:

1. Is this expected behavior in DevLake v1.0.2?
2. Is there any configuration that allows DevLake to re-check previously skipped in_progress GitHub workflow runs during subsequent incremental syncs?
3. Is there a recommended timeAfter configuration or backfill/full-sync approach that can be scheduled automatically for this scenario?
4. If a full/backfill sync is the recommended workaround, how can we configure this in DevLake v1.0.2?
5. Is it possible to configure a backfill/time window that only covers the period containing these long-running workflow runs, rather than performing a complete full sync?
6. Has this behavior been fixed or changed in a newer DevLake version?
7. If a permanent fix is planned, is there a target version in which we could expect this behavior to be addressed?

Production requirement

This is particularly important for our production environment because our workflows can run for 3–4 hours or longer, while our normal GitHub Workflow Runs sync runs every 2 hours.

For example:

10:00 - Workflow starts
10:00 - DevLake sync runs → workflow is in_progress → DevLake skips it
12:00 - DevLake sync runs again
14:00 - Workflow finally completes
14:00+ - Subsequent incremental sync does not appear to revisit the previously skipped run

### What do you expect to happen

We expect DevLake to retain/track GitHub Actions workflow runs while they are `in_progress` and re-check them during subsequent incremental syncs.

Once the workflow run changes to `completed` in GitHub, DevLake should update the existing workflow run record with its final status and information.

This would ensure that long-running GitHub Actions workflows are not missed by incremental syncs when they are still `in_progress` during the initial collection.

### How to reproduce

1. Configure a GitHub connection in DevLake v1.0.2.
2. Configure the GitHub Workflow Runs sync policy with an incremental sync frequency of every 2 hours (`0 */2 * * *`).
3. Start a GitHub Actions workflow that takes several hours to complete.
4. Run the DevLake "Collect Workflow Runs" task while the GitHub workflow is still `in_progress`.
5. Observe that DevLake logs:
   `Skipping run{id: ..., number: ...} with status in_progress`
6. Allow the GitHub Actions workflow to finish and become `completed`.
7. Run the next incremental sync.
8. Observe that the previously skipped workflow run does not appear to be collected or updated in DevLake.

Expected result:
The workflow run should be retained/re-checked and updated once its GitHub status changes from `in_progress` to `completed`.

Actual result:
The workflow run is skipped while `in_progress` and does not appear to be revisited by subsequent incremental syncs after completion.

### Anything else

Skipping run{id: 3372***, number: 1158***} with status in_progress

### Version

v1.0.2

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)