Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Bug][refdiff] Incorrect deployment diffs stay cached after missing commit parents are collected

Abierto
  • ¿Algo no está bien?
#9,187 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
55/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
go
Área
backend, data-engineering

Línea de trabajo

Start with backend/plugins/refdiff/utils/commit_node_graph.go and run the provided incomplete_graph_test.go reproducer to understand the missing-parent result. Then inspect backend/plugins/refdiff/tasks/deployment_commit_diff_calculator.go and helpers/pluginhelper/api/batch_save.go for completion markers and replacement behavior. Done means incomplete diffs are not finalized, corrected parent history causes affected pairs to be recalculated, and obsolete rows are removed.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Search before asking
  • I had searched in the issues and found no similar issues.

I searched open and closed issues and PRs, including related comments and patches. Similar symptoms have been reported, but I did not find a report or PR covering this specific refdiff cache behavior after missing parent history is collected. Related fixes and their different scope are listed below.

What happened

calculateDeploymentCommitsDiff can save an incorrect result when DevLake's stored commit history is incomplete. After the missing parent relationships are collected, the old result is still treated as finished, including during Collect Data in Full Refresh Mode.

Here is a small example. Letters are placeholders, not real commit SHAs. Lines show commits leading to newer commits, from left to right.

Complete Git history:

R --- H --- A -------- B
       \              /
        ----- F ------

A = previous successful production deployment
B = next successful production deployment

Already in A: R, H, A
New in B:     F, B

Suppose DevLake has not collected the edge between A and its parent H:

DevLake's incomplete stored graph:

R --- H    [missing edge]    A --- B
       \                         /
        ------------ F ----------

From A, DevLake can only find A.
From B, it can still find F, H, and R through the other branch.

Expected new commits: B, F
Calculated commits:  B, F, H, R

Refdiff saves the incorrect result and writes a completion marker for (new=B, old=A). When the missing edge is later collected, the marker causes the pair to be skipped. The incorrect rows remain in commits_diffs.

This can link old PRs/MRs to a much later deployment and greatly inflate Lead Time for Changes.

In an affected instance, Full Refresh collected the missing parent data but did not repair the cached diff. Removing both the affected pair's diff rows and completion marker, then rerunning refdiff and DORA, corrected the metric. A later normal incremental run preserved the correction.

This report focuses on refdiff's handling of incomplete data and its cache. The exact reason the collector initially missed the parent data is not established here.

What do you expect to happen
  • If required commit history is missing, collect it or report an incomplete result. Do not mark an unverified diff as finished.
  • When the parent graph is corrected, invalidate or revalidate affected cached pairs. Full Refresh should not silently preserve a known incorrect result.
  • Recalculation should replace the pair's old result, so commits that no longer belong in the diff are removed.
How to reproduce

Graph calculation: use the example above with CommitNodeGraph.

  1. Add the parent edges H -> R, F -> H, B -> A, and B -> F. Here, child -> parent means a call to AddParent(child, parent).
  2. Do not add A -> H yet.
  3. Call CalculateLostSha("A", "B"). It returns B, F, H, R without reporting incomplete history.
  4. Add A -> H and call it again. It now returns the correct result, B, F.

Cache behavior: in a disposable test project, create two successful production deployment records for A and B, with B's previous successful deployment pointing to A. Make the example commits and parent edges available through that project's repository mapping.

  1. Run calculateDeploymentCommitsDiff while A -> H is missing. The incorrect diff is stored and the pair is marked finished.
  2. Add the missing parent edge.
  3. Rerun the task for the same pair, including with fullSync=true. The completion-marker check skips the pair, so the old diff remains.

The graph calculation was reproduced locally using the upstream code. The stale-cache behavior was observed on the affected instance; the small example is a synthetic illustration, not its private data.

Small graph-only reproducer

Save this as incomplete_graph_test.go in backend/plugins/refdiff/utils, then run:

go test -v commit_node_graph.go incomplete_graph_test.go

package utils

import "testing"

func TestIncompleteDeploymentHistory(t *testing.T) {
    g := NewCommitNodeGraph()
    g.AddParent("H", "R")
    g.AddParent("F", "H")
    g.AddParent("B", "A")
    g.AddParent("B", "F")
    // A is not a root commit, but its parent edge has not been collected.
    before, _, _ := g.CalculateLostSha("A", "B")

    g.AddParent("A", "H")
    after, _, _ := g.CalculateLostSha("A", "B")

    t.Logf("missing A -> H: %v", before)
    t.Logf("after collecting A -> H: %v", after)
    if len(before) != 4 || len(after) != 2 {
        t.Fatalf("unexpected result: before=%v after=%v", before, after)
    }
}

Output:

missing A -> H: [B F H R]
after collecting A -> H: [B F]

This test documents the current graph behavior. It does not test the database cache or mean that the bug is fixed.

Anything else

Relevant code in v1.0.3-beta15:

  • CalculateLostSha treats missing parent information as the end of history and does not return a completeness error.
  • calculateDeploymentCommitsDiff skips pairs with completion markers, without checking graph changes or FullSync.
  • BatchSave uses CreateOrUpdate. Removing only the completion marker is therefore not enough: recalculation does not delete old diff rows that are absent from the new result.

The graph algorithm and deployment-diff task have the same logic in v1.0.3-beta18 and the main revision reviewed. This was a source comparison, not a runtime test on those versions.

Related reports:

  • #8434, #8818, and #8857 concern missing commit data. This report starts after a data gap exists and focuses on refdiff's incorrect result and stale cache. Merged PR #8959 improves GitLab MR commit collection; it does not repair refdiff's cache.
  • #8188, #8249, and #8790 describe similar symptoms of incorrect PR-to-deployment mapping. Merged PR #8942 adds a two-phase DORA lookup: direct SHA match, then diff-based fallback. The beta15 DORA source already contains this fix, but the fallback still reads commits_diffs. The PR does not invalidate or replace refdiff's cached results.
  • #9182 / #9183 concern repeated calculation when there is no previous successful deployment; PR #9183 is still open. Here, both A and B exist and both SHAs are non-empty; the problem is that an incorrect result is not recalculated.

All commit labels and diagrams above are synthetic. No internal project names, URLs, commit SHAs, deployment IDs, dates, logs, or credentials are included.

Version

v1.0.3-beta15 (MySQL 8, GitLab integration)

Are you willing to submit PR?
  • Yes I am willing to submit a PR!
Code of Conduct
Lenguaje dominante
Go
Estrellas
3.1k
Forks
812
Merge medio
2 d 8 h
PR fusionados (30 d)
50

Preparar el entorno

Aún no hemos revisado los archivos de configuración de este proyecto. Empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/devlake

Todos los issues de apache/devlake

Issues similares

Más issues de Go

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.