[Bug][Tempo] A team scope's pipeline adds every Tempo team as a scope and resets their scope config

Geschlossen
  • Stimmt etwas nicht?
#9,318 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 2 Tagen

Ein zugehöriger Pull Request wurde bereits gemerged.

  • #9319 von @ShivamSharma43 — gemerged

Bewertung

Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Anfängerfreundlichkeit
25/100
Issue-Typ
Bug
Klarheit
Klar beschrieben
Aktivitätsstatus
Veraltet
Tech-Stack
go
Bereich
data-engineering

Rechercherichtung

Start with the Tempo tasks in backend/plugins/tempo/tasks, especially collect-teams and extract_teams, and compare their team scoping with the worklog collector. Use the reproduction described in the issue and the raw fixture e2e/raw_tables/_raw_tempo_api_teams.csv to verify behavior; done means running one team scope leaves other scopes and its scope config unchanged. A pull request is already open for this issue.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Search before asking
  • I had searched in the issues and found no similar issues.
What happened

Running a blueprint with one Tempo team scope adds every team of the Tempo instance as a data scope of the connection, and resets the scope config of the team scopes.

The _tool_tempo_teams table is both the scope table (TempoTeam is the plugin's ToolLayerScope) and the output of the extract_teams subtask. In backend/plugins/tempo/tasks:

  • collect-teams calls GET /4/teams without a team filter, so it collects all teams of the instance, with connection-wide raw params {"ConnectionId":1,"TeamId":0} (the team id of the scope isn't set, unlike the worklog collector).
  • extract_teams turns every collected team into a TempoTeam with ScopeConfigId 0. Because the params are connection-wide, the extractor's batch save first deletes every _tool_tempo_teams row with those params (i.e. every team scope extracted before, for any blueprint of the connection) and re-inserts all teams.

Result after one pipeline run:

  1. Teams the user never selected show up under the connection's data scopes (GET /plugins/tempo/connections/1/scopes).
  2. The scope_config_id of the selected team scopes is overwritten with 0, so the scope config assignment is lost on every run.
  3. A team scope the user deletes from the connection comes back on the next run of any other team scope.

Reproduced on main (84e3707) with the plugin's e2e helper on PostgreSQL: with only team 71 in _tool_tempo_teams (scope_config_id = 3) and the raw rows of e2e/raw_tables/_raw_tempo_api_teams.csv (teams 17 and 71), running ExtractTeams with {ConnectionId: 1, TeamId: 71} leaves:

team_id=17 name="Abstract BE Team"        scope_config_id=0 params={"ConnectionId":1,"TeamId":0}
team_id=71 name="SmartShopper (RTL+WHS)"  scope_config_id=0 params={"ConnectionId":1,"TeamId":0}

Also, collect-teams is a stateful collector that sends updatedFrom on incremental runs, but GET /4/teams has no such parameter (the Tempo OpenAPI spec, https://apidocs.tempo.io/tempo-openapi.yaml, lists offset, limit, teamMembers, name and teamIds). So an incremental run fetches every team again and appends it to the raw table instead of replacing it.

What do you expect to happen

A team scope's pipeline only collects and extracts its own team. The other team scopes of the connection, and teams that were never selected, are left alone, and the team scope keeps its scope config.

How to reproduce
  1. Create a Tempo connection to an instance with several teams.
  2. Add one team (e.g. id 71) as a data scope, assign it a scope config, and add it to a blueprint.
  3. Run the blueprint.
  4. SELECT team_id, scope_config_id, _raw_data_params FROM _tool_tempo_teams WHERE connection_id = 1 lists every team of the instance, all with scope_config_id = 0; the connection's data scope list in config-ui shows all of them.
Anything else

Proposed fix (I have a PR ready):

  • Scope the team raw data and the extracted rows per team ({ConnectionId, TeamId}, like TempoTeam.ScopeParams() and the worklog collector), so extracting one team scope only replaces its own row.
  • collect-teams: request only the scope's team with teamIds, drop the unsupported updatedFrom and use a plain (non-incremental) collector so each run refreshes the raw rows.
  • extract_teams: skip teams other than the scope's team and keep the existing scope_config_id.
  • Unit tests and an e2e test.

Related: #9284 (team summary longer than 255 characters). Since every team of the instance is extracted today, one long summary on any team can break extract_teams for every team scope of the connection.

Version

main

Are you willing to submit PR?
  • Yes I am willing to submit a PR!
Code of Conduct
Vorherrschende Sprache
Go
Sterne
3.2k
Forks
832
Ø Merge
2 T. 6 Std.
Gemergte PRs (30 T.)
86

Entwicklungsumgebung

In Codespaces öffnen

Startet den Dev-Container des Projekts im Browser, mit Ihrem eigenen GitHub-Konto.

  • Kein Dockerfile und keine Docker-Compose-Datei
  • Hat eine Pull-Request-Vorlage
  • Kein Beitragsleitfaden

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus apache/devlake

Alle Issues in apache/devlake

Ähnliche Issues

Weitere Issues zu Go

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.