Evaluate selecting test configs separately for each confidence level

Offen
  • Stimmt etwas nicht?
#7,386 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 2 Tagen

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Anfängerfreundlichkeit
25/100
Issue-Typ
Feature
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
python
Bereich
machine-learning

Rechercherichtung

Start by locating the HTTP service that returns config_groups, the config selection model evaluation, and the Firefox taskgraph strategies named bugbug-reduced-manifests-config-selection-*. Measure selection cost and regressions caught for each level against a selection using only groups at or above that level. The issue is resolved when the measurements show whether separate per-level selections are worthwhile and, if so, the service replies and taskgraph strategies use them.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

The HTTP service selects the configs of each confidence level (very_high, high, medium, low, very_low) after the ones of the levels above, reusing them, and returns a single config_groups for all levels. The Firefox taskgraph strategies that select configs (bugbug-reduced-manifests-config-selection-*) run the groups at or above their level on these configs.

This keeps the configs of a level independent from the groups below it, which don't run with it. However, a strategy at a lower level runs the groups of the levels above too, and their configs were selected without knowing which configs the lower groups would use: e.g. for the low strategy, a medium group can't reuse a config selected for a low group, so the low strategy may run more configs than a selection made for its groups alone.

We should measure how much this costs, e.g. in the config selection model's evaluation, comparing the cost and the regressions caught at each level with a selection made for the groups at or above it only.

If the difference is significant, the HTTP service could return a separate config selection for each level (a selection over all the groups at or above it), and the taskgraph strategies could read the one for their level. This would increase the size of the replies and the number of config selection model calls, and require changes in the taskgraph.

Vorherrschende Sprache
Python
Sterne
578
Forks
366
Ø Merge
2 T. 15 Std.
Gemergte PRs (30 T.)
127

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus mozilla/bugbug

Alle Issues in mozilla/bugbug

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.