Check the demo apps' measurement end to end: results, statistics and revenue from app traffic

Offen
  • Stimmt etwas nicht?
#1,369 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Anfängerfreundlichkeit
35/100
Issue-Typ
Dokumentation
Klarheit
Klar beschrieben
Aktivitätsstatus
Aktiv
Tech-Stack
python
Bereich
testing

Rechercherichtung

Start with the nightly CI journeys for the ShopLab and StreamPulse demo apps, including the existing delivery checks for assignment, targeting, rollouts, and rollback. Compare app-traffic results, statistics, revenue, guardrails, and sample-ratio checks with independently computed answers using fixed seeds. The plan is done when it prioritizes the gaps, identifies any required product fixes, defines where each nightly check runs, and specifies how planted defects prove every check fails reliably.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

enhancement

What is thin today

The demo apps exercise the delivery side of the platform well, and a nightly CI job drives it end to end: assignment, feature flags, targeting, staged rollouts and automatic rollback. The measurement side is checked much less:

  • No check reads app results. No CI step reads the results of any ShopLab or StreamPulse experiment, so app traffic is never shown to reach the results page in the right arm.
  • Statistics are never checked. The demo seeds turn on sequential testing, Bayesian analysis, CUPED and planned sample sizes, but nothing checks them against app traffic.
  • Revenue is analysed as a rate. ShopLab's revenue metric is analysed as a conversion rate (#439).
  • CUPED has nothing to work with. It has no pre-period data: the simulated traffic is mostly new users at a flat rate, with no returning visitors.
  • Some checks are never exercised. Guardrail metrics and the sample-ratio check are never exercised from app traffic.

Goal

A plan that decides:

  • which of these gaps are checked first;
  • which product fixes they need, if any;
  • where each check runs (the nightly journeys, not the per-PR suites, where possible).

The plan's rules for the checks:

  • An independent oracle. Each check compares numbers produced by app traffic with an answer computed independently.
  • Tamper-tested. A planted defect turns each check red.
  • Not flaky. Fixed seeds, no single wall-clock timing, and no threshold on random traffic.

Process

A reviewed plan comes before any build. Related: the guided-examples page, which may show results only where these checks hold.

Vorherrschende Sprache
Python
Sterne
0
Forks
4
Ø Merge
7 Std. 47 Min.
Gemergte PRs (30 T.)
427

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus Controlvar/controlvar

Alle Issues in Controlvar/controlvar

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.