Check the demo apps' measurement end to end: results, statistics and revenue from app traffic

Ouverte
  • Quelque chose ne va pas ?
#1,369 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
5/5
Temps estimé
Plus d'une semaine
Accessibilité débutants
35/100
Type d'issue
Documentation
Clarté
Clairement spécifiée
Activité
Active
Stack technique
python
Domaine
testing

Piste de recherche

Start with the nightly CI journeys for the ShopLab and StreamPulse demo apps, including the existing delivery checks for assignment, targeting, rollouts, and rollback. Compare app-traffic results, statistics, revenue, guardrails, and sample-ratio checks with independently computed answers using fixed seeds. The plan is done when it prioritizes the gaps, identifies any required product fixes, defines where each nightly check runs, and specifies how planted defects prove every check fails reliably.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

enhancement

What is thin today

The demo apps exercise the delivery side of the platform well, and a nightly CI job drives it end to end: assignment, feature flags, targeting, staged rollouts and automatic rollback. The measurement side is checked much less:

  • No check reads app results. No CI step reads the results of any ShopLab or StreamPulse experiment, so app traffic is never shown to reach the results page in the right arm.
  • Statistics are never checked. The demo seeds turn on sequential testing, Bayesian analysis, CUPED and planned sample sizes, but nothing checks them against app traffic.
  • Revenue is analysed as a rate. ShopLab's revenue metric is analysed as a conversion rate (#439).
  • CUPED has nothing to work with. It has no pre-period data: the simulated traffic is mostly new users at a flat rate, with no returning visitors.
  • Some checks are never exercised. Guardrail metrics and the sample-ratio check are never exercised from app traffic.

Goal

A plan that decides:

  • which of these gaps are checked first;
  • which product fixes they need, if any;
  • where each check runs (the nightly journeys, not the per-PR suites, where possible).

The plan's rules for the checks:

  • An independent oracle. Each check compares numbers produced by app traffic with an answer computed independently.
  • Tamper-tested. A planted defect turns each check red.
  • Not flaky. Fixed seeds, no single wall-clock timing, and no threshold on random traffic.

Process

A reviewed plan comes before any build. Related: the guided-examples page, which may show results only where these checks hold.

Langage dominant
Python
Étoiles
0
Forks
4
Merge moyen
5 h 19 min
PR mergées (30 j)
503

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de Controlvar/controlvar

Toutes les issues de Controlvar/controlvar

Issues similaires

Plus d'issues Python

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.