Check the demo apps' measurement end to end: results, statistics and revenue from app traffic

Abierto
  • ¿Algo no está bien?
#1,369 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
35/100
Tipo de issue
Documentación
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
python
Área
testing

Línea de trabajo

Start with the nightly CI journeys for the ShopLab and StreamPulse demo apps, including the existing delivery checks for assignment, targeting, rollouts, and rollback. Compare app-traffic results, statistics, revenue, guardrails, and sample-ratio checks with independently computed answers using fixed seeds. The plan is done when it prioritizes the gaps, identifies any required product fixes, defines where each nightly check runs, and specifies how planted defects prove every check fails reliably.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

enhancement

What is thin today

The demo apps exercise the delivery side of the platform well, and a nightly CI job drives it end to end: assignment, feature flags, targeting, staged rollouts and automatic rollback. The measurement side is checked much less:

  • No check reads app results. No CI step reads the results of any ShopLab or StreamPulse experiment, so app traffic is never shown to reach the results page in the right arm.
  • Statistics are never checked. The demo seeds turn on sequential testing, Bayesian analysis, CUPED and planned sample sizes, but nothing checks them against app traffic.
  • Revenue is analysed as a rate. ShopLab's revenue metric is analysed as a conversion rate (#439).
  • CUPED has nothing to work with. It has no pre-period data: the simulated traffic is mostly new users at a flat rate, with no returning visitors.
  • Some checks are never exercised. Guardrail metrics and the sample-ratio check are never exercised from app traffic.

Goal

A plan that decides:

  • which of these gaps are checked first;
  • which product fixes they need, if any;
  • where each check runs (the nightly journeys, not the per-PR suites, where possible).

The plan's rules for the checks:

  • An independent oracle. Each check compares numbers produced by app traffic with an answer computed independently.
  • Tamper-tested. A planted defect turns each check red.
  • Not flaky. Fixed seeds, no single wall-clock timing, and no threshold on random traffic.

Process

A reviewed plan comes before any build. Related: the guided-examples page, which may show results only where these checks hold.

Lenguaje dominante
Python
Estrellas
0
Forks
4
Merge medio
5 h 36 min
PR fusionados (30 d)
509

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de Controlvar/controlvar

Todos los issues de Controlvar/controlvar

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.