Check the demo apps' measurement end to end: results, statistics and revenue from app traffic
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
调研方向
Start with the nightly CI journeys for the ShopLab and StreamPulse demo apps, including the existing delivery checks for assignment, targeting, rollouts, and rollback. Compare app-traffic results, statistics, revenue, guardrails, and sample-ratio checks with independently computed answers using fixed seeds. The plan is done when it prioritizes the gaps, identifies any required product fixes, defines where each nightly check runs, and specifies how planted defects prove every check fails reliably.
由索引模型根据 Issue 内容生成。
描述
What is thin today
The demo apps exercise the delivery side of the platform well, and a nightly CI job drives it end to end: assignment, feature flags, targeting, staged rollouts and automatic rollback. The measurement side is checked much less:
- No check reads app results. No CI step reads the results of any ShopLab or StreamPulse experiment, so app traffic is never shown to reach the results page in the right arm.
- Statistics are never checked. The demo seeds turn on sequential testing, Bayesian analysis, CUPED and planned sample sizes, but nothing checks them against app traffic.
- Revenue is analysed as a rate. ShopLab's revenue metric is analysed as a conversion rate (#439).
- CUPED has nothing to work with. It has no pre-period data: the simulated traffic is mostly new users at a flat rate, with no returning visitors.
- Some checks are never exercised. Guardrail metrics and the sample-ratio check are never exercised from app traffic.
Goal
A plan that decides:
- which of these gaps are checked first;
- which product fixes they need, if any;
- where each check runs (the nightly journeys, not the per-PR suites, where possible).
The plan's rules for the checks:
- An independent oracle. Each check compares numbers produced by app traffic with an answer computed independently.
- Tamper-tested. A planted defect turns each check red.
- Not flaky. Fixed seeds, no single wall-clock timing, and no threshold on random traffic.
Process
A reviewed plan comes before any build. Related: the guided-examples page, which may show results only where these checks hold.
- 主要语言
- Python
- 星标
- 0
- 派生
- 4
- 平均合并
- 6 小时 5 分钟
- 30 天内合并 PR
- 453
环境准备
- 提供 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
Controlvar/controlvar 的其他 Issue
-
Dashboard: an archived flag shows Off with an on/off switch the API refuses可能已有人在做 @amarkanday 今天认领。 未关闭bug
难度 2/5 1-3 小时 新手友好度 72/100
Controlvar/controlvar#1748 · 1 条评论 ·
维护者通常 1 天内回复
-
Dashboard: the sequential panel's "Ready to stop" banner shows no sample-ratio-mismatch caution可能已有人在做 @amarkanday 今天认领。 未关闭bug
难度 2/5 1-3 小时 新手友好度 68/100
Controlvar/controlvar#1737 · 1 条评论 ·
维护者通常 1 天内回复
-
dashboard: a split URL draft's variant edit shows generic copy when the API refuses it可能已有人在做 @amarkanday 今天认领。 未关闭bug
难度 2/5 1-3 小时 新手友好度 70/100
Controlvar/controlvar#1562 · 1 条评论 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 90/100
Controlvar/controlvar#1447 · 2 条评论 ·
维护者通常 1 天内回复
-
Experiment setup: emptying a traffic share box and typing a number shows a leading zero ("034")可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭bug
难度 2/5 1-3 小时 新手友好度 72/100
Controlvar/controlvar#1428 · 1 条评论 ·
维护者通常 1 天内回复
查看 Controlvar/controlvar 的全部 Issue
相似的 Issue
-
documentation good first issue
难度 1/5 1 小时以内 新手友好度 88/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 84/100
MarketSquare/robotframework-browser#5479 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 78/100
endoflife-date/endoflife.date#11561 ·
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 76/100
OCA/l10n-brazil#5602 ·
维护者通常 1 天内回复
-
documentation review-2.0.0
难度 2/5 1-3 小时 新手友好度 78/100
dkuenzel/meshcore-home-gateway#106 ·
维护者通常 1 天内回复