arms: significance and interval from one sign-flip test (D018 for eval/stats.py)

Aperta
  • Qualcosa non va?
#421 0 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@antonwolfy ci sta già lavorando.

Dal 6/11/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

enhancement

Problem

D018 (#292, #409) made every repo compare row take its 95% interval from inverting its sign-flip test, so the interval excludes 0 exactly when p < 0.05. It leaves the arms track (dokimi/eval/stats.py) to an issue of its own; this is that issue.

The arms track has the same kind of split, in a different form. _describe_deltas takes the interval from mean_ci (Student t), which agrees with p_ttest, but significant and significant_adjusted need both p_ttest and p_wilcoxon below 0.05. Near the boundary the interval and the verdict disagree. On main 37c0c21, paired_test(d, [0] * len(d)) gives:

deltas t interval p_ttest p_wilcoxon significant sign-flip p inverted interval
[0, 0, 0.2, 0.3, 0.25, 0.3] [+0.028, +0.323] 0.028 0.125 False 0.125 [0.000, +0.300]
[0, 0, 0, 0.2, 0.3, 0.25, 0.3, 0.2] [+0.044, +0.269] 0.014 0.0625 False 0.0625 [0.000, +0.275]
[-0.05, 0.2, 0.3, 0.25, 0.3, 0.2] [+0.063, +0.337] 0.013 0.0625 False 0.0625 [-0.050, +0.300]
[0.2, 0.3, 0.25, 0.3, 0.2] (5 pairs) [+0.188, +0.312] None None None 0.0625 open both ends

In each of the first three rows the interval excludes 0 and the row is not significant. With fewer than MIN_PAIRS_FOR_TEST (6) pairs no test runs, but a bounded t interval is still printed.

Callers: arm_significance (arms run, eval/arm_runner.py), paired_test (report/eval_report.py, report/model_compare.py), and eval/interactions.py, which builds on the same helpers.

Proposal

Follow D018 in the arms track: one sign-flip test on the per-question deltas decides significance, and its inversion gives the interval.

  • Reuse repos/compare_interval.py (ShiftTest, sign_flip_interval) and the sign-flip p, moved to a module both tracks can import within the layers in tests/test_import_architecture.py. eval sits below repos, so it cannot import them where they are now.
  • Holm across arms (_apply_correction) corrects the sign-flip p, as p_holm does in repo compare. The interval is uncorrected and says so.
  • Each result names its interval method (ci_method), as it does today.

Open questions

  • p_ttest and p_wilcoxon: keep them as descriptive fields next to the sign-flip p, or drop them? Dropping them changes the eval and arms records: check the schemas in dokimi/schemas/ and the readers through records.py.
  • Interactions: do the 2x2 contrasts in eval/interactions.py move too, in this issue or a later one?
  • Cohen's d_z stays a descriptive effect size, as it is now?
  • Without SciPy, p_ttest and p_wilcoxon are None today; the sign-flip test needs only NumPy, so a run without SciPy would get a verdict.

Done when

Arms-track significance and its interval come from the same sign-flip test, and pinned tests show they agree, including the rows above. The stats.py docstrings and the docs that describe arms statistics say so, and D018 is updated or followed by a record for the arms track.

Lingua principale
Python
Stelle
11
Fork
9
Merge medio
1g 11h
PR unite (30g)
335

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di intel/dokimi

Tutte le issue di intel/dokimi

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.