Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

repo compare: an interval that agrees with the sign-flip p

Abierto
  • ¿Algo no está bien?
#292 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
38/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
numpy, python
Área
data

Línea de trabajo

Start with repos/compare.py and its paired_summary logic to trace how rows currently get intervals and sign-flip p-values; read §9 of the contract and the research-register guidance before choosing the method. Measure the proposed inversion cost and resolve the open questions about discreteness, the bootstrap flag, and other tracks. Done means every repo compare row uses an inverted sign-flip interval, tests cover known delta sets and agreement with p, and the contract, module docstring, and research record are updated.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Problem

repo compare takes a row's 95% interval from one method and its p from another. Gate and score rows use a percentile bootstrap, the other rows a t interval, and p always comes from a sign-flip test (repos/compare.py docstring and paired_summary). Near 0.05 the two disagree, and a reader can't tell which to believe. Two rows from a replay of IntelPython/dpnp#3099 (26 review_replay tasks):

row delta interval p (sign-flip)
metric:n_findings, 26 tasks +0.89 t: [−0.091, +1.860] (includes 0) 0.0497 (t-test 0.074)
score, 6 tasks +0.139 bootstrap: [+0.028, +0.250] (excludes 0) 0.25

In the second row, three non-zero deltas, all positive, can't give an exact sign-flip p below 0.25, while a bootstrap over six tasks is too narrow. A separate change adds a note line wherever the two disagree. This issue is about removing the cause.

Proposal

Take every row's interval from the sign-flip test itself, by inverting it.

  • The interval: every shift θ for which the test on d − θ does not reject at 0.05. Find each end by bisection between the mean and a far bound.
  • Exactness: exact up to EXACT_MAX (16) non-zero deltas, as the p is. Above that, Monte Carlo with the fixed seed and common random numbers across θ, so the ends are stable.
  • The result: "interval excludes 0" holds exactly when p < 0.05 (exactly in the exact range, up to Monte Carlo error above it), on every row.
  • Few non-zero deltas: the interval is wide, or open on one side, which is the honest answer.

This is a change of method, so it goes with a research-register decision record and with §9 of the contract.

Open questions

  • Discreteness: the exact test's p jumps as θ crosses a delta, so the interval ends sit at data points. Is that acceptable as is, or should a mid-p version be used?
  • Speed: 2^16 sign patterns × ~30 bisection steps × 2 ends × rows. Fine in NumPy? Measure it.
  • Comparability: keep the bootstrap behind a flag, so a comparison can be redone the old way?
  • Other tracks: should the arms track (eval/stats.py) follow too?

Done when

repo compare computes every row's interval by inversion. Tests pin the interval on known delta sets, including the two rows above, and check that it agrees with p. §9 and the module docstring describe it, and a research record holds the decision.

Lenguaje dominante
Python
Estrellas
9
Forks
9
Merge medio
1 d 7 h
PR fusionados (30 d)
249

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de intel/dokimi

Todos los issues de intel/dokimi

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.