Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Sigmoid test adequacy

Abierto
  • ¿Algo no está bien?
#412 1 comentario 0 reacciones 1 asignado Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
25/100
Tipo de issue
Nueva funcionalidad
Claridad
Necesita aclaración
Estado de actividad
Activo
Stack tecnológico
python
Área
data-visualization, testing-qa

Línea de trabajo

The issue does not name files, tests, or entry points. Start by locating the current causal test adequacy calculation based on bootstrapped causal-effect kurtosis and the plots that display it; clarify the sigmoid transformation, score direction, and proposed stability labels before changing behavior.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Summary

Current causal test adequacy measurement, based on the kurtosis of bootstrapped causal effect estimates is unintuitive. It can be arbitrarily negative or positive, and zero is the best possible score (although because it's a statistical value, this isn't ever perfectly achievable except with infinite resamples of and infinite dataset). To make it more intuitive, we discussed putting it through two "half sigmoids" to bring positive values into the range [0, 1] and negative values into the range [0, -1]. We also discussed adding the labels "suspiciously stable" and "suspiciously unstable" to the plots.

Considerations

  • Target - Typically test adequacy is a "numbers go up" game where 100% is the goal. At the moment, we have 0 being the goal. The obvious easy thing to do here, based on the solution above, would be to simply transform the raw kurtosis number to bring it into the range [-1, 1], multiply by 100, and there's the "percentage" (although it's not a percentage of anything), so 0% is still the goal. If we can work out how, it'd be really cool to make it so that high numbers are better so that 100% is good (i.e., represents 0 kurtosis) and -100% is bad (i.e. represents -1 kurtosis). I'm not sure how you'd implement this conceptually, though.
  • What should be exponentially harder to reach - @SylviaWhittle from your explanation, it seems like the "obvious easy solution" mentioned above would make it exponentially more difficult to reach the worse values of test adequacy (corresponding to kurtosis values of -1 and 1). From the perspective of "once the raw values get sufficiently large, we can't really scream YOU NEED MORE DATA any louder", this sort of makes sense, but I can also see an argument for making it exponentially harder to achieve the best possible adequacy (corresponding to kurtosis of 0) to reflect the diminishing returns of additional data values. Or would this just serve to further diminish the diminishing returns.
    Update: This was a stupid idea - of course it makes sense to have the "linear" part of the sigmoid be approaching zero kurtosis rather than the "exponential" bit since the "final approach" to zero (which can never actually be reached because statistics) will be more or less the same for every system. While I have seen kurtosis values of 50+, this isn't super common, and especially not if you've got anywhere near enough data. Based on "vibes", I'd guess that having the linear bit start around 1 (or -1) would be about right, but I have absolutely no formal basis for saying that. If you can find anything theoretical to justify that, that'd be cool. If not, we can look at this dataset and try to find a suitable value and empirical justification. I do still think 100% rather than 0% should represent "good" though if we can.

@SylviaWhittle, I'm happy to discuss either of these points further if you'd like, but I'm also keen not to micromanage you and to give you some creative freedom here to play with stuff and see what works for you. It's great to have your input here, since you're a much better representative of the sort of person who we're hoping will eventually use the framework than I am (albeit an extremely capable and eager one).

Lenguaje dominante
Python
Estrellas
19
Forks
7
Merge medio
8 d 21 h
PR fusionados (30 d)
2

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de CITCOM-project/CausalTestingFramework

Todos los issues de CITCOM-project/CausalTestingFramework

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.