Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Proposal] Sparse probing: optional groups argument so rows from one prompt can't straddle the split

Abierto
  • ¿Algo no está bien?
#1,813 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@lorenzozanee ya está trabajando en esto.

Desde el 26/9/2026.

  • #1824 de @lorenzozanee — abierto

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
25/100
Tipo de issue
Nueva funcionalidad
Claridad
Bien especificado
Estado de actividad
Estancado
Stack tecnológico
python, pytorch
Área
machine-learning

Línea de trabajo

Start with sparse_probing.py:251 and inspect fit_sparse_probe and sweep_sparse_probe, then review the guide’s leakage section and existing split tests. An open linked pull request (#1824) is already working on this proposal, so check its changes before considering any contribution. Done means group IDs do not cross the split, invalid groupings raise clearly, the supplied fixture scores near chance, the guide covers grouped splitting, and the listed checks pass.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

complexity-simple enhancement help wanted TransformerBridge
Proposal

Add an optional groups: Integer[torch.Tensor, "example"] to fit_sparse_probe and sweep_sparse_probe that holds out whole groups, defaulting to today's row-level split when omitted (sparse_probing.py:251).

Motivation

The guide already warns that rows sharing a source prompt must not straddle the split, but nothing in the API lets a caller act on it. Flattening [batch, pos, d_model], where every position of a document carries that document's label, is the normal way to build features and has no safe form today.

Pitch

On 40 groups of 8 rows: each group a shared identity vector plus noise, labels assigned per group at random, so the honest answer is no signal:

def grouped(n_groups=40, per_group=8, d=64, seed=0):
    g = torch.Generator().manual_seed(seed)
    ident = torch.randn(n_groups, d, generator=g) * 3.0
    rows = ident.repeat_interleave(per_group, 0) + torch.randn(n_groups * per_group, d, generator=g)
    labels = (torch.rand(n_groups, generator=g) < 0.5).long().repeat_interleave(per_group)
    return rows, labels, torch.arange(n_groups).repeat_interleave(per_group)

X, y, groups = grouped(seed=0)
fit_sparse_probe(X, y, k=8, seed=0).metrics.f1   # 0.805

Row-level F1 is 0.72–0.81 across seeds 0-3 where a group-held-out split gives 0.38–0.70. The probe is reading group identity out of the training rows of the same group, and nothing in the result says so.

  • groups assigns each row a group id; the stratified split partitions groups instead of rows, both classes still on both sides.
  • Omitting it changes nothing, so no existing result moves.

Acceptance:

  • No group id appears in both train_indices and test_indices
  • Clear raise when the grouping can't keep both classes on both sides
  • The fixture above scores near chance with groups supplied
  • Guide's leakage section shows the groups form
  • make unit-test passes
  • uv run mypy . passes
Checklist
  • I have checked that there is no similar issue in the repo (required)
Lenguaje dominante
Python
Estrellas
3.9k
Forks
708
Merge medio
1 d 18 h
PR fusionados (30 d)
65

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

  • Sin Dockerfile ni archivo de Docker Compose
  • Tiene una plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de TransformerLensOrg/TransformerLens

Todos los issues de TransformerLensOrg/TransformerLens

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.