Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

groupby().agg() silently ignores weights, producing incorrect results

Abierto
  • ¿Algo no está bien?
#264 2 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
3/5
Tiempo estimado
1-2 días
Aptitud para principiantes
65/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Activo
Stack tecnológico
pandas, python
Área
backend-api-design, data, data-engineering

Línea de trabajo

The issue is in microdf/microdataframe.py lines 644-680, where the MicroDataFrameGroupBy class does not override .agg(). Start by reading the existing weighted method overrides (like sum). Determine how to parse aggregation specs and apply weights, or implement a clear error/warning. Run the provided test case to verify the fix.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Problem

MicroDataFrame's groupby operations silently ignore weights when using .agg(), producing incorrect (unweighted) results without any warning or error.

Example
import microdf as mdf
import numpy as np

df = mdf.MicroDataFrame(
    {"group": ["A", "A", "B", "B"], "value": [10, 20, 30, 40]},
    weights=np.array([2, 3, 1, 4])
)

# CORRECT (weighted):
df.groupby("group").value.sum()
# A: 10*2 + 20*3 = 80.0
# B: 30*1 + 40*4 = 190.0

# INCORRECT (unweighted) - no warning!
df.groupby("group").agg({'value': 'sum'})
# A: 10 + 20 = 30  ❌
# B: 30 + 40 = 70  ❌
Impact

This is a critical data correctness issue because:

  1. Silent failure: No error or warning - just wrong numbers
  2. Natural usage pattern: .agg() is a standard pandas idiom for multi-column aggregation
  3. Plausible results: The unweighted numbers look reasonable, making bugs hard to detect
  4. Real-world consequences: Users analyzing survey data (CPS, ACS, etc.) will get wildly incorrect population estimates
Root Cause

Looking at microdf/microdataframe.py:644-680, the MicroDataFrameGroupBy class:

  • ✓ Overrides specific methods like sum(), mean(), etc. to apply weights
  • ✗ Does NOT override .agg() or .aggregate(), so they fall back to pandas' unweighted implementation
Proposed Solutions

Option 1: Override .agg() to apply weights (Best)

  • Implement MicroDataFrameGroupBy.agg() to properly handle weights
  • Parse the aggregation specifications and route to weighted methods

Option 2: Raise an error (Safer than current behavior)

def agg(self, *args, **kwargs):
    raise NotImplementedError(
        "MicroDataFrameGroupBy.agg() does not support weights. "
        "Use df.groupby(col).column.sum() instead."
    )

Option 3: Emit a loud warning

def agg(self, *args, **kwargs):
    warnings.warn(
        "MicroDataFrameGroupBy.agg() ignores weights! Results will be unweighted.",
        UserWarning,
        stacklevel=2
    )
    return super().agg(*args, **kwargs)
Related Issues

This extends #193, which identified similar problems with .groupby()[[cols]].sum() but didn't specifically address .agg().

Additional Test Cases Needed
def test_agg_with_weights():
    """Test that .agg() applies weights correctly or raises an error"""
    df = mdf.MicroDataFrame(
        {"group": ["A", "A", "B"], "value": [10, 20, 30]},
        weights=np.array([2, 3, 4])
    )
    
    # These should either work correctly or raise NotImplementedError
    result = df.groupby("group").agg({'value': 'sum'})
    
    # If implemented, should equal weighted sums
    # A: 10*2 + 20*3 = 80
    # B: 30*4 = 120
    expected = pd.DataFrame({'value': [80.0, 120.0]}, index=['A', 'B'])
    expected.index.name = 'group'
    
    # Should NOT be unweighted sums (30, 30)
    assert not result.equals(pd.DataFrame({'value': [30, 30]}))
Priority

HIGH - This is a data correctness bug that produces silently wrong results in a library designed for weighted survey analysis.

Lenguaje dominante
Python
Estrellas
16
Forks
10
Merge medio
5 d 3 h
PR fusionados (30 d)
21

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de PolicyEngine/microdf

Todos los issues de PolicyEngine/microdf

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.