Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Standardize temporal column statistics across pandas, Polars, and the UI

Abierto
  • ¿Algo no está bien?
#1,737 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 2 días

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
32/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
pandas, python, typescript
Área
backend-api-design, data-engineering, frontend, testing

Línea de trabajo

Start by comparing the pandas and Polars statistics modules and their listed tests, then inspect DAGWorksDescribe.tsx and the observability schema or generated types. Run the focused SDK tests and existing frontend tests to establish current payload and rendering behavior. Done means the documented temporal contract is implemented across both producers, schemas, renderers, and cross-backend/frontend tests, including timezone and malformed-value cases.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

core-work enhancement SDK UI

Problem

Hamilton's observability schema currently treats several distinct temporal concepts as if they were JavaScript-compatible datetimes. pandas and Polars consequently produce different payloads for equivalent columns, and some payloads either cannot be serialized or render incorrectly in the UI.

This was identified while reviewing #1675 and Jernej's comment in https://github.com/apache/hamilton/pull/1675#discussion_r3795651892. PR #1675 should remain a focused fix: Polars Time and Duration are routed to the existing unhandled path so they no longer break datetime rendering. This issue tracks the complete cross-backend solution.

Current behavior
Logical type pandas Polars UI consequence
Datetime base_data_type=\"datetime\"; ISO-like statistics base_data_type=\"datetime\"; ISO-like statistics Existing datetime renderer generally works
Date Python date object columns normally become unhandled base_data_type=\"datetime\"; min/max are YYYY-MM-DD new Date(\"YYYY-MM-DD\") is interpreted as UTC and can display the previous day in negative-offset timezones
Time Object columns normally become unhandled unhandled after #1675 No rich statistics, but safe
Duration/timedelta base_data_type=\"datetime\"; min/max/mean/quantiles contain raw pd.Timedelta unhandled after #1675 pandas payload fails ordinary JSON serialization; duration strings are not valid JavaScript Dates anyway

The frontend datetime path in ui/frontend/src/components/dashboard/Runs/Task/result-summaries/DAGWorksDescribe.tsx calls new Date(...) for range, quantile, and histogram values. That operation does not preserve Date semantics and cannot parse Time or Duration values.

The generated frontend datetime type and backend Pydantic datetime model also do not declare all fields that producers currently emit, which prevents static typing from catching contract drift.

Desired solution

Define one explicit temporal statistics contract shared by pandas, Polars, the backend schema, generated frontend types, and UI renderers.

Logical types

Use distinct base_data_type values for:

  1. datetime
  2. date
  3. time
  4. duration

Do not route Date, Time, or Duration through the datetime renderer.

Canonical JSON representations

The precise representation should be documented and applied identically by both producers. A suggested starting point:

  • Datetime: ISO 8601 datetime strings with an explicit, documented timezone policy. Preserve offsets for timezone-aware values.
  • Date: YYYY-MM-DD strings. Treat them as calendar dates in the UI; do not pass them directly to new Date(string).
  • Time: ISO local-time strings such as HH:mm:ss[.ffffff], with an explicit policy for timezone-aware time values if supported.
  • Duration: a numeric canonical unit, preferably integer microseconds, for min/max/mean/quantiles and charting. Avoid display-formatted strings as the transport representation. If ISO 8601 durations are selected instead, the UI must parse them explicitly and tests must cover ordering/charting.

All emitted statistics—including min, max, mean, standard deviation, quantiles, and histogram boundaries—must be JSON-safe and use the canonical representation for their logical type.

A schema-version bump may be required if adding base_data_type values is not compatible with the current dagworks_describe v0.0.3 contract.

UI behavior

Add explicit dispatch and rendering for each type:

  • Datetime: retain the existing renderer after confirming timezone behavior.
  • Date: render calendar dates without UTC conversion or timezone shifting. Construct from components or display the canonical value through a date-aware formatter.
  • Time: render wall-clock values without inventing a date or applying a browser timezone.
  • Duration: format canonical numeric values into readable durations and chart them numerically.

Unsupported or malformed values should degrade to the existing unhandled presentation rather than producing Invalid Date or NaN.

Schema and generated types

Update the backend observability models and generated frontend API types so each temporal statistics shape declares the fields actually emitted and consumed. Avoid casting datetime records to numeric statistics types in the frontend.

Acceptance criteria

  • Equivalent pandas and Polars columns produce the same logical base_data_type and value shape for Datetime, Date, Time, and Duration.
  • Every pandas and Polars temporal payload passes JSON serialization without custom fallback encoders.
  • Datetime timezone/offset behavior is documented and tested.
  • Date values display the same calendar day in positive and negative UTC-offset browser timezones.
  • Time values display without an invented date or timezone conversion.
  • Duration values render and sort/chart numerically with documented precision and unit.
  • The frontend never invokes the datetime renderer for Date, Time, or Duration.
  • Malformed/unsupported temporal values degrade safely instead of rendering Invalid Date/NaN.
  • Backend and generated frontend types match the actual payload fields.
  • Cross-backend contract tests cover min, max, mean, std, quantiles, histogram boundaries where applicable, missing values, and empty/all-null columns.
  • Frontend tests run under at least one negative-offset timezone (for example America/Los_Angeles) and verify Date behavior.
  • Upgrade/schema compatibility is documented, including how older SDK payloads are rendered.

Suggested implementation areas

  • pandas routing and serialization:
    • ui/sdk/src/hamilton_sdk/tracking/pandas_stats.py
    • ui/sdk/src/hamilton_sdk/tracking/pandas_col_stats.py
  • Polars routing and serialization:
    • ui/sdk/src/hamilton_sdk/tracking/polars_stats.py
    • ui/sdk/src/hamilton_sdk/tracking/polars_col_stats.py
  • Observability schema/models and generated API types
  • UI grouping and renderers:
    • ui/frontend/src/components/dashboard/Runs/Task/result-summaries/DAGWorksDescribe.tsx
  • Tests:
    • ui/sdk/tests/tracking/test_pandas_stats.py
    • ui/sdk/tests/tracking/test_pandas_col_stats.py
    • ui/sdk/tests/tracking/test_polars_stats.py
    • ui/sdk/tests/tracking/test_polars_col_stats.py
    • focused frontend temporal renderer/contract tests

Incremental path

Until this contract and its renderers exist, the safe schema-v0.0.3 fallback is for only true Datetime columns to use the datetime path and for unsupported Date/Time/Duration values to use unhandled. PR #1675 implements that fallback for Polars Time/Duration and should not be expanded into this larger schema/UI project.

Lenguaje dominante
Jupyter Notebook
Estrellas
2.6k
Forks
221
Merge medio
2 d 1 min
PR fusionados (30 d)
17

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de apache/hamilton

Todos los issues de apache/hamilton

Issues similares

Más issues de Backend & API Design

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.