Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Standardize temporal column statistics across pandas, Polars, and the UI

Aperta
  • Qualcosa non va?
#1,737 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
5/5
Tempo stimato
Più di una settimana
Idoneità per principianti
32/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
pandas, python, typescript
Ambito
backend-api-design, data-engineering, frontend, testing

Direzione di ricerca

Start by comparing the pandas and Polars statistics modules and their listed tests, then inspect DAGWorksDescribe.tsx and the observability schema or generated types. Run the focused SDK tests and existing frontend tests to establish current payload and rendering behavior. Done means the documented temporal contract is implemented across both producers, schemas, renderers, and cross-backend/frontend tests, including timezone and malformed-value cases.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

core-work enhancement SDK UI

Problem

Hamilton's observability schema currently treats several distinct temporal concepts as if they were JavaScript-compatible datetimes. pandas and Polars consequently produce different payloads for equivalent columns, and some payloads either cannot be serialized or render incorrectly in the UI.

This was identified while reviewing #1675 and Jernej's comment in https://github.com/apache/hamilton/pull/1675#discussion_r3795651892. PR #1675 should remain a focused fix: Polars Time and Duration are routed to the existing unhandled path so they no longer break datetime rendering. This issue tracks the complete cross-backend solution.

Current behavior
Logical type pandas Polars UI consequence
Datetime base_data_type=\"datetime\"; ISO-like statistics base_data_type=\"datetime\"; ISO-like statistics Existing datetime renderer generally works
Date Python date object columns normally become unhandled base_data_type=\"datetime\"; min/max are YYYY-MM-DD new Date(\"YYYY-MM-DD\") is interpreted as UTC and can display the previous day in negative-offset timezones
Time Object columns normally become unhandled unhandled after #1675 No rich statistics, but safe
Duration/timedelta base_data_type=\"datetime\"; min/max/mean/quantiles contain raw pd.Timedelta unhandled after #1675 pandas payload fails ordinary JSON serialization; duration strings are not valid JavaScript Dates anyway

The frontend datetime path in ui/frontend/src/components/dashboard/Runs/Task/result-summaries/DAGWorksDescribe.tsx calls new Date(...) for range, quantile, and histogram values. That operation does not preserve Date semantics and cannot parse Time or Duration values.

The generated frontend datetime type and backend Pydantic datetime model also do not declare all fields that producers currently emit, which prevents static typing from catching contract drift.

Desired solution

Define one explicit temporal statistics contract shared by pandas, Polars, the backend schema, generated frontend types, and UI renderers.

Logical types

Use distinct base_data_type values for:

  1. datetime
  2. date
  3. time
  4. duration

Do not route Date, Time, or Duration through the datetime renderer.

Canonical JSON representations

The precise representation should be documented and applied identically by both producers. A suggested starting point:

  • Datetime: ISO 8601 datetime strings with an explicit, documented timezone policy. Preserve offsets for timezone-aware values.
  • Date: YYYY-MM-DD strings. Treat them as calendar dates in the UI; do not pass them directly to new Date(string).
  • Time: ISO local-time strings such as HH:mm:ss[.ffffff], with an explicit policy for timezone-aware time values if supported.
  • Duration: a numeric canonical unit, preferably integer microseconds, for min/max/mean/quantiles and charting. Avoid display-formatted strings as the transport representation. If ISO 8601 durations are selected instead, the UI must parse them explicitly and tests must cover ordering/charting.

All emitted statistics—including min, max, mean, standard deviation, quantiles, and histogram boundaries—must be JSON-safe and use the canonical representation for their logical type.

A schema-version bump may be required if adding base_data_type values is not compatible with the current dagworks_describe v0.0.3 contract.

UI behavior

Add explicit dispatch and rendering for each type:

  • Datetime: retain the existing renderer after confirming timezone behavior.
  • Date: render calendar dates without UTC conversion or timezone shifting. Construct from components or display the canonical value through a date-aware formatter.
  • Time: render wall-clock values without inventing a date or applying a browser timezone.
  • Duration: format canonical numeric values into readable durations and chart them numerically.

Unsupported or malformed values should degrade to the existing unhandled presentation rather than producing Invalid Date or NaN.

Schema and generated types

Update the backend observability models and generated frontend API types so each temporal statistics shape declares the fields actually emitted and consumed. Avoid casting datetime records to numeric statistics types in the frontend.

Acceptance criteria

  • Equivalent pandas and Polars columns produce the same logical base_data_type and value shape for Datetime, Date, Time, and Duration.
  • Every pandas and Polars temporal payload passes JSON serialization without custom fallback encoders.
  • Datetime timezone/offset behavior is documented and tested.
  • Date values display the same calendar day in positive and negative UTC-offset browser timezones.
  • Time values display without an invented date or timezone conversion.
  • Duration values render and sort/chart numerically with documented precision and unit.
  • The frontend never invokes the datetime renderer for Date, Time, or Duration.
  • Malformed/unsupported temporal values degrade safely instead of rendering Invalid Date/NaN.
  • Backend and generated frontend types match the actual payload fields.
  • Cross-backend contract tests cover min, max, mean, std, quantiles, histogram boundaries where applicable, missing values, and empty/all-null columns.
  • Frontend tests run under at least one negative-offset timezone (for example America/Los_Angeles) and verify Date behavior.
  • Upgrade/schema compatibility is documented, including how older SDK payloads are rendered.

Suggested implementation areas

  • pandas routing and serialization:
    • ui/sdk/src/hamilton_sdk/tracking/pandas_stats.py
    • ui/sdk/src/hamilton_sdk/tracking/pandas_col_stats.py
  • Polars routing and serialization:
    • ui/sdk/src/hamilton_sdk/tracking/polars_stats.py
    • ui/sdk/src/hamilton_sdk/tracking/polars_col_stats.py
  • Observability schema/models and generated API types
  • UI grouping and renderers:
    • ui/frontend/src/components/dashboard/Runs/Task/result-summaries/DAGWorksDescribe.tsx
  • Tests:
    • ui/sdk/tests/tracking/test_pandas_stats.py
    • ui/sdk/tests/tracking/test_pandas_col_stats.py
    • ui/sdk/tests/tracking/test_polars_stats.py
    • ui/sdk/tests/tracking/test_polars_col_stats.py
    • focused frontend temporal renderer/contract tests

Incremental path

Until this contract and its renderers exist, the safe schema-v0.0.3 fallback is for only true Datetime columns to use the datetime path and for unsupported Date/Time/Duration values to use unhandled. PR #1675 implements that fallback for Polars Time/Duration and should not be expanded into this larger schema/UI project.

Lingua principale
Jupyter Notebook
Stelle
2.6k
Fork
221
Merge medio
2g 1m
PR unite (30g)
17

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/hamilton

Tutte le issue di apache/hamilton

Issue simili

Altre issue su Backend & API Design

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.