Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Detail Bug] Usage ingestion publishes table usage and join dates one day earlier on UTC-negative host timezones

Abierto Apto para principiantes
  • ¿Algo no está bien?
#36,604 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

@detail-app ya está trabajando en esto.

Desde el 30/10/2026.

  • #36623 de @detail-app — abierto

Evaluación

Dificultad
2/5
Tiempo estimado
1-3 horas
Aptitud para principiantes
72/100
Tipo de issue
Error
Claridad
Bien especificado
Estado de actividad
Estancado
Stack tecnológico
python
Área
backend, data

Línea de trabajo

Comienza por ingestion/src/metadata/ingestion/bulksink/metadata_usage.py, donde se decodifican UsageRequest.date y TableJoins.startDate, y compara los helpers de conversión de ingestion/src/metadata/utils/time_utils.py. Agrega o actualiza las aserciones en ingestion/tests/unit/bulksink/test_metadata_usage.py para verificar que las fechas sigan siendo correctas con un TZ negativo respecto a UTC. El trabajo se considera terminado cuando ambas fechas publicadas conserven el día calendario UTC del parser de consultas y las pruebas específicas pasen.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Detail Bug Report

https://app.detail.dev/org_3377c26d-da48-4ccd-b83a-22c542f4fe83/bugs/bug_f7dcffa8-1094-41f0-8c79-c59da9727e59

Introduced in #17981 by @sushi30 on Oct 10, 2024

Summary

  • Context: The usage workflow encodes each query's analysis date as a UTC-midnight epoch-millisecond string in the query parser (ingestion/src/metadata/ingestion/processor/query_parser.py, via datetime_to_timestamp from ingestion/src/metadata/utils/time_utils.py), stages it in files, and the usage bulk sink (ingestion/src/metadata/ingestion/bulksink/metadata_usage.py) decodes it back to a calendar date when publishing UsageRequest (table usage) and TableJoins.startDate (frequent joins) to the OpenMetadata server.
  • Bug: The bulk sink decodes the UTC-encoded epoch with datetime.fromtimestamp() without a tz argument, which interprets the timestamp in the host's local timezone, while the producer attached UTC.
  • Actual vs. expected: On a host running a UTC-negative timezone (e.g. America/New_York), UTC midnight renders as 19:00 of the previous day, so usage counted for 2025-06-05 is published for 2025-06-04; the published date should be the same day that the query parser and stage files were keyed on.
  • Impact: Table usage statistics and column-join information are systematically attributed to the wrong calendar day for every deployment whose ingestion host runs a UTC-offset timezone — silently corrupting usage analytics, dashboards, and percentile/tier computations built on those dates.

Code with Bug

table_usage_request = UsageRequest(
    date=datetime.fromtimestamp(convert_timestamp(value_dict["usage_date"])).strftime("%Y-%m-%d"),  # <-- BUG 🔴 naive fromtimestamp decodes the UTC epoch in host-local time
    count=value_dict["usage_count"],
)

# TODO: Clean up how we are passing dates from query parsing to here to use timestamps instead of strings
start_date = datetime.fromtimestamp(int(table_usage.date) / 1000)  # <-- BUG 🔴 same host-local decode of a UTC epoch for the joins startDate

Explanation

The query parser encodes the analysis date as an epoch derived from a UTC-midnight datetime. The sink later decodes that epoch using datetime.fromtimestamp(...) with no timezone, so Python interprets the epoch in the machine’s local timezone.

On UTC-negative timezones (e.g. America/New_York), “UTC midnight” becomes the prior local evening, and formatting the decoded datetime as YYYY-MM-DD produces the previous day. The OpenMetadata server stores the date values as provided, so the off-by-one day error persists into stored usage/join records.

Codebase Inconsistency

ingestion/src/metadata/utils/time_utils.py already provides tested UTC-safe helpers (utc_from_timestamp(), timestamp_to_datetime(..., timezone_str="UTC")), and utc_from_timestamp is used elsewhere with the same convert_timestamp(...) pattern; only this sink decodes via bare datetime.fromtimestamp.

Recommended Fix

Decode the epoch as UTC in both sink locations using the existing helpers:

from metadata.utils.time_utils import convert_timestamp, utc_from_timestamp

date=utc_from_timestamp(convert_timestamp(value_dict["usage_date"])).strftime("%Y-%m-%d"),  # <-- FIX 🟢
...
start_date = utc_from_timestamp(int(table_usage.date) / 1000)  # <-- FIX 🟢

Also add assertions in ingestion/tests/unit/bulksink/test_metadata_usage.py that pin UsageRequest.date and TableJoins.startDate under a UTC-negative TZ.

History

This bug was introduced in commit 68e71cb3dc (PR #17981, "GEN-970: Refactor redshift system metrics to support freshness test", Oct 10 2024). That commit updated datetime_to_timestamp to attach UTC to naive datetimes, making the producer emit UTC-midnight epochs while the sink continued to decode epochs in host-local time via datetime.fromtimestamp(...), creating an asymmetric round-trip that shifts dates back one day on UTC-negative hosts.

Lenguaje dominante
TypeScript
Estrellas
15.4k
Forks
2.4k
Merge medio
23 h 39 min
PR fusionados (30 d)
663

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de open-metadata/OpenMetadata

Todos los issues de open-metadata/OpenMetadata

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.