Support encoding times with units finer than the data resolution

Abierto
#11,920 0 comentarios 1 reacción 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
45/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
numpy, pandas, python
Área
backend, data

Línea de trabajo

Start with encode_cf_datetime in xarray/coding/times.py and trace how datetime and timedelta encoders convert values to the requested unit, including the cftime fallback. Review the handling of overflow and float dtypes, then inspect the existing encoder tests. Done means finer requested units work where representable, overflow behavior is decided and covered, and datetime and timedelta encode/decode round trips remain exact.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Is your feature request related to a problem?

Since #11907, encoding datetime64 or timedelta64 values with units finer than the resolution of the data raises a ValueError. Before that, every value was silently encoded as NaT. @kmuehlbauer asked me to open an issue to discuss how a follow-up PR could support this case.

import numpy as np
from xarray.coding.times import encode_cf_datetime

times = np.array(["2000-01-02T00:00:01"], dtype="datetime64[s]")
encode_cf_datetime(times, "milliseconds since 2000-01-01")
# ValueError: Times can't be serialized with requested units 'milliseconds since 2000-01-01', because these units are finer than the resolution of the data ('seconds'). ...
Describe the solution you'd like

The first version of #11907 (e2f70ea, 819f73f) converted the time deltas to the finer unit with TimedeltaIndex.as_unit before the division. @spencerkclark wrote that this looks like the right approach for a full solution. Units equal to or coarser than the data took the same path as before.

Points that I think need a decision:

  1. Values that do not fit. The cast overflows when the values do not fit into int64 with the finer units, for example times 500 years from the reference date with nanoseconds units. The first version raised OutOfBoundsDatetime or OutOfBoundsTimedelta, also when a float dtype was requested. Should a float dtype give floats instead?
  2. cftime fallback. In the datetime encoder the overflow error is caught by the except (OutOfBoundsDatetime, OverflowError, ValueError) clause and sent to the cftime fallback, which then fails with KeyError: 'nanoseconds'. The first version raised the error after the try block, the same way #11907 does now.
  3. Tests. The first version tested the six unit pairs (seconds to milliseconds, microseconds and nanoseconds, milliseconds to microseconds and nanoseconds, microseconds to nanoseconds) with three dtypes, for both encoders. Would you prefer a smaller set?
Describe alternatives you've considered

Keep the error from #11907 and leave it to the user to cast the data to a finer resolution before writing.

Additional context

With the conversion applied on top of #11906, I checked an encode and decode round trip for four data resolutions, seven units from days to nanoseconds, three dtypes, and both datetimes and timedeltas, each with one NaT. All 168 combinations came back exact.

Empty and all-NaT arrays already encode today and would not need to change.

I'm happy to open the PR once the approach is agreed.

Lenguaje dominante
Python
Estrellas
4.2k
Forks
1.5k
Merge medio
2 d 21 h
PR fusionados (30 d)
74

Preparar el entorno

Abrir en Codespaces

Inicia el contenedor de desarrollo del proyecto en tu navegador, con tu propia cuenta de GitHub.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de pydata/xarray

Todos los issues de pydata/xarray

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.