query_pandas() fails with HOUR_TIMESTAMP dimension due to timezone mismatch

Open Beginner friendly
#12 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
74/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Quiet
Tech stack
pandas, python
Domain
api, data

Research direction

Start in src/shopifyql/results.py at ShopifyQLPandasResult.TYPES_MAP and from_table_data, especially lines 50-53 cited in the issue. Reproduce query_pandas() with an HOUR_TIMESTAMP dimension and inspect how sub-day timestamp strings are cast. Done means the query completes without the timezone ValueError and the returned timestamp columns retain a consistent timezone-aware representation.

Written by the indexing model from the issue text.

Description

Issue

client.query_pandas() raises a ValueError when the query groups by hour (or any sub-day timeseries dimension).

The library's ShopifyQLPandasResult.TYPES_MAP maps HOUR_TIMESTAMP → "datetime64[ns]" (timezone-naive), but the Shopify API returns timezone-aware ISO 8601 strings for that column (e.g. "2026-05-01T13:00:00+00:00"). When df.astype(data_types) is called inside ShopifyQLPandasResult.from_table_data, pandas refuses to cast a tz-aware string into a tz-naive dtype.

Swapping the dimension to day works without error, because DAY_TIMESTAMP values appear to be returned as date-only strings with no timezone offset.

Error

ValueError: cannot supply both a tz and a timezone-naive dtype (i.e. datetime64[ns]):
Error while type casting for column 'hour'
Root cause (results.py:50-53)

All *_TIMESTAMP types in TYPES_MAP are mapped to "datetime64[ns]". Sub-day granularities (HOUR_TIMESTAMP, MINUTE_TIMESTAMP, SECOND_TIMESTAMP, TIMESTAMP) include a UTC offset in the API response, making the cast incompatible with a tz-naive dtype.

Expected behavior

query_pandas() should handle tz-aware timestamp strings returned natively by the API

Workaround

  1. Override problematic types inline:
_SUB_DAY_TS = {'TIMESTAMP', 'SECOND_TIMESTAMP', 'MINUTE_TIMESTAMP', 'HOUR_TIMESTAMP'}
for _k in _SUB_DAY_TS:
    ShopifyQLPandasResult.TYPES_MAP[_k] = pd.DatetimeTZDtype(tz='UTC')
  1. Subclass ShopifyQLPandasResult to override the from_table_data class method.
    Generally:
class _FixedPandasResult(ShopifyQLPandasResult):
    @classmethod
    def from_table_data(cls, table_data):
        data_types = cls._pandas_dtypes_from_columns(table_data["columns"])
        column_names = [str(c.get("name", "")) for c in table_data["columns"]]
        df = pd.DataFrame(table_data["rows"], columns=column_names)

        ts_cols = {col for col, dtype in data_types.items() if dtype == "datetime64[ns]"}
        df = df.astype({col: dtype for col, dtype in data_types.items() if col not in ts_cols})
        for col in ts_cols:
            df[col] = pd.to_datetime(df[col], utc=True).dt.tz_convert("America/Regina").dt.tz_localize(None)

        return df
Dominant language
Python
Stars
7
Forks
2
PR merge metrics
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Shopify/shopifyql-py

All issues in Shopify/shopifyql-py

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.