UHUGEINT values>= 2^127 return negative w/ to_arrow_table

Open
#568 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
55/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
python
Domain
databases

Research direction

Start with the supplied Python reproduction script and trace the to_arrow_table() entry point for UHUGEINT values. Verify the Arrow schema and round-trip behavior for values at or above 2^127, including the maximum value; done means they no longer return as negative and are represented with sufficient precision and signedness.

Written by the indexing model from the issue text.

Description

needs triage
What happens?

When a UHUGEINT column is exported to Arrow, values >= 2^127 come back negative, yet fetchall returns the correct value.

UHUGEINT is exported as decimal128(38, 0), and that schema misdescribes both the data's signedness and precision.

pyarrow properly raises an error in the same case.

See the Decimal('-1') in the output below for the error:

============ SELECT ~(0::UHUGEINT) AS v ==========
con.execute(sql).fetchall()=[(340282366920938463463374607431768211455,)]
con.sql(sql).to_arrow_table().to_pylist()=[{'v': Decimal('-1')}]
============ SELECT 340282366920938463463374607431768211455 AS v ==========
con.execute(sql).fetchall()=[(340282366920938463463374607431768211455,)]
con.sql(sql).to_arrow_table().to_pylist()=[{'v': Decimal('-1')}]
============ pyarrow representability ==========
val=340282366920938463463374607431768211455
decimal128(38, 0) -> ArrowInvalid: Decimal type with precision 39 does not fit into precision inferred from first array element: 38
pa.array([decimal.Decimal(val)], pa.decimal256(39, 0)).to_pylist()=[Decimal('340282366920938463463374607431768211455')]

To Reproduce
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.12"
# dependencies = ["duckdb==1.5.5", "pyarrow>=22"]
# ///
import decimal

import duckdb
import pyarrow as pa

con = duckdb.connect()
sql = "SELECT ~(0::UHUGEINT) AS v"  # all 128 bits set, i.e. UHUGEINT max

# Test 1
print("============", sql, "==========")
print(f"{con.execute(sql).fetchall()=}")
print(f"{con.sql(sql).to_arrow_table().to_pylist()=}")

# Test 2
val = 2**128-1
sql = f"SELECT {val} AS v"  
print("============", sql, "==========")
print(f"{con.execute(sql).fetchall()=}")
print(f"{con.sql(sql).to_arrow_table().to_pylist()=}")


# Test 3: Arrow can represent this value, just not as decimal128(38, 0)
val = 2**128-1
print("============ pyarrow representability ==========")
print(f"{val=}")
try:
    print(f"{pa.array([decimal.Decimal(val)], pa.decimal128(38, 0))=}")
except Exception as e:
    print(f"decimal128(38, 0) -> {type(e).__name__}: {e}")
print(f"{pa.array([decimal.Decimal(val)], pa.decimal256(39, 0)).to_pylist()=}")
OS:

Linux

DuckDB Package Version:

1.5.5 and 1.6.0.dev341

Python Version:

3.14

Full Name:

Paul T

Affiliation:

Iqmo

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a nightly build

Did you include all relevant data sets for reproducing the issue?

Yes

Did you include all code required to reproduce the issue?
  • Yes, I have
Did you include all relevant configuration to reproduce the issue?
  • Yes, I have
Dominant language
Python
Stars
186
Forks
113
Avg merge
20h 58m
Merged PRs (30d)
11

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from duckdb/duckdb-python

All issues in duckdb/duckdb-python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.