UHUGEINT values>= 2^127 return negative w/ to_arrow_table
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 55/100
Research direction
Start with the supplied Python reproduction script and trace the to_arrow_table() entry point for UHUGEINT values. Verify the Arrow schema and round-trip behavior for values at or above 2^127, including the maximum value; done means they no longer return as negative and are represented with sufficient precision and signedness.
Written by the indexing model from the issue text.
Description
What happens?
When a UHUGEINT column is exported to Arrow, values >= 2^127 come back negative, yet fetchall returns the correct value.
UHUGEINT is exported as decimal128(38, 0), and that schema misdescribes both the data's signedness and precision.
pyarrow properly raises an error in the same case.
See the Decimal('-1') in the output below for the error:
============ SELECT ~(0::UHUGEINT) AS v ==========
con.execute(sql).fetchall()=[(340282366920938463463374607431768211455,)]
con.sql(sql).to_arrow_table().to_pylist()=[{'v': Decimal('-1')}]
============ SELECT 340282366920938463463374607431768211455 AS v ==========
con.execute(sql).fetchall()=[(340282366920938463463374607431768211455,)]
con.sql(sql).to_arrow_table().to_pylist()=[{'v': Decimal('-1')}]
============ pyarrow representability ==========
val=340282366920938463463374607431768211455
decimal128(38, 0) -> ArrowInvalid: Decimal type with precision 39 does not fit into precision inferred from first array element: 38
pa.array([decimal.Decimal(val)], pa.decimal256(39, 0)).to_pylist()=[Decimal('340282366920938463463374607431768211455')]
To Reproduce
#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.12"
# dependencies = ["duckdb==1.5.5", "pyarrow>=22"]
# ///
import decimal
import duckdb
import pyarrow as pa
con = duckdb.connect()
sql = "SELECT ~(0::UHUGEINT) AS v" # all 128 bits set, i.e. UHUGEINT max
# Test 1
print("============", sql, "==========")
print(f"{con.execute(sql).fetchall()=}")
print(f"{con.sql(sql).to_arrow_table().to_pylist()=}")
# Test 2
val = 2**128-1
sql = f"SELECT {val} AS v"
print("============", sql, "==========")
print(f"{con.execute(sql).fetchall()=}")
print(f"{con.sql(sql).to_arrow_table().to_pylist()=}")
# Test 3: Arrow can represent this value, just not as decimal128(38, 0)
val = 2**128-1
print("============ pyarrow representability ==========")
print(f"{val=}")
try:
print(f"{pa.array([decimal.Decimal(val)], pa.decimal128(38, 0))=}")
except Exception as e:
print(f"decimal128(38, 0) -> {type(e).__name__}: {e}")
print(f"{pa.array([decimal.Decimal(val)], pa.decimal256(39, 0)).to_pylist()=}")
OS:
Linux
DuckDB Package Version:
1.5.5 and 1.6.0.dev341
Python Version:
3.14
Full Name:
Paul T
Affiliation:
Iqmo
What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.
I have tested with a nightly build
Did you include all relevant data sets for reproducing the issue?
Yes
Did you include all code required to reproduce the issue?
- Yes, I have
Did you include all relevant configuration to reproduce the issue?
- Yes, I have
- Dominant language
- Python
- Stars
- 186
- Forks
- 113
- Avg merge
- 20h 58m
- Merged PRs (30d)
- 11
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from duckdb/duckdb-python
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
duckdb/duckdb-python#627 ·
-
needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
duckdb/duckdb-python#576 · 3 comments ·
-
needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
duckdb/duckdb-python#534 ·
-
needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
duckdb/duckdb-python#386 ·
-
needs triage
Difficulty 4/5 3-5 days Newbie friendliness 48/100
duckdb/duckdb-python#622 ·
All issues in duckdb/duckdb-python
Similar issues
-
documentation help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
simonw/sqlite-utils#872 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100