Raw numpy scalar values get silently stored as corrupted BLOBs instead of being converted
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 75/100
Research direction
Look at the jsonify_if_needed() function in db.py to see how values are converted before insertion. The issue is that numpy scalar types like np.int64 are not recognized and get stored as binary. Add handling for numpy scalar types by checking for np.generic or using .item() to extract the Python native value. Test by running the provided repro script and verifying the output is {'col': 5}.
Written by the indexing model from the issue text.
Description
While working on #872, I found that passing raw numpy scalar values (e.g. np.int64(5), not run through pandas or .item()) directly into insert_all() doesn't get converted to a normal SQLite INTEGER/REAL — instead it silently stores garbage binary data as a BLOB. Repro:
python
import sqlite_utils, numpy as np
db = sqlite_utils.Database(memory=True)
db["t"].insert_all([{"col": np.int64(5)}])
print(list(db["t"].rows))
{'col': b'\x05\x00\x00\x00\x00\x00\x00\x00'} instead of {'col': 5}
This likely affects anyone using numpy arrays directly without pandas. Might be worth adding numpy-scalar handling to jsonify_if_needed() in db.py.
- Dominant language
- Python
- Stars
- 2.2k
- Forks
- 172
- Avg merge
- 9m
- Merged PRs (30d)
- 1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from simonw/sqlite-utils
-
Difficulty 2/5 1-3 hours Newbie friendliness 90/100
simonw/sqlite-utils#872 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
simonw/sqlite-utils#857 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
simonw/sqlite-utils#849 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
simonw/sqlite-utils#588 · 4 comments · 1 reaction ·
-
bug documentation help wanted
Difficulty 1/5 Under an hour Newbie friendliness 88/100
simonw/sqlite-utils#493 · 5 comments ·
All issues in simonw/sqlite-utils
Similar issues
-
agent-ready documentation needs-triage
Difficulty 1/5 1-3 hours Newbie friendliness 88/100
-
documentation
Difficulty 1/5 Under an hour Newbie friendliness 91/100
-
workflow-status page template still says reusable workflows are "triggered only by workflow_call:" Open
Difficulty 1/5 Under an hour Newbie friendliness 92/100
-
instance instance add
Difficulty 1/5 Under an hour Newbie friendliness 72/100
searxng/searx-instances#939 · 1 comment ·
-
area-deployment area-integrations triage:bot-seen
Difficulty 2/5 Half a day Newbie friendliness 86/100