bug: zero max_rows_per_file panics, zero V1 max_rows_per_group writes nothing
Maintainers usually reply within 2 days
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 72/100
Research direction
Start in chunker.rs at the break_stream and chunk_stream paths named in the issue, then trace how Dataset::write and FragmentCreateBuilder::write pass row limits for V1 and V2. Check the existing compaction validation for a precedent. Done means zero limits are rejected clearly by both APIs instead of panicking or reporting a successful empty write; add coverage for the described cases.
Written by the indexing model from the issue text.
Description
Dataset::write and FragmentCreateBuilder accept a zero row limit that the writers cannot honor:
lance.write_dataset(tbl, "/tmp/t", max_rows_per_file=0)
# panics in break_stream (chunker.rs:209), a remainder by zero
lance.write_dataset(tbl, "/tmp/t", data_storage_version="legacy", max_rows_per_group=0)
# returns, and the dataset has 0 rows
With the default V2 format, max_rows_per_file=0 reaches break_stream(data, 0), which takes % max_rows on the first batch. With the legacy format, max_rows_per_group=0 reaches chunk_stream(data, 0), which yields no batches, so the write commits an empty dataset and reports success. max_rows_per_file=0 does the same there, because the write folds it into the group size with min. FragmentCreateBuilder::write panics the same way on V2, where its file limit is min(128 * 1024, max_rows_per_file), and on V1 a zero group size reports Input data was empty.
The only existing check is for compaction's file_row_counts path.
- Dominant language
- Rust
- Stars
- 7.2k
- Forks
- 882
- Avg merge
- 4d 2h
- Merged PRs (30d)
- 241
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- Ships a Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from lance-format/lance
-
feature performance
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
lance-format/lance#10137 ·
Maintainers usually reply within 2 days
-
perf: miniblock binary decode copies the chunk's value bytes instead of slicing themPossibly taken @LuciferYang claimed this 6 days ago. Openperformance
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
lance-format/lance#9928 ·
Maintainers usually reply within 2 days
-
After initialize_mem_wal, every FTS query logs "ERROR Index __lance_mem_wal has no fields"Possibly taken @lance-gatefixer claimed this 17 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
lance-format/lance#9655 ·
Maintainers usually reply within 2 days
-
bug: main does not compile, MinHash calls build_prefilter without the metrics argumentPossibly taken @lance-gatefixer claimed this 25 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
lance-format/lance#9506 · 1 comment ·
Maintainers usually reply within 2 days
-
perf: mapping row offsets rescans the fragment for every offset when deletions are set-backedPossibly taken @LuciferYang claimed this 25 days ago. Openperformance
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
lance-format/lance#9498 ·
Maintainers usually reply within 2 days
All issues in lance-format/lance
Similar issues
-
area:desktop priority:p2 type:bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Difficulty 1/5 1-3 hours Newbie friendliness 88/100
fritzprix/libr-agent#2165 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 69/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
-
area:ecosystem bug
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
Chelis-Lang/chelis#3942 ·
Maintainers usually reply within 1 day