[Bug] range-bitmap file index: IS NULL returns only the first and last row when a column is entirely NULL in a file
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 88/100
Research direction
Start in org.apache.paimon.fileindex.rangebitmap.RangeBitmap#isNull and inspect the cardinality <= 0 branch. Run the provided Java API reproduction with five entirely NULL values; done means c IS NULL returns all five rows, with regression coverage for the range-bitmap case.
Written by the indexing model from the issue text.
Description
Search before asking
- I searched in the issues and found nothing similar.
Paimon version
2.0.0 (paimon-bundle-2.0.0)
Compute Engine
None — reproduced with the Java API (ReadBuilder.withFilter), local filesystem catalog, Java 21.
Minimal reproduce step
Create an append table with a range-bitmap file index on c, write 5 rows where c is NULL in every row, then read with c IS NULL:
Identifier id = Identifier.create("db", "t_isnull");
catalog.createTable(
id,
Schema.newBuilder()
.column("id", DataTypes.INT())
.column("c", DataTypes.INT())
.option("file-index.range-bitmap.columns", "c")
.build(),
false);
Table table = catalog.getTable(id);
List<GenericRow> rows = new ArrayList<>();
for (int i = 0; i < 5; i++) {
rows.add(GenericRow.of(i, null));
}
write(table, rows);
Predicate isNull = new PredicateBuilder(table.rowType()).isNull(1);
System.out.println(count(table, isNull)); // prints 2
System.out.println(count(table, null)); // prints 5
Helper methods:
static void write(Table table, List<GenericRow> rows) throws Exception {
BatchWriteBuilder builder = table.newBatchWriteBuilder();
try (BatchTableWrite write = builder.newWrite();
BatchTableCommit commit = builder.newCommit()) {
for (GenericRow row : rows) write.write(row);
commit.commit(write.prepareCommit());
}
}
static long count(Table table, Predicate filter) throws Exception {
ReadBuilder builder = table.newReadBuilder();
if (filter != null) builder = builder.withFilter(filter);
long[] count = new long[1];
try (RecordReader<InternalRow> reader =
builder.newRead().createReader(builder.newScan().plan())) {
reader.forEachRemaining(row -> count[0]++);
}
return count[0];
}
The same table with file-index.bitmap.columns or file-index.bsi.columns instead of range-bitmap returns 5.
What doesn't meet your expectations?
c IS NULL returns 2 rows; all 5 rows match. The rows returned are the first and the last row of the file; every row in between is dropped.
The same table with file-index.bitmap.columns or file-index.bsi.columns instead of range-bitmap returns 5.
Anything else?
Root cause is in org.apache.paimon.fileindex.rangebitmap.RangeBitmap#isNull. When the column has no non-null value in the file (cardinality <= 0), it returns:
RoaringBitmap32.bitmapOf(0, rid - 1)
bitmapOf(int...) takes a list of positions, so this is the two-element set {0, rid - 1}, not the range [0, rid). Any file with three or more rows where the indexed column is entirely NULL loses its middle rows for IS NULL.
Suggested fix: return the full range, e.g. RoaringBitmap32.bitmapOfRange(0, rid). isNotNull on the same reader is correct (it returns an empty bitmap for such a file), so the complement of isNotNull over [0, rid) gives the right answer too.
Are you willing to submit a PR?
- I'm willing to submit a PR!
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 365
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/paimon
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day
-
[Bug] Hive query on Paimon table fails with 'Proxy user is not supported' when Kerberos proxy user is usedPossibly taken @sohurdc claimed this 88 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/paimon#9006 · 1 comment ·
Maintainers usually reply within 1 day
-
[Bug] FileIndexFormat.Reader leaks input stream when construction fails with a runtime exceptionMay be free again A pull request for this issue was closed without being merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
Maintainers usually reply within 1 day
-
[Bug] Bitmap benchmarks do not close seekable input streamsMay be free again A pull request for this issue was closed without being merged. Open
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
floci-io/floci#8791 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 75/100
quarkusio/quarkus-insights#336 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 77/100
Maintainers usually reply within 1 day
-
affects/8.10 affects/8.8 affects/8.9 area/reliability component/zeebe kind/bug likelihood/low severity/high
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day