Empty ListVector and LargeListVector can expose offset buffers with writerIndex greater than capacity
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 72/100
Research direction
Start by inspecting setReaderAndWriterIndex() in org.apache.arrow.vector.complex.ListVector and LargeListVector, then trace how their offset buffers are allocated for valueCount == 0. Verify that the exported buffer preserves the leading zero offset and satisfies writerIndex <= capacity for both vector types without shrinking future offset allocation.
Written by the indexing model from the issue text.
Description
Describe the bug, including details regarding any error messages, version, and platform.
ListVector and LargeListVector can expose an invalid offset buffer state when valueCount == 0.
For an empty list vector, the logical offset buffer should still contain the leading offset entry:
ListVector:(valueCount + 1) * 4 == 4bytesLargeListVector:(valueCount + 1) * 8 == 8bytes
However, in the empty-vector path, the offset buffer can have:
readerIndex: 0
writerIndex: 4
capacity: 0
or the equivalent writerIndex: 8, capacity: 0 for LargeListVector.
This violates the normal buffer invariant:
0 <= readerIndex <= writerIndex <= capacity
Downstream consumers that unwrap or serialize the Arrow buffer through Netty can then fail with:
IndexOutOfBoundsException: readerIndex: 0, writerIndex: 4
(expected: 0 <= readerIndex <= writerIndex <= capacity(0))
The issue is that setReaderAndWriterIndex() sets the offset buffer writer index based on valueCount * OFFSET_WIDTH, which is 0 for empty vectors. But list vectors still require one offset slot even when there are no values.
The same issue applies to both:
org.apache.arrow.vector.complex.ListVectororg.apache.arrow.vector.complex.LargeListVector
Expected behavior
For valueCount == 0, the offset buffer should still have enough capacity and readable bytes for the leading zero offset:
(valueCount + 1) * OFFSET_WIDTH
So:
- empty
ListVectorshould expose at least 4 bytes for offset[0] - empty
LargeListVectorshould expose at least 8 bytes for offset[0]
The first offset value should be zero.
Actual behavior
An empty list vector can expose an offset buffer with a non-zero writer index but zero capacity, causing Netty buffer validation to fail when the buffer is unwrapped or consumed.
Suggested fix
Update ListVector.setReaderAndWriterIndex() and LargeListVector.setReaderAndWriterIndex() so the offset buffer writer index is based on:
(valueCount + 1) * OFFSET_WIDTH
For the valueCount == 0 case, ensure the offset buffer has enough capacity for the leading zero offset before setting the writer index.
Care should be taken not to shrink the vector's future offset allocation size when allocating this empty sentinel offset buffer.
Additional context
This was observed downstream in Dremio after upgrading Arrow Java. The failure occurred while sending a record batch containing an empty list vector, where the send path unwraps Arrow buffers through Netty.
The downstream error was:
SYSTEM ERROR: IndexOutOfBoundsException: readerIndex: 0, writerIndex: 4
(expected: 0 <= readerIndex <= writerIndex <= capacity(0))
This issue is distinct from #1125. That issue involves UnionListReader.setPosition on a post-IPC empty list. This issue is about the offset buffer exported by empty ListVector / LargeListVector instances having an invalid writer-index/capacity relationship.
- Dominant language
- Java
- Stars
- 95
- Forks
- 154
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 11
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/arrow-java
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
apache/arrow-java#1261 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/arrow-java#1236 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/arrow-java#1230 ·
-
Type: bug
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
apache/arrow-java#1205 ·
-
Type: bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
apache/arrow-java#1196 · 1 comment ·
All issues in apache/arrow-java
Similar issues
-
bug untriaged
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
opensearch-project/ml-commons#5094 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
-
emitter:client:csharp feature
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
affects/8.10 affects/8.9 component/clients kind/bug likelihood/mid severity/mid
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Two open-case totals on one screen: the Programs tile says 15,858 and the nav badge says 15,868 Openbug frontend maui-pilot
Difficulty 2/5 1-3 hours Newbie friendliness 72/100