VectorAppender fails with OversizedAllocationException when appending to empty UnionVector
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 55/100
Research direction
Start with the VectorAppender.java loop around line 556 and inspect UnionVector.getValueCapacity(), including NonNullableStructVector.java around line 373. Run the provided testAppendEmptyTargetUnionVector reproduction and compare capacity before and after reAlloc(). Done means appending to an empty UnionVector completes without OversizedAllocationException and preserves all five float values.
Written by the indexing model from the issue text.
Description
Describe the bug, including details regarding any error messages, version, and platform.
Description:
When appending a UnionVector to an empty target UnionVector, this while loop runs infinitely until the exception occurs:
org.apache.arrow.vector.util.OversizedAllocationException: Unable to expand the buffer
It seems to be a issue with the UnionVector.getValueCapacity() method. Logging the capacity before and after calling reAlloc() shows that it's always 0.
Debugging further, it seems like the value capacity of the internalStruct in the UnionVector always returns 0 because it has no children so size is 0.
@Override
public int getValueCapacity() {
return Math.min(getTypeBufferValueCapacity(), internalStruct.getValueCapacity());
}
Steps to reproduce
@Test
public void testAppendEmptyTargetUnionVector() {
final int length = 5;
try (final UnionVector target = UnionVector.empty("target", allocator);
final UnionVector delta = UnionVector.empty("delta", allocator)) {
// populate the delta vector
delta.setType(0, Types.MinorType.FLOAT4);
delta.setType(1, Types.MinorType.FLOAT4);
delta.setType(2, Types.MinorType.FLOAT4);
delta.setType(3, Types.MinorType.FLOAT4);
delta.setType(4, Types.MinorType.FLOAT4);
Float4Vector deltaFloatVector = delta.getFloat4Vector();
deltaFloatVector.allocateNew();
ValueVectorDataPopulator.setVector(deltaFloatVector, 1f, 2f, 3f, 4f, 5f);
assertEquals(length, deltaFloatVector.getValueCount());
delta.setValueCount(length);
VectorAppender appender = new VectorAppender(target);
delta.accept(appender, null);
assertEquals(length, target.getValueCount());
for (int i = 0; i < length; i++) {
Object floatObj = target.getObject(i);
assertTrue(floatObj instanceof Float);
assertEquals(i+1, ((Float) floatObj).intValue());
}
}
}
This issue does not occur if any value is set in the target vector before appending.
- Dominant language
- Java
- Stars
- 95
- Forks
- 154
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 11
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from apache/arrow-java
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
apache/arrow-java#1261 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/arrow-java#1236 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
apache/arrow-java#1230 ·
-
Type: bug
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
apache/arrow-java#1205 ·
-
Type: bug
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
apache/arrow-java#1196 · 1 comment ·
All issues in apache/arrow-java
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
-
Two open-case totals on one screen: the Programs tile says 15,858 and the nav badge says 15,868 Openbug frontend maui-pilot
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
objectionary/eo-graphs#74 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100