VectorAppender fails with OversizedAllocationException when appending to empty UnionVector

Open
#812 0 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
55/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Stale
Tech stack
java
Domain
data

Research direction

Start with the VectorAppender.java loop around line 556 and inspect UnionVector.getValueCapacity(), including NonNullableStructVector.java around line 373. Run the provided testAppendEmptyTargetUnionVector reproduction and compare capacity before and after reAlloc(). Done means appending to an empty UnionVector completes without OversizedAllocationException and preserves all five float values.

Written by the indexing model from the issue text.

Description

Type: bug
Describe the bug, including details regarding any error messages, version, and platform.

Description:
When appending a UnionVector to an empty target UnionVector, this while loop runs infinitely until the exception occurs:

org.apache.arrow.vector.util.OversizedAllocationException: Unable to expand the buffer

It seems to be a issue with the UnionVector.getValueCapacity() method. Logging the capacity before and after calling reAlloc() shows that it's always 0.
Debugging further, it seems like the value capacity of the internalStruct in the UnionVector always returns 0 because it has no children so size is 0.

 @Override
  public int getValueCapacity() {
    return Math.min(getTypeBufferValueCapacity(), internalStruct.getValueCapacity());
  }

Steps to reproduce

  @Test
  public void testAppendEmptyTargetUnionVector() {
    final int length = 5;

    try (final UnionVector target = UnionVector.empty("target", allocator);
         final UnionVector delta = UnionVector.empty("delta", allocator)) {

      // populate the delta vector
      delta.setType(0, Types.MinorType.FLOAT4);
      delta.setType(1, Types.MinorType.FLOAT4);
      delta.setType(2, Types.MinorType.FLOAT4);
      delta.setType(3, Types.MinorType.FLOAT4);
      delta.setType(4, Types.MinorType.FLOAT4);

      Float4Vector deltaFloatVector = delta.getFloat4Vector();
      deltaFloatVector.allocateNew();
      ValueVectorDataPopulator.setVector(deltaFloatVector, 1f, 2f, 3f, 4f, 5f);
      assertEquals(length, deltaFloatVector.getValueCount());
      delta.setValueCount(length);

      VectorAppender appender = new VectorAppender(target);
      delta.accept(appender, null);

      assertEquals(length, target.getValueCount());

      for (int i = 0; i < length; i++) {
        Object floatObj = target.getObject(i);
        assertTrue(floatObj instanceof Float);
        assertEquals(i+1, ((Float) floatObj).intValue());
      }
    }
  }

This issue does not occur if any value is set in the target vector before appending.

Dominant language
Java
Stars
95
Forks
154
Avg merge
2d 10h
Merged PRs (30d)
11

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/arrow-java

All issues in apache/arrow-java

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.