Bug: `ValueWriters.FixedByteBufferWriter` uses `writeBytes()` instead of `writeFixed()` for Avro `fixed[N]` fields

Open Beginner friendly
#17,501 12 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
86/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
java
Domain
databases

Research direction

Start in core/src/main/java/org/apache/iceberg/avro/ValueWriters.java at the FixedByteBufferWriter inner class, comparing it with FixedWriter. Reproduce the manifest write/read case described for a FIXED(3) identity partition, then add a regression test that preserves the partition bytes and record_count. Done means fixed fields are encoded without a length prefix and the manifest round trip retains both values.

Written by the indexing model from the issue text.

Description

bug
Apache Iceberg version

1.11.0 (latest release)

Query engine

Snowflake

Please describe the bug 🐞

Bug description

ValueWriters.FixedByteBufferWriter.write(ByteBuffer, Encoder) calls encoder.writeBytes(bytes)
instead of encoder.writeFixed(arr, 0, length). For Avro fixed[N] fields this is wrong:
writeBytes prepends a zigzag-encoded length (e.g. 0x06 for a 3-byte value) before the data,
but the Avro reader consumes exactly N bytes for a fixed field and has no length prefix to skip.
The stray prefix byte spills into the next field in the record.

Affected version: 1.11.0 (also present in prior versions)

Where the bug is

core/src/main/java/org/apache/iceberg/avro/ValueWriters.java, inner class FixedByteBufferWriter:

private static class FixedByteBufferWriter implements ValueWriter<ByteBuffer> {
  private final int length;

  @Override
  public void write(ByteBuffer bytes, Encoder encoder) throws IOException {
    Preconditions.checkArgument(bytes.remaining() == length, ...);
    encoder.writeBytes(bytes);   // ← BUG: should be writeFixed
  }
}

The sibling class FixedWriter (which handles byte[]) already does it correctly:

private static class FixedWriter implements ValueWriter<byte[]> {
  @Override
  public void write(byte[] bytes, Encoder encoder) throws IOException {
    Preconditions.checkArgument(bytes.length == length, ...);
    encoder.writeFixed(bytes);   // ← correct
  }
}

Concrete corruption example

Writing a FIXED(3) partition value [AB CD EF] via FixedByteBufferWriter:

Encoding Output bytes What the reader sees
writeBytes (current) 06 AB CD EF fixed[3] consumes 06 AB CD; EF spills into the next field
writeFixed (correct) AB CD EF fixed[3] consumes AB CD EF

When the spill byte EF prefixes the subsequent record_count varint 08 (zigzag for 4 rows),
the combined varint EF 08 decodes to raw value 1135, which zigzag-decodes to -568. Every
reader (Java, native, Spark) trusts this value from the manifest and sees record_count = -568
for a file that has 4 rows.

How to reproduce

  1. Create a partitioned Iceberg table with a FIXED(3) identity-partition column.
  2. Write a data file via the Iceberg Java writer (triggers InternalWriterFixedByteBufferWriter).
  3. Read the written manifest Avro directly and inspect the data_file.record_count field for the
    partition entry — it will be a large negative number instead of the actual row count.
  4. Alternatively: observe that data_file.partition.val8 in the manifest contains 06abcd instead
    of abcdef for a [AB CD EF] partition value.

Proposed fix

In FixedByteBufferWriter.write(), extract the bytes from the ByteBuffer and call
encoder.writeFixed():

@Override
public void write(ByteBuffer bytes, Encoder encoder) throws IOException {
  Preconditions.checkArgument(bytes.remaining() == length,
      "Cannot write byte buffer of length %s as fixed[%s]", bytes.remaining(), length);
  byte[] arr = new byte[length];
  bytes.duplicate().get(arr);
  encoder.writeFixed(arr);
}

A regression test should assert that a round-trip write/read of a manifest entry with a FIXED(N)
identity-partition column preserves the exact partition value bytes and the correct record_count.

Willingness to contribute
  • I can contribute a fix for this bug independently
  • I would be willing to contribute a fix for this bug with guidance from the Iceberg community
  • I cannot contribute a fix for this bug at this time
Dominant language
Java
Stars
9.3k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
143

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from apache/iceberg

All issues in apache/iceberg

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.