Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

Big-endian compatibility failures in bucket hashing and Avro/deletion-vector serialization

Aberta
  • Algo errado?
#1,000 2 comentários 1 reação 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 2 dias

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
4/5
Tempo estimado
3-5 dias
Facilidade para iniciantes
35/100
Tipo de issue
Bug
Clareza
Claramente especificada
Status de atividade
Ativa
Stack de tecnologia
cpp
Domínio
backend, databases, testing

Direção de pesquisa

Start with the linked MurmurHash block loads in src/iceberg/util/murmurhash3_internal.cc and the bucket hash golden vectors; the issue reports native word reads on big-endian systems. Then inspect the two decimal writers, the pinned Avro float/double routines, and the CRoaring serialization calls in src/iceberg/deletes/roaring_position_bitmap.cc. Add cross-endian fixtures for each reported wire format; done means the valid little-endian fixtures match and decode correctly on big-endian systems.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

Summary

Junwang (@zhjwpku) raised the big-endian question in the review of #998, which prompted this investigation.

An endian audit of main at 44e29e109e461e7c9069855c9a90442002580d23 found four compatibility problems on big-endian systems:

  1. Bucket hashes differ from Iceberg's specified values.
  2. Both Avro decimal writers produce incorrect big-endian decimal bytes.
  3. The configured Avro dependency uses native byte order for float/double serialization.
  4. The configured CRoaring dependency's portable serialization is little-endian-only, affecting deletion-vector bitmap payloads.

These are pre-existing findings, independent of #998. Representative failures were reproduced by executing actual s390x big-endian binaries under QEMU. This was a focused reproduction, not a full Iceberg build or integration test.

Verification scope

  • Ubuntu 22.04.5 ARM64 container on a Mac; QEMU 6.2.0; s390x GCC/G++ 11.4.0; statically linked target binaries.
  • ELF headers identify IBM S/390, big-endian, and the harness checks native byte order at runtime. No physical big-endian hardware was used.
  • Native little-endian control tests pass.
  • Murmur: complete, unchanged Iceberg source; blob 14cedabf4ac7b4ece349b52d05e254b86d2a5e8d matches upstream.
  • Decimal: actual Arrow 25.0.0 BasicDecimal128 storage plus the original statements from both Iceberg writers. Full Arrow arrays and Iceberg writers were not instantiated.
  • Avro: four unchanged float/double routine bodies from pinned commit 209a3735ec330679790824da54b7558db55e4e7f, using byte-copy stream substitutes. The full Avro library was not linked.
  • CRoaring: complete, unchanged 4.4.3 library and its original C++ wrapper, with no stubs.

Dependency findings below apply to the configured bundled versions. Custom dependency versions may differ.

1. Bucket hashing reads little-endian input as native words

getblock32 and getblock64 return p[i] without endian conversion. The x86_32 block loop uses these native loads. Although HashLong prepares little-endian input, the block reads interpret it incorrectly on a big-endian host.

Observed signed 32-bit hashes and bucket[100] results:

Input Expected hash Actual s390x hash Expected bucket Actual bucket
int/long 34 2017239379 1403955681 79 81
binary 00 01 02 03 -188683207 749933555 41 55
string iceberg 1210000089 1578596666 89 66

The UUID golden vector also fails; a three-byte tail-only control passes. The failure is consistent with native full-block loads, rather than the byte-based tail handling.

This affects bucket transforms for numeric/date/time/timestamp/UUID inputs and byte/string inputs with full four-byte blocks. The same underlying hash code also affects object-store path hashing, with a lower interoperability impact.

Suggested direction: explicit little-endian, alignment-safe block loads, consistent with Iceberg's Appendix B 32-bit hash definition. Merely converting the input buffer to little-endian does not fix the subsequent native loads.

2. Avro decimal writers unconditionally reverse native bytes

Both direct encoder and data utility writer reverse the native decimal bytes unconditionally.

For a 16-byte decimal with unscaled value 1:

Expected big-endian wire: 00000000000000000000000000000001
Actual s390x wire:       01000000000000000000000000000000
Actual interpreted value: 2^120

On big-endian Arrow storage, the reversal corrupts the intended big-endian wire representation. Suggested direction: reverse only on little-endian hosts, or use an explicit big-endian conversion. The reader-side nanoarrow decimal copy is not the source of this failure.

3. Pinned Avro float/double routines use native wire order

The configured Avro revision's encoder and decoder copy native floating-point bytes, while Avro requires little-endian wire order.

Value Expected wire hex Actual s390x wire hex s390x decoding of valid little-endian wire
float 1.0 0000803f 3f800000 4.6006029882248069e-41
double 1.0 000000000000f03f 3ff0000000000000 3.0386519416174186e-319

Same-host encode/decode round trips pass, hiding the cross-endian incompatibility. Both Iceberg Avro backends use the affected dependency. Suggested direction: fix/update the dependency or explicitly reject unsupported big-endian configurations until this is resolved.

4. CRoaring portable serialization is incompatible on big-endian hosts

Iceberg writes and reads the inner bitmap through CRoaring's portable serialization APIs.

With the complete CRoaring 4.4.3 library and bitmap {1}, the s390x execution:

  • emits bytes different from the valid little-endian portable fixture;
  • rejects that valid little-endian fixture when reading;
  • successfully round-trips its own output.

CRoaring's 4.4.3 README explicitly limits serialization support to little-endian systems. Its writer and reader use native byte order.

Iceberg's outer deletion-vector header endian handling is correct; the inner serialized bitmap remains incompatible. Suggested direction: guard big-endian use of this path until the dependency provides compatible serialization.

Regression coverage

Cross-endian golden fixtures are needed in addition to same-host round trips: bucket hash vectors, decimal wire bytes, Avro float/double fixtures, and deletion-vector bitmap fixtures. A full Iceberg integration run on a big-endian target remains untested.

Linguagem predominante
C++
Estrelas
226
Forks
133
Merge médio
6d 5h
PRs com merge (30d)
40

Preparar o ambiente

Este projeto não oferece contêiner de desenvolvimento, Dockerfile nem guia de contribuição, então a configuração fica por sua conta: comece pelo README e veja nosso guia da primeira contribuição para os passos gerais.

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de apache/iceberg-cpp

Todas as issues de apache/iceberg-cpp

Issues semelhantes

Mais issues de C++

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.