Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Big-endian compatibility failures in bucket hashing and Avro/deletion-vector serialization

Aperta
  • Qualcosa non va?
#1,000 1 commento 1 reazione 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
35/100
Tipo di issue
Bug
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
cpp
Ambito
backend, databases, testing

Direzione di ricerca

Start with the linked MurmurHash block loads in src/iceberg/util/murmurhash3_internal.cc and the bucket hash golden vectors; the issue reports native word reads on big-endian systems. Then inspect the two decimal writers, the pinned Avro float/double routines, and the CRoaring serialization calls in src/iceberg/deletes/roaring_position_bitmap.cc. Add cross-endian fixtures for each reported wire format; done means the valid little-endian fixtures match and decode correctly on big-endian systems.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

Summary

Junwang (@zhjwpku) raised the big-endian question in the review of #998, which prompted this investigation.

An endian audit of main at 44e29e109e461e7c9069855c9a90442002580d23 found four compatibility problems on big-endian systems:

  1. Bucket hashes differ from Iceberg's specified values.
  2. Both Avro decimal writers produce incorrect big-endian decimal bytes.
  3. The configured Avro dependency uses native byte order for float/double serialization.
  4. The configured CRoaring dependency's portable serialization is little-endian-only, affecting deletion-vector bitmap payloads.

These are pre-existing findings, independent of #998. Representative failures were reproduced by executing actual s390x big-endian binaries under QEMU. This was a focused reproduction, not a full Iceberg build or integration test.

Verification scope

  • Ubuntu 22.04.5 ARM64 container on a Mac; QEMU 6.2.0; s390x GCC/G++ 11.4.0; statically linked target binaries.
  • ELF headers identify IBM S/390, big-endian, and the harness checks native byte order at runtime. No physical big-endian hardware was used.
  • Native little-endian control tests pass.
  • Murmur: complete, unchanged Iceberg source; blob 14cedabf4ac7b4ece349b52d05e254b86d2a5e8d matches upstream.
  • Decimal: actual Arrow 25.0.0 BasicDecimal128 storage plus the original statements from both Iceberg writers. Full Arrow arrays and Iceberg writers were not instantiated.
  • Avro: four unchanged float/double routine bodies from pinned commit 209a3735ec330679790824da54b7558db55e4e7f, using byte-copy stream substitutes. The full Avro library was not linked.
  • CRoaring: complete, unchanged 4.4.3 library and its original C++ wrapper, with no stubs.

Dependency findings below apply to the configured bundled versions. Custom dependency versions may differ.

1. Bucket hashing reads little-endian input as native words

getblock32 and getblock64 return p[i] without endian conversion. The x86_32 block loop uses these native loads. Although HashLong prepares little-endian input, the block reads interpret it incorrectly on a big-endian host.

Observed signed 32-bit hashes and bucket[100] results:

Input Expected hash Actual s390x hash Expected bucket Actual bucket
int/long 34 2017239379 1403955681 79 81
binary 00 01 02 03 -188683207 749933555 41 55
string iceberg 1210000089 1578596666 89 66

The UUID golden vector also fails; a three-byte tail-only control passes. The failure is consistent with native full-block loads, rather than the byte-based tail handling.

This affects bucket transforms for numeric/date/time/timestamp/UUID inputs and byte/string inputs with full four-byte blocks. The same underlying hash code also affects object-store path hashing, with a lower interoperability impact.

Suggested direction: explicit little-endian, alignment-safe block loads, consistent with Iceberg's Appendix B 32-bit hash definition. Merely converting the input buffer to little-endian does not fix the subsequent native loads.

2. Avro decimal writers unconditionally reverse native bytes

Both direct encoder and data utility writer reverse the native decimal bytes unconditionally.

For a 16-byte decimal with unscaled value 1:

Expected big-endian wire: 00000000000000000000000000000001
Actual s390x wire:       01000000000000000000000000000000
Actual interpreted value: 2^120

On big-endian Arrow storage, the reversal corrupts the intended big-endian wire representation. Suggested direction: reverse only on little-endian hosts, or use an explicit big-endian conversion. The reader-side nanoarrow decimal copy is not the source of this failure.

3. Pinned Avro float/double routines use native wire order

The configured Avro revision's encoder and decoder copy native floating-point bytes, while Avro requires little-endian wire order.

Value Expected wire hex Actual s390x wire hex s390x decoding of valid little-endian wire
float 1.0 0000803f 3f800000 4.6006029882248069e-41
double 1.0 000000000000f03f 3ff0000000000000 3.0386519416174186e-319

Same-host encode/decode round trips pass, hiding the cross-endian incompatibility. Both Iceberg Avro backends use the affected dependency. Suggested direction: fix/update the dependency or explicitly reject unsupported big-endian configurations until this is resolved.

4. CRoaring portable serialization is incompatible on big-endian hosts

Iceberg writes and reads the inner bitmap through CRoaring's portable serialization APIs.

With the complete CRoaring 4.4.3 library and bitmap {1}, the s390x execution:

  • emits bytes different from the valid little-endian portable fixture;
  • rejects that valid little-endian fixture when reading;
  • successfully round-trips its own output.

CRoaring's 4.4.3 README explicitly limits serialization support to little-endian systems. Its writer and reader use native byte order.

Iceberg's outer deletion-vector header endian handling is correct; the inner serialized bitmap remains incompatible. Suggested direction: guard big-endian use of this path until the dependency provides compatible serialization.

Regression coverage

Cross-endian golden fixtures are needed in addition to same-host round trips: bucket hash vectors, decimal wire bytes, Avro float/double fixtures, and deletion-vector bitmap fixtures. A full Iceberg integration run on a big-endian target remains untested.

Lingua principale
C++
Stelle
226
Fork
133
Merge medio
5g 10h
PR unite (30g)
33

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di apache/iceberg-cpp

Tutte le issue di apache/iceberg-cpp

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.