Hacktoberfest 2026: die Issues, die Maintainer für den Oktober markiert haben – offen und einsteigerfreundlich. Hacktoberfest-Issues durchsuchen

Big-endian compatibility failures in bucket hashing and Avro/deletion-vector serialization

Offen
  • Stimmt etwas nicht?
#1,000 1 Kommentar 1 Reaktion 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 2 Tagen

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Anfängerfreundlichkeit
35/100
Issue-Typ
Bug
Klarheit
Klar beschrieben
Aktivitätsstatus
Aktiv
Tech-Stack
cpp
Bereich
backend, databases, testing

Rechercherichtung

Start with the linked MurmurHash block loads in src/iceberg/util/murmurhash3_internal.cc and the bucket hash golden vectors; the issue reports native word reads on big-endian systems. Then inspect the two decimal writers, the pinned Avro float/double routines, and the CRoaring serialization calls in src/iceberg/deletes/roaring_position_bitmap.cc. Add cross-endian fixtures for each reported wire format; done means the valid little-endian fixtures match and decode correctly on big-endian systems.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Summary

Junwang (@zhjwpku) raised the big-endian question in the review of #998, which prompted this investigation.

An endian audit of main at 44e29e109e461e7c9069855c9a90442002580d23 found four compatibility problems on big-endian systems:

  1. Bucket hashes differ from Iceberg's specified values.
  2. Both Avro decimal writers produce incorrect big-endian decimal bytes.
  3. The configured Avro dependency uses native byte order for float/double serialization.
  4. The configured CRoaring dependency's portable serialization is little-endian-only, affecting deletion-vector bitmap payloads.

These are pre-existing findings, independent of #998. Representative failures were reproduced by executing actual s390x big-endian binaries under QEMU. This was a focused reproduction, not a full Iceberg build or integration test.

Verification scope

  • Ubuntu 22.04.5 ARM64 container on a Mac; QEMU 6.2.0; s390x GCC/G++ 11.4.0; statically linked target binaries.
  • ELF headers identify IBM S/390, big-endian, and the harness checks native byte order at runtime. No physical big-endian hardware was used.
  • Native little-endian control tests pass.
  • Murmur: complete, unchanged Iceberg source; blob 14cedabf4ac7b4ece349b52d05e254b86d2a5e8d matches upstream.
  • Decimal: actual Arrow 25.0.0 BasicDecimal128 storage plus the original statements from both Iceberg writers. Full Arrow arrays and Iceberg writers were not instantiated.
  • Avro: four unchanged float/double routine bodies from pinned commit 209a3735ec330679790824da54b7558db55e4e7f, using byte-copy stream substitutes. The full Avro library was not linked.
  • CRoaring: complete, unchanged 4.4.3 library and its original C++ wrapper, with no stubs.

Dependency findings below apply to the configured bundled versions. Custom dependency versions may differ.

1. Bucket hashing reads little-endian input as native words

getblock32 and getblock64 return p[i] without endian conversion. The x86_32 block loop uses these native loads. Although HashLong prepares little-endian input, the block reads interpret it incorrectly on a big-endian host.

Observed signed 32-bit hashes and bucket[100] results:

Input Expected hash Actual s390x hash Expected bucket Actual bucket
int/long 34 2017239379 1403955681 79 81
binary 00 01 02 03 -188683207 749933555 41 55
string iceberg 1210000089 1578596666 89 66

The UUID golden vector also fails; a three-byte tail-only control passes. The failure is consistent with native full-block loads, rather than the byte-based tail handling.

This affects bucket transforms for numeric/date/time/timestamp/UUID inputs and byte/string inputs with full four-byte blocks. The same underlying hash code also affects object-store path hashing, with a lower interoperability impact.

Suggested direction: explicit little-endian, alignment-safe block loads, consistent with Iceberg's Appendix B 32-bit hash definition. Merely converting the input buffer to little-endian does not fix the subsequent native loads.

2. Avro decimal writers unconditionally reverse native bytes

Both direct encoder and data utility writer reverse the native decimal bytes unconditionally.

For a 16-byte decimal with unscaled value 1:

Expected big-endian wire: 00000000000000000000000000000001
Actual s390x wire:       01000000000000000000000000000000
Actual interpreted value: 2^120

On big-endian Arrow storage, the reversal corrupts the intended big-endian wire representation. Suggested direction: reverse only on little-endian hosts, or use an explicit big-endian conversion. The reader-side nanoarrow decimal copy is not the source of this failure.

3. Pinned Avro float/double routines use native wire order

The configured Avro revision's encoder and decoder copy native floating-point bytes, while Avro requires little-endian wire order.

Value Expected wire hex Actual s390x wire hex s390x decoding of valid little-endian wire
float 1.0 0000803f 3f800000 4.6006029882248069e-41
double 1.0 000000000000f03f 3ff0000000000000 3.0386519416174186e-319

Same-host encode/decode round trips pass, hiding the cross-endian incompatibility. Both Iceberg Avro backends use the affected dependency. Suggested direction: fix/update the dependency or explicitly reject unsupported big-endian configurations until this is resolved.

4. CRoaring portable serialization is incompatible on big-endian hosts

Iceberg writes and reads the inner bitmap through CRoaring's portable serialization APIs.

With the complete CRoaring 4.4.3 library and bitmap {1}, the s390x execution:

  • emits bytes different from the valid little-endian portable fixture;
  • rejects that valid little-endian fixture when reading;
  • successfully round-trips its own output.

CRoaring's 4.4.3 README explicitly limits serialization support to little-endian systems. Its writer and reader use native byte order.

Iceberg's outer deletion-vector header endian handling is correct; the inner serialized bitmap remains incompatible. Suggested direction: guard big-endian use of this path until the dependency provides compatible serialization.

Regression coverage

Cross-endian golden fixtures are needed in addition to same-host round trips: bucket hash vectors, decimal wire bytes, Avro float/double fixtures, and deletion-vector bitmap fixtures. A full Iceberg integration run on a big-endian target remains untested.

Vorherrschende Sprache
C++
Sterne
226
Forks
133
Ø Merge
5 T. 10 Std.
Gemergte PRs (30 T.)
33

Entwicklungsumgebung

Dieses Projekt bietet weder Dev-Container noch Dockerfile noch Beitragsleitfaden – die Einrichtung liegt bei Ihnen. Beginnen Sie mit der README; die allgemeinen Schritte stehen in unserem Leitfaden für den ersten Beitrag.

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus apache/iceberg-cpp

Alle Issues in apache/iceberg-cpp

Ähnliche Issues

Weitere Issues zu C++

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.