CI: add a big-endian job (s390x, cross-compiled, tests under qemu-user)

Aperta
#6,115 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
45/100
Tipo di issue
Funzionalità
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
cmake, cpp, github-actions
Ambito
ci-cd, testing

Direzione di ricerca

Start with cmake/ci.cmake and .github/workflows/ubuntu.yml to see how existing CI targets are defined, then check the s390x cross-compile and qemu-user commands in the issue's sketch, which are not yet run as written. Decide how much of the suite to run, since the runtime figures are in the issue. Done means a big-endian job builds all test binaries and passes under qemu-user, and on a branch that reverts #6074 the build fails in tests/src/unit-alt-string.cpp.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

CI kind: enhancement/improvement tests
Description

No CI job builds the library or its tests for a big-endian target. Every job in .github/workflows/ (Ubuntu x86-64, the native ci_test_arm64, macOS, Windows, Emscripten, CUDA, ...) runs on a little-endian target. The library has big-endian code paths: the byte swaps of the binary formats (CBOR, MessagePack, UBJSON, BJData, BSON, BON8), the SWAR scans guarded by JSON_LITTLE_ENDIAN_TARGET, and json_view, whose save()/load() do not exist on big-endian targets by design (basic_json_document::load: "Little-endian targets only."). None of this is compiled in CI on such a target.

That gap let a test break the build on big-endian targets without anyone noticing: tests/src/unit-alt-string.cpp called alt_editable_document::load(d.save()) without the little-endian guard that the other image tests have, so the test suite of develop did not compile on s390x. A bug hunt found it by hand; #6074 fixed it. Packagers who build and run the tests on s390x (Debian, Fedora, Ubuntu and SUSE all ship s390x) would have hit it with the 3.13.0 release.

Since when: always (no big-endian job has existed). The concrete breakage came with the json_view alt-string test and was fixed by #6074 (merged 2026-11-11).
Security: Not a vulnerability: a CI coverage gap.
Related: #6074 (the breakage, which says a CI job "is not part of this PR"), #5920 (no save()/load() on big-endian targets).

Possible fix: a job that cross-compiles the tests natively for s390x and runs them under qemu-user. Cross-compiling keeps the compile at native speed; only the test binaries run emulated. Building inside an emulated s390x container (uraimo/run-on-arch-action, or docker/setup-qemu-action plus an s390x/ubuntu image) would also run the compiler under QEMU, which is much slower for a suite of 180 test binaries; I did not measure that variant.

s390x rather than big-endian ppc64: Debian and Ubuntu ship the s390x cross compiler (g++-s390x-linux-gnu) and qemu-user in their main archives; big-endian ppc64 is only a Debian ports architecture (ppc64el, which they ship, is little-endian).

A sketch in the style of ubuntu.yml, with a target in cmake/ci.cmake like the other jobs:

add_custom_target(ci_test_big_endian
    COMMAND ${CMAKE_COMMAND}
        -DCMAKE_BUILD_TYPE=Debug -GNinja
        -DCMAKE_SYSTEM_NAME=Linux -DCMAKE_SYSTEM_PROCESSOR=s390x
        -DCMAKE_CXX_COMPILER=s390x-linux-gnu-g++
        "-DCMAKE_CROSSCOMPILING_EMULATOR=qemu-s390x\;-L\;/usr/s390x-linux-gnu"
        -DJSON_BuildTests=ON -DJSON_FastTests=ON
        -S${PROJECT_SOURCE_DIR} -B${PROJECT_BINARY_DIR}/build_big_endian
    COMMAND ${CMAKE_COMMAND} --build ${PROJECT_BINARY_DIR}/build_big_endian
    COMMAND cd ${PROJECT_BINARY_DIR}/build_big_endian && ${CMAKE_CTEST_COMMAND} --parallel ${N} --output-on-failure -LE git_required
    COMMENT "Cross-compile for s390x (big-endian) and run the tests under qemu-user"
)
  # Big-endian target: the tests are cross-compiled for s390x and run under
  # qemu-user (only the test binaries are emulated, not the compiler).
  ci_test_big_endian:
    runs-on: ubuntu-26.04
    container: debian:trixie
    steps:
      - name: Install the s390x cross compiler and qemu-user
        run: |
          apt-get update
          apt-get install -y --no-install-recommends g++-s390x-linux-gnu qemu-user git ca-certificates
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      - name: Get latest CMake and ninja
        uses: lukka/get-cmake@f86dd60acae0d132056913e248bdabf0e0580b7a # v4.4.5
      - name: Check that the target is big-endian
        run: echo | s390x-linux-gnu-g++ -dM -E - | grep -q '__BYTE_ORDER__ __ORDER_BIG_ENDIAN__'
      - name: Run CMake
        run: cmake -S . -B build -DJSON_CI=On
      - name: Build
        run: cmake --build build --target ci_test_big_endian

(Not run as written: I ran the same cmake configure, build and ctest commands directly in the container, not through a ci.cmake target or a workflow, so the \; escaping of the emulator list in add_custom_target needs a check. The -LE git_required follows the note for exported trees; inside the CI checkout it could be dropped. git is needed to download the test data.)

Runtime (measured; Debian trixie arm64 container on an Apple M1 Max under colima, limited to 4 CPUs like a GitHub runner, GCC 14.2 cross compiler, Debug, JSON_FastTests=ON, test data from a local copy):

step time
apt-get install of the toolchain and qemu-user 39 s
build of all 181 test binaries, -j 4 592 s
ctest -j 4 -LE git_required, 197 tests 653 s (all passed)

test-unicode_cpp11 alone takes 404 s under qemu, so it sets the wall time of the test step; the next slowest are test-json_patch_cpp11 (62 s) and test-ordered_json_cpp11 (56 s). The binary-format and json_view tests (CBOR, MessagePack, UBJSON, BJData, BSON, BON8, all json_view* tests and the binary writer tests: 40 tests) take 410 s of CPU time together. A GitHub-hosted x86-64 runner compiles slower than an M1 Max core and also emulates s390x, so expect the job to take more than the 21 minutes above, roughly 30 to 45 minutes (an estimate, not measured on GitHub). Options to shorten it:

  • Compile all, run a subset: build everything (this is what would have caught #6074), then ctest -R 'cbor|msgpack|ubjson|bjdata|bson|bon8|binary|json_view|alt-string|serialization|class_parser'. That keeps the endianness-relevant tests and drops test-unicode (the code it covers is byte-order independent).
  • Exclude only the slow one: ctest -E 'test-unicode', which cuts the test step to about the time of the next slowest tests divided over 4 jobs.
  • Run the job only on pushes to develop and on a schedule, not on every PR push. I would still run at least the build on PRs, since compiling is what broke.

With json_view images not available on big-endian targets, test-json_view_image_cpp11 there checks only that save()/load() do not exist (0.04 s in the run above); that is intended.

Test: the job itself. To check that it catches the #6074 kind of breakage, run it once on a branch that reverts #6074; the build must fail in unit-alt-string.cpp (see the error below).

Reproduction steps

How the breakage happened and how it shows on a big-endian target:

  1. Check out the commit before #6074 (e0b76bac31^).
  2. In a Debian trixie container: apt-get install g++-s390x-linux-gnu.
  3. s390x-linux-gnu-g++ -std=c++11 -fsyntax-only -w -I include -I tests/thirdparty/doctest -I tests/src tests/src/unit-alt-string.cpp
  4. Every CI job passed on that commit, because none compiles for a big-endian target.
Expected vs. actual results
  • Expected: a CI job compiles the test suite for a big-endian target, so a test that uses a little-endian-only API without the guard fails CI.
  • Actual: CI was green; the test suite did not compile on s390x until #6074, found only by a manual cross build.
Minimal code example
// tests/src/unit-alt-string.cpp before #6074 (line 534): no little-endian guard
const alt_editable_document loaded = alt_editable_document::load(d.save());
Error messages

Before #6074 (e0b76bac31^), s390x cross compiler (shortened aka types):

tests/src/unit-alt-string.cpp:534:69: error: 'load' is not a member of 'alt_editable_document' {aka 'nlohmann::json_abi_v3_12_0::basic_json_document<nlohmann::json_abi_v3_12_0::basic_json<std::map, std::vector, alt_string, ...>, nlohmann::json_abi_v3_12_0::view_mode::editable>'}
tests/src/unit-alt-string.cpp:534:76: error: 'using alt_editable_document = class nlohmann::json_abi_v3_12_0::basic_json_document<...>' {aka 'class nlohmann::json_abi_v3_12_0::basic_json_document<...>'} has no member named 'save'

develop (5bad6c518b), the proposed configuration:

build (-j 4): 592 s, 181 test binaries
100% tests passed, 0 tests failed out of 197
Total Test time (real) = 652.71 sec
Compiler and operating system

s390x-linux-gnu-g++ (Debian 14.2.0-19) 14.2.0 with qemu-user, Debian trixie arm64 container (colima) on macOS 27.0.1, Apple M1 Max

Library version

develop (5bad6c518b); 3.12.0: n/a (CI gap; the specific breakage did not exist in 3.12.0)


Found in a bug hunt on develop (2026-11-10); drafted with Claude Code.

Lingua principale
C++
Stelle
50.9k
Fork
7.5k
Merge medio
1g 3h
PR unite (30g)
217

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di nlohmann/json

Tutte le issue di nlohmann/json

Issue simili

Altre issue su C++

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.