Add Documentation: Native Binaries, manylinux, ISA Compatibility & PyPy Support

Đang mở
  • Có gì không ổn?
#1,820 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
45/100
Loại issue
Tài liệu
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Đình trệ
Công nghệ
python
Lĩnh vực
build-system, documentation

Hướng nghiên cứu

Bắt đầu bằng việc xác định tài liệu Sphinx, tài liệu cài đặt và tài liệu phát triển hoặc dành cho contributor, nơi có thể thêm và tham chiếu một trang mới. Kiểm tra hành vi được mô tả trong tài liệu đối chiếu với nội dung của wheel, quy trình manylinux và auditwheel, entry point flatc được đóng gói, và hỗ trợ PyPy được mô tả trong issue; công việc được xem là hoàn tất khi quy trình phân phối native và các ràng buộc tương thích được giải thích rõ ràng.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

docs enhancement

📘 Add Documentation: Native Binaries, manylinux, ISA Compatibility & PyPy Support

See also: https://github.com/crossbario/zlmdb/issues/99

Summary

Both zlmdb and autobahn-python ship native code and native executables as part of their Python distributions:

  • CFFI-based native extensions (LMDB)
  • A bundled FlatBuffers compiler (flatc)
  • Native wheels for x86-64 and ARM64
  • First-class PyPy support
  • Fully manylinux-compliant wheels verified with auditwheel

This is powerful — but also subtle and non-obvious.

This issue proposes adding a dedicated documentation page explaining how and why native artifacts are built, bundled, verified, and distributed the way they are.

Motivation

From a user’s point of view, installation is intentionally simple:

pip install zlmdb
# or
pip install autobahn

Behind the scenes, however, a lot of careful engineering ensures that:

  • wheels install without compilation
  • binaries work across distributions
  • PyPy works just as well as CPython
  • auditwheel repair succeeds
  • ISA incompatibilities (e.g. x86_64_v2) are avoided
  • users do not need system FlatBuffers, LMDB, or compilers

Without documentation, it is difficult for users and contributors to understand:

  • why flatc is bundled at all
  • why it is shipped inside the Python package
  • why manylinux containers matter
  • why baseline ISA flags are required
  • why CFFI was chosen over CPython C-API
  • why PyPy support “just works”
What This Documentation Should Explain
1. Native Components Shipped in the Wheel

Each wheel contains:

  • A CFFI-based LMDB extension (_lmdb_cffi.*.so)
  • A native flatc executable, invoked via a Python wrapper
  • Vendored FlatBuffers Python runtime and reflection data

All of these are installed transparently into the environment.

2. manylinux and ISA Compatibility

Key constraints:

  • Wheels must run on a wide range of Linux distributions

  • auditwheel enforces:

    • ABI compatibility
    • baseline CPU instruction sets

  • Compiling on a “modern” host can silently introduce:

    • x86_64_v2 instructions (SSE4.2, etc.)
    • too-new symbol versions

Therefore:

  • Builds are done in manylinux containers
  • Baseline architecture flags are enforced
  • Toolchains are chosen carefully
  • Wheels are verified with auditwheel repair

This ensures wheels are installable on older systems, not just CI hosts.

3. Bundled flatc: Why and How

Why bundle flatc?

  • Avoid system dependencies
  • Ensure schema ↔ compiler compatibility
  • Guarantee reproducible builds
  • Support hermetic CI and user installs
  • Enable schema-driven workflows out of the box

How it is exposed:

  • flatc is shipped as a native executable inside the package
  • A Python console script (flatc) dispatches to it
  • Works identically on CPython and PyPy
  • No PATH manipulation required
4. PyPy as a First-Class Target

PyPy support is not accidental:

  • LMDB is accessed via CFFI, not CPython C-API
  • Native wheels are built and published for PyPy
  • No fallback-to-sdist compilation required for users

From a user’s perspective:

pip install zlmdb
flatc --version

Just works — even on PyPy.

This deserves to be documented explicitly.

5. Dynamic vs Static Linking (and Why It Matters)

The documentation should clarify:

  • Which libraries are dynamically linked (e.g. libc)
  • Which are intentionally not statically linked
  • Why auditwheel compatibility matters more than “fully static” binaries
  • How symbol versions and ISA levels interact with manylinux

This context explains earlier design decisions and prevents future regressions.

Example: What Users Actually Get

After installation, users have:

  • A working flatc executable
  • A native LMDB backend
  • No system-level dependencies required
  • Identical behavior on CPython and PyPy

This is the goal, and the build system exists to make this boringly reliable.

Proposed Documentation Location

Add a new Sphinx page such as:

  • “Native Binaries & manylinux Compatibility”
  • “Native Code, Wheels, and PyPy Support”
  • “How Native Components Are Built and Distributed”

This page should be referenced from:

  • Installation docs
  • Development / contributor docs
Why This Matters

Documenting this will:

  • Reduce confusion around build complexity
  • Help contributors avoid breaking manylinux compliance
  • Explain why certain CI constraints exist
  • Make PyPy support feel intentional (because it is)
  • Turn “tribal knowledge” into durable documentation
Outcome

A clear explanation of:

  • native binaries in Python wheels
  • manylinux constraints
  • ISA compatibility
  • PyPy support
  • why flatc is bundled

This complements the higher-level architecture & data-plane documentation and completes the picture from design → implementation → distribution.

Checklist

  • I have searched existing issues to avoid duplicates
  • I have described the problem clearly
  • I have provided use cases
  • I have considered alternatives
  • I have assessed impact and breaking changes
Ngôn ngữ chính
Python
Star
2.5k
Fork
766
Merge trung bình
12 giờ 40 phút
Pull request đã merge (30 ngày)
20

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của crossbario/autobahn-python

Tất cả issue của crossbario/autobahn-python

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.