Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Add Architecture & Design Documentation: FlatBuffers-based Zero-Copy Data Plane for WAMP

Aperta
  • Qualcosa non va?
#1,819 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
42/100
Tipo di issue
Documentazione
Chiarezza
Abbastanza chiara
Stato di attività
Ferma
Stack tecnologico
python
Ambito
documentation

Direzione di ricerca

Inizia esaminando le sorgenti FlatBuffers vendorizzate, flatc incluso, il controllo di sincronizzazione della versione di FlatBuffers zlmdb.check_autobahn_flatbuffers_version_in_sync() e i riferimenti a native wheel e CFFI menzionati nell’issue. Aggiungi in entrambi i progetti una pagina di architettura Sphinx coerente che descriva il flusso di dati condiviso, gli obiettivi zero-copy, la motivazione degli strumenti e le garanzie di sincronizzazione.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

docs enhancement

📘 Add Architecture & Design Documentation: FlatBuffers-based Zero-Copy Data Plane for WAMP

See also: https://github.com/crossbario/zlmdb/issues/98

Summary

Both zlmdb and autobahn-python implement a shared architectural vision:

A schema-first, zero-copy, end-to-end data plane for WAMP, using FlatBuffers as the single source of truth for data representation — across storage and transport, or data-at-rest and data-in-transit.

This design is non-trivial, highly intentional, and spans multiple layers of the system.
At the moment, this architecture exists implicitly in the code and build system, but is not documented in one coherent place.

This issue proposes adding a dedicated Sphinx documentation page explaining the high-level goals, architecture, and design rationale.

Motivation

New contributors and advanced users currently have to infer the design from:

  • vendored FlatBuffers sources
  • bundled flatc compiler
  • native wheels (x86-64 + ARM64)
  • CFFI-based LMDB integration
  • FlatBuffers reflection usage
  • version synchronization checks between projects

This is powerful, but not obvious without context.

A clear architecture document would:

  • Explain why FlatBuffers is central (not incidental)
  • Explain why flatc is bundled
  • Explain why both projects vendor FlatBuffers
  • Clarify how data-at-rest and data-in-transit share the same schema
  • Make the zero-copy goal explicit
  • Reduce the learning curve for advanced users
  • Serve as a reference for future maintenance decisions
High-Level Architecture

At a conceptual level, the system looks like this:

LMDB (data-at-rest)
   ↓ zero-copy
FlatBuffers object graph
   ↓ zero-copy
WAMP RPC / PubSub (data-in-transit)
   ↓ zero-copy
WebSocket (or other WAMP-)transport

Key properties:

  • FlatBuffers schemas are the single source of truth

  • The same schema is used for:

    • persistent storage (zlmdb)
    • network transport (autobahn-python)

  • No intermediate serialization formats are introduced

  • No JSON / MsgPack / Protobuf translation layers

  • Memory layouts are shared as much as possible

Design Principles
1. Schema-First

  • FlatBuffers schemas define:

    • structure
    • evolution rules
    • compatibility guarantees

  • Code generation is derived from schemas — not vice versa

  • Reflection (reflection.fbs / .bfbs) enables dynamic and tooling use cases

2. Zero-Copy by Design
  • LMDB provides memory-mapped access to data
  • FlatBuffers allows reading structured data directly from memory
  • WAMP messages can be transmitted without re-encoding
  • Goal: avoid deserialize → reserialize cycles
3. Unified Data Model for Storage and Transport

  • zlmdb focuses on data-at-rest

  • autobahn-python focuses on data-in-transit

  • Both operate on the same FlatBuffers data model

  • This enables patterns such as:

    • reading a database record
    • returning it directly as a WAMP RPC result
4. Hermetic Tooling

  • FlatBuffers is vendored to ensure:

    • deterministic builds
    • version consistency
    • reproducibility

  • flatc is bundled inside wheels:

    • avoids system dependencies
    • avoids PATH issues
    • ensures schema/compiler compatibility

  • Version synchronization is explicitly checked at runtime (e.g. between zlmdb and autobahn-python)

5. First-Class Native Support

  • Native wheels for:

    • x86-64
    • ARM64 (AArch64)

  • manylinux-compliant builds

  • PyPy supported via CFFI (no CPython-specific APIs)

  • C++ usage isolated behind stable interfaces

Why This Matters

This architecture enables:

  • High-performance WAMP applications
  • Predictable memory usage
  • Efficient large-payload handling
  • Long-term schema evolution without breaking consumers
  • A coherent mental model across persistence and messaging

It is intentionally not a generic “serialization library integration”, but a vertically integrated data plane.

Proposed Documentation Work

Add a new Sphinx documentation page (in both projects) covering:

  • Architectural goals
  • Data flow diagram
  • Role of FlatBuffers
  • Relationship between zlmdb and autobahn-python
  • Rationale for bundling flatc
  • Zero-copy design considerations
  • Version synchronization guarantees

Possible page titles:

  • “Architecture & Design”
  • “FlatBuffers-based Data Plane”
  • “Zero-Copy Data Flow for WAMP”
Related Code / References

  • FlatBuffers version synchronization check:

    • zlmdb.check_autobahn_flatbuffers_version_in_sync()

  • Vendored FlatBuffers + bundled flatc

  • Native wheel build & auditwheel verification

  • CFFI-based LMDB integration

Outcome

Documenting this architecture will:

  • Make the design explicit and durable
  • Improve onboarding for advanced users
  • Provide context for future contributors
  • Reduce accidental regressions
  • Showcase the uniqueness of the approach

If you want, next I can:

  • tailor two slightly different versions (one zlmdb-centric, one autobahn-centric), or
  • draft the actual Sphinx page content instead of an issue description.

Either way — this is absolutely worth documenting.

Checklist

  • I have searched existing issues to avoid duplicates
  • I have described the problem clearly
  • I have provided use cases
  • I have considered alternatives
  • I have assessed impact and breaking changes
Lingua principale
Python
Stelle
2.5k
Fork
769
Merge medio
1g 11h
PR unite (30g)
16

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di crossbario/autobahn-python

Tutte le issue di crossbario/autobahn-python

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.