Support storing Arrow-IPC blobs in the LMDB backend (first-class / registry-selectable)

Start with get_async_backend_cls("x.lmdb"), _async_adapters.py, _adapters.py, and lmdb/_blob_backend.py to compare the registry path with the native raw-byte backend. Decide which proposed public path to pursue, then verify that the registry or documented constructor can store and retrieve opaque Arrow-IPC bytes without msgpack conversion.

Summary

Add first-class support for storing opaque Arrow-IPC blobs in the LMDB backend (one pre-serialized IPC stream per frame), reachable through the normal AsyncBlobIO/registry API.

The low-level capability already exists — LMDBBlobBackend round-trips arbitrary bytes — but the standard construction path routes *.lmdb through the object (msgpack) backend, which rejects non-msgpack values. So today you can only get there by hand-wiring internals.

Motivation

While benchmarking read paths for a read-only frame CDN (zndraw-cdn: Lance + Arrow-IPC over REST) against zndraw's LMDB + msgpack path, the fastest option turned out to be pre-serialized Arrow-IPC blobs stored in LMDB: build each frame's IPC stream once at write time, store the bytes, serve them verbatim. That gives LMDB-get latency and the Arrow wire format with zero per-request conversion — effectively a durable version of an IPC byte-cache.

Read latency, single 1000-atom frame (+forces), through a REST server, random access over 1000 frames (median ms):

approach	median	p99
LMDB + msgpack (AsyncBlobIO.get)	1.55	6.6
LMDB + Arrow-IPC blob (precomputed, raw lmdb)	1.67	5.5
LMDB + Arrow IPC (decode→atoms_to_table→encode at read)	2.20	15.5
Lance + Arrow IPC (take([i]), positional)	3.20	15.8
Lance + Arrow IPC (__HASH__ CAS filter)	5.40	25.5

Converting msgpack→Arrow at read time costs ~0.6 ms/frame (and ~32 ms for a 200-frame batch); storing the IPC blob removes that entirely.

Current behavior / limitation

The common path (get_async_backend_cls("x.lmdb") → LMDBObjectBackend, wrapped by AsyncObjectToBlobReadWriteAdapter) msgpack-decodes every value on write, so an opaque Arrow-IPC blob raises:

File ".../asebytes/_async_adapters.py", line 253, in extend
File ".../asebytes/_adapters.py", line 32, in <dictcomp>
    k.decode(): msgpack.unpackb(v, object_hook=m.decode) for k, v in raw.items()
msgpack.exceptions.ExtraData: unpack(b) received extra data.

i.e. the blob value contract is implicitly "msgpack-numpy bytes" (_serialize_row/_deserialize_row in _adapters.py).

What already works (so the ask is small)

The native LMDBBlobBackend (asebytes/lmdb/_blob_backend.py) is a ReadWriteBackend[bytes, bytes] that txn.puts raw bytes — no msgpack. Constructing AsyncBlobIO over it directly round-trips an opaque IPC blob perfectly:

from asebytes import AsyncBlobIO
from asebytes.lmdb import LMDBBlobBackend
from asebytes._async_backends import sync_to_async

io = AsyncBlobIO(sync_to_async(LMDBBlobBackend.from_uri("frames.lmdb", group="room")))
await io.extend([{b"arrow": ipc_bytes}])          # store
blob = (await io.get(0))[b"arrow"]                 # exact same bytes back

Verified: 57704-byte IPC stream stored and returned byte-identical, decodes to a valid 1-row table. The problem is purely that this isn't reachable via the registry/documented API and relies on private imports (_async_backends.sync_to_async).

Proposal (any one of these would unblock it)

1. Register a native async LMDB blob backend so the async resolver can return raw-blob mode for *.lmdb (e.g. via a uri flag/scheme such as lmdb+blob:// or from_uri(..., layer="blob")), without the msgpack object adapter.
2. An Arrow-IPC frame codec for the high-level IO: encode Atoms → atoms_to_table → IPC on write and IPC → Atoms on read, so users keep ASEIO ergonomics but get Arrow on the wire (and an endpoint can stream the stored bytes verbatim).
3. At minimum, document that LMDBBlobBackend stores raw bytes and the supported way to build an AsyncBlobIO over it without the object adapter (promote sync_to_async or add a public constructor).

Tradeoffs to call out in docs

• Each blob re-ships its own ~2 KB Arrow schema block → wasteful for tiny frames and you lose single-schema/columnar batch/range reads (you'd fetch N blobs and re-concat).
• No cross-frame columnar slicing (column[[5, 112, ...]]).
• Write-once: partial field updates (e.g. a PATCH that touches only arrays.positions) require a full re-encode of the blob — so this mode suits immutable / read-only data.

Prior art

KV/embedded stores that expose Arrow to minimize copies: UKV (LevelDB/RocksDB/in-memory, Arrow-native), Plasma (shared-memory Arrow object store), TileDB; and Arrow-IPC/Feather files are themselves mmap-zero-copy. For "Arrow + random access + columnar" the idiomatic choice is a columnar store like Lance — so this LMDB+IPC-blob mode is specifically for the fast immutable single-frame read path, not a replacement for columnar storage.

Environment

asebytes 0.3.2 (also reproduced conceptually on 0.3.1), lmdb 2.2.0, pyarrow 24, Python 3.11, macOS.