Feature/API request: portable camera capture across ESP-IDF, Zephyr, and parallel interfaces

This is a design proposal rather than an implementation task; start by reviewing the existing imagecapture.ParallelImageCapture bindings and RP2 backend linked in the issue, alongside the espcamera and Spresense camera interfaces. The proposal explicitly says the API is not implemented or hardware-tested and calls for separate review of the first receiver implementations. Done would require maintainer agreement on the contract and a scoped implementation plan; no specific patch or test is identified.

Proposal: a portable camera capture API for CircuitPython

Status: Discussion draft, 5 October 2026. No new API is implemented by this proposal.

AI assistance: Prepared with Codex at Ladyada's request to propose one camera interface for native ESP-IDF and Zephyr ports, covering parallel and MIPI CSI-2 cameras. Existing bindings, backend code, and vendor documentation were inspected. The proposed API and examples have not been built or hardware-tested.

Recommendation

Define a camera source protocol with common configuration, capture, streaming, and lifetime semantics. Implement it over ESP-IDF, Zephyr Video, and native parallel capture. Keep hardware construction specific to the board, sensor, and transport.

Prefer extending imagecapture with shared format and frame types over introducing another module immediately. Document the source interface as circuitpython_typing.CameraSource; implementations need not inherit from a new Python base class. The namespace is a maintainer decision, separate from the ownership contract below.

Provide two complementary operations:

1. capture_into(buffer) gives the application an image it owns, using an existing buffer. This is the portable baseline for snapshots, processing, and ordinary displayio use.
2. acquire_frame() / Frame.release() provides an optional streaming path for native consumers without requiring a full-frame copy. Ownership is explicit, and acquisition never invalidates an earlier unreleased frame.

Preserve existing espcamera, imagecapture.ParallelImageCapture, and Spresense camera programs. Add adapters and shared implementation incrementally; do not change their established return types or buffer lifetimes in place.

Why this belongs partly in core

Applications should be able to capture and process frames without knowing whether pixels arrived through RP2 PIO, an Espressif parallel camera controller, MIPI CSI-2, or a Zephyr video device. This does not imply that every backend supports every sensor, pixel format, resolution, or streaming mode.

DMA, cache synchronization, interrupt completion, frame ownership, and background capture need native support. Sensor register tables, board wiring, application policy, and most convenience functions can remain in Python libraries. Reuse vendor or Zephyr sensor drivers where those already own the sensor; do not require a second Python implementation of every sensor.

The immediate motivating target is the Metro ESP32-P4 with a Raspberry Pi Camera v1.3 and DSI display. Its earlier Arduino bring-up establishes useful hardware experience, not evidence that this CircuitPython API works. The design should also be exercised on a Zephyr camera backend and an existing parallel camera before being treated as portable.

Existing interfaces and constraints

Existing implementation	What can be reused	What needs care
espcamera.Camera	Sensor controls, configuration, and camera-driver buffers; take() returns a bitmap or JPEG memoryview.	The inspected implementation returns the previous driver buffer at the next take(). A new explicit-lease API must not inherit that lifetime implicitly.
imagecapture.ParallelImageCapture	Caller-provided buffers and transport-specific capture. Python libraries such as adafruit_ov5640 manage sensor settings.	Continuous-capture methods exist in the bindings, but the inspected RP2 implementation uses the default unsupported implementations. Their presence does not establish streaming support.
Spresense camera.Camera	A snapshot interface that writes into an application buffer and returns its length.	camera is already an occupied public namespace. Preserve that API.
Zephyr Video	Format/capability queries, sensor controls, stream control, and buffer enqueue/dequeue.	Device wiring and sensor composition are commonly configured in devicetree. Driver capabilities differ.
ESP-IDF P4 camera controller	CSI/DVP controllers, capture transactions, completion callbacks, and suitable buffer allocation.	Sensor setup and ISP configuration are additional responsibilities. This is a different integration from simply enabling the existing espcamera binding.

The RP2 continuous-capture limitation corrects an earlier assumption in our discussion. See the default implementations and the RP2 backend.

Construction and layering

Unify behavior after construction; do not require one constructor with every platform's pins, device names, and driver flags.

Backend	Construction responsibility	Common interface provider
RP2 / other native parallel capture	Python sensor library configures I2C, clocks, reset, and a parallel receiver.	An updated sensor-library object, delegating capture to native code.
Existing Espressif parallel cameras	Native adapter reuses the ESP camera library's sensor and receiver integration.	An opt-in adapter with the new ownership semantics.
ESP32-P4 CSI or DVP	Board/sensor helper composes sensor setup, receiver, and ISP when required.	A native capture object, optionally wrapped by a Python sensor helper.
Zephyr	Firmware/devicetree defines the video device graph; a binding opens an exposed capture device.	A native Zephyr adapter implementing the same protocol.

An application can import a board-specific open_camera() helper once and keep the rest of its capture code unchanged. Such helpers belong in board/sensor libraries; their names below are placeholders, not existing APIs. A connector alone does not identify which removable sensor is attached.

For native parallel/CSI receivers, retain a lower-level path for Python sensor drivers. A proposed MipiImageCapture receiver would configure the physical receiver, not scan arbitrary sensors or contain a universal sensor database. Its detailed constructor needs a separate review of the first two implementations.

Exactly one layer owns each sensor, I2C transaction path, clock, and receiver. Construction failures unwind acquired resources. Supplied shared I2C buses remain caller-owned and are not deinitialized when the camera closes. Sensor changes affecting timing or output format must go through the managed camera configuration while it is stopped.

Proposed source contract

Names below are proposed. The protocol is descriptive; it does not require runtime inheritance or a Python callback for every frame.

# Interface sketch, not executable implementation.
class CameraSource:
    def supports(self, *, width, height, pixel_format,
                 frame_rate=None) -> bool: ...

    def configure(self, *, width, height, pixel_format,
                  frame_rate=None) -> Format: ...

    def capture_into(self, buffer, *, timeout=1.0) -> int | None: ...

    # Optional streaming capability:
    def start(self, *, buffer_count=2) -> None: ...
    def acquire_frame(self, *, timeout=1.0) -> Frame | None: ...
    def stop(self) -> None: ...

    def deinit(self) -> None: ...

    # Read-only properties:
    format: Format | None
    streaming_supported: bool
    streaming: bool
    frame_available: bool

Sources support context-manager cleanup. Construction leaves capture stopped. format is None until configured. Capture and start require successful configuration; otherwise they raise RuntimeError. frame_available is informative and is false while stopped; callers still handle an acquisition timeout.

Configuration

configure() sets the application's output dimensions and pixel format. Dimensions describe the delivered image, not necessarily the sensor's readout size. Internal sensor mode selection and ISP processing remain backend responsibilities.

• supports() checks the complete sensor/receiver/processing path without starting capture. It does not merely report that the chip has a CSI controller.
• Unsupported dimensions or formats raise ValueError; there is no silent resolution reduction or software JPEG/color-conversion fallback.
• frame_rate is in frames per second. None selects a supported default. A specified rate must be a supported nominal mode; Format.frame_rate reports the configured nominal rate, not a measured delivery guarantee. Hardware clock tolerance is not an API error. Do not silently substitute 15 fps for 30 fps.
• Configuration requires capture stopped and all frame leases released. Validate known-invalid combinations before touching hardware. A runtime configuration failure leaves capture stopped; the application must reconfigure or deinitialize before capturing again.
• Capability queries describe the modes this adapter actually implements. A small query interface avoids eagerly expanding Zephyr's resolution ranges into a large Python list. Full mode enumeration can be added if applications need it.

Format is a small immutable descriptor: width, height, pixel_format, frame_rate, and buffer_size. frame_rate may be None when the backend cannot report a nominal rate. buffer_size is the required capacity for capture_into(), including a configured maximum for compressed output. It is not necessarily the native DMA allocation size.

Snapshots and application-owned pixels

capture_into() writes one complete image and returns the number of meaningful bytes. For uncompressed images, rows in this destination are tightly packed; a backend may copy rows from a padded DMA buffer. JPEG returns its actual encoded length. It performs no implicit color conversion beyond the output format selected at configuration.

When stopped, it performs a bounded one-shot operation and returns to the stopped state. When streaming, it copies one completed frame and releases that internal lease. A direct DMA path is allowed when the destination is suitable; zero-copy is not guaranteed for arbitrary Python buffers.

The baseline destination is a writable byte buffer. A displayio.Bitmap convenience path must explicitly validate bit depth and row layout, handle differing row strides, and mark the bitmap dirty. It must reject incompatible formats rather than treating packed RGB888 as an interchangeable bitmap representation.

Timeouts are seconds: 0 polls, positive values wait up to that duration, and None waits interruptibly. A stopped one-shot request with timeout=0 returns None without starting hardware. A timeout returns None; the destination may contain partial data and must be ignored. Before returning, the backend must cancel or fence any operation that could still write the caller's buffer. Hardware cleanup may add a short bounded delay beyond the wait deadline.

Invalid or undersized destinations raise ValueError. Device/transfer failures raise OSError. Compressed-frame overflow is an error, never a successfully returned truncated JPEG. Do not overwrite the buffer beyond its supplied capacity.

# Proposed API; board_camera is a board/sensor helper to be written.
import imagecapture
from board_camera import open_camera

with open_camera() as camera:
    fmt = camera.configure(
        width=640, height=480,
        pixel_format=imagecapture.PixelFormat.RGB565_LE,
    )
    pixels = bytearray(fmt.buffer_size)
    count = camera.capture_into(pixels, timeout=1.0)
    if count is not None:
        # pixels remains application-owned after the next capture or deinit.
        # Reusing pixels as a destination will, of course, replace its contents.
        print(count)

Streaming and frame ownership

start(buffer_count=2) allocates or reserves a bounded native pool and begins background capture. Calling it while already streaming raises RuntimeError; a failed start leaves the source stopped. Unsupported streaming raises NotImplementedError. Reject an unsupported buffer count instead of silently allocating more frames. A backend must disclose additional scratch/backup memory; buffer_count counts deliverable frame slots, not all memory used by the pipeline.

acquire_frame() requires an active stream and returns the oldest available completed frame, or None on timeout. It does not perform a fresh snapshot or release any earlier frame. Pool exhaustion may drop incoming frames or safely suspend reception; it must never overwrite a leased frame. A backend that cannot safely handle starvation must not advertise this streaming capability.

Use FIFO delivery initially. A latest-frame policy can be added later with explicit drop semantics; neither mode would promise lossless recording when consumers are too slow.

# Proposed streaming API. No frame-sized allocation occurs in this loop.
camera.start(buffer_count=2)
try:
    while True:
        frame = camera.acquire_frame(timeout=1.0)
        if frame is None:
            continue
        with frame:
            print(frame.sequence, frame.timestamp_ns)
            frame.copy_into(pixels)
finally:
    camera.stop()

Frame is a lease, not an independently owned photograph. Proposed members:

Member	Meaning
format	Immutable description of this frame's output mode.
stride	Native bytes per row; None for compressed data. May include padding.
byte_count	Meaningful byte count returned by copy_into(), excluding row padding.
sequence	Increasing completion sequence within this stream session. It does not count unobserved sensor exposures.
timestamp_ns	Capture-completion time in the time.monotonic_ns() domain, or None if unavailable. Not exposure-start time.
copy_into(buffer)	Copies/repacks into application storage, with the same layout rules as capture_into().
release()	Idempotently ends this Python lease; also performed on context-manager exit.

Allocation of a small lease wrapper per Python acquisition is acceptable initially. Reusing one Python Frame object and making old references become valid again is not. Native-to-native consumers should be able to use the internal frame protocol without allocating a Python wrapper per frame.

The ownership lifecycle is:

free -> capturing -> completed -> leased -> free
                                      |
                                      +-> native consumer reference
                                          (reuse waits for completion)

stop() is idempotent and synchronously stops capture DMA/callback access. It discards unclaimed completed frames; already leased frames remain readable until released. Restart/reconfiguration requires all leases and native consumer references released. deinit() stops capture, revokes outstanding Python frame operations, and releases resources only after hardware can no longer access them. Access through a released/revoked frame raises RuntimeError.

Future asynchronous native consumers must hold an owner reference and be stopped/detached before source deinitialization. An explicit deinit() with such a consumer still attached must raise RuntimeError without freeing its storage; it must not wait forever for a display still scanning that storage. VM reset cleanup must stop consumers before producers. This is an integration requirement for the later accelerated display path, not an already implemented facility.

Explicit release is the normal path. Garbage collection is a cleanup fallback, not the mechanism that makes a stream keep up. Ctrl-C, reload, exceptions, and soft reset must stop native work before dropping GC roots or freeing memory.

Do not export an unsafe borrowed memoryview

For the first implementation, Frame should not expose a raw buffer protocol, borrowed memoryview, or borrowed displayio.Bitmap. Offer copy_into() for Python access and an internal native frame protocol for accelerated consumers.

A read-only view prevents Python writes, but does not prevent DMA reuse. Clearing a Frame reference or marking it released does not revoke a previously exported view. The inspected CircuitPython memoryview implementation also must not be assumed to retain an arbitrary native owner or release a camera lease.

This restriction is a deliberate initial tradeoff: Python processing uses owned pixels; native processing can avoid a copy. A future view API needs a demonstrated owner-retention and buffer-reuse design, including derived views and deinitialization. If leads prefer caller-owned queued buffers instead, that is a viable alternative, but then the application must honor exclusive access while buffers are queued; arbitrary existing views cannot be made inaccessible automatically.

Formats, controls, and CSI processing

Start with explicitly defined PixelFormat values needed by current users: RGB565_LE, RGB565_BE, RGB888 (R, G, B byte order), GRAY8, and JPEG. A backend implements only its supported subset. RGB565 byte order is part of the format, not a hidden consequence of the CPU or display driver. Initial raw RGB/gray output uses a documented full-range representation; it does not promise calibrated colors across different sensors.

Do not add a generic RAW or YUV value. Future Bayer support needs CFA order, bit depth, packing, and valid image dimensions. YUV needs packing/planes, range, and color-matrix metadata. Those are separate additions to the shared format model.

The selected output format is distinct from the CSI wire format. A Bayer sensor may feed a receiver and ISP that deliver RGB to the application. On P4, the ISP is part of the CSI integration, including when configured for bypass. Demosaicing, tuning, and automatic exposure/white-balance algorithms are not implied just by receiving CSI packets. Neither JPEG output nor a scaler is mandatory for a camera backend.

Keep sensor-specific controls in existing sensor objects/libraries initially. As an additive portable subset, propose auto_exposure, exposure_time in seconds, auto_white_balance, flip_x, and flip_y, advertised through supported_controls. A property exists only when its units and behavior can be represented honestly. Do not relabel an undocumented vendor register value as seconds or map unrelated gain ranges to an invented universal 0–100 scale. Range discovery and auto/manual interaction need agreement before these controls become required API.

For a composed camera, common controls may be provided by the sensor or ISP. Applications must not operate a second competing auto-exposure loop. Controls that change buffer sizing, Bayer order, or format require stopping and reconfiguring; ordinary exposure changes may affect later frames without promising which exact frame.

Mapping to the backends

Zephyr: configure the selected capture device through its format and capability API. Queue available buffers, dequeue completed ones, and requeue only after the last lease/consumer releases them. Adapt to the Zephyr revision pinned by CircuitPython rather than copying an arbitrary current header signature. Drivers returning frame fragments must assemble a complete frame or reject this API mode. Normalize timestamps and handle counter rollover; do not interpret a driver's millisecond timestamp as nanoseconds. These mechanisms follow Zephyr's buffer API, format descriptor, and buffer metadata.

ESP32-P4: configure the sensor, CSI/DVP receiver, and ISP in the required order. Use camera transactions and native callbacks to move buffers between the internal states. Handle DMA alignment, cache maintenance, and backup-buffer behavior inside the backend. Never run Python or sensor I2C operations from a capture ISR. Existing IDF examples provide a starting point, not a drop-in portable binding.

Existing ESP camera library: map native frame get/return to acquire/release. Share low-level driver setup where practical, but do not implement leases by repeatedly calling the old Python take() method, which automatically returns its previous buffer. Existing and new interfaces cannot simultaneously own the same camera hardware.

RP2 parallel: retain the native one-shot capture primitive and Python sensor configuration. Add timeout/cancellation support needed by the portable contract; do not assume the present primitive already has it. Initially report streaming unsupported. A later PIO/DMA streaming implementation must satisfy the same lease and starvation rules rather than simulating background capture in a Python polling loop.

Display, scaling, playback, and audio

Keep capture separate from display attachment and transforms. The first DSI preview can use an owned bitmap and normal displayio, with copying disclosed and measured. Fast preview needs a native consumer that holds the frame until its DMA/readout operation completes. Ordinary displayio assignment alone does not establish that ownership contract.

A Python object implementing these method names does not automatically implement a native producer protocol. Initial native consumers can accept native Frame objects. Continuous source-to-display attachment needs an explicit native adapter and shutdown contract; it must not call arbitrary Python from an ISR. Resolve that interface with the first real accelerated consumer.

A future scaler can consume the same frame metadata and produce a separate frame. P4's PPA has buffer and scaling restrictions; a common API must expose supported operations rather than imply arbitrary in-place resizing. Prefer sensor-native output sizes when appropriate, but keep display fitting policy outside camera configuration.

Borrow the useful pattern from CircuitPython audio: native background work, source/consumer separation, explicit status, and deterministic cleanup. A live camera has no rewind or looping playback position, so do not copy play(loop=True) or audio pause semantics into capture. A video player can later expose playback controls while sharing frame ownership with cameras and decoders.

Audio devices remain in audioio/audiobusio and related modules. Future recording/playback synchronization requires a common clock and actual audio progress; camera completion timestamps alone do not provide A/V sync. File containers, codecs, audio mixing, USB webcams, and a general processing graph are outside this initial proposal.

Migration and implementation order

1. Agree on the source contract, namespace, and frame-lifetime model. Retain all existing camera APIs and document adapters without an immediate deprecation schedule.
2. Implement the minimum shared bindings/types and one P4 CSI backend. Exercise configuration, one-shot capture, bounded waits, and cleanup on the Metro with the previously used OV5647 camera. Verify a safe copied DSI preview first.
3. Implement a backend for an actually supported Zephyr camera configuration. This is a portability gate, not a claim that Zephyr already supports the Metro P4/OV5647 pairing. Zephyr migration of the Metro is not required.
4. Adapt an existing ESP parallel camera and RP2 sensor library to the baseline interface. Verify old examples remain unchanged. Add streaming only where the backend can meet its contract.
5. Add and measure a native display/scaler consumer. Revisit shared frame metadata before adding video decoders or recording APIs.

Keep these as reviewable changes with their own evidence, rather than one simultaneous rewrite of every camera module. Shared Python bindings belong in shared-bindings/, portable validation and frame bookkeeping in shared-module/, and hardware operations in the appropriate port implementation. Expose an internal native protocol only where native consumers actually need it.

Acceptance checks before claiming support

• Capture known color bars/patterns; verify dimensions, RGB565 byte order, row padding, compressed lengths, and no writes beyond caller capacity.
• Hold one or all frame leases while capture continues. Verify their contents remain unchanged, starvation is safe, and release resumes useful capture.
• Exercise timeout, absent/disconnected sensor, Ctrl-C, GC, stop/start, failed configuration, deinit with an outstanding frame, and soft reload. No native access may survive freed buffers or reset pins.
• Verify released frame objects remain invalid after their native slot is reused. Add retained/derived-view tests before ever introducing a public borrowed-view API.
• On each backend, record frame rate, capture-to-consumer latency, frame drops known to the driver, peak memory, and full-frame copies. Separate calculated memory sizes from measured results; no universal fps promise.
• Run the same capture application after swapping only its construction helper, and retain regression tests for existing APIs.

Decisions requested from the CircuitPython leads

1. Namespace and scope: extend imagecapture with a documented camera-source protocol, or prefer a new cameraio module? Recommendation: reuse imagecapture; keep the Spresense camera API intact.
2. Ownership: accept capture_into() plus optional explicit frame leases, with copied Python access and native consumer access initially? Recommendation: yes; defer borrowed views until their lifetime is demonstrably safe.
3. Sensor boundary: allow both Python sensor libraries and native composite drivers to implement the same source interface? Recommendation: yes; standardize behavior without requiring identical constructors or duplicate sensor drivers.
4. Portability gate: require a native ESP-IDF backend and a real Zephyr backend before stabilizing the new streaming API? Recommendation: yes; use RP2 to validate the simpler snapshot path and honest capability reporting.

Evidence and limits

Local source inspected: CircuitPython verification checkout 100848b3c7adec504efddddcf2bcfb2c3958eddb; ESP camera submodule 303985ab678391b78ebe0baff98edcc863f296c2; ESP-IDF v6.0.1-9-g577a17f7e4.

The checkout pins Adafruit Zephyr to 70f1a63cc8bc96ea53ab0c7983f3f8ff60f438bb. Its public video header and driver interface were checked alongside current documentation. Links to main/latest above are explanatory references and may evolve.

This work produced a design proposal only. No camera binding was changed, firmware flashed, performance measured, or Zephyr camera hardware tested during its preparation.