rvf: a typed custom-segment API (append and read application segments in a container)

Start with crates/rvf/rvf-types/src/segment_type.rs, rvf-wire/src/writer.rs, and the compaction paths in rvf-runtime/src/store.rs and rvf-runtime/src/store/policy_api.rs. The issue offers a first PR covering the reserved type range, single-store append and read APIs, and compaction; first resolve the listed design questions, especially signed compaction and strict policy behavior. Done means custom records are persisted, readable, integrity checked, and retained according to the agreed compaction rules.

I'd like to propose a small public API for appending and reading application-defined segments in an RVF container. The format and compaction already tolerate this today. Only the API is missing. All references are at fefc2ec9.

What I found

Items are CODE-DERIVED unless tagged otherwise.

1. SegmentType (crates/rvf/rvf-types/src/segment_type.rs:9) is a closed #[repr(u8)] enum, and TryFrom<u8> returns Err(value) for anything unlisted. Assigned values are 0x00-0x11, 0x20-0x23 and 0x30-0x36.

2. RvfStore has no public way to append a segment of a chosen type. The typed embed_kernel, embed_ebpf, wasm and dashboard embeds exist, but there is nothing general.

3. rvf-wire write_segment(seg_type: u8, payload, flags, segment_id) (rvf-wire/src/writer.rs:28) takes a raw u8, so the wire layer already allows any type value.

4. Compaction already carries unknown types. scan_preservable_segments (rvf-runtime/src/store.rs:4567) returns every segment that is not Vec, Manifest or Journal, and compact() (store.rs:1687) copies them verbatim, dropping only Index and Meta. The doc comments at store.rs:1686 and store.rs:4565 name "vendor extensions".

5. MEASURED at fefc2ec9: a segment of type 0xE0 appended to a store file with rvf_wire::writer::write_segment, followed by RvfStore::open, ingest_batch, delete and compact, is still in the file byte for byte afterwards. The store reopens read-only with the expected vectors. So the format and compaction tolerate custom types today. The reproducer is an integration test in rvf-runtime/tests:

   let seg = rvf_wire::writer::write_segment(0xE0, b"app-record-v1", SegmentFlags::empty(), 9_000);
   // create store, ingest [1,2], close; append `seg` to the file; then:
   let mut s = RvfStore::open(&p)?;       // opens fine
   s.ingest_batch(&[&[1.0, 1.0]], &[3], None)?;
   s.delete(&[1])?;
   s.compact()?;
   s.close()?;
   // scan the file for a segment header with seg_type 0xE0 and that payload: found
   

6. UNVERIFIED: signed stores do not trust file bytes during compaction. verified_compaction_sources() (rvf-runtime/src/store/policy_api.rs:324, called at the top of compact() when a signing key is present) takes the segments to carry from the directory, each checked against the committed hash. I did not check whether a hand-appended segment is in that directory. I would expect it is not, which would mean it neither survives signed compaction nor is covered by the signature. I'd want a failing test before relying on this.

Use case

Applications that carry their own typed records inside the container, keyed by an application key. Examples are per-document annotations, schema or version stamps, small indexes and provenance notes. They should:

• (a) live in the same file as the vectors, so one file is the unit of copy, backup and signing;
• (b) be inherited by a COW branch the way vectors are;
• (c) be covered by the same integrity and signing story as built-in segments;
• (d) survive compaction.

Today an application has to keep a sidecar file, or hand-append raw bytes and hope.

Proposal

This is a proposal, not a finished design. The open questions below are real.

Reserve 0xC0-0xFF for vendor and application types, never assigned by the format. SegmentType stays closed. Add a checked wrapper:

pub struct CustomType(u8);                       // checked: 0xC0..=0xFF
impl CustomType { pub fn new(v: u8) -> Result<Self, RvfError>; }

And a store API:

impl RvfStore {
    /// Append a record. `key` is the application key (bounded length, e.g. <= 255 bytes);
    /// `payload` is opaque. Returns the segment id.
    pub fn append_segment(&mut self, ty: CustomType, key: &[u8], payload: &[u8]) -> Result<u64, RvfError>;

    /// Latest record for (ty, key), following the branch chain if this store is a COW child.
    pub fn read_segment(&self, ty: CustomType, key: &[u8]) -> Result<Option<Vec<u8>>, RvfError>;

    /// All live records of a type, in segment order (own records, then parents').
    pub fn iter_segments(&self, ty: CustomType) -> impl Iterator<Item = (Vec<u8>, Vec<u8>)>;

    /// Tombstone a key (an empty-payload record with a delete flag), so a child can hide an inherited one.
    pub fn remove_segment(&mut self, ty: CustomType, key: &[u8]) -> Result<(), RvfError>;
}

Payload framing inside the segment is u16 key_len | key | payload, one record per segment. That way scan_preservable_segments and the manifest directory need no changes to carry them.

Rules:

• Write path. It goes through the same writer as built-in segments: wire dialect header, checksum algo 1 normally and 2 (SHAKE-256-128) when the store is signed, a directory entry in the next manifest, and fsync before the manifest that publishes it (the same ordering as the metadata generations).
• Integrity and signing. In a signed store the segment is in the directory with its committed hash, so the Level-0 signature covers it through the manifest, and verified_compaction_sources treats it like any other carried segment. For an unsigned store only the header content hash applies. Under Strict, I have not checked how unknown types are treated today. The proposal is that types in the reserved range are accepted if the hash matches (see question 4).
• Branch inheritance. read_segment and iter_segments on a COW child look in the child, then the parent chain. Precedence is the same as vectors: the child wins, and a remove_segment tombstone hides the inherited record. Records are not copied into the child at branch time.
• Compaction guarantee. Live records (latest per (type, key), not tombstoned) are carried. Superseded versions of a key are dropped, which is where compaction reclaims space. Today everything unknown is carried forever, so this needs deciding.
• Older readers. A runtime that does not know the API still opens the file and carries the segments verbatim (item 4), so the feature is backward compatible.

Open questions

1. Reserved range: is 0xC0-0xFF acceptable, or should vendor types be namespaced (a u16, or a 4-byte vendor tag in the payload) so two applications cannot collide?
2. Latest-wins per key with superseded versions dropped at compaction, versus keeping all records forever as today?
3. Key and payload size limits. One record per segment, or batched records for many small values?
4. Should custom segments be visible to Strict and Paranoid as an explicit allow-list entry rather than a blanket reserved range?
5. Should iteration across a branch chain merge child and parent records, or only look at the nearest store?
6. rvf-node and rvf-wasm bindings: in scope, or later?
7. Item 6 above (signed compaction of hand-appended segments) is unverified. Does someone already know the answer?

Offer

I'm happy to implement this. A first PR would cover the reserved range, append_segment, read_segment and iter_segments for a single store, and the compaction guarantee. A second would cover branch inheritance and tombstones. I'll start once the shape is agreeable.