Rust transform SDK discards the input record offset that the ABI provides (Go SDK exposes Record.Offset)

Start in src/transform-sdk/rust/core-sys/src/lib.rs at abi::read_next_record and trace the offset alongside timestamp into WrittenRecord or WriteEvent. Compare the Rust record API with src/transform-sdk/go/transform/sdk.go, then verify that the public Rust record accessor exposes the ABI-provided offset without changing existing fields.

Summary

The Rust transform SDK reads the input record's offset from the ABI and then discards it, so there is no way to obtain it from a transform. The Go SDK exposes it as Record.Offset. This makes the two SDKs non-equivalent for any transform that needs to report where a record came from.

The offset is already available — this is a plumbing gap in the safe wrapper, not an ABI limitation.

Where

abi::read_next_record takes an out-param for the offset and Redpanda populates it. core-sys passes a local, then drops it:

https://github.com/redpanda-data/redpanda/blob/efd61acff989b8c25d99757711edf44a14d13903/src/transform-sdk/rust/core-sys/src/lib.rs#L156-L182

let mut offset: i64 = 0;
let errno_or_amt = unsafe {
    abi::read_next_record(&mut attr, &mut timestamp, &mut offset, ...)  // populated here
};
...
let ts = SystemTime::UNIX_EPOCH + Duration::from_millis(timestamp as u64);
cb(
    WriteEvent {
        record: WrittenRecord::from_record(record, ts),   // ...and dropped here
    },
    writer,
)

timestamp makes it through via ts; offset does not. Nothing else in the public API surfaces it — WrittenRecord exposes key/value/headers/timestamp only.

For comparison, the Go SDK has it on the record struct:

https://github.com/redpanda-data/redpanda/blob/efd61acff989b8c25d99757711edf44a14d13903/src/transform-sdk/go/transform/sdk.go#L102

Why it matters

We are porting a production Go transform to Rust (TinyGo's allocator doubles its arena on growth, which makes the 3 MiB data_transforms_per_function_memory_limit hard to live under). The port reproduces the Go output byte-for-byte except for one field.

When a record exceeds our size guard we dead-letter it with a diagnostic payload. In Go that is:

{"error":"record_too_large","size":214003,"offset":171835412,"limit":200000}

In Rust the offset cannot be populated. It is the only field that locates the offending record in the input topic — the key identifies which record but not where to read it, so recovering the original bytes becomes a timestamp-window scan rather than a direct seek.

More generally, any transform emitting error or audit records, or correlating output back to an input position, needs this. It is also the natural thing to include in a log line when a transform panics on a specific record.

Suggested fix

Thread offset through the same path ts already takes: add it to WrittenRecord (or WriteEvent) with an accessor, and pass it in from_record. That appears additive — WrittenRecord::from_record is pub(crate), so the change is internal apart from the new accessor.

Happy to open a PR if that shape is agreeable, or if you would prefer it on WriteEvent rather than on the record.

Version

redpanda-transform-sdk 1.1.0 / redpanda-transform-sdk-sys 1.1.0 (crates.io), target wasm32-wasip1. The same code is present on the tip of the default branch as of this report.