Motivation

Version metadata can be used for debugging and lineage tracking

Proto definition

1. Deprecate https://github.com/lancedb/lance/blob/cb092db625baade0b5568c1a98039e5a991d48e8/protos/table.proto#L134-L139
2. Make it an optional field instead

Write

1. Add Rust struct for VersionAuxData (https://github.com/eto-ai/lance/blob/main/protos/format.proto#L79) along with pb<>rust conversion traits
2. Write the aux data section similar to how index metadata is written (https://github.com/eto-ai/lance/blob/main/rust/src/io/writer.rs#L79). write_manifest should take additional version metadata as parameter note make sure the version_aux_data file position is tested

Read

1. Add fn to read version aux data in the Version struct (https://github.com/eto-ai/lance/blob/main/rust/src/dataset.rs#L66)
2. Instead of returning a PyDict for each version (https://github.com/eto-ai/lance/blob/main/python/src/dataset.rs#L182), create a pyo3 struct Version that is exposed to python. This struct should have a function to read the aux data and cache it in memory.