Question on decoding large data structures

示されている Rust Ephemeris および Spline Encode/Decode の実装、特に decoder.sequence と peek_header を確認します。要求されている数千件の spline のゼロコピー復号が、割り当てなしでサポートされているかを判断し、推奨されるアプローチを特定します。両方の質問に確立された回答が得られた時点で、issue は解決されます。

Hi there,

I'm trying to use ASN1 to store a large data set (up to 200 MB or so). This in replacement of a file that is currently packed struct stored as bytes to disk. I was wondering if you had any hints on the best approach to do this.

The data I have is an "ephemeris" which stores several "splines." Before starting the encoding, I do not know how many splines I'll need to encode/decode, but it could be several thousands. The reference file I'm using defines that length up front and then libraries perform a direct access to the correct spline (there's a specific algorithm on how to retrieve the correct spline number from some input information, and then seeking through the file is sufficient to grab the data).

In an attempt to mimic this behavior, I'm trying the following structure:

pub struct Ephemeris<'a> {
    pub spline_duration_s: f64,
    pub splines: &'a [Spline<'a>], // We can't do SequenceOf<Spline<'a>, N>, because N would be too large to fit on the stack
}

And the following encoding and decoding implementations (the encoding works, but the decoding fails)

impl<'a> Encode for Ephemeris<'a> {
    fn encoded_len(&self) -> der::Result<der::Length> {
        self.spline_duration_s.encoded_len()?
            + self.splines.iter().fold(Length::new(2), |acc, spline| {
                (acc + spline.encoded_len().unwrap()).unwrap()
            })
    }

    fn encode(&self, encoder: &mut der::Encoder<'_>) -> der::Result<()> {
        encoder.encode(&self.spline_duration_s)?;
        encoder.sequence(Length::new(self.splines.len() as u16), |encoder| {
            for spline in self.splines {
                encoder.encode(spline)?;
            }
            Ok(())
        })
    }
}

impl<'a> Decode<'a> for Ephemeris<'a> {
    fn decode(decoder: &mut Decoder<'a>) -> der::Result<Self> {
        let spline_duration_s = decoder.decode()?;
        let expected_len: u32 = decoder.peek_header().unwrap().length.into();
        dbg!(expected_len);
        // XXX: how can I point each spline to the input buffer? I can't perform an alloc, nor can I 
        let mut splines: &'a [Spline<'a>; 1000]; // XXX: How do I even initialize this?
        decoder.sequence(|decoder| {
            for idx in 0..expected_len {
                splines[idx as usize] = decoder.decode()?;
            }
            Ok(())
        });
        Ok(Self {
            spline_duration_s,
            splines,
        })
    }
}

Question

1. Is this a bad approach and if so, what would you recommend instead?
2. If this is a reasonable approach, how can I get the splines field to point to the decoded splines?

Appendix

For reference, here is how a spline is defined, where x,y,z are encoded as OctetStrings.

pub struct Spline<'a> {
    pub start_epoch: Epoch,
    pub end_epoch: Epoch,
    /// State information (km)
    pub x: &'a [u8],
    /// State information (km)
    pub y: &'a [u8],
    /// State information (km)
    pub z: &'a [u8],

Thanks for your help