Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Update Zstd decompression for "unknown decompressed size" when streaming API was used for compression

Aperta
  • Qualcosa non va?
#116 17 commenti 0 reazioni 1 assegnatario Vedi su GitHub

@ajelenak ci sta già lavorando.

Dal 2/1/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

Filter - ZSTD Priority - 1. High 🔼 Type - Improvement

Introduction

The Zstandard plugin for HDF5 should be modified to allow for an unknown decompressed size in the frame header.

Currently, the Zstd decompression scheme, following from the original implemention, uses ZSTD_getDecompressedSize to obtain the size of the decompressed buffer. The returned value is not validated and passed directly to malloc.

https://github.com/HDFGroup/hdf5_plugins/blob/770d70ae73587714629cf5ec139d482c1562e7c1/ZSTD/src/H5Zzstd.c#L59-L60

ZSTD_getDecompressedSize returns 0 if the decompressed size is empty, unknown, or an error has occured. If malloc is asked to allocate 0 bytes, it will return NULL, resulting in returning an error condition. This is an incorrect result if the decompressed size is actually empty or unknown and there is no actual error. ZSTD_getDecompressedSize is obsolete.

ZSTD_getFrameContentSize should replace the use of ZSTD_getDecompressedSize. ZSTD_getFrameContentSize distinguishes between empty, unknown, or an error. The unknown or error states are indicated by a return value of ZSTD_CONTENTSIZE_UNKNOWN or ZSTD_CONTENTSIZE_ERROR, respectively.

The unknown decompression state is common. This occurs when the compression is done via the streaming API via ZSTD_compressStream or ZSTD_compressStream2. ZSTD_compressStream2 in particular only stores the frame size when either ZSTD_e_end is provided on the initial call or ZSTD_CCtx_setPledgedSrcSize is used.

Tasks

  • Use ZSTD_getFrameContentSize instead of the obsolete ZSTD_getDecompressedSize to correctly distinguish between empty, unknown, or error states when determining the decompressed size.
  • Recognize the unknown size state by checking the return value of ZSTD_getFrameContentSize against ZSTD_CONTENTSIZE_UNKNOWN
  • Address unknown size state by growing buffer if needed or using stream decompression API via ZSTD_decompressStream
    • The HDF5 library should know the expected number of bytes for a chunk. We should not be relying on the filter to figure this out.

References

[1] https://facebook.github.io/zstd/zstd_manual.html

Lingua principale
CMake
Stelle
21
Fork
27
Nessuna PR unita negli ultimi 30g

Preparare l'ambiente

Questo progetto non fornisce container di sviluppo, Dockerfile né guida per i contributori, quindi l'ambiente è a tuo carico: parti dal suo README e consulta la nostra guida al primo contributo per i passaggi generali.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di HDFGroup/hdf5_plugins

Tutte le issue di HDFGroup/hdf5_plugins

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.