ZSTD_compressSequencesAndLiterals() does not enforce its documented litCapacity >= litSize + 8 contract — the implemented check is 8 bytes weaker than the API documentation, its own error message, and the introducing commit

Open Beginner friendly
#4,779 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
1/5
Estimated time
Under an hour
Newbie friendliness
88/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
c
Domain
backend

Research direction

Start at the litCapacity check in lib/compress/zstd_compress.c:8128 and compare it with the contract in lib/zstd.h. Build with make -C lib, run the reproducer described in the issue, and verify that capacities below litSize + 8 are rejected while litSize + 8 still succeeds.

Written by the indexing model from the issue text.

Description

Describe the bug

ZSTD_compressSequencesAndLiterals() does not enforce its own documented
litBufCapacity >= litSize + 8 contract: the implemented check in
lib/compress/zstd_compress.c:8128 only tests litCapacity < litSize,
i.e. it is 8 bytes weaker than the contract stated in three independent
places:

  1. API documentationlib/zstd.h line 1700:

    @litBufCapacity must be at least 8 bytes larger than @litSize.

  2. The introducing commitb7a9e69d ("added parameter litCapacity
    to ZSTD_compressSequencesAndLiterals() to enforce the litCapacity >=
    litSize+8 condition
    .")
  3. The check's own error message

    "literals buffer is not large enough: must be at least 8 bytes larger
    than litSize
    (risk of read out-of-bound)"

/* lib/compress/zstd_compress.c:8126-8130 */
if (litCapacity < litSize) {   /* <-- should be: litCapacity < litSize + 8 */
    RETURN_ERROR(workSpace_tooSmall,
        "literals buffer is not large enough: "
        "must be at least 8 bytes larger than litSize (risk of read out-of-bound)");
}

Calls that violate the documented contract by up to 8 bytes
(litSize <= litCapacity < litSize + 8) are silently accepted and produce
valid frames, instead of being rejected with the documented
ZSTD_error_workSpace_tooSmall.

To Reproduce

Steps to reproduce the behavior:

  1. Clone zstd and build the static library:
git clone https://github.com/facebook/zstd && cd zstd
make -C lib libzstd.a    # or: make -C lib
  1. Save the following as reproducer.c in the repo root:
/* reproducer.c — demonstrates that ZSTD_compressSequencesAndLiterals()
 * accepts litCapacity values that violate its documented
 * "at least 8 bytes larger than litSize" contract. */
#define ZSTD_STATIC_LINKING_ONLY
#include "zstd.h"
#include <stdio.h>

int main(void)
{
    /* one explicit-delimiter block: 284 literals + a 16-byte match at
     * offset 8, then 16 trailing literals carried by the delimiter */
    ZSTD_Sequence seqs[2] = {
        { /*offset=*/8, /*litLength=*/284, /*matchLength=*/16, /*rep=*/0 },
        { /*offset=*/0, /*litLength=*/ 16, /*matchLength=*/ 0, /*rep=*/0 },
    };
    unsigned char literals[300 + 8];        /* +8 so the probe itself is memory-safe */
    unsigned char dst[1024];
    const size_t litSize = 300, srcSize = 316;
    size_t i, r;
    ZSTD_CCtx* cctx = ZSTD_createCCtx();

    for (i = 0; i < litSize; i++) literals[i] = (unsigned char)('a' + (i % 26));

    ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, 3);
    ZSTD_CCtx_setParameter(cctx, ZSTD_c_blockDelimiters, ZSTD_sf_explicitBlockDelimiters);

    /* documented minimum is litSize + 8; pass litSize + 7 on purpose */
    r = ZSTD_compressSequencesAndLiterals(cctx, dst, sizeof(dst),
                                          seqs, 2, literals, litSize,
                                          litSize + 7, srcSize);
    if (ZSTD_isError(r))
        printf("litCapacity = litSize+7 : rejected (%s) -> contract enforced\n",
               ZSTD_getErrorName(r));
    else
        printf("litCapacity = litSize+7 : ACCEPTED (%zu-byte frame) -> "
               "documented contract NOT enforced\n", r);

    ZSTD_freeCCtx(cctx);
    return 0;
}
  1. Compile and run:
gcc -O2 -Ilib reproducer.c lib/libzstd.a -o reproducer -lpthread
./reproducer
  1. Scroll up on the log to the reproducer's output line:
litCapacity = litSize+7 : ACCEPTED (223-byte frame) -> documented contract NOT enforced
  1. See error: there is no error — the contract-violating call is
    accepted and produces a valid, round-tripping 223-byte frame. The same
    happens with litCapacity = litSize (violating by the full 8 bytes).
    With the one-line fix below, both calls are rejected with
    ZSTD_error_workSpace_tooSmall while a compliant
    litCapacity = litSize + 8 call keeps working.

Expected behavior

ZSTD_compressSequencesAndLiterals() should reject litCapacity values
below litSize + 8 with ZSTD_error_workSpace_tooSmall, exactly as its
documentation (lib/zstd.h:1700), its introducing commit (b7a9e69d) and
its own error message all promise:

--- a/lib/compress/zstd_compress.c
+++ b/lib/compress/zstd_compress.c
@@ -8125,7 +8125,7 @@ ZSTD_compressSequencesAndLiterals(ZSTD_CCtx* cctx,
     /* Transparent initialization stage, same as compressStream2() */
     DEBUGLOG(4, "ZSTD_compressSequencesAndLiterals (dstCapacity=%zu)", dstCapacity);
     assert(cctx != NULL);
-    if (litCapacity < litSize) {
+    if (litCapacity < litSize + 8) {
         RETURN_ERROR(workSpace_tooSmall, "literals buffer is not large enough: must be at least 8 bytes larger than litSize (risk of read out-of-bound)");
     }
     FORWARD_IF_ERROR(ZSTD_CCtx_init_compressStream2(cctx, ZSTD_e_end, decompressedSize), "CCtx initialization failed");

Screenshots and charts

Not applicable — the reproducer output above shows the behavior
(ACCEPTED ... contract NOT enforced on current dev; after the one-line
fix the same call prints rejected (workSpace buffer is not large enough) -> contract enforced).

Desktop (please complete the following information):

  • OS: Linux (x86-64)
  • Version: Ubuntu 20.04
  • Compiler: gcc 9.4.0
  • Flags: -O2 for the reproducer; the fuller PoC (see Additional context) was additionally run with -g -O1 -fsanitize=address,undefined -fno-sanitize-recover=all
  • Other relevant hardware specs: n/a (no hardware dependency)
  • Build system: make -C lib (static library lib/libzstd.a); library sources compiled directly in the PoC verification as well

Additional context

  • Affected version: current dev (verified byte-identical at
    lib/compress/zstd_compress.c:8128 on dev @ d9c0c7e2, 2026-09-09).
    I could not find an existing issue or PR covering this mismatch (closest:
    open PR #4741 fixes a different validation gap in the same function).
  • Full verification PoC (attached): a three-phase test that (1) confirms a
    compliant litCapacity = litSize + 8 call succeeds, (2) demonstrates both
    litSize + 7 and litSize violations are accepted on the unpatched
    build and rejected with ZSTD_error_workSpace_tooSmall after the
    one-line fix, and (3) runs an ASan sweep of 360 configurations
    (30 literal sizes x 3 data patterns x 4 levels, literals buffer of
    exactly litSize bytes) probing for reads inside the documented-but-
    unenforced margin [litSize, litSize+8).
  • Honest impact statement: the sweep produced zero ASan reports —
    literal reads are currently bounded by litSize in the existing encoder
    paths, so this is a contract-enforcement / defense-in-depth gap rather
    than a demonstrated out-of-bounds read. The +8 slack exists because the
    literal encoders historically used 8-byte-wide loads near the tail; the
    check is the only guard between the caller and
    ZSTD_entropyCompressSeqStore_internal(), and today it enforces none of
    the slack that the documentation, the error message and commit
    b7a9e69d all promise.
  • Suggested title for the issue: "ZSTD_compressSequencesAndLiterals() does
    not enforce the documented litBufCapacity >= litSize + 8 contract (check
    is missing the + 8)".
Dominant language
C
Stars
27.9k
Forks
2.6k
Avg merge
1d 3h
Merged PRs (30d)
8

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from facebook/zstd

All issues in facebook/zstd

Similar issues

More C issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.