Consider a Zstd.compressFile(Path, Path, int) convenience wrapper - scope question

Open
#94 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
38/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Quiet
Tech stack
java
Domain
api

Research direction

Read ZstdCompressStream and the existing ZstdOutputStream/ZstdInputStream wrappers first, then the prototype loop in the issue body, to understand what Zstd.compressFile(Path, Path, int) would wrap. The blocking step is the maintainer's own open scope question: decide whether the method or a documented recipe in docs/zero-copy.md/how-to.md is wanted, plus the buffer-size and posix_madvise sub-questions. Done means a maintainer decision on the API surface; no pull request should be written before that.

Written by the indexing model from the issue text.

Description

enhancement help wanted

What

ZstdCompressStream already has everything needed for zero-copy file-to-file compression of arbitrarily large files - it's the exact engine behind every timing harness in a recent mmap-vs-zstd-jni investigation:

try (FileChannel ch = FileChannel.open(src, StandardOpenOption.READ);
     Arena arena = Arena.ofConfined();
     ZstdCompressStream cs = new ZstdCompressStream(level)) {
    MemorySegment mapped = ch.map(FileChannel.MapMode.READ_ONLY, 0, ch.size(), arena);
    MemorySegment dst = arena.allocate(dstBufferSize);
    long off = 0;
    ZstdStreamResult r;
    do {
        r = cs.compress(dst, mapped.asSlice(off), ZstdEndDirective.END);
        off += r.bytesConsumed();
        // drain dst to the destination file, once opened outside the loop
    } while (!r.isComplete());
}

Every caller who wants this today has to hand-roll it. A Zstd.compressFile(Path src, Path dst, int level) (+ matching decompressFile) would remove that boilerplate, and is what backs the actual measured win in the investigation this came out of: mmap + posix_madvise(WILLNEED) beat zstd-jni's stream API by ~45% at 2.25GiB and ~13-24% at 10GiB (macOS, single-machine, see the mmap-vs-zstd-jni session for full numbers and caveats - not yet cross-OS validated).

Open scope question (why this isn't a clear-cut yes)

A Path, Path signature only covers "both ends are real, mappable files." It cannot express - and structurally never will, since you can't mmap a socket or a pipe - use cases like "read from a network socket, compress, write to a file" or "compress from an in-memory buffer, stream the result to a client." Those are already served today by the existing ZstdOutputStream/ZstdInputStream java.io wrappers, which accept arbitrary InputStream/OutputStream:

try (InputStream in = socket.getInputStream();
     ZstdOutputStream out = new ZstdOutputStream(Files.newOutputStream(dst), level)) {
    in.transferTo(out);   // works today, just without the mmap/madvise win - and note
                           // transferTo()'s hardcoded 8KB buffer measurably hurts vs. a
                           // larger explicit buffer, see the same investigation
}

So the real question isn't "can we support arbitrary sources/destinations" (already true) - it's whether the narrower "both ends are local files, and I want the fastest path" case is common and valuable enough to earn a dedicated convenience method on top of what ZstdCompressStream already provides, or whether that's better left as a documented recipe (e.g. in docs/zero-copy.md or how-to.md) rather than new permanent API surface.

If pursued, related open questions

  • Should the destination scratch-buffer size be caller-configurable (compressFile(Path, Path, int level, long dstBufferSize)), or fixed with a sane default? (Buffer size measurably affects throughput - see the transferTo() 8KB-vs-8MiB finding above.)
  • Should it apply posix_madvise/equivalent internally? The measured win is real on macOS but untested on Linux (same POSIX API, plausibly less headroom since Linux already won without any hint) and has no direct equivalent on Windows (PrefetchVirtualMemory is a structurally different API, untested).
Dominant language
Java
Stars
5
Forks
1
Avg merge
13d 18h
Merged PRs (30d)
2

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from dfa1/zstd-ffm

All issues in dfa1/zstd-ffm

Similar issues

More Java issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.