Consider a Zstd.compressFile(Path, Path, int) convenience wrapper - scope question
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 38/100
Research direction
Read ZstdCompressStream and the existing ZstdOutputStream/ZstdInputStream wrappers first, then the prototype loop in the issue body, to understand what Zstd.compressFile(Path, Path, int) would wrap. The blocking step is the maintainer's own open scope question: decide whether the method or a documented recipe in docs/zero-copy.md/how-to.md is wanted, plus the buffer-size and posix_madvise sub-questions. Done means a maintainer decision on the API surface; no pull request should be written before that.
Written by the indexing model from the issue text.
Description
What
ZstdCompressStream already has everything needed for zero-copy file-to-file compression of arbitrarily large files - it's the exact engine behind every timing harness in a recent mmap-vs-zstd-jni investigation:
try (FileChannel ch = FileChannel.open(src, StandardOpenOption.READ);
Arena arena = Arena.ofConfined();
ZstdCompressStream cs = new ZstdCompressStream(level)) {
MemorySegment mapped = ch.map(FileChannel.MapMode.READ_ONLY, 0, ch.size(), arena);
MemorySegment dst = arena.allocate(dstBufferSize);
long off = 0;
ZstdStreamResult r;
do {
r = cs.compress(dst, mapped.asSlice(off), ZstdEndDirective.END);
off += r.bytesConsumed();
// drain dst to the destination file, once opened outside the loop
} while (!r.isComplete());
}
Every caller who wants this today has to hand-roll it. A Zstd.compressFile(Path src, Path dst, int level) (+ matching decompressFile) would remove that boilerplate, and is what backs the actual measured win in the investigation this came out of: mmap + posix_madvise(WILLNEED) beat zstd-jni's stream API by ~45% at 2.25GiB and ~13-24% at 10GiB (macOS, single-machine, see the mmap-vs-zstd-jni session for full numbers and caveats - not yet cross-OS validated).
Open scope question (why this isn't a clear-cut yes)
A Path, Path signature only covers "both ends are real, mappable files." It cannot express - and structurally never will, since you can't mmap a socket or a pipe - use cases like "read from a network socket, compress, write to a file" or "compress from an in-memory buffer, stream the result to a client." Those are already served today by the existing ZstdOutputStream/ZstdInputStream java.io wrappers, which accept arbitrary InputStream/OutputStream:
try (InputStream in = socket.getInputStream();
ZstdOutputStream out = new ZstdOutputStream(Files.newOutputStream(dst), level)) {
in.transferTo(out); // works today, just without the mmap/madvise win - and note
// transferTo()'s hardcoded 8KB buffer measurably hurts vs. a
// larger explicit buffer, see the same investigation
}
So the real question isn't "can we support arbitrary sources/destinations" (already true) - it's whether the narrower "both ends are local files, and I want the fastest path" case is common and valuable enough to earn a dedicated convenience method on top of what ZstdCompressStream already provides, or whether that's better left as a documented recipe (e.g. in docs/zero-copy.md or how-to.md) rather than new permanent API surface.
If pursued, related open questions
- Should the destination scratch-buffer size be caller-configurable (
compressFile(Path, Path, int level, long dstBufferSize)), or fixed with a sane default? (Buffer size measurably affects throughput - see thetransferTo()8KB-vs-8MiB finding above.) - Should it apply
posix_madvise/equivalent internally? The measured win is real on macOS but untested on Linux (same POSIX API, plausibly less headroom since Linux already won without any hint) and has no direct equivalent on Windows (PrefetchVirtualMemoryis a structurally different API, untested).
- Dominant language
- Java
- Stars
- 5
- Forks
- 1
- Avg merge
- 13d 18h
- Merged PRs (30d)
- 2
Getting set up
This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from dfa1/zstd-ffm
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
FasterXML/jackson-databind#6443 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
finos/fluxnova-bpm-platform#387 ·
Maintainers usually reply within 4 days
-
Difficulty 1/5 Under an hour Newbie friendliness 88/100
-
Difficulty 2/5 Under an hour Newbie friendliness 84/100
cpp-best-practices/project_validator#251 ·
Maintainers usually reply within 1 day