Remote file access: parallel ranged downloads — downloadTo(...) with parallelism(K), ~3× on modern hosts

Start by reviewing the existing ranged openStream() and downloadTo flow from #146 and #147, then inspect the builder's credential lifecycle before settling how sibling connections are created. Use FakeWsmanServer and WinRMLiveTest for the concurrent-range, quota, cleanup, and checksum cases. Done means the stated acceptance criteria pass and files.md/file-transfers.md document the threshold and quota behavior.

Follow-up of #147 (downloadFile), building on the byte-range reads of #146. Related: #175 (raw bytes instead of base64), which raises the per-connection rate and composes with this.

Context

A remote file read through the command shell is capped per output stream: the WinRM service reads a command's stdout pipe itself, at most 32 KiB per read and about 60 reads per second. With base64 that gives about 1.45 MB/s of file data per connection, whatever the network or the client (see the Performance section of files.md).

Each command gets its own pipe, so several commands in parallel, each reading one byte range on its own connection, add up. Measured with a 40 MiB random file split into K ranges, one WinRMClient per range (file(p).offset(o).length(l).openStream()), reassembled result verified by SHA-256 (all OK):

• Near-linear up to K=2, then a second ceiling around 4.6–4.9 MB/s per host from K=4 on (per-connection rate drops to ~0.6 MB/s at K=8). Its cause is not identified yet (host CPU on a 4-vCPU VM, NTLM sealing in the WinRM service, network?). The client is not it (it takes in 24–31 MB/s against a local fake server).
• Opening, authenticating and creating a shell for all K connections took 0.2–0.9 s in total.
• Windows 2008 R2's default MaxShellsPerUser is 5, and the caller's own client already holds one: at most 4 extra connections there. Its MaxConcurrentOperationsPerUser (15) did not trigger at K=4.

Proposal

client.file("D:\\logs\\huge.log")
    .parallelism(4)                 // default 1 = today's single stream
    .downloadTo(Path.of("huge.log"));

1. One setup call on the caller's client: the file size and the remote SHA-256 (the digest #147 already probes).
2. Split into K ranges; each worker opens its own connection (same host, credentials and settings as the caller's client), reads its range with the existing ranged openStream(), and writes it at its position in the temporary destination file (FileChannel.write(buffer, position)), so no reordering buffer is needed.
3. Once all ranges are in, the #147 flow unchanged: digest the whole temporary file, compare, fsync, ATOMIC_MOVE. A file modified during the transfer shows up as a digest mismatch.

Rules:

• Only above a size threshold (e.g. 8 MiB): below it, the extra connections cost more than they save.
• Graceful degradation: when opening a worker connection is refused by a quota (2150859173 concurrent shells, 2150859174 concurrent operations), continue with the workers already open instead of failing — and never retry past the quota in a tight loop.
• Recommended value 4 (the measured sweet spot); document that more does not help and that 2008 R2 allows at most 4 extra.
• Memory bounded by one buffer per worker; the temporary file is deleted on any failure (as in #147).
• openStream() stays single-stream (parallel streaming would need a reordering buffer — out of scope).

Design question to settle first

A WinRMClient is a serial channel and cannot open further connections today: the password is consumed by build(). Options:

• the builder keeps the credentials (it already needs them for re-authentication?) so the client can open sibling connections internally — security review needed for how long the char[] lives and how it is wiped on close();
• or the caller passes the extra clients/a factory (parallelism(Supplier<WinRMClient>, 4)) — no credential retention, clumsier API.

Tests & docs

• FakeWsmanServer: several concurrent connections, ranges reassembled byte-exact, a quota refusal on the 3rd connection → download completes with 2 workers, a worker failure midway → no destination file and no temporary file left.
• WinRMLiveTest: parallel download of a ≥ 40 MiB file on a real host, SHA-256 verified.
• files.md / file-transfers.md: the scaling table above (re-measured), the threshold, the quota limits.

Acceptance criteria

• downloadTo with parallelism(4) is ≥ 2.5× faster than parallelism(1) on a 40 MiB file on a modern host, byte-exact.
• On Windows 2008 R2 with default quotas, parallelism(8) degrades to what the quota allows instead of failing.
• mvn verify site green.

🤖 Generated with Claude Code