Hacktoberfest 2026: die Issues, die Maintainer für den Oktober markiert haben – offen und einsteigerfreundlich. Hacktoberfest-Issues durchsuchen

Remote file access: parallel ranged downloads — downloadTo(...) with parallelism(K), ~3× on modern hosts

Offen
  • Stimmt etwas nicht?
#176 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Anfängerfreundlichkeit
38/100
Issue-Typ
Feature
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
java
Bereich
documentation, networking, testing

Rechercherichtung

Start by reviewing the existing ranged openStream() and downloadTo flow from #146 and #147, then inspect the builder's credential lifecycle before settling how sibling connections are created. Use FakeWsmanServer and WinRMLiveTest for the concurrent-range, quota, cleanup, and checksum cases. Done means the stated acceptance criteria pass and files.md/file-transfers.md document the threshold and quota behavior.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Follow-up of #147 (downloadFile), building on the byte-range reads of #146. Related: #175 (raw bytes instead of base64), which raises the per-connection rate and composes with this.

Context

A remote file read through the command shell is capped per output stream: the WinRM service reads a command's stdout pipe itself, at most 32 KiB per read and about 60 reads per second. With base64 that gives about 1.45 MB/s of file data per connection, whatever the network or the client (see the Performance section of files.md).

Each command gets its own pipe, so several commands in parallel, each reading one byte range on its own connection, add up. Measured with a 40 MiB random file split into K ranges, one WinRMClient per range (file(p).offset(o).length(l).openStream()), reassembled result verified by SHA-256 (all OK):

Host K=1 K=2 K=4 K=8
Windows Server 2022, PS 5.1 1.46 MB/s 2.95 (2.0×) 4.92 (3.4×) 4.78 (no gain)
Windows Server 2019, PS 5.1 1.43 2.67 (1.9×) 4.57 (3.2×) 4.64 (no gain)
Windows Server 2008 R2, PS 2.0 1.46 2.82 (1.9×) 3.73 (2.6×) fails: MaxShellsPerUser = 5 (WSManFault 2150859173)
  • Near-linear up to K=2, then a second ceiling around 4.6–4.9 MB/s per host from K=4 on (per-connection rate drops to ~0.6 MB/s at K=8). Its cause is not identified yet (host CPU on a 4-vCPU VM, NTLM sealing in the WinRM service, network?). The client is not it (it takes in 24–31 MB/s against a local fake server).
  • Opening, authenticating and creating a shell for all K connections took 0.2–0.9 s in total.
  • Windows 2008 R2's default MaxShellsPerUser is 5, and the caller's own client already holds one: at most 4 extra connections there. Its MaxConcurrentOperationsPerUser (15) did not trigger at K=4.

Proposal

client.file("D:\\logs\\huge.log")
    .parallelism(4)                 // default 1 = today's single stream
    .downloadTo(Path.of("huge.log"));
  1. One setup call on the caller's client: the file size and the remote SHA-256 (the digest #147 already probes).
  2. Split into K ranges; each worker opens its own connection (same host, credentials and settings as the caller's client), reads its range with the existing ranged openStream(), and writes it at its position in the temporary destination file (FileChannel.write(buffer, position)), so no reordering buffer is needed.
  3. Once all ranges are in, the #147 flow unchanged: digest the whole temporary file, compare, fsync, ATOMIC_MOVE. A file modified during the transfer shows up as a digest mismatch.

Rules:

  • Only above a size threshold (e.g. 8 MiB): below it, the extra connections cost more than they save.
  • Graceful degradation: when opening a worker connection is refused by a quota (2150859173 concurrent shells, 2150859174 concurrent operations), continue with the workers already open instead of failing — and never retry past the quota in a tight loop.
  • Recommended value 4 (the measured sweet spot); document that more does not help and that 2008 R2 allows at most 4 extra.
  • Memory bounded by one buffer per worker; the temporary file is deleted on any failure (as in #147).
  • openStream() stays single-stream (parallel streaming would need a reordering buffer — out of scope).

Design question to settle first

A WinRMClient is a serial channel and cannot open further connections today: the password is consumed by build(). Options:

  • the builder keeps the credentials (it already needs them for re-authentication?) so the client can open sibling connections internally — security review needed for how long the char[] lives and how it is wiped on close();
  • or the caller passes the extra clients/a factory (parallelism(Supplier<WinRMClient>, 4)) — no credential retention, clumsier API.

Tests & docs

  • FakeWsmanServer: several concurrent connections, ranges reassembled byte-exact, a quota refusal on the 3rd connection → download completes with 2 workers, a worker failure midway → no destination file and no temporary file left.
  • WinRMLiveTest: parallel download of a ≥ 40 MiB file on a real host, SHA-256 verified.
  • files.md / file-transfers.md: the scaling table above (re-measured), the threshold, the quota limits.

Acceptance criteria

  • downloadTo with parallelism(4) is ≥ 2.5× faster than parallelism(1) on a 40 MiB file on a modern host, byte-exact.
  • On Windows 2008 R2 with default quotas, parallelism(8) degrades to what the quota allows instead of failing.
  • mvn verify site green.

🤖 Generated with Claude Code

Vorherrschende Sprache
Java
Sterne
15
Forks
4
Ø Merge
11 Std. 58 Min.
Gemergte PRs (30 T.)
20

Entwicklungsumgebung

Dieses Projekt bietet weder Dev-Container noch Dockerfile noch Beitragsleitfaden – die Einrichtung liegt bei Ihnen. Beginnen Sie mit der README; die allgemeinen Schritte stehen in unserem Leitfaden für den ersten Beitrag.

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus MetricsHub/winrm-java

Alle Issues in MetricsHub/winrm-java

Ähnliche Issues

Weitere Issues zu Java

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.