Hacktoberfest 2026 : les issues que les mainteneurs ont marquées pour octobre, ouvertes et accessibles aux débutants. Parcourir les issues Hacktoberfest

Remote file access: parallel ranged downloads — downloadTo(...) with parallelism(K), ~3× on modern hosts

Ouverte
  • Quelque chose ne va pas ?
#176 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
5/5
Temps estimé
Plus d'une semaine
Accessibilité débutants
38/100
Type d'issue
Fonctionnalité
Clarté
Plutôt claire
Activité
Active
Stack technique
java
Domaine
documentation, networking, testing

Piste de recherche

Start by reviewing the existing ranged openStream() and downloadTo flow from #146 and #147, then inspect the builder's credential lifecycle before settling how sibling connections are created. Use FakeWsmanServer and WinRMLiveTest for the concurrent-range, quota, cleanup, and checksum cases. Done means the stated acceptance criteria pass and files.md/file-transfers.md document the threshold and quota behavior.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

Follow-up of #147 (downloadFile), building on the byte-range reads of #146. Related: #175 (raw bytes instead of base64), which raises the per-connection rate and composes with this.

Context

A remote file read through the command shell is capped per output stream: the WinRM service reads a command's stdout pipe itself, at most 32 KiB per read and about 60 reads per second. With base64 that gives about 1.45 MB/s of file data per connection, whatever the network or the client (see the Performance section of files.md).

Each command gets its own pipe, so several commands in parallel, each reading one byte range on its own connection, add up. Measured with a 40 MiB random file split into K ranges, one WinRMClient per range (file(p).offset(o).length(l).openStream()), reassembled result verified by SHA-256 (all OK):

Host K=1 K=2 K=4 K=8
Windows Server 2022, PS 5.1 1.46 MB/s 2.95 (2.0×) 4.92 (3.4×) 4.78 (no gain)
Windows Server 2019, PS 5.1 1.43 2.67 (1.9×) 4.57 (3.2×) 4.64 (no gain)
Windows Server 2008 R2, PS 2.0 1.46 2.82 (1.9×) 3.73 (2.6×) fails: MaxShellsPerUser = 5 (WSManFault 2150859173)
  • Near-linear up to K=2, then a second ceiling around 4.6–4.9 MB/s per host from K=4 on (per-connection rate drops to ~0.6 MB/s at K=8). Its cause is not identified yet (host CPU on a 4-vCPU VM, NTLM sealing in the WinRM service, network?). The client is not it (it takes in 24–31 MB/s against a local fake server).
  • Opening, authenticating and creating a shell for all K connections took 0.2–0.9 s in total.
  • Windows 2008 R2's default MaxShellsPerUser is 5, and the caller's own client already holds one: at most 4 extra connections there. Its MaxConcurrentOperationsPerUser (15) did not trigger at K=4.

Proposal

client.file("D:\\logs\\huge.log")
    .parallelism(4)                 // default 1 = today's single stream
    .downloadTo(Path.of("huge.log"));
  1. One setup call on the caller's client: the file size and the remote SHA-256 (the digest #147 already probes).
  2. Split into K ranges; each worker opens its own connection (same host, credentials and settings as the caller's client), reads its range with the existing ranged openStream(), and writes it at its position in the temporary destination file (FileChannel.write(buffer, position)), so no reordering buffer is needed.
  3. Once all ranges are in, the #147 flow unchanged: digest the whole temporary file, compare, fsync, ATOMIC_MOVE. A file modified during the transfer shows up as a digest mismatch.

Rules:

  • Only above a size threshold (e.g. 8 MiB): below it, the extra connections cost more than they save.
  • Graceful degradation: when opening a worker connection is refused by a quota (2150859173 concurrent shells, 2150859174 concurrent operations), continue with the workers already open instead of failing — and never retry past the quota in a tight loop.
  • Recommended value 4 (the measured sweet spot); document that more does not help and that 2008 R2 allows at most 4 extra.
  • Memory bounded by one buffer per worker; the temporary file is deleted on any failure (as in #147).
  • openStream() stays single-stream (parallel streaming would need a reordering buffer — out of scope).

Design question to settle first

A WinRMClient is a serial channel and cannot open further connections today: the password is consumed by build(). Options:

  • the builder keeps the credentials (it already needs them for re-authentication?) so the client can open sibling connections internally — security review needed for how long the char[] lives and how it is wiped on close();
  • or the caller passes the extra clients/a factory (parallelism(Supplier<WinRMClient>, 4)) — no credential retention, clumsier API.

Tests & docs

  • FakeWsmanServer: several concurrent connections, ranges reassembled byte-exact, a quota refusal on the 3rd connection → download completes with 2 workers, a worker failure midway → no destination file and no temporary file left.
  • WinRMLiveTest: parallel download of a ≥ 40 MiB file on a real host, SHA-256 verified.
  • files.md / file-transfers.md: the scaling table above (re-measured), the threshold, the quota limits.

Acceptance criteria

  • downloadTo with parallelism(4) is ≥ 2.5× faster than parallelism(1) on a 40 MiB file on a modern host, byte-exact.
  • On Windows 2008 R2 with default quotas, parallelism(8) degrades to what the quota allows instead of failing.
  • mvn verify site green.

🤖 Generated with Claude Code

Langage dominant
Java
Étoiles
15
Forks
4
Merge moyen
11 h 58 min
PR mergées (30 j)
20

Préparer son environnement

Ce projet ne fournit ni conteneur de développement, ni Dockerfile, ni guide de contribution : l'installation est à votre charge. Commencez par son README, et consultez notre guide de la première contribution pour les étapes générales.

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de MetricsHub/winrm-java

Toutes les issues de MetricsHub/winrm-java

Issues similaires

Plus d'issues Java

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.