Hacktoberfest 2026: as issues que os mantenedores marcaram para outubro, abertas e boas para iniciantes. Ver issues do Hacktoberfest

Remote file access: parallel ranged downloads — downloadTo(...) with parallelism(K), ~3× on modern hosts

Aberta
  • Algo errado?
#176 0 comentários 0 reações 0 responsáveis Ver no GitHub

Mantenedores costumam responder em até 1 dia

Ninguém assumiu esta issue ainda.

Avaliação

Dificuldade
5/5
Tempo estimado
Mais de uma semana
Facilidade para iniciantes
38/100
Tipo de issue
Funcionalidade
Clareza
Razoavelmente clara
Status de atividade
Ativa
Stack de tecnologia
java
Domínio
documentation, networking, testing

Direção de pesquisa

Start by reviewing the existing ranged openStream() and downloadTo flow from #146 and #147, then inspect the builder's credential lifecycle before settling how sibling connections are created. Use FakeWsmanServer and WinRMLiveTest for the concurrent-range, quota, cleanup, and checksum cases. Done means the stated acceptance criteria pass and files.md/file-transfers.md document the threshold and quota behavior.

Escrita pelo modelo de indexação a partir do texto da issue.

Descrição

Follow-up of #147 (downloadFile), building on the byte-range reads of #146. Related: #175 (raw bytes instead of base64), which raises the per-connection rate and composes with this.

Context

A remote file read through the command shell is capped per output stream: the WinRM service reads a command's stdout pipe itself, at most 32 KiB per read and about 60 reads per second. With base64 that gives about 1.45 MB/s of file data per connection, whatever the network or the client (see the Performance section of files.md).

Each command gets its own pipe, so several commands in parallel, each reading one byte range on its own connection, add up. Measured with a 40 MiB random file split into K ranges, one WinRMClient per range (file(p).offset(o).length(l).openStream()), reassembled result verified by SHA-256 (all OK):

Host K=1 K=2 K=4 K=8
Windows Server 2022, PS 5.1 1.46 MB/s 2.95 (2.0×) 4.92 (3.4×) 4.78 (no gain)
Windows Server 2019, PS 5.1 1.43 2.67 (1.9×) 4.57 (3.2×) 4.64 (no gain)
Windows Server 2008 R2, PS 2.0 1.46 2.82 (1.9×) 3.73 (2.6×) fails: MaxShellsPerUser = 5 (WSManFault 2150859173)
  • Near-linear up to K=2, then a second ceiling around 4.6–4.9 MB/s per host from K=4 on (per-connection rate drops to ~0.6 MB/s at K=8). Its cause is not identified yet (host CPU on a 4-vCPU VM, NTLM sealing in the WinRM service, network?). The client is not it (it takes in 24–31 MB/s against a local fake server).
  • Opening, authenticating and creating a shell for all K connections took 0.2–0.9 s in total.
  • Windows 2008 R2's default MaxShellsPerUser is 5, and the caller's own client already holds one: at most 4 extra connections there. Its MaxConcurrentOperationsPerUser (15) did not trigger at K=4.

Proposal

client.file("D:\\logs\\huge.log")
    .parallelism(4)                 // default 1 = today's single stream
    .downloadTo(Path.of("huge.log"));
  1. One setup call on the caller's client: the file size and the remote SHA-256 (the digest #147 already probes).
  2. Split into K ranges; each worker opens its own connection (same host, credentials and settings as the caller's client), reads its range with the existing ranged openStream(), and writes it at its position in the temporary destination file (FileChannel.write(buffer, position)), so no reordering buffer is needed.
  3. Once all ranges are in, the #147 flow unchanged: digest the whole temporary file, compare, fsync, ATOMIC_MOVE. A file modified during the transfer shows up as a digest mismatch.

Rules:

  • Only above a size threshold (e.g. 8 MiB): below it, the extra connections cost more than they save.
  • Graceful degradation: when opening a worker connection is refused by a quota (2150859173 concurrent shells, 2150859174 concurrent operations), continue with the workers already open instead of failing — and never retry past the quota in a tight loop.
  • Recommended value 4 (the measured sweet spot); document that more does not help and that 2008 R2 allows at most 4 extra.
  • Memory bounded by one buffer per worker; the temporary file is deleted on any failure (as in #147).
  • openStream() stays single-stream (parallel streaming would need a reordering buffer — out of scope).

Design question to settle first

A WinRMClient is a serial channel and cannot open further connections today: the password is consumed by build(). Options:

  • the builder keeps the credentials (it already needs them for re-authentication?) so the client can open sibling connections internally — security review needed for how long the char[] lives and how it is wiped on close();
  • or the caller passes the extra clients/a factory (parallelism(Supplier<WinRMClient>, 4)) — no credential retention, clumsier API.

Tests & docs

  • FakeWsmanServer: several concurrent connections, ranges reassembled byte-exact, a quota refusal on the 3rd connection → download completes with 2 workers, a worker failure midway → no destination file and no temporary file left.
  • WinRMLiveTest: parallel download of a ≥ 40 MiB file on a real host, SHA-256 verified.
  • files.md / file-transfers.md: the scaling table above (re-measured), the threshold, the quota limits.

Acceptance criteria

  • downloadTo with parallelism(4) is ≥ 2.5× faster than parallelism(1) on a 40 MiB file on a modern host, byte-exact.
  • On Windows 2008 R2 with default quotas, parallelism(8) degrades to what the quota allows instead of failing.
  • mvn verify site green.

🤖 Generated with Claude Code

Linguagem predominante
Java
Estrelas
15
Forks
4
Merge médio
11h 58min
PRs com merge (30d)
20

Preparar o ambiente

Este projeto não oferece contêiner de desenvolvimento, Dockerfile nem guia de contribuição, então a configuração fica por sua conta: comece pelo README e veja nosso guia da primeira contribuição para os passos gerais.

Primeiros passos

  1. Leia a issue inteira e depois o guia de contribuição do projeto.
  2. Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
  3. Faça um fork do repositório e trabalhe em uma branch.
  4. Abra um pull request que referencie o número da issue.

Mais de MetricsHub/winrm-java

Todas as issues de MetricsHub/winrm-java

Issues semelhantes

Mais issues de Java

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.