Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

Remote file access: parallel ranged downloads — downloadTo(...) with parallelism(K), ~3× on modern hosts

Đang mở
  • Có gì không ổn?
#176 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
38/100
Loại issue
Tính năng
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
java
Lĩnh vực
documentation, networking, testing

Hướng nghiên cứu

Start by reviewing the existing ranged openStream() and downloadTo flow from #146 and #147, then inspect the builder's credential lifecycle before settling how sibling connections are created. Use FakeWsmanServer and WinRMLiveTest for the concurrent-range, quota, cleanup, and checksum cases. Done means the stated acceptance criteria pass and files.md/file-transfers.md document the threshold and quota behavior.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Follow-up of #147 (downloadFile), building on the byte-range reads of #146. Related: #175 (raw bytes instead of base64), which raises the per-connection rate and composes with this.

Context

A remote file read through the command shell is capped per output stream: the WinRM service reads a command's stdout pipe itself, at most 32 KiB per read and about 60 reads per second. With base64 that gives about 1.45 MB/s of file data per connection, whatever the network or the client (see the Performance section of files.md).

Each command gets its own pipe, so several commands in parallel, each reading one byte range on its own connection, add up. Measured with a 40 MiB random file split into K ranges, one WinRMClient per range (file(p).offset(o).length(l).openStream()), reassembled result verified by SHA-256 (all OK):

Host K=1 K=2 K=4 K=8
Windows Server 2022, PS 5.1 1.46 MB/s 2.95 (2.0×) 4.92 (3.4×) 4.78 (no gain)
Windows Server 2019, PS 5.1 1.43 2.67 (1.9×) 4.57 (3.2×) 4.64 (no gain)
Windows Server 2008 R2, PS 2.0 1.46 2.82 (1.9×) 3.73 (2.6×) fails: MaxShellsPerUser = 5 (WSManFault 2150859173)
  • Near-linear up to K=2, then a second ceiling around 4.6–4.9 MB/s per host from K=4 on (per-connection rate drops to ~0.6 MB/s at K=8). Its cause is not identified yet (host CPU on a 4-vCPU VM, NTLM sealing in the WinRM service, network?). The client is not it (it takes in 24–31 MB/s against a local fake server).
  • Opening, authenticating and creating a shell for all K connections took 0.2–0.9 s in total.
  • Windows 2008 R2's default MaxShellsPerUser is 5, and the caller's own client already holds one: at most 4 extra connections there. Its MaxConcurrentOperationsPerUser (15) did not trigger at K=4.

Proposal

client.file("D:\\logs\\huge.log")
    .parallelism(4)                 // default 1 = today's single stream
    .downloadTo(Path.of("huge.log"));
  1. One setup call on the caller's client: the file size and the remote SHA-256 (the digest #147 already probes).
  2. Split into K ranges; each worker opens its own connection (same host, credentials and settings as the caller's client), reads its range with the existing ranged openStream(), and writes it at its position in the temporary destination file (FileChannel.write(buffer, position)), so no reordering buffer is needed.
  3. Once all ranges are in, the #147 flow unchanged: digest the whole temporary file, compare, fsync, ATOMIC_MOVE. A file modified during the transfer shows up as a digest mismatch.

Rules:

  • Only above a size threshold (e.g. 8 MiB): below it, the extra connections cost more than they save.
  • Graceful degradation: when opening a worker connection is refused by a quota (2150859173 concurrent shells, 2150859174 concurrent operations), continue with the workers already open instead of failing — and never retry past the quota in a tight loop.
  • Recommended value 4 (the measured sweet spot); document that more does not help and that 2008 R2 allows at most 4 extra.
  • Memory bounded by one buffer per worker; the temporary file is deleted on any failure (as in #147).
  • openStream() stays single-stream (parallel streaming would need a reordering buffer — out of scope).

Design question to settle first

A WinRMClient is a serial channel and cannot open further connections today: the password is consumed by build(). Options:

  • the builder keeps the credentials (it already needs them for re-authentication?) so the client can open sibling connections internally — security review needed for how long the char[] lives and how it is wiped on close();
  • or the caller passes the extra clients/a factory (parallelism(Supplier<WinRMClient>, 4)) — no credential retention, clumsier API.

Tests & docs

  • FakeWsmanServer: several concurrent connections, ranges reassembled byte-exact, a quota refusal on the 3rd connection → download completes with 2 workers, a worker failure midway → no destination file and no temporary file left.
  • WinRMLiveTest: parallel download of a ≥ 40 MiB file on a real host, SHA-256 verified.
  • files.md / file-transfers.md: the scaling table above (re-measured), the threshold, the quota limits.

Acceptance criteria

  • downloadTo with parallelism(4) is ≥ 2.5× faster than parallelism(1) on a 40 MiB file on a modern host, byte-exact.
  • On Windows 2008 R2 with default quotas, parallelism(8) degrades to what the quota allows instead of failing.
  • mvn verify site green.

🤖 Generated with Claude Code

Ngôn ngữ chính
Java
Star
15
Fork
4
Merge trung bình
11 giờ 58 phút
Pull request đã merge (30 ngày)
20

Chuẩn bị môi trường

Dự án này không cung cấp dev container, Dockerfile hay hướng dẫn đóng góp, nên bạn cần tự thiết lập môi trường: hãy bắt đầu từ README và xem hướng dẫn đóng góp lần đầu của chúng tôi để biết các bước chung.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của MetricsHub/winrm-java

Tất cả issue của MetricsHub/winrm-java

Issue tương tự

Thêm issue về Java

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.