Device zerocopy source callbacks: no way to tell which buffer completed, and no per-send completion

Đang mở
  • Có gì không ổn?
#3,966 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
22/100
Loại issue
Tính năng
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
cpp
Lĩnh vực
distributed-systems, hpc

Hướng nghiên cứu

This is API design for Charm++ device zerocopy source callbacks, not a first-timer patch. Read the issue’s two problems (empty callback vs refnum; per-buffer vs all-done) and related #3961/#3963. Entry points named in the text include CkDeviceBuffer, deviceMintAck, DevicePendingAck, deviceDrainAcksInto, and the host path invokeSourceCallback/constructSourceBufferObject. Done means maintainers pick an option (document refnum, diagnose, payload, and/or coalesced send-complete callback) and implement it in the runtime; per-buffer completion must remain.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Two rough edges in the device zerocopy source-callback interface, raised while reviewing #3961. Neither is a defect in that PR — both predate it — but applications are about to start building habits around this API, so they are worth settling now.

1. The callback does not say which buffer completed

A device source callback is delivered as an empty message. It reports that a buffer is free, not which one. With several buffers in flight the sender has to distinguish them itself, by reference number:

CkCallback cb(CkIndex_Block::ghostFree(NULL), thisProxy[thisIndex]);
cb.setRefnum(LEFT);                    // the user sets it, before handing the callback over
... CkDeviceBuffer(d_send_left_ghost, cb, comm_stream) ...

entry void ghostFree(AckMsg*);         // must take a message, or the closure carries no refnum
when ghostFree[LEFT](AckMsg*) { ... }

This is arguably the better mechanism — the refnum rides in an envelope that was being sent anyway, where the host path instead builds and delivers a whole CkNcpyBuffer (invokeSourceCallback → constructSourceBufferObject) so the receiver knows which buffer completed.

The problem is that nothing announces it, and getting it wrong fails silently. A zero-argument entry method carries no reference number, so when ghostFree[LEFT]() never matches and the program hangs with no diagnostic and no error. That has already cost two jobs to diagnose once.

Options, roughly in increasing cost:

  • document the refnum idiom as the supported way;
  • diagnose it — a refnum-bearing callback whose target entry method takes no message is always a bug, and could be caught at runtime or by charmxi;
  • deliver an identifying payload as the host path does (consistent with host semantics, but heavier than a refnum).

2. There is no "all buffers done" callback

Callbacks are minted per buffer — deviceMintAck is called once per CkDeviceBuffer in the sender loop — so an entry method carrying four zerocopy buffers produces four callbacks. An application that only wants to know "the send is finished" has to count them or thread them through SDAG.

Worth noting what is and is not already shared: every ack id from one send already rides back in a single piggybacked block (deviceDrainAcksInto(dest_pe, buffers[0])), so the wire cost is one block regardless. What is still one-for-one is delivery — the sender resolves each id and sends each callback separately.

Coalescing is therefore cheap, with one gap: DevicePendingAck records the pointer, size, registration handle and callback, but not which send an id came from, so there is nothing to group by today. Two shapes:

  • record a send id on each pending ack and fire one callback when the last id of a send resolves; or
  • let the API take one callback for the whole invocation, which gives the user the semantics directly rather than making the runtime infer them.

Per-buffer completion should stay available rather than be replaced. A buffer whose transfer finishes early can be reused early, and that is occasionally what an application wants. The point is that "all done" is the common case and should be the easy thing to ask for.

Context

Background and the full registration/reuse picture are written up separately; the relevant part here is that the source callback is the zerocopy analogue of ownership transfer. With a message you hand the buffer over at send and "when may I reuse it?" is moot; with zerocopy the application keeps the buffer, so the callback is the runtime's answer to a question that is genuinely live. Related: #3963 (shipped examples repack send buffers with no reuse guarantee).

🤖 Generated with Claude Code

Ngôn ngữ chính
C++
Star
241
Fork
62
Merge trung bình
1 ngày 1 giờ
Pull request đã merge (30 ngày)
125

Chuẩn bị môi trường

Dự án này không cung cấp dev container, Dockerfile hay hướng dẫn đóng góp, nên bạn cần tự thiết lập môi trường: hãy bắt đầu từ README và xem hướng dẫn đóng góp lần đầu của chúng tôi để biết các bước chung.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của charmplusplus/charm

Tất cả issue của charmplusplus/charm

Issue tương tự

Thêm issue về C++

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.