todo: Ascend P/D request fails 500 when the decode dials a KV port nothing announced

Open
  • Is something wrong?
#765 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
28/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
go, kubernetes, python
Domain
distributed-systems, networking

Research direction

Reproduce the md-q10-t5 Encode/Prefill/Decode shape and capture the prefill payload at set_xfer_handshake_metadata plus the decode-resolved KV endpoint before each request. Start with wave10-logs/t5-run.log, then trace mooncake_connector.py and the serving POST /v1/chat/completions failure path. Done means determining whether 20091 is dynamically bound or incorrectly supplied by handshake metadata, with evidence identifying the responsible path.

Written by the indexing model from the issue text.

Description

area/vendor/ascend todo

What happened

Two Ascend 910B2 runs returned 500 Internal server error from a ModelDeployment.
One of the two has a complete saved chain; the other does not.

Measured, with the chain saved

Deployment md-q10-t5 (Encode + Prefill + Decode, llm-d-router), 2026-10-16. Two requests
failed, 26.4 s and 20.2 s wall time:

E1016 13:02:03 transfer_executor_base.cpp:263 Failed to connect to target: 10.42.227.218:20091,
                                    status: 503900, errmsg: E19999: Inner Error!
E19999 Call llm::HcclAdapter::GetInstance().HcclCommPrepare(...) fail, ret: 0x9
       [FUNC:PrepareHcclComm][FILE:comm_channel.cc][LINE:105]
I1016 13:02:03 transfer_executor_base.cpp:587 Retry transfer to: 10.42.227.218:20091
E1016 13:02:13 transfer_executor_base.cpp:263 Failed to connect to target: 10.42.227.218:20091
(EngineCore) ERROR mooncake_connector.py:1044 Mooncake transfer failed ... ret=-1
(EngineCore) ERROR scheduler.py:3231 Failing 1 request(s) due to KV load failure
                              (failure_policy=fail, 40 tokens affected)
(APIServer)  ERROR serving.py:207   failed with an internal error during generation
             POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error

The open question is the port. The decode dialed 10.42.227.218:20091. In the same window the
prefill announced only:

Transfer Engine RPC using P2P handshake, listening on 10.42.227.218:16640
install AscendDirectTransport for: 10.42.227.218:16640
KVCacheSendingThread started listening on path: tcp://10.42.227.218:8998

No saved line binds or announces 20091. Either something allocates it dynamically and the saved
window is too narrow, or the handshake metadata carries a wrong endpoint. The saved files do not
decide between those.

Not reproduced

The same shape with a fresh prompt returned 200 with a real KV transfer on later runs
(142.38 ms / 145.90 ms with an encoder; 6.06 ms without), at prefill size 1 and at prefill
size 2. A second incident (a 3-role EPD deployment) returned one 500 and then two 200s, and it
has no saved cause.

What would close it

Reproduce on the md-q10-t5 shape and capture, before the request: the prefill's full
set_xfer_handshake_metadata payload, and the decode's resolved KV endpoint. If 20091 appears in
neither, the defect is in how the handshake names the peer.

Provenance

Measured: the quoted chain and the 500s, from wave10-logs/t5-run.log. Inferred: that 20091 is
unannounced — that is an absence in the saved window, not a proof it was never bound.

Dominant language
Go
Stars
5
Forks
6
Avg merge
2h 59m
Merged PRs (30d)
71

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from gpustack/gpustack-operator

All issues in gpustack/gpustack-operator

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.