pytorch/serve

Confused about Cumulative Inference Duration vs. PredictionTime

Open

#1,698 建立於 2022年6月20日

在 GitHub 查看
 (3 留言) (0 反應) (0 負責人)Java (3,844 star) (790 fork)batch import
help wantedquestion

描述

📚 The doc issue

I am running a model on TorchServe and I am trying to see how long it takes for inference. If I use logging and view the logs, then I can see there is something called PredictionTime: image

However, if I use the Metrics API, then I got something called "Cumulative Inference Duration" image

And in terms of values those 2 are very different. So I am not sure which one should I use to measure the total inference time for my requests?

Btw, there is also something else called HandlerTime in the logs image

What does it mean? Where can I find related information about what are the meanings of these metrics?

Thanks,

Suggest a potential alternative/fix

No response

貢獻者指南

技術棧
pythonpytorch
領域
documentation
議題類型
documentation
難度面向新貢獻者的預計實作難度，1 表示很小改動，5 表示專家級工作。
2
預計時間有經驗貢獻者完成調查、實作、測試並準備 pull request 的粗略時間範圍。
1-3 hours
活動狀態議題目前的可參與程度：新鮮、活躍、陳舊、阻塞或等待維護者輸入。
needs maintainer response
清晰度議題是否清楚說明預期改動、驗收標準和下一步。
mostly clear
前置要求
無
新手友善度1-100 的估計分數，表示該議題對首次貢獻者的友善程度。
40
研究方向
The user is confused about the difference between PredictionTime, Cumulative Inference Duration, and HandlerTime metrics. To clarify, one should look at the TorchServe metrics documentation (e.g., docs/metrics.md) and the source code where these metrics are logged (e.g., ts/metrics.py or ts/handler.py). A maintainer should explain which metric corresponds to request level inference time and how they relate to each other.