pytorch/serve

Confused about Cumulative Inference Duration vs. PredictionTime

Open

#1.698 geöffnet am 20. Juni 2022

Auf GitHub ansehen
 (3 Kommentare) (0 Reaktionen) (0 zugewiesene Personen)Java (3.844 Stars) (790 Forks)batch import
help wantedquestion

Beschreibung

📚 The doc issue

I am running a model on TorchServe and I am trying to see how long it takes for inference. If I use logging and view the logs, then I can see there is something called PredictionTime: image

However, if I use the Metrics API, then I got something called "Cumulative Inference Duration" image

And in terms of values those 2 are very different. So I am not sure which one should I use to measure the total inference time for my requests?

Btw, there is also something else called HandlerTime in the logs image

What does it mean? Where can I find related information about what are the meanings of these metrics?

Thanks,

Suggest a potential alternative/fix

No response

Contributor Guide

Tech Stack
pythonpytorch
Domain
documentation
Issue Type
documentation
SchwierigkeitGeschätzte Implementierungsschwierigkeit für neue Contributors, von 1 für sehr kleine Änderungen bis 5 für Expertenarbeit.
2
Geschätzte ZeitEin grober Zeitrahmen, um zu recherchieren, zu implementieren, zu testen und einen Pull Request vorzubereiten.
1-3 hours
AktivitätsstatusWie verfügbar das Issue gerade wirkt: frisch, aktiv, stale, blockiert oder wartet auf Maintainer-Input.
needs maintainer response
KlarheitWie klar das Issue die erwartete Änderung, Akzeptanzkriterien und den nächsten Schritt erklärt.
mostly clear
Voraussetzungen
Keine
EinsteigerfreundlichkeitEin Score von 1 bis 100, der schätzt, wie zugänglich dieses Issue für First-Time Contributors ist.
40
Research-Richtung
The user is confused about the difference between PredictionTime, Cumulative Inference Duration, and HandlerTime metrics. To clarify, one should look at the TorchServe metrics documentation (e.g., docs/metrics.md) and the source code where these metrics are logged (e.g., ts/metrics.py or ts/handler.py). A maintainer should explain which metric corresponds to request level inference time and how they relate to each other.