pytorch/serve

how does `default_response_timeout` work?

Open

#2452 opened on Jul 8, 2023

View on GitHub
 (6 comments) (0 reactions) (0 assignees)Java (3,844 stars) (790 forks)batch import
documentationgood first issuetriaged

Description

📚 The doc issue

I set the value of default_response_timeout to 4 i.e. 4 seconds. At the start of the model load, this happens after 4 (ish) seconds:

org.pytorch.serve.wlm.WorkerInitializationException: Backend worker did not respond in given time

My guess is because the model takes a while to load (more than 4 seconds), the worker gets killed. Is there a way to set a larger initial delay i.e. differentiate these two scenarios:

  • account for the initial model load with a number different from default_response_timeout
  • if model doesn't response in default_response_timeout after the initial load, then kill the worker

Suggest a potential alternative/fix

No response

Contributor guide

Tech stack
pythonpytorchjava
Domain
documentationbackend
Issue type
documentation
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
2
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
clear
Prerequisites
PyTorch Serveconfiguration
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
80
Research direction
Examine the issue comments for any maintainer suggestions or workarounds. Look at the PyTorch Serve source code, particularly the worker initialization files (e.g., `wlm.py` for Python backend or relevant Java classes) to understand how `default response timeout` is currently implemented. Propose adding a separate configuration parameter like `initial load timeout` to differentiate between model loading and subsequent timeouts. Update the documentation to clarify this behavior.