good first issuehelp wanted
仓库指标
- 星标
- (64 个星标)
- PR 合并指标
- (30 天内没有已合并 PR)
描述
Description
Implement request timeout handling to prevent long-running requests from blocking the server.
Current Status - Partially Implemented
Model-level timeout exists but HTTP-level timeout handling is not implemented.
Already Implemented
- Model-level timeout in predict_batch_with_timeout method
- Timeout logging and monitoring
- Graceful timeout error handling at model level
Remaining Work
- Add configurable HTTP request timeout (default 30 seconds)
- Return HTTP 408 Request Timeout for exceeded requests
- Add timeout configuration to CLI arguments
- Add timeout middleware to Axum router
- Add tests for HTTP timeout behavior
- Update documentation with timeout information
Implementation Guidance
- Use tower::timeout::TimeoutLayer for HTTP-level timeouts
- Add --request-timeout CLI argument
- Consider different timeouts for different endpoints
- Integrate with existing model timeout for layered timeout handling
Estimated Difficulty
Easy - Half day (reduced due to existing timeout infrastructure)