pytorch/serve

Enabling HTTP caching of inference results

Open

Aperta il 23 mar 2022

Vedi su GitHub
 (2 commenti) (4 reazioni) (1 assegnatario)Java (3844 star) (790 fork)batch import
enhancementhelp wanted

Descrizione

Is your feature request related to a problem? Please describe.

Currently, TorchServe adds headers that prevent from caching the inference results: https://github.com/pytorch/serve/blob/30f83500b0850e26ec55581f48a9307b1986f9f9/frontend/server/src/main/java/org/pytorch/serve/util/NettyUtils.java#L187-L190

This prevents some reverse-proxies like nginx from caching the results

Describe the solution

It would be great to have a way to override this behavior, either from the model handler, or through some configuration key.

Describe alternatives solution

N/A

Guida contributor

Tech stack
javapython
Dominio
backendperformance
Tipo issue
feature
DifficoltàQuanto dovrebbe essere impegnativa per un nuovo contributor.
3
Tempo stimatoTempo stimato per completare e verificare uno scope piccolo.
half day
Stato attivitàQuanto è probabile che la issue sia ancora attiva e reviewable.
stale
ChiarezzaQuanto chiaramente la issue descrive problema, scope e risultato atteso.
clear
Prerequisiti
JavaNettyTorchServe internalsHTTP caching concepts
Adatta ai principiantiQuanto la issue è adatta a contributor alla prima esperienza.
40
Direzione di ricerca
Investigate the file frontend/server/src/main/java/org/pytorch/serve/util/NettyUtils.java, specifically lines 187-190 where caching prevention headers are set. Consider adding a configuration option or model handler method to allow overriding these headers. Review existing configuration mechanisms in TorchServe to understand how to implement a toggle for caching. Look at discussions in the issue comments for any additional context or proposed solutions.