pytorch/serve

Having two different endpoints for a single model

Open

Aperta il 5 gen 2023

Vedi su GitHub
 (2 commenti) (0 reazioni) (0 assegnatari)Java (3844 star) (790 fork)batch import
enhancementhelp wantedtriaged

Descrizione

Hello!

Is it possible to create two different endpoints for a single model? So it can work in two separate modes and batch accordingly depending on the input (e.g., it can generate an image or perform upscaling given an input). I can make two different models but then the memory will be double, while actually no extra memory is needed (as the model weights are the same). Ideally, I want to have the same model weights but two different endpoints with different batching settings.

Best, Valentin

Guida contributor

Tech stack
pythonpytorchjava
Dominio
backendapi
Tipo issue
feature
DifficoltàQuanto dovrebbe essere impegnativa per un nuovo contributor.
4
Tempo stimatoTempo stimato per completare e verificare uno scope piccolo.
1-2 days
Stato attivitàQuanto è probabile che la issue sia ancora attiva e reviewable.
stale
ChiarezzaQuanto chiaramente la issue descrive problema, scope e risultato atteso.
mostly clear
Prerequisiti
TorchServe basicsModel serving conceptsPython/Java knowledge
Adatta ai principiantiQuanto la issue è adatta a contributor alla prima esperienza.
30
Direzione di ricerca
Investigate TorchServe's model registration and endpoint configuration in files like 'frontend/model server.py' or 'backend/...'. Look for how batching is tied to endpoints. The issue may require creating a configuration to allow multiple endpoints per model with separate batching parameters. Links to relevant code should be examined.