kubeflow/trainer

Support Elastic PyTorch in TrainJob

Open

#2.903 aberto em 21 de out. de 2025

Ver no GitHub
 (14 comments) (7 reactions) (1 assignee)Go (964 forks)github user discovery
area/apiarea/controllerhelp wantedkind/feature

Métricas do repositório

Stars
 (2.112 stars)
Métricas de merge de PR
 (Métricas PR pendentes)

Description

What you would like to be added?

As part of Kubeflow Trainer API, we designed the ElasticPolicy API which should allow users to run PyTorch in elastic mode: https://docs.pytorch.org/docs/stable/elastic/run.html

It would be nice if someone could drive KEP for that, and propose API changes to the next release: Trainer v2.2

elasticPolicy:
  minNodes: 2
  maxNodes: 5
  metrics:
    - type: Resource
      resource:
        name: nvidia.com/gpu
        target:
          type: Utilization
          averageUtilization: 75

Potentially, we should support elastic JobSet for that: https://github.com/kubernetes-sigs/jobset/issues/463

cc @kubeflow/kubeflow-trainer-team

/area api /area controller /help

Why is this needed?

We should support Elastic TrainJobs like in Training Operator V1 PyTorchJob.

Love this feature?

Give it a 👍 We prioritize the features with most 👍

Guia do colaborador