kubeflow/trainer

Support Elastic PyTorch in TrainJob

Open

#2903 aperta il 21 ott 2025

Vedi su GitHub
 (14 commenti) (7 reazioni) (1 assegnatario)Go (964 fork)github user discovery
area/apiarea/controllerhelp wantedkind/feature

Metriche repository

Star
 (2112 star)
Metriche merge PR
 (Metriche PR in attesa)

Descrizione

What you would like to be added?

As part of Kubeflow Trainer API, we designed the ElasticPolicy API which should allow users to run PyTorch in elastic mode: https://docs.pytorch.org/docs/stable/elastic/run.html

It would be nice if someone could drive KEP for that, and propose API changes to the next release: Trainer v2.2

elasticPolicy:
  minNodes: 2
  maxNodes: 5
  metrics:
    - type: Resource
      resource:
        name: nvidia.com/gpu
        target:
          type: Utilization
          averageUtilization: 75

Potentially, we should support elastic JobSet for that: https://github.com/kubernetes-sigs/jobset/issues/463

cc @kubeflow/kubeflow-trainer-team

/area api /area controller /help

Why is this needed?

We should support Elastic TrainJobs like in Training Operator V1 PyTorchJob.

Love this feature?

Give it a 👍 We prioritize the features with most 👍

Guida contributor