kubeflow/trainer

Remove Finalizer from Runtimes

Closed

#3,715 opened on Jul 10, 2026

View on GitHubContributor guide
 (4 comments) (2 reactions) (1 assignee)Go (964 forks)github user discovery
area/controllergood first issuehelp wantedkind/feature

Repository metrics

Stars
 (2,112 stars)
PR merge metrics
 (PR metrics pending)

Description

What you would like to be added?

Once https://github.com/kubeflow/trainer/pull/3580 is merged, we no longer need the finalizer mechanism on runtimes. We should remove it.

What needs doing:

  1. update the runtime snapshot kep https://github.com/kubeflow/trainer/tree/master/proposals/2599-mutable-runtimes
  2. remove the logic that adds finalizers to runtimes when they are in use.
  3. provide a safe upgrade path: we need to automatically remove the finalizers from any existing runtimes when users upgrade.

Why is this needed?

Currently, if a user uninstalls the trainer whilst they have TrainJobs running then any finalizers on the runtimes are never removed. This means that these runtimes are never actually deleted, preventing users from re-installing the trainer without manually removing the finalizers.

Love this feature?

Give it a 👍 We prioritize the features with most 👍

Contributor guide

Research direction
Examine the runtime controller code to identify where finalizers are added and removed. Ensure the removal is safe during upgrades by checking existing finalizer removal logic and providing a migration path.
Tech stack
go
Domain
backendcloud
Issue type
Refactor
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Half day
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
GoKubernetescontroller runtime
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
60