Remove Finalizer from Runtimes

What you would like to be added?

Once https://github.com/kubeflow/trainer/pull/3580 is merged, we no longer need the finalizer mechanism on runtimes. We should remove it.

What needs doing:

1. update the runtime snapshot kep https://github.com/kubeflow/trainer/tree/master/proposals/2599-mutable-runtimes
2. remove the logic that adds finalizers to runtimes when they are in use.
3. provide a safe upgrade path: we need to automatically remove the finalizers from any existing runtimes when users upgrade.

Why is this needed?

Currently, if a user uninstalls the trainer whilst they have TrainJobs running then any finalizers on the runtimes are never removed. This means that these runtimes are never actually deleted, preventing users from re-installing the trainer without manually removing the finalizers.

Love this feature?

Give it a 👍 We prioritize the features with most 👍