Upgrade: a failed system VM template step is skipped on restart, as the schema steps are already marked Complete

Start by reading DatabaseUpgradeChecker.upgrade(), executeUpgradeCleanup(), VersionDaoImpl.getCurrentVersion(), and SystemVmTemplateRegistration.updateSystemVmTemplates(). Trace how upgrade state is read and written when template registration fails, then compare the two recovery approaches described in the issue. Done means a chosen approach retries the failed template step safely without rerunning non-idempotent SQL; the issue calls for agreeing on the approach and testing it on a real upgrade.

problem

If the system VM template registration step of a database upgrade fails, the upgrade can't be completed by fixing the cause and restarting the management server. The restart skips the template step, and the server comes up on the new version with the old system VM template still configured. The only way back we found was to restore the database backup and run the upgrade again.

The cause is the order of operations in DatabaseUpgradeChecker.upgrade():

executeProcedureScripts();
final DbUpgrade[] upgrades = executeUpgrades(dbVersion, currentVersion);
executeViewScripts();
updateSystemVmTemplates(upgrades);

• executeUpgrades() runs each step of the path. executeUpgrade() commits the schema and data changes and inserts the version row, and executeUpgradeCleanup() then sets that row to Complete and commits.
• Only after every step is Complete does updateSystemVmTemplates() run. It calls SystemVmTemplateRegistration.updateSystemVmTemplates(), which looks for the template under the name from metadata.ini (for example systemvm-kvm-4.22.0-x86_64). If no template has that name, it registers one from the packaged template file. If that fails, it throws, and the management server stops.
• On the next start, VersionDaoImpl.getCurrentVersion() returns the newest Complete row, which is now the code version. doUpgrades() logs "DB version and code version matches so no upgrade needed." and returns. updateSystemVmTemplates() is only reached from upgrade(), so it doesn't run again.

The result is a management server on the new version with router.template.* and minreq.sysvmtemplate.version still at the old values, and nothing that marks the upgrade as incomplete. Nothing in the startup path checks for it, and nothing retries it.

In our case, the template step failed because the new template had been registered ahead of the upgrade under a name other than the one the upgrade expects (systemvm-kvm-4.22.0 instead of systemvm-kvm-4.22.0-x86_64). That was an operator error. But any failure in that step leaves the database in the same state, for example a missing or corrupt template file, or a secondary storage that can't be mounted.

versions

Seen on an upgrade from 4.20.3 to 4.22.1 on KVM with NFS secondary storage. The same ordering is on the 4.22 branch and on main. It has been there since template registration became part of the database upgrade in 4.16 (#4329).

The steps to reproduce the bug

1. Run a 4.20 setup with at least one KVM cluster.
2. Make the template registration step fail. One way is to register the 4.22 system VM template ahead of the upgrade under a different name, such as systemvm-kvm-4.22.0, and move the packaged KVM template file out of /usr/share/cloudstack-management/templates/systemvm/.
3. Install the 4.22 packages and start the management server. The schema upgrade runs, then the template step fails with "Failed to register SystemVM Template. Upgrade failed" and the management server stops.
4. Check the version table. The rows for every step up to 4.22.x are Complete.
5. Fix the cause, for example by renaming the template to systemvm-kvm-4.22.0-x86_64, and start the management server again.
6. It logs "DB version and code version matches so no upgrade needed." and starts. The template step doesn't run. router.template.kvm and minreq.sysvmtemplate.version still hold the 4.20 values, and the template is not marked as a system template.

What to do about it?

Two ways to fix it come to mind. Both change upgrade or startup behaviour, so I'd rather agree on the approach here first, and then test it on a real upgrade.

1. Retry on startup. When the database version matches the code version, check whether the template named in metadata.ini is registered for each hypervisor and architecture in use, and run the registration step if one is missing. This is the smallest change, but it runs on every start. It has to tell a failed upgrade apart from a setup that is missing the template for another reason, for example a cluster of a new architecture added after the upgrade. It also has to decide whether a failure blocks startup.
2. Mark the last step Complete only after the template step succeeds. If the template step fails, leave the last version row at Cleanup. On the next start, when the newest row is the code version at Cleanup, run only the template step and then mark it Complete. Its SQL must not run again, because the scripts aren't idempotent. This keeps the retry to failed upgrades only. But getCurrentVersion() would return the previous version while the schema is already at the new one, so every reader of that value needs checking, and so does downgrading to the old packages.

Until this is fixed, the only recovery we found is to restore the database backup, fix the cause, and run the upgrade again.