Upgrade: a failed system VM template step is skipped on restart, as the schema steps are already marked Complete

Đang mở
  • Có gì không ổn?
#14,626 1 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
25/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
java
Lĩnh vực
backend, infrastructure

Hướng nghiên cứu

Start by reading DatabaseUpgradeChecker.upgrade(), executeUpgradeCleanup(), VersionDaoImpl.getCurrentVersion(), and SystemVmTemplateRegistration.updateSystemVmTemplates(). Trace how upgrade state is read and written when template registration fails, then compare the two recovery approaches described in the issue. Done means a chosen approach retries the failed template step safely without rerunning non-idempotent SQL; the issue calls for agreeing on the approach and testing it on a real upgrade.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

problem

If the system VM template registration step of a database upgrade fails, the upgrade can't be completed by fixing the cause and restarting the management server. The restart skips the template step, and the server comes up on the new version with the old system VM template still configured. The only way back we found was to restore the database backup and run the upgrade again.

The cause is the order of operations in DatabaseUpgradeChecker.upgrade():

executeProcedureScripts();
final DbUpgrade[] upgrades = executeUpgrades(dbVersion, currentVersion);
executeViewScripts();
updateSystemVmTemplates(upgrades);
  • executeUpgrades() runs each step of the path. executeUpgrade() commits the schema and data changes and inserts the version row, and executeUpgradeCleanup() then sets that row to Complete and commits.
  • Only after every step is Complete does updateSystemVmTemplates() run. It calls SystemVmTemplateRegistration.updateSystemVmTemplates(), which looks for the template under the name from metadata.ini (for example systemvm-kvm-4.22.0-x86_64). If no template has that name, it registers one from the packaged template file. If that fails, it throws, and the management server stops.
  • On the next start, VersionDaoImpl.getCurrentVersion() returns the newest Complete row, which is now the code version. doUpgrades() logs "DB version and code version matches so no upgrade needed." and returns. updateSystemVmTemplates() is only reached from upgrade(), so it doesn't run again.

The result is a management server on the new version with router.template.* and minreq.sysvmtemplate.version still at the old values, and nothing that marks the upgrade as incomplete. Nothing in the startup path checks for it, and nothing retries it.

In our case, the template step failed because the new template had been registered ahead of the upgrade under a name other than the one the upgrade expects (systemvm-kvm-4.22.0 instead of systemvm-kvm-4.22.0-x86_64). That was an operator error. But any failure in that step leaves the database in the same state, for example a missing or corrupt template file, or a secondary storage that can't be mounted.

versions

Seen on an upgrade from 4.20.3 to 4.22.1 on KVM with NFS secondary storage. The same ordering is on the 4.22 branch and on main. It has been there since template registration became part of the database upgrade in 4.16 (#4329).

The steps to reproduce the bug
  1. Run a 4.20 setup with at least one KVM cluster.
  2. Make the template registration step fail. One way is to register the 4.22 system VM template ahead of the upgrade under a different name, such as systemvm-kvm-4.22.0, and move the packaged KVM template file out of /usr/share/cloudstack-management/templates/systemvm/.
  3. Install the 4.22 packages and start the management server. The schema upgrade runs, then the template step fails with "Failed to register SystemVM Template. Upgrade failed" and the management server stops.
  4. Check the version table. The rows for every step up to 4.22.x are Complete.
  5. Fix the cause, for example by renaming the template to systemvm-kvm-4.22.0-x86_64, and start the management server again.
  6. It logs "DB version and code version matches so no upgrade needed." and starts. The template step doesn't run. router.template.kvm and minreq.sysvmtemplate.version still hold the 4.20 values, and the template is not marked as a system template.
What to do about it?

Two ways to fix it come to mind. Both change upgrade or startup behaviour, so I'd rather agree on the approach here first, and then test it on a real upgrade.

  1. Retry on startup. When the database version matches the code version, check whether the template named in metadata.ini is registered for each hypervisor and architecture in use, and run the registration step if one is missing. This is the smallest change, but it runs on every start. It has to tell a failed upgrade apart from a setup that is missing the template for another reason, for example a cluster of a new architecture added after the upgrade. It also has to decide whether a failure blocks startup.
  2. Mark the last step Complete only after the template step succeeds. If the template step fails, leave the last version row at Cleanup. On the next start, when the newest row is the code version at Cleanup, run only the template step and then mark it Complete. Its SQL must not run again, because the scripts aren't idempotent. This keeps the retry to failed upgrades only. But getCurrentVersion() would return the previous version while the schema is already at the new one, so every reader of that value needs checking, and so does downgrading to the old packages.

Until this is fixed, the only recovery we found is to restore the database backup, fix the cause, and run the upgrade again.

Ngôn ngữ chính
Java
Star
3.1k
Fork
1.4k
Merge trung bình
7 ngày 16 giờ
Pull request đã merge (30 ngày)
15

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của apache/cloudstack

Tất cả issue của apache/cloudstack

Issue tương tự

Thêm issue về Java

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.