openvmm: in virt_whp, a VP saved in `cli; hlt` runs past the HLT after restore, but still reads back as `Halted`

Open
#1,474 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
rust

Research direction

Start with vmm_core/virt_whp/src/apic.rs, especially restore_activity and the WHP APIC state handling described in the issue; then inspect vmm_core/virt/src/state.rs for the restore checks. Reproduce on a Windows host with WHP using the vp_state tests from repro-test.patch, which is not included in the repository yet. Done means halted and idle VPs remain suspended across all three restore paths, while state read-back and the existing single-field mutation behavior remain consistent.

Written by the indexing model from the issue text.

Description

Summary

This happens with the default --hypervisor whp (the hypervisor APIC), with or without --hv. OpenVMM saves a VP that is halted with interrupts off as MpState::Halted, which is correct. Restoring that state with restore_all succeeds, and activity() still returns Halted. But on its next run the VP executes the instructions after the HLT.

A VP idling through HV_X64_MSR_GUEST_IDLE (saved as Idle) behaves the same way. With sti; hlt, only a restore onto the same VP wakes it. KVM keeps the VP halted in all these cases.

Repro

This needs a Windows host with WHP. repro-test.patch adds vmm_core/virt_whp/tests/vp_state.rs, which has small helpers for building partitions, running guest code, and saving and restoring VPs. The guest is real-mode code: out 0x80, 'H'; cli; hlt; out 0x80, 'W'. The test below is abridged from that file.

fn suspended_vp_stays_suspended(config: Config, code: &'static [u8], mark: u8, fresh: bool) {
    with_time(move |time| {
        let mut vm = run_to_suspend(config, code, mark, time); // run until 'H', stop
        let saved = vm.save(0);
        let expected = vm.activity(0).mp_state; // Halted
        if fresh {
            // As `--restore-snapshot`: a new partition with the saved RAM.
            let ram = vm.ram();
            drop(vm);
            let mut hv = whp(config);
            vm = Vm::new(&mut hv, config, 1, time);
            vm.mem.write_at(0, &ram).unwrap();
        }
        vm.restore(0, &saved).unwrap();
        assert_eq!(vm.activity(0).mp_state, expected, "read back after restore_all");
        let woke = vm.run_until_mark(b'W', Duration::from_secs(1));
        assert!(!woke, "the VP was saved as {expected:?} but ran past the suspending instruction");
    })
}

#[test]
fn halted_vp_stays_halted_after_restore_into_fresh_partition() {
    suspended_vp_stays_suspended(WHP, HALT, b'H', true);
}
// Also: ..._onto_same_vp, and idle_vp_stays_idle_after_restore_into_fresh_partition (WHP_HV1).
$ cargo test -p virt_whp --test vp_state -- halted_vp idle_vp
the VP was saved as Halted but ran past the suspending instruction after restore_all (marks "W")

All three tests fail on main 42d4fe619. The read-back right after restore_all passes in each of them.

Cause

The behavior comes from WHP itself. restore_all sets registers, then activity (halt_suspend = 1), then the APIC page (apic.rs#L292-L319). We tested this by restoring subsets of the elements in different orders:

  • Restoring any single element does not wake the VP.
  • Restoring registers and apic together does, with or without activity.
  • No order keeps the VP halted. If activity is written after apic, the halt is dropped and reads back as Running. That would trip restore_all's debug check (state.rs#L98-L113), so changing the order is not a fix.

Fix Suggestion

fix.patch works around this inside virt_whp, for the hypervisor APIC only. On restore of Halted or Idle, if the instruction before RIP is HLT (or the RDMSR of the idle MSR), it moves RIP back onto that instruction and restores the VP as Running. The VP then suspends itself again on its first instruction. Until the VP runs, the getters keep returning the saved mp_state and RIP, so restore_all's check and a re-save are unchanged. With the patch, all three restore paths keep the VP suspended, and a sweep of single-field state mutations gives the same results as before.

fix.patch (click to expand)
diff --git a/vmm_core/virt_whp/src/apic.rs b/vmm_core/virt_whp/src/apic.rs
index 60fcb73ae..4df6f36d7 100644
--- a/vmm_core/virt_whp/src/apic.rs
+++ b/vmm_core/virt_whp/src/apic.rs
@@ -175,7 +175,100 @@ fn now(&mut self) -> VmTime {
     }
 }
 
+/// A halt or idle suspend state that `restore_activity` turned into a
+/// re-execution of the suspending instruction.
+///
+/// With the hypervisor's APIC, WHP does not keep a VP in a halt or idle
+/// suspend state that is restored along with the VP's register and APIC
+/// state: on its next run the VP continues after the HLT (or the RDMSR of
+/// HV_X64_MSR_GUEST_IDLE), even though the activity register still reads
+/// back as suspended, and a suspend state written after the APIC state is
+/// dropped. Instead of restoring the suspend state, RIP is pointed back at
+/// the suspending instruction and the VP is restored as running, so that it
+/// executes the instruction again and the hypervisor suspends it itself.
+///
+/// Until the VP runs, the state accessors still report the saved state:
+/// `mp_state` and `rip` (the architectural RIP, after the instruction).
+#[derive(Debug, Copy, Clone)]
+pub(crate) struct RestoredSuspend {
+    pub mp_state: vp::MpState,
+    pub rip: u64,
+}
+
 impl WhpProcessor<'_> {
+    /// Points RIP back at the instruction that put the VP into `mp_state`
+    /// (HLT for halt, RDMSR for the guest idle MSR) if that is the
+    /// instruction just before RIP. Returns the original RIP if it did.
+    fn rewind_to_suspend_instruction(
+        &mut self,
+        vtl: Vtl,
+        mp_state: vp::MpState,
+    ) -> Result<Option<u64>, Error> {
+        let expected: &[u8] = match mp_state {
+            vp::MpState::Halted => &[0xf4],     // hlt
+            vp::MpState::Idle => &[0x0f, 0x32], // rdmsr
+            vp::MpState::Running | vp::MpState::WaitForSipi => return Ok(None),
+        };
+        let wvp = self.vp.whp(vtl);
+        let (cs, rip, efer) = get_registers!(
+            wvp,
+            [
+                whp::RegisterSegment::Cs,
+                whp::Register64::Rip,
+                whp::Register64::Efer,
+            ]
+        )
+        .for_op("get registers")?;
+        let long_mode = efer & x86defs::X64_EFER_LMA != 0
+            && x86defs::SegmentAttributes::from(cs.Attributes).long();
+        let start = rip.wrapping_sub(expected.len() as u64);
+        for (i, &byte) in expected.iter().enumerate() {
+            let offset = start.wrapping_add(i as u64);
+            let gva = if long_mode {
+                offset
+            } else {
+                cs.Base.wrapping_add(offset) & 0xffff_ffff
+            };
+            let Ok(gpa) = wvp
+                .translate_gva(
+                    gva,
+                    whp::abi::WHvTranslateGvaFlagValidateExecute
+                        | whp::abi::WHvTranslateGvaFlagPrivilegeExempt,
+                )
+                .for_op("translate gva")?
+            else {
+                return Ok(None);
+            };
+            let mut b = [0];
+            if self.vp.partition.gm.read_at(gpa, &mut b).is_err() || b[0] != byte {
+                return Ok(None);
+            }
+        }
+        wvp.set_register(whp::Register64::Rip, start)
+            .for_op("set rip")?;
+        Ok(Some(rip))
+    }
+
+    /// Sets the register state. With the hypervisor's APIC, a new RIP
+    /// invalidates the check in [`Self::rewind_to_suspend_instruction`], so
+    /// redo it for a VP that is in a halt or idle suspend state.
+    pub(crate) fn restore_registers(
+        &mut self,
+        vtl: Vtl,
+        value: &vp::Registers,
+    ) -> Result<(), Error> {
+        let LocalApicKind::Offloaded = self.vp.partition.vtlp(vtl).lapic else {
+            return self.vp.set_register_state(vtl, value);
+        };
+        let activity = self.save_activity(vtl)?;
+        self.state.vtls[vtl].restored_suspend = None;
+        self.vp.set_register_state(vtl, value)?;
+        if matches!(activity.mp_state, vp::MpState::Halted | vp::MpState::Idle) {
+            self.restore_activity(vtl, &activity)?;
+        }
+        Ok(())
+    }
+
     pub(crate) fn save_activity(&mut self, vtl: Vtl) -> Result<vp::Activity, Error> {
         let activity = match self.vp.partition.vtlp(vtl).lapic {
             LocalApicKind::Emulated(_) => {
@@ -203,6 +296,9 @@ pub(crate) fn save_activity(&mut self, vtl: Vtl) -> Result<vp::Activity, Error>
                     &self.vp.whp(vtl).get_apic().for_op("get apic state")?,
                 );
                 activity.nmi_pending = apic.hv_apic_nmi_pending();
+                if let Some(restored) = self.state.vtls[vtl].restored_suspend {
+                    activity.mp_state = restored.mp_state;
+                }
                 activity
             }
         };
@@ -247,7 +343,35 @@ pub(crate) fn restore_activity(&mut self, vtl: Vtl, value: &vp::Activity) -> Res
                 self.vp.set_register_state(vtl, &value)?;
             }
             LocalApicKind::Offloaded => {
-                self.vp.set_register_state(vtl, value)?;
+                // Undo an earlier rewind so that RIP is architectural again.
+                if let Some(restored) = self.state.vtls[vtl].restored_suspend.take() {
+                    self.vp
+                        .whp(vtl)
+                        .set_register(whp::Register64::Rip, restored.rip)
+                        .for_op("set rip")?;
+                }
+                let mut mp_state = value.mp_state;
+                if let Some(rip) = self.rewind_to_suspend_instruction(vtl, mp_state)? {
+                    self.state.vtls[vtl].restored_suspend = Some(RestoredSuspend { mp_state, rip });
+                    mp_state = vp::MpState::Running;
+                } else if matches!(mp_state, vp::MpState::Halted | vp::MpState::Idle) {
+                    tracelimit::warn_ratelimited!(
+                        ?mp_state,
+                        "restoring a suspend state without its instruction before rip; \
+                         the hypervisor may resume the vp"
+                    );
+                }
+                self.vp.set_register_state(
+                    vtl,
+                    &vp::Activity {
+                        mp_state,
+                        nmi_pending: value.nmi_pending,
+                        nmi_masked: value.nmi_masked,
+                        interrupt_shadow: value.interrupt_shadow,
+                        pending_event: value.pending_event,
+                        pending_interruption: value.pending_interruption,
+                    },
+                )?;
                 // Set the NMI pending bit via the APIC.
                 let mut apic = vp::ApicRegisters::from_page(
                     &self.vp.whp(vtl).get_apic().for_op("get apic state")?,
diff --git a/vmm_core/virt_whp/src/lib.rs b/vmm_core/virt_whp/src/lib.rs
index f3088f4f4..871e2cb1f 100644
--- a/vmm_core/virt_whp/src/lib.rs
+++ b/vmm_core/virt_whp/src/lib.rs
@@ -249,6 +249,11 @@ struct PerVtlRunState {
     deliverability_notifications: HvDeliverabilityNotificationsRegister,
     // Only used when `hv` is `None`.
     vp_assist_page: u64,
+    /// A restored halt or idle state that is being re-entered by executing
+    /// the suspending instruction again (see `apic::RestoredSuspend`).
+    #[cfg(guest_arch = "x86_64")]
+    #[inspect(debug)]
+    restored_suspend: Option<apic::RestoredSuspend>,
 }
 
 impl PerVtlRunState {
@@ -269,6 +274,8 @@ pub fn new(lapic: &LocalApicKind, hv: &Hv1State, vp_info: &TargetVpInfo, vtl: Vt
             },
             deliverability_notifications: 0.into(),
             vp_assist_page: 0,
+            #[cfg(guest_arch = "x86_64")]
+            restored_suspend: None,
         }
     }
 
@@ -359,6 +366,8 @@ fn reset(&mut self, is_bsp: bool, prot_access: &mut dyn hv1_emulator::VtlProtect
             hv,
             deliverability_notifications,
             vp_assist_page,
+            #[cfg(guest_arch = "x86_64")]
+            restored_suspend,
         } = self;
 
         #[cfg(guest_arch = "x86_64")]
@@ -375,6 +384,10 @@ fn reset(&mut self, is_bsp: bool, prot_access: &mut dyn hv1_emulator::VtlProtect
 
         *deliverability_notifications = 0.into();
         *vp_assist_page = 0;
+        #[cfg(guest_arch = "x86_64")]
+        {
+            *restored_suspend = None;
+        }
     }
 }
 
diff --git a/vmm_core/virt_whp/src/vp.rs b/vmm_core/virt_whp/src/vp.rs
index 30549ffe6..6e57e33b5 100644
--- a/vmm_core/virt_whp/src/vp.rs
+++ b/vmm_core/virt_whp/src/vp.rs
@@ -371,6 +371,14 @@ pub(crate) async fn run_vp(
             // Set the lazy EOI bit just before running.
             let lazy_eoi = self.sync_lazy_eoi();
 
+            // From here on the VP executes the instruction that a restore
+            // rewound to, so its register state is architectural again.
+            #[cfg(guest_arch = "x86_64")]
+            {
+                let vtl = self.state.active_vtl;
+                self.state.vtls[vtl].restored_suspend = None;
+            }
+
             let mut runner = self.current_whp().runner();
             let exit = runner
                 .run()
diff --git a/vmm_core/virt_whp/src/vp_state.rs b/vmm_core/virt_whp/src/vp_state.rs
index a187c12d6..434ef1808 100644
--- a/vmm_core/virt_whp/src/vp_state.rs
+++ b/vmm_core/virt_whp/src/vp_state.rs
@@ -89,11 +89,15 @@ fn commit(&mut self) -> Result<(), Self::Error> {
         }
 
         fn registers(&mut self) -> Result<vp::Registers, Self::Error> {
-            self.run.vp.get_register_state(self.vtl)
+            let mut regs: vp::Registers = self.run.vp.get_register_state(self.vtl)?;
+            if let Some(restored) = self.run.state.vtls[self.vtl].restored_suspend {
+                regs.rip = restored.rip;
+            }
+            Ok(regs)
         }
 
         fn set_registers(&mut self, value: &vp::Registers) -> Result<(), Self::Error> {
-            self.run.vp.set_register_state(self.vtl, value)
+            self.run.restore_registers(self.vtl, value)
         }
 
         fn activity(&mut self) -> Result<vp::Activity, Self::Error> {

Tested on Windows 11 26H2 (build 26300), on a Xeon 8573C host and an EPYC 9V74 host.

Dominant language
Rust
Stars
848
Forks
63
Avg merge
13h 37m
Merged PRs (30d)
571

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/nvx

All issues in microsoft/nvx

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.