openvmm: in virt_whp, a VP saved in `cli; hlt` runs past the HLT after restore, but still reads back as `Halted`
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- rust
- Domain
- backend, operating-systems
Research direction
Start with vmm_core/virt_whp/src/apic.rs, especially restore_activity and the WHP APIC state handling described in the issue; then inspect vmm_core/virt/src/state.rs for the restore checks. Reproduce on a Windows host with WHP using the vp_state tests from repro-test.patch, which is not included in the repository yet. Done means halted and idle VPs remain suspended across all three restore paths, while state read-back and the existing single-field mutation behavior remain consistent.
Written by the indexing model from the issue text.
Description
Summary
This happens with the default --hypervisor whp (the hypervisor APIC), with or without --hv. OpenVMM saves a VP that is halted with interrupts off as MpState::Halted, which is correct. Restoring that state with restore_all succeeds, and activity() still returns Halted. But on its next run the VP executes the instructions after the HLT.
A VP idling through HV_X64_MSR_GUEST_IDLE (saved as Idle) behaves the same way. With sti; hlt, only a restore onto the same VP wakes it. KVM keeps the VP halted in all these cases.
Repro
This needs a Windows host with WHP. repro-test.patch adds vmm_core/virt_whp/tests/vp_state.rs, which has small helpers for building partitions, running guest code, and saving and restoring VPs. The guest is real-mode code: out 0x80, 'H'; cli; hlt; out 0x80, 'W'. The test below is abridged from that file.
fn suspended_vp_stays_suspended(config: Config, code: &'static [u8], mark: u8, fresh: bool) {
with_time(move |time| {
let mut vm = run_to_suspend(config, code, mark, time); // run until 'H', stop
let saved = vm.save(0);
let expected = vm.activity(0).mp_state; // Halted
if fresh {
// As `--restore-snapshot`: a new partition with the saved RAM.
let ram = vm.ram();
drop(vm);
let mut hv = whp(config);
vm = Vm::new(&mut hv, config, 1, time);
vm.mem.write_at(0, &ram).unwrap();
}
vm.restore(0, &saved).unwrap();
assert_eq!(vm.activity(0).mp_state, expected, "read back after restore_all");
let woke = vm.run_until_mark(b'W', Duration::from_secs(1));
assert!(!woke, "the VP was saved as {expected:?} but ran past the suspending instruction");
})
}
#[test]
fn halted_vp_stays_halted_after_restore_into_fresh_partition() {
suspended_vp_stays_suspended(WHP, HALT, b'H', true);
}
// Also: ..._onto_same_vp, and idle_vp_stays_idle_after_restore_into_fresh_partition (WHP_HV1).
$ cargo test -p virt_whp --test vp_state -- halted_vp idle_vp
the VP was saved as Halted but ran past the suspending instruction after restore_all (marks "W")
All three tests fail on main 42d4fe619. The read-back right after restore_all passes in each of them.
Cause
The behavior comes from WHP itself. restore_all sets registers, then activity (halt_suspend = 1), then the APIC page (apic.rs#L292-L319). We tested this by restoring subsets of the elements in different orders:
- Restoring any single element does not wake the VP.
- Restoring
registersandapictogether does, with or withoutactivity. - No order keeps the VP halted. If
activityis written afterapic, the halt is dropped and reads back asRunning. That would triprestore_all's debug check (state.rs#L98-L113), so changing the order is not a fix.
Fix Suggestion
fix.patch works around this inside virt_whp, for the hypervisor APIC only. On restore of Halted or Idle, if the instruction before RIP is HLT (or the RDMSR of the idle MSR), it moves RIP back onto that instruction and restores the VP as Running. The VP then suspends itself again on its first instruction. Until the VP runs, the getters keep returning the saved mp_state and RIP, so restore_all's check and a re-save are unchanged. With the patch, all three restore paths keep the VP suspended, and a sweep of single-field state mutations gives the same results as before.
fix.patch (click to expand)
diff --git a/vmm_core/virt_whp/src/apic.rs b/vmm_core/virt_whp/src/apic.rs
index 60fcb73ae..4df6f36d7 100644
--- a/vmm_core/virt_whp/src/apic.rs
+++ b/vmm_core/virt_whp/src/apic.rs
@@ -175,7 +175,100 @@ fn now(&mut self) -> VmTime {
}
}
+/// A halt or idle suspend state that `restore_activity` turned into a
+/// re-execution of the suspending instruction.
+///
+/// With the hypervisor's APIC, WHP does not keep a VP in a halt or idle
+/// suspend state that is restored along with the VP's register and APIC
+/// state: on its next run the VP continues after the HLT (or the RDMSR of
+/// HV_X64_MSR_GUEST_IDLE), even though the activity register still reads
+/// back as suspended, and a suspend state written after the APIC state is
+/// dropped. Instead of restoring the suspend state, RIP is pointed back at
+/// the suspending instruction and the VP is restored as running, so that it
+/// executes the instruction again and the hypervisor suspends it itself.
+///
+/// Until the VP runs, the state accessors still report the saved state:
+/// `mp_state` and `rip` (the architectural RIP, after the instruction).
+#[derive(Debug, Copy, Clone)]
+pub(crate) struct RestoredSuspend {
+ pub mp_state: vp::MpState,
+ pub rip: u64,
+}
+
impl WhpProcessor<'_> {
+ /// Points RIP back at the instruction that put the VP into `mp_state`
+ /// (HLT for halt, RDMSR for the guest idle MSR) if that is the
+ /// instruction just before RIP. Returns the original RIP if it did.
+ fn rewind_to_suspend_instruction(
+ &mut self,
+ vtl: Vtl,
+ mp_state: vp::MpState,
+ ) -> Result<Option<u64>, Error> {
+ let expected: &[u8] = match mp_state {
+ vp::MpState::Halted => &[0xf4], // hlt
+ vp::MpState::Idle => &[0x0f, 0x32], // rdmsr
+ vp::MpState::Running | vp::MpState::WaitForSipi => return Ok(None),
+ };
+ let wvp = self.vp.whp(vtl);
+ let (cs, rip, efer) = get_registers!(
+ wvp,
+ [
+ whp::RegisterSegment::Cs,
+ whp::Register64::Rip,
+ whp::Register64::Efer,
+ ]
+ )
+ .for_op("get registers")?;
+ let long_mode = efer & x86defs::X64_EFER_LMA != 0
+ && x86defs::SegmentAttributes::from(cs.Attributes).long();
+ let start = rip.wrapping_sub(expected.len() as u64);
+ for (i, &byte) in expected.iter().enumerate() {
+ let offset = start.wrapping_add(i as u64);
+ let gva = if long_mode {
+ offset
+ } else {
+ cs.Base.wrapping_add(offset) & 0xffff_ffff
+ };
+ let Ok(gpa) = wvp
+ .translate_gva(
+ gva,
+ whp::abi::WHvTranslateGvaFlagValidateExecute
+ | whp::abi::WHvTranslateGvaFlagPrivilegeExempt,
+ )
+ .for_op("translate gva")?
+ else {
+ return Ok(None);
+ };
+ let mut b = [0];
+ if self.vp.partition.gm.read_at(gpa, &mut b).is_err() || b[0] != byte {
+ return Ok(None);
+ }
+ }
+ wvp.set_register(whp::Register64::Rip, start)
+ .for_op("set rip")?;
+ Ok(Some(rip))
+ }
+
+ /// Sets the register state. With the hypervisor's APIC, a new RIP
+ /// invalidates the check in [`Self::rewind_to_suspend_instruction`], so
+ /// redo it for a VP that is in a halt or idle suspend state.
+ pub(crate) fn restore_registers(
+ &mut self,
+ vtl: Vtl,
+ value: &vp::Registers,
+ ) -> Result<(), Error> {
+ let LocalApicKind::Offloaded = self.vp.partition.vtlp(vtl).lapic else {
+ return self.vp.set_register_state(vtl, value);
+ };
+ let activity = self.save_activity(vtl)?;
+ self.state.vtls[vtl].restored_suspend = None;
+ self.vp.set_register_state(vtl, value)?;
+ if matches!(activity.mp_state, vp::MpState::Halted | vp::MpState::Idle) {
+ self.restore_activity(vtl, &activity)?;
+ }
+ Ok(())
+ }
+
pub(crate) fn save_activity(&mut self, vtl: Vtl) -> Result<vp::Activity, Error> {
let activity = match self.vp.partition.vtlp(vtl).lapic {
LocalApicKind::Emulated(_) => {
@@ -203,6 +296,9 @@ pub(crate) fn save_activity(&mut self, vtl: Vtl) -> Result<vp::Activity, Error>
&self.vp.whp(vtl).get_apic().for_op("get apic state")?,
);
activity.nmi_pending = apic.hv_apic_nmi_pending();
+ if let Some(restored) = self.state.vtls[vtl].restored_suspend {
+ activity.mp_state = restored.mp_state;
+ }
activity
}
};
@@ -247,7 +343,35 @@ pub(crate) fn restore_activity(&mut self, vtl: Vtl, value: &vp::Activity) -> Res
self.vp.set_register_state(vtl, &value)?;
}
LocalApicKind::Offloaded => {
- self.vp.set_register_state(vtl, value)?;
+ // Undo an earlier rewind so that RIP is architectural again.
+ if let Some(restored) = self.state.vtls[vtl].restored_suspend.take() {
+ self.vp
+ .whp(vtl)
+ .set_register(whp::Register64::Rip, restored.rip)
+ .for_op("set rip")?;
+ }
+ let mut mp_state = value.mp_state;
+ if let Some(rip) = self.rewind_to_suspend_instruction(vtl, mp_state)? {
+ self.state.vtls[vtl].restored_suspend = Some(RestoredSuspend { mp_state, rip });
+ mp_state = vp::MpState::Running;
+ } else if matches!(mp_state, vp::MpState::Halted | vp::MpState::Idle) {
+ tracelimit::warn_ratelimited!(
+ ?mp_state,
+ "restoring a suspend state without its instruction before rip; \
+ the hypervisor may resume the vp"
+ );
+ }
+ self.vp.set_register_state(
+ vtl,
+ &vp::Activity {
+ mp_state,
+ nmi_pending: value.nmi_pending,
+ nmi_masked: value.nmi_masked,
+ interrupt_shadow: value.interrupt_shadow,
+ pending_event: value.pending_event,
+ pending_interruption: value.pending_interruption,
+ },
+ )?;
// Set the NMI pending bit via the APIC.
let mut apic = vp::ApicRegisters::from_page(
&self.vp.whp(vtl).get_apic().for_op("get apic state")?,
diff --git a/vmm_core/virt_whp/src/lib.rs b/vmm_core/virt_whp/src/lib.rs
index f3088f4f4..871e2cb1f 100644
--- a/vmm_core/virt_whp/src/lib.rs
+++ b/vmm_core/virt_whp/src/lib.rs
@@ -249,6 +249,11 @@ struct PerVtlRunState {
deliverability_notifications: HvDeliverabilityNotificationsRegister,
// Only used when `hv` is `None`.
vp_assist_page: u64,
+ /// A restored halt or idle state that is being re-entered by executing
+ /// the suspending instruction again (see `apic::RestoredSuspend`).
+ #[cfg(guest_arch = "x86_64")]
+ #[inspect(debug)]
+ restored_suspend: Option<apic::RestoredSuspend>,
}
impl PerVtlRunState {
@@ -269,6 +274,8 @@ pub fn new(lapic: &LocalApicKind, hv: &Hv1State, vp_info: &TargetVpInfo, vtl: Vt
},
deliverability_notifications: 0.into(),
vp_assist_page: 0,
+ #[cfg(guest_arch = "x86_64")]
+ restored_suspend: None,
}
}
@@ -359,6 +366,8 @@ fn reset(&mut self, is_bsp: bool, prot_access: &mut dyn hv1_emulator::VtlProtect
hv,
deliverability_notifications,
vp_assist_page,
+ #[cfg(guest_arch = "x86_64")]
+ restored_suspend,
} = self;
#[cfg(guest_arch = "x86_64")]
@@ -375,6 +384,10 @@ fn reset(&mut self, is_bsp: bool, prot_access: &mut dyn hv1_emulator::VtlProtect
*deliverability_notifications = 0.into();
*vp_assist_page = 0;
+ #[cfg(guest_arch = "x86_64")]
+ {
+ *restored_suspend = None;
+ }
}
}
diff --git a/vmm_core/virt_whp/src/vp.rs b/vmm_core/virt_whp/src/vp.rs
index 30549ffe6..6e57e33b5 100644
--- a/vmm_core/virt_whp/src/vp.rs
+++ b/vmm_core/virt_whp/src/vp.rs
@@ -371,6 +371,14 @@ pub(crate) async fn run_vp(
// Set the lazy EOI bit just before running.
let lazy_eoi = self.sync_lazy_eoi();
+ // From here on the VP executes the instruction that a restore
+ // rewound to, so its register state is architectural again.
+ #[cfg(guest_arch = "x86_64")]
+ {
+ let vtl = self.state.active_vtl;
+ self.state.vtls[vtl].restored_suspend = None;
+ }
+
let mut runner = self.current_whp().runner();
let exit = runner
.run()
diff --git a/vmm_core/virt_whp/src/vp_state.rs b/vmm_core/virt_whp/src/vp_state.rs
index a187c12d6..434ef1808 100644
--- a/vmm_core/virt_whp/src/vp_state.rs
+++ b/vmm_core/virt_whp/src/vp_state.rs
@@ -89,11 +89,15 @@ fn commit(&mut self) -> Result<(), Self::Error> {
}
fn registers(&mut self) -> Result<vp::Registers, Self::Error> {
- self.run.vp.get_register_state(self.vtl)
+ let mut regs: vp::Registers = self.run.vp.get_register_state(self.vtl)?;
+ if let Some(restored) = self.run.state.vtls[self.vtl].restored_suspend {
+ regs.rip = restored.rip;
+ }
+ Ok(regs)
}
fn set_registers(&mut self, value: &vp::Registers) -> Result<(), Self::Error> {
- self.run.vp.set_register_state(self.vtl, value)
+ self.run.restore_registers(self.vtl, value)
}
fn activity(&mut self) -> Result<vp::Activity, Self::Error> {
Tested on Windows 11 26H2 (build 26300), on a Xeon 8573C host and an EPYC 9V74 host.
- Dominant language
- Rust
- Stars
- 848
- Forks
- 63
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 571
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from microsoft/nvx
-
area: ci
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
area: ci
Difficulty 1/5 Under an hour Newbie friendliness 90/100
Maintainers usually reply within 1 day
-
openvmm: virtiofs: aggregate synthetic root st_nlink counts single-file shares as subdirectoriesOpenarea: openvmm
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
microsoft/nvx#1632 · 1 comment ·
Maintainers usually reply within 1 day
-
area: openvmm
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
microsoft/nvx#1628 · 1 comment ·
Maintainers usually reply within 1 day
-
area: openvmm
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
microsoft/nvx#1629 · 1 comment ·
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
obi1kenobi/cargo-semver-checks#1814 ·
Maintainers usually reply within 1 day
-
community differential help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
hffgzktbcc-blip/ounce-core#489 · 1 comment ·
Maintainers usually reply within 1 day
-
needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
pbakaus/impeccable#1560 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 68/100
pact-foundation/pact-plugins#125 ·