openvmm: in virt_whp, a VP saved in `cli; hlt` runs past the HLT after restore, but still reads back as `Halted`
I maintainer di solito rispondono entro 1 giorno
Nessuno ha ancora preso questa issue.
Valutazione
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Idoneità per principianti
- 35/100
- Tipo di issue
- Bug
- Chiarezza
- Specificata chiaramente
- Stato di attività
- Attiva
- Stack tecnologico
- rust
- Ambito
- backend, operating-systems
Direzione di ricerca
Start with vmm_core/virt_whp/src/apic.rs, especially restore_activity and the WHP APIC state handling described in the issue; then inspect vmm_core/virt/src/state.rs for the restore checks. Reproduce on a Windows host with WHP using the vp_state tests from repro-test.patch, which is not included in the repository yet. Done means halted and idle VPs remain suspended across all three restore paths, while state read-back and the existing single-field mutation behavior remain consistent.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Summary
This happens with the default --hypervisor whp (the hypervisor APIC), with or without --hv. OpenVMM saves a VP that is halted with interrupts off as MpState::Halted, which is correct. Restoring that state with restore_all succeeds, and activity() still returns Halted. But on its next run the VP executes the instructions after the HLT.
A VP idling through HV_X64_MSR_GUEST_IDLE (saved as Idle) behaves the same way. With sti; hlt, only a restore onto the same VP wakes it. KVM keeps the VP halted in all these cases.
Repro
This needs a Windows host with WHP. repro-test.patch adds vmm_core/virt_whp/tests/vp_state.rs, which has small helpers for building partitions, running guest code, and saving and restoring VPs. The guest is real-mode code: out 0x80, 'H'; cli; hlt; out 0x80, 'W'. The test below is abridged from that file.
fn suspended_vp_stays_suspended(config: Config, code: &'static [u8], mark: u8, fresh: bool) {
with_time(move |time| {
let mut vm = run_to_suspend(config, code, mark, time); // run until 'H', stop
let saved = vm.save(0);
let expected = vm.activity(0).mp_state; // Halted
if fresh {
// As `--restore-snapshot`: a new partition with the saved RAM.
let ram = vm.ram();
drop(vm);
let mut hv = whp(config);
vm = Vm::new(&mut hv, config, 1, time);
vm.mem.write_at(0, &ram).unwrap();
}
vm.restore(0, &saved).unwrap();
assert_eq!(vm.activity(0).mp_state, expected, "read back after restore_all");
let woke = vm.run_until_mark(b'W', Duration::from_secs(1));
assert!(!woke, "the VP was saved as {expected:?} but ran past the suspending instruction");
})
}
#[test]
fn halted_vp_stays_halted_after_restore_into_fresh_partition() {
suspended_vp_stays_suspended(WHP, HALT, b'H', true);
}
// Also: ..._onto_same_vp, and idle_vp_stays_idle_after_restore_into_fresh_partition (WHP_HV1).
$ cargo test -p virt_whp --test vp_state -- halted_vp idle_vp
the VP was saved as Halted but ran past the suspending instruction after restore_all (marks "W")
All three tests fail on main 42d4fe619. The read-back right after restore_all passes in each of them.
Cause
The behavior comes from WHP itself. restore_all sets registers, then activity (halt_suspend = 1), then the APIC page (apic.rs#L292-L319). We tested this by restoring subsets of the elements in different orders:
- Restoring any single element does not wake the VP.
- Restoring
registersandapictogether does, with or withoutactivity. - No order keeps the VP halted. If
activityis written afterapic, the halt is dropped and reads back asRunning. That would triprestore_all's debug check (state.rs#L98-L113), so changing the order is not a fix.
Fix Suggestion
fix.patch works around this inside virt_whp, for the hypervisor APIC only. On restore of Halted or Idle, if the instruction before RIP is HLT (or the RDMSR of the idle MSR), it moves RIP back onto that instruction and restores the VP as Running. The VP then suspends itself again on its first instruction. Until the VP runs, the getters keep returning the saved mp_state and RIP, so restore_all's check and a re-save are unchanged. With the patch, all three restore paths keep the VP suspended, and a sweep of single-field state mutations gives the same results as before.
fix.patch (click to expand)
diff --git a/vmm_core/virt_whp/src/apic.rs b/vmm_core/virt_whp/src/apic.rs
index 60fcb73ae..4df6f36d7 100644
--- a/vmm_core/virt_whp/src/apic.rs
+++ b/vmm_core/virt_whp/src/apic.rs
@@ -175,7 +175,100 @@ fn now(&mut self) -> VmTime {
}
}
+/// A halt or idle suspend state that `restore_activity` turned into a
+/// re-execution of the suspending instruction.
+///
+/// With the hypervisor's APIC, WHP does not keep a VP in a halt or idle
+/// suspend state that is restored along with the VP's register and APIC
+/// state: on its next run the VP continues after the HLT (or the RDMSR of
+/// HV_X64_MSR_GUEST_IDLE), even though the activity register still reads
+/// back as suspended, and a suspend state written after the APIC state is
+/// dropped. Instead of restoring the suspend state, RIP is pointed back at
+/// the suspending instruction and the VP is restored as running, so that it
+/// executes the instruction again and the hypervisor suspends it itself.
+///
+/// Until the VP runs, the state accessors still report the saved state:
+/// `mp_state` and `rip` (the architectural RIP, after the instruction).
+#[derive(Debug, Copy, Clone)]
+pub(crate) struct RestoredSuspend {
+ pub mp_state: vp::MpState,
+ pub rip: u64,
+}
+
impl WhpProcessor<'_> {
+ /// Points RIP back at the instruction that put the VP into `mp_state`
+ /// (HLT for halt, RDMSR for the guest idle MSR) if that is the
+ /// instruction just before RIP. Returns the original RIP if it did.
+ fn rewind_to_suspend_instruction(
+ &mut self,
+ vtl: Vtl,
+ mp_state: vp::MpState,
+ ) -> Result<Option<u64>, Error> {
+ let expected: &[u8] = match mp_state {
+ vp::MpState::Halted => &[0xf4], // hlt
+ vp::MpState::Idle => &[0x0f, 0x32], // rdmsr
+ vp::MpState::Running | vp::MpState::WaitForSipi => return Ok(None),
+ };
+ let wvp = self.vp.whp(vtl);
+ let (cs, rip, efer) = get_registers!(
+ wvp,
+ [
+ whp::RegisterSegment::Cs,
+ whp::Register64::Rip,
+ whp::Register64::Efer,
+ ]
+ )
+ .for_op("get registers")?;
+ let long_mode = efer & x86defs::X64_EFER_LMA != 0
+ && x86defs::SegmentAttributes::from(cs.Attributes).long();
+ let start = rip.wrapping_sub(expected.len() as u64);
+ for (i, &byte) in expected.iter().enumerate() {
+ let offset = start.wrapping_add(i as u64);
+ let gva = if long_mode {
+ offset
+ } else {
+ cs.Base.wrapping_add(offset) & 0xffff_ffff
+ };
+ let Ok(gpa) = wvp
+ .translate_gva(
+ gva,
+ whp::abi::WHvTranslateGvaFlagValidateExecute
+ | whp::abi::WHvTranslateGvaFlagPrivilegeExempt,
+ )
+ .for_op("translate gva")?
+ else {
+ return Ok(None);
+ };
+ let mut b = [0];
+ if self.vp.partition.gm.read_at(gpa, &mut b).is_err() || b[0] != byte {
+ return Ok(None);
+ }
+ }
+ wvp.set_register(whp::Register64::Rip, start)
+ .for_op("set rip")?;
+ Ok(Some(rip))
+ }
+
+ /// Sets the register state. With the hypervisor's APIC, a new RIP
+ /// invalidates the check in [`Self::rewind_to_suspend_instruction`], so
+ /// redo it for a VP that is in a halt or idle suspend state.
+ pub(crate) fn restore_registers(
+ &mut self,
+ vtl: Vtl,
+ value: &vp::Registers,
+ ) -> Result<(), Error> {
+ let LocalApicKind::Offloaded = self.vp.partition.vtlp(vtl).lapic else {
+ return self.vp.set_register_state(vtl, value);
+ };
+ let activity = self.save_activity(vtl)?;
+ self.state.vtls[vtl].restored_suspend = None;
+ self.vp.set_register_state(vtl, value)?;
+ if matches!(activity.mp_state, vp::MpState::Halted | vp::MpState::Idle) {
+ self.restore_activity(vtl, &activity)?;
+ }
+ Ok(())
+ }
+
pub(crate) fn save_activity(&mut self, vtl: Vtl) -> Result<vp::Activity, Error> {
let activity = match self.vp.partition.vtlp(vtl).lapic {
LocalApicKind::Emulated(_) => {
@@ -203,6 +296,9 @@ pub(crate) fn save_activity(&mut self, vtl: Vtl) -> Result<vp::Activity, Error>
&self.vp.whp(vtl).get_apic().for_op("get apic state")?,
);
activity.nmi_pending = apic.hv_apic_nmi_pending();
+ if let Some(restored) = self.state.vtls[vtl].restored_suspend {
+ activity.mp_state = restored.mp_state;
+ }
activity
}
};
@@ -247,7 +343,35 @@ pub(crate) fn restore_activity(&mut self, vtl: Vtl, value: &vp::Activity) -> Res
self.vp.set_register_state(vtl, &value)?;
}
LocalApicKind::Offloaded => {
- self.vp.set_register_state(vtl, value)?;
+ // Undo an earlier rewind so that RIP is architectural again.
+ if let Some(restored) = self.state.vtls[vtl].restored_suspend.take() {
+ self.vp
+ .whp(vtl)
+ .set_register(whp::Register64::Rip, restored.rip)
+ .for_op("set rip")?;
+ }
+ let mut mp_state = value.mp_state;
+ if let Some(rip) = self.rewind_to_suspend_instruction(vtl, mp_state)? {
+ self.state.vtls[vtl].restored_suspend = Some(RestoredSuspend { mp_state, rip });
+ mp_state = vp::MpState::Running;
+ } else if matches!(mp_state, vp::MpState::Halted | vp::MpState::Idle) {
+ tracelimit::warn_ratelimited!(
+ ?mp_state,
+ "restoring a suspend state without its instruction before rip; \
+ the hypervisor may resume the vp"
+ );
+ }
+ self.vp.set_register_state(
+ vtl,
+ &vp::Activity {
+ mp_state,
+ nmi_pending: value.nmi_pending,
+ nmi_masked: value.nmi_masked,
+ interrupt_shadow: value.interrupt_shadow,
+ pending_event: value.pending_event,
+ pending_interruption: value.pending_interruption,
+ },
+ )?;
// Set the NMI pending bit via the APIC.
let mut apic = vp::ApicRegisters::from_page(
&self.vp.whp(vtl).get_apic().for_op("get apic state")?,
diff --git a/vmm_core/virt_whp/src/lib.rs b/vmm_core/virt_whp/src/lib.rs
index f3088f4f4..871e2cb1f 100644
--- a/vmm_core/virt_whp/src/lib.rs
+++ b/vmm_core/virt_whp/src/lib.rs
@@ -249,6 +249,11 @@ struct PerVtlRunState {
deliverability_notifications: HvDeliverabilityNotificationsRegister,
// Only used when `hv` is `None`.
vp_assist_page: u64,
+ /// A restored halt or idle state that is being re-entered by executing
+ /// the suspending instruction again (see `apic::RestoredSuspend`).
+ #[cfg(guest_arch = "x86_64")]
+ #[inspect(debug)]
+ restored_suspend: Option<apic::RestoredSuspend>,
}
impl PerVtlRunState {
@@ -269,6 +274,8 @@ pub fn new(lapic: &LocalApicKind, hv: &Hv1State, vp_info: &TargetVpInfo, vtl: Vt
},
deliverability_notifications: 0.into(),
vp_assist_page: 0,
+ #[cfg(guest_arch = "x86_64")]
+ restored_suspend: None,
}
}
@@ -359,6 +366,8 @@ fn reset(&mut self, is_bsp: bool, prot_access: &mut dyn hv1_emulator::VtlProtect
hv,
deliverability_notifications,
vp_assist_page,
+ #[cfg(guest_arch = "x86_64")]
+ restored_suspend,
} = self;
#[cfg(guest_arch = "x86_64")]
@@ -375,6 +384,10 @@ fn reset(&mut self, is_bsp: bool, prot_access: &mut dyn hv1_emulator::VtlProtect
*deliverability_notifications = 0.into();
*vp_assist_page = 0;
+ #[cfg(guest_arch = "x86_64")]
+ {
+ *restored_suspend = None;
+ }
}
}
diff --git a/vmm_core/virt_whp/src/vp.rs b/vmm_core/virt_whp/src/vp.rs
index 30549ffe6..6e57e33b5 100644
--- a/vmm_core/virt_whp/src/vp.rs
+++ b/vmm_core/virt_whp/src/vp.rs
@@ -371,6 +371,14 @@ pub(crate) async fn run_vp(
// Set the lazy EOI bit just before running.
let lazy_eoi = self.sync_lazy_eoi();
+ // From here on the VP executes the instruction that a restore
+ // rewound to, so its register state is architectural again.
+ #[cfg(guest_arch = "x86_64")]
+ {
+ let vtl = self.state.active_vtl;
+ self.state.vtls[vtl].restored_suspend = None;
+ }
+
let mut runner = self.current_whp().runner();
let exit = runner
.run()
diff --git a/vmm_core/virt_whp/src/vp_state.rs b/vmm_core/virt_whp/src/vp_state.rs
index a187c12d6..434ef1808 100644
--- a/vmm_core/virt_whp/src/vp_state.rs
+++ b/vmm_core/virt_whp/src/vp_state.rs
@@ -89,11 +89,15 @@ fn commit(&mut self) -> Result<(), Self::Error> {
}
fn registers(&mut self) -> Result<vp::Registers, Self::Error> {
- self.run.vp.get_register_state(self.vtl)
+ let mut regs: vp::Registers = self.run.vp.get_register_state(self.vtl)?;
+ if let Some(restored) = self.run.state.vtls[self.vtl].restored_suspend {
+ regs.rip = restored.rip;
+ }
+ Ok(regs)
}
fn set_registers(&mut self, value: &vp::Registers) -> Result<(), Self::Error> {
- self.run.vp.set_register_state(self.vtl, value)
+ self.run.restore_registers(self.vtl, value)
}
fn activity(&mut self) -> Result<vp::Activity, Self::Error> {
Tested on Windows 11 26H2 (build 26300), on a Xeon 8573C host and an EPYC 9V74 host.
- Lingua principale
- Rust
- Stelle
- 848
- Fork
- 63
- Merge medio
- 13h 37m
- PR unite (30g)
- 571
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Ha un modello di pull request
- Nessuna guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di microsoft/nvx
-
area: ci
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
I maintainer di solito rispondono entro 1 giorno
-
area: ci
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 90/100
I maintainer di solito rispondono entro 1 giorno
-
openvmm: virtiofs: aggregate synthetic root st_nlink counts single-file shares as subdirectoriesApertaarea: openvmm
Difficoltà 2/5 1-3 ore Idoneità per principianti 72/100
microsoft/nvx#1632 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
area: openvmm
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
microsoft/nvx#1628 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
area: openvmm
Difficoltà 2/5 1-3 ore Idoneità per principianti 80/100
microsoft/nvx#1629 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di microsoft/nvx
Issue simili
-
priority: low
Difficoltà 1/5 1-3 ore Idoneità per principianti 83/100
storytold/vectorcraft#8387 · 2 commenti ·
I maintainer di solito rispondono entro 1 giorno
-
area:desktop kind:test priority:p3 ready
Difficoltà 2/5 1-3 ore Idoneità per principianti 91/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 68/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
obi1kenobi/cargo-semver-checks#1814 ·
I maintainer di solito rispondono entro 1 giorno
-
COMPARE.md symlink is brokenAperta
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 85/100
BurntSushi/jiff#715 ·
I maintainer di solito rispondono entro 1 giorno