Bare Metal Gotchas

Hard-won lessons from real hardware debugging. Violating any of these will crash on bare metal while appearing to work fine in VMs. CLAUDE.md carries the short-form list as a per-turn reminder; the full incident histories live here.

ISR / IDT

• No LAPIC TPR writes in ISR path. The IDT isr_handler must track IRQL in software only -- no lapic_write(LAPIC_REG_TPR, ...) on interrupt entry/exit. The LAPIC hardware handles vector priority masking via ISR/PPR. TPR writes break emulated LAPIC on WHPX/VBox/TCG.
• The LAPIC TPR has exactly three writers -- do not add a fourth without updating current_irql. KeRaiseIrql/KeLowerIrql (irql.c) skip the TPR MMIO store when an IRQL transition does not change the TPR class (PASSIVE<->APC, the 0xFF system band, nested same-level). This is only safe because the hardware TPR always mirrors irql_to_tpr(current_irql): the sole writers are irql_write_tpr_value() and the two lapic.c init writes (both TPR=0=PASSIVE). A new out-of-band lapic_write(LAPIC_REG_TPR, ...) that sets the TPR without updating current_irql would make a later same-class KeLowerIrql skip a now-necessary write and under-mask interrupts. test_sched.c pins the mapping half of the invariant.
• No CLAC/STAC without SMAP CPUID. clac and stac cause #UD on CPUs without SMAP in CPUID -- including QEMU TCG and VirtualBox NEM. The ISR common stub must NOT use clac/stac until SMAP is actually enabled via per-process page tables. This was the root cause of the 2026-03-28 "hardware interrupts crash on bare metal/TCG" issue.
• GS_BASE must be set before any interrupt fires. smp_early_bsp_init() is called as the first thing in boot_phase0() -- before serial init. On bare metal, GS_BASE defaults to 0; smp_this_cpu() reads garbage from the real-mode IVT at physical address 0 instead of NULL, crashing the IRQL tracking in isr_handler.

MMU / Memory

• No SMEP/SMAP until per-process page tables. The shared identity-mapped address space uses 2 MiB pages; user stacks are kmalloc'd from the kernel heap, so user and kernel data share the same 2 MiB pages. Clearing User bit from "kernel" pages also blocks user-mode stack access. hv_supports_cr4_smep_smap() returns 0 for PLATFORM_BARE_METAL.
• No MMIO through WB-cached pages. HPET, ECAM, NVMe BARs, and future GPU BARs must use vmm_map_mmio_uc() with UC (uncacheable) attributes. The bootloader identity-maps everything as WB. LAPIC/IOAPIC work only because MTRRs override those ranges to UC. HPET calibration uses vmm_map_mmio_uc() (fixed 2026-03-29).
• CR3 reloads on WHPX can reset per-vCPU MSRs. write_cr3(read_cr3()) (used by vmm_flush_tlb_all) triggers a VMEXIT on WHPX that can reset the PAT MSR (and potentially other per-vCPU MSRs) to Intel defaults. Any MSR that must persist across TLB flushes must be re-programmed after CR3 reloads. Discovered 2026-04-13: cpu_configure_pat() wrote WC to PAT entry 1, then vmm_promote_to_1g() flushed TLB via CR3 reload, and PAT readback showed WT (Intel default). Fix: re-program PAT after page table modifications that flush TLB.
• User-mode ELF range (0x800000--0x900000). Constants defined in include/kernel/mm/user_range.h (single source of truth). Three C files include it: vmm.c, pmm.c, task.c. user/user.ld must be updated manually if the base changes. Static asserts, runtime PMM bitmap verify, guard page at 0x900000, and unit test enforce sync.

SMP Startup

• No Init Level De-Assert IPI. The broadcast Init Level De-Assert (ICR: INIT | ALL | LEVEL_DEASSERT) was deprecated since Intel P6 (1995) and is a hardware no-op on all x86-64 CPUs. On WHPX with 2+ vCPUs it hangs because the hypervisor traps the broadcast and stalls waiting for the not-yet-booted AP. Removed entirely 2026-04-01. The per-AP INIT->de-assert->SIPI sequence in lapic_send_init() is unrelated and required.

Devices

• NVMe I/O unreliable on QEMU WHPX. QEMU's emulated NVMe controller processes doorbell MMIO writes asynchronously through its event loop under WHPX. The vCPU polls the CQ at native speed before the main thread processes the command, causing intermittent admin and I/O timeouts. Not a driver bug -- the NVMe driver has correct barriers (wmb/rmb). Real NVMe hardware handles PCIe DMA synchronously with cache snooping. (This line previously also credited the driver with clflush, which it has never contained: until the cross-boot durability work below there was no clflush anywhere in the tree, and the only cache maintenance today lives in src/kernel/cache.c and its cross-boot evidence callers. A doc claim that a subsystem performs cache maintenance is the kind that survives precisely because a repo-wide grep for the instruction now returns hits somewhere else.) NVMe test uses TCG; normal boot (SATA) uses WHPX fine.

Cross-boot durability

• A record that must survive a reset needs a cache writeback, not a barrier. mb()/wmb()/rmb() order VISIBILITY between CPUs and say nothing about durability: on x86 every CPU already sees a store through the coherent cache, so a modified line can sit dirty indefinitely while satisfying every fence in the tree. Intel SDM Vol. 3A Table 9-1 footnote 6 states that internal caches are invalid after power-up and RESET, and nothing in the SDM writes modified lines back first -- so a crash record published with a plain store into write-back memory and followed by a platform reset can never reach DRAM. Every emulator this repo gates on models no cache hierarchy, so QEMU, KVM and WHPX all pass the sequence whether or not the bug exists; it appears only on real silicon. Use cache_writeback_range() from include/kernel/cache.h. Found 2026-08-15 on the cross-boot evidence page at 0x80000 and the klog crash region, both of which had exactly this defect.
• Publish a cross-boot record by storing its publication word LAST, not by ordering flushes. Durably un-publish, write and flush the whole body while the publication word reads zero, then store and flush that word. Ordering flushes behind an already-stored publication word (body lines first, publication line last) looks equivalent and is not: it holds only where CLFLUSH gives line granularity and collapses on the WBINVD fallback, which commits many lines with no ordering boundary and can land a valid magic and CRC over a still-stale body.
• Durably clearing a valid record requires an owner. The un-publish flush is safe only where something arbitrates who may replace the record, because a writer that dies between the clear and the republish has destroyed the old record and written nothing. The panic evidence page has that arbitration and does flush its un-publish; the klog crash region has none and deliberately does not, letting the body flush carry the clear instead. Both fail closed on CRC; only one of them can also lose a complete record to a nested panic.
• CLFLUSH is CPUID-gated and its fencing is vendor-dependent. Gate on CPUID.01H:EDX[19] and take the line size from CPUID.01H:EBX[15:8] (in 8-byte units), on the EXECUTING CPU -- a cached BSP-built feature table is the wrong shape for a panic path that can run on an AP or before enumeration. Intel documents CLFLUSH as ordered against prior writes; AMD documents it as ordered by MFENCE and explicitly NOT by SFENCE, so bracket a batch with MFENCE rather than SFENCE.
• Never bound a line-flush loop by an end address. p += line wraps to 0 at the top of the address space and a p < end loop then restarts from the bottom and never terminates. The kernel image window ends at UINT64_MAX, so this is reachable. Count the lines instead, and decide "does this range wrap" from the LAST BYTE, never the exclusive end -- a range ending exactly at UINT64_MAX has an exclusive end of 0 and an end-based test misreads it as a wrap.

FPU / SIMD

• FXSAVE/XSAVE buffer: set BOTH FCW and MXCSR defaults. task_alloc_xsave() zeroes the buffer, then must set FCW at offset 0 to 0x037F (all x87 exceptions masked, Intel SDM reset value) and MXCSR at offset 24 to 0x1F80 (all SIMD exceptions masked). Zeroed FCW causes #MF (vector 16) on the first imprecise x87 FP op; zeroed MXCSR causes #XM (vector 19) on the first SSE instruction. On WHPX (has XSAVE), XRSTOR init optimization masked the FCW=0 bug. On TCG (uses FXRSTOR), FCW=0 was loaded directly, freezing boot at the font renderer (2026-04-13). MXCSR=0 caused WHPX BSOD the same day.
• CR0.TS must be cleared before ALL FPU instructions. XSAVE, XRSTOR, FXSAVE, and FXRSTOR all fault with #NM when CR0.TS=1 -- not just restores. Both the preemptive schedule() and cooperative schedule_now() must CLTS before any save or restore. Missing CLTS before XRSTOR caused the original WHPX freeze (2026-04-13).
• Never compile SSE2 fallback code with -mavx2. The compiler emits VEX-encoded instructions (vmovdqu instead of movdqu) for all SSE operations when -mavx2 is active, even around inline asm and in scalar tails. On CPUs without AVX (TCG qemu64, pre-Sandy Bridge bare metal), VEX instructions cause #UD. Split SIMD files: AVX2 functions in a -mavx2 translation unit, SSE2 fallbacks and dispatch in a -msse2 translation unit. This crashed TCG boot when memops_sse.c was compiled as part of memops.c with -mavx2 (2026-04-13).

MSR Probe

• msr_try_read() is a no-crash guarantee, not an existence check. WHPX silently absorbs reads of unknown MSRs (returns 0, no #GP), so msr_try_read() returning 0 does NOT prove the MSR exists. Always gate on cpu_has() or CPUID first; use msr_try_read() only as a secondary safety net. Pattern: if (!cpu_has(FEATURE)) return; if (msr_try_read(MSR, &val) != 0) { fallback; }. Discovered 2026-04-13 when msr_try_read(0xFFFFFFFF) returned success on WHPX.
• msr_try_read() requires the kernel IDT to be loaded. The probe installs a custom #GP handler in a C-level handlers[] table consulted only by our ISR stubs, which only dispatch after idt_init() has lidt-ed the kernel IDT. Before that, IDTR still points at the UEFI IDT, and any real #GP (KVM trapping IA32_MPERF, real hardware trapping an unavailable MSR) is caught by UEFI's handler and halts the system. msr_try_read() now gates on idt_is_loaded() and returns -1 pre-IDT; callers must have a fallback path. simd_enable_avx512() was moved from boot_phase0 to boot_phase1 (after idt_init()) so its throttle guard keeps working on bare metal. Discovered 2026-04-17 when the KVM smoke test crashed at MPERF probe in phase 0. Under virtualization CPUID lies (KVM with -cpu host passes AVX512F through but traps MPERF); always pair CPUID gate with msr_try_read() fallback.

Init Ordering

• No thread_create for deferred init. boot_run_deferred() MUST run inline in Phase 3, not on a background thread. compositor_run() is an infinite event loop on the BSP that starves any kernel thread created just before it. Moving deferred inits to a thread caused VirtIO input and VBox mouse drivers to never initialize, breaking absolute cursor positioning on QEMU/VBox (2026-04-05). Per-thread fault isolation requires SEH (TODO-10).

Disk-Sourced Config

• Disk-sourced config files get dynamic buffers + hard-fail overflow. resources/boot/boot.conf was originally read into a 4096-byte stack buffer with a silent [WARN] truncating on overflow. On 2026-04-21 doc-comment growth pushed the file to 4.5 KiB; the truncation dropped the patch-appended test=1 line at EOF and every test=1 boot ran zero tests without any visible error. Fixed by switching bootx64.c boot_conf_load() to gBS->AllocatePool(FileSize + 1) via a GetInfo(&EFI_FILE_INFO_ID, ...) size probe, with a 1 MiB sanity cap that boot_fatal()s on hit (never [WARN] + continue). Codified in boot-code-quality Gate 14. Any future parser reading disk/user input should follow the same pattern: dynamic-size first, sanity cap second, silent truncation never.

Boot Handoff Trust

• A producer-set BOOT_PAYLOAD_FLAG_RESERVED bit is NOT proof the reservation pass pinned the range. The Phase-0 pass now declines to pin a payload whose declared length falls outside its type's contract (boot_seed_length_reservable() for BOOT_PAYLOAD_RANDOM_SEED, src/kernel/mm/boot_reserved.c), because the alternative is worse: reserving an attacker-declared gigabyte span permanently starves the PMM and the boot dies in heap_init rather than degrading. The flag stays SET on the descriptor it skipped, so any consumer that reads FLAG_RESERVED as "the PMM owns this" dereferences memory nothing reserved. Every consumer of a bounded type must apply the SAME predicate the pass applied. Shipped 2026-08-29 with 01-boot-platform/TODO-12 section 11, and the follow-on [high] was found in the very next review round: canary_seed_desc_ok() still used its own length >= 16 and had to be routed through the shared predicate. The remaining eight payload types are still unbounded (01-boot-platform/TODO-01 section 28).
• canary_init() dereferences a loader-published range BEFORE the IDT exists. It runs from kernel_main between Phase 0 and Phase 1, after pmm/vmm/heap init and after the loader-owned boot stack's guard page is live, but with no interrupt handlers installed. A fault there is a triple fault with no diagnostic, so the pre-IDT peek must refuse anything it cannot prove is pinned rather than trusting a bound check of its own. No emulator produces a malformed descriptor, so this whole class is invisible under QEMU, WHPX and VirtualBox alike -- OVMF always hands over a well-formed one.

Uninitialised .bss in the Loader

• UEFI does not zero a loader image's .bss, so an early static reads FIRMWARE POISON, not 0. Edk2-derived firmware fills freed pool memory with 0xAF bytes, and BOOTX64's .bss is ordinary pool memory the firmware never scrubs; on a warm reboot it can instead hold whatever the previous boot left at that address. So a static consulted before its initialiser has run holds 0xAFAFAFAF or last boot's value. This is invisible under every emulator: OVMF under QEMU happens to hand over zeroed pages, and the class only appears on real firmware. Two live defects came from it, both shipped 2026-09-03 with 01-boot-platform/TODO-14 sections 18 and 20: serial_early_print drove port 0xAFAF as if it were a configured UART, and boot_entries_parser.c skipped its CRC table build, computed every CRC over a poisoned table, and rejected a VALID boot store as CRC_MISMATCH -- sending the loader to its invalid-store fallback, which can select a different kernel than the store asked for.
• Reset first; the cookie is defence in depth, and a BOOLEAN buys nothing. early_diag_reset() runs as the first statement of efi_main, ahead of any code that can emit a diagnostic (src/boot/uefi/bootx64.c:505), and the parser resets at its own entry -- that reset is the load-bearing half, because a constant cookie discriminates poison from ready and can never discriminate THIS boot from the last one. Readiness is then a wide exact-match cookie (EARLY_DIAG_READY at bootx64.c:422, BOOT_ENTRIES_CRC32_READY at boot_entries_parser.c:59), tested with !=, never with !x: 0xAF is non-zero and reads as "configured", so a boolean flag is exactly the wrong shape. A path that somehow runs ahead of the reset then degrades to silence instead of to port I/O on an address the poison spells.
• The cookie must be volatile or the compiler deletes it. These objects have internal linkage and never have their address taken, so clang may assume nothing outside the program writes them and narrow the representation to the single bit it can prove is needed. Measured 2026-09-03 on the real -O2 build objects: the non-volatile static int g_crc32_ready was emitted at SIZE 1, and both bootx64 cookies became 1-byte objects with cmpb $1 consumers, the 32-bit constant nowhere in the image -- silently reducing the wide compare back to the boolean it exists to avoid. That narrowing is sound only under the assumption firmware poison violates. volatile states the truth and keeps the wide compare. A plain = 0 initialiser is not a third option: it is documentation, since .bss is what the initialiser nominally targets.