Boot Validation and Certification Matrix

What is it?

The certification matrix answers one question: is this build boot-certified across QEMU, VirtualBox, Hyper-V and real hardware? It is a schema-checked table (tools/boot-cert/boot-cert.yml) that maps every boot-platform roadmap file to at least one certification row, a result format that automated suites and manual bare-metal runs both use, and a lint that fails when the matrix, a row or a result is malformed or incomplete.

The schemas, the lint and the automated reboot-reliability suite ship and run in the tooling tests. The storage, security, recovery, network-boot and firmware-sanity suites that would fill the matrix with real results are open, as is the release dashboard, so today the matrix proves its own shape and coverage rather than certifying a release.

How does it work?

boot-cert.yml has 29 rows, nine platform classes (QEMU and VM classes plus five bare-metal classes) and three tiers (nightly, rc, stable). Each row names its owning roadmap file, the platforms it runs on, the tiers that require it, how automated it is, and the artifact the release gate reads.

lint.py validates the file against boot-cert.schema.json and then checks two rules a schema cannot: a row required by any tier must name its evidence, and every boot-platform roadmap file must have at least one row. It reads the roadmap file list from the tree on each run, so a new roadmap file without a row fails immediately. A single run's result must match result.schema.json (row, status of pass, fail, skip or degraded, build, machine, tier, platform class, timestamp, optional logs and reason).

The automated suite is boot_reliability.py. It keeps a launcher per platform class: qemu-tcg uses scripts/machines/boot-test-qemu.sh, virtualbox uses scripts/release/boot-test-vbox.sh, and qemu-whpx and hyperv have no Linux-host launcher yet. Every boot is a fresh VM launch (QEMU runs with -no-reboot), so no guest-initiated reboot is exercised. On QEMU, a cold boot starts from fresh firmware variables and the warm series shares one variables file, seeded fresh on its first boot and reused by the later ones. VirtualBox has no cold and warm distinction: its launcher creates and destroys a throwaway VM on every run. It runs a number of cold and warm boots, classifies each captured serial log, and reduces the series to a decision against a per-tier flake threshold (one flake for nightly and rc, none for stable). The aggregate matches reliability.schema.json. Its --self-check mode validates the launcher table and schemas without booting anything, which is what the tooling suite runs.

The manual half is the Bare-Metal Boot Lab: five hardware classes mapped to the bare-metal platform classes, a per-machine inventory, and a run record in the result.schema.json shape.

scripts/test-smoke-matrix.sh is related but separate. It is the four-leg smoke gate (KVM and TCG, one and two CPUs) run for boot-path changes; it does not produce matrix results.

What are its interfaces?

How do I use it?

python3 tools/boot-cert/lint.py
python3 tools/boot-cert/boot_reliability.py --self-check --platform qemu-tcg --tier stable
python3 tools/boot-cert/test_boot_reliability.py

The self-check prints boot-reliability self-check OK. scripts/test-tooling.sh runs all three and reports boot-cert lint passes (schema + coverage + evidence) among its checks.

On a host with QEMU, a live reliability run boots the image repeatedly:

python3 tools/boot-cert/boot_reliability.py --platform qemu-tcg --tier stable --row-id boot-cert-matrix \
    --build-id <build> --machine-id <host> --cold 3 --warm 2

Pass --row-id with a row that exists in boot-cert.yml: the default, boot-reliability, is not a matrix row, so python3 tools/boot-cert/lint.py --results <run-dir>/results.json rejects results written with it.

For the separate smoke gate, bash scripts/test-smoke-matrix.sh prints one line per leg and ends with SMOKE MATRIX PASSED or SMOKE MATRIX FAILED.

What is not implemented yet?

• Storage and media boot suite: Storage/Media Boot Suite.
• Security boot suite (Secure Boot on and off, signed and unsigned, dbx, SBAT, TPM present and absent): Security Boot Suite.
• Recovery and rollback suite (A/B rollback, recovery fallback, watchdog reboot, hibernation fallback): Recovery and Rollback Suite.
• Network boot suite (PXE, HTTP Boot, local fixtures): Network Boot Suite.
• A share-safe boot support bundle with redaction: Boot Support Bundle Collector.
• The release gate and dashboard that consume results: Release Gate and Dashboard.
• An operator guide for certification runs: Certification Docs.
• A firmware-sanity gate that fails a release on degraded firmware tables: Firmware Sanity Certification Gate.
• A pixel check on captured error screens, so a screen drawn in the wrong place fails: Error-Screen Pixel Oracle.
• Only QEMU warm boots after the first reuse firmware NVRAM, VirtualBox reuses none, and a guest-initiated reboot (reset path, device state across reset) is not tested: VM Automation Suite.
• WHPX and Hyper-V have no launcher on a Linux host; they wait on the Windows-side runner: CI Boot Matrix for Every Artifact.

How does it compare with Windows 11 and Linux?

Windows 11 hardware certification runs through the Windows Hardware Lab Kit, a closed test suite vendors run against published requirements, including reliability testing. Linux has no single equivalent: each distribution runs its own CI, and the Firmware Test Suite (FWTS) checks ACPI and UEFI tables separately. Impossible OS has an open, roadmap-mapped matrix and a repeat-boot flake gate, but without the remaining suites and the release gate it does not yet certify a build end to end.

See also

• Boot Validation and Certification Matrix roadmap
• Bare-Metal Boot Lab, the manual deep dive
• A/B Dual-Slot Boot and Automatic Rollback
• Network Boot (PXE and HTTP)