Down the rabbit hole of tboot, E820 maps, and Xen PV PCI-passthrough domains

From an OpenXT bug report:

TL;DR: a minor adjustment had to be made in tboot so that it picks the right memory protection for itself in the E820 map. The bug only affected PV Linux guests with PCI-passthrough devices as correctly guessed above.[…]

OpenXT: adding anti-evil maid support, using UEFI and Xen in place of TXT

[…]In this PR OpenXT is extended to allow booting under UEFI while maintaining current security properties.,Changes to Measured Launch,,Booting OpenXT under UEFI introduces significant changes to the Measured Launch system of OpenXT by switching to the use of SRTM PCRs. Performing DRTM with TXT when booting under UEFI is not supported but can be implemented at a later date. OpenXT under UEFI relies on the shim EFI application to perform necessary measurements of critical boot components of OpenXT. OpenXT’s static PCR list is extended to include PCR4, PCR5, PCR7 and PCR8. Specific to OpenXT’s context, PCR4 holds the measurements of the shim, Xen and the dom0 kernel while PCR8 holds the measurements of openxt.cfg, the dom0 initrd image and the XSM policy. Any change in these components, selecting a boot entry other then the one used when Sealing took place and/or booting any application before the shim will result in the Measured Launch system tripping.[…]


Xen Security Advisory XSA-254 updated to V12 (on Spectre/Meltdown)

The XSA on Spectre/Meltdown has been updated again, with more info on ARM firmware:

Xen Security Advisory CVE-2017-5753,CVE-2017-5715,CVE-2017-5754 / XSA-254
version 12

Information leak via side effects of speculative execution


Corrections to ARM SP2 information:
* ARM 32-bit requires new firmware on some CPUs.
* Provide link to the ARM firmware page, accordingly.
* ARM 32-bit mitigations are complete for Cortex-A CPUs.
We do not have information for other ARM CPUs at this time.

Systems running all versions of Xen are affected. For SP1 and SP2, both Intel and AMD are vulnerable. Vulnerability of ARM processors to SP1 and SP2 varies by model and manufacturer. ARM has information on affected models on the following website. For SP3, only Intel processors are vulnerable. (The hypervisor cannot be attacked using SP3 on any ARM processors, even those that are listed as affected by SP3.) Furthermore, only 64-bit PV guests can exploit SP3 against Xen. PVH, HVM, and 32-bit PV guests cannot exploit SP3.

xen-uefi: Instructions and tools to boot Xen in UEFI mode with TPM measurements of Xen and dom0

Instructions and tools to boot Xen in UEFI mode with TPM measurements of Xen and dom0

This repository contains tools and instructions for installing Xen and dom0 with UEFI/SecureBoot such that all critical components of Xen and the dom0 kernel get SecureBoot verified and measured into the TPM.

Includes an updated Shim.

OpenXT has a nice article on Xen, Linux, TPM, and TXT. It also mentions the OpenXT toolkit.

OpenXT is an open-source development toolkit for hardware-assisted security research and appliance integration. Released as Open-Source Software (OSS) in June 2014, OpenXT stands on the shoulders of Xen Project and OpenEmbedded. It is derived from XenClient XT, which was first released in May 2011. It includes hardened Xen VMs that can be configured as a user-facing virtualization appliance, for client devices with Linux and/or Windows guests. It has been used to develop managed software appliances to isolate demanding graphics workloads, untrusted workloads and multiple networks on a single laptop or desktop. OpenXT is optimized for x86 devices with Intel VT-d, TXT (Trusted Execution Technology) and a TPM. OpenXT is being developed to meet the varied needs of the security and virtualization communities, as a toolkit for the configurable disaggregation of operating systems and user workflows. Client appliances developed on OpenXT can contain a mixture of open-source and proprietary software, supporting a range of business models.[…]


Pandavirtualization: Exploiting the Xen hypervisor

Pandavirtualization: Exploiting the Xen hypervisor
Posted by Jann Horn, Project Zero

On 2017-03-14, I reported a bug to Xen’s security team that permits an attacker with control over the kernel of a paravirtualized x86-64 Xen guest to break out of the hypervisor and gain full control over the machine’s physical memory. The Xen Project publicly released an advisory and a patch for this issue 2017-04-04. To demonstrate the impact of the issue, I created an exploit that, when executed in one 64-bit PV guest with root privileges, will execute a shell command as root in all other 64-bit PV guests (including dom0) on the same physical machine.[…]



guestrace: VM guess whole-system-call tracer

guestrace: A whole-system system-call tracer for VM guests
Ryan Johnson began writing guestrace as a prototype for a research project. Since then, we have packaged guestrace as a stand-alone utility. A properly-configured guestrace will print as they occur the system calls which processes invoke within a guest host. The guestrace utility relies on libvmi to perform virtual-machine introspection. Guestrace also provides a library, libguestrace, which gives programmers access to the guestrace engine. This is useful for programs which must trace system calls and do more than merely print them. […]

new CHIPSEC test for Xen XSA-188

Proof-of-concept module for Xen XSA-188 (
CVE-2016-7154: “use after free in FIFO event channel code”
Discovered by Mikhail Gorobets
This module triggers host crash on vulnerable Xen 4.4
“ -m tools.vmm.xen.xsa188“


“Xenpwn is a toolkit for memory access tracing using hardware assisted virtualization. It runs as a normal user space application inside the management domain (dom0) of a Xen hypervisor and can be used to trace any memory accesses performed by another VM running on the same hypervisor. The toolkit uses libvmi for interaction with the Xen hypervisor API and relies on simutrace for efficient storage of memory traces. Xenpwn was used to discover double fetch vulnerabilities in the inter domain communication of the Xen hypervisor resulting in XSA 155. Further research on identifying double fetches in other software is still ongoing.[…]”

CHIPSEC updates

The CHIPSEC team have tweeted about an upcoming 1.2.3 release with more Xen, Hyper-V, IOMMU, EPT support.

Also, Yuriy Bulygin of the Intel CHIPSEC team has posted some videos of their REcon training showing CHIPSEC usage:
It looks like their last checkin to the public git repo was in April:

Description of Xen exploit XSA-105

Jérémie Boutoille has a new blog post with information on Xen, with a video at the beginning for those who are too busy to read the entire article:

Xen exploitation part 1: XSA-105, from nobody to root

This blog post describes the exploitation of Xen Security Advisory 105 (XSA-105)  (CVE-2014-7155). This post explains the environment setup and shows the development of a fully working exploit on Linux 4.4.5. We are not aware of any public exploit for this vulnerability, although Andrei Lutas wrote excellent articles  describing the root cause of the vulnerability and how to trigger it. This post explains the environment setup and shows the development of a fully working exploit on Linux 4.4.5 (it probably works with many others versions). […]

Xen patches for AMD FIP/FCS/FDP/FDS/FOP instruction leak

The Xen security team posted an updated security advisory with a workaround to an AMD CPU issue. Heavily-edited announcement follows, see the OSS-security list post for the full announcement and patches:

Xen Security Advisory CVE-2016-3158,CVE-2016-3159 / XSA-172
version 3
broken AMD FPU FIP/FDP/FOP leak workaround

There is a workaround in Xen to deal with the fact that AMD CPUs don’t load the x86 registers FIP (and possibly FCS), FDP (and possibly FDS), and FOP from memory (via XRSTOR or FXRSTOR) when there is no pending unmasked exception.  (See XSA-52.) However, this workaround does not cover all possible input cases. This is because writes to the hardware FSW.ES bit, which the current workaround is based on, are ignored; instead, the CPU calculates FSW.ES from the pending exception and exception mask bits.  Xen therefore needs to do the same. Note that part of said workaround was the subject of XSA-52. This can leak register contents from one guest to another.  The registers in question are the FPU instruction and data pointers and opcode.

A malicious domain is able to obtain address space usage and timing information, about another domain, at a fairly low rate. The leaked address information might be used to help defeat address space randomisation in order to enable another attack.  The leaked address and timing information forms a low-bandwidth covert channel which might be used to gain information about the operation of a target guest. The affected FPU facility would not normally be used by cryptographic operations, as it does not provide cryptographically-relevant SIMD functions. It appears to us very unlikely that the leak might directly compromise sensitive information such as cryptographic keys, although (without knowledge of the guest software) this cannot be ruled out.  (This is notwithstanding the contrary statement in `Impact’ in XSA-52.)

Xen versions 4.0 and onwards are vulnerable.  Any kind of guest can exploit the vulnerability. The vulnerability is exposed only on AMD x86 systems.  Intel and ARM systems do not expose this vulnerability. Both PV and HVM guests are affected. The vulnerability can be avoided if the guest kernel is controlled by the host rather than guest administrator, provided that further steps are taken to prevent the guest administrator from loading code into the kernel (e.g. by disabling loadable modules etc) or from using other mechanisms which allow them to run code at kernel privilege. On Xen versions 4.3 and earlier, turning off XSAVE support via the “no-xsave” hypervisor command line option will avoid the vulnerability. On Xen versions 4.4 and onwards there is no other known mitigation.

This issue was discovered by Jan Beulich from SUSE.


Critical bug in Xen hypervisor

Wow, Joanna of ITL says “IMHO this is the worst bug affecting Xen, ever.”

Excerpt from Qubes Security Bulletin #22:

Critical Xen bug in PV memory virtualization code (XSA 148)

The Xen Security Team has announced a critical security bug (XSA 148) in the hypervisor code handling memory virtualization for the PV VMs [1]:

| The code to validate level 2 page table entries is bypassed when
| certain conditions are satisfied.  This means that a PV guest can
| create writeable mappings using super page mappings.
| Such writeable mappings can violate Xen intended invariants for pages
| which Xen is supposed to keep read-only.

The above is a political way of stating the bug is a very critical one. Probably the worst we have seen affecting the Xen hypervisor, ever. Sadly.

Full advisory:

Rekall forensics of XEN

There’s a nice blog in the Rekall forensics blog about memory forensics of XEN systems:

If you’ve ever taken a memory image of a Linux virtual machine that’s running under XEN in paravirtualization mode and you’ve tried to analyze it you’ll have noticed most of your plugins don’t work (if any). The reason is that XEN’s page tables are funky. XEN uses a technique known as direct mapping which significantly differs from how memory management is done in many other virtualization solutions.

More Information:

Qubes 3.0-RC alpha of LiveUSB release

Joanna Rutkowska of Invisible Things Lab posted a message to the qubes-users mailing list today, announcing a new Live USB image format of Qubes OS.

“We have built and uploaded the first ever working Qubes Live USB image! 🙂 It’s based on the recently released 3.0-rc2 release. Now you should be able to run and try Qubes OS of any laptop without needing to install it anywhere!”

Note that it currently does not work with UEFI:

“We have faced several challenges when making this Live USB edition of Qubes OS, which traditional Linux distro don’t need to bother with:
1. We needed to ensure Xen is properly started when booting the stick. In fact we still don’t support UEFI boot for the sitck for this reason, even though the Fedora liveusb creator we used does support it. Only legacy boot for this version, sorry.
Current limitations
7. UEFI boot doesn’t work, and if you try booting it via UEFI Xen will not be started, rendering the whole experiment unusable.”

Read the full announcement here:!topic/qubes-users/IQdCEpkooto


Today on the UEFI development list, Laszlo Ersek of Redhat announced an OVMF BOF at the upcoming KVM Forum, including Paolo Bonzini speaking on adding SMM to OVMF for KVM and Tianocore. This is a massive checkin, and having SMM in OVMF makes it a lot easier to trace and fuzz. Event aside, the list of the last year’s worth of changes to  OVMF/AVMF makes for an interesting read. Heavily-edited announcement follows:

Let’s do an OVMF BoF at this year’s KVM Forum too. Paolo will present “Securing secure boot: system management mode in KVM and Tiano Core” on Thursday, August 20, in the 5:00pm – 5:30pm time slot. Right after that, the BoF section starts at 5:30pm. We should convene and discuss stuff. I don’t have an agenda, so people should bring their ideas and questions (famous last words).

As food for thought, I tried to collect the feature-looking patches from the git history that have been committed since last year’s KVM Forum, and to match them against patch sets on the mailing list:

git log –reverse –oneline –since=2014-10-14 — OvmfPkg/ ArmVirtPkg/ ArmPlatformPkg/ArmVirtualizationPkg/

I attempted to sort them into categories. You can see the list below. The ordering is totally random, it’s just what I ended up with. Corrections / additions welcome. One (missing) feature I’d like to see discussed is: “SataControllerDxe in OVMF”. SMM will require Q35, and the only “IDE” that Q35 speaks is SATA / AHCI. (And you can’t disable that controller on Q35.)

Features completed (… unless marked [pending])

– Xen guest:

  – PV block driver:
    [PATCH v4 00/19] Introducing Xen PV block driver to OVMF

  – Xen for ARM:
    [PATCH v5 00/29] Xen/ARM guest support

– PCI / hw related:

  – PCI on ARM; detect VGA and USB keyboard:
    [PATCH v3 00/28] ArmVirtualizationPkg/ArmVirtualizationQemu: enable PCI
    [PATCH 0/4] ArmVirtualizationPkg: PlatformIntelBdsLib: dynamic console setup

  – support for Q35:
    [PATCH v6 0/9] OVMF: Add support for Qemu Q35 machine type
    [PATCH 1/1] OvmfPkg: QemuBootOrderLib: parse OFW device path nodes of PCI bridges

  – USB3 (ARM and x86):
    [PATCH v2 2/4] ArmVirtualizationPkg/ArmVirtualizationQemu: include XHCI driver
    [PATCH v2 4/4] OvmfPkg: include XHCI driver

  – support TCO watchdog emulation features:
    [PATCH v5 2/2] OvmfPkg/PlatformPei: Initialise RCBA (B0:D31:F0 0xf0) register

  – virtio-vga:
    [PATCH] Add virtio-vga support

  – support extra PCI root buses for NUMA-locality with assigned devices:
    [PATCH v3 00/23] OvmfPkg: support extra PCI root buses

– QEMU config integration:

  – fw_cfg, boot order, and -kernel booting on ARM:
    [PATCH v4 00/13] ArmVirtualizationQemu: support fw_cfg, bootorder, ‘-kernel’
    [PATCH 0/3] ArmVirtPkg: drop support for the ARM BDS

  – support for “-boot menu=on[,splash-time=N]”:
    [PATCH v2 0/3] OVMF, ArmVirt: consume QEMU’s “-boot menu=on[,splash-time=N]”

  – ACPI tables for ARM:
    [PATCH v2 0/3] ACPI over fw_cfg for ARM/AARCH64 qemu guests

  – SMBIOS features: Type 0 default, and SMBIOS 3.0 support on ARM and x86:
    [PATCH] OvmfPkg/SMBIOS: Provide default Type 0 (BIOS Information) structure
    [PATCH v2 0/6] ArmVirtPkg/ArmVirtQemu: support SMBIOS
    [PATCH 0/9] OvmfPkg, ArmVirtPkg: SMBIOS 3.0, round 2

– ARM specific:

  – “fun” with the caches:
    [PATCH v4 0/5] ArmVirtualizationPkg: explicit cache maintenance

  – secure boot:
    [PATCH v3 0/3] ArmVirtualizationQemu: enable support for UEFI Secure Boot

  – performance optimization:
    [PATCH v2 0/6] ArmPkg/ArmVirtPkg: GIC revision detection

  – better handling for the typical Linux terminal (generic driver code, hooked up to ArmVirt):
    [PATCH V4 0/5] Add TtyTerm terminal type

– SMM for OVMF (in progress):
    [PATCH 00/11] Bits and pieces
    [PATCH 00/58] OvmfPkg: support SMM for better security (single VCPU, IA32) [pending]

– Build system:

  – moving to NASM:
    [PATCH 0/7] Convert OVMF assembly to NASM
    [PATCH v2 0/6] OvmfPkg/XenBusDxe: Convert *.asm to NASM.

  – accept UTF-8 in .uni files:
    [PATCH v4 00/10] Support UTF-8 in .uni string files

  – LLVM/clang support for AARCH64 (in progress):
    [PATCH v4 00/13] BaseTools: unify all GCC linker scripts
    [PATCH v4 0/7] small model and clang support for AARCH64 [pending]

– UEFI compliance:

  – support for OsIndications:
    [PATCH v2 0/9] OvmfPkg: PlatformBdsLib cleanups and improvements

  – signal ReadyToBoot:
    [PATCH 1/8] OvmfPkg/PlatformBdsLib: Signal ReadyToBoot before booting QEMU kernel

  – signal EndOfDxe:
    [PATCH v2] ArmVirtPkg: signal EndOxDxe event in PlatformBsdInit
    [PATCH v2 0/6] OvmfPkg: save S3 state at EndOfDxe

  – fix Serial IO Protocol issues flagged by SCT
    [PATCH V4 0/5] Some improvements on serial terminal

– other

  – big OVMF guests:
    [PATCH v2 0/4] OvmfPkg: enable >= 64 GB guests

  – IPv6 (conditionally enabled):
    [PATCH v2] OvmfPkg: enable the IPv6 support

  – many fixes for toolchain warnings and C language misuse

More Information:

Xen releases Libbdvmi, Bitdefender VM introspection library

Xen has created a new library, Libbdvmi, the Bitdefender VM introspection library, an x86-centric library, supported by Xen 4.6.

In our quest for zero-footprint guest introspection using Xen, we needed to be able to decide whether changes to the guest OS are benign or malicious, and, once decided, to control whether they are allowed to happen, or rejected. To that end, we needed a way to be notified of interesting Xen guest events, with a very low overhead, that we could use in dom0 (or a similarly privileged dedicated domain), and that would connect our introspection logic component to the guest. We’re very happy to announce that the library we’ve created to help us perform virtual machine introspection, libbdvmi, is now on GitHub, under the LGPLv3 license, here. The library is x86-specific, and while there’s some #ifdeferry suggesting that the earliest supported version is Xen 4.3, only Xen 4.6 will work out-of-the-box with it. It has been written from scratch, using libxenctrl and libxenstore directly. Libbdvmi aims to provide a very efficient way of working with Xen to access guest information in an OS-agnostic manner:

* it only connects an introspection logic component to the guest, leaving on-the-fly OS detection and decision-making to it;
* provides a xenstore-based way to know when a guest has been started or stopped;
* has as few external library dependencies as possible – to that end, where LibVMI has used Glib for caching, we’ve only used STL containers, and the only other dependencies are libxenctrl and libxenstore;
* allows mapping guest pages from userspace, with the caveat that this implies mapping a single page for writing, where LibVMI’s buffer-based writes would possibly touch several non-contiguous pages;
* it works as fast as possible – we get a lot of events, so any unnecessary userspace / hypervisor context switches may incur unacceptable penalties (this is why one of our first patches had vm_events carry interesting register values instead of querying the quest from userspace after receiving the event);
* last but not least, since the Xen vm_event code has been in quite a bit of flux, being able to immediately modify the library code to suit a new need, or to update the code for a new Xen version, has been a bonus to the project’s development pace.

CrowdStrike announces Venom vulnerability

As reported by Robert Hackett at Fortune, Crowdstrike has research on a new vulnerability that impacts virtualization. Venom stands for “virtualized environment neglected operations manipulation”. It impacts QEMU, Xen, KVM, and VirtualBox, among others.

(It must be a big deal, as it already has an icon. I think Heartbleed took longer for it’s icon.)

More information: