GPU Passthrough with QEMU on Arch Linux

QEMU Arch Linux logo

GPU passthrough allows native graphics performance in the virtual machine which opens up new possibilities. One major use case is virtualized gaming as well as hardware acceleration for certain applications like Adobe Creative Suite. Steam In-Home Streaming opens up some odd but functional configurations by leveraging the VM to stream to clients on a local network. A major advantage of running Steam in a VM is that it does not prevent the use of the host system while streaming. In practice this could mean that two different users could simultaneously make use of the same system.

 

Before proceeding there are a few things to check to confirm that at least essential hardware requirements are met to allow GPU passthrough. For any virtualization to be possible both the motherboard and the processor must support x86 virtualization. Different manufacturers have different terms for this technology. Intel systems must support VT-x while AMD system must support AMD-V. Ensure that AMD-VI/VT-D is enabled in the BIOS settings. Further more to make GPU passthrough possible the CPU must also support IOMMU which is referred to as VT-d for Intel platforms or AMD-Vi for AMD platforms. On the software side the only requirement is kernel 4.1 or newer. Older kernels can be used as well but do not have vfio-pci driver and would require the use of a more cumbersome pci-stub driver. There is little reason to use kernel older than 4.1 at this point.

 

Hardware used was Asus Z170 Deluxe motherboard with Intel Skylake 6700K CPU and 16GB DDR4 Corsair RAM. Motherboard did not require ACS patch for GPU passthrough as well as on of the USB controller and one of the ethernet ports but did require the patch for integrated audio passthrough. Overall Skylake platform is not recommended for Linux due to abysmal Linux driver support from Intel. This will likely cause massive issue when integrated Intel graphics is used for Linux host. Many recent motherboards should work without patches and most should work with a patch. Practically all AMD CPU’s and Intel Skylake CPU’s support virtualization. Older Intel CPU’s mostly only support VT-d on non-K models.

The system was successfully tested with Radeon HD7970 / R9 280X and Radeon RX 480 GPUs. Initially RX 480 caused the client system to freeze on VM shutdown and then the host system when it is shutdown as well. Shutting down the host system while the VM was not shut down caused no issues and neither did rebooting the VM. This seems to have been mostly rectified in a recent firmware update.

 

Check the current kernel version.

uname -r

 

Check if vfio-pci is available on the system.

modprobe vfio-pci

An absence of any output means vfio-pci is available while modprobe: FATAL: Module vfio-pci not found means it is not available.

 

Enable IOMMU

There are several boot loaders that can be used with Arch Linux. The newest and simplest boot loader is Systemd-boot and is used here for those reasons. Regardless of which boot loader is used the required configuration is very similar.

 

Open the configuration file.

sudo nano /boot/loader/entries/arch.conf

Different flavours of Arch Linux may have an alternatively named configuration file. For example, under Antergos Linux the path should be substituted with  /boot/loader/entries/antergos. Append iommu=on for AMD processor and intel_iommu=on for Intel processor to the end of the options line and save the file. Both parameters can be appended with no negative effects.

 

Reboot for the changes to take effect.

reboot

 

Check to confirm IOMMU has been enabled successfully.

sudo dmesg | grep -e DMAR -e IOMMU

Look for DMAR: IOMMU enabled line in the output to confirm IOMMU is supported and enabled.

 

Add Kernel Modules

Open the configuration file.

sudo nano /etc/mkinitcpio.conf

Prepend vfio, vfio_iommu_type1, vfio_pci and vfio_virqfd modules to the MODULES list. Existing listed modules like i915, radeon, amdgpu, nouveau or others should come last in the list.

 

Save the changes to the initial ramdisk environment.

sudo mkinitcpio -p linux

The -p parameter value should be the kernel currently in use. Typically this will be linux, linux-lts, linux-vfio or linux-vfio-lts.

 

Reboot for the changes to take effect.

sudo reboot

 

Isolate Devices

List all devices present in the system.

lspci -nnk

Find the devices that are to be passed through and take note of the device IDs at the end. Graphics devices will often have a separate audio device that must also be passed through for audio output and often for the graphics card to work at all.

 

Sample output for Radeon RX 480 Graphics card with the PCI device claimed by VFIO driver.

01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Polaris10] [1002:67df] (rev c7)
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:0b37]
	Kernel driver in use: vfio-pci
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aaf0]
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aaf0]
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

 

Open VFIO configuration file.

sudo nano /etc/modprobe.d/vfio.conf

 

Append vfio-pci command followed by the list of comma separated device IDs to the options line and save the file.

options vfio-pci ids=1002:67df,1002:aaf0

Devices other than a GPU such as network interfaces or USB controllers can also be bound. Since virt-manager passes devices on demand they do not need to be bound and will remain attached to the host unit the VM is started. When using QEMU directly devices need to be bound before hand and thus those devices will be unavailable to the host at any point.

As an alternative to binding the modules could also be blacklisted. This method however has distinct disadvantages. For one any device using a blacklisted module will no longer function. This is not an issue if for example there is only a single device that uses this module. Another downside of blacklisting is that when using qemu directly the devices will still need to be bound to vfio-pci and so require an additional scripting.

 

Reboot for the changes to take effect.

sudo reboot

 

Verify that the devices have been successfully assigned.

sudo dmesg | grep -i vfio

Device IDs should be listed in the output if it was successfully assigned.

 

IOMMU / PCI Groups

List all PCI devices and their groups they are in.

find /sys/kernel/iommu_groups/ -type l

A group is considered clean when only related devices are in the same PCI group.

 

List all devices in a PCI group of a device.

ls -l /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices/

Replace 0000:01:00.0 with the ID of the device to be passed through from the previous section. Most graphics cards will have two devices, the GPU itself and HDMI audio device. More than the expected devices in the group means the IOMMU grouping needs to be fixed.

 

An example of a dirty IOMMU group. The ACS patch will be needed to clean it up.

lrwxrwxrwx 1 root root 0 Apr 12 15:47 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0
lrwxrwxrwx 1 root root 0 Apr 12 15:47 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
lrwxrwxrwx 1 root root 0 Apr 12 15:47 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1

 

To bypass IOMMU grouping a patch needs to be applied. The simplest method is to install VFIO Linux Kernel with the patch already applied and enable it in the boot manager.

 

Enable ACS Override Patch with Systemd-boot

Open systemd-boot boot manager configuration file.

sudo nano /boot/loader/entries/arch.conf

Enable the ACS Override patch by appending pcie_acs_override=downstream to the command line options line.

 

Restart for the changes to take effect.

sudo reboot

 

Install OVMF

Install Packer if needed or use another package wrapper like Yaourt.

 

Install OVMF binaries with packer.

packer -S ovmf-git

 

Copy the OVMF variable file to the desired location.

cp /usr/share/ovmf/x64/ovmf_vars_x64.bin /var/lib/libvirt/images/ovmf_vars_x64.bin

Default libvirt image directory is used here but any directory with appropriate permissions will work.

 

Install QEMU

Install QEMU with pacman.

sudo pacman -S qemu

 

Install optional QEMU dependencies.

sudo pacman -S dmidecode dnsmasq firewalld

These dependencies are more to prevent errors from being logged than anything else.

 

Open QEMU configuration file.

sudo nano /etc/libvirt/qemu.conf

 

Modify nvram parameters to match the location of OVMF files and save the file.

nvram = [
     "/usr/share/ovmf/x64/ovmf_x64.bin:/var/lib/libvirt/images/ovmf_vars_x64.bin"
]

 

Run the VM

There are two method of running Virtual Machine with GPU Passthrough using QEMU. The first method is to to run entirely from command line which allows a great level of control and makes troubleshooting easier. The second method involves the use of a Graphical User Interface namely virt-manager. A desktop environment needs to be installed  when GUI approach is used but on the positive side it is significantly more convenient and easier to manage. Certain advanced or uncommon setups may still require manual configuration from the terminal.

 

Setup QEMU Virtual Machine with virt-manager on Arch Linux (unpublished)

dom