FLUSP logo FLUSP - FLOSS at USP

Use QEMU and libvirt to setup a Linux kernel test environment

Written on , last modified on

This tutorial describes how to setup a test environment for Linux kernel development using QEMU and libvirt. This tutorial was originally thought to be part of a set of tutorials tailored to aid newcomers to develop for the Linux kernel Industrial I/O subsystem.

Command Summary

If you did not read this tutorial yet, skip this section. This section was added as a summary for those that already went through this tutorial and just want to remember a specific command.


virt-filesystems --long -h --all -a <disk_image>

virt-ls -a <disk_image> -m <disk_partition> <directory>

virt-copy-out -a <disk_image> <disk_directory_or_file> <local_directory>

sudo systemctl start libvirtd
sudo virsh net-start default

sudo virsh list --all
sudo virsh start --console iio-arm64
sudo virsh net-dhcp-leases default
ssh -i ~/.ssh/rsa_iio_arm64_virt root@<vm-ip-address>

mount -t virtiofs mount_tag /mnt/mount/shared

Setup the test environment

Summary of the parts of this tutorial:

  1. Set up a Virtual Machine (VM)
  2. Set up ssh access to the VM
  3. Set up host <-> VM file sharing (optional)
  4. Get the list of VM modules (optional)

The setup for Linux kernel development requires a few things. Choose or create a directory for this workshop to keep things barely organized. Example:

mkdir $HOME/iio_workshop
export IIO_DIR="$HOME/iio_workshop"
cd $IIO_DIR

Choose carefully. The naming conventions defined here will be followed in further tutorials related to Linux kernel development. From now on, we will use IIO_DIR to refer to the directory where the content related to Linux IIO development will be.

1) Set up a virtual machine (VM)

1.1) Get a disk image for the virtual machine.

An operating system (OS) image will be needed to test the kernels we will produce. You may create a virtual disk and do a fresh installation or download a disk image from the Internet. Many Linux distributions provide minimal disk images for download, some of them are Debian, Ubuntu, Fedora, and Suse. Your author is familiar with debian distributions so the following steps will describe how to set up a test VM with a debian operating system. Help enhance this workshop by adapting the following setup instructions to your favorite distro.

Download a disk image.

mkdir $IIO_DIR/vm_dir/
export VM_DIR=$IIO_DIR/vm_dir
cd $VM_DIR
wget http://cdimage.debian.org/cdimage/cloud/bookworm/daily/20230305-1310/debian-12-nocloud-arm64-daily-20230305-1310.qcow2
mv debian-12-nocloud-arm64-daily-20230305-1310.qcow2 base_iio_arm64.qcow2 # for legibility

The downloaded VM disk should be 2GB size. It is large enough to do a modules_install, but not enough to have more than one extra set of kernel modules. To avoid lack of disk space when playing with Linux modules, let’s resize the disk image to make it a little bigger.

qemu-img create -f qcow2 -o preallocation=metadata iio_arm64.qcow2 4G
virt-resize --expand /dev/sda1 base_iio_arm64.qcow2 iio_arm64.qcow2

1.2) Extract the kernel and initrd image.

We are interested in testing custom Linux kernels and we will do it by emulating machines with QEMU. Our emulation software of choice allows us to specify the kernel image through the -kernel flag. We also have to pass an initial ram disk with the modules needed to load the kernel. For now, we will use the kernel image and initrd that come with the VM we got. Later, we will replace the distro kernel by the one we will build.

See the available filesystem in disk.

virt-filesystems --long -h --all -a iio_arm64.qcow2

The boot filesystem should be /dev/sda1, rootfs should be /dev/sda2. See which kernel and initrd image(s) are inside the boot directory.

virt-ls -a iio_arm64.qcow2 -m /dev/sda2 /boot/

Copy out the kernel and initrd image. To keep things organized, let’s create a directory for our boot artifacts.

mkdir $VM_DIR/iio_arm64_boot
virt-copy-out -a iio_arm64.qcow2 /boot/initrd.img-6.1.0-5-arm64 iio_arm64_boot/
virt-copy-out -a iio_arm64.qcow2 /boot/vmlinuz-6.1.0-5-arm64 iio_arm64_boot/

Test your initial setup. Create a script called launch_vm_iio_workshop.sh and fill it with the following.

#!/bin/bash

IIO_DIR=$HOME/iio_workshop
VM_DIR=$IIO_DIR/vm_dir/
BOOT_DIR=$VM_DIR/iio_arm64_boot/

qemu-system-aarch64 \
  -M virt,gic-version=3 \
  -m 2G -cpu cortex-a57 \
  -smp 2 \
  -netdev user,id=net0 -device virtio-net-device,netdev=net0 \
  -initrd $BOOT_DIR/initrd.img-6.1.0-5-arm64 \
  -kernel $BOOT_DIR/vmlinuz-6.1.0-5-arm64 \
  -append "console=ttyAMA0 loglevel=8 root=/dev/vda2 rootwait" \
  -device virtio-blk-pci,drive=hd \
  -drive if=none,file=$VM_DIR/iio_arm64.qcow2,format=qcow2,id=hd \
  -nographic

Run the launch_vm_iio_workshop.sh launch script and see the VM boots. The default login user is root with no password (empty password). See the Troubleshooting section for help if the VM doesn’t boot.

1.3) Setup libvirt

libvirt is a virtual machine manager we will use to enable remote connection to our VM and file sharing. Install dependencies for the libvirt setup.

sudo apt-get install qemu libvirt-clients libvirt-daemon-system \
  bridge-utils virtinst libvirt-daemon guestfs-tools libguestfs-tools
sudo apt install qemu-system-arm
sudo apt install virt-manager # only if you want GUI for managing VMs

Start the libvirt service:

sudo systemctl start libvirtd
systemctl status libvirtd

The libvirt daemon is necessary for the virsh commands we will run next. Thus remind to start libvirtd every time you reboot your machine or enable the service (sudo systemctl enable libvirtd) to start automatically at system init.

Some installation may or may not allow nonroot users to manage virsh by default. If you happen to be using such a system, know that it is possible to enable nonroot users to manage virsh by setting libvirt default URI (export LIBVIRT_DEFAULT_URI=qemu:///system), by adding --connect qemu:///system to each virsh command. or by prepending sudo to run virsh commands with root privileges. If you can run virsh without sudo, do it for all virsh commands. For this tutorial, we will show virsh commands prepended with sudo since it’s not hard to forget the URI export and have some operations ran with and without the right permissions. You may end up seeing a set of VMs with virsh list --all and another set by running sudo virsh list --all.

Start an internal network managed by libvirt. Our VM will be connected to this virtual network.

sudo virsh net-start default

Similarly to the libvirt, the virsh default network will not start automatically after reboot unless you configure it with sudo virsh net-autostart default.

It’s easy to mess up with long commands so I suggest using a script for creating the virtual machine. I will call it create_vm_virsh_iio_workshop.sh. Fill it with the following.

#!/bin/bash

if [ "$EUID" -ne 0 ]
    MESSAGE="Running with sudo (or as root). "
    MESSAGE+="Don't forget to run following virsh commands with sudo too."
    echo "${MESSAGE}"
fi

## Part 1 version - distribution kernel - adapted to run with sudo/root
IIO_DIR=<full_path_to_your_iio_workshop_directory>
VM_DIR=$IIO_DIR/vm_dir/
BOOT_DIR=$VM_DIR/iio_arm64_boot/

virt-install \
    --name "iio-arm64" \
    --memory 1024 \
    --arch aarch64 --machine virt \
    --osinfo detect=on,require=off \
    --import \
    --features acpi=off \
    --disk path=$VM_DIR/iio_arm64.qcow2 \
    --boot kernel=$BOOT_DIR/vmlinuz-6.1.0-5-arm64,initrd=$BOOT_DIR/initrd.img-6.1.0-5-arm64,kernel_args="console=ttyAMA0 loglevel=8 root=/dev/vda2 rootwait" \
    --network bridge:virbr0 \
    --graphics none

We avoid using user dependent variables or expressions such as $HOME or ~/ because they will evaluate to different paths when running the script as root.

Run the VM create script.

sudo ./create_vm_virsh_iio_workshop.sh

You shall see the virtual machine start booting after running the creation script.

Useful libvirt/virsh commands:

2) Set up ssh access to the VM

Edit the virtual machine /etc/ssh/sshd_config file to allow root login. Set yes to the line that defines root login policy. If there is no line seting the policy, add one.

PermitRootLogin yes

Inside of the VM, configure the ssh daemon host keys and start the ssh server.

# Run these inside the virtual machine
dpkg-reconfigure openssh-server
systemctl enable sshd
systemctl restart sshd

Alike the libvirt service ran in the host machine, the ssh daemon service won’t start automatically at every VM boot if one does not run systemctl enable sshd.

Poweroff the VM with either the poweroff command from inside the VM or with sudo virsh shutdown iio-arm64. If poweroff does not succeed, try sudo virsh destroy iio-arm64.

Generate a ssh-key and copy it to the VM.

ssh-keygen -f ~/.ssh/rsa_iio_arm64_virt -t rsa -N ""

sudo virt-sysprep -a iio_arm64.qcow2 --ssh-inject root:file:/home/$USER/.ssh/rsa_iio_arm64_virt.pub

Start the VM again and find its IP address.

sudo virsh start --console iio-arm64
sudo virsh net-list
sudo virsh net-dhcp-leases default

Certify the SSH daemon is running inside the virtual machine.

systemctl status sshd

If sshd status is not active see the Troubleshooting section for tips in bringing sshd up.

And then we ssh with the IP shown by the net-dhcp-leases command.

ssh -i ~/.ssh/rsa_iio_arm64_virt root@<vms-ip-address-here>

Example:

ssh -i ~/.ssh/rsa_iio_arm64_virt root@192.168.122.55

See this cyberciti.biz post for a reference ssh setup example.

3) Set up host<->VM file sharing (optional)

To create a libvirt shared directory, we will edit the VM description.

sudo EDITOR=vim; sudo virsh edit iio-arm64

Add the memoryBacking node within the domain node and add the filesystem node within the devices node.

<domain>
  ...
  <memoryBacking>
    <source type='memfd'/>
    <access mode='shared'/>
  </memoryBacking>
  ...
  <devices>
    ...
    <filesystem type='mount' accessmode='passthrough'>
      <driver type='virtiofs' />
      <source dir='/path'/>
      <target dir='mount_tag'/>
    </filesystem>
    ...
  </devices>
</domain>

The /path in stands for the location (path) in the host machine that will be shared with the guest (virtual) machine. For instance, one can do

cd $IIO_DIR
mkdir shared_arm64

then use that directory for the sharing on the host side .

Inside the VM, mount the shared directory.

# Run these inside the virtual machine
mount -t virtiofs mount_tag /mnt/mount/shared

Now, the contents shared between host and virtual machine will be available under $IIO_DIR/mountpoint_arm64/ on the host and under /mnt/mount/shared/ on the virtual machine side.

See the libvirt documentation for reference on sharing files with virtiofs.

4) Get the list of VM modules (optional)

Logged in the virtual machine, run lsmod to list the currently loaded kernel modules and save that list into a file.

ssh -i ~/.ssh/rsa_iio_arm64_virt root@192.168.122.55
# Run lsmod inside the virtual machine
lsmod > vm_mod_list

The list of loaded kernel modules will later help reduce Linux build time in part 2 of this workshop.

Complementary Commands

Domain information commands:

virsh desc iio-arm64 --edit
virsh backup-cumpxml --wrap iio-arm64
virsh dominfo iio-arm64
virsh dumpxml iio-arm64
virsh edit iio-arm64

Use undefine to start over the virsh VM creation:

virsh undefine iio-arm64

virt-install flag to pass qemu options is --qemu-commandline=. Example: --qemu-commandline="-display gtk,gl=on"

Start and console connect separate commands:

sudo virsh start iio-arm64
sudo virsh console iio-arm64

Troubleshooting

Virtual machines doesn’t boot

The VM might fail to boot if the boot and rootfs partitions are not correctly specified in QEMU or libvirt scripts.

Compare the output of

virt-filesystems --long -h --all -a base_iio_arm64.qcow2

with the output from

virt-filesystems --long -h --all -a iio_arm64.qcow2

In the original VM disk, the boot filesystem should be sda15 and the root filesystem should be sda1. virt-resize might change the filesystem naming to sda1 for boot and sda2 for rootfs or maybe to something else. Check the output of virt-filesystems to figure out the correct partition names.

Unable to connect through ssh

Beware the messages highligted by journalctl -xeu ssh.service may be misleading when troubleshooting sshd startup. A common error/warning message seen in journalctl logs is ssh.service: Start request repeated too quickly.

If you go for that message, you may find a popular Stack Overflow thread suggesting to edit the virtual machine /etc/systemd/system/multi-user.target.wants/ssh.service file to set a higher start interval limit to the server.

However, that doesn’t solve the particular quick service restart problem we often encounter with this setup. By reading through earlier journal logs, if you find the message sshd: no hostkeys available – exiting try (re)generate OpenSSH host keys either with ssh-keygen -A or with dpkg-reconfigure openssh-server. See How To: Ubuntu / Debian Linux Regenerate OpenSSH Host Keys for additional explanation.

Cannot share directories between host and VM

The shared directory implementation for this setup is provided by virtiofs which requires Linux 5.4 or later with CONFIG_VIRTIO_FS and other configs enabled. You may see an error stating Unable to find a satisfying virtiofsd if the shared directory setup fail. See Standalone virtiofs usage for details on proper virtiofs setup.

Conclusion

This post described how to set up a virtual machine with both QEMU and libvirt. The virtual machine will be useful for testing changes to the Linux kernel. By this point, you should be able to start the virtual machine either with QEMU or libvirt. You should also enjoy ssh connection whenever starting the VM through libvirt.

History

  1. V1: Release
comments powered by Disqus