This tutorial describes setting up a Linux kernel development test environment using QEMU and libvirt to safely and efficiently compile and install custom kernels from source code. It is the first of a series of 4 tutorials introducing newcomers to Linux kernel development using the Industrial I/O (IIO) subsystem as a practical workspace for hands-on learning.

Command Summary

Skip this section if you have not completed this tutorial yet. This is a reference for those who have gone through this tutorial and want to remember specific commands.

### CREATE TEST ENV DIR AND SET PERMISSIONS
sudo mkdir /home/lk_dev # creates directory `lk_dev` inside `/home`
sudo chown -R libvirt-qemu:libvirt-qemu /home/lk_dev # change ownership of `/home/lk_dev` to `libvirt-qemu` user and group
sudo chmod -R 2770 /home/lk_dev # modify permissions of `/home/lk_dev` and make new sub-files/dirs inherit group 
sudo usermod -aG libvirt-qemu "$USER" # add your user to the `libvirt-qemu` group

### MANIPULATE DISK IMAGES
qemu-img info "${VM_DIR}/base_arm64_img.qcow2" # shows infos about the disk image
virt-filesystems --long --human-readable --all --add "${VM_DIR}/base_arm64_img.qcow2" # shows info about partitions and filesystems
qemu-img create -f qcow2 -o preallocation=metadata "${VM_DIR}/arm64_img.qcow2" 5G # creates new QCOW2 image with 5GB
virt-resize --expand /dev/<rootfs> "${VM_DIR}/base_arm64_img.qcow2" "${VM_DIR}/arm64_img.qcow2" # makes a copy of the image expanding the `rootfs`
virt-ls --add "${VM_DIR}/arm64_img.qcow2" --mount /dev/<rootfs> /boot # lists the files in `/boot` of partition `rootfs`
virt-copy-out --add "${VM_DIR}/arm64_img.qcow2" /boot/<kernel> "$BOOT_DIR"
virt-copy-out --add "${VM_DIR}/arm64_img.qcow2" /boot/<initrd> "$BOOT_DIR"

### MANAGE VMS
sudo systemctl start libvirtd # starts the libvirt daemon
sudo virsh net-start default # starts `virsh` default network
sudo virsh list --all # list all registered VMs
sudo virsh start --console arm64 # start a registered instance attaching console
sudo virsh console arm64 # attach console to running instance
sudo virsh shutdown arm64 # gracefully shutdown instance
sudo virsh net-dhcp-leases default # list `default` network
ssh root@<VM-IP-address> # SSH into VM of given IP

Introduction

This section is vital to understanding the motivation behind why we need a testing environment in the first place and how this environment will be, so don’t skip it!

When developing any software, it is essential to have a testing environment where you can execute the program you are developing to test the changes you make in the codebase. The testing environment should be safe and efficient. Safe means isolated from the development system - the system you use to develop the software - to avoid (1) adding setup biases (the “works on my machine” syndrome) and (2) messing up the development system (like creating files inside the home directory). Efficient means the testing environment should be quick and straightforward to set up (start) and teardown (stop), and provide fast feedback on changes, as there will be many cycles of making and then testing modifications. In Linux kernel development, this is truer than ever.

Most computers (certainly the ones you use to develop software) have an operating system (OS), and the kernel is its core component. So, when modifying the Linux kernel, you can use the OS the development system is running (if it is compatible with the Linux kernel) or use an OS in another system other than the development one. Using the OS of the development system isn’t safe or efficient, as a minor mistake can break the system, and you would need to reboot the system every time you want to test your change (there are workarounds, but in those cases, you are actually making robust testing environments). On the other hand, buying and configuring a new physical machine seems overkill and expensive, even more so when you are just curious about Linux development. This is where virtualization comes to save us!

Virtual machines (VMs) will be the testing environments for this series of tutorials. Although some may be tempted to use containers, it wouldn’t work, as it would essentially be the same as using the development system’s OS, as containers share the underlying kernel.

We will use QEMU to run VMs, in other words, to provide a virtual model of an entire machine (CPU, memory, and emulated devices) that runs on top of the actual machine’s OS, like an application. libvirt simplifies management of VMs and allows automation. Together, they offer a safe and efficient way to test kernel modifications without risking the stability of the development system.

Installing Tutorial Dependencies

Correctly installing the dependencies is vital to avoid problems during this tutorial, so install the following packages according to your Linux distribution. Note that these packages are “bloated”, in the sense that some of the commands we will use could be installed with a number of specific packages. Also, even if your distro fits the instructions below, the packages could have different names and/or be arranged differently.

# Arch-based distros
sudo pacman -Syy && sudo pacman -S qemu-full libvirt virt-install guestfs-tools wget
# Debian-based distros
sudo apt update && sudo apt install qemu-system libvirt-daemon-system virtinst libguestfs-tools wget
# Fedora-based distros
sudo dnf update && sudo dnf install qemu libvirt-daemon virt-install guestfs-tools wget

Setting Up the Testing Environment

In this series, host refers to the physical machine, while guest refers to the VM.

Summary of this tutorial:

1. Preparing testing environment directory and “all-in-one” script
2. Set up and configure a VM running a guest OS
3. Configure SSH access from the host to the VM
4. Fetch the list of modules loaded in the guest kernel
5. Set up host <-> VM file sharing (optional)

My journey on the Google Summer of Code project passed by so fast… This is my last week on the GSoC and those 14 weeks flew by! A lot of stuff happened during those three months, and as I’m writing this blog post, I feel quite nostalgic about this three months.