This tutorial describes how to build - in other words, compile - the Linux kernel for the ARM architecture and boot test it in a Virtual Machine (VM) using kw to streamline and automate some processes we will be doing. Basic kernel build configuration is covered too.

This tutorial was originally thought to be part of a set of tutorials tailored to aid newcomers in developing the Linux kernel Industrial I/O subsystem (IIO). It is the second in the series and a continuation of the “Setting up a test environment for Linux Kernel Dev using QEMU and libvirt” tutorial. This post is an adaptation of “Building and booting a custom Linux kernel for ARM” to use kw wherever is convenient.

Command Summary

Skip this section if you have not completed this tutorial yet. This is a reference for those who have gone through this tutorial and want to remember specific commands.

### INSTALL `kw`
git clone https://github.com/kworkflow/kworkflow.git "$KW_DIR"
cd "$KW_DIR"
git switch unstable # changes to the branch `unstable`
./setup.sh --full-installation # install `kw` w/ all dependencies for building Linux kernel

### SHALLOWLY CLONING THE IIO TREE W/ ACTIVE DEVELOPMENT BRANCH
git clone git://git.kernel.org/pub/scm/linux/kernel/git/jic23/iio.git "${IIO_TREE}" --branch testing --single-branch --depth 10

### CONFIGURING `kw`
cd "$IIO_TREE" # go to the IIO tree directory
kw init # initialize local configurations for the IIO tree
kw remote --add arm64 root@<VM-IP-address> --set-default # adds a remote named `arm64` to local context and set it to default
kw config build.arch 'arm64' # defines the target architecture to ARM64
kw config build.cross_compile 'aarch64-linux-gnu-' # defines the cross-compiler
kw config build.kernel_img_name 'Image.gz' # defines the kernel image format
kw config --show build # check `kw build` configurations

### CREATING A LEAN `.config` file
cd "$IIO_TREE" # go to the IIO tree directory
make ARCH=arm64 defconfig # create predefined defconfig
make ARCH=arm64 olddefconfig # update defconfig with new default values
kw ssh --get '~/vm_mod_list' # copy VM modules list to IIO tree
make ARCH=arm64 LSMOD=vm_mod_list localmodconfig
kw kernel-config-manager --save arm64-optimized # saves `.config` w/ name `arm64-optimized`
kw kernel-config-manager --list # lists all managed `.config`s
kw kernel-config-manager --get arm64-optimized # gets managed `.config` w/ name `arm64-optimized`

### BUILDING A LINUX KERNEL FROM SOURCE CODE
cd "$IIO_TREE" # go to the IIO tree directory
kw build --menu # safely modify `.config` w/ a TUI
kw build # compile Linux kernel from source considering local configurations

### INSTALLING LINUX KERNEL MODULES
cd "$IIO_TREE" # go to the IIO tree directory
kw deploy --modules # only install modules into the VM

Introduction

After setting up a test environment with VMs using QEMU and libvirt, we begin compiling (also called building) a custom Linux kernel from source and booting it. This involves taking the source code developed by the Linux community, compiling it into a monolithic executable that can be loaded into memory and run on hardware (virtualized or not), and using it to start a system. Additionally, we will compile and install kernel modules, smaller units that can be dynamically loaded and unloaded while the system is running. Cross-compilation will also be a factor, as we assume that the machine used to compile the source code to these artifacts has an AMD64 architecture, while the target architecture is ARM64.

kworkflow, or just kw, is a Free/Libre and Open Source Software (FLOSS) Developer Automation Workflow System (DAWS), which has the mission of reducing the environment and setup overhead of developing for Linux. Similar to how libvirt facilitated our management of VMs - albeit after some overhead setting its permissions to work - the kw project helps kernel developers in their day-to-day tasks, like streamlining and automating building and deploying custom Linux kernels from source. In this tutorial, kw will help us fetch an optimized .config for our VM, build the custom kernel image and modules, and install the modules inside the VM. It will also ease our life in terms of installing necessary dependencies to compile a Linux kernel.

Configuring, building, and booting a custom Linux kernel

In this section we will go through the steps on how to configure, compile, and boot customized Linux images in ARM64 VMs using kw.

Summary of this tutorial:

1. Installing kw
2. Cloning a Linux kernel tree
3. Configure kw in a local context for IIO development
4. Configuring the Linux kernel compilation
5. Building a custom Linux kernel
6. Installing modules and booting the custom-built Linux kernel

Before proceding remember to have the activate.sh script we’ve created available to be modified and active for all the commands run in the host system. Also assure that you have your testing environment set up from the last tutorial.

1) Installing kw

The easiest way to install kw is to clone it using git and run the setup.sh script. First, let’s add an env var for the path of kw repository in the activate.sh script.

--- a/activate.sh
+++ b/activate.sh
@@ -4,6 +4,7 @@
 export LK_DEV_DIR='/home/lk_dev' # path to testing environment directory
 export VM_DIR="${LK_DEV_DIR}/vm" # path to VM directory
 export BOOT_DIR="${VM_DIR}/arm64_boot" # path to boot artifacts
+export KW_DIR="${LK_DEV_DIR}/kw" # path to `kw` repository

 # utility functions

Rerun the activate.sh after that and clone the kw repository

git clone https://github.com/kworkflow/kworkflow.git "$KW_DIR"

Then, go to the repository’s root and change it to the unstable branch. The default master is OK for this tutorial, but the latest kw development is pretty stable, although the name of the branch suggests otherwise.

cd "$KW_DIR"
git switch unstable # changes to the branch `unstable`

Finally, do the full installation of kw to include dependencies for building the kernel. If you are using a Arch, Debian, or Fedora-based distro, it should detect it and prompt you to enter the superuser credentials to start installing all the packages necessary.

./setup.sh --full-installation # install `kw` w/ all dependencies for building Linux kernel

From now on, you can update kw with itself! Running kw self-update updates kw with the latest development on the master branch, while kw self-update --unstable does the same for the unstable branch.

You may need to exit and rerun the activate.sh script for kw to show on your PATH. To ensure your kw installation was successful, run the following to see information about the release.

kw --version # print information about the `kw` release installed

2) Cloning a Linux kernel tree

There are several repositories that contain the source code for the Linux kernel. These repositories are known as Linux kernel trees, in short, kernel trees or just trees, because, as every software project, its source code is arranged in a tree-like file hierarchy. Some trees are widely known such as Linus Torvalds’ tree (known as mainline) that represent the most updated official version of Linux and the Linux stable tree that represents the stable releases (the 6.10, 6.11, 6.12, etc.). In general, a Linux tree is a repository where some development for the kernel happens and each tree follows its own development ruled by its own community. Many of those repos are at kernel.org.

Some examples of Linux kernel trees are:

• Linus Torvalds’ tree (mainline)
• Linux-stable tree
• Linux-next tree
• IIO subsystem tree
• Raspberry Pi tree
• Analog Devices tree

For this workshop, we’ll be using the Industrial I/O (IIO) subsystem tree so download (clone) it with git. First, modify your activate.sh script

--- a/activate.sh
+++ b/activate.sh
@@ -4,6 +4,7 @@
 export LK_DEV_DIR='/home/lk_dev' # path to testing environment directory
 export VM_DIR="${LK_DEV_DIR}/vm" # path to VM directory
 export BOOT_DIR="${VM_DIR}/arm64_boot" # path to boot artifacts
 export KW_DIR="${LK_DEV_DIR}/kw" # path to `kw` repository
+export IIO_TREE="${LK_DEV_DIR}/iio" # path to IIO subsystem Linux kernel tree

 # utility functions

then rerun the activate.sh script and clone the tree.

git clone git://git.kernel.org/pub/scm/linux/kernel/git/jic23/iio.git "${IIO_TREE}" --branch testing --single-branch --depth 10

In terms of source code, Linux kernel trees are already huge (a couple of GB in disk space), but the git history is even bigger at this point (around 5GB). To (hopefully) not use too much disk space and avoid longer download times, we use the --branch option, which tells git we want the testing branch of the remote pointed by the URL, and the --depth=10 option, which limits the git history to the latest 10 commits in the branch to be fetched. We’ve chosen the testing branch instead of the default master because the former is where the actual development happens. Hence, as in this series, we aim to (possibly) develop and propose changes to IIO, the testing branch is the one we are interested in.

If you have plenty of disk space (and probably some spare time), I suggest you clone the tree without the --branch and --depth flags because commit logs are often a good source of information when trying to understand kernel code. By the time this post was written, the IIO tree (with its entire git history) was sizing roughly 5GB.

3) Configure kw in a local context for IIO development

kw can create specific isolated configurations for each Linux kernel tree in the user’s filesystem. It can actually create multiple environments for a single tree, but this is too advanced for this tutorial. For now, let’s move into the IIO tree and initialize local configurations for that tree.

cd "$IIO_TREE" # go to the IIO tree directory
kw init # initialize local configurations for the IIO tree

For all kw commands run inside this tree, local configurations will override global ones whilst keeping them encapsulated in this context. This is similar to how git config and git config --global work.

Before moving forward, let’s start the VM running ARM64, which we set up in the last tutorial. Note that if either libvirtd, the default network, or the VM are already started, some commands will raise errors, indicating that these are already active.

@host
sudo systemctl start libvirtd # starts the libvirt daemon
sudo virsh net-start default # starts `virsh` default network
sudo virsh start --console arm64 # start a registered instance attaching console
sudo virsh net-dhcp-leases default # list `default` network to get VM IP

Inside the IIO tree, add the VM remote configurations for kw to manage it seamlessly.

@host
cd "$IIO_TREE"
### ADAPT THE COMMAND BELOW ###
kw remote --add arm64 root@<VM-IP-address> --set-default # adds a remote named `arm64` to local context and set it to default

To test if the remote addition was correctly done, run the following to initiate an SSH connection inside the IIO tree.

@host
cd "$IIO_TREE"
kw ssh # initiate an SSH connection to the default remote

You can check the remotes available in the current local context with kw remote --list.

4) Configuring the Linux kernel compilation

The Kernel Build System (kbuild) is based on make and other GNU tools and allows a highly modular and customizable build process for the Linux kernel. By default, kbuild uses the configuration options stored in the .config file under the root directory of the Linux kernel tree, which shouldn’t be present in a freshly cloned tree. Those options hold values for configuration symbols associated with kernel resources such as drivers, tools, and features in general. Nearly all directories inside a kernel tree have a Kconfig file, which defines the symbols for the resources at the same level. Top Kconfig files include (source) Kconfig files from subdirectories thus creating a tree of configuration symbols. When needed, kbuild generates configuration options from Kconfig symbols and stores their values in a .config file. kbuild Makefiles then use these configuration values to compile code conditionally and to decide which objects to include in the monolithic kernel image and which objects to become modules [1] [2].

There are predefined .configs for building kernels for different architectures and purposes. These are called defconfig (default configuration) files. defconfig files store only specific non-default values for configuration symbols. For instance, one can find defconfig files for ARM architecture machines under arch/arm/configs. We will create a .config file from the arm64 defconfig.

@host
cd "$IIO_TREE"
make ARCH=arm64 defconfig # create predefined defconfig
make ARCH=arm64 olddefconfig # update defconfig with new default values

Fetch the saved list of modules you’ve generated in the first tutorial of the series and generate an optimized .config based on it. Refer to section 4 of the first tutorial, in case you don’t have this file.

@host
cd "$IIO_TREE"
kw ssh --get '~/vm_mod_list' # copy VM modules list to IIO tree
make ARCH=arm64 LSMOD=vm_mod_list localmodconfig

kw kernel-config-manager helps us manage multiple configs without having to manually move and rename files. Below are some commands to help you more robustly manage these .configs. Note that .configs are considered in a “global” context, so files managed by kw are accessible no matter the kernel tree you are.

@host
kw kernel-config-manager --save arm64-optimized # saves `.config` w/ name `arm64-optimized`
kw kernel-config-manager --list # lists all managed `.config`s
kw kernel-config-manager --get arm64-optimized # gets managed `.config` w/ name `arm64-optimized`

It is possible to open the .config file and directly edit it, but this isn’t recommended. A safer and more palatable way is to use one of the Terminal User Interfaces (TUI) provided by the Linux project

@host
cd "$IIO_TREE"
kw build --menu # open a TUI to safely edit the `.config`

Use the arrow keys to navigate and ENTER to change/toggle a configuration. You can take some time to familiarize with these menus and see the available configurations.

For this tutorial, we want to make a tiny customization: change the kernel image name. In the menu, select General setup, then in Local version - append to kernel release, put whatever you want to be appended after the kernel version, and disable Automatically append version information to the version string. Be aware that the string you input will be part of the name of your first custom kernel, so treat it well :)

To exit, hit F9, which will prompt you to save the changes. The figure below illustrates the steps.

You can check how many modules will be compiled along with other informations using kw.

@host
cd "$IIO_TREE"
kw build --info

5) Building a custom Linux kernel

Different processor architectures have distinct instruction sets and register names. Due to that, the binaries produced by a compiler for architecture A will not work on a machine of architecture B. So, we need to use a compiler that produces binaries compatible with the instruction set of the machine we want to run our kernel. Essentially, this concept is called Cross-Compilation.

In our case, we assume we have a development system of the AMD64 (x86_64) architecture, and the target machine is a VM of the ARM64 architecture.

Most distros should have a GCC package with a compiler for AMD64 host machines that produces binaries for ARM64 targets. On Debian and Fedora, the package name is gcc-aarch64-linux-gnu, while in Arch the package name is aarch64-linux-gnu-gcc.

# Arch-based distros
sudo pacman -Syy && sudo pacman -S aarch64-linux-gnu-gcc
# Debian-based distros
sudo apt update && sudo apt install gcc-aarch64-linux-gnu
# Fedora-based distros
sudo dnf update && sudo dnf install gcc-aarch64-linux-gnu

See the Complementary Commands section for advice if you are having problems in finding an equivalent package to your distro.

We just need to configure kw a little bit more to be good to go. As said, the IIO tree has local configurations, and we need to set some related to kw build, the feature that deals with everything related to compiling kernels. kw config --show build shows the status of configurations available.

@host
cd "$IIO_TREE"
kw config build.arch 'arm64' # defines the target architecture to ARM64
kw config build.cross_compile 'aarch64-linux-gnu-' # defines the cross-compiler
kw config build.kernel_img_name 'Image.gz' # defines the kernel image format
kw config --show build # check `kw build` configurations

Finally, let’s build our custom Linux kernel from source code. Mind that, although we made a really lean .config, compiling the kernel still is a heavy task, and even more so when cross-compiling.

@host
cd "$IIO_TREE"
kw build # compile Linux kernel from source considering local configurations

If everything goes right, you should see an arch/arm64/boot/Image, modules.order and alike files under the Linux source root directory.

6) Installing modules and booting the custom-built Linux kernel

Before booting the custom-built kernel, we need to install the modules into the VM, i.e., we need to move the module objects compiled to the right place inside the VM’s filesystem. To do this, ensure the VM is running, and the kw remote configuration is correct. Then run

@host
kw deploy --modules # only install modules into the VM

There may be some errors thrown about strip or make not being able to do some things, but they may not indicate a real error.

Finally, update your activate.sh script to boot the VM (both through pure QEMU and with libvirt) with the custom-built kernel. Don’t forget to adapt the modification to have the correct initrd and the correct vda partition depending on your VM particularities!

--- a/activate.sh
+++ b/activate.sh
@@ -23,7 +23,7 @@ function launch_vm_qemu() {
         -smp 2 \
         -netdev user,id=net0 -device virtio-net-device,netdev=net0 \
         -initrd "${BOOT_DIR}/<initrd>" \
-        -kernel "${BOOT_DIR}/<kernel>" \
+        -kernel "${IIO_TREE}/arch/arm64/boot/Image" \
         -append "loglevel=8 root=/dev/<vdaX> rootwait" \
         -device virtio-blk-pci,drive=hd \
         -drive if=none,file="${VM_DIR}/arm64_img.qcow2",format=qcow2,id=hd \
@@ -41,7 +41,7 @@ function create_vm_virsh() {
         --import \
         --features acpi=off \
         --disk path="${VM_DIR}/arm64_img.qcow2" \
-        --boot kernel=${BOOT_DIR}/<kernel>,initrd=${BOOT_DIR}/<initrd>,kernel_args="loglevel=8 root=/dev/<vdaX> rootwait" \
+        --boot kernel=${IIO_TREE}/arch/arm64/boot/Image,initrd=${BOOT_DIR}/<initrd>,kernel_args="loglevel=8 root=/dev/<vdaX> rootwait" \
         --network bridge:virbr0 \
         --graphics none
 }

Rerun the activate.sh script. Shutdown the VM and undefine it from libvirt managed VMs. Then, created it again

@host
sudo virsh shutdown arm64
sudo virsh undefine arm64
create_vm_virsh

Log into the VM and run

@VM
uname --kernel-release

to check that you are now running the custom kernel we just built. The output of this command should be something in the format of

<major-release-nr>.<minor-release-nr>.<patch-release-nr>-rc<one-through-eight>-<the-name-you-wrote>+

like, for example,

6.14.0-rc1-free-software+

Congratulations, you’ve just compiled and boot-tested a Linux kernel! From now on, whenever you make a change on the Linux kernel tree, you can “just” compile it, install the modules, and then shutdown, with sudo virsh shutdown arm64, and start the VM, with sudo virsh start arm64 (rebooting it doesn’t works). It will run the new custom kernel we will be capable to run validations inside this testing environment!

6.1) Installing the kernel image

Often, kernel developers also need to explicitly install the Linux kernel image to their target test machines. Notice that here, we are not copying or moving the Linux kernel image to nowhere inside the VM like we did when installing modules. Neither we are dealing with a physical machine.

Essentially, installing a new kernel image would be to just replace the vmlinuz/Image/zImage/bzImage/uImage file, which contains the Linux boot executable program. However, some platforms (such as AMD64 and ARM64) have fancy boot procedures with boot loaders that won’t find kernel images without very specific configuration pointing to them (e.g., GRUB), which might mount temporary file systems (initrd), load drivers prior to mounting the rootfs, and so on. To help setup those additional boot files and configuration, the Linux kernel has a install rule. So, kernel developers may also run make install or make install INSTALL_PATH=<path_to_bootfs> when deploying kernels to those platforms.

For this setup, we shall not bother with that. We don’t need to run the installation rule because we instructed QEMU (with -kernel) and libvirt (with --boot kernel=...) to pick up the kernel image in the tree that resulted from the compilation, and we are reusing the initrd file from the original guest OS that came with disk image.

This shows the plurality of the umbrella Linux project; Linux is composed of many subprojects (called subsystems), each one with their own development contexts, accomplishing similar tasks in the most varied ways, sometimes even duplicating (or wasting) efforts.

Complementary Commands

One may also download cross compiler toolchains from different vendors. For instance, ARM provides an equivalent cross compiler that you may download if having trouble finding a proper distro package.

wget -O "${LK_DEV_DIR}/gcc-aarch64-linux-gnu.tar.xz" https://developer.arm.com/-/media/Files/downloads/gnu-a/10.3-2021.07/binrel/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu.tar.xz
tar -xf -C "$LK_DEV_DIR" "${LK_DEV_DIR}/gcc-aarch64-linux-gnu.tar.xz"

Sometimes identifying the cross compiler for your combination of host and target machines may require some understanding of what is called the compiler triplet. Conceptually, the compiler triplet should contain three fields: the name of the CPU family/model, the vendor, and the operating system name [3]. However, sometimes the vendor is omitted so one may find a triplet like x86_64-freebsd (FreeBSD kernel for 64-bit x86 CPUs) [3]. It is also common to see the operating system information split into two separate fields, one for indicating the kernel and the other for describing the runtime environment or C library which is being used. The the debian package for x86-64 gcc is an example of this triplet format mutation: gcc-x86-64-linux-gnu (compiler for 64-bit x86 targets that will run a Linux kernel and have GNU glibc in their runtime). But things can get even more unintuitive when system call conventions or Application Binary Interfaces (ABI) are specified in the OS field as in arm-linux-gnueabi (compiler for 32-bit ARM targets that will run Linux using the EABI system call convention) or as in arm-none-eabi (compiler for 32-bit ARM that will run no OS (bare-metal) using the EABI system call convention).

Anyways, you may point to the generic cross compiler name when using compilers not under your PATH. For example:

export CROSS_COMPILE="${LK_DEV_DIR}/gcc-aarch64-linux-gnu/bin/aarch64-none-linux-gnu-"

Conclusion

This tutorial described how to configure and build a custom Linux kernel and boot-test it into a VM. To accomplish that, we covered basic concepts of Linux kernel build configuration to guide readers into generating feasible .config files, and some cross-compilation concepts. By this point, you should be able to configure, build, and boot-test a custom Linux kernel from source code in a safe, while not-so-much efficient (we will work on that), testing environment.

We also introduced kw, a feature-rich system with the mission of helping kernel developers do their day-to-day tasks more efficiently, error-free, and intuitively.

History

1. V1: Release

References

[1] Javier Martinez Canillas. “Kbuild: the Linux Kernel Build System”. (2012) URL: https://www.linuxjournal.com/content/kbuild-linux-kernel-build-system. ⤴

[2] Michael Elizabeth Chastain and Kai Germaschewski and Sam Ravnborg. “Linux Kernel Makefiles”. (2023) URL: https://www.kernel.org/doc/html/latest/kbuild/makefiles.html. ⤴

[3] . “Target Triplet”. (2019) URL: https://wiki.osdev.org/Target_Triplet. ⤴

This tutorial explains a few aspects of Linux character devices with a basic character device driver example. This can be a continuation for the “Introduction to kernel build configuration and modules” tutorial.