This tutorial shows how to make a simple Linux kernel module and how to create build configurations for new kernel features.

This tutorial was originally thought to be part of a set of tutorials tailored to aid newcomers to develop for the Linux kernel Industrial I/O subsystem. This is a continuation for the “Building and booting a custom Linux kernel for ARM” and “Building and booting a custom Linux kernel for ARM using kw” tutorial.

Command Summary

If you did not read this tutorial yet, skip this section. This section was added as a summary for those that already went through this tutorial and just want to remember a specific command.

### MAKE COMMANDS TO CONFIGURE, COMPILE, AND INSTALL MODULES
make -C "$IIO_TREE" menuconfig # manage `.config` w/ a TUI
make -C "$IIO_TREE" -j$(nproc) Image.gz modules # compile kernel image and modules
make -C "$IIO_TREE" modules_install # install modules
make -C "$IIO_TREE" modules_prepare # refresh modules for compilation
make -C "$IIO_TREE" M=<path-of-dir-to-build> # compile only a subset of the tree

### MANIPULATING MODULES
modinfo <module_name> # see information about module of given name
insmod <module_file.ko> # loads module at given location
rmmod <module_name> # unloads module of given name
modprobe <module_name> # loads module of given name and its dependencies 
modprobe -r <module_name> # unloads module of given name
depmod --quick # updates the module dependency list only if a module is added or removed

### SEE KERNEL LOGS
dmesg | tail # see last logs

Introduction

Kernel modules are code segments that can be dynamically inserted into or removed from the kernel as needed. They enhance the kernel’s capabilities without requiring a system reboot. A prime example is a device driver module, which enables the kernel to communicate with connected hardware. Without modules, systems typically rely on monolithic kernels, where new features must be directly incorporated into the kernel image. This results in larger kernel sizes and demands recompilation and a system restart whenever additional functionality is introduced.

Starting to tinker with the Linux codebase through modules is interesting as we can have an isolated context to understand other concepts about Linux development, and this is precisely what we will do in this tutorial.

Introduction to Linux kernel build configuration and modules

This part shows how to build and test a simple kernel module and explores the Linux kernel build configuration further by explaining how to use menuconfig to enable kernel features and how to create your own build configuration for a simple example module.

Summary of the parts of this tutorial:

1. Creating a simple example module
2. Creating Linux kernel configuration symbols
3. Configuring the Linux kernel build with menuconfig
4. Installing Linux kernel modules
5. Dependencies between kernel features

1) Creating a simple example module

Don’t forget to have the activate.sh script built in the last tutorials running.

From the root of the Linux kernel source code, create the file ${IIO_TREE}/drivers/misc/simple_mod.c and add the code for the simple mod in there.

#include <linux/module.h>
#include <linux/init.h>

static int __init simple_mod_init(void)
{
	pr_info("Hello world\n");
	return 0;
}

static void __exit simple_mod_exit(void)
{
	pr_info("Goodbye world\n");
}

module_init(simple_mod_init);
module_exit(simple_mod_exit);

MODULE_LICENSE("GPL");

2) Creating Linux kernel configuration symbols

Now, let’s create a Kconfig configuration symbol for our simple module and add the associated Kbuild configuration option to build it. When adding new Kconfig symbols we usually write them in the Kconfig file that stands in the same directory of the thing we want to build. The simple_mod.c module is under ${IIO_TREE}/drivers/misc/ so we will add a configuration symbol for that in ${IIO_TREE}/drivers/misc/Kconfig.

When adding entries to kbuild Kconfig and Makefiles we also follow the convention of keeping the entries in alphabetical order. The convention is not enforced by the build system so out of order entries will not prevent us from building the kernel. Nevertheless, keeping entries in order definitely helps developers find build configurations when looking for them. Also, the Linux kernel community will ask code submitters to keep things organized when upstreaming new configuration symbols. Let’s keep the good practices.

config SIMPLE_MOD
	tristate "Simple example Linux kernel module"
	default	n
	help
	  This option enables a simple module that says hello upon load and
	  bye on unloading.

The config keyword defines a new configuration symbol. Further, Kbuild will generate a configuration option for that symbol which in turn will be stored as a configuration entry in the .config file. Configuration options will also show in kernel configuration tools such as menuconfig, nconfig, or during the compilation process. In particular, the SIMPLE_MOD configuration symbol has the following attributes:

• tristate: the type for the configuration option. It declares that this symbol stands for something that may be compiled as a module (m), built-in compiled (y) (i.e., included in the kernel image), or not compiled at all (n). The type definition also accepts an optional input prompt to set the option name that kernel configuration tools display.
• default: the value that should be selected by kernel config tools if no explicit value has been assigned to the associated configuration option (such as when applying a defconfig).
• help: defines a help text to be displayed as auxiliary info.

Other common attributes for configuration symbols are:

• bool: type for features that can only be either enabled or disabled.
• depends on: list of dependency symbols. If its dependencies are not satisfied, this symbol may become non-visible during configuration or compilation time. As an experiment, try to disable SPI support at Device Drivers. Many ADCs will no longer be listed at Device Drivers -> Industrial I/O support -> Analog to digital converters.
• select: when the symbol containing the select list is enabled, the symbols from its select list will also be enabled. Note the symbols in this list will not be disabled if the symbol containing the select list is later disabled.

Now we add the simple_mod module to the list of build objects in ${IIO_TREE}/drivers/misc/Makefile.

obj-$(CONFIG_SIMPLE_MOD)		+= simple_mod.o

That’s all we need for enabling the configuration of our simple_mod with kbuild.

3) Configuring the Linux kernel build with menuconfig

Run menuconfig and enable our example module.

make -C "$IIO_TREE" menuconfig

Type forward slash (/) to search by symbol name. In the search screen, type simple_mod then enter. The description of the simple_mod will appear. Type 1 to go to the configuration option. With selection over the simple_mod option, type m to enable it as a module. Save the configuration and exit menuconfig.

Before building the image and modules, let’s clean any artifacts from previous compilations. This isn’t strictly necessary, but it helps us ensure we are on the same page as much as possible.

Without kw

make -C "$IIO_TREE" -j$(nproc) clean

With kw

cd "$IIO_TREE"
kw build --clean

Build image and modules again.

Without kw

make -C "$IIO_TREE" -j$(nproc) Image.gz modules

With kw

cd "$IIO_TREE"
kw build

Install new kernel modules. No need to copy or install the kernel image since virt will pick the generated image file at ${IIO_TREE}/arch/arm/boot/Image.

Without kw

### ADAPT THE COMMAND BELOW ###
sudo guestmount --rw --add "${VM_DIR}/arm64_img.qcow2" --mount /dev/<rootfs> "${VM_MOUNT_POINT}" # mount the VM `rootfs` to the given mount point in read and write mode (this could take a while)
sudo --preserve-env make -C "${IIO_TREE}" modules_install # install modules to inside the VM
sudo guestunmount "$VM_MOUNT_POINT" # unmount the VM `rootfs`

With kw

mkdir "${VM_DIR}/arm64_rootfs"
### ADAPT THE COMMAND BELOW ###
sudo guestmount --rw --add "${VM_DIR}/arm64_img.qcow2" --mount /dev/<rootfs> "${VM_DIR}/arm64_rootfs" # mount the VM `rootfs` to the given mount point in read and write mode (this could take a while)
sudo --preserve-env make -C "${IIO_TREE}" INSTALL_MOD_PATH="${VM_DIR}/arm64_rootfs" modules_install # install modules to inside the VM
sudo guestunmount "$VM_MOUNT_POINT" # unmount the VM `rootfs`

Connect to the VM with ssh.

Without kw

sudo virsh start arm64
sudo virsh start arm64
sudo virsh net-dhcp-leases default
ssh root@<VM-IP-address>

With kw

sudo virsh start arm64
kw ssh

Verify the kernel version.

@VM
uname --all
cat /proc/version

4) Installing Linux kernel modules

Run modinfo which shows main info related to a kernel module. When known, modinfo will show the module file name, module author, module description, license, ailas, dependencies, signature, and signer.

@VM
modinfo simple_mod

List currently loaded kernel modules.

@VM
lsmod

You should have an output in similar to the below.

root@localhost:~# lsmod
Module                  Size  Used by
crct10dif_ce           12288  1
cfg80211              409600  0
rfkill                 28672  2 cfg80211
drm                   577536  0
dm_mod                131072  0
ip_tables              28672  0
x_tables               40960  1 ip_tables

Notice our simple_mod is not loaded. Let’s take care of it. There are two ways of loading a Linux kernel module: insmod and modprobe.

insmod takes a path to a module file (.ko) and loads that into the running kernel. The kernel object file doesn’t really need to have been installed as we did with modules_install. rmmod unloads the module. Load our example module with insmod.

@VM
insmod /lib/modules/$(uname -r)/kernel/drivers/misc/simple_mod.ko # loads module at given location

Then run dmesg to see kernel log messages. You should have something like this

root@localhost:~# dmesg | tail
<snipped>
[ 3962.547283] Hello world

Remove the module with rmmod.

@VM
rmmod simple_mod # unloads module of name `simple_mod`

Then run dmesg again to see kernel log messages. You should have something like this

root@localhost:~# dmesg | tail
<snipped>
[ 3973.986089] Goodbye world

We can do the same with modprobe.

@VM
modprobe simple_mod # loads module of name `simple_mod` 
dmesg | tail
modprobe -r simple_mod # unloads module of name `simple_mod` 
dmesg | tail

Instead of a module file, modprobe takes the module name as argument. For that to work, the module has to be installed within the kernel and module tracking files (such as modules.dep) must contain references to the requested module. The advantage of having that is that modprobe will look for module dependencies and (if any) properly load them before loading the requested module [1]. insmod does not check for any module dependencies.

5) Dependencies between kernel features

Let’s increment our example module to export a function that can be called by other modules.

#include <linux/module.h>
#include <linux/init.h>

void simple_mod_func(void);

void simple_mod_func(void)
{
	pr_info("Called %s, %s function\n", KBUILD_MODNAME, __func__);
}
EXPORT_SYMBOL_NS_GPL(simple_mod_func, "IIO_WORKSHOP_SIMPLE_MOD");

static int __init simple_mod_init(void)
{
	pr_info("Hello from %s module\n", KBUILD_MODNAME);
	return 0;
}

static void __exit simple_mod_exit(void)
{
	pr_info("Goodbye from %s\n", KBUILD_MODNAME);
}

module_init(simple_mod_init);
module_exit(simple_mod_exit);

MODULE_LICENSE("GPL");

Rebuild the example module and copy it to the virtual machine.

Without kw

make -C "$IIO_TREE" M="${IIO_TREE}/drivers/misc/"
scp "${IIO_TREE}/drivers/misc/simple_mod.ko" root@<VM-IP-address>:~/

With kw

cd "$IIO_TREE"
kw build
kw ssh --send drivers/misc/simple_mod.ko

The M= option specify a directory for external module build. With that, we can only rebuild the modules of a child directory such as ${IIO_TREE}/drivers/misc. Inside the virtual machine, test the new simple_mod version. No need to reboot.

Inside the VM, do

@VM
cp simple_mod.ko /lib/modules/`uname -r`/kernel/drivers/misc/
depmod --quick
modprobe simple_mod
modprobe -r simple_mod
dmesg | tail

Now, let’s add another module to call the exported simple_mod function. Create a file ${IIO_TREE}/drivers/misc/simple_mod_part.c with the following contents.

#include <linux/module.h>
#include <linux/init.h>

extern void simple_mod_func(void);

static int __init simple_mod_part_init(void)
{
	pr_info("Hello from %s module\n", KBUILD_MODNAME);
	simple_mod_func();
	return 0;
}

static void __exit simple_mod_part_exit(void)
{
	pr_info("Goodbye from %s\n", KBUILD_MODNAME);
}

module_init(simple_mod_part_init);
module_exit(simple_mod_part_exit);

MODULE_LICENSE("GPL");
MODULE_IMPORT_NS("IIO_WORKSHOP_SIMPLE_MOD");

Also add entries to ${IIO_TREE}/drivers/misc/Kconfig and ${IIO_TREE}/drivers/misc/Makefile as we did for simple_mod.

config SIMPLE_MOD_PART
	tristate "Simple Test Partner Module"
	depends on SIMPLE_MOD
	help
	   Enable this configuration option to enable the simple test partern
	   module.

obj-$(CONFIG_SIMPLE_MOD_PART) += simple_mod_part.o

Run menuconfig again to enable simple_mod_part to build as a module.

make -C "$IIO_TREE" menuconfig

You will notice though, if you try to build the modules with make M=drivers/misc/ that it will build simple_mod but won’t build simple_mod_part. To get simple_mod_part built and setup as we have simple_mod, run the modules_prepare rule. Alternatively, we can rebuild the kernel and install the modules again. Let’s go for the first option which is faster.

Without kw

make -C "$IIO_TREE" modules_prepare
make -C "$IIO_TREE" M=drivers/misc/
scp "${IIO_TREE}/drivers/misc/simple_mod_part.ko" root@<VM-IP-address>:~/

With kw

cd "$IIO_TREE"
kw build
kw ssh --send drivers/misc/simple_mod_part.ko

Load simple_mod_part and check out the output in kernel logs.

@VM
cp simple_mod_part.ko /lib/modules/`uname -r`/kernel/drivers/misc/
depmod --quick
modinfo simple_mod_part
modprobe simple_mod_part
modprobe -r simple_mod_part
dmesg | tail

You should have an output like

root@localhost:~# dmesg | tail
[  119.775025] Hello from simple_mod module
[  119.777102] Hello from simple_mod_part module
[  119.777290] Called simple_mod, simple_mod_func function
[  128.551064] Goodbye from simple_mod_part
[  128.563960] Goodbye from simple_mod

Note you don’t have to explicitly ask to load simple_mod if using modprobe. To summarize, when adding new modules we need to prepare them for inclusion or rebuild the whole kernel to get the modules correctly setup. After the modules have been setup, we can modify modules, build only those that were updated, copy them to the VM, update module dependancies with depmod, then test.

Proposed Exercices

1. The .config file that comes with the arm64 VM is bloated with features built together with the kernel image (y config value) which is why even after make localmodconfig the .config file did not reduce significantly and the build took a lot of time. Run make allmodconfig to turn builtin configuration values into module values when possible. After that, boot the VM, regenerate the list of needed modules as described in Part 1, run make localmodconfig with new list of modules. Did it reduce .config size further? How much? Does the whole kernel build take less time with the new .config? Does the resulting kernel still boot?

2. Sometimes developers lose track of what .config was used to generate a running kernel after messing arround for enough time. The in kernel configuration config (IKCONFIG) exports (imports?) the .config file used to build the kernel into the kernel image and make it later availabe as /proc/config.gz file. Enable IKCONFIG, rebuild the kernel and read your .config from /proc/config.gz within the VM.

3. Customize the log messages for simple_mod and simple_mod_part. Add #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt at the top of the module source files. KBUILD_MODNAME will expand to the module source file name resulting in every message logged through pr_info() (and friends) being prepended by the module name. See include/linux/printk.h for documentation. Tip, add the above define before header inclusions to avoid build warnings.

This tutorial describes how to build - in other words, compile - the Linux kernel for the ARM architecture and boot test it in a Virtual Machine (VM). Basic kernel build configuration is covered too.

This tutorial was originally thought to be part of a set of tutorials tailored to aid newcomers to develop for the Linux kernel Industrial I/O subsystem (IIO). It is the second in the series and a continuation of the “Setting up a test environment for Linux Kernel Dev using QEMU and libvirt” tutorial.

Command Summary

Skip this section if you have not completed this tutorial yet. This is a reference for those who have gone through this tutorial and want to remember specific commands.

### SHALLOWLY CLONING THE IIO TREE W/ ACTIVE DEVELOPMENT BRANCH
git clone git://git.kernel.org/pub/scm/linux/kernel/git/jic23/iio.git "${IIO_TREE}" --branch testing --single-branch --depth 10

### CREATING A LEAN `.config` file
make -C "$IIO_TREE" defconfig # create predefined defconfig
make -C "$IIO_TREE" olddefconfig # update defconfig with new default values
make -C "$IIO_TREE" localmodconfig # reduce compilation to minimal set of modules

### SAFELY MODIFYING THE `.config` W/ A TUI
make -C "$IIO_TREE" nconfig

### BUILDING A LINUX KERNEL FROM SOURCE CODE
make -C "$IIO_TREE" -j$(nproc) Image.gz modules

### MOUNTING VM `rootfs` AND INSTALLING MODULES
sudo guestmount --rw --add "${VM_DIR}/arm64_img.qcow2" --mount /dev/<rootfs> "${VM_MOUNT_POINT}" # mount the VM `rootfs` to the given mount point in read and write mode (this could take a while)
sudo --preserve-env make -C "${IIO_TREE}" modules_install # install modules to inside the VM
sudo guestunmount "$VM_MOUNT_POINT" # unmount the VM `rootfs`

Introduction

After setting up a test environment with VMs using QEMU and libvirt, we begin compiling (also called building) a custom Linux kernel from source and booting it. This involves taking the source code developed by the Linux community, compiling it into a monolithic executable that can be loaded into memory and run on hardware (virtualized or not), and using it to start a system. Additionally, we will compile and install kernel modules, smaller units that can be dynamically loaded and unloaded while the system is running. Cross-compilation will also be a factor, as we assume that the machine used to compile the source code to these artifacts has an AMD64 architecture, while the target architecture is ARM64.

Configuring, building, and booting a custom Linux kernel

In this section we will go through the steps on how to configure, compile, and boot customized Linux images in ARM64 VMs.

Summary of this tutorial:

1. Cloning a Linux kernel tree
2. Configuring the Linux kernel compilation
3. Building a custom Linux kernel
4. Installing modules and booting the custom-built Linux kernel

Before proceding remember to have the activate.sh script we’ve created available to be modified and active for all the commands run in the host system. Also assure that you have your testing environment set up from the last tutorial.

1) Cloning a Linux kernel tree

There are several repositories that contain the source code for the Linux kernel. These repositories are known as Linux kernel trees, in short, kernel trees or just trees, because, as every software project, its source code is arranged in a tree-like file hierarchy. Some trees are widely known such as Linus Torvalds’ tree (known as mainline) that represent the most updated official version of Linux and the Linux stable tree that represents the stable releases (the 6.10, 6.11, 6.12, etc.). In general, a Linux tree is a repository where some development for the kernel happens and each tree follows its own development ruled by its own community. Many of those repos are at kernel.org.

Some examples of Linux kernel trees are:

• Linus Torvalds’ tree (mainline)
• Linux-stable tree
• Linux-next tree
• IIO subsystem tree
• Raspberry Pi tree
• Analog Devices tree

For this workshop, we’ll be using the Industrial I/O (IIO) subsystem tree so download (clone) it with git. First, modify your activate.sh script

--- a/activate.sh
+++ b/activate.sh
@@ -4,6 +4,7 @@
 export LK_DEV_DIR='/home/lk_dev' # path to testing environment directory
 export VM_DIR="${LK_DEV_DIR}/vm" # path to VM directory
 export BOOT_DIR="${VM_DIR}/arm64_boot" # path to boot artifacts
+export IIO_TREE="${LK_DEV_DIR}/iio" # path to IIO subsystem Linux kernel tree

 # utility functions

then rerun the activate.sh script and clone the tree.

git clone git://git.kernel.org/pub/scm/linux/kernel/git/jic23/iio.git "${IIO_TREE}" --branch testing --single-branch --depth 10

In terms of source code, Linux kernel trees are already huge (a couple of GB in disk space), but the git history is even bigger at this point (around 5GB). To (hopefully) not use too much disk space and avoid longer download times, we use the --branch option, which tells git we want the testing branch of the remote pointed by the URL, and the --depth=10 option, which limits the git history to the latest 10 commits in the branch to be fetched. We’ve chosen the testing branch instead of the default master because the former is where the actual development happens. Hence, as in this series, we aim to (possibly) develop and propose changes to IIO, the testing branch is the one we are interested in.

If you have plenty of disk space (and probably some spare time), I suggest you clone the tree without the --branch and --depth flags because commit logs are often a good source of information when trying to understand kernel code. By the time this post was written, the IIO tree (with its entire git history) was sizing roughly 5GB.

2) Configuring the Linux kernel compilation

The Kernel Build System (kbuild) is based on make and other GNU tools and allows a highly modular and customizable build process for the Linux kernel. By default, kbuild uses the configuration options stored in the .config file under the root directory of the Linux kernel tree, which isn’t present in a freshly cloned tree. Those options hold values for configuration symbols associated with kernel resources such as drivers, tools, and features in general. Nearly all directories inside a kernel tree have a Kconfig file, which defines the symbols for the resources at the same level. Top Kconfig files include (source) Kconfig files from subdirectories thus creating a tree of configuration symbols. When needed, kbuild generates configuration options from Kconfig symbols and stores their values in a .config file. kbuild Makefiles then use these configuration values to compile code conditionally and to decide which objects to include in the monolithic kernel image and which objects to become modules [1] [2].

There are predefined .configs for building kernels for different architectures and purposes. These are called defconfig (default configuration) files. defconfig files store only specific non-default values for configuration symbols. For instance, one can find defconfig files for ARM architecture machines under arch/arm/configs. We will create a .config file from the arm64 defconfig. For it, we must specify our target architecture for the build. Once again, modify the activate.sh script and rerun it.

--- a/activate.sh
+++ b/activate.sh
@@ -5,6 +5,8 @@ export LK_DEV_DIR='/home/lk_dev' # path to testing environment directory
 export VM_DIR="${LK_DEV_DIR}/vm" # path to VM directory
 export BOOT_DIR="${VM_DIR}/arm64_boot" # path to boot artifacts
 export IIO_TREE="${LK_DEV_DIR}/iio" # path to IIO subsystem Linux kernel tree
+## Linux kernel tree build variables
+export ARCH=arm64 # variable defining target architecture

 # utility functions

Then create the .config defconfig file for the ARM architecture.

make -C "$IIO_TREE" defconfig # create predefined defconfig
make -C "$IIO_TREE" olddefconfig # update defconfig with new default values
ls -alFh "${IIO_TREE}/.config" # list file infos of generated `.config`

The -C option, like with git, tells make to run the commands as if it was in the path passed as value. This series of tutorials does this (and other things) to avoid requiring commands to be run in implied directories. Ideally, you should always be aware of where you are executing a command and if it makes sense. Nevertheless, to avoid unexpected troubles, we present make commands with “mistake mitigation” flags.

Before moving forward, let’s start the VM running ARM64, which we set up in the last tutorial. Note that if either libvirtd, the default network, or the VM are already started, some commands will raise errors, indicating that these are already active.

@host
sudo systemctl start libvirtd # starts the libvirt daemon
sudo virsh net-start default # starts `virsh` default network
sudo virsh start --console arm64 # start a registered instance attaching console

After the VM is running, fetch the saved list of modules you’ve generated in the first tutorial of the series. Refer to section 4 of the first tutorial, in case you don’t have this file.

@host
sudo virsh net-dhcp-leases default # list `default` network to get VM IP
scp root@<VM-IP-address>:~/vm_mod_list "${IIO_TREE}" # copy VM modules list to IIO tree

Now, use this file to make a minimal set of modules selected for compilation, thus reducing the time to build the kernel and the amount of VM disk space required to install the modules. First, modify and rerun the activate.sh script

--- a/activate.sh
+++ b/activate.sh
@@ -7,6 +7,7 @@ export BOOT_DIR="${VM_DIR}/arm64_boot" # path to boot artifacts
 export IIO_TREE="${LK_DEV_DIR}/iio" # path to IIO subsystem Linux kernel tree
 ## Linux kernel tree build variables
 export ARCH=arm64 # variable defining target architecture
+export LSMOD="${IIO_TREE}/vm_mod_list" # path to list of minimal set of modules

 # utility functions

then, optimize the .config file with the minimal set of modules to be built

make -C "$IIO_TREE" localmodconfig

New options might be prompted due to the differences in the files. You can just spam the ENTER key for the default values.

It is possible to open the .config file and directly edit it, but this isn’t recommended. A safer and more palatable way is to use one of the Terminal User Interfaces (TUI) provided by the Linux project

make -C "$IIO_TREE" nconfig

Use the arrow keys to navigate and ENTER to change/toggle a configuration. You can take some time to familiarize with these menus and see the available configurations.

For this tutorial, we want to make a tiny customization: change the kernel image name. In the menu, select General setup, then in Local version - append to kernel release, put whatever you want to be appended after the kernel version, and disable Automatically append version information to the version string. Be aware that the string you input will be part of the name of your first custom kernel, so treat it well :)

To exit, hit F9, which will prompt you to save the changes. The figure below illustrates the steps.

3) Building a custom Linux kernel

Different processor architectures have distinct instruction sets and register names. Due to that, the binaries produced by a compiler for architecture A will not work on a machine of architecture B. So, we need to use a compiler that produces binaries compatible with the instruction set of the machine we want to run our kernel. Essentially, this concept is called Cross-Compilation.

In our case, we assume we have a development system of the AMD64 (x86_64) architecture, and the target machine is a VM of the ARM64 architecture.

Most distros should have a GCC package with a compiler for AMD64 host machines that produces binaries for ARM64 targets. On Debian and Fedora, the package name is gcc-aarch64-linux-gnu, while in Arch the package name is aarch64-linux-gnu-gcc.

# Arch-based distros
sudo pacman -Syy && sudo pacman -S aarch64-linux-gnu-gcc
# Debian-based distros
sudo apt update && sudo apt install gcc-aarch64-linux-gnu
# Fedora-based distros
sudo dnf update && sudo dnf install gcc-aarch64-linux-gnu

See the Complementary Commands section for advice if you are having difficulties in finding an equivalent package to your distro.

Modify the activate.sh script (don’t forget to rerun it) to include a env variable that defines our cross-compiler

--- a/activate.sh
+++ b/activate.sh
@@ -5,10 +5,12 @@ export IIO_TREE="${LK_DEV_DIR}/iio" # path to IIO subsystem Linux kernel tree
 ## Linux kernel tree build variables
 export ARCH=arm64 # variable defining target architecture
 export LSMOD="${IIO_TREE}/vm_mod_list" # path to list of minimal set of modules
+export CROSS_COMPILE=aarch64-linux-gnu- # cross-compiler

# utility functions

The kernel has many build targets, though we will only use the Image.gz and modules targets. Use make help to view a list of available targets. Finally, let’s build our custom Linux kernel from source code. Mind that, although we made a really lean .config, compiling the kernel still is a heavy task, and even more so when cross-compiling.

make -C "$IIO_TREE" -j$(nproc) Image.gz modules

It is likely that the above command will fail due to the software required for the build being missing. Yet, kbuild does a good job of telling what is missing from the Linux build. So, one may often identify what to install after analyzing the errors in the build output. On Debian-based OSes, developers often need to install flex, bison, and ncurses.

There is also a minimal requirements to compile the kernel page with a list of software required to build Linux and how to check your system has the minimal required versions of them.

The make command will instruct kbuild Makefiles to start the build process. The main goal of the kbuild Makefiles is to produce the kernel image (vmlinuz) and modules [2]. Akin to Kconfig files, kbuild Makefiles are also present in most kernel directories, often working with the values assigned for the symbols defined by the former.

The whole build is done recursively — a top Makefile descends into its sub- directories and executes each subdirectory’s Makefile to generate the binary objects for the files in that directory. Then, these objects are used to generate the modules and the Linux kernel image. [1]

If everything goes right, you should see an arch/arm64/boot/Image, modules.order and alike files under the Linux source root directory.

4) Installing modules and booting the custom-built Linux kernel

Before booting the custom-built kernel, we need to install the modules into the VM, i.e., we need to move the module objects compiled to the right place inside the VM’s filesystem. To do this, we mount the VM rootfs (it should be the same sdaX partition from the first tutorial), which essentially attaches the virtual disk representing the rootfs into a mount point that can be accessed by our host filesystem.

First, alter the activate.sh script to include an env variable for the mount point and the path to install the modules used by the make command we will run.

--- a/activate.sh
+++ b/activate.sh
@@ -5,10 +5,12 @@ export LK_DEV_DIR='/home/lk_dev' # path to testing environment directory
 export VM_DIR="${LK_DEV_DIR}/vm" # path to VM directory
 export BOOT_DIR="${VM_DIR}/arm64_boot" # path to boot artifacts
 export IIO_TREE="${LK_DEV_DIR}/iio" # path to IIO subsystem Linux kernel tree
+export VM_MOUNT_POINT="${VM_DIR}/arm64_rootfs" # path to mount point for VM rootfs
 ## Linux kernel tree build variables
 export ARCH=arm64 # variable defining target architecture
 export LSMOD="${IIO_TREE}/vm_mod_list" # path to list of minimal set of modules
 export CROSS_COMPILE=aarch64-linux-gnu- # cross-compiler
+export INSTALL_MOD_PATH="$VM_MOUNT_POINT" # path to install modules inside VM

 # utility functions

Rerun the activate.sh script, then create a mount point, mount the VM, and install the modules. The commands below are being run with superuser privileges due to actions related to mounting, manipulating mounted directories, and unmounting not being available for normal users. ASSURE THAT THE VM IS SHUT DOWN BEFORE PROCEEDING.

mkdir "$VM_MOUNT_POINT" # creates mount point
### ADAPT THE COMMAND BELOW ###
sudo guestmount --rw --add "${VM_DIR}/arm64_img.qcow2" --mount /dev/<rootfs> "${VM_MOUNT_POINT}" # mount the VM `rootfs` to the given mount point in read and write mode (this could take a while)
sudo --preserve-env make -C "${IIO_TREE}" modules_install # install modules to inside the VM
sudo guestunmount "$VM_MOUNT_POINT" # unmount the VM `rootfs`

ASSURE THAT THE VM ROOTFS IS UNMOUNTED BEFORE PROCEDING.

Finally, update your activate.sh script to boot the VM (both through pure QEMU and with libvirt) with the custom-built kernel. Don’t forget to adapt the modification to have the correct initrd and the correct vda partition depending on your VM particularities!

--- a/activate.sh
+++ b/activate.sh
@@ -23,7 +23,7 @@ function launch_vm_qemu() {
         -smp 2 \
         -netdev user,id=net0 -device virtio-net-device,netdev=net0 \
         -initrd "${BOOT_DIR}/<initrd>" \
-        -kernel "${BOOT_DIR}/<kernel>" \
+        -kernel "${IIO_TREE}/arch/arm64/boot/Image" \
         -append "loglevel=8 root=/dev/<vdaX> rootwait" \
         -device virtio-blk-pci,drive=hd \
         -drive if=none,file="${VM_DIR}/arm64_img.qcow2",format=qcow2,id=hd \
@@ -41,7 +41,7 @@ function create_vm_virsh() {
         --import \
         --features acpi=off \
         --disk path="${VM_DIR}/arm64_img.qcow2" \
-        --boot kernel=${BOOT_DIR}/<kernel>,initrd=${BOOT_DIR}/<initrd>,kernel_args="loglevel=8 root=/dev/<vdaX> rootwait" \
+        --boot kernel=${IIO_TREE}/arch/arm64/boot/Image,initrd=${BOOT_DIR}/<initrd>,kernel_args="loglevel=8 root=/dev/<vdaX> rootwait" \
         --network bridge:virbr0 \
         --graphics none
 }

Rerun the activate.sh script. We need to undefine our libvirt managed VM, then created it again

@host
sudo virsh undefine arm64
create_vm_virsh

Log into the VM and run

@VM
uname --kernel-release

to check that you are now running the custom kernel we just built. The output of this command should be something in the format of

<major-release-nr>.<minor-release-nr>.<patch-release-nr>-rc<one-through-eight>-<the-name-you-wrote>+

like, for example,

6.14.0-rc1-free-software+

Congratulations, you’ve just compiled and boot-tested a Linux kernel! From now on, whenever you make a change on the Linux kernel tree, you can “just” compile it, install the modules, and then start the VM to boot the new custom kernel and be able to run validations inside the testing environment!

4.1) Installing the kernel image

Often, kernel developers also need to explicitly install the Linux kernel image to their target test machines. Notice that here, we are not copying or moving the Linux kernel image to nowhere inside the VM like we did when installing modules. Neither we are dealing with a physical machine.

Essentially, installing a new kernel image would be to just replace the vmlinuz/Image/zImage/bzImage/uImage file, which contains the Linux boot executable program. However, some platforms (such as AMD64 and ARM64) have fancy boot procedures with boot loaders that won’t find kernel images without very specific configuration pointing to them (e.g., GRUB), which might mount temporary file systems (initrd), load drivers prior to mounting the rootfs, and so on. To help setup those additional boot files and configuration, the Linux kernel has a install rule. So, kernel developers may also run make install or make install INSTALL_PATH=<path_to_bootfs> when deploying kernels to those platforms.

For this setup, we shall not bother with that. We don’t need to run the installation rule because we instructed QEMU (with -kernel) and libvirt (with --boot kernel=...) to pick up the kernel image that resulted from the compilation, and we are reusing the initrd file from the original guest OS that came with the disk image.

Complementary Commands

One may also download cross compiler toolchains from different vendors. For instance, ARM provides an equivalent cross compiler that you may download if having trouble finding a proper distro package.

wget -O "${LK_DEV_DIR}/gcc-aarch64-linux-gnu.tar.xz" https://developer.arm.com/-/media/Files/downloads/gnu-a/10.3-2021.07/binrel/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu.tar.xz
tar -xf -C "$LK_DEV_DIR" "${LK_DEV_DIR}/gcc-aarch64-linux-gnu.tar.xz"

Sometimes identifying the cross compiler for your combination of host and target machines may require some understanding of what is called the compiler triplet. Conceptually, the compiler triplet should contain three fields: the name of the CPU family/model, the vendor, and the operating system name [3]. However, sometimes the vendor is omitted so one may find a triplet like x86_64-freebsd (FreeBSD kernel for 64-bit x86 CPUs) [3]. It is also common to see the operating system information split into two separate fields, one for indicating the kernel and the other for describing the runtime environment or C library which is being used. The the debian package for x86-64 gcc is an example of this triplet format mutation: gcc-x86-64-linux-gnu (compiler for 64-bit x86 targets that will run a Linux kernel and have GNU glibc in their runtime). But things can get even more unintuitive when system call conventions or Application Binary Interfaces (ABI) are specified in the OS field as in arm-linux-gnueabi (compiler for 32-bit ARM targets that will run Linux using the EABI system call convention) or as in arm-none-eabi (compiler for 32-bit ARM that will run no OS (bare-metal) using the EABI system call convention).

Anyways, you may point to the generic cross compiler name when using compilers not under your PATH. For example:

export CROSS_COMPILE="${LK_DEV_DIR}/gcc-aarch64-linux-gnu/bin/aarch64-none-linux-gnu-"

Conclusion

This tutorial described how to configure and build a custom Linux kernel and boot-test it into a VM. To accomplish that, we covered basic concepts of Linux kernel build configuration to guide readers into generating feasible .config files, and some cross-compilation concepts. By this point, you should be able to configure, build, and boot-test a custom Linux kernel from source code in a safe testing environment.

History

1. V1: Release
2. V2: Review for the Free Software Development course offered in 2025-1 at IME-USP
3. V3: Incorporate Marcelo Schmitt feedbacks from Merge Request 135

References

[1] Javier Martinez Canillas. “Kbuild: the Linux Kernel Build System”. (2012) URL: https://www.linuxjournal.com/content/kbuild-linux-kernel-build-system. ⤴

[2] Michael Elizabeth Chastain and Kai Germaschewski and Sam Ravnborg. “Linux Kernel Makefiles”. (2023) URL: https://www.kernel.org/doc/html/latest/kbuild/makefiles.html. ⤴

[3] . “Target Triplet”. (2019) URL: https://wiki.osdev.org/Target_Triplet. ⤴