Learn how to generate test coverage reports for KUnit tests using the rather old GCC 6.

Test coverage, also called code coverage, is a metric that indicates how much source code is covered by tests. Low test coverage is a valuable indicator that some code needs more testing. On the other hand, having high test coverage doesn’t necessarily ensure good quality testing, since these tests may, for instance, fail to detect regressions, test irrelevant parts or miss some edge cases. According to “Unit Testing: Principles, Practices, and Patterns” by Vladimir Khorikov, test coverage shouldn’t be used to evaluate the quality of tests, and should only be “a first step on the way to a quality test suite”. In this sense, having test coverage reports is still an essential part in software testing.

KUnit, the unit testing framework for the Linux kernel, allows calculating test coverage, and we’ll use it to see some test coverage reports. KUnit uses gcov to generate the test coverage reports, which is a code coverage tool that can be used with GCC (and also LLVM) not only for measuring test coverage, but also as a profiling tool, showing information such as lines of code that ran and how much time that took. For more information on gcov, you may want to read the official documentation, and also the Linux kernel page that explains how it can be used there – in this last case, in a more general scope than the KUnit one.

How To

There are two limitations to generating test coverage reports with KUnit’s kunit_tool¹: we have to use a GCC version older than 7, and the tests can only be run under the UML architecture.

Disclaimer
1) If you try this tutorial with GCC 9+ versions, you will stumble upon a linking problem as, in GCC 9+ versions, mangle_path from gcc/gcov-io.* conflicts with the function of same name defined in the Linux kernel’s fs/seq_file.c.
2) In this context, GCC 7+ versions aren’t able to generate gcov’s coverage information files, making it impossible to retrieve coverage information. This happens because of the way gcov’s exit handler is dealt ².
3) Finally, we can only obtain coverage under UML because kunit_tool still lacks a way to copy the files produced by gcov in the QEMU VM³ and, for UML, it is rather easier, as it works pretty much as an executable program.

Install GCC 6

We’ll install GCC 6.5.0 from source. Open your terminal, and start by downloading it:

wget https://ftp.gnu.org/gnu/gcc/gcc-6.5.0/gcc-6.5.0.tar.gz

Extract it:

tar -xvf gcc-6.5.0.tar.gz

Create a build folder inside the gcc-6.5.0 one that you just extracted, and name it however you like. I’ll make things simple and call it build:

mkdir -p gcc-6.5.0/build

Then enter the gcc-6.5.0 directory:

cd  gcc-6.5.0

Now it’s time to download some prerequisites required by GCC. Run the following script:

./contrib/download_prerequisites

After you’re done with downloading the prerequisites, navigate into the build directory we created earlier:

cd build

We need to configure GCC before building it. Notice that the command below must be run inside the build folder, even though configure is located one directory above it:

../configure --enable-languages=c,c++ --disable-libsanitizer --disable-multilib --prefix=/usr/bin/gcc6.5.0

--enable-languages: in this case, only the compilers for C and C++ will be built.
--disable-libsanitizer: I added this to avoid some warnings thrown during installation, no noble reason here.
--disable-multilib: build only a 64-bit compiler.
--prefix: this states where we’ll store GCC, this is important since you probably already have a newer version of GCC installed in your system, and we want to avoid any conflict.

Still inside the build directory, build GCC. Keep in mind this takes a long time.

make -j$(nproc)

Finally, install GCC:

sudo make install

Grab the Linux kernel source code

For this part, we’ll download a Linux kernel version containing KUnit tests for AMDGPU. At the moment, AMDGPU doesn’t offer support for UML, but the branch we’re cloning has some tweaks to enable compilation under this architecture. Run the command below to have it:

git clone --depth 1 --branch kunit-coverage-post https://gitlab.freedesktop.org/magali/linux.git

Generate coverage reports

Go into the Linux kernel directory you just cloned:

cd linux

To generate the reports, we’ll first use kunit_tool, which will build and run the tests:

./tools/testing/kunit/kunit.py run --kunitconfig=drivers/gpu/drm/amd/display/tests/ \
  --make_options=CC=/usr/bin/gcc6.5.0/bin/gcc --kconfig_add CONFIG_VIRTIO_UML=y     \
  --kconfig_add CONFIG_UML_PCI_OVER_VIRTIO=y --kconfig_add CONFIG_DEBUG_KERNEL=y    \
  --kconfig_add CONFIG_DEBUG_INFO=y \
  --kconfig_add CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y \
  --kconfig_add CONFIG_GCOV=y

--kunitconfig: this points to a directory with a .kunitconfig file, which contains config entries needed to run the tests.
--make_options: specify the executable to compile the kernel, in this case, we installed GCC 6 in the /usr/bin/gcc6.5.0/ directory, so set the CC variable to /usr/bin/gcc6.5.0/bin/gcc.
--kconfig_add: add and set config entries.

CONFIG_VIRTIO_UML, CONFIG_UML_PCI_OVER_VIRTIO: config entries to enable CONFIG_PCI, which AMDGPU depends on.
CONFIG_DEBUG_KERNEL, CONFIG_DEBUG_INFO, CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT, CONFIG_GCOV: config entries to enable GCOV and allow test coverage generation.

After running the command above, gcov will have generated .gcda and .gcno files inside the .kunit directory – which, in this case, is where the kernel is being compiled to. These files contain information about the lines of code that were run during the execution of the tests. Check, for instance, the files inside .kunit/drivers/gpu/drm/amd/display/dc/dml/dcn20/ and notice some *.gcda and *.gcno among the other expected and familiar object files. Now the next step is to retrieve the information in these files.

lcov is the tool we’ll need to extract the gcov data generated in the previous step. You may need to install the lcov package first (sudo apt install lcov on Ubuntu or sudo dnf install lcov on Fedora). Let’s call our tests “kunit_tests” with the -t option, write the output to coverage.info, set -c to capture coverage data, point .kunit as the directory containing the gcov files, and, finally, specify the location to gcov 6:

lcov -t "kunit_tests" -o coverage.info -c -d .kunit/ --gcov-tool=/usr/bin/gcc6.5.0/bin/gcov

To generate the test coverage reports in HTML, run:

genhtml -o /tmp/coverage_html coverage.info

The reports can be found under /tmp/coverage_html, and you can find a live version of these reports here.

This tutorial was based on Tips For Running KUnit Tests.