Developer Tutorial: Live patching Debian 10 Linux kernel with Kpatch

Published: Feb 18, 2020 2:28:00 PM / Last update: Jun 25, 2020 / by Paul Jacobs

Debian10.2

Live patching is a way of updating a Linux kernel without interruption.

Because kernel updates don’t take effect until the system is rebooted, Linux kernel live patching is most commonly used to patch severe Linux kernel vulnerabilities without rebooting servers.

Aside from improved service continuity and uptime, organizations with large server fleets also use live patching to avoid the administrative overhead associated with the coordination and planning needed to reboot multiple systems.

This tutorial will show how to use Kpatch to change the behavior of a running Debian 10 kernel without stopping it, changing the contents of /proc/uptime (and the uptime command) so that the system’s reported uptime is 10 years greater.

Kpatch was created by Red Hat and works on RHEL and its derivatives. Red Hat offer a commercial live patch service for RHEL customers, as do the following companies, who each focus on different distributions:

Click to compare live patching tools

 

I have chosen Kpatch for this tutorial because it is one of the few solutions whose source code is freely available and regularly updated.

Prerequisites

Here are the system prerequisites for following this tutorial.

  • A test (non-production) system running Debian Buster (10.1 or 10.2) on an x86_64/amd64 architecture.
  • 20 Gb of free disk space. (The Linux kernel source code takes up around 909 Mb on disk, growing to 17 Gb when compiled.)
  • Your kernel has not been customized; you are using the standard one supplied by Debian
  • Your kernel has live patching built in. Use this command and expect to see two values set to y for CONFIG_HAVE_LIVEPATCH and CONFIG_LIVEPATCH:
grep LIVEPATCH /boot/config-$(uname -r)
  • Version of gcc installed matches that used to build the original kernel. (The kpatch-build command will fail if the versions don’t match. This can be overridden with the option --skip-gcc-check, although use of it is discouraged.)
    • To see the version of gcc installed: gcc --version
    • To see the version of gcc used to compile the current kernel: cat /proc/version

1. Install dependency packages

  1. Install and configure sudo.

    As root:

    apt-get install sudo
    adduser <user> sudo

    where <user> is the username for a normal user. (All subsequent commands should be done as this user.)

  2. Install packages.

    sudo apt-get -y update
    sudo apt-get -y upgrade
    sudo apt-get -y install build-essential devscripts \
    ccache # kpatch-build uses this if present \
    gawk libelf-dev libssl-dev linux-source

2. Install Kpatch

Debian’s kpatch package is out of date, so you must install it from source.

git clone https://github.com/dynup/kpatch.git
cd kpatch && make && sudo make install

3. Get a copy of the Linux kernel source code

  1. (Optional) Make and move into a working directory.

    mkdir kernel && cd $_
  2. Extract the Linux kernel source code.

    tar xaf /usr/src/linux-source-4.19.tar.xz

    Note: 4.19 is the Linux kernel version for Debian 10.2 at the time of writing. You should check and substitute the most recent version present in /usr/src.

4. Create the Linux kernel configuration file

  1. The Linux kernel is compiled using settings in a configuration file supplied with your distribution. Take a copy and change some settings so kpatch-build can compile a Linux kernel with the same settings as your running kernel.

    cd linux-source-4.19
    cp /boot/config-$(uname -r) .config
  2. Check that the required kernel settings are enabled for using Kpatch. All should return ‘y’.

    scripts/config -s DYNAMIC_FTRACE_WITH_REGS
    scripts/config -s FUNCTION_TRACER
    scripts/config -s HAVE_DYNAMIC_FTRACE_WITH_REGS
    scripts/config -s HAVE_FENTRY
    scripts/config -s HAVE_LIVEPATCH
    scripts/config -s KALLSYMS_ALL
    scripts/config -s KALLSYMS_ALL
    scripts/config -s LIVEPATCH
    scripts/config -s MODULES
    scripts/config -s MODULE_SIG
    scripts/config -s SYSFS
    scripts/config -s SYSTEM_TRUSTED_KEYRING
  3. Change the value of one kernel configuration item.

    scripts/config --set-str SYSTEM_TRUSTED_KEYS ""
  4. Leave the source directory.

    cd ..

5. Create a patch

A patch source file is the output from the diff command run on the original and the changed source code files.

The patching example shown in the ‘Quick start’ section of the kpatch github page changes the output of /proc/meminfo. Many other Kpatch articles reproduce this example, so I wanted something different and a little more interesting, yet still safe.

This example changes the output of the uptime command to give them illusion that your server’s uptime has increased by a decade.

  1. Still in your working directory, copy a file.

    cp linux-source-4.19/fs/proc/uptime.c .
  2. Edit it. At line 26, change:

    (unsigned long) uptime.tv_sec,

    to

    (unsigned long) uptime.tv_sec + 315576000,

    Save the file.

  3. Create the patch file.

    diff -u linux-source-4.19/fs/proc/uptime.c ./uptime.c > uptime.patch
  4. Create the patch module. (The first time you do this, it will take some hours, as the kernel source code must be compiled. Subsequent builds are significantly faster, in the order of minutes.)

    kpatch-build -s linux-source-4.19 -v /usr/lib/debug/lib/modules/$(uname -r)/vmlinux uptime.patch
  5. When done, you will have a Linux Loadable Kernel Module file (.ko) for the patch.

    ls -l *.ko

6. Test the patch

  1. Before loading the patch module, check the current uptime.

    cat /proc/uptime && uptime -p
  2. Load the patch module.

    sudo kpatch load livepatch-uptime.ko
  3. Check the uptime again.

    cat /proc/uptime && uptime -p

    You should see your uptime is ten years better. (The internal value hasn’t been changed, only what is printed.)

  4. Unload the patch module.

    sudo kpatch unload livepatch-uptime.ko
  5. Check the uptime has returned to its former value.

    cat /proc/uptime && uptime -p

Conclusion

Live patching the Linux kernel with Kpatch isn’t hard. The difficult lies in writing a patch that won’t crash the system and will work with other patches that come later. Most patches are created from more than a simple diff and need thorough testing across multiple kernel versions running on various distributions.

Patch writers need to be both accomplished C programmers and experienced Linux kernel developers, and compiling the kernel and testing patches for every kernel version takes a big investment in hardware and automation tools. The rarity of skills and the cost of infrastructure means that vendors must charge for live patching services.

 

Learn more about KernelCare

 

And don't forget to read our explicit guide on How to Apply Linux Kernel Security Patches: 3 Different Ways. The article explains how to update Linux kernels without rebooting, covering three different methods for some of the most popular Linux kernels. 

Topics: Developer Blog

Paul Jacobs

Written by Paul Jacobs

With more than a quarter of a century in IT, Paul brings with him a kaleidoscope of experiences and insight which he uses to drill into and pick apart the complexities of Linux server security and hosting issues, as Technical Evangelist and Content Writer for CloudLinux.

    cover for blog

    Download Whitepaper

    Subscribe to Email Updates

    Recent Posts