Updating Linux kernels is a routine – as dull as taxes and only slightly less inconvenient than death. New security vulnerabilities in the Linux kernel seem to appear with tedious regularity and even get fancy names. In most but not all cases, the patches needed to fix them follow swiftly after. There is work involved in patching the kernel the latest Linux kernel security updates, and danger if you delay–leave it too long and bad actors might take advantage of the period of vulnerability.
In our previous blog post about, we discussed How to update Linux kernel with 3 different ways, two of which (using command line/yum and kexec) require a server reboot.
It is time to review another way of Linux kernel security updates - rebootless live kernel patching. Read further to learn more about each live patching tool and alternatives.
There are times when security patching is super-critical, but so are the processes that stop when you reboot. If you're running an 'always-on' or 'high-availability' system, you'll already be familiar with this dilemma.
Rebootless Linux kernel updates are not a replacement for full kernel upgrades, as it only applies patches for security vulnerabilities or critical bug fixes. But, in many cases, this is all you need, and it is possible to keep a server safe and running for years between reboots using these methods.
A number of leading Linux vendors offer rebootless kernel updates. The one you choose depends on the distribution you run and on your budget. In the remainder of this article we'll talk about the following products:
Ksplice was the first commercially-available implementation of rebootless kernel updating. Ksplice Inc. was eventually acquired by Oracle so that now it is only available (unsurprisingly) on Oracle Linux and RedHat Enterprise Linux distributions, and the deployment needs a license from Oracle.
To deploy it, run:
sudo wget -N https://ksplice.oracle.com/uptrack/install-uptrack-oc
sudo sh install-uptrack-oc -autoinstall
Note, there is no reboot command, and you only need to run the install script once in the lifetime of the server. After that, the Uptrack service will automatically detect new kernel updates and deploy them for you. There’s no scheduling, no downtime, and nothing more to do.
This is Canonical’s technology for live-patching kernels. You can even create your own patches, although it can be difficult, time-consuming work. Some vendors will create an Ubuntu upgrade kernel for you, for a fee. The service is available for Ubuntu 16.04 and later.
It's deployed like this:
The Canonical Livepatch service is free for up to 3 machines for personal use or up to 50 machines for Ubuntu Community Members. You can sign up for a token here.sudo snap install canonical-livepatch sudo canonical-livepatch enable [TOKEN]
This is Red Hat's own kernel patching tool. It was announced in 2014 and has been ported to work on others in the same family (Fedora, CentOS) as well as for some Debian-based systems (Ubuntu, Gentoo).
Here's an example of deploying it on RHEL 7:
sudo yum install kpatch
sudo yum install kpatch-patch-X.X.X.el7.x86_64.rpm
Unlike Ubuntu's Livepatch service or Oracle's Ksplice, it's not automatic, and you must manually check for and install each kernel patch as it becomes available.
Developed and announced at almost the same time as Red Hat's solution, Kgraft is SUSE's live patching offering (known as SUSE Linux Enterprise Live Patching). It's only for SUSE's own Linux Enterprise Server 12, and comes preinstalled, so there's really nothing to do (except pay for it). It works on a different principle to most other approaches but has a feature-set comparable with Kpatch.
Also launched in 2014, KernelCare's Linux kernel live patching service stands out among the kernel patching solutions in its OS coverage, which includes CentOS, RHEL, Oracle Linux, Debian, Ubuntu and others. And like Oracle's solution, KernelCare supports the older 2.6.32 kernels from RHEL 6.
Here's how to install KernelCare:
wget -qq -O -- https://kernelcare.com/installer | bash
sudo /usr/bin/kcarectl --register <your key>
For <your key> get your trial key here.
KernelCare is an 'install and forget' solution. Once installed, KernelCare automatically downloads and applies new kernel security patches, without rebooting the server.
But in contrast to its closest competitors, KernelCare can handle some of the more complex patches for vulnerabilities such as Meltdown (CVE-2017-5754), Spectre (CVE-2017-5753 & CVE-2017-5715), and more recently, the Linux kernel buffer overflow flaw known, romantically, as Mutagen Astronomy (CVE-2018-14634). KernelCare supports custom patch configurations, fixed-date patches, delayed patches, and rebootless rollbacks, i.e. patch removals.
Like the other vendors considered here, KernelCare also springs from a good blood line–its creator is CloudLinux, the leading web hosting Linux-based OS vendor.
If your server is non-critical and can endure a period offline, updating the kernel is relatively painless using the standard tools on the command line.
If you're running an always-on system, (i.e. you can't or won't reboot), take a look at live kernel patching solutions. Of these, there are three kinds:
If you want to learn more about live patching technology and how it enables your infrastructure security - read our most popular blog posts:
Have you ever had a chance to use Linux kernel live patching tools? Which one did you find the most useful for your business? Share your thoughts in comments.