Administrators responsible for patching Linux know that it’s practically a full-time job in a large enterprise environment. To patch just one system, the administrator must identify that a patch is available, download it, and then deploy it to the system. In an enterprise environment, there could be hundreds of servers to manage, so the job of patch management becomes an all-day responsibility with the added risk of reboot fails after installation. Instead of manual updates, administrators can free up time and organize patches using automation tools.
- What is Patch Management
- Why is Patch Management Important?
- How Often Should Patch Management Be Performed?
- What are Linux Patch Management Strategies?
- How to Patch Your Linux Systems Manually?
- Patching Best Practices
- How Does Automated Linux Patch Management Software Work?
- How Live Patching Fits into the Patch Management Framework?
- How KernelCare Works with Patch Management Tools?
What is Patch Management?
Before getting into patch automation, administrators should understand the importance of patching Linux regularly. Administrators could simply patch a Linux system manually, but this leads to human errors, and rollbacks due to issues after installation are tricky. Human errors could lead to severely long downtimes when mistakes are made. It’s also time consuming to manually patch when several patches are necessary.
Patch management benefits administrators by automating the entire process. Integrating a patch management system will automatically detect updates, download them, and then deploy them to all servers. Live patching adds to these benefits by eliminating the reboot process necessary after updating Linux.
Why is Patch Management Important?
Unpatched public-facing web servers are a critical issue for cybersecurity, but cybersecurity isn’t the only reason to patch Linux. Patching also remediates bugs and adds functionality to software. Some patches fix issues with drivers and software running on the system. Large updates add functionality to the operating system.
The longer administrators wait to patch a system, the more patches will be needed to get the system up to date. This issue increases the time it takes to fully patch a Linux server. Hotfixes available from vendors and distro developers are the most important, as they fix critical issues within the operating system.
How Often Should Patch Management Be Performed?
Installing anything on a production server should be done after thorough testing. In a large enterprise environment, it’s possible for new updates to be available every day, which means constant testing and deployment. Manually checking for new patches every day is tedious and requires unnecessary overhead when patch management automation is available.
To add to the overhead, patches should be deployed only after being tested in a staging environment. Staging environments should be a replica of production to ensure that it’s a 1:1 match during testing or errors could cause downtime in production. Even though testing is important, a good rule of thumb is to apply patches within 30 days of vendors making them available.
For security patches, it’s critical that administrators test and deploy them as soon as possible. Zero-day vulnerabilities are a real threat to organizations and their digital assets. When zero-day vulnerabilities are announced, threat actors quickly create exploits to take advantage of unpatched systems. Several recent data breaches were the result of exploits on unpatched systems. To lower the risk of a data breach, organizations should rapidly deploy security patches as soon as they are available.
What are Linux Patch Management Strategies?
Unlike closed-source operating systems like Windows, Linux patching can be a bit more unpredictable and complex. Open-source has its advantages, but one disadvantage is running an operating system with several possible changes made by various contributors. Just one incompatible change could affect your entire organization.
To alleviate some of the overhead and hassles of poor patch management, here are a few strategies and best practices to incorporate into your procedures:
- Create a patch management policy. This policy should include every step including quality assurance (QA) testing, frequency of patching, any rollback procedures, and who must sign off on operating system changes.
- Use scanning tools that find vulnerabilities. Whether it’s public-facing servers or internal hosts running corporate applications, vulnerability scans will find unpatched systems to avoid common exploits.
- Use reporting to identify failed patches. How do you know that a patch installation was successful? A good patch management solution has a central dashboard that displays reports on successful and failed patch installations so administrators can review and manually patch the system if necessary.
- Deploy patches as soon as testing is complete. Testing is important before deployment, but as soon as testing provides a green light for deployment, patches should be installed across the entire environment.
- Document changes to the environment. Usually, documentation is done in the form of change control where authorized employees sign off on updates to the environment. This step is important when reviewing downtime and performing root cause analysis. It’s also important for auditing and compliance reasons.
How to Patch Your Linux Systems Manually?
Even with patch automation, manual updates are occasionally necessary. After a failed update, administrators may need to manually patch the system. Manual updates might be necessary in a testing environment. The commands to update Linux depend on your distribution, but here are the commands for some common distributions.
For Debian-based distributions (e.g. Debian, Ubuntu, Mint), the following commands will let you view available patches and update packages and the operating system:
sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade
For Red Hat Linux distributions (e.g. RedHat, CentOS, Oracle), the following commands check for updates and patches the system:
For Suse-based Linux (e.g. Suse Linux Enterprise, OpenSuse), the following commands check for updates and patch the system:
Patching Best Practices
The SysAdmin, Audit, Network, and Security (SANS) organization lays out best practices for patch management. These best practices give administrators guidance on how to implement a corporate policy that documents, audits, and assesses risk across the organization to determine when and how patches should be deployed.
The eight best practices are:
- Inventory your environment. Before you know what needs to be patched, you need an audited list of all Linux systems on the network.
- Assign risk levels to each server. Risk levels tell administrators which servers are most important and should be prioritized. All systems should be patched, but targeting the most important servers will lower risk of them being compromised while testing and other patching is underway.
- Consolidate patch management software into one solution. Automation tools are beneficial, but too many different tools making changes to the environment can lead to errors and possible race conditions.
- Review vendor patch announcements regularly. Automation tools will download updates automatically, but administrators should still be aware when new patches are available, especially critical ones.
- Mitigate risks of patch failures. It’s not uncommon for administrators to halt updates due to an issue with exceptions. When this happens, servers should be locked down to avoid exploit potential.
- Always test patches in staging first. A staging environment should replicate production, so patches can be tested and lower the risk of downtime.
- Patch systems as quickly as possible. The longer a server is unpatched, the bigger the risk of compromise due to a known vulnerability.
- Use automation tools. Automation tools take a lot of the overhead from administrators and automatically deploy patches when they are available.
Related read: Enabling Compliance with Faster Patch Management
How Does Automated Linux Patch Management Software Work?
To avoid becoming the next newsworthy data breach, organizations must do vulnerability scans on every device. Vulnerability scans identify if patches are missing, so administrators can deploy them as soon as possible. There are a few good vulnerability scanners available that make this first step much more efficient and convenient. These scanners are:
With a scan complete, it’s time for patch management tools to take over. Several tools on the market make patching much more convenient for administrators. They report on successful and failed patches so that administrators know when manual updates are necessary, and they can get an update on the current cybersecurity health of the environment.
A few tools available to manage patches include:
The above tools primary advantage is organization. These tools download updates and then report results to administrators taking away the disorganization of patch management across a large environment. Administrators can also schedule patches, choose their own deployment policies, test and then approve updates before deployment.
How Live Patching Fits into the Patch Management Framework?
Patch management tools offer organization to administrators, but reboots are still an issue. Rebooting a critical Linux server means downtime for the organization and scheduling patches for a time during off-peak hours. This means that patches could be postponed until it’s convenient, which leaves unpatched servers vulnerable.
Live patching improves the entire process by eliminating the reboot process. The reboot process brings its own set of risks. What if the system doesn’t restart? What if there are several critical servers that must be patched simultaneously? You could potentially have several critical servers that power the entire organization that need patching, and there is risk that several of them don’t restart without issues. With live patching, this risk is eliminated.
How KernelCare Works with Patch Management Tools?
KernelCare is a Linux live patching tool that integrates into current patch management solutions. Patching is still scheduled, tested, downloaded, and deployed from the patch management tool, but KernelCare offers live patching results to eliminate reboot requirements.
Here is how KernelCare live patching works:
- Allocate kernel memory and load new security code into memory.
- Temporarily freeze all processes in safe mode.
- Modify functions and jump to new secure code, which plugs the vulnerability.
- Unfreezes processes and resumes activity.
If you’re using any of the aforementioned patching tools (e.g. Ansible, Puppet, Chef, SaltStack), these tools can be used to deploy KernelCare rather than install it manually on each server. With these tools, administrators can:
- Distribute the KernelCare agent package (only necessary for servers with no internet access)
- Distribute the KernelCare agent configuration file /etc/sysconfig/kcare/kcare.conf.
- Set environment variables.
- Install the KernelCare agent from either local or remote download servers.
- Register KernelCare with key-based or IP-based licenses.
In addition to easy distribution and integration into current patch deployment applications, KernelCare also reports a safe kernel to any vulnerability scanners that poll servers for vulnerabilities. With KernelCare, your Linux servers are automatically patched and vulnerability scanners will report them as updated and current.
Integrating KernelCare into patch management reduces risk, improves cybersecurity of your Linux servers, and provides convenience to administrators. KernelCare seamlessly works with your current patching process to introduce rebootless updates.
We have customers with Linux servers that haven’t been rebooted in over six years across several different distributions. With KernelCare, data centers with over 300,000 supported servers keep their SOC2 compliance status with our live patching framework. Try KernelCare for 30 days and remove a lot of overhead and time-consuming processes from your administrators.