Patching Linux - Pain or Gain?

Problems with Patching

So you think that patching a Linux server is pretty straight-forward ? You’re probably 90% right in that assumption, but there are several issues that need to planned for or addressed when you are considering running updates on your servers.

Networks : The Bigger They Are…

If you’re running a small shop, patching your 5 to 10 servers will probably not be a big problem. Given a particular span of time, you should be able to have them all updated before the next big patch cycle. Still, it will take some coordination among your users, developers and management to minimize the effects of scheduled downtime. You’ll have to help specify how long the machines will be unavailable, have an idea of the number of reboots involved, and make sure that someone knowledgeable enough with the machine is ready to test whatever application the machine is running once the patching is done.

Now, take that same process and multiply the number of those machines by 10—or even 20. Big shops will not only have a large number of machines to patch, but the number of those affected by downtime will surely rise as these enterprise level services will now affect users numbering in the hundreds. You will also have to deal with larger scale development schedules, and the fun politics that can come from the management and community.

There is no perfect solution for this: strategies and non-strategies will dictate the tools you’ll need to get all these servers patched. Concepts from clustering servers to social engineering will help get you around the various roadblocks you’ll encounter as you try to coordinate and compromise on reboot times ranging across all hours of the day and night. You’ll need to provide assurances about when the systems will come back up, since the word “patching” can be synonymous with “un-scheduled downtimes” due to either incompatibilities between vendor software and OS updates, or just human error. You’ll also need to prepare for such occasions by having reliable backups available, prepped fail-over systems that are ready to go if the primary system doesn’t come back up, and the experience of a well-educated sys admin who can help troubleshoot problem updates after their installation.

These are just some of the issues I run into when trying to manage a large network of Linux servers in an enterprise environment. Without the right size staff, a lot this would be difficult to get done, making the management of a small IT shop look like a piece of cake by comparison.

Another problem with big shops is that they tend to have a variety of configurations when it comes to their OS installs. Because of this, it’s hard to pick and choose what patches should be applied to your machines—you don’t really have the time to sit there and pick and choose which patches need to be installed first. In response to this dilemma, you may opt to just install all of the available updates you need. The world would be perfect if we could have servers running a single upgradable app, but it just isn’t so. Instead, you have machines running Tomcat or Apache HTTP Server as your primary app, yet you also rely on the services provided by ntp, openssh or xinetd. These services, as well as many others, are updated over time, and should be upgraded when given the chance. So, don’t just update that one important app—update them all. It may save you potential headaches in the future.

The only exception to this “patch everything” rule is kernel patches. They usually require a restart of the system, whereas other patches require a daemon restart or no restart at all. If updating the machine means not having to reboot the server, then that’s just one less thing to worry about, so save the kernel patching for a better time.

Disk Space

One of the problems I’ve run into when it comes to patching a Linux OS is disk space. Hard drive capacity to cost ratios are getting better all the time: the amount of money spent on a 20 GB hard drive a few years ago will now buy you a 500 GB drive, so it’s never a bad idea to have too much disk space. Application logs, home directories, dump files and third party software will always be there. This requires you to keep track of the available disk space you have on your machine, though; remember to always have enough space when you download your patches. Thanks to the large disk capacities of today, running out of disk space is probably the last thing on your mind, but in the cases of virtualized machines or older hard drives, capacity can be an issue. If a machine is in the process of downloading a large number of patches, it’s going to need the space to store these packages so that the installation process can access the files and execute the install.

redhat yum

redhat yum

The contents of the /var/cache/yum sub-directories can show lots of valuable space wasted away holding old RPMs you don’t need anymore.

In SuSE’s Enterprise Server, for example, all of these files are stored in organized local directories on the updated server itself. Once YaST Online Update has been executed and these RPMs are used, they can be manually removed if needed. In older versions of YaST’s Online Update, you could check the “Remove installation files” box at the end of the update, which would remove the downloaded RPM files once the machine was done installing them.

  • Darkk
    Very nice article. Patching is just a way of life of sys admins everywhere regardless of what flavor of server and desktop OS. Least the article explains in detail what to expect and the gotchas.

    Good job!

  • Patching harder on Linux than Windows?!?
    Maybe I'm biased, but updating Debian or ArchLinux (more of a desktop distro) has been so easy as not to even think about it.
  • You really don't know what you're talking about here..and readers should avoid this article.

    If you buy Red Hat Enterprise Linux with a Satellite subscription Red Hat does the patching.

    If you have Novell - ZenWorks will do the trick.

    If you're running a non-commercial unsupported version than sure some of the options you mention might make sense but a simple cron job with yum/apt will do it all with one command line.

  • Was this article written by Steve Ballmer? And interestingly there is only a single line about the Debian-based distros? Why Mr. Anderson, why didn't you mention the details about APT? Now Steve Ballmer, let me tell you something - my close friend is a sysadmin of my university (University Of Toronto) and he doesnt even bother patching the systems because they are fully automated (over 200 machines). As well he deploys 50 machines with brand new OS installation with no more than 5 lines of commands.

    Sad Tom's.. sad.. you have been an amazing site once upon a time ...
  • hmmm.
    I'm a Windows guy most of the time but I enjoy playing with Linux from time to time.
    Actually as a Linux starter(some time ago) I had no problem patching my Linux.
    It was very easy...
    I do not remember reading or doing something special before patching it at that first time.
  • Correction, Patch Quest by Advent Net was cited as patching only RedHat which is incorrect. It also patches Debian. In my experience finding a patch solution for your particular OS has not be that terribly difficult. Finding one that has robust scheduling, push on demand, and can handle the multitude of necessary evil apps, such as Adobe Reader, Quicktime, Realplayer, Instant Messaging, etc... is the real challenge.
  • GoK
    The author of this article needs to go back through their information, and edit this article. It is highly inaccurate! The fact that he gave Debian-based GNU/Linux flavors (ie., Ubuntu&Gentoo) less time in the article than his praise for Mircosofts upstream ability for patches, seems a bad sign for this article.

    Patching most GNU/Linux installs is a simple task, which is highly scalable, and that can be fully automated through the use of CRON scheduling, etc. NO EXTRA SOFTWARE should be required to update/maintain ANY enterprise level GNU/Linux server distro (also if you server has a GUI on it, its not running in an enterprise level configuration).

    I find the mention of Windows Server strange in the article, since it can't run services like Bind9 (DNS), it only makes up roughly 38% of the current market share of net servers, and since it can't run Bind9, it runs NONE of the internet backbone (DNS routing server).

    I am a huge fan of Tom's, but this article should never have been published.
  • nochternus
    While there are many Linux solutions, everybody will find what works best for them. I myself have become a fan of distributions like ArchLinux. I use it on my 3 servers at work and on my desktop and server at home. the package manager, pacman, is by far the best I've ever used. While it may not categorize some things into software groups, it does have it broken down into core, extra and then everything else. It is also extremely easy to configure and create wrappers or optional interfaces that utilize pacman (just like some of the others mentioned. There is also a package called the "arch build system" that allows you to create your own packages from source with the simple modifications of a PKGBUILD file, making recompiling and rebuilding easy and efficient. My latest server was not fully supported by a vanilla or even a patched kernel so a few quick modifications to the PKGBUILD and the kernel config and one command later, the package was compiled from source and installed without me sweating, swearing or crying.

    I don't want this to come off as a "YAY ARCH - EVERYBODY SWITCH" comment so much as a "do a little more research, or even a community probe could get you better information" comment. The concept of the article wasn't bad just slightly "mis-informative". Especially seeing as how not everything that is open-source and is an OS is linux/unix. Most are linux-like or unix-like (as is the nature of progression.

    As a note for the naysayers, I've used Windows Server, Debian, Gentoo, RedHat, SuSE, ubuntu, FreeBSD, OpenBSD, Solaris and many spin offs of some of those. All of them have their strengths and weaknesses (most notably the flaw of the Windows Server platform would be any machine that loads it - THAT is a biased opinion.)
  • malici0usc0de
    With Ubuntu you can also set it up to silently install them in the background, it just prompts for a password then goes away. I don't know how long Ubuntu has had this but I have been using it as my only OS at home for about 2 years now and have never had a problem with patches. I use XP at work as almost everyone does and I notice it operates almost exactly the same way except it doesn't ask you for a password. So if it works for the less techie MS user base then I don't see why so many problems are occurring with this same basic system running under Linux. sudo apt-get install brain
  • resistance
    The writer of this article has 0% knowledge of _present-day_ GNU/Linux or this article was sponsored by software monopolist.

    in Debian based distros like *ubuntu you can set automatically daily updates without _any_ user intervension and without installing additional software.

    Its a first time I see such badly written article on tomsharware.