[AUUG-Talk]: Patch mgmt approaches?

[Stuart Remphrey]

Hi David,

Hmm, haven't tried the Sun N1 stuff yet, might be worth a look.

We've used the earlier Sun CST stuff a bit * pretty web GUI,
and we could query from a script, but only for [any rev|a specific rev]
of a requested pkg/patch, not "what's the latest rev of this patch".
Still installed, summarises install/change dates etc, but we don't
use it much (in fact I'd forgotten about it).


We've not had any problem with Sun patches for quite a while,
apart from a few systems affected by the SSH patch debacle
(holding off on the rest at the moment). I definitely relate to
the problem getting access and/or being able to change
corporate dev or prod systems! Some we need a fortnight's
notice to reboot (unless it's urgent, like already down).

There have been a couple of cases where we've had downtime
due to a SAN driver bug for which there has been a patch
already available, but hadn't gotten far enough through
the process to get it onto the affected systems in time :-(


Gradually moving from "if it hasn't broke don't fix it" closer to
"just because we haven't hit this known bug/patch yet doesn't
mean we won't".

Rgds, Stuart.


>>> 
From: 	David J N Begley <d.begley at auug.org.au>
To:	Stuart Remphrey <stuart.remphrey at rmit.edu.au>
Date: 	18-Oct-06 1:54 am
Subject: 	Re: [AUUG-Talk]: Patch mgmt approaches?

Quoting Stuart Remphrey <stuart.remphrey at rmit.edu.au>:

> Along the lines of the last "what resources do you use" survey,
> anyone like to contribute thoughts on pkg/patch mgmt on Unices?
> Good tools or other resources, pet hates, policy/planning...

For my Solaris (9) boxen (x14 - I leave the rest to the Enterprise  
Team as long as they leave my boxes to me), I must admit to being  
somewhat slack with patches simply because I've been bitten too often  
with Sun patches that either trample all over local configs (eg.,  
restarting services that were disabled) or introduce new bugs that  
completely screw-up production applications (eg., the on-going SunSSH  
debacle).  It's actually easier to run third-party software and  
rebuild it from source, which is rather sad.

That being the case, I may go 6-12 months before applying the latest  
Recommended Patch Cluster, though steps are taken to protect the  
machine (from both external and internal threats) in the interim.   
There are exceptions when something is definitely broken and requires  
patching (eg., during the switch to GCCFSS or yet more SunSSH problems).

Every month an old script runs to compare our patch levels to those in  
ye olde patch xref file and mails a report so we can see just how far  
behind we've gone.  I haven't gotten around to changing the patch  
regime according to SunSolve's New Patch Order.

The Enterprise Team are trying to use a more sophisticated method with  
Sun N1 SPS (or whatever the heck it's called this week) that allows  
for centralised monitoring of changes and application of patches, but  
I think they're only really enjoying success with this on new  
deployments of Solaris 10 - most Solaris 9 or older hosts are proving  
too troublesome.

For the few production (SuSE) Linux boxen I admin (x2 critical, x1  
partial), it's a hell of a lot easier (and I've never encountered the  
same sort of problems with patches that I've had with Sun).

Every morning an rsync job keeps our local copy of the patch archive  
updated with any overnight changes in the main mirror in Germany;   
from there we can apply (automatically in some cases, manually in  
others) patches to keep the machines up-to-date to within a few days  
at the most.

> A patch cycle is followed per application group: apply to test
> systems, test, approve, apply *same* to prod and dev; on to next app group.
> So they're in-sync within an application group, as far as hardware allows,
> but not necessarily across disjoint applications.

For Solaris, I have a couple of development machines on which patches  
are first tested (where possible) before deployment to all the others;  
  for SuSE Linux I don't have the same luxury so patches are applied  
directly to the production servers - touch wood, this hasn't burnt me  
(yet!).  The only time a reboot is required is when the kernel itself  
is patched.

The Enterprise Team are in a worse situation (in terms of  
applications, etc.);  they can throw a patch at a desktop workstation  
they use but due to the difficulty in approving downtime of _anything_  
(even development systems), patches often end up going directly on to  
production systems (with mixed results - we once had an E10k domain  
that refused to boot after being patched).  As a result, their  
machines often end up _way_ behind in patches because the apps people  
won't let _anything_ change.

Cheers..