Patch Management: Unix
Patches are applied to managed Unix servers on a regular schedule.
This article applies to: Managed Servers
This process involves applying operating system patches directly from the vendor and patches or upgrades to tools or utilities provided locally by the Systems Support group. See the current patch set at http://sysdocs.cit.cornell.edu/twiki/bin/view/Documentation/CurrentPatchSet.
Patching Automation and Scheduling
Given the number of Unix servers we manage and the time-consuming nature of patching, the Systems Support group has employed an automated patching system. This automation allows us greater flexibility when scheduling patch application and provides us with a mechanism for patching systems more efficiently. In rare cases where some unique aspect of a managed Unix server prevents us from using this automation, we will work to have staff available for running through the process by hand.
You can see the patching times for the servers assigned to you on this page: http://sfinfo.cit.cornell.edu/sfinfo_app/htdocs/areamgrpatch.php. The Area Manager can modify the patching times. To add a server to your list, email firstname.lastname@example.org.
Mirrored Boot Environment
Every managed Unix server has two hard drives working together to create a mirrored boot environment. This boot environment does not hold any customer application data but does contain all of the operating system's executables, libraries and system configuration files. One of the disk drives is split off from the boot environment mirror prior to patching, leaving us an untouched copy of the operating system. Operating system patches are then applied to the remaining boot drive and the server is rebooted. Customers are encouraged to provide a list of email contacts, as our patch application process will send out notifications after the patch process finishes to confirm successful patch application.
Post-patching Verification and Backing Out Changes before Re-synchronization
In the event of a patch application failure, the patch process will attempt to recover the system by booting from the drive containing the un-patched, detached mirror. Once the system is back and stable, the customer will be notified and further discussion will be required to determine the cause of the failure and to schedule another attempt for patch application. Barring any patch application failure, the detached mirror is left untouched for 24 hours following patching to allow customers the opportunity to test their application in the updated environment. Unless instructed otherwise, the split-off mirror will be added back into the boot environment and re-synchronized. Once this synchronization occurs, customers have limited, time-consuming options for "backing out" the patch set, so it's important to verify application functionality during our default 24-hour window.
You can request to recover your server from the detached mirror, or if testing is taking longer than expected, you may ask for the mirror to remain detached longer than our standard 24-hour period. Send e-mail within 24 hours to email@example.com with the subject line: ServerName Do Not Re-synchronize. A member of our Systems Administration team will contact you to confirm the next steps. In cases where you make this decision after business hours or dangerously close to the end of our 24-hour window, you are welcome to contact the NOC at 255-9900 and have them confirm your request with our on-call administrator.
If you have any questions, send e-mail to firstname.lastname@example.org.
For CIT Customers Only: Patching and the CIT Change Advisory Board (CCAB)
CCAB announcements about patches are the responsibility of the service owner. If you determine that this OS patching maintenance should be included for the CCAB process, please submit a change request in Remedy. The usual lead time for CCAB submissions is one week before the maintenance is scheduled.