TruCluster Server High Availability Case Study Patching Guidelines and Practices

• The factors involved in deciding to patch
• A recommended patching process
• How to find the patch kit that applies to your Tru64 UNIX operating system

These guidelines do not describe how to install a patch kit. Patch kit installation is a function of dupatch, a patch management utility that uses setld to install a patch kit on the Tru64 UNIX operating system. Each patch kit is specific to a system version and contains the patches appropriate to that version. The patch kit includes a copy of dupatch tailored for that patch kit and the user documentation describing how to run dupatch.

HP Tru64 UNIX engineering provides patch kits that correct critical software problems. Patch kits sometimes contain new system or product functions; however, to add functions, you should upgrade your Tru64 UNIX operating system or TruCluster Software product to the latest version.

To successfully patch a Tru64 UNIX system, you must meet the following requirements:

Deciding to Patch

When and how to patch the Tru64 UNIX operating system and TruCluster product depends on how you use your system, the operations your system performs, and the level of risk you and your environment can accept. The following sections provide a context within which you can think about your approach to patching and make recommendations for a patching process. The sections also describe the several ways in which you can obtain a patch kit from HP Computer Corporation.

Weighing the Risks of Patching

Patches began as reactive fixes to system or product defects. Typically, a customer encounters a problem, informs the vendor, and the vendor develops and sends a patch. In some cases, the patch has already been developed, but the customer receives it only upon request.

As system support became formal, customers received defect and patch notifications, which led some customers to apply proactive patches to their systems to prevent any problem from occurring.

Installing a patch changes the operating system or software product. The change corrects a functional defect, but it requires you to take the system out of operation while you install the patch and includes the risk of unplanned down time, a reduction in system performance, or differences in how functions interact.

We provide patch kits that we have tested on specific system versions. Installing an HP patch kit reduces the risk of system performance reductions and functional interactions.

You must balance the risk of making a change and the risk of continuing with the defect. Our approach to managing the risk of patching the production-level cluster includes:

• A patching process that includes installation of the complete patch kit
• A test bed for measuring the impact of patch kits
• System backups and patch reversibility to preserve the pre-patched environment

A Recommended Patching Process

Apply patch kits in the context of a well-defined and consistent system management process that includes:

• A plan that defines the patching strategy and a schedule for its implementation.
• Identification of, and access to, the appropriate patch kit. HP provides patch kits from the Services Software Patches Web Site or on the Patch Kit CD-ROM as described in Finding the Correct Patch Kit.
• An analysis to determine whether the required patches have dependencies on other patches or if there are any conflicts among patches. Because we perform pre-release testing on the aggregate patch kit, this will not be an issue if you install the complete patch kit.
• A review of any system customizations or manually installed patches that can be undone by a patch kit installation. To prevent problems, you should back up your system prior to patching, select installation of reversible patches, and, if appropriate, baseline your system as described in the Tru64 UNIX and TruCluster Software Product Patch Kit Installation Instructions included with the patch kit.
• A review of the Patch Summary and Release Notes and any special instructions included with the patch kit.
• Identification of the storage space you require to back up your system and install a reversible patch kit, as described in the section, System Backups and Reversing Patches.
• Assigned roles and responsibilities for installing and testing the patch kit in a test bed, logging any system changes, and maintaining contingency plans.
• Documented procedures for installing patch kits in the production environment.

HP Service Contracts

A Service Contract creates a partnership between you and HP, and gives you access to expertise that supplements your in-house resources. For example, your service representative can advise you on patch kit installation and patch maintenance requirements.

We provide support packages that vary depending on geography. In North America, HP offers four level of support (Bronze, Silver, Gold, and Platinum). For more information, refer to the HP Software Support Services Web site at the following URL:

http://www.hp.com/hps/software/

Test Bed

A test bed is a non-production environment that resembles the hardware, software, firmware, and applications found on your production system or cluster. At a minimum, your test bed should approximate your production system and run your business critical applications to the degree that you can run the patched environment for a period of time under simulated loads before you deploy the patch kit to the production environment.

For information on the value and use of a test bed and a description of the test bed hardware we use with our production-level cluster, see the TruCluster High Availability Test Bed page.

The following procedure describes how patch kits are installed on our TruCluster High Availability production-level server:

1. Measure performance on the production and test bed environments prior to patching.
2. Use the dupatch utility to install the patch kit on the test bed, document each step, and record the down time.
3. Test critical business applications in the patched configuration for at least several days. Also, perform standard and automated tests that exercise the subsystems that were patched. When possible, run stress tests, such as production environment simulations, that test the business application in the patched configuration.
4. Re-measure test bed performance and compare it against the initial measurement. If satisfactory, use the down-time records to schedule the deployment of the patch kit to the production environment.
5. Using dupatch and the documented steps from the test bed installation, install the patch kit on the production system.

System Backups and Reversing Patches

Applying a patch kit alters operating system and product code, and backups provide a way to recover from any problems that alteration might cause. Before you install a patch kit, we recommend that you back up your system. At a minimum, back up the root (/), /usr, and /var file systems.

The dupatch utility installs reversible patches by default. We recommend that you accept the default; thus, if something goes wrong with the installation, you can return the system to its prior state.

When dupatch installs reversible patches, it compresses and saves a copy of the files that will be patched. If you then install the patch kit, but later go back and use dupatch to delete the patches, the utility reinstates the saved, pre-patch, files.

If you install non-reversible patches and then attempt to use dupatch to delete them, dupatch disallows the delete operation because it is unable to return the system to its original state.

By default, the reversible files are saved in /var/adm/patch/backup. See the Tru64 UNIX and TruCluster Software Products Patch Kit Installation Instructions included with each patch kit for a description of dupatch and reversibility.

Finding the Correct Patch Kit

How do you know that you have a problem and how do you know there is a patch kit that fixes it? If you have a HP Service Contract, the answer is simple. You can call a service representative, describe your situation, and discuss the availability of a patch kit and the advisability of installing it.

Alternatively, you can link to the HP Services Software Patches Web Site.

http://www.support.compaq.com/patches/unix/

Whether you have a service contract or not, you can access the Web site and:

• Obtain pricing and ordering information for the Patch Kit CD-ROM described in CD-ROM Access to Patch Kits.
• Download the latest patch kit, specific to your Tru64 UNIX operating system version, as described in Web Site Access to Patch Kits.
• Link to the "Search Patches" page, select Tru64 UNIX, and obtain a list of all the released patch kits. The list links you to README files that describes the date, size, and reason for each patch in the patch kit.

You can link to the HP Services Software Patches Web Site and search through the patch kits released for your operating system version. The system version indexes contain README files that describe current problems and solutions.

The patch kit you obtain, whether from the Web site or from the Patch CD-ROM, contains patches and Release Notes that are specific to that version of the operating system and software product. The patch kit also contains dupatch, the patch management tool.

Selecting the Correct Patch Kit

A patch kit is a collection of all current released patches that we have tested, in the aggregate, on the operating system.

Because patch kits are created and tested for specific versions, you must make sure you install the correct patch kit, whether it is a kit you download from the Web site or install from the CD-ROM. To help you find the patches appropriate for your operating system version, consider the naming convention for patch kits:

os or product | version | kit type | kit no. | mfg date | .filetype

For example, the patch kit t64v50as0002-20000324.tar is:

• t64 — Tru64 UNIX operating system (some versions of Tru64 UNIX may have the DU label).
• v50 — Version 5.0.
• as — Aggregate Selective, the cumulative patch kit for this version.
• 0002 — The kit number. This is the second patch kit issued for this version.
• 20000324 — Manufactured on March 24, 2000.
• .tar — A tar file.

The operating system version public index on the HP Services Software Patches Web Site can contain other kit types in addition to the Aggregate Selective patch kit. We recommend that you download the Aggregate Selective patch kit appropriate for your operating system version.

Also, patch kits are placed on the HP Services Software Patches Web Site as soon as they are approved for distribution while patch kit CD-ROMs are issued quarterly. Thus, patch kits on the Web may be more current than those on the CD-ROM. We recommend that you compare the patch kit file name for your operating system version as listed on the Web site with the patch kit file name for your operating system version as listed in the 00-READ-ME-FIRST.txt file on the CD-ROM. If the file name on the Web site is more recent, download and install the patch kit from the Web site.

For example, the patch kit file name for the Tru64 UNIX Version 4.0D Patch Kit on the December 1999 Patch CD-ROM is: DUV40DAS0005-19991007. The patch kit file name for the Tru64 UNIX Version 4.0D Patch Kit on the HP Services Software Patches Web Site (May 10, 2000) was: DUV40DAS0006-20000209. In this case, you download and install the patch kit from the Web Site.

After you identify the most recent patch kit appropriate to your operating system, compare it to the patch kits currently installed on your operating system to ensure that you are installing the most up-to-date kit.

If you have patch kits installed on your system, examine the /var/adm/patch/log/event.log file, which records the date, patch kit name, and patches installed. The installation information is appended to the log, so examine the bottom of the file for the most recently installed patch kit. Compare the last installed patch kit name with the patch kit name on the CD-ROM or the Web site to determine if the installed patch kit is current.

When you run the dupatch utility to install the patch kit, the dialogue prompts you to specify whether you want to install all of the patches on the patch kit. Because the patches have been tested and verified as a package, accept the dupatch default to install the complete patch kit.

CD-ROM Access to Patch Kits

HP issues a Patch CD-ROM four times a year. Each CD-ROM contains all the release patches for the previous 3 months that are available to North American customers. To order the Patch CD-ROM from HP, call 1-800-752-0900.

Because of encryption restrictions, customers outside of North America must contact a local HP sales office or authorized reseller for pricing and ordering information.

The patch kits on the CD-ROM are organized by Tru64 UNIX operating system and TruCluster Software Product version. The CD-ROM also contains a READ-ME-FIRST.txt file listing the patches and directing you to the patch kit appropriate to your system and product version.

The CD-ROM packaging contains instructions for mounting the media and viewing the READ-ME-FIRST.txt file.

Unlike the Web-based patch kit tar file, the CD-ROM patch kits are expanded and may be more convenient to use.

Web Site Access to Patch Kits

Patch kits are available from the HP Services Software Patches Web site after HP tests and approves them for release. Thus, the Web site provides you with the earliest access to patch kits.

To access the HP Services Software Patches Web Site, link to the following URL:

http://www.support.compaq.com/patches/

If you have an HP Software Services Contract and an assigned ftp user name and password, you can use ftp to access all of the released patch kits available to North American customers. For information on obtaining an ftp user name and password, select contract access from the Associated Links menu of the HP Services Software Patches Web Site.

If you do not have a valid HP Software Services Contract, you can use the Web site to access the public patch indexes, which include all the released patch kits available to North American customers that are not restricted for license, export, or business reasons.

As with the patch kit CD-ROM, Internet access to patches is restricted to North American customers. If you are not a North American customer, you must contact your local HP sales office or authorized reseller for pricing and ordering information.

To access the released public patch kits from the HP Services Software Patches Web Site:

• From the Associated Links menu, select browse patch tree.
• From the public patch index, select the appropriate operating system (/Digital_UNIX). (The /Digital_UNIX directory contains patches for Digital UNIX and Tru64 UNIX operating systems.)
• From the operating system index, select the appropriate version number (for example, /v5.0).

Each operating system index contains the released patch kits for that operating system version. Each operating system index also contains the following documents:

• 00-READ-ME-FIRST — a text file that describes preinstallation steps.
• PatchInstallGuide — an HTML and PDF version of the Tru64 UNIX and TruCluster Software Products Patch Kit Installation Instructions that describes the use of dupatch to install the patch kit.
• ReleaseNotes — an HTML and PDF version of the Patch Summary and Release Notes for this version of the operating system and for TruCluster Software Products.

From the operating system index, select the tar file for the appropriate patch kit and use ftp to download it to your Tru64 UNIX system or cluster. You can use the patch kit number (explained in Selecting the Correct Patch Kit) to identify the patch kit appropriate to your product. You can also use the Web browser on your UNIX system to download the tar file. Keep in mind that a patch kit tar file can be quite large; for example, the patch kit for Version 4.0D issued February 25, 2000 (duv40fas0003-20000225.tar) is 64.1 MB compressed. The amount of time required to download the tar file depends on the speed of your network connection.

Each patch kit contains the following files (where kit_number is the patch kit Identifier):

• kit_number.CHKSUM — A file containing a checksum used to verify the data in the tar file.
• kit_number.CONTENTS — A list of the patch kit's documentation files and their paths.
• kit_number.README — A description of the patch kit, including a kit summary, installation notes, a list of superseded patches, and any known problems.
• kit_number.html — An HTML version of .README.
• kit_number.pdf — A PDF version of the Patch Summary and Release Notes for this patch kit. The Release Notes summarize each patch in the patch kit.
• kit_number.tar — The patch kit tar file that you ftp to your Tru64 UNIX machine. Use binary ftp to transfer the file to your UNIX machine and the tar command to expand the file.

  A typical patch kit tar file expands to the following files and directories:

  • ReadMeFirst — A text document that describes preinstallation steps.
  • PatchInstallGuide — An HTML version of Tru64 UNIX and TruCluster Software Products Patch Kit Installation Instructions that describes how to run the dupatch utility.
  • dupatch — Used to update the dupatch utility and patch tools, if necessary.
  • The version-specific operating system patch directory — Contains /doc and /kit subdirectories. The /doc subdirectory contains documentation, in various forms, describing each patch. The /kit subdirectory contains the install scripts and source files that dupatch and setld use to install the patch kit.

    In addition to the /kit and /doc subdirectories, the operating system directory may also contain a document named Special_Instructions.txt that contains patch-specific information. If this document is included, make sure you read it before installing the patch kit.

  • The version-specific TruCluster Software Product directory — Similar to the operating system directory.

  • A patch _tools directory — Contains up-to-date versions of dupatch and other patch tools.

To expand tar files from the HP Services Software Patches Web Site on any NFS-mountable file system, we recommend that you create a temporary directory, copy the tar file to that directory, and expand the file.

After you have expanded the tar file, use the mount command to make the temporary patch directory available to the system you are patching.

Patch kits are specific to operating system versions. Also, any upgrades to the dupatch utility are included as part of the patch kit and are incorporated when you install the patch kit. Because the Tru64 UNIX and TruCluster Software Products Patch Kit Installation Instructions, which is also included with each patch kit, is an up-to-date description of the patch kit and the dupatch utility, refer to that document for information on installing the patch kit.

Return to TruCluster Server High Availability Case Study.