The AIX 5L Service Strategy and Best Practices: AIX 5L Service Scenarios and Tools

IBM Corporation

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

Table of Contents
1 INTRODUCTION ................................................................................................. 4 1.1 NEW TERMS AND CONCEPTS ............................................................................. 5 1.1.1 Technology Level (TL) ............................................................................. 5 1.1.2 Service Pack (SP)..................................................................................... 5 1.1.3 Concluding Service Pack (CSP) ............................................................... 6 1.1.4 Interim Fix (IF) ........................................................................................ 6 2 MAINTENANCE STRATEGY MODELS........................................................... 8 2.1 2.2 2.3 2.4 3 4 5 6 3.1 LATEST LEVEL ................................................................................................. 8 MAXIMUM STABILITY ....................................................................................... 9 YEARLY UPDATE .............................................................................................. 9 MODEL OVERVIEW ......................................................................................... 10 REDUCING THE AMOUNT OF CHANGE BETWEEN TECHNOLOGY LEVELS ............ 11

RELEASE SCHEDULES.................................................................................... 11 SECURITY FIXES.............................................................................................. 12 TESTING OF PTF UPDATES AND TECHNOLOGY LEVELS ..................... 13 AIX 5L PACKAGING AND FIX ARCHITECTURE ....................................... 14 6.1 6.2 PTF................................................................................................................ 14 APAR ............................................................................................................ 14
ALT_DISK_INSTALL......................................................................................... 15 MKSYSB.......................................................................................................... 15 MULTIBOS....................................................................................................... 16

SYSTEM BACKUP AND RECOVERY............................................................. 15 7.1 7.2 7.3

8 9

PREVIEWING UPDATES ................................................................................. 17 TOOLS................................................................................................................. 18 9.1 SUMA ........................................................................................................... 18 9.1.1 Functional Highlights ............................................................................ 18 9.2 COMPARE_REPORT ......................................................................................... 18 9.2.1 Functional Highlights ............................................................................ 19

10

USAGE SCENARIOS ..................................................................................... 20

10.1 OVERVIEW ..................................................................................................... 20 10.1.1 Awareness.............................................................................................. 20

aixseStrategywp020706.doc

Page 2

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

10.1.2 Decision................................................................................................. 20 10.1.3 Implement .............................................................................................. 20 10.1.4 Verify ..................................................................................................... 21 10.2 PROACTIVE MAINTENANCE WITH SUMA ........................................................ 22 10.2.1 Latest Level............................................................................................ 22 10.2.2 Maximum Stability ................................................................................. 23 10.2.3 Yearly Update ........................................................................................ 23 10.3 STANDALONE WITH INTERNET ACCESS.......................................................... 24 10.3.1 Download the latest fixes periodically (using suma command line) ........ 24 10.3.2 Download a specific APAR (using suma SMIT easy menus) ................... 25 10.4 STANDALONE WITH NO INTERNET ACCESS..................................................... 27 10.4.1 Download the latest fixes (using compare_report command line) ........... 27 10.4.2 Download a specific APAR (using the IBM Support Web site) ................ 28 10.5 MORE THAN ONE SYSTEM WITH INTERNET ACCESS ........................................ 29 10.5.1 Download the latest fixes (setup suma on all NIM clients) ...................... 30 10.5.2 Download the latest fixes (setup suma only on NIM master) ................... 30 10.6 MORE THAN ONE SYSTEM WITH NO INTERNET ACCESS ................................... 31 10.6.1 Download the latest fixes (compare_report using inventory from clients)31

aixseStrategywp020706.doc

Page 3

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

Introduction

In response to client feedback related to the frequency of required maintenance, and the desire to maintain a more stable environment for longer periods of time, AIX 5L will change some of its current service strategy directions and institute new release rules. One issue that has made it increasingly difficult to control the amount of change introduced with an update is the increased activity in the service update stream between maintenance levels. Delivering too many fixes in the update stream can cause instability and regression risk to existing function. Delivering fixes only for critical and pervasive problems will help reduce the amount of risk when a service update is applied. Another issue is the amount of change in Maintenance Levels and the frequency with which they are released. Starting in 2006, a Maintenance Level will be referred to as a Technology Level and will only be released twice per year. The first Technology Level of the year will be focused on hardware enablement for new systems and adapters that are being introduced. The second Technology Level, released in the second half of the year, will include both hardware and software features. Having a year between major releases of new software features also allows for more extensive testing of these releases. For clients who wish to install only one major level per year, the concepts of a Service Pack and Concluding Service Pack will be introduced. A Concluding Service Pack will allow support on a Technology Level for up to fourteen months from the date the Technology Level was released. Finally, to address security issues, fixes for security problems will be shipped on multiple Technology Levels, so you will not be required to jump up a level just to patch a security hole. Note: Changes to the Fix Central Web site to reflect the new service strategy will be made in early 2006, when the first Technology Level is released.

aixseStrategywp020706.doc

Page 4

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

1.1

New Terms and Concepts

1.1.1 Technology Level (TL) A Technology Level is the new term for the twice yearly releases which contain new hardware and software features, and service updates. The first TL will be restricted to hardware features and enablement, as well as software service. The second TL will include new hardware features and enablement, software service, and new software features, making it the larger of the two yearly releases. In the past, we have used the term Maintenance Level (ML) even though the release contained code for new features and functions. Technology Level is a more appropriate term for this code. The Technology Level (e.g., 5300-04) will be displayed using the oslevel -r AIX 5L command, just as this command can currently be used to display the Maintenance Level. Starting in 2006, installing a Technology Level should be viewed as an all or nothing operation, meaning that requisites will be added so that the whole Technology Level is installed, and not allow a TL to be partially installed. The reason for this change is to avoid getting into unsupported configurations. Technology/Maintenance Levels are tested as a unit, with minimal testing of individual updates. We believe that most clients install a new level as a unit, and supporting individual PTFs is no longer a focus. It is more of a risk to have only half of a Technology Level installed than it is to install the entire Technology Level, and as a result, install some additional updates that you feel you might not need. It is preferred to have systems that are at a known tested level. We do not recommend attempting to reject a Technology Level once it has been applied to an AIX 5L system. Since Technology Levels are usually large (with respect to the amount of code shipped) it is faster and less risky to fall back to the previous level using other methods. To fall back with a reboot, use alt_disk_install or the new multibos function shipped with 5300-03. To fall back with a restore, create a backup (mksysb) to NIM or bootable media (CD, DVD, or tape) before applying the Technology Level. Having a good recovery plan in place is critical, especially when dealing with maintenance windows on production systems. As with any new technology, careful testing should be performed before putting a new Technology Level into production. 1.1.2 Service Pack (SP) The Service Pack concept will allow service-only updates (as known as PTFs) that are released between Technology Levels to be grouped together for easier identification. These fixes will be for highly pervasive, critical, or security related issues. Service Packs will be provided for the N and N-1 releases (e.g., V5.3 and V5.2) on the latest Technology Level for each release (e.g. 5300-04 and 5200-08), and they will be released approximately every 4-6 weeks after the release of a Technology Level. Applying and rejecting an individual service update (PTF) is still a supported and recommended method of removing an update if there is a problem or a regression after it is installed. Since Service Packs can also be rejected, it is recommended that before applying a Service Pack or PTF update, that all other updates on the system are
aixseStrategywp020706.doc Page 5

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

committed (put in the commit state) to allow for easy identification of the Service Pack updates. The commit operation (fastpath smitty commit) will remove saved files and free up space in the /usr file system. It is also highly recommended that system administrators always preview an update (apply or commit) operation to ensure all requisites are met before performing the actual operation. Set the PREVIEW only? field in SMIT to yes to perform a preview operation once the updates for apply or reject have been selected. Applying the latest level of available updates will move the system to the latest Service Pack. To see which Service Pack is currently installed, run the oslevel -s command. Sample output for a V5.3 system, with Technology Level 4, and Service Pack 2 installed would be: oslevel s
5300-04-02

Service Packs allow easier identification of a systems current level. Instead of referring to a systems level as being at 5300-04 plus 15 PTFs, the system can be referenced to be at the 5300-04-02 level. The oslevel -s command also has options to list installed filesets that are less than or greater than the level contained within a Service Pack so additional PTFs that have been applied can be easily identified. In addition, Service Packs are cumulative, so if Service Pack 3 is applied, all of the previous critical fixes from Service Packs 1 and 2 will also be applied. Critical Fix Packs will no longer be shipped, since all of the updates that are contained in a Service Pack have been deemed critical or pervasive. 1.1.3 Concluding Service Pack (CSP) Concluding Service Pack is the term that will identify the last Service Pack on a Technology Level. The CSP will contain fixes for highly pervasive, critical, or security related issues, just like an SP, but it may also contain fixes from the newly released Technology Level that fall into these categories. Therefore, a CSP will contain a very small subset of service that was just released as a part of a new Technology Level. The Concluding Service Pack will be available shortly after a new Technology Level is released. For example (dates may change), if Technology Level 5300-04 is released in February of 2006, the CSP for the previous release, 5300-03, will be available approximately 4-8 weeks later. It will have a specific level designation of CSP. For example, the output of running the command oslevel -s would return 5300-03-CSP. Concluding Service Packs will allow for extended service on a Technology Level through the utilization of Interim Fixes. 1.1.4 Interim Fix (IF) The term Interim Fix is used in AIX 5L as a replacement for emergency fix or efix in order to simplify terminology across IBM and not cause confusion when dealing with

aixseStrategywp020706.doc

Page 6

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

other products. While the term emergency fix is still applicable in some situations (a fix given in the middle of the night with minimal testing), the term Interim Fix is more descriptive in that it implies a temporary state until an update can be applied that has been through more extensive testing. Interim fixes that address non-security related issues will be provided for the two most recent supported releases (e.g., V5.3 and V5.2) on the last two Technology Levels for each release. All interim fixes are applied using the emgr tool (also known as fixmgr). The emgr tool has many benefits, including: Standard way to apply all fixes. In the past, applying a fix or patch meant moving and replacing files manually. The emgr command will save and replace files, and has other options, such as over mounting, to allow for testing. Tracks fixes applied on a system. In the past, there was no way to tell if an efix (interim fix) was applied. Now, the emgr l command will list all of the interim fixes on a system. No more overwriting fixes with service updates (PTFs). The emgr command will lock a fileset and not allow an update to be applied that may overwrite an interim fix. For more information on the emgr command, see its man page (man emgr) or visit the pSeries and AIX Information Center and search on Interim Fix Management Solution.

aixseStrategywp020706.doc

Page 7

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

Maintenance Strategy Models

The type of maintenance strategy that you adopt depends on the timeliness and frequency that you choose to move to the next Technology Level. This section describes different options that may be considered. As described earlier in this document, there will be only two AIX 5L releases per year. The examples below depict this by showing first half (e.g., 1H06) and second half releases (e.g., 2H06). When looking at the AIX Releases line below, the examples show a 1H06 release of Technology Level 4 (TL4, also referred to as 5300-04) and a 2H06 release of Technology Level 5 (TL5 or 5300-05). Roughly 4-8 weeks after the release of a TL, the Concluding Service Pack (CSP) for the previous TL will be released. For example, CSP4, the CSP for TL4, will be released approximately 4-8 weeks after TL5 is released. In between Technology Levels, one or more Service Packs (SPs), a tested group of PTFs, will be released in support of the current Technology Level.

2.1

Latest Level

The Latest Level line shown below depicts a maintenance strategy model for clients who are upgrading to hardware that requires a new Technology Level, or who wish to utilize new hardware or software features being introduced in a TL. This model would typically entail a move to a new TL shortly after its release. The first half Technology Level (e.g., TL4, TL6, etc.) is a smaller TL since it is restricted to hardware features and enablement, and software service. The second half TL (e.g., TL5) also includes new software features, and thus will be a larger release. In between Technology Levels, Service Packs and interim fixes (to address security and other issues) can be utilized in support of the current Technology Level.
Release Date AIX Releases 1H06 2H06 1H07 ...

TL4

CSP3

SPs

TL5

CSP4

SPs

TL6

CSP5

Latest Level

TL4

SPs

TL5

SPs

TL6

...

aixseStrategywp020706.doc

Page 8

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

2.2

Maximum Stability

Not all clients will move to a new Technology Level when it is released. For those wishing to delay their adoption of a new TL by a period of six to eight months, and take advantage of the latest tested group of PTFs available in a Concluding Service Pack, the Maximum Stability model may be appropriate. In between Technology Levels, interim fixes (to address security and other issues) can be utilized in support of the Technology Level that the system is running.
Release Date AIX Releases 1H06 2H06 1H07 ...

TL4

CSP3

SPs

TL5

CSP4

SPs

TL6

CSP5

Maximum Stability

ML3 CSP3

IF

IF

TL4 CSP4

IF

IF

TL5 CSP5

...

2.3

Yearly Update

For clients who are stable and wish to stay on a Technology Level for up to a year, they may choose to adopt the Yearly Update model. This model shows an annual move to a new Technology Level, and thereby skips the direct move to one of the two TLs that are released during the year, instead picking up this function when the next TL is installed, as TLs are cumulative. In between Technology Levels, Service Packs (a tested group of PTFs), individual PTFs, and interim fixes (to address security and other issues) can be utilized to maintain service on a Technology Level for up to one year. When moving to a new TL annually, it is recommended to utilize a first half TL (e.g., TL4, TL6, etc.) because it has the advantage of being a smaller release. When installing a first half TL (e.g. TL6) it will contain all the new AIX 5L software enhancements that were released with the previous TL (e.g. TL5), however TL5 will have already been in the field for six to eight months, allowing it to become more stable.
Release Date AIX Releases 1H06 2H06 1H07 ...

TL4

CSP3

SPs

TL5

CSP4

SPs

TL6

CSP5

Yearly Update

TL4

SPs

PTFs

CSP4

IF

TL6
Page 9

...

aixseStrategywp020706.doc

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

2.4

Model Overview

The diagram in this section shows a comparative view of the three models described. In general, clients do not need to move to the latest Technology Level when it is released. The exception to this are clients who are upgrading hardware that requires the new TL, or want to use new hardware or software features introduced in a TL. If a client is stable, they may choose to remain on a Technology Level as long as possible. Depending on when the TL is released will indicate the total length of time it will be supported. If a Concluding Service Pack is applied, then service will be offered via interim fixes so that clients are not required to move to a new Technology Level. This will give clients a maintenance only stream for 6-8 months after the release of a CSP. The frequency of your maintenance window will determine how often you move to a Technology Level. Some clients may choose to move to the Technology Level that is released in the first half of the year (e.g. TL4, TL6, etc.) on a yearly basis since it will have a considerably smaller amount of change than the second half Technology Level. And, by choosing to apply the first half TL as part of a yearly maintenance strategy, this also allows the larger second half TL an additional six months of field exposure and testing before utilizing the new enhancements it contains. These models can be modified to suit specific client environments.
Release Date AIX Releases 1H06 2H06 1H07

TL4

CSP3

SPs

TL5

CSP4

SPs

TL6

CSP5

...

Latest Level

TL4

SPs

TL5

SPs

TL6

...

Maximum Stability

ML3 CSP3

IF

IF

TL4 CSP4

IF

IF

TL5 CSP5

...

Yearly Update

TL4

SPs

PTFs

CSP4

IF

TL6

...

aixseStrategywp020706.doc

Page 10

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

Release Schedules

There will be two AIX 5L Technology Levels each year on the N and N-1 levels, with N being the latest release (for example, AIX 5L V5.3) and N-1 being the previous release (AIX 5L V5.2). While the N release will always be fully enhanced, over time the N-1 release will see less and less of hardware enablement and software features (much like what has happened with AIX 5L V5.1 today). Fixes will still be produced and shipped for the N-1 release, but over time it will not be enhanced with new hardware enablement and software features. Technology Levels will be approximately six months apart, giving additional time for testing, especially for the larger Technology Level that is released in the second half of the year.

3.1

Reducing the Amount of Change between Technology Levels

As we have seen in the past, changing too much code with a limited time to test can be risky. When a change is necessary to resolve a system crash or security problem, the fix needs to be made available quickly. In such cases, an interim fix is initially released, and later, after additional testing, the fix becomes available in a PTF update. However, neither interim fixes nor service updates receive the level of testing of a Technology Level. Therefore, to improve stability between Technology Levels, and to reduce the amount of change brought in by individual fixes, each APAR will now be closely scrutinized to decide if it meets the criteria of having a PTF update provided to address the issue. If an APAR does not meet the criteria of being fixed in a service PTF, it will be fixed in the next Technology Level, allowing it to go through a full integration test cycle.

aixseStrategywp020706.doc

Page 11

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

Security Fixes

AIX 5L has always produced emergency fixes for security problems. These fixes usually go out with a CERT advisory for the three supported releases (e.g., V5.3, V5.2 and V5.1) on their latest Maintenance Level. This process will change so that security fixes will now be shipped as interim fixes for the last three Technology Levels on the two most recent supported releases (e.g., V5.3 and V5.2), thereby allowing clients to install a fix without updating to the latest Technology Level. Since the oldest supported release (e.g., V5.1) does not change as often, security fixes will only be provided for the latest TL. For example, if a security problem was reported at the end of 2006, interim fixes would be made available for 5300-03, 5300-04, 5300-05, 5200-07, 5200-08, 5200-09 and the latest level of V5.1. In some cases, depending on the code affected, you will need to update to a CSP or latest level of a fileset for that component. Also, the same interim fix may be appropriate for all levels of a release if the code has not changed since it was initially released. Interim fixes for non-security related issues will be provided for the two most recent supported releases (e.g., V5.3 and V5.2) on the last two Technology Levels for each release, and on the latest level for the oldest supported release (e.g., V5.1).

aixseStrategywp020706.doc

Page 12

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

Testing of PTF updates and Technology Levels

The faster an update or fix is released, the less testing it is going to get. Some interim fixes, like security problems, will go thru more extensive regression testing because they will be released to the general public. Other interim fixes will receive testing that is more appropriate for the code change and severity of the problem. Your service representative will be able to tell you the amount testing the interim fix has undergone. In some cases, the developer may choose to send the interim fix thru the Functional Verification Test (FVT) suite before sending it out, if there is time and if this additional testing is deemed necessary based on the scope of the change and the severity of the problem. There are four labs that perform different levels of testing: FVT lab Functional Verification Test will test the component for any regressions. It does not (in most cases) include integrated testing with other components. This is appropriate for a change that will only affect a single component, or have limited interaction with other components. ART lab This level of testing is required for all PTF updates that are shipped. PTF updates go thru the ART lab (APAR Regression Test lab) and receive a longer period of test time and integrated testing with thousands of test cases. PTF updates also receive more integrated testing on different types of hardware made available in the ART lab. Security Fix lab Runs a subset of ART lab tests on interim fixes for security problems. System Test The System Test lab performs testing on each new Technology Level for the N and N-1 level of AIX 5L (for example, V5.3 and V5.2). Because Technology Levels are only released twice a year, this gives changes more time to be tested with other fixes. Technology Levels also receive additional testing with applications and are tested on the latest level of hardware and firmware, which is usually required for new systems or upgrades. System Test takes over four months with each Technology Level.

aixseStrategywp020706.doc

Page 13

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

AIX 5L Packaging and Fix Architecture

AIX 5L packaging architecture divides the system into filesets, each of which contains a group of logically related client deliverable files. For example, files required for TCP/IP client functionality are packaged in the bos.net.tcp.client fileset, where files required for TCP/IP server functionality are packaged in the bos.net.tcp.server fileset. Each fileset can be separately installed and updated, allowing for more granular installation and smaller update packages. Changes to each fileset are tracked by a Version, Release, Maintenance, and Fix level, referred to as a VRMF (e.g., bos.rte 5.3.0.0 which maps to Version 5, Release 3, Maintenance level 0, and Fix level 0). In addition to filesets in installp format, other installers are supported to allow installation of other products and packages on AIX 5L. The InstallShield installer adds entries to the same native database that installp uses, while the rpm installer has its own separate database to store what has been installed. Some AIX 5L commands display information that combines data from both databases. For example, lslpp L will display all the products that have been installed on the system by the installp, rpm and InstallShield installers. Filesets can have relationships to one another called requisites. For example, if fileset B needs fileset A to function, a requisite would be added requiring that fileset A is installed along with fileset B. The lslpp p command can be used to list requisites for installed filesets.

6.1

PTF

A PTF (Program Temporary Fix) is a fileset update at a specific VRMF level. PTF numbers are only used for the purpose of distribution, and are not tracked in the AIX 5L software vital product database. Installed filesets and their VRMF levels are tracked by AIX 5L.

6.2

APAR

An APAR (Authorized Program Analysis Report) is a description of a problem and its resolution. An APAR fix can involve one or more fileset updates (PTFs). Some APARs are classified for certain conditions. These special classifications include: Security - those related to Security issues. It is recommended that these be installed immediately. HIPER/PE - those marked as HIPER (High Impact and/or Pervasive) or PE (PTFs in Error a PTF that causes a regression of previous existing function). It is recommended that these be installed immediately if you utilize the function affected by the problem. Although not all PEs are critical in nature, their installation is recommended so the regression does not cause additional problems.

aixseStrategywp020706.doc

Page 14

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

System Backup and Recovery

Having a backup and recovery plan in place is one of the most important aspects of system administration. Things can go wrong, everything from power outages and disk drive failures, to a root user running commands in the wrong directory. Performing system maintenance is one of the times that you want to be sure you have a reliable and fast recovery scenario in place. Introducing any change to a system is a risk, and some changes can be easier to back out than others. For example, the emgr -m (mount) option can be used when applying an interim fix, meaning that the interim fix files are mounted over the target files instead of replacing them. This process of applying a fix is very useful for special situations such as installing an interim fix containing libc.a, because once executables run and link to libc.a it may be difficult to remove the interim fix and return to the original libc.a if problems are encountered. Utilizing the emgr mount option allows a system reboot to restore the original files automatically since a reboot will unmount all existing mounted files and only mount what is specified in /etc/filesystems when the system is restarted. Most files can be easily rolled back, since interim fixes and PTF updates are created to allow files to be easily restored. So even though it is recommended to have a backup scenario in place, the first option for recovery when applying a PTF update or an interim fix should be to reject the update or to remove the interim fix. Since the reject operation is not recommended for Technology Levels, a recovery plan should be in place in case problems are encountered when upgrading to a new TL. The AIX 5L alt_disk_install, mksysb, and multibos commands are also very useful for backup and recovery planning.

7.1

alt_disk_install

The alt_disk_install command allows users a way to update the operating system to the next release or Technology Level without taking the machine down for an extended period of time. This can be done in two ways: by installing a mksysb image on a separate disk, or by cloning the current system and then applying updates to get to the next Technology Level on a separate disk. If a problem is encountered with the new level, the bootlist command can be run after the new disk has been booted, and the bootlist can be changed to boot back to the original disk in order to get the system back to the original level.

7.2

mksysb

The mksysb command creates a backup of the operating system (specifically, the root volume group). You can use this backup to reinstall a system to its original state after it has been corrupted. If you create the backup on tape, the tape is bootable and includes the installation programs needed to install from the backup.
aixseStrategywp020706.doc Page 15

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

7.3

multibos

Beginning with AIX 5L 5300-03, the multibos utility allows the root level administrator to create and maintain two bootable instances of the AIX 5L Base Operating System (BOS) within the same root volume group (rootvg). This utility is provided primarily as an upgrade vehicle. The multibos utility allows the administrator to access, install maintenance, update, and customize the standby instance of BOS (during setup or in subsequent customization operations) without affecting production on the running instance. Migration to later releases of AIX 5L will be supported when they are available, so that will be a future option to keep in mind. The file systems /, /usr, /var, /opt, and /home, along with the boot logical volume must exist privately in each instance of BOS. The administrator has the ability to share or keep private all other data in the rootvg. As a general rule, shared data should be limited to file systems and logical volumes containing data not affected by an upgrade or modification of private data. When updating the non-running BOS instance, it is best to first update the running BOS instance with the latest available version of multibos (which is in the bos.rte.bosinst fileset).

aixseStrategywp020706.doc

Page 16

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

Previewing Updates

When PTF updates are downloaded from the IBM Support Web site (Fix Central) or using SUMA (Service Update Management Assistant), or retrieved from media, you should always perform a preview installation before attempting any software install. The preview option is available from SMIT, WebSM, and using the install_all_updates command (with the -p option). The preview option will verify that all of the updates are available, including any requisite updates. While installp will always attempt to apply as many updates as it can, it is not recommended to install any updates if everything is not in place for a successful install, especially with Technology Levels. Keeping updates in separate repositories (or LPP_SOURCES for NIM) is a good idea if you only want to apply specific updates to your machines. If you keep all of your updates in one repository, you may get warnings about superseded updates when trying to apply specific fixes. The lppmgr command can help maintain your repository by cleaning up superseded updates.

aixseStrategywp020706.doc

Page 17

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

Tools

There are numerous tools available to assist with keeping current software updates on your system.

9.1

SUMA

As system administration becomes more time consuming and complex, it is often a roadblock that prevents maintaining systems with current software fixes. The Service Update Management Assistant (SUMA) moves the system administrator away from the manual task of retrieving maintenance updates from the Web. SUMA provides clients with flexible, task-based options allowing them to perform unattended downloads of AIX 5L software updates from the IBM Support Web site, thereby allowing clients to move toward an automatic maintenance strategy which helps reduce the time spent on system administration. The SUMA AIX 5L Web site contains a technical white paper with a high-level overview of SUMA functionality (http://www.ibm.com/servers/AIX/whitepapers/suma.html). 9.1.1 Functional Highlights SUMA offers system administrators the power to setup the capability of automating the download of maintenance fixes onto a system and supports a comprehensive set of features: Automated task-based retrieval of multiple fix types (APAR, PTF, Critical, Security, Latest, Fileset, Maintenance Level) Three task actions allow for download preview, actual download of updates, or combining the download with a fix repository cleanup (utilizing lppmgr) A scheduling module allows policies to be run at various intervals in order to conform to necessary maintenance windows E-mail notification of update availability and task completion Support for HTTP, HTTPS, and FTP transfer protocols and proxy servers Filtering options allow comparisons against an installed software inventory or a maintenance level

9.2

Compare_report

The AIX 5L compare_report command allows clients to maintain a proactive fix strategy by providing them an easy way to ensure their systems are at the latest maintenance level or latest level. A comparison can be done between the filesets installed on a standalone system and the contents of an image repository or a list of fixes available from the IBM Support Web site to determine the fixes that need to be downloaded. The following site contains additional details on using the compare_report command: http://www-912.ibm.com/eserver/support/fixes/techTips.html

aixseStrategywp020706.doc

Page 18

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

9.2.1 Functional Highlights The compare_report command provides the following functions: Compares the filesets installed on a system to a list of fixes available from the IBM Support Web site or a fix repository Generates reports detailing which fixes are required for a system to be at the latest level or latest maintenance level Reports may be uploaded to the Compare report page (select based on your desired OS level) on the IBM Support Web site (http://www.ibm.com/eserver/support/fixes/FixReleaseInfo.jsp?system=2&release=5.2) and fixes may then be downloaded utilizing normal Web site interfaces

aixseStrategywp020706.doc

Page 19

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

10 Usage Scenarios
10.1 Overview
Determining a method to keep your system up-to-date with AIX 5L software updates is an important part of system administration. By having a better idea of how to maintain your system, you will be able to take advantage of the benefits of having current software updates, including increased security and reliability benefits, as well as the reduced downtime and total cost of ownership that come with keeping current fixes on a system. When utilizing the scenarios in this section it is recommended to establish a process that groups the steps into stages. These stages can provide a structure for business best practices and are described in general below: 10.1.1 Awareness The Awareness stage centers on the ability to perform preventative maintenance by being able to determine if a fix is available for your system. Examples include: Being notified by an automated SUMA task that a specific fix or group of fixes is available or has been downloaded Subscription service Running the compare_report command to determine the fixes available to get a system to the latest level or latest maintenance level and being able to download them CERT Advisories

10.1.2 Decision The Decision stage relates to assessing the business impact of installing the available fixes. Examples include: Performing a preview installation of the available fixes to gather information about requisite, space, and reboot requirements Obtaining information about the fixes, such as what has been fixed or changed Testing the fixes on non-production systems and confirming their readiness for deployment. For example, adopting a level and a timeframe of testing on bronze, silver, and gold systems. Or, to put it another way, a development system, a test system and a production system.

10.1.3 Implement The Implement stage focuses on performing the deployment and installation of the available fixes. Examples include:

aixseStrategywp020706.doc

Page 20

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

Performing a NIM installation of clients within an available maintenance window Using the alt_disk_install command to perform the update in order to minimize the impact to the running system, and then planning the switch to the new software level during the weekend

10.1.4 Verify The Verify stage involves utilizing tools to ensure that systems are in compliance with established criteria. Examples include: Checking that a system is in compliance with the approved level of security fixes Running the compare_report command to confirm that two clients are at the same software inventory level Verifying that a systems level of software meets an established baseline

aixseStrategywp020706.doc

Page 21

10.2 Proactive Maintenance with SUMA

Using some of the sample timeframes described in the Maintenance Strategy Models section for when clients may choose to move to the next Technology Level, this section gives some recommendations for using SUMA to monitor for the availability of new Technology Levels, Service Packs, and Concluding Service Packs. 10.2.1 Latest Level Clients who choose to move to a new TL when it is released may find setting up the following types of SUMA tasks helpful. Check monthly for a new TL Since a TL is released approximately every 6 months, a monthly check may be appropriate. You may schedule a SUMA task with a command similar to the one below to check monthly (for example, on the 15th of every month at 2:30am) whether a new TL has been released. An email notification will be sent with the results of the check. You may also elect to automatically download the filesets in this Technology Level by setting Action equal to Download instead of Preview in the command below. In this case, the filesets will only be downloaded, and no installation will occur. suma s 30 2 15 * * a Action=Preview a RqType=ML a RqName=5300-04 a FilterML=5300-03 a NotifyEmail=bob.smith@host3 Note: Currently SUMA uses the previous terminology of ML (Maintenance Level) instead of TL (Technology Level). Both will be supported in future versions. This command performs a Preview (no download will occur) to check if TL 5300-04 has been released. The FilterML setting specifies that the client already has filesets in the 5300-03 level. If 5300-04 has been released, the notification email will contain the list of filesets in TL 5300-04 that would be or were downloaded. If 5300-04 is not yet available, the email notification will contain a message similar to Invalid requested ML level:V530004. Check weekly for a new SP Since a SP is released approximately every 4-6 weeks, a weekly check may be appropriate. To check weekly (for example, every Thursday at 3:00am) and receive an email notification indicating whether a new SP has been released, and have the Service Pack automatically downloaded if it is found, you may schedule a SUMA task with a command similar to: suma s 0 3 * * 4 a Action=Download a RqType=SP a RqName=5300-04-01 a FilterML=5300-04 a NotifyEmail=bob.smith@host3 This command will download Technology Level 5300-04, Service Pack 1, when it becomes available. The FilterML setting specifies that the client already has filesets in the 5300-04 level.

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

10.2.2 Maximum Stability Clients following a Maximum Stability maintenance model may wish to monitor when the Concluding Service Pack is released. When the CSP comes out, they could utilize SUMA to download both the N-1 TL and its corresponding CSP. Since those adhering to this model are not moving to the latest TL when it is released, they would not have to utilize SUMA to regularly check for the release of a TL or Service Packs. Check monthly for a new CSP Since a CSP is released approximately every 6 months, a monthly check may be appropriate. You may schedule a SUMA task with a command similar to the one below to check monthly (for example, on the 15th of every month at 2:30am) whether a new CSP has been released. An email notification will be sent with the results of the check. You may also elect to automatically download the filesets in this CSP by setting Action equal to Download instead of Preview in the command below. In this case, the filesets will only be downloaded, and no installation will occur. suma s 30 2 15 * * a Action=Preview a RqType=SP a RqName=5300-04-CSP a FilterML=5300-04 a NotifyEmail=bob.smith@host3 a DLTarget=/tmp/530004 The above suma command will return a SUMA task ID that may later be used to perform an immediate download of a schedule task. For example, the following command could be used to immediately download the 5300-04-CSP that had the Preview action scheduled above to check for its release. (Assumes task ID of 4 was returned.) suma x a Action=Download 4 Once you have been notified that a new CSP has been released, you can also perform an immediate download of the corresponding TL using the following command: suma x a Action=Download a RqType=ML a RqName=5300-04 a FilterML=5300-03 a DLTarget=/tmp/530004 Note: Currently SUMA uses the previous terminology of ML (Maintenance Level) instead of TL (Technology Level). Both will be supported in future versions. The command will perform a download of the 5300-04 TL into the /tmp/530004 directory. Both the 5300-04 TL and CSP may then be installed at the same time by utilizing the SMIT menus found at the smitty install_all fast path. 10.2.3 Yearly Update Clients following the Yearly Update maintenance model may wish to monitor when Technology Levels and Concluding Service Packs are released.

aixseStrategywp020706.doc

Page 23

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

The process of utilizing SUMA to automatically check for when a TL or CSP is released is similar to what is described in the preceding Latest Level (to check monthly for a new TL) and Maximum Stability (to check monthly for a new CSP) sections, so please refer to those sections.

10.3 Standalone with Internet access

This scenario will detail the steps to retrieve the latest level of software updates for a standalone system running AIX 5L ML 5200-06. Since the AIX 5L system has Internet access, it can use the AIX 5L suma command to download updates. Prior to running the suma command, any authentication should be performed to ensure the system has Internet access. To verify the system is connected to the Internet, which will allow for the download of fixes from the IBM Support Web site, enter the following command (performs a Preview download with no files being downloaded): suma x a Action=Preview a RqType=Latest If the following message is returned, the system is not authenticated to access the Internet and you should contact your administrator to determine the steps necessary to allow your system to access the Internet. 0500-013 Failed to retrieve list from fix server. 10.3.1 Download the latest fixes periodically (using suma command line) Perform the following steps to setup a SUMA task to download the latest updates periodically: 1. To create and schedule a task (it will not run the task) that will check monthly (for example, on the 15th of every month at 2:30am) for the latest updates, and send email notifications to users on a remote system, type: suma s 30 2 15 * * a RqType=Latest a NotifyEmail=bob.smith@host3 Note: A task ID will be returned for this newly created task. This example will utilize SUMA task defaults, as displayed by suma D, for any Field=Value pair not specified on the command line. For example, with the task default of DLTarget=/usr/sys/inst.images, the updates in installp format will be downloaded into /usr/sys/inst.images/installp/ppc. A SUMA task has now been created and scheduled that will automatically run every month to check for the latest level of software updates available and download any new updates found. The inventory of the system will be used to determine if more recent updates are available.

aixseStrategywp020706.doc

Page 24

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

2. If you wish to run this newly created task immediately, and perform a Preview operation instead of a Download, type the following command to return a list of the latest level updates that are available for the filesets installed on the system (assume task ID 2 was returned from the task created in step 1): suma x a Action=Preview 2 Note: Email notifications are only sent for scheduled tasks, not those run immediately. 2a. If you wish to return a list of all the latest updates, including those for filesets not installed on the system from which suma is being run, enter the following command, which sets the FilterSysFile field to ignore the system inventory: suma x a Action=Preview a FilterSysFile=/dev/null 2 3. If you wish to list information on the above SUMA task, type: suma 1 2 10.3.2 Download a specific APAR (using suma SMIT easy menus) Perform the following steps to download a specific APAR using the SUMA SMIT menus: 1. Type the following smitty fast path to view the current SUMA task defaults. These will be utilized as default values when performing operations from the SUMA Easy menu without being asked for. smitty suma_task_defaults

aixseStrategywp020706.doc

Page 25

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

Pay attention to Action (specify whether a Download or Preview is preferred) and Directory for item storage (the DLTarget where the APAR will be downloaded). For example, with the task default of DLTarget=/usr/sys/inst.images, updates in installp format will be downloaded into /usr/sys/inst.images/installp/ppc. There is no need to change the Type of item to request field value to APAR (from the default setting of Security) as this will automatically be changed to APAR once the APAR selection is made from the Easy menu. 2. To access the Download Updates Now (Easy) menu, type: smitty suma_easy

aixseStrategywp020706.doc

Page 26

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

3. Select Download by APAR Number 4. Type the APAR number (e.g. IY12345), and press Enter. The APAR will now be immediately downloaded into the DLTarget directory, if available.

10.4

Standalone with no Internet access

This scenario will detail the steps to retrieve the latest level of software updates for a system running AIX 5L ML 5200-06. Since the AIX 5L system has no Internet access, it can use the AIX 5L compare_report command, along with a client having Internet access, to download updates. 10.4.1 Download the latest fixes (using compare_report command line) Perform the following steps to download the latest updates: 1. Utilizing a client having Internet access, download the appropriate latest fix data file from the Compare report page of the IBM Support Web site by selecting the Data file for AIX 5.2 link. This file contains a list of the latest available updates.
http://www-912.ibm.com/eserver/support/fixes/FixReleaseInfo.jsp?system=2&release=5.2

aixseStrategywp020706.doc

Page 27

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

2. Copy the LatestFixData52 file to your AIX 5L system. 3. Run the compare_report command to compare the filesets installed on the AIX 5L system (-s option) with the filesets available in the latest fix data file (-r option) to generate a report of the latest fixes not currently installed on your AIX 5L system. Using the l option will generate a report file lowerthanlatest1.rpt (in /tmp by default) that lists the filesets on the system that are at a lower level than the latest level. /usr/sbin/compare_report s r /tmp/LatestFixData52 l 4. Upload the lowerthanlatest1.rpt report file to the Compare report page of the IBM Support Web site. The PTF numbers contained in the report file will be used to provide the requested fixes. 5. Use the normal IBM Support Web site interfaces to download the needed fixes. 10.4.2 Download a specific APAR (using the IBM Support Web site) Perform the following steps to download a specific APAR using the IBM Support Web site:

aixseStrategywp020706.doc

Page 28

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

1. Go to the Specific fixes page for your desired OS level (e.g. AIX 5L V5.2). http://www.ibm.com/eserver/support/fixes/search.jsp?system=2&release=5.2 Note: Future enhancements to the Fix Central Web site will allow for selection of a Specific fix from the main Fix Central page for Unix servers. 2. Select to Search by APAR number. 3. Type in the APAR number (e.g. IY12345), and press Go.

4. Use the normal IBM Support Web site interfaces to download the APAR.

10.5 More than one system with Internet access

This scenario will detail the steps to retrieve the latest level of software updates for multiple systems running AIX 5L ML 5200-06. Since the AIX 5L systems have Internet access, they can use the AIX 5L suma command to download updates and NIM to distribute the updates to multiple systems. Prior to running the suma command, any authentication should be performed to ensure systems have Internet access. To verify systems are connected to the Internet, which will allow for the download of fixes from the IBM Support Web site, enter the following command (performs a Preview download, with no files being downloaded): suma x a Action=Preview a RqType=Latest
aixseStrategywp020706.doc Page 29

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

If the following message is returned, the system is not authenticated to access the Internet and you should contact your administrator to determine the steps necessary to allow your system to access the Internet. 0500-013 Failed to retrieve list from fix server. 10.5.1 Download the latest fixes (setup suma on all NIM clients) To setup a SUMA task to download the latest updates periodically, the same steps documented in the Standalone with Internet access section may be utilized. Note: Depending on the number of systems you wish to maintain with the latest fixes, you can either setup a suma task on all NIM clients, or setup a suma task only on the NIM master (described in the next section): For a small number of NIM clients, it is recommended that a SUMA task be setup on each NIM client. This provides the advantage of being able to download the specific fixes that each client needs directly to the client automatically. 10.5.2 Download the latest fixes (setup suma only on NIM master) For a large number of NIM clients, where it might not be feasible to setup a suma task on each client, you may download all of the latest fixes available for each OS release level by setting up suma tasks on the NIM master. The disadvantage of this approach is that it doesnt utilize the client inventories to allow for download of only applicable clients updates, but rather downloads all available updates for an OS release level. Sample commands to do this, similar to the commands shown in the Standalone with Internet access section, would be the following to create a new SUMA task: suma s 30 2 15 * * a RqType=Latest a NotifyEmail=bob.smith@host3 Then to establish NIM lpp_source directories containing fixes for each OS release level supported by the NIM master, type (assuming task ID 2 was returned from the above command): suma x a Action=Preview a FilterSysFile=/dev/null a FilterML=5300-02 a DLTarget=/lpp_source53 2 suma x a Action=Preview a FilterSysFile=/dev/null a FilterML=5200-06 a DLTarget=/lpp_source52 2 Setting Action equal to Download instead of Preview in the command above will perform the actual download and establish separate lpp_source directories on the NIM master that will support the various OS release levels of the NIM clients.

aixseStrategywp020706.doc

Page 30

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

10.6

More than one system with no Internet access

10.6.1 Download the latest fixes (compare_report using inventory from clients) This scenario will detail the steps to retrieve the latest level of software updates for multiple systems not having Internet access. The AIX 5L compare_report command, along with a machine having Internet access, can be used to download updates. Perform the following steps to download the latest updates: 1. Utilizing a machine having Internet access, download the appropriate latest fix data file from the Compare report page of the IBM Support Web site by selecting the Data file for AIX link. This file contains a list of the latest available updates. For example, for AIX 5L V5.2 the file can be found here:
http://www.ibm.com/eserver/support/fixes/FixReleaseInfo.jsp?system=2&release=5.2

2. Copy the LatestFixData52 file to a NIM master so that it can be made available to multiple NIM clients. (E.g. /52/LatestFixData52)

aixseStrategywp020706.doc

Page 31

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

3. NFS export the /52 directory to allow access by other NFS clients. Enter one of the following commands as root to allow read-write access, so clients can later copy results to the exported directory. /usr/sbin/mknfsexp d /52 t rw Or to give access to specific clients: /usr/sbin/mknfsexp d /52 t rw c <client1.host> <client2.host> Note: To stop exporting the directory to clients, type: /usr/sbin/rmnfsexp d /52 4. NFS mount the directory on the client by typing the following on the client: mount <master>:/52 /mnt 5. Run the compare_report command to compare the filesets installed on the AIX 5L system (-s option) with the filesets available in the latest fix data file (-r option) to generate a report of the latest fixes not currently installed on your AIX 5L system. Using the l option will generate a report file lowerthanlatest1.rpt (in the /tmp by default) that lists the filesets on the system that are at a lower level than the latest level. /usr/sbin/compare_report s r /mnt/LatestFixData52 l 6. Give the lowerthanlatest1.rpt report file a unique name (e.g. client1.rpt). 7. Copy the uniquely named report file to the NIM master (e.g. /mnt/client1.rpt) that will receive the report files from all the NIM clients at specific OS release levels. /usr/bin/cp /tmp/client1.rpt /mnt 8. Repeat steps 4-7 above for each NIM client or system that you wish to have installed with the latest level of fixes, altering the unique name given in step 6 and the copy command in step 7 (e.g. cp /tmp/client2.rpt /mnt, etc.). Note: For a V5.3 system, step 2 would copy the LatestFixData53 file, and the other steps would utilize a /53 directory so the AIX 5L V5.3 based files can be handled separate from the AIX 5L V5.2 based report files. 9. For each OS release level, combine all of the report files on the master into a single file, sorting to remove duplicate entries, using a command similar to below: cd /52; cat client1.rpt client2.rpt client3.rpt | sort u > full52.rpt OR

aixseStrategywp020706.doc

Page 32

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

cd /52; cat * | sort -u > full52.rpt 10. Upload the full52.rpt report file to the Compare report page of the IBM Support Web site (shown in step 1). The PTF numbers contained in the report file will be used to provide the requested fixes. 11. Use the normal IBM Support Web site interfaces to download the needed fixes to an OS level location on the NIM master. For example, the fixes requested by full52.rpt can be copied into a /lpp_source52 directory on the NIM master, the fixes requested by full53.rpt can be copied into a /lpp_source53 directory, etc. The different lpp_source directories will then hold the latest level of fixes for a specific OS release level and can be used to support clients at varying OS release levels. 12. To setup a directory (e.g. /lpp_source52) as a NIM lpp_source resource, type the following where 52_lpp is the NIM resource name being assigned: nim -d define -t lpp_source -a server=<master> -a location=/lpp_source52 52_lpp 13. To setup the NIM clients as a machine group (so they can be updated with one command), type the following, where 52_client_grp is the name of the machine group being created: nim -o define -t mac_group -a add_member=<client1.host> \ -a add_member=<client2.host> 52_client_grp For an environment with a large number of NIM clients it may be more convenient to setup the machine group thru the SMIT menus instead of listing each client on the command line. In this case, type: smitty nim_mkgrp_standalone 14. To perform a NIM operation to update all software on a machine group of NIM clients or a single NIM client, type the following: To update the 52_client_grp machine group, type: nim -o cust -a lpp_source=52_lpp -a filesets=all 52_client_grp To update a single client, type: nim -o cust -a lpp_source=52_lpp -a filesets=all <client1.host>

aixseStrategywp020706.doc

Page 33

THE AIX 5L SERVICE STRATEGY AND BEST PRACTICES

IBM Corporation 2006

IBM Corporation Marketing Communications Systems and Technology Group Route 100 Somers, New York 10589 Produced in the United States of America February 2006 All Rights Reserved This document was developed for products and/or services offered in the United States. IBM may not offer the products, features, or services discussed in this document in other countries. The information may be subject to change without notice. Consult your local IBM business contact for information on the products, features and services available in your area. All statements regarding IBM future directions and intent are subject to change or withdrawal without notice and represent goals and objectives only. IBM, the IBM logo, AIX 5L logo, AIX, AIX 5L, pSeries are trademarks or registered trademarks of International Business Machines Corporation in the United States or other countries or both. A full list of U.S. trademarks owned by IBM may be found at: http://www.ibm.com/legal/copytrade.shtml. Other company, product, and service names may be trademarks or service marks of others. Information concerning non-IBM products was obtained from the suppliers of these products or other public sources. Questions on the capabilities of the non-IBM products should be addressed with the suppliers. The IBM home page on the Internet can be found at http://www.ibm.com. The System p5 and ~ p5 home page on the Internet can be found at http:www.ibm.com/systems/p.

aixseStrategywp020706.doc

Page 34