Installation Guide QLogic Fabric SW v50 RevA

QLogic Fabric Software Installation Guide
Version 5.0
D000084-002 Rev A
Information furnished in this manual is believed to be accurate and reliable. However, QLogic Corporation assumes no responsibility for its use, nor for any infringements of patents or other rights of third parties which may result from its use. QLogic Corporation reserves the right to change product specifications at any time without notice. Applications described in this document for any of these products are for illustrative purposes only. QLogic Corporation makes no representation nor warranty that such applications are suitable for the specified use without further testing or modification. QLogic Corporation assumes no responsibility for any errors that may appear in this document.
Document Revision History

Revision A, May, 2009 Changes Sections Affected
Page ii
D000084-002 Rev A
Table of Contents
Preface
Intended Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Related Materials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Documentation Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . License Agreements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Technical Support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Contact Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix ix ix x x x xi 1-1 1-1 1-2 1-3 1-3 1-4 2-1 2-2 2-4 2-4 2-5 2-5 2-5 2-13 2-13
Software Installation Overview

Component Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . QLogic IB Basic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Installation Prerequisites. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . QLogic InfiniBand Fabric Suite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Installation Prerequisites. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Installation Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Install the Fabric Software

Design the Fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Set Up the Fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Install QLogicIB-IFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pre-installation Steps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Install QLogicIB-IFS on IB Management Node . . . . . . . . . . . . . . . . . . Unpack the Tar File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Install QLogicIB-IFS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Configure QLogic and SilverStorm IB Chassis. . . . . . . . . . . . . . . . . . . . . . . QLogic and SilverStorm IB Chassis Configuration Pre-Requisites . . .
D000084-002 Rev A
iii
S
2-15 2-16 2-17 2-18 2-19 2-20 2-21 2-21 2-22 2-23 2-23 2-23 2-24 2-24 2-26 2-26 2-26 2-27 2-28 2-28 2-29 2-30 2-31 2-31 2-31 2-32 2-32 2-32 2-33 2-33 2-33 2-33 2-34 2-35 2-35 2-36 2-36 2-36 2-37
Configure Chassis Using Fast Fabric . . . . . . . . . . . . . . . . . . . . . . . . . Edit Config and Select/Edit Chassis Files . . . . . . . . . . . . . . . . . Verify Chassis via Ethernet ping . . . . . . . . . . . . . . . . . . . . . . . . . Update Chassis Firmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Set Up Chassis Basic Configuration. . . . . . . . . . . . . . . . . . . . . . Setup Password-less ssh/scp. . . . . . . . . . . . . . . . . . . . . . . . . . . Reboot Chassis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Configure Chassis Fabric Manager . . . . . . . . . . . . . . . . . . . . . . Run a command on all chassis. . . . . . . . . . . . . . . . . . . . . . . . . . Install and Configure the Fabric Manager . . . . . . . . . . . . . . . . . . . . . . . . . . Configure Firmware on the Externally Managed IB Switches . . . . . . . . . . . IB Switches Configuration Pre-Requisites. . . . . . . . . . . . . . . . . . . . . . Configure Externally Managed Switches. . . . . . . . . . . . . . . . . . . . . . . Edit Config and Select/Edit Switch Files. . . . . . . . . . . . . . . . . . . Test for Switch Presence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Verify Switch Firmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Update Switch Firmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Set Up Switch Basic Configuration . . . . . . . . . . . . . . . . . . . . . . . Reboot Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Report Switch Firmware and Hardware Info. . . . . . . . . . . . . . . . Install InfiniBand on the Remaining Servers . . . . . . . . . . . . . . . . . . . . . . . . Edit Config and Select/Edit Hosts Files . . . . . . . . . . . . . . . . . . . . . . . . Verify Hosts via Ethernet ping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Setup Password-less ssh/scp . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Copy /etc/hosts to all hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Show uname -a for all hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Install/Upgrade QLogic IB Software . . . . . . . . . . . . . . . . . . . . . . . . . . Configure IPoIB IP Address. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Build MPI Test Apps and Copy to Hosts . . . . . . . . . . . . . . . . . . . . . . . Reboot Hosts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Run a command on all hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Verifying InfiniBand on the Remaining Servers . . . . . . . . . . . . . . . . . . . . . . Edit Config and Select/Edit Hosts Files . . . . . . . . . . . . . . . . . . . . . . . . Verify Hosts via Ethernet ping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Summary of Fabric Components. . . . . . . . . . . . . . . . . . . . . . . . . . . . . Check Status of IB Ports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Verify Hosts see each other. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Verify Hosts ping via IPoIB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Refresh ssh Known Hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Page iv
D000084-002 Rev A
Check MPI Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Check Overall Fabric Health . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Complete Installation of additional IB Management Nodes . . . . . . . . . . . . . Configure and Initialize Health Check Tools. . . . . . . . . . . . . . . . . . . . . . . . . Running High Performance Linpack. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2-37 2-38 2-38 2-40 2-41 3-1 3-3 3-3 3-3 4-1
Upgrade the Management Node

Preinstallation Steps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . QLogicIB-IFS Upgrade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Unpack the Tar File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Upgrade QLogicIB-IFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4 A
Upgrade the Fabric

Upgrade QLogic InfiniBand Fabric Suite . . . . . . . . . . . . . . . . . . . . . . . . . . .
Infiniband Fabric Suite Installation Checklist

Pre-Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Install QLogicIB-IFS on Management Node. . . . . . . . . . . . . . . . . . . . . . . . . Configure and verify Firmware on the IB Chassis . . . . . . . . . . . . . . . . . . . . Pre-Configuration Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Configuration and Verification of Chassis . . . . . . . . . . . . . . . . . . . . . . Install and Configure the Fabric Manager . . . . . . . . . . . . . . . . . . . . . . . . . . Installing and Verifying Firmware on the IB Switches. . . . . . . . . . . . . . . . . . Pre-Configuration Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Configuration and Verification of Switches . . . . . . . . . . . . . . . . . . . . . Install Infiniband on the Remaining Servers. . . . . . . . . . . . . . . . . . . . . . . . . Verifying Infiniband on the Remaining Servers . . . . . . . . . . . . . . . . . . . . . . Complete Installation of additional IB Management Nodes . . . . . . . . . . . . . Configure and initialize health check tools . . . . . . . . . . . . . . . . . . . . . . . . . . A-1 A-2 A-2 A-2 A-2 A-2 A-3 A-3 A-3 A-3 A-4 A-4 A-4
Fast Fabric Configuration Files

fastfabric.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iba_mon.conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Host List Files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chassis List Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Selection of slots within a chassis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Switch List Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Port List Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . topology.0:0.xml . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B-2 B-10 B-12 B-13 B-14 B-15 B-16 B-17
D000084-002 Rev A
Page v
S
C-1 C-4
Multi-Subnet Fabrics
Primarily Independent Subnets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Overlapping Subnets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D E
Install a Previous Version of Software Install QLogicIB-Basic

Install QLogicIB-Basic on Cluster Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . Unpack the Tar File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Install QLogicIB-IFS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E-1 E-1 E-1
QLogicIB-Basic to IFS Upgrade

Unpack the Tar File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Install QLogicIB-IFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F-1 F-1
./Install Syntax List of Figures
Figure Page 2-1 QLogic IB Software Main Menu (Example) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-5 2-2 QLogic IB Install Menu (Example) (Screen 1 of 3). . . . . . . . . . . . . . . . . . . . . . . . . . 2-6 2-3 QLogic IB Install Menu (Example) (Screen 2 of 3). . . . . . . . . . . . . . . . . . . . . . . . . . 2-7 2-4 QLogic IB Install Menu (Example) (Screen 3 of 3). . . . . . . . . . . . . . . . . . . . . . . . . . 2-8 2-5 QLogic Fast Fabric InfiniBand Tools Menu (Example) . . . . . . . . . . . . . . . . . . . . . . . 2-15 2-6 QLogic Fast Fabric IB Chassis Setup/Admin Menu . . . . . . . . . . . . . . . . . . . . . . . . . 2-16 2-7 QLogic Fast Fabric IB Switch Setup/Admin Menu . . . . . . . . . . . . . . . . . . . . . . . . . . 2-24 2-8 QLogic Fast Fabric IB Host Setup Menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-29 2-9 QLogic Fast Fabric IB Host Verification/Admin Menu . . . . . . . . . . . . . . . . . . . . . . . 2-34 3-1 QLogic IB Software Main Menu (Example) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-3 3-2 QLogic IB Install Menu (Example) (Screen 1 of 3). . . . . . . . . . . . . . . . . . . . . . . . . . 3-4 3-3 QLogic IB Install Menu (Example) (Screen 2 of 3). . . . . . . . . . . . . . . . . . . . . . . . . . 3-5 3-4 QLogic IB Install Menu (Example) (Screen 3 of 3). . . . . . . . . . . . . . . . . . . . . . . . . . 3-5 E-1 QLogic InfiniBand Main Menu (Example). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E-2 E-2 QLogicIB-Basic Install Menu (Example) (Screen 1 of 3) . . . . . . . . . . . . . . . . . . . . . E-2 E-3 QLogicIB-Basic Install Menu (Example) (Screen 2 of 3) . . . . . . . . . . . . . . . . . . . . . E-3 E-4 QLogicIB-Basic Install Menu (Example) (Screen 3 of 3) . . . . . . . . . . . . . . . . . . . . . E-4 F-1 QLogic InfiniBand Main Menu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F-2 F-2 QLogicIB-IFS Install Menu (Example) (Screen 1 of 2) . . . . . . . . . . . . . . . . . . . . . . . F-2 F-3 Installation Complete . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F-3
Tables
Table Page 2-1 Performance Impact. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-38
Page vi
D000084-002 Rev A
S
B-1
Fast Fabric Configuration Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B-1
D000084-002 Rev A
Page vii
Page viii
D000084-002 Rev A
Preface
Intended Audience
This manual is intended to provide network administrators and other qualified personnel a reference for installation and configuration of the QLogicIB-Basic and QLogicIB-IFS software.
Related Materials

QLogic Fast Fabric Users Guide QLogic Fabric Manager Users Guide QLogic Fast Fabric CLI Reference Guide
Documentation Conventions
This guide uses the following documentation conventions: NOTE: provides additional information. CAUTION! indicates the presence of a hazard that has the potential of causing damage to data or equipment. WARNING!! indicates the presence of a hazard that has the potential of causing personal injury. Text in blue font indicates a hyperlink (jump) to a figure, table, or section in this guide, and links to Web sites are shown in underlined blue. For example: Table 9-2 lists problems related to the user interface and remote agent. See Installation Checklist on page 3-6. For more information, visit www.qlogic.com.
Text in bold font indicates user interface elements such as a menu items, buttons, check boxes, or column headings. For example: Click the Start button, point to Programs, point to Accessories, and then click Command Prompt. Under Notification Options, select the Warning Alarms check box.
D000084-002 Rev A
ix
A
Contact Information
Please feel free to contact your QLogic approved reseller or QLogic Technical Support at any phase of integration for assistance. QLogic Technical Support can be reached by the following methods:
Web Email http://support.qlogic.com support@qlogic.com
The QLogic knowledge database contains troubleshooting information for the QLogic products. Access the data base from the QLogic Support Web page, http://support.qlogic.com. Use the Support Center search engine to look for specific troubleshooting information.
D000084-002 Rev A
xi
xii
D000084-002 Rev A
S
Text in Courier font indicates a file name, directory path, or command line text. For example: To return to the root directory from anywhere in the file structure: Type cd /root and press ENTER. Enter the following command: sh ./install.bin
Key names and key strokes are indicated with UPPERCASE: Press CTRL+P. Press the UP ARROW key.
Text in italics indicates terms, emphasis, variables, or document titles. For example: For a complete listing of license agreements, refer to the QLogic Software End User License Agreement. What are shortcut keys? To enter the date type mm/dd/yyyy (where mm is the month, dd is the day, and yyyy is the year).
Topic titles between quotation marks identify related topics either within this manual or in the online help, which is also referred to as the help system throughout this document.
License Agreements
Refer to the QLogic Software End User License Agreement for a complete listing of all license agreements affecting this product.
Technical Support
Customers should contact their authorized maintenance provider for technical support of their QLogic switch products. QLogic-direct customers may contact QLogic Technical Support; others will be redirected to their authorized maintenance provider. Visit the QLogic support Web site listed in Contact Information for the latest firmware and software updates.
Availability
QLogic Technical Support for products under warranty is available during local standard working hours excluding QLogic Observed Holidays.
D000084-002 Rev A
Software Installation Overview

There are two main types of software installation for a QLogic fabric. The basic installation installs the QLogic InfiniBand (IB) Basic components that are required to run a fabric. The basic software installation is provided at no charge with the QLogic products. The QLogic IB Fabric Suite (IFS) is a chargeable set of software products that provide a number of value added features that includes everything in the basic installation package along with the QLogic Fast Fabric Toolset (FF) and the Fabric Manager (FM). The following sub-sections list the components of the two installation packages.
Component Overview
QLogic IB Basic
The QLogic IB Basic (QLogicIB-Basic.DISTRO.VERSION.tgz) installation package can install the following components: OFED IB Stack TrueScale HCA Libs QLogic IB Tools OFED IB Development QLogic SRP The VirtualHBA (i.e., SRP) ULP works in concert with the QLogic Fibre Channel Virtual I/O Controller (FVIC), providing virtual Fibre Channel connectivity.
QLogic VNIC The VirtualNIC (VNIC) upper layer protocol (ULP) works in concert with the QLogic Ethernet Virtual I/O Controller (EVIC), providing virtual Ethernet connectivity.

OFED IP over IB OFED SDP
D000084-002 Rev A
1-1
Software Installation Overview Component Overview

OFED uDAPL MVAPICH for gcc MVAPICH2 for gcc OpenMPI for gcc MVAPICH/PSM for gcc MVAPICH/PSM for PGI MVAPICH/PSM for PSc (eg. PathScale compiler) MVAPICH/PSM for Intel OpenMPI/PSM for gcc OpenMPI/PSM for PGI OpenMPI/PSM for PSc (eg. PathScale compiler) OpenMPI/PSM for Intel MPI Source OFED RDS OFED SRP OFED SRP Target OFED iSER OFED iSER Target OFED iWARP OFED Open SM OFED Debug Info
NOTE: There are separate variations of the installation package (QLogicIB-Basic.DISTRO.VERSION.tgz) for each of the following Linux distributions: Red Hat EL 4 Red Hat EL 5 SLES 10
Installation Prerequisites
In addition to normal OS installation options, the following OS rpms are also required to be installed prior to the QLogic IB Basic installation.
1-2
D000084-002 Rev A
A
tcl tcl-devel tk libstdc++
Software Installation Overview Component Overview
QLogic InfiniBand Fabric Suite

The QLogic InfiniBand Fabric Suite (QLogicIB-IFS.DISTRO.VERSION.tgz) installation package can install all of the components found in the QLogic InfiniBand Basic installation package plus: QLogic Fast Fabric Toolset QLogic Fabric Manager
For details on using QLogic Fast Fabric Toolset (FF), refer to the Fast Fabric Users Guide. For details on using QLogic Fabric Manager (FM), refer to the Fabric Manager Users Guide. NOTE: There are separate variations of the installation package (QLogicIB-IFS.DISTRO.VERSION.tgz) for each of the following Linux distributions: Red Hat EL 4 Red Hat EL 5 SLES 10
Installation Prerequisites
In addition to normal OS installation options, the following OS rpms are also required to be installed prior to the QLogic IB Fabric Suite installation. tcl tcl-devel tk libstdc++ expect
D000084-002 Rev A
1-3
Software Installation Overview Installation Overview
Installation Overview
The installation of the QLogic InfiniBand software provides a description of the installation, prerequisites for the installation and an easy to follow set of instructions to install the InfiniBand Fabric Suite (IFS) package with the user in mind. It also takes you through the configuration of all of the components in the fabric so that when finished, the fabric is ready for traffic. Installation procedures for the Basic package are provided in Appendix E.
1-4
D000084-002 Rev A
D000084-002 Rev A
1-5
Notes
1-6
D000084-002 Rev A
Install the Fabric Software

This section provides the information and procedures needed to install, configure, and verify the fabric software. The Site Implementation Engineer must perform the tasks described in this section to correctly install and configure the fabric. To aid in keeping track of steps performed, a checklist is provided in Appendix A that can be copied to use online and/or printed. When installing the QLogicIB-IFS package, the Fast Fabric (FF) configuration files that must be edited or created are described in Appendix B. The procedures below describe the IFS installation, configuration, and verification sequence for a typical single IB-subnet fabric. For information on installation and verification of multiple IB-subnet fabrics, see Appendix C. Some of the tasks are only applicable when Linux is being used and will be marked with (Linux). Other tasks are only applicable when QLogic OFED, the QLogic IB packaging of OFED, or QuickSilver Linux IB software is being used on the hosts and will be marked with (Host). All tasks which are applicable only when QLogic IB Switches, QLogic IB Chassis, SilverStorm IB Switches or SilverStorm IB Chassis are being used will be marked with (Switch). All remaining tasks are generally applicable to all environments and will be marked with (All). NOTE: Some of the Linux steps may be applicable to other Unix-like operating systems if it is desired to enable use of non-IB specific Fast Fabric tools (such as cmdall) against the given hosts.
Design the Fabric

Prior to beginning the installation and setup of the fabric, its important to carefully design and plan the installation. Part of the design plan must include identification of which servers will be the administration nodes for the cluster and consequently where the IFS will be installed.
D000084-002 Rev A
2-1
Install the Fabric Software Set Up the Fabric
For large clusters, cable, power, and cooling plans are very important and must be carefully considered. These plans drive the ultimate layout of equipment in the racks. A typical configuration involves edge switches and servers in the same racks, with core switches in centrally located racks. This minimizes both cable lengths and complexity. It is also recommended to place the IB switches at the bottom of a rack. This allows inter-rack cables to be cleanly routed below the floor (some sites use cable routing above the racks, in which case placing the IB switches near the top of the rack is recommended). NOTE: The overall physical design has many complex aspects, such as power, cooling and rack layout which are beyond the scope of this document.
Set Up the Fabric

The following steps provide the information to set up the fabric. For information about the configuration files used by Fast Fabric refer to Appendix B 1. (All) The first step in any installation is to physically install the hardware: Servers Core and edge InfiniBand switches, such as the SilverStorm 9024 and 9000 Multi-Protocol Fabric Directors (9020, 9040, 9080, 9120 and 9240). Virtual I/O systems, such as the EVIC and FVIC cards for the SilverStorm 9000 Multi-Protocol Fabric Directors Series.
NOTE: When installing externally managed switches (such as the QLogic 12200 or SilverStorm 9024-FC switch), take note of the Node GUID. This is typically on a label on the case of the switch. The Node GUID will be needed later to configure and manage the switch(es) with IFS. 2. (All) Within each server a host channel adapter (HCA), such as the QLogic 7000 series HCA, must be installed. Refer to the QuickSilver Fabric Access Quick Start Guide or the QLogic OFED Software Users Guide for instructions. (All) Prior to installing software, the hardware configuration should be reviewed to ensure everything was installed according to plan. Later during the installation, FF tools may also be used to help verify the installation.
3.
2-2
D000084-002 Rev A
A
4.
Install the Fabric Software Set Up the Fabric
(Linux) Install the desired Linux OS version (with the same kernel distribution) on all hosts. The IB Management node(s) (i.e., the host(s) which will run Fast Fabric) should have a full install and must include the TCL, and Expect packages. If Redhat Enterprise Linux is being installed, only the TCL and Expect packages are required. For MPI clusters, install the C and Fortran compilers along with their associated tools on the IB Management node(s). NOTE: All hosts must have a command-line prompt ending in # or $". Make certain there is a space after either # or $. This prompt is used for the root user and all other user codes.
NOTE: To simplify the use of Fast Fabric to set up ssh security, it is recommended to install all servers with the same root password. After ssh has been set up using Fast Fabric, the user can change the root passwords.
NOTE: Consult the QLogic Infiniband Fabric Suite Release Notes for a list of supported OS versions. 5. (Linux) Enable remote login as root to each host: In order for FF to manage the hosts, the IB Management Node must be able to securely log in as root to each host. This can be accomplished using ssh. 6. (All) TCP/IP Host Name resolution: Fast Fabric and TCP/IP will need to resolve hostnames to the Management Network and/or IPoIB IP addresses. If the management network is not IPoIB, each host will need both a management network name and an IPoIB network name. A recommended convention is to use the actual hostname as the management network name and HOSTNAME-ib as the IPoIB network name (where HOSTNAME is the management network name of the given host).
D000084-002 Rev A
2-3
Install the Fabric Software Install QLogicIB-IFS
Name resolution is accomplished by configuring a DNS server on the management network with both management network and IPoIB addresses for each host and each QLogic internally managed IB chassis. An /etc/hosts file may be created on the IB Management node. Fast Fabric can then propagate this /etc/hosts file to all the other hosts. If using the /etc/hosts approach: On the master node, add all the Ethernet and IPoIB addresses into the /etc/hosts file. For the IPoIB convention, use HOSTNAME-ib. The localhost line should not be edited. The /etc/hosts file should not have any node-specific data. Copy the file to all of the nodes as described in Copy /etc/hosts to all hosts on page 2-31. If using DNS: Consult the documentation for the Domain Name System (DNS) server being used. Make sure to edit the /etc/resolv.conf configuration on the IB Management Node to use the proper DNS server. Consult the Linux OS documentation for more information on configuring /etc/resolv.conf. This file is typically configured during OS installation. If /etc/resolv.conf must be manually configured for each host, Fast Fabric can aid in copying the file to all the hosts. The /etc/resolv.conf file created on the IB Management Node must not have any node-specific data and must be appropriate for use on all hosts. A later section will step through the task of copying this file to all the nodes. 7. (All) NTP setup: Configure an NTP server for the cluster, and have all the Linux hosts and Internally Managed Chassis synchronize their clocks with the NTP server.
Setup of the Internally Managed chassis is described in the Configure QLogic and SilverStorm IB Chassis on page 2-13.
8. 9.
(All) Install the software: On the IB Management node, install the QLogicIB-IFS software.
Install QLogicIB-IFS
Pre-installation Steps
Prior to installing QLogicIB-IFS, ensure the following have been performed: Review the Release Notes for a list of compatible software. QLogicIB-IFS includes a compatible version of QLogic OFED. Consult the QLogic OFED Users Guide for more information on QLogic OFED.
2-4
D000084-002 Rev A
NOTE: When managing a cluster where the IPoIB settings on the compute nodes are incompatible with the IB Management node (for example when a 4K MTU is used on the compute nodes and a 2K MTU is used on the management nodes), it is recommended not to run IPoIB on the IB management nodes.
Install QLogicIB-IFS on IB Management Node

On hosts where the full QLogic InfiniBand Fabric Suite (IFS) package has been purchased use the package file, QLogicIB-IFS.DISTRO.VERSION.tgz. Using the menus, select to install the desired components (at least OFED IB Stack, QLogic IB Tools and QLogic Fast Fabric) as described below.
Unpack the Tar File

Use the following procedure to unpack the QLogicIB-IFS.DISTRO.VERSION.tgz tar file. 1. 2. 3. 4. Login as user root. Copy the tar file QLogicIB-IFS.DISTRO.VERSION.tgz to the /root directory. Type cd /root and press ENTER. Type tar xvfz QLogicIB-IFS.DISTRO.VERSION.tgz and press ENTER. This unpacks the tar file to the QLogicIB-IFS.DISTRO.VERSION directory.
1. 2. Type cd QLogicIB-IFS.DISTRO.VERSION and press ENTER Type ./INSTALL and press ENTER. Displays the QLogic Inc. Infiniband VERSION Software main menu (Figure 2-1).
QLogic Inc. InfiniBand VERSION Software 1) 2) 3) 4) 5) 6) Install/Uninstall Software Reconfigure OFED IP over IB Reconfigure Driver Autostart Update HCA Firmware Generate Supporting Information for Problem Report Fast Fabric (Host/Chassis/Switch Setup/Admin)
X) Exit
Figure 2-1 QLogic IB Software Main Menu (Example)
D000084-002 Rev A
2-5
3.
Press 1 Displays screen 1 of 3 of the QLogic Inc. IB Install menu (Figure 2-2)
QLogic Inc. IB Install (VERSION release) Menu Please Select Install Action (screen 0) OFED IB Stack [ Install 1) TrueScale HCA Libs [ Install 2) QLogic IB Tools [ Install 3) OFED IB Development [ Install 4) QLogic Fast Fabric [ Install 5) QLogic SRP [ Install 6) QLogic Virtual NIC [ Install 7) OFED IP over IB [ Install 8) OFED SDP [ Install 9) OFED uDAPL [ Install a) QLogic FM [ Install b) MVAPICH (gcc) [ Install c) MVAPICH2 (gcc) [ Install d) OpenMPI (gcc) [ Install N) P) R) X) Next Screen Perform the selected actions Re-Install All Return to Previous Menu (or ESC) 1 of 3): ][Available] ][Available] ][Available] ][Available] ][Available] ][Available] ][Available] ][Available] ][Available] ][Available] ][Available] ][Available] ][Available] ][Available]
VERSION VERSION VERSION VERSION VERSION VERSION VERSION VERSION VERSION VERSION VERSION VERSION VERSION VERSION
I) Install All U) Uninstall All
Figure 2-2 QLogic IB Install Menu (Example) (Screen 1 of 3)

NOTE: In Figure 2-2 and Figures 2-3 and 2-4 below, the TrueScale and PSM components are only available if a QLogic TrueScale HCA is found in the given server. 4. 5. Verify that all items that are selected to be installed in the default menu are correct. Select any items that are required and not selected by default. NOTE: To change the selection of any of the items in the different menus select the number associated with the item. This toggles the selection from Install to Don't Install and back. 6. Press N.
2-6
D000084-002 Rev A
A
QLogic Inc. IB Install (VERSION release) Menu Please Select Install Action (screen 2 of 3): 0) MVAPICH/PSM (gcc) [ Install ][Available] 1) MVAPICH/PSM (PGI) [ Install ][Available] 2) MVAPICH/PSM (PSc) [ Install ][Available] 3) MVAPICH/PSM (Intel) [ Install ][Available] 4) OpenMPI/PSM (gcc) [ Install ][Available] 5) OpenMPI/PSM (PGI) [ Install ][Available] 6) OpenMPI/PSM (PSc) [ Install ][Available] 7) OpenMPI/PSM (Intel) [ Install ][Available] 8) MPI Source [ Install ][Available] 9) OFED RDS [ Install ][Available] a) OFED SRP [ Install ][Available] b) OFED SRP Target [Don't Install][Available] c) OFED iSER [Don't Install][Available] d) OFED iSER Target [Don't Install][Available] N) P) R) X) Next Screen Perform the selected actions Re-Install All Return to Previous Menu (or ESC)
Displays screen 2 of 3 of the QLogic Inc. IB Install menu (Figure 2-3)
VERSION.sles10 VERSION.sles10 VERSION.sles10 VERSION.sles10 VERSION.sles10 VERSION.sles10 VERSION.sles10 VERSION.sles10 VERSION VERSION VERSION VERSION VERSION VERSION

7. 8. 9. Verify that all items that are selected to be installed in the default menu are correct. Select any items that are required and not selected by default. Press N. Displays screen 3 of 3 of the QLogic Inc. IB Install menu (Figure 2-4)
D000084-002 Rev A
2-7
QLogic Inc. IB Install (VERSION release) Menu Please Select Install Action (screen 3 of 3): 0) OFED iWARP [Don't Install][Available] VERSION 1) OFED Open SM [Don't Install][Available] VERSION 2) OFED Debug Info [Don't Install][Not Avail] N) P) R) X) Next Screen Perform the selected actions Re-Install All Return to Previous Menu (or ESC)

10. 11. 12. Verify that all items that are selected to be installed in the default menu are correct. Select any items that are required and not selected by default. Press P. Installs all of the software selected from all screens. The following statement displays:
About to Uninstall previous InfiniBand Software Installations... Hit any key to continue...
13.
Press any key. The system proceeds with the installation. During the installation, the following prompts will be displayed. For each prompt select the default by pressing enter.
Rebuild OFED SRPMs (a=all, p=prompt per SRPM, n=only as needed?) [n]: Permit non-root users to query the fabric? (y) Enable OFED SMI/GSI renice (RENICE_IB_MAD))? [y]: Install MPI with prefix compatible with mpi-selector (/usr/mpi/qlogic) [y]: Enable IPoIB Connected Mode (SET_IPOIB_CM)? [y]: Configure OFED IP over IB IPV4 addresses now? [n]:
2-8
D000084-002 Rev A
A
If
IPoIB will be enabled and run on the IB Management Node. IPoIB will not be run on the IB Management Node.
Then
Continue with Step 14 Skip to Step 21
14.
Type y and press ENTER to configure IPV4 addresses. System prompts:

How many OFED IP over IB interfaces would you like to configure? [1]:
15.
Type the number of IB interfaces that need to be configured and press ENTER. System prompts:
Use interface name ib0? [y]:
16.
Press ENTER. System prompts:

Enter IPV4 address in dot notation (or dhcp) for ib0:
17.
Enter IB address in dot notation format and press ENTER. System prompts:
Is IPV4 address '172.26.16.173' correct? (y/n):
18.
Type y and press ENTER. System prompts:

Enter IPV4 netmask in dot notation for ib0 172.26.16.173 [255.255.0.0]:
19.
Enter IB netmask in dot notation format and press ENTER. System prompts:
Is IPV4 netmask '255.255.240.0' correct? (y/n):
20.
Type y and press ENTER. The system proceeds with the installation. System prompts:
Enable OFED SRP High Availability deamon (SRPHA_ENABLE)? [n]:
D000084-002 Rev A
2-9
Skip to Step 22 21. Press ENTER to select default (n). The system displays ifcfg file requirements as shown below and proceeds with the installation.
OFED IP over IB requires an ifcfg file for each OFED IP over IB device instance. Manually create files such as '//etc/sysconfig/network-scripts/ifcfg-ib0'
System prompts:
22.
Press ENTER to select default (n). System prompts:

Enable OFED IB Stack (openibd) to autostart? [y]:
23.
Press ENTER to accept default. System prompts:

Enable IB Port Monitor (iba_mon) to autostart? [y]: .
NOTE: If Fast Fabric will be used to monitor fabric health and/or check the fabric for errors, its recommended to answer n to this prompt. 24. Press ENTER to accept default (Press N to adhere to the note above). System prompts:
Enable S20 Port Tuner (s20tune) to autostart? [y]:
25.
Press ENTER to accept default.
2-10
D000084-002 Rev A
The following series of prompts will allow selection of which drivers will be started at system boot time. It is recommended to press ENTER for a default answer of yes to each of the following prompts.
Enable QLogic SRP (qlgc_srp) to autostart? [y]: Enable QLogic Virtual NIC (qlgc_vnic) to autostart? [y]: Enable OFED IP over IB (openibd) to autostart? [y]: Enable OFED SDP (openibd) to autostart? [y]: Enable QLogic FM (iview_fm) to autostart? [y]: Enable QLogic FM SNMP Agent (iview_snmpd) to autostart? [y]: Enable OFED RDS (openibd) to autostart? [y]: Enable OFED SRP (openibd) to autostart? [y]: Hit any key to continue...
26.
Press any key. System prompts:

/usr/bin/qlgc_firmware_tool -i -l //var/log/iba.log Firmware is not required for the QLogic HCA(s) in this system. Press any key to continue.
The above lines display when using QLogic HCAs. Skip to Step 30 or
/usr/bin/qlgc_firmware_tool -i -l //var/log/iba.log One HCA was found:
The above line displays the number of HCAs found. The following line(s) list each HCA name and firmware, and displays one of the following scenarios:
D000084-002 Rev A
2-11
S
No firmware available: This will display if the HCA is not a Connect-X HCA (Example).
0: MT_0390140002 (MHGA28-XTC/A4/A0) Firmware : No firmware available. Contact your vendor for firmware updates for this HCA. No firmware available for HCAs in your system. Contact your vendor for firmware updates for this system. Press any key to continue.
Update available (Example):
0: MT_04A0110002 (MHGH28-XTC/X4/A0) Firmware 2.2.0: Update to 2.5.0 available. To update an HCA, or to install different firmware on an HCA, type its number. To quit, enter 'Q':
Firmware is up to date (Example):
0: MT_04A0110002 (MHGH28-XTC/X4/A0) Firmware 2.5.0: Okay.
To update an HCA, or to install different firmware on an HCA, type its number. To quit, enter 'Q':
If
No firmware is available You need to upgrade the firmware You do not need to upgrade the firmware Skip to Step 30
Then
Proceed with Step 27 Skip to Step 29
27.
Select a number corresponding to the HCA that needs to be upgraded. System prompts (Example):
MT_04A0110002 (MHGH28-XTC/X4/A0) Firmware 2.2.0 The following firmware revision(s) are available for this HCA: 0: MT_04A0110002: standard firmware Select firmware version, or Q to cancel:
28.
Select the number corresponding to the firmware revision required for the HCA. The firmware is installed on the HCA
2-12
D000084-002 Rev A
A
System prompts:
Install the Fabric Software Configure QLogic and SilverStorm IB Chassis
If
You need to upgrade the firmware in another HCA You do not need to upgrade the firmware on any other HCAs
Then
Repeat Steps 27 and 28 Continue with Step 29
29.
Press Q The installation completes and displays the main menu Skip to Step 31
30. 31. 32.
Press any key. The installation completes and displays the main menu: Press X to exit. Reboot the server.
Configure QLogic and SilverStorm IB Chassis

If the fabric contains QLogic 12000 or SilverStorm 9000 series internally managed IB switches, Fast Fabric is used to aid the installation and configuration of the switches.
QLogic and SilverStorm IB Chassis Configuration Pre-Requisites

Prior to using Fast Fabric, the following steps need to be performed. Refer to the QLogic 12000 Hardware Installation Guide and SilverStorm 9000 Hardware Installation Guide for further details: 1. (Switch) Connect each QLogic or SilverStorm chassis to the management network via its Ethernet management port. Chassis with redundant management should have both Ethernet management ports connected. (Switch) Assign each QLogic or SilverStorm chassis a unique IP address and appropriately configure the chassis Ethernet management port network settings.
2.
D000084-002 Rev A
2-13
3.
(Switch) Select a unique name which will be used for each QLogic or SilverStorm Chassis. This name should be configured in DNS or /etc/hosts as the TCP/IP name for the chassis Ethernet management port. When Virtual I/O controllers (VICs) are installed in a chassis, each VIC should also be assigned a unique IB Node Description via the chassis CLI or GUI. There is no need to set the Chassis IB Node Description at this time. It will be set in subsequent steps described below.
4.
(Switch) Configure the administrator password on each QLogic or SilverStorm Chassis. NOTE: All versions of QLogic 12000 chassis firmware and newer versions of SilverStorm 9000 chassis firmware permit SSH keys to be configured within the chassis for secure password-less login. To simplify the use of Fast Fabric to set up ssh security, it is recommended to install all chassis with the same admin password (it is simplest if the password is left as its default adminpass until after Fast Fabric has set up ssh keys for password-less login to the chassis by the IB Management Node). After ssh has been set up using Fast Fabric, it is recommended to change the admin passwords.
NOTE: When using versions of SilverStorm chassis firmware that do not support SSH keys, to simplify the use of Fast Fabric it is recommended to install all chassis with the same admin password. 5. (Switch) Copy the relevant chassis firmware files onto the Fast Fabric management node. During the steps below, the *.pkg files will be used to upgrade the firmware on each chassis. NOTE: When copying files it is best to place all files at a given firmware level into a single directory whose name indicates the firmware revision number.
2-14
D000084-002 Rev A
Configure Chassis Using Fast Fabric

Once the pre-requisites have been completed, configure the Chassis using Fast Fabric in the following procedure. NOTE: Refer to the Fast Fabric User Guide or more information on how to use the Fast Fabric TUI 1. (Switch) Type fastfabric and press ENTER. Displays the QLogic Fast Fabric Infiniband Tools menu (Figure 2-5)
QLogic Inc. Fast Fabric InfiniBand Tools Version: VERSION 1) 2) 3) 4) Chassis Setup/Admin Externally Managed Switch Setup/Admin Host Setup Host Verification/Admin
X) Exit
Figure 2-5 QLogic Fast Fabric InfiniBand Tools Menu (Example)

2. (Switch) Press 1. Displays the Fast Fabric IB Chassis Setup/Admin Menu (Figure 2-6).
D000084-002 Rev A
2-15
S
] ] ] ] ] ] ] ] ] ] ] ]
Fast Fabric IB Chassis Setup/Admin Menu Chassis List: /etc/sysconfig/iba/chassis Setup: 0) Edit Config and Select/Edit Chassis Files [ Skip 1) Verify Chassis via Ethernet ping [ Skip 2) Update Chassis Firmware [ Skip 3) Set up Chassis Basic Configuration [ Skip 4) Setup Password-less ssh/scp [ Skip 5) Reboot Chassis [ Skip 6) Configure Chassis Fabric Manager [ Skip Admin: 7) Check Status of IB Ports [ Skip 8) Control Chassis Fabric Manager [ Skip 9) Generate all Chassis Problem Report Info [ Skip a) Run a command on all chassis [ Skip Review: b) View iba_chassis_admin result files [ Skip P) Perform the selected actions X) Return to Previous Menu (or ESC)
N) Select None
Figure 2-6 QLogic Fast Fabric IB Chassis Setup/Admin Menu

3. 4. (Switch) Select the items in the Setup section of the menu above. Press P Perform the items that were selected in the sub-sections below.
Edit Config and Select/Edit Chassis Files

(Switch) The Edit Config and Select/Edit Chassis Files selection will permit the chassis, ports, and Fast Fabric configuration files to be edited. When placed in the editor for fastfabric.conf, review all the settings. Especially review the FF_CHASSIS_LOGIN_METHOD and FF_CHASSIS_ADMIN_PASSWORD. Consult Appendix B for more information about fastfabric.conf. NOTE: Fast Fabric will provide the opportunity to enter the chassis password interactively when needed. It is not necessary to place it within fastfabric.conf. If the QLogic chassis admin password is placed in fastfabric.conf, then change the fastfabric.conf permissions to be 0x600 (e.g., root-only access).
2-16
D000084-002 Rev A
NOTE: All versions of QLogic 12000 chassis firmware and newer versions of SilverStorm 9000 chassis firmware permit ssh keys to be configured within the chassis for secure password-less login, There is no need to configure a FF_CHASSIS_ADMIN_PASSWORD, and FF_CHASSIS_LOGIN_METHOD can be set to ssh (the default) when using the newer versions of the chassis firmware. When using older versions of SilverStorm 9000 chassis firmware, the FF_CHASSIS_LOGIN_METHOD must be set to telnet. Fast Fabric can set up these keys as described later in this section. Consult the SilverStorm 9000 Users Guide for more information. When placed in the editor for ports, review the file. For typical single-subnet clusters, the default of 0:0 may be used. This will use the first active IB port on the IB Management Node to access the fabric. For more information on configuring a cluster with multiple subnets, see Appendix C. For further details about the file format, refer to the Fast Fabric Command Line Interface Reference Guide. When placed in the editor for chassis, create the file with a list of the chassis names (the TCP/IP Ethernet management port names assigned above) or IP addresses (use of names is recommended). Enter one chassis name or IP address per line. For example:
Chassis1 Chassis2
NOTE: Do not list externally managed switches, such as the QLogic 12200 or SilverStorm 9024FC switches in this file. Those will be covered in the next section. For further details about the file format refer to the Fast Fabric Command Line Interface Reference Guide.
Verify Chassis via Ethernet ping

(Switch) The Verify Chassis via Ethernet ping selection will ping each selected chassis over the management network. If all chassis were found, continue to the next step. If some chassis were not found, abort out of the menu and review the following for those chassis which were not found: Is chassis powered on and booted Is chassis connected to management network
D000084-002 Rev A
2-17

Are chassis IP address and network settings consistent with DNS or

/etc/hosts
Is Management node connected to the management network Are Management node IP address and network settings correct Is the management network itself up (switches, routers, etc) Is correct set of chassis listed in the chassis file (the previous step may be repeated to review and edit the file as needed)?
Update Chassis Firmware

(Switch) The Update Chassis Firmware selection will permit the chassis firmware version to be verified and updated as needed. NOTE: The SilverStorm 9000 chassis must be running firmware version 4.0.0.4.3 or later to perform this function. If the chassis is not up to this level, it will need to be manually updated via the chassis GUI. See the SilverStorm 9000 Users Guide for more information.
NOTE: Consult the relevant chassis firmware release notes to ensure any prerequisites for the upgrade to the new firmware level have been met prior to performing the upgrade via Fast Fabric. 1. When this procedure is started the following message will be displayed:
Multiple Firmware files and/or Directories may be space separated Shell wildcards may be used For Directories all .pkg files in the directory tree will be used Enter Files/Directories to use (or none):
2.
Specify the directory where the relevant firmware files have been stored and press ENTER. This can be the mount point of the CD or the directory to which the files were copied in a previous step. Prompt displays:
Would you like to run the firmware now? [n]:
3.
Type y and press ENTER since the fabric is not yet operational. Fast Fabric will ensure that all chassis are running the firmware level provided and install and/or reboot each chassis as needed.
2-18
D000084-002 Rev A
A
Set Up Chassis Basic Configuration
If any chassis fails to be updated, use the View iba_chassis_admin result files option to review the result files from the update. Refer to the Fast Fabric Command Line Interface Reference Guide for more details. (Switch) The Setup Chassis Basic Configuration will permit the typical chassis setup operations to be performed for all chassis. Perform the following procedure: 1. When this procedure is started the following message will be displayed:
Do you wish to configure a syslog server? [y]:
2.
Press ENTER to select default (y). The following message will display:
Enter IP address for syslog server:
3.
Enter the IP address of a syslog server which is to receive log messages from all chassis. The following message will display:
Do you wish to configure an NTP server? [y]:
4.
Enter IP address for NTP server:
5.
Enter the IP address of an NTP server which can supply a consistent time base for use by all chassis. The following message will display:
Do you wish to configure timezone and DST information? [y]:
6.
Do you want to use the local timezone information from the local server? [y]:
7.
Press ENTER to select default (y). This will cause the time zone of the local server (e.g., the IB Management Node) to be replicated to all the chassis to specify their time zones.
D000084-002 Rev A
2-19
The following message will display:

Do you wish to configure the chassis maximum packet MTU size? [n]:
8.
Press ENTER to select default (n). This will cause the default MTU of 2048 to be used for all chassis. If chassis have previously been manually configured for a different MTU size, this option will force all chassis back to a consistent MTU size. See the QLogic 12000 Users Guide or SilverStorm 9000 Users Guide for more information. The following message will display:
Do you wish to configure IB Node Desc to match ethernet chassis name? [y]:
9.
Press ENTER to select default (y). This will cause the chassis name entered in the /etc/sysconfig/iba/chassis file to be used as the IB Node Description for the chassis, making the management network and IB network names for the chassis consistent. If the /etc/sysconfig/iba/chassis file has IP addresses instead of names, enter n to this question The following message will display:
Do you wish to configure IB Node Desc Format? [y]:
10.
Press ENTER to select default (y) The following message will display:
Do you wish to use concise IB Node Desc format? [y]:
11.
Press ENTER to select default (y). This will cause the chassis IB Node Descriptions to use concise naming for the Leafs and Spines such as L01 or S01A (as opposed to "Leaf 1" or "Spine 1, Chip A").
Setup Password-less ssh/scp

(Switch) The Setup Password-less ssh/scp selection will set up secure password-less ssh such that the IB Management Node can securely log in to all the chassis as admin via the management network without requiring a password.
2-20
D000084-002 Rev A
A
Reboot Chassis
(Switch) The Reboot Chassis selection will reboot all the selected chassis and ensure they go down and come back up (as verified via ping over the management network). When the chassis come back up, they will be running with all the new configuration settings.
Configure Chassis Fabric Manager

(Switch) The Configure Chassis Fabric Manager selection will assist in configuring the Fabric Manager for any QLogic 12000 chassis with appropriate license keys. You will first be prompted with:
Performing Chassis Admin: Configure Chassis Fabric Manager Enter FM Config file to use (or none or generate):
It is recommended to enter generate. This will perform the config_generate operation to guide the user through selecting FM configuration options. See the QLogic Fabric Manager Users Guide for more information about config_generate. After responding to the prompts for config_generate, the following message will display:
You have selected to use: ./qlogic_fm.xml Syntax Checking ./qlogic_fm.xml... Executing: /opt/iba/fm_tools/config_check -s -c ./qlogic_fm.xml Valid FM Config file: ./qlogic_fm.xml After push, the FM may be started/restarted Would you like to restart the FM? [n]:
It is recommended to select y, this will cause the FM to be started with the new configuration. The following message will display:
Would you like to run the FM on slave MMs? [n]:
In a fabric with only a single chassis running the FM, it is recommended to select y, this will cause the FM also be started on slave MMs in the applicable chassis. However in a fabric with multiple chassis running the FM, it is recommended to select n, this will cause the FM to only be started on the master MM in the applicable chassis.
D000084-002 Rev A
2-21

There will be a disruption as FMs are restarted Doing the operation in parallel (on multiple chassis) will finish the fastest Doing it serially may reduce disruption Would you like to do the operation in parallel? [y]:
It is recommended to press ENTER to select the default y. The following message will display:
You have selected to perform the push, and FM restart in parallel Would you like to enable FM start at boot? [n]:
It is recommended to select y, this will cause the FM to be started on all applicable chassis each time those chassis boot. The following message will display:
Would you like to enable FM start on slave MMs at boot? [n]:
In a fabric with only a single chassis running the FM, it is recommended to select y, this will cause the FM also be started at chassis boot on slave MMs in the applicable chassis. However in a fabric with multiple chassis running the FM, it is recommended to select n, this will cause the FM to only be started on the master MM in the applicable chassis. The following message will display:
Would you like to be prompted for chassis' password? [n]:
It is recommended to press ENTER to select the default n.
Run a command on all chassis

(Switch) If there are any other operations which need to be performed on all chassis, they may be performed using the Run a command on all chassis option. Each time this is executed, a single chassis CLI command may be specified to be executed against all selected chassis. When using these commands, additional setup or verification of the chassis may be performed.
2-22
D000084-002 Rev A
Install the Fabric Software Install and Configure the Fabric Manager
Install and Configure the Fabric Manager

(All) At this point the Fabric Manager (FM) for the fabric must be enabled. If using a Host FM, section Install QLogicIB-IFS on page 2-5 installed and enabled the QLogic FM using the default configuration file. If using an embedded QLogic Fabric Manager the, Configure Chassis Fabric Manager on page 2-21 will have configured the QLogic Fabric Manager. Refer to the QLogic Fabric Manager Users Guide for information on how to configure the FM. When using the QLogic Fabric Manager, a typical installation will place the Fast Fabric and the FM on the same IB Management Node. If desired, it is also valid to place Fast Fabric on its own independent management node, perhaps along with other 3rd party management applications (such as MPI job schedulers, etc). The following procedures require that a subnet manager be operational within the fabric.
Configure Firmware on the Externally Managed IB Switches

If the fabric contains QLogic 12200 or SilverStorm 9024FC series externally managed switches, Fast Fabric is used to aid in the installation and configuration of the switches.
IB Switches Configuration Pre-Requisites

Prior to using Fast Fabric, the following minimal steps need to be performed: 1. (Switch) Select a unique name to be used for each Switch. This name will be configured as the IB Node Description for the switch in the steps below. NOTE: Externally managed switches do not have an Ethernet port and therefore will not have a TCP/IP name. 2. (Switch) Copy the relevant switch firmware files onto the Fast Fabric management node. For the steps below, the *.emfw files will be used to upgrade the firmware on each switch. NOTE: When copying files, it is best to place all files at a given firmware level into a single directory whose name indicates the firmware revision number
D000084-002 Rev A
2-23
Install the Fabric Software Configure Firmware on the Externally Managed IB Switches
Configure Externally Managed Switches

Once the pre-requisites have been completed, configure the switches using Fast Fabric in the following procedure. 1. 2. (Switch) If the QLogic Fast Fabric Infiniband Tools menu is not displayed type fastfabric and press ENTER. (Switch) Press 2. Displays the Fast Fabric IB Switch Setup/Admin Menu (Figure 2-6).
Fast Fabric IB Switch Setup/Admin Menu Externally Managed Switch List: /etc/sysconfig/iba/ibnodes Setup: 0) Edit Config and Select/Edit Switch Files 1) Test for Switch Presence 2) Verify Switch Firmware 3) Update Switch Firmware 4) Setup Switch Basic Configuration 5) Reboot Switch 6) Report Switch Firmware & Hardware Info Admin: 7) Report Switch VPD Information Review: 8) View iba_switch_admin result files P) Perform the selected actions X) Return to Previous Menu (or ESC)
[ [ [ [ [ [ [
Skip Skip Skip Skip Skip Skip Skip
] ] ] ] ] ] ] ] ]
[ Skip [ Skip
N) Select None
Figure 2-7 QLogic Fast Fabric IB Switch Setup/Admin Menu

3. 4. 5. (Switch) Select the items 0 through 4 in the Setup section of the menu. Press P. Perform the items selected above in the sub-sections below.
Edit Config and Select/Edit Switch Files

(Switch) The Edit Config and Select/Edit Switch Files selection will permit the ibnodes, ports, and Fast Fabric configuration files to be edited. When placed in the editor for fastfabric.conf, review all the settings. Refer to Appendix B for more information about fastfabric.conf.
2-24
D000084-002 Rev A
When placed in the editor for ports, review the file. For typical single-subnet clusters, the default of 0:0 may be used. This will use the first active IB port on the IB Management Node to access all externally managed switches. For more information on configuring a cluster with multiple subnets, see Appendix C. For further details about the file format, refer to the Fast Fabric Command Line Interface Reference Guide. When placed in the editor for ibnodes, create the file with a list of the switch node GUID and desired switch names, Enter one switch node GUID and desired switch name per line. For example:
0x00066a00d9000138,edge1 0x00066a00d9000139,edge2
NOTE: Per the example above, when typing a new name, do not use any spaces before or after the comma.
NOTE: Do not list internally managed chassis, such as the QLogic 12000 or SilverStorm 9000 chassis in this file. Those were covered in a previous section.
NOTE: If the IB path from the IB Management node to other switch nodes is through a 12200 or 9024FC which is to be updated, the ibnodes file should omit that 12200 or 9024FC switch at this time. Otherwise, the reboot for the 12200 or 9024FC in the path could disrupt the updates of other switches. For further details about the file format, refer to the Fast Fabric Command Line Interface Reference Guide. If needed, an SA query such as the following can be used to get a list of all switches. This includes both internally and externally managed switches, and consequently the output must be edited to leave only the QLogic and SilverStorm externally managed switches:
iba_saquery -t sw -o nodeguid
D000084-002 Rev A
2-25
Test for Switch Presence

(Switch) The Test for Switch Presence selection will verify that each Externally Managed Switch specified in the ibnodes file can be accessed by the IB Management Node via the IB Network. If all switches were found, continue to the next step. If some switches were not found, abort out of the menu and review the following for those switches which were not found: Is switch powered on and booted Is switch connected to IB Fabric Is Subnet Manager running Is IB Management nodes IB Port active Is IB Management node connected to the correct IB Fabric Is correct set of switches listed in the ibnodes file (the previous step may be repeated to review and edit the file as needed)?
Verify Switch Firmware

(Switch) The Verify Switch Firmware selection will verify each externally managed switch is operational and its firmware is valid and accessible.
Update Switch Firmware

(Switch) The Update Switch Firmware selection will permit the switch firmware version to be updated and the switch node name set. NOTE: Consult the relevant switch firmware release notes to ensure any prerequisites for the upgrade to the new firmware level have been met prior to performing the upgrade via Fast Fabric. Perform the following procedure: 1. When this procedure is started the following message will be displayed:
Multiple Firmware files and/or Directories may be space separated Shell wildcards may be used For Directories all .emfw files in the directory tree will be used Enter Files/Directories to use (or none):
2.
Specify the directory where the relevant firmware files have been stored. This can be the mount point of the CD or the directory to which the files were copied in a previous step.
2-26
D000084-002 Rev A
A
3. Enter y

After upgrade, the switch may be optionally rebooted Would you like to reboot the switch after the update? [n]:
The fabric is not yet operational The following message will display:
You have selected to update the switch firmware and reboot. There will be a disruption as switch or switches are rebooted Doing the operation in parallel (on multiple switches) will finish the fastest Doing it serially may reduce disruption Would you like to do the operation in parallel? [y]:
4.
If there are no selected externally managed switches in the path from the IB Management node to any other externally managed switch (for example, if the IB Management node is connected directly to a core switch and externally managed switches are only at the edges), parallel operations can be established: Press ENTER. or Type n and press ENTER if in doubt. NOTE: Be aware that non-parallel operation for a fabric with many externally managed switches could take a significant amount of time. Fast Fabric will update the firmware on all switches and set the node names as per the ibnodes file created in a previous step. Each switch will then be rebooted. If any switch fails to be updated, use the View iba_switch_admin result files option to review the result files from the update. Refer to the Fast Fabric Command Line Interface Reference Guide for more details.
Set Up Switch Basic Configuration

(Switch) The Setup Switch Basic Configuration will permit the typical switch setup operations to be performed for all switch.
D000084-002 Rev A
2-27
Perform the following procedure: 1. When this procedure is started the following message will be displayed:
Do you wish to configure the switch maximum MTU size? [n]:
2.
Press ENTER to select default (n). This will cause the default MTU of 2048 to be used for all switches. If chassis have previously been manually configured for a different MTU size, this option will force all switches back to a consistent MTU size. See the QLogic 12000 Users Guide for more information. The following message will display:
Do you wish to configure the switch VL Capability? [n]:
3.
Press ENTER to select default (n). This will cause the default VL Capability of 1 data VL to be used for all switches. If switches have previously been manually configured for a different VL Capability, this option will force all switches back to a consistent VL Capability. See the QLogic 12000 Users Guide for more information. NOTE: This operation is only applicable to QLogic 12000 switches. Any SilverStorm 9024FC switches specified will be skipped for this operation.
Reboot Switch
(Switch) The Reboot Switch will reboot all switches, this will ensure that all the configuration changes become effective and are discovered by the Fabric Manager.
Report Switch Firmware and Hardware Info

(Switch) The Report Switch Firmware and Hardware Info selection will report the firmware and hardware versions for each switch, along with the Capability (DDR or SDR), Fan Status and Power Supply Status. Review the results against the expected models and firmware versions. (Switch) If any 12200 or 9024FC switches were skipped, these sections should be repeated for those switches. In this case it is recommended to create a separate file with a name other than ibnodes. An alternate name may be specified at the prompt:
Select Switch File to Use/Edit [/etc/sysconfig/iba/ibnodes]:
2-28
D000084-002 Rev A
Install the Fabric Software Install InfiniBand on the Remaining Servers
Install InfiniBand on the Remaining Servers

Fast Fabric may now be used to install and configure the remaining hosts and verify overall operation of the fabric. NOTE: The procedure presented below is for the QLogic IB packaging of OFED or the QuickSilver IB Stack. Fast Fabric may also be used to install the IB Stack Tools on the remaining hosts when using other variations of OFED. In this case, OFED must be installed on each host manually. 1. 2. (All) If the QLogic Fast Fabric Infiniband Tools menu is not displayed, type fastfabric and press ENTER. (Linux) Press 3. Displays the Fast Fabric IB Host Setup Menu (Figure 2-8).
Fast Fabric IB Host Setup Menu Host List: /etc/sysconfig/iba/hosts Setup: 0) Edit Config and Select/Edit Hosts Files 1) Verify Hosts via Ethernet ping 2) Setup Password-less ssh/scp 3) Copy /etc/hosts to all hosts 4) Show uname -a for all hosts 5) Install/Upgrade QLogic IB Software 6) Configure IPoIB IP Address 7) Build MPI Test Apps and Copy to Hosts 8) Reboot Hosts Admin: 9) Refresh ssh Known Hosts a) Rebuild MPI Library and Tools b) Run a command on all hosts c) Copy a file to all hosts Review: d) View iba_host_admin result files P) Perform the selected actions X) Return to Previous Menu (or ESC)
[ [ [ [ [ [ [ [ [ [ [ [ [
Skip Skip Skip Skip Skip Skip Skip Skip Skip Skip Skip Skip Skip
] ] ] ] ] ] ] ] ] ] ] ] ] ]
[ Skip
N) Select None
Figure 2-8 QLogic Fast Fabric IB Host Setup Menu

3. Select items 0 through 2 and 4 through 8.
D000084-002 Rev A
2-29
4.
Press P. NOTE: If /etc/hosts will be used for name resolution (as opposed to using DNS), also select Copy /etc/hosts to all hosts
5.
Perform the items selected above in the sub-sections below.
Edit Config and Select/Edit Hosts Files

(All) The Edit Config and Select/Edit Hosts Files selection will permit the hosts and Fast Fabric configuration files to be edited. When placed in the editor for fastfabric.conf, review all the settings. Especially review the FF_IPOIB_SUFFIX, ff_host_basename_to_ipoib, ff_host_basename, FF_IPOIB_NETMASK, FF_PRODUCT, FF_PACKAGES, FF_INSTALL_OPTIONS, FF_UPGRADE_OPTIONS, and FF_ALL_ANALYSIS files. Consult Appendix B for more information about fastfabric.conf.
.
NOTE: During setup of password-less ssh, Fast Fabric will provide the opportunity to enter the host root password interactively when needed. Therefore, it is recommended not to place it within fastfabric.conf. If it is desired to instead keep the root password for the hosts in fastfabric.conf, its recommended to change the fastfabric.conf permissions to be 0x600 (e.g. root-only access). When placed in the editor for hosts, create the file with a list of the hosts names (the TCP/IP management network names) except the IB Management node from which Fast Fabric is presently being run, Enter one hosts name per line. For example:
host1 host2
NOTE: Do not list the IB Management Node itself (i.e., the node where Fast Fabric is currently running). If additional IB Management Nodes are to be used, they may be listed at this time and Fast Fabric can aid in their initial installation and verification. For further details about the file format, refer to the Fast Fabric Command Line Interface Reference Guide.
2-30
D000084-002 Rev A
A
Verify Hosts via Ethernet ping
(All) The Verify Hosts via Ethernet ping selection will ping each selected host over the management network. If all hosts were found, continue to the next step. If some hosts were not found, abort out of the menu and review the following for those hosts which were not found: Is host powered on and booted Is host connected to management network Is host management network IP address and network settings consistent with DNS or /etc/hosts Is management node connected to the management network Are management node IP address and network settings correct Is management network itself up (switches, routers, etc) Is correct set of hosts listed in the hosts file (the previous step may be repeated to review and edit the file as needed)?
Setup Password-less ssh/scp

(Linux) The Setup Password-less ssh/scp section will set up secure password-less ssh such that the IB Management Node can securely log in to all the other hosts as root via the management network without requiring a password. Password-less ssh is required by Fast Fabric, MPI test applications and most versions of MPI (including QuickSilver MPI, OFED openmpi, OFED mvapich, and OFED mvapich2).
Copy /etc/hosts to all hosts

(Linux) The Copy /etc/hosts to all hosts section will copy the /etc/hosts file on this host to all the other selected hosts. NOTE: If DNS is being used, this step should be skipped.
NOTE: Typically, /etc/resolv.conf is set up as part of OS installation for each host. However, if /etc/resolv.conf was not setup on all the hosts during OS installation, the Fast Fabric Copy a file to all hosts operation could be done at this time to copy /etc/resolv.conf from the IB Management Node to all the other nodes.
D000084-002 Rev A
2-31
Show uname -a for all hosts

(Linux) The Show uname -a for all hosts selection will show the OS version on all the hosts. Review the results carefully to verify all the hosts have the expected OS version. In typical clusters, all hosts will be running the same OS and kernel version. If any hosts are identified with an incorrect OS version, the OS on those hosts should be corrected at this time and operation of this sequence should be aborted when prompted. As necessary, all the preceding setup steps should then be repeated for those hosts (there is no harm in repeating them for all the hosts).
Install/Upgrade QLogic IB Software

(Host) The Install/Upgrade QLogic IB Software selection will install the IB software on all the hosts. By default it will look in the current directory for the QLogicIB-Basic.DESTRO.VERSION.tgz file. If it is not found in the current directory, it will prompt for input of a directory name where this file can be found. When prompted, select to do an initial installation as follows:
Would you like to do an upgrade install? [y]: n Would you like to do an initial install/load? [n]: y
NOTE: An initial installation will uninstall any existing QLogic IB OFED, QuickSilver or OFED software on the selected hosts. An upgrade install is not appropriate at this step. If any hosts fail to be installed, use the View iba_host_admin result files option to review the result files from the update. For more details, refer to the Fast Fabric Command Line Interface Reference Guide.
Configure IPoIB IP Address

(Host) The Configure IPoIB IP Address selection will create the ifcfg-ib0 files on each host (previous non-OFED releases created the ifcfg-ib1 file). The file will be created with a statically assigned IP address. The IPoIB IP address for each host will be determined by the resolver (Linux host command). If not found via the resolver, /etc/hosts on the given host will be checked.
2-32
D000084-002 Rev A
Install the Fabric Software Verifying InfiniBand on the Remaining Servers
Build MPI Test Apps and Copy to Hosts

(Host) The Build MPI Test Apps and Copy to Hosts selection will build the MPI sample benchmarks on the IB Management Node and copy the resulting object files to all the hosts. This is in preparation for execution of MPI performance tests and benchmarks in a later step. NOTE: This option is available for the QLogic IB packaging of OFED, but is not presently available for other packagings of OFED.
Reboot Hosts
(Linux) The Reboot Hosts selection will reboot all the selected hosts and ensure they go down and come back up (as verified via ping over the management network). When the hosts come back up, they will be running the IB software installed.
Run a command on all hosts

(Linux) If there are any other setup operations which need to be performed on all hosts, they may be performed using the Run a command on all hosts option. Each time this is executed a Linux shell command (or sequence of commands separated by semicolons) may be specified to be executed against all selected hosts. NOTE: It is recommended at this time to run the date command to verify the date and time is consistent on all hosts. If needed Copy a file to all hosts option may be used to copy the appropriate files to all hosts to enable and configure NTP.
Verifying InfiniBand on the Remaining Servers

Upon completion of the preceding sections, the hosts are all booted, installed and operational. The subsequent steps will verify the operation of the hosts and fabric. 1. 2. (All) If the QLogic Fast Fabric Infiniband Tools menu is not displayed type fastfabric and press ENTER. (All) Press 4. Displays the Fast Fabric IB Host Verification/Admin Menu (Figure 2-8).
D000084-002 Rev A
2-33
S
[ [ [ [ [ [ [ [ [ Skip Skip Skip Skip Skip Skip Skip Skip Skip ] ] ] ] ] ] ] ] ] ] ] ]
Fast Fabric IB Host Verification/Admin Menu Host List: /etc/sysconfig/iba/allhosts Validation: 0) Edit Config and Select/Edit Hosts Files 1) Verify Hosts via Ethernet ping 2) Summary of Fabric Components 3) Check Status of IB Ports 4) Verify Hosts see each other 5) Verify Hosts ping via IPoIB 6) Refresh ssh Known Hosts 7) Check MPI Performance 8) Check Overall Fabric Health Admin: 9) Generate all Hosts Problem Report Info a) Run a command on all hosts Review: b) View iba_host_admin result files P) Perform the selected actions X) Return to Previous Menu (or ESC)
[ Skip [ Skip [ Skip
N) Select None
Figure 2-9 QLogic Fast Fabric IB Host Verification/Admin Menu

3. 4. Select the items 0 through 8 in the Validation section of the menu Press P.
Edit Config and Select/Edit Hosts Files

(All) The Edit Config and Select/Edit Hosts Files section will permit the hosts, ports, and Fast Fabric configuration files to be edited. When placed in the editor for fastfabric.conf, review all the settings. Especially review the FF_TOPOLOGY_FILE, FF_IPOIB_SUFFIX, ff_host_basename_to_ipoib, and ff_host_basename. Refer to Appendix B for more information about fastfabric.conf. If desired, a Fast Fabric topology file may be created as /etc/sysconfig/iba/topology.0:0.xml to describe the intended topology of the fabric and augment assorted fabric reports with customer-specific information such as cable labels and additional details about nodes, SMs, links, ports and cables, etc. Refer to the Fast Fabric Command Line Interface Reference Guide for more information about topology verification files. Also review the following parameters which will be used for overall fabric health checks: FF_ANALYSIS_DIR, FF_ALL_ANALYSIS, FF_FABRIC_HEALTH, FF_CHASSIS_CMDS, FF_CHASSIS_HEALTH, and FF_ESM_CMDS. FF_ALL_ANALYSIS should be updated to reflect the type of SM (esm or hostsm).
2-34
D000084-002 Rev A
When placed in the editor for ports, review the file. For typical single-subnet clusters, the default of 0:0 may be used. This will use the first active IB port on the IB Management node to access the fabric. For more information on configuring a cluster with multiple subnets, see Appendix C. For further details about the file format, refer to the Selection of Ports section in the Fast Fabric Command Line Interface Reference Guide. When placed in the editor for allhosts, create the file with the IB Management node's hosts names (the TCP/IP management network names) (shown as mgmthost below for example) and include the hosts file previously created, enter one per line. For example:
mgmthost include /etc/sysconfig/iba/hosts
For further details about the file format refer to the Selection of Hosts section in the Fast Fabric Command Line Interface Reference Guide.
Verify Hosts via Ethernet ping

(All) The Verify Hosts via Ethernet ping selection will ping each selected host over the management network. If all hosts were found, continue to the next step. If some hosts were not found, abort out of the menu and review the following for those hosts which were not found: Is the host powered on and booted Is the host connected to management network Is the host management network IP address and network settings consistent with DNS or /etc/hosts Is the Management node connected to the management network Are the Management node IP address and network settings correct Is the management network itself up (switches, routers, etc) Is the correct set of hosts listed in the hosts file (the previous step may be repeated to review and edit the file as needed)?
Summary of Fabric Components

(All) The Summary of Fabric Components selection will provide a brief summary of the counts of components in the fabric, including how many switch chips, hosts, and links are in the fabric. It will also indicate if any 1x links were found (which could indicate a poorly seated or bad cable). Review the results against the expected configuration of the cluster. If components are missing or 1x links are found, they should be corrected. Subsequent steps will aid in locating any 1x links.
D000084-002 Rev A
2-35
Check Status of IB Ports

(Host) The Check Status of IB Ports selection will allow the state and symbol error counts of all ports to be checked and reviewed. When prompted, select to do error and link speed analysis select the defaults by pressing enter at each prompts:
Would you like to perform fabric error analysis? [y]: Clear error counters after generating report? [y]: Would you like to perform fabric link speed error analysis? [y]: Check for links configured to run slower than supported? [n]: Check for links connected with mismatched speed potential? [n]:
This will check all the ports in the fabric for any links which have high error rates or are running at a lower speed than expected. Any such identified links should be diagnosed and corrected. NOTE: If the fabric is homogeneous and all links are expected to be running at full speed, answer y to the last two questions.
Verify Hosts see each other

(Host) The Verify Hosts see each other selection will verify that each host can see all the others via queries to the Subnet Administrator.
Verify Hosts ping via IPoIB

(Host) The Verify Hosts ping via IPoIB selection will verify that IPoIB is properly configured and running on all the hosts. This is accomplished via the IB management node pinging each host via IPoIB. NOTE: Use of this operation requires that IPoIB be enabled on the IB Management Node as well as each host selected for verification. 1) The management host needs to have ipoib configured 2) Depending on the MTU of the fabric, this may not be successful.
2-36
D000084-002 Rev A
A
Refresh ssh Known Hosts
(Linux) The Refresh ssh Known Hosts selection will refresh the ssh known_hosts file on the IB management node to include the IPoIB hostnames of all the hosts. NOTE: Use of this operation requires that IPoIB be enabled on the IB Management Node as well as each host selected for verification.
Check MPI Performance

(Host) The MPI Performance selection will do a quick check of PCI and MPI performance via end-to-end latency and bandwidth tests. NOTE: This option is available for the QLogicIB packaging of OFED, but is not presently available for other packagings of OFED. When MPI Performance is selected it displays a prompt as follows:
Test Latency and Bandwidth deviation between all hosts? [y]:
At the prompt press ENTER to select default (y) This displays the results of pairwise analysis of latency and bandwidth for the selected hosts and reports pairs outside an acceptable tolerance range. By default performance is compared relative to other hosts in the fabric (with the assumption that all hosts selected for a given run should have comparable IB performance). Failing hosts will be clearly indicated. The results are also written to the test.res file which may be viewed via the View iba_host_admin result files option. Refer to the Fast Fabric Command Line Interface Reference Guide for more details. If any hosts fail, carefully examine the failing hosts to verify the HCA models, PCI slot used, BIOS settings and any motherboard jumpers related to devices on PCI buses or slot speeds. Also verify the HCA and riser cards are properly seated. The bandwidth reported should also be checked against the practical PCI speeds in the Performance Impact table (Table 2-1). If all pairs are not in the expected performance range, carefully examine all hosts to verify the HCA models, PCI slot used, BIOS settings and any motherboard jumpers related to devices on PCI buses or slot speeds.
D000084-002 Rev A
2-37
Install the Fabric Software Complete Installation of additional IB Management Nodes
S
Theoretical Max
2000MB/sec 1000MB/sec 1000MB/sec 1000MB/sec 1064MB/sec 8000MB/sec 532MB/sec
Table 2-1. Performance Impact PCI Speed

PCIe x8 PCIe x4 PCIe x8 PCIe x4 133 100 66
IB Speed
DDR DDR SDR SDR SDR SDR SDR
Practical Bandwidth
1300-1500 MB/sec 800-900 MB/sec 900-1000 MB/sec 800-900 MB/sec 800-900 MB/sec 600-680 MB/sec 400-450 MB/sec
Check Overall Fabric Health

(ALL) The Check Overall Fabric Health selection will permit the present fabric configuration to be baselined for use in future fabric health checks. This should be performed after configuring any additional IB management Nodes. See Configure and Initialize Health Check Tools on page 2-40 for more information.
Complete Installation of additional IB Management Nodes

If the fabric is to have more than one IB Management Node, the setup of the additional management nodes may be completed now. The previous steps will have performed basic software installation, setup and verification on those nodes. Now the management software itself must be installed and configured. NOTE: The steps below assume a symmetrical configuration where all IB management nodes have the same connectivity and capabilities. In asymmetrical configurations where the IB management nodes are not all connected to the same set of management networks and IB subnets, the files copied to each management node may need to be slightly different. For example, configuration files for fabric_analysis may indicate different port numbers, or host files used for Fast Fabric and MPI may need to list different hosts. For multiple-subnet configurations, refer to Multi-Subnet Fabrics on page C-1. Repeat the following steps on each additional IB Management Node:
2-38
D000084-002 Rev A
A
1.
Install the Fabric Software Complete Installation of additional IB Management Nodes
(All) Upgrade the QLogicIB- Basic to QLogicIB-IFS software to add additional components using the procedure documented in Appendix F. The IB Management node must have at least Fast Fabric, the IB Stack and should have IPoIB installed and configured. For MPI clusters the IB Management node should also include at least OFED openmpi, OFED mvapich, or OFED mvapich2, and if the user desires to rebuild MPI itself, the OFED IB Development package and MPI Source packages will also be required. NOTE: Do not uninstall or replace existing configuration files which were previous created, especially IPoIB-related configuration files.
2.
(All) Copy the Fast Fabric configuration files from the initial IB Management Node. At least the following files should be copied:
/etc/sysconfig/fastfabric.conf /etc/sysconfig/iba/ports /etc/sysconfig/iba/topology*.xml /etc/sysconfig/iba/hosts /etc/sysconfig/iba/allhosts /etc/sysconfig/iba/ibnodes /etc/sysconfig/iba/chassis
After copying the files, edit the hosts and allhosts files such that the file on each IB Management node omits itself from the hosts files (but lists all other IB Management nodes) and specifies itself in the allhosts file. See Appendix B for a complete list of Fast Fabric configuration files. 3. (All) If the QLogic Fabric Manager is also going to be run, copy the QLogic FM configuration file (/etc/sysconfig/qlogic_fm.xml) from the initial IB Management Node. After copying the file, edit the file on each IB Management node as needed. Consult the QLogic Fabric Manager Users Guide for more information on how to configure the FM. 4. (Linux) Perform Setup Password-less ssh/scp option in the Host Setup via Fast Fabric menu and Refresh ssh Known Hosts option in the Host Admin via Fast Fabric menu.
D000084-002 Rev A
2-39
Install the Fabric Software Configure and Initialize Health Check Tools
Configure and Initialize Health Check Tools

For more information on the health check tools, see the detailed discussion in Fast Fabric Command Line Interface Reference Guide. The Health check tools may be run on one or more IB management nodes within the cluster. This procedure should be followed on each IB management node from which the health check tools will be used. 1. (All) Edit fastfabric.conf and review the following parameters: FF_ANALYSIS_DIR, FF_ALL_ANALYSIS, FF_FABRIC_HEALTH, FF_CHASSIS_CMDS, FF_CHASSIS_HEALTH, and FF_ESM_CMDS. FF_ALL_ANALYSIS should be updated to reflect the type of SM (esm or hsm).
/etc/sysconfig/iba/esm_chassis
2.
(All) If using Embedded SM(s) in QLogic IB Chassis, create listing the chassis which are running SMs. Create the file with a list of the chassis names (the TCP/IP Ethernet management port names assigned above) or IP addresses (use of names is recommended). Enter one name or IP address per line. For example:
Chassis1 Chassis2
For further details about the file format, refer to Fast Fabric Command Line Interface Reference Guide. 3. (All) Perform a health check using: all_analysis -e. If any errors are encountered, resolve the errors and rerun all_analysis -e until a clean run occurs. (All) Create a cluster configuration baseline using: all_analysis -b. NOTE: This may also be done using the Fast Fabric menu system by selecting Check Overall Fabric Health and enter y to Baseline present configuration? [n]: 5. (All) If desired, schedule regular runs of all_analysis via cron or other mechanisms. Consult the Linux OS documentation for more information on cron. Also consult the Fast Fabric Command Line Interface Reference Guide for more information about all_analysis and its automated use.
4.
2-40
D000084-002 Rev A
Install the Fabric Software Running High Performance Linpack
Running High Performance Linpack

As part of the installation process, a set of common MPI benchmarks have been installed. One of the more popular measures of overall performance is High Performance Linpack (HPL). This is the application used to rate systems on the Top 500 list. The steps allow some initial runs of HPL to be made and provide some initial baseline numbers. The defaults provided should perform within 10 20% of optimal HPL results for the cluster. Tuning for that additional 10 20% is beyond the scope of this document. 1. (Host) To run HPL, first select a configuration file appropriate to your cluster. It is best to start with a small configuration to verify HPL has been properly compiled:
cd /opt/iba/src/mpi_apps ./config_hpl 2t
will configure a two process test run of HPL. 2. 3. (Host) Create the file /opt/iba/src/mpi_apps/mpi_hosts listing the host names of all the hosts. (Host) Run HPL:
./run_hpl 2
Since this is a very small problem size, the performance of the run will be much lower than the potential of the machine. So do not worry about performance, just whether or not the run was successful. At this point the user is ready to move onto full scale HPL runs. Assorted sample files are provided in /opt/iba/src/mpi_apps/hpl-config. These files are a good starting point for most clusters and should get within 10 20% of the optimal performance for the cluster. The problem sizes used assume a cluster with 1GB of physical memory per processor (e.g., for a 2 processor node, 2GB of node memory is assumed). For each cluster size, 4 files are provided:
HPL.dat
t - a very small test run (5000 problem size) s - a small problem size on the low end of optimal problem sizes m - a medium problem size l - a large problem size These can be selected using config_hpl. The following command displays the pre-configured problem sizes available:
./config_hpl
For example, to do a small run for a 256 processor cluster (i.e., 128 nodes of dual CPU systems):
D000084-002 Rev A
2-41
Install the Fabric Software Running High Performance Linpack
1. 2.
Type ./config_hpl 256s and press ENTER. Type ./run_hpl 256 and press ENTER. During these runs the user should use top on a node to monitor memory and CPU usage. The xhpl should use 98 99% of the CPU. If any other processes are taking more than 1 2%, review the host configuration and stop these extra processes if possible. HPL is very sensitive to swapping. If a lot of swapping is seen, and xhpl is dropping below 97% for long durations, this may indicate a problem size that is too large for the memory and OS configuration. At this point the user can continue to tune HPL to refine performance. Parameters in HPL.dat can all affect HPL performance. In addition, the selection of compiler and BLAS Math library may also significantly affect performance. The new HPL.dat files may be placed in /opt/iba/src/mpi_apps/hpl-config. Use config_hpl to select them and copy them to all nodes in the run. Alternately, scpall may be used to copy the file to all nodes. Refer to Fast Fabric Command Line Interface Reference Guide for more information on scpall.
2-42
D000084-002 Rev A
Upgrade the Management Node

This procedure provides discussion and step-by-step directions to upgrade a IB Management Node from a previous QLogicIB software version to the latest QLogicIB-IFS application.
Preinstallation Steps
Prior to upgrading to QLogicIB-IFS, ensure the following have been performed: Review the Release Notes for a list of compatible software. Any versions of Mellanox VAPI installed on the system must be uinstalled first. This includes removing /usr/mellanox and export of MT* environment variables in /etc/bashrc, /etc/profile, and /etc/csh.cshrc. Any versions of 3rd party Infiniband stacks must be uninstalled first. The configuration files should be backed up prior to upgrading the QLogicIB software or upgrading the OS. This is for extra protection incase there is a problem or a mistake made during the upgrade. The following is a list of the files that should be backed up:
/etc/sysconfig/iview_fm.config /etc/sysconfig/iba/iba_mon.conf /etc/sysconfig/fastfabric.conf /etc/sysconfig/iba/ports /etc/sysconfig/iba/chassis /etc/sysconfig/iba/hosts /etc/sysconfig/iba/esm_chassis /var/opt/iba/analysis/baseline/*

Consult the OS documentation for a list of any other OS specific files that should be included in any backups.
D000084-002 Rev A
3-1
Upgrade the Management Node Preinstallation Steps
QLogicIB-IFS includes a compatible version of QLogic OFED. Consult the QLogic HCA and QLogic OFED Software Users Guide for more information on QLogic OFED.
NOTE: When managing a cluster where the IPoIB settings on the compute nodes are incompatible with the IB Management node (for example when a 4K MTU is used on the compute nodes and a 2K MTU is used on the management nodes), it is recommended not to run IPoIB on the IB management nodes.
3-2
D000084-002 Rev A
A
QLogicIB-IFS Upgrade
Upgrade the Management Node QLogicIB-IFS Upgrade
To install the QLogicIB-IFS software in a node with existing QLogicIB-IFS software perform the following steps. Use the package file, QLogicIB-IFS.DISTRO.VERSION.tgz on host where the full QLogic InfiniBand Fabric Suite (IFS) package has been purchased. Using the menus, select to install the desired components (at least OFED IB Stack, QLogic IB Tools and QLogic Fast Fabric) as described below.
Unpack the Tar File

Use the following procedure to unpack the QLogicIB-IFS.DISTRO.VERSION.tgz tar file. 1. 2. 3. 4. Login as user root. Copy the tar file QLogicIB-IFS.DISTRO.VERSION.tgz to the /root directory. Type cd /root and press ENTER. Type tar xvfz QLogicIB-IFS.DISTRO.VERSION.tgz and press ENTER. This unpacks the tar file to the QLogicIB-IFS.DISTRO.VERSION directory.
Upgrade QLogicIB-IFS
1. 2. Type cd QLogicIB-IFS.DISTRO.VERSION and press ENTER Type ./INSTALL and press ENTER. Displays the QLogic Inc. InfiniBand VERSION Software main menu (Figure 3-1).
X) Exit
Figure 3-1 QLogic IB Software Main Menu (Example)

3. Press 1
D000084-002 Rev A
3-3
Upgrade the Management Node Upgrade QLogicIB-IFS
S
1 of 3): ][Available] ][Available] ][Available] ][Available] ][Available] ][Available] ][Available] ][Available] ][Available] ][Available] ][Available] ][Available] ][Available] ][Available]
Displays screen 1 of 3 of the QLogic Inc. IB Install Menu (Figure 3-2). The install actions that have a previous installation show Upgrade.
QLogic Inc. IB Install (VERSION release) Menu Please Select Install Action (screen 0) OFED IB Stack [ Upgrade 1) TrueScale HCA Libs [ Upgrade 2) QLogic IB Tools [ Upgrade 3) OFED IB Development [ Upgrade 4) QLogic Fast Fabric [ Upgrade 5) QLogic SRP [ Upgrade 6) QLogic Virtual NIC [ Upgrade 7) OFED IP over IB [ Upgrade 8) OFED SDP [ Upgrade 9) OFED uDAPL [ Upgrade a) QLogic FM [ Upgrade b) MVAPICH (gcc) [ Upgrade c) MVAPICH2 (gcc) [ Upgrade d) OpenMPI (gcc) [ Upgrade N) P) R) X) Next Screen Perform the selected actions Re-Install All Return to Previous Menu (or ESC)

NOTE: In Figures 3-2, 3-3, and 3-4 below, the TrueScale and PSM components are only available if a QLogic TrueScale HCA is found in the given server. 4. Verify that all items that are selected to be upgraded in the menu are correct. NOTE: To change the selection of any of the items in the different menus select the number associated with the item. This toggles the selection from Upgrade to Don't Install and back. 5. 6. Select any items that are required and not selected by default. Press N. Displays screen 2 of 3 of the QLogicIB-IFS Install menu (Figure 3-3)
3-4
D000084-002 Rev A
A
QLogic Inc. IB Install (VERSION release) Menu Please Select Install Action (screen 2 of 3): 0) MVAPICH/PSM (gcc) [ Install ][Available] 1) MVAPICH/PSM (PGI) [ Install ][Available] 2) MVAPICH/PSM (PSc) [ Install ][Available] 3) MVAPICH/PSM (Intel) [ Install ][Available] 4) OpenMPI/PSM (gcc) [ Install ][Available] 5) OpenMPI/PSM (PGI) [ Install ][Available] 6) OpenMPI/PSM (PSc) [ Install ][Available] 7) OpenMPI/PSM (Intel) [ Install ][Available] 8) MPI Source [ Upgrade ][Available] 9) OFED RDS [ Upgrade ][Available] a) OFED SRP [ Upgrade ][Available] b) OFED SRP Target [Don't Install][Available] c) OFED iSER [Don't Install][Available] d) OFED iSER Target [Don't Install][Available] N) P) R) X) Next Screen Perform the selected actions Re-Install All Return to Previous Menu (or ESC)

7. 8. 9. Verify that all items that are selected to be upgraded in the menu are correct. Select any items that are required and not selected by default. Press N. Displays screen 3 of 3 of the QLogicIB-IFS Install menu (Figure 3-4)
D000084-002 Rev A
3-5
10. 11. 12.
Verify that all items that are selected to be upgraded in the menu are correct. Select any items that are required and not selected by default. Press P. Upgrades all of the software selected from both screens. System Starts the upgrade: During the upgrade, the following prompts will occur. For each prompt, the default is recommended, press enter to accept the default
Rebuild OFED SRPMs (a=all, p=prompt per SRPM, n=only as needed?) [n]: Permit non-root users to query the fabric? [y]: Do you want to keep //etc/security/limits.conf? [y]: Enable OFED SMI/GSI renice (RENICE_IB_MAD)? [y]: Install MPI with prefix compatible with mpi-selector (/usr/mpi/qlogic) [y]: Do you want to keep //etc/sysconfig/iba/iba_mon.conf? [y]: Do you want to keep //etc/sysconfig/fastfabric.conf? [y]: Do you want to keep //etc/sysconfig/iba/ports? [y]: Do you want to keep //etc/sysconfig/iba/iba_stat.conf? [y]: Do you want to keep //etc/sysconfig/qlgc_srp.cfg? [y]: Enable IPoIB Connected Mode (SET_IPOIB_CM)? [y]: Do you want to keep OFED IP over IB ifcfg files (//etc/sysconfig/network/ifcfg-ib[0-9]*)? [y]: Do you want to keep Old QLogic FM configuration file (//etc/sysconfig/iview_fm.config)? [y]: Enable OFED SRP High Availability deamon (SRPHA_ENABLE)? [n]:
3-6
D000084-002 Rev A
The following prompts will allow the selection of which Drivers and Applications will be started at boot time. The default is recommended, to select the default press enter for each prompt
Enable OFED IB Stack (openibd) to autostart? [y]: Enable IB Port Monitor (iba_mon) to autostart? [y]: Enable S20 Port Tuner (s20tune) to autostart? [y]: Enable QLogic SRP (qlgc_srp) to autostart? [y]: Enable QLogic Virtual NIC (qlgc_vnic) to autostart? [y]: Enable OFED IP over IB (openibd) to autostart? [y]: Enable OFED SDP (openibd) to autostart? [y]: Enable QLogic FM (qlogic_fm) to autostart? [y]: Enable QLogic FM SNMP Agent (iview_snmpd) to autostart? [y]: Enable OFED RDS (openibd) to autostart? [y]: Enable OFED SRP (openibd) to autostart? [y]: Hit any key to continue...
13.
Press any key. System prompts:

The above lines display when using QLogic HCAs. Skip to Step 17
D000084-002 Rev A
3-7
or
The above line displays the number of HCAs found. The following line(s) list each HCA name and firmware, and displays one of the following scenarios: No firmware available: This will display if the HCA is not a Connect-X HCA (Example).
If
Then
3-8
D000084-002 Rev A
A
14. System prompts (Example):
Select a number corresponding to the HCA that needs upgraded.

15.
Select the number corresponding to the firmware revision required for the HCA. The firmware is installed on the HCA System prompts:
If
Then
16.
17. 18. 19. 20.
Press any key. The installation completes and displays the main menu: Press X to exit. Reboot the server. (All) Compare all configuration files with the -sample configuration files to ensure they have the latest information and data.
D000084-002 Rev A
3-9
NOTE: If FastFabric is being used, after the upgrade review the FF_PRODUCT parameter in /etc/sysconfig/fastfabric.conf. This parameter must be adjusted to match value shown in /etc/sysconfig/fastfabric.conf-sample.
3-10
D000084-002 Rev A
Upgrade the Fabric
Upgrade QLogic InfiniBand Fabric Suite

If an existing fabric which has been installed and verified needs to have InfiniBand software upgraded, the following steps may be followed. 1. (All) On each IB Management Node, perform an upgrade installation of the Fabric Access Software using the procedure documented in the QLogic OFED Software Users Guide as appropriate. For QLogic OFED systems, also perform an upgrade installation of Fast Fabric for QLogic OFED. Each IB Management Node must have at least Fast Fabric, the IB Stack and IPoIB installed and configured. For MPI clusters using QuickSilver or QLogic OFED software, the IB Management Nodes should also include the MPI Runtime and MPI Development packages. If the user desires to rebuild MPI itself, the IB Development package and MPI Source packages will also be required. After completing the install, reboot each of the IB Management Nodes to ensure they are running the new IB software. NOTE: Ensure that existing configuration is appropriately upgraded, especially Fast Fabric and IPoIB related configuration files. Consult the Fabric Access Software Users Guide and release notes for further information. 2. (All) Select the Host Setup option from the Fast Fabric menu.
D000084-002 Rev A
4-1
Upgrade the Fabric Upgrade QLogic InfiniBand Fabric Suite
3.
Select the items shown as Perform in the menu below and press P to perform them:
Fast Fabric IB Host Setup Menu Host List: /etc/sysconfig/iba/hosts Setup: 0) Edit Config and Select/Edit Hosts Files 1) Verify Hosts via Ethernet ping 2) Setup Password-less ssh/scp 3) Copy /etc/hosts to all hosts 4) Show uname -a for all hosts 5) Install/Upgrade QLogic IB Software 6) Configure IPoIB IP Address 7) Build MPI Test Apps and Copy to Hosts 8) Reboot Hosts Admin: 9) Refresh ssh Known Hosts a) Rebuild MPI Library and Tools b) Run a command on all hosts c) Copy a file to all hosts Review: d) View iba_host_admin result files P) Perform the selected actions X) Return to Previous Menu (or ESC)
[Perform] [ Skip ] [ Skip ] [ Skip ] [ Skip ] [Perform] [ Skip ] [ Skip ] [Perform] [ [ [ [ Skip Skip Skip Skip ] ] ] ] ]
[ Skip
N) Select None
4.
(All) Edit Config and Select/Edit Hosts Files will permit the hosts and fastfabric.conf files to be edited. When placed in the editor for fastfabric.conf, review all the settings. Especially review the FF_PRODUCT, FF_PACKAGES, and FF_UPGRADE_OPTIONS. See Appendix B for more information about fastfabric.conf. Select a hosts list file which lists all the hosts except the IB Management nodes. If necessary, create a new file at this time, potentially based on the existing /etc/sysconfig/iba/hosts file. NOTE: Do not list any of IB Management Nodes (e.g., the nodes which have Fast Fabric installed).
NOTE: The file may list the Management Network or IPoIB hostnames for the selected hosts
4-2
D000084-002 Rev A
A
5.
(Host) Install/Upgrade QLogic IB Software will upgrade the IB software on all the selected hosts. By default it will look in the current directory for the $FF_PRODUCT.$FF_PRODUCT_VERSION.tgz file. If it is not found in the current directory, it will prompt for input of a directory name where this file can be found. When prompted, select to do an upgrade installation as follows:
Would you like to do an upgrade install? [y]: y
NOTE: An upgrade installation will update any existing QLogic IB software on the selected hosts. An upgrade install is only valid for hosts which already have a previous version of QLogic IB or InfiniServ software installed. If any hosts fail to be updated, use the View iba_host_admin result files option to review the result files from the update. Refer to the Fast Fabric Command Line Interface Reference Guide for more details. NOTE: When using the QLogic IB packaging of OFED, Fast Fabric may be used to upgrade the IB stack on the remaining hosts. When using other packaging of OFED, Fast Fabric may be used to upgrade the IB Stack Tools (InfiniServTools.$FF_PRODUCT_VERSION.tgz) on the remaining hosts. OFED must be upgraded on each host manually. 6. (Linux) If any other setup operations need to be performed on all hosts, use the Run a command on all hosts menu option. This option executes a the specified Linux shell command (or sequence of commands separated by semicolons) against all selected hosts. NOTE: Check the relevant release notes for the new InfiniServ release being installed for any such additional required steps. 7. (Linux) Reboot Hosts will reboot all the selected hosts and ensure they go down and come back up (as verified via ping over the management network). When the hosts come back up, they will be running the IB software installed. Repeat the verification steps for the fabric as discussed in Verifying InfiniBand on the Remaining Servers on page 2-33.
8.
D000084-002 Rev A
4-3
4-4
D000084-002 Rev A
Infiniband Fabric Suite Installation Checklist

The sections below provide a checklist to aid in tracking the steps as they are completed for Fabric Setup, Installation and verification. Check off each step as its performed. Refer to Install the Fabric Software on page 2-1 for a more detailed explanation of each step.
Pre-Installation
1. 2. 3. Hardware installed, cabled, powered. HCA installed in each server. Desired Linux OS installed on each server with the following options: a. For root user command prompt ends in "# " or "$". NOTE: There must be a space after # or $. b. 4. a. b. 5. a. b. TCL and Expect packages installed on all IB Management Nodes. Ensure SSH server enabled All servers configured with the same root password If using /etc/hosts the /etc/hosts file updated on IB Management Node If using DNS 6. All Management Network and IPoIB hostnames added to DNS
/etc/resolv.conf file configured on IB Management Node.
Remote login as root enabled.
TCP/IP Host Name Resolution
NTP server setup.
D000084-002 Rev A
A-1
Infiniband Fabric Suite Installation Checklist Install QLogicIB-IFS on Management Node
Install QLogicIB-IFS on Management Node

1. 2. Unpack Tar file on the Management Node Install QLogicIB-IFS on IB Management Node.
Configure and verify Firmware on the IB Chassis

Pre-Configuration Requirements
1. 2. 3. 4. All Chassis connected to management network. Unique IP address configured for each chassis. Unique name selected for each chassis and entered in DNS or /etc/hosts. Administrator passwords configured the same on all chassis.
Configuration and Verification of Chassis

5. 6. 7. 8. 9.
fastfabric.conf file reviewed. /etc/sysconfig/iba/chassis file created listing all chassis.
Chassis verified via Ethernet ping. Chassis firmware updated and running. Chassis Basic Configuration setup via Fast Fabric including: a. a. b. c. d. e. configuration of syslog server configuration of NTP server configuration of timezone configuration of MTU configuration of IB Node Desc configuration of IB Node Desc format
10. 11. 12.
Setup password-less SSH/SCP. Reboot chassis. Configure Chassis Fabric Manager (if using embedded FM)
Install and Configure the Fabric Manager

1. 2. Fabric Manager installed/enabled. Fabric Manager started and running.
A-2
D000084-002 Rev A
Infiniband Fabric Suite Installation Checklist Installing and Verifying Firmware on the IB Switches
Installing and Verifying Firmware on the IB Switches

Pre-Configuration Requirements
1. All switches connected to IB fabric.
Configuration and Verification of Switches

2. 3. 4. 5. 6. 7. 8.
/etc/sysconfig/fastfabric.conf file reviewed. /etc/sysconfig/iba/ibnodes file created listing all switches and their desired unique IB Node Desc names.
Switch presence verified. Switch firmware verified. Switch firmware updated and running. Switch firmware and hardware info reviewed Any 12200 or 9024FC switches in the IB Management Node path to fabric updated.
Install Infiniband on the Remaining Servers

1. 2. 3. 4. 5.
/etc/sysconfig/fastfabric.conf file reviewed. /etc/sysconfig/iba/hosts file created listing all hosts except IB
Management Node. Verify Hosts via Ethernet ping. Setup password-less SSH/SCP. TCP/IP Name Resolution configured on all hosts. a. b. 6. 7. 8. 9. 10. 11. if using /etc/hosts - Copy /etc/hosts to all hosts. if using DNS - /etc/resolv.conf copied or configured on all hosts.
Show uname -a for all hosts and verify OS version installed. Install QLogicIB-Basic. Configure IPoIB IP Addresses. Build MPI Test Apps and copy to hosts. Reboot hosts. Any additional host configuration completed.
D000084-002 Rev A
A-3
Infiniband Fabric Suite Installation Checklist Verifying Infiniband on the Remaining Servers
Verifying Infiniband on the Remaining Servers

1. 2. 3. 4. 5. 6. 7. 8. 9.
/etc/sysconfig/fastfabric.conf file reviewed. /etc/sysconfig/iba/allhosts file created listing all hosts including IB
management nodes. Verify hosts via Ethernet ping. Summary of fabric components reviewed. Check Status of Host IB Ports. Verify hosts can see each other. Verify hosts ping via IPoIB. Refresh SSH known hosts. Check MPI performance.
Complete Installation of additional IB Management Nodes

For Each IB management node: 1. 2. 3. 4. 5. Install additional QLogicIB-IFS software components. Copy Fast Fabric configuration files from initial IB management node and edit host lists. If QLogic Fabric Manager will be used, copy Fabric Manager configuration file (/etc/sysconfig/qlogic_fm.xml) from Initial IB Management Node Setup password-less SSH/SCP. Refresh SSH known hosts.
Configure and initialize health check tools

This procedure should be followed on each IB management node from which the health check tools will be used. 1. 2. 3. 4. 5. Edit /etc/sysconfig/fastfabric.conf and review the health check tools parameters. If using embedded SM(s), create /etc/sysconfig/iba/esm_chassis. Perform a health check using all_analysis -e. Create a cluster configuration baseline using all_analysis -b. If desired, schedule regular runs of all_analysis via cron or other mechanisms.
A-4
D000084-002 Rev A
Fast Fabric Configuration Files

The following configuration files are used by Fast Fabric: Table B-1. Fast Fabric Configuration Files Configuration File
/etc/sysconfig/fastfabric.conf /etc/sysconfig/iba/iba_mon.conf /etc/sysconfig/iba/allhosts /etc/sysconfig/iba/hosts /etc/sysconfig/iba/chassis /etc/sysconfig/iba/ibnodes /etc/sysconfig/iba/esm_chassis /etc/sysconfig/iba/ports /etc/sysconfig/iba/topology.0:0.xml
Description
Overall configuration file Error thresholds List of all hosts managed by Fast Fabric including the localhost List of all hosts managed by Fast Fabric except the localhost List of all chassis managed by Fast Fabric List of all externally managed switches managed by Fast Fabric List of all chassis running an embedded SM which are to be monitored via esm_analysis List of local HCA ports (e.g. subnets) to be used for fabric health analysis Fabric topology input file used by iba_reports and fabric health tools
D000084-002 Rev A
B-1
Fast Fabric Configuration Files fastfabric.conf
fastfabric.conf
The Fast Fabric tools support a configuration file /etc/sysconfig/fastfabric.conf. This file can be used to provide default settings for most of the Fast Fabric command line options. The config file is a bash shell script which will be included by each tool. As such the file should be implemented such that environment variables defined before the config file is executed will not be altered. The sample displayed below makes use of the bash syntax:
var= {var:-value}
such that only uninitialized variables are overwritten by the configuration file. A sample file is provided below, and matches the internal defaults of the Fast Fabric tools. For a given release consult /etc/sysconfig/fastfabric.conf-sample for a sample file with the defaults of the given release. If fastfabric.conf does not assign a value to a given configuration variable, the default value will be used. NOTE: Do not edit /etc/sysconfig/fastfabric.conf-sample.
B-2
D000084-002 Rev A
A
#!/bin/bash
The use of various configuration variables are discussed in the Environment Variables section for each command.
# [ICS VERSION STRING: @(#) ./Fast Fabric/ib_tools/defaults VERSION [DATE TIME] # This is a bash sourced config file which defines variables used in # Fast Fabric tools. settings. Command line args will override these
# Assignments should be scripted such that this file does not override # exported environment settings, as shown in the defaults below if [ "$CONFIG_DIR" = "" ] then if [ -d /etc/sysconfig ] then CONFIG_DIR=/etc/sysconfig else CONFIG_DIR=/etc fi export CONFIG_DIR fi # Override default location for HOSTS_FILE export HOSTS_FILE=${HOSTS_FILE:-$CONFIG_DIR/iba/hosts} # Override default location for CHASSIS_FILE export CHASSIS_FILE=${CHASSIS_FILE:-$CONFIG_DIR/iba/chassis} # Override default location for ESM_CHASSIS_FILE export ESM_CHASSIS_FILE=${ESM_CHASSIS_FILE:-$CONFIG_DIR/iba/esm_chassis} # Override default location for IBNODES_FILE export IBNODES_FILE=${IBNODES_FILE:-$CONFIG_DIR/iba/ibnodes} # Override default location for PORTS_FILE export PORTS_FILE=${PORTS_FILE:-$CONFIG_DIR/iba/ports}
D000084-002 Rev A
B-3
# Default suffix for IPoIB host names # the special value "NONE" is used to indicate a value of "" export FF_IPOIB_SUFFIX=${FF_IPOIB_SUFFIX:--ib} # Default management host export MGMT_HOST=${MGMT_HOST:-localhost}
# The shell funcs below are only defined if no existing function/command # with given name, hence allowing use of shell functions or creation of a # command for this operation # shell Function to convert a basic hostname into an IPoIB hostname # if FF_IPOIB_SUFFIX is "", this should return $1 unmodified # such that commands can be used with -i "" to skip IPoIB operations # (special value of NONE will be converted to "" before this is called) if ! type ff_host_basename_to_ipoib >/dev/null 2>/dev/null then function ff_host_basename_to_ipoib() { # $1 = hostname provided echo "$1$FF_IPOIB_SUFFIX" } fi # shell Function to convert a hostname into a basic hostname # (eg. remove IPoIB suffix, etc) should match "hostname -s" on host # if FF_IPOIB_SUFFIX is "", this should return $1 unmodified # such that commands can be used with -i "" to skip IPoIB operations # (special value of NONE will be converted to "" before this is called) if ! type ff_host_basename >/dev/null 2>/dev/null then function ff_host_basename() { # $1 = hostname provided (could be enet or IPoIB
B-4
D000084-002 Rev A
A
name) } fi # IP netmask for IPoIB subnet [-m option]
echo "$1"|sed -e "s/$FF_IPOIB_SUFFIX\$//"
# if "" default will be determined based on class of IP address [A, B, C] export FF_IPOIB_NETMASK=${FF_IPOIB_NETMASK:-} # Maximum parallel processes for ibtest and -p option on other commands # set to 1 to avoid parallel execution export FF_MAX_PARALLEL=${FF_MAX_PARALLEL:-20} # If the systems are slow for some reason, this can be used to provide a # multiplier for all timeouts in ibtest, iba_host_admin, # iba_chassis_admin, iba_switch_admin export FF_TIMEOUT_MULT=${FF_TIMEOUT_MULT:-2} # InfiniServ product to install during iba_host_admin load and upgrade export FF_PRODUCT=${FF_PRODUCT:-QLogicIB-Basic.`cat /etc/sysconfig/iba/osid_wrapper`} # InfiniServ product version to install during iba_host_admin load # and upgrade export FF_PRODUCT_VERSION=${FF_PRODUCT_VERSION:-`cat /etc/sysconfig/iba/version_wrapper 2>/dev/null`} # InfiniServ packages to install during iba_host_admin load [-P option] export FF_PACKAGES=${FF_PACKAGES:-iba ipoib mpi} # Additional INSTALL options during iba_host_admin load [-I option] export FF_INSTALL_OPTIONS=${FF_INSTALL_OPTIONS:-} # Additional INSTALL options during iba_host_admin upgrade [-U option]
D000084-002 Rev A
B-5
export FF_UPGRADE_OPTIONS=${FF_UPGRADE_OPTIONS:-} # where to upload server specific files to during uploadall, captureall # [-d option] export UPLOADS_DIR=${UPLOADS_DIR:-./uploads} # where to download server specific files from during downloadall # [-d option] export DOWNLOADS_DIR=${DOWNLOADS_DIR:-./downloads} # where to save history and baselines for cluster analysis and health # checks [-d option] export FF_ANALYSIS_DIR=${FF_ANALYSIS_DIR:-/var/opt/iba/analysis} # topology input file(s) to use for fabric_analysis, all_analysis and # iba_reports operations. The %P marker will be replaced with the # local HCA port/fabric selection string (for example 0:0 or 1:2) for # the fabric being selected (see PORTS_FILE for more information) # if this file is not found, or the value of this parameter is "NONE" # no topology input file will be used export FF_TOPOLOGY_FILE=${FF_TOPOLOGY_FILE:-$CONFIG_DIR/iba/topology.%P.x ml} # How to login to host systems during iba_host_admin # can be ssh, rsh or telnet # Note: at present cmdall and scpall always assume password-less ssh/scp # Note: check_rsh always assumes password-less rsh/rcp # Note: arguments (-s) to setup_ssh select ssh setup via rsh/rcp # or ssh/scp export FF_LOGIN_METHOD=${FF_LOGIN_METHOD:-ssh} # initial user name (cannot be root if CFG_LOGIN_METHOD=telnet) export FF_USERNAME=${FF_USERNAME:-root}
B-6
D000084-002 Rev A
A
# configured) export FF_PASSWORD=${FF_PASSWORD:-}
# password for FF_USERNAME (not needed if password-less ssh/rsh
# if FF_USERNAME is not root, what is the root password needed when # suing to root export FF_ROOTPASS=${FF_ROOTPASS:=} # How to login to chassis # can be ssh or telnet export FF_CHASSIS_LOGIN_METHOD=${FF_CHASSIS_LOGIN_METHOD:-telnet} # password for admin (not needed if password-less ssh configured) export FF_CHASSIS_ADMIN_PASSWORD=${FF_CHASSIS_ADMIN_PASSWORD:-adminpass} #base device number for IPoIB devices - IPoIB devices in the systems will # be named "ib($FF_IPOIB_BASE_DEV_NUM) ib($FF_IPOIB_BASE_DEV_NUM+1) ..." # On Silvertstorm stack it will be "ib1 ib2 ...." # On OFED stack it will be "ib0 ib1 ..." export FF_IPOIB_BASE_DEV_NUM=${FF_IPOIB_BASE_DEV_NUM:-0} # shell Function to return the base IPoIB device number for this stack type. # For Silverstorm stack installations this function should return 1, for # OFED stack installations this should return 0. if ! type ff_ipoib_base_dev_num >/dev/null 2>/dev/null then function ff_ipoib_base_dev_num() { # No arguments to this function # $host is supplied as an arguemnt when this function is called # but it is ignored here. This is a hack to limit the number of # changes required. echo "$FF_IPOIB_BASE_DEV_NUM" }
D000084-002 Rev A
B-7
fi # options for iba_report during fabric_analysis health check # alternative would be to use "-i seconds" instead of -C # depending on fabric design, more strict link speed checks, such as # -o slowconfiglinks or -o slowconnlinks may be appropriate export FF_FABRIC_HEALTH=${FF_FABRIC_HEALTH:- -s -C -o errors -o slowlinks} # list of CLI commands to issue during chassis_analysis export FF_CHASSIS_CMDS=${FF_CHASSIS_CMDS:-showInventory fwVersion showIBNodeDesc ismShowPStatThresh ismChassisSet12x timeZoneConf
B-8
D000084-002 Rev A
A
# ismIslSet12x, ismIslSetSpeed
timeDSTConf snmpCommunityConf snmpTargetAddr showChassisIpAddr showDefaultRoute} # other possible additions (if running newer chassis FW which supports these)
# single CLI command to issue to check overall health during chassis_analysis # hwCheck is prefered, but is not supported on older chassis FW export FF_CHASSIS_HEALTH=${FF_CHASSIS_HEALTH:-hwCheck} #export FF_CHASSIS_HEALTH=${FF_CHASSIS_HEALTH:-fruInfo} # list of CLI commands to issue during esm_analysis export FF_ESM_CMDS=${FF_ESM_CMDS:-smShowSMParms smShowDefBcGroup} # list of analysis to perform during all_analysis # pick appropriate type of SM to analyze #export FF_ALL_ANALYSIS=${FF_ALL_ANALYSIS:-fabric chassis hostsm esm} export FF_ALL_ANALYSIS=${FF_ALL_ANALYSIS:-fabric chassis} # command to perform text files for baseline vs latest snapshot during # fabric health analysis commands #export FF_DIFF_CMD=${FF_DIFF_CMD:-diff} export FF_DIFF_CMD=${FF_DIFF_CMD:-diff -C 1} # configuration of arguments for MPI performance deviation testing # via /opt/iba/src/mpi_apps/deviation/deviation # The defaults should work for most fabrics. stricter This can be made
# and/or specific to the fabric by using arguments such as -h hostname # -bwthres and -latthres. for more # information. #export FF_DEVIATION_ARGS=${FF_DEVIATION_ARGS:--bwtol 20 -lattol 50 -c -v} #export FF_DEVIATION_ARGS=${FF_DEVIATION_ARGS:--bwtol 20 -lattol 50 -c -h hostname} export FF_DEVIATION_ARGS=${FF_DEVIATION_ARGS:--bwtol 20 -lattol 50 -c} See run_deviation or the user's guide
D000084-002 Rev A
B-9
Fast Fabric Configuration Files iba_mon.conf
iba_mon.conf
The /etc/sysconfig/iba/iba_mon.conf configuration file defines port statistics thresholds for use by iba_report, fabric_analysis, all_analysis, and iba_mon. This file lists a threshold for each port statistic. If the threshold for a given statistic is not defined or is set to 0, the given statistic will not be checked.
B-10
D000084-002 Rev A
Fast Fabric Configuration Files iba_mon.conf
A sample file is provided below, and matches the internal defaults of the Fast Fabric tools. For a given release consult /etc/sysconfig/iba/iba_mon.conf-sample for a sample file with the defaults of the given release. NOTE: Do not edit /etc/sysconfig/iba/iba_mon.conf-sample.
# This file controls the iba_mon Port Counter monitoring Thresholds. # [ICS VERSION STRING: unknown] # # Error Counters are specified in absolute number of errors over Interval. # All Data Movement thresholds are specified in terms of average data/second # over the monitoring interval. # # Setting a threshold to 0 disables monitoring of the given counter # # Output is generated when a threshold is exceeded. # # Counters for which a non-zero threshold is specified will be cleared by # iba_mon and may impact any remote Performance Managers which are monitoring # the given Counter Interval seconds SyslogFacility disable # Normal Data Movement PortXmitData PortRcvData PortXmitPkts PortRcvPkts # Error Counters 0 0 0 0 # as MB/second # as MB/second # as packets/second # as packets/second 10 # monitoring interval in
local6
# syslog facility code, or
D000084-002 Rev A
B-11
Fast Fabric Configuration Files Host List Files
S
100 3 3 100 100 100 # Anafa2 issue, incorrectly 100 10 10 3 3 100
SymbolErrorCounter LinkErrorRecoveryCounter LinkDownedCounter PortRcvErrors PortRcvRemotePhysicalErrors #PortRcvSwitchRelayErrors increments PortXmitDiscards PortXmitConstraintErrors PortRcvConstraintErrors LocalLinkIntegrityErrors ExcessiveBufferOverrunErrors VL15Dropped
NOTE: When this file is used by iba_mon, the thresholds represent counts per "Interval". However, when used by iba_report or fabric health tools, the counts are absolute values and are applied against the counters as found in the system
Host List Files

The /etc/sysconfig/iba/hosts and /etc/sysconfig/iba/allhosts files are used to specify the hosts which Fast Fabric will operate against for many operations. If desired alternate filenames may be specified in fastfabric.conf, via environment variables or on the command line. Refer to Fast Fabric Command Line Interface Reference Guide for more information. Below is a sample host list file:
# this is a comment 192.168.0.4 # host identified by IP address n001 # host identified by resolvable TCP/IP name include /etc/sysconfig/iba/hosts-mpi# included file
Each line of the host list file may specify a single host, a comment or another host list file to include.
B-12
D000084-002 Rev A
Fast Fabric Configuration Files Chassis List Files
Hosts may be specified by IP address or a resolvable TCP/IP hostname. Typically hostnames are used for readability. Also, some Fast Fabric tools will translate the supplied host names to IPoIB hostnames, in which case names are generally easier to translate than numeric IP addresses. Typically management network host names are specified. However, if desired, IPoIB hostnames or IP addresses may be used. This can accelerate large file transfers and other operations. Files to be included may be specified via an include directive followed by a file name. File names specified should generally be absolute path names. If relative path names are used, they will be searched for within the current directory then /etc/sysconfig/iba. Comments may be placed on any line by using a # to precede the comment. On lines with hosts or include directives, the # must be white-space separated from any preceding host name, IP address or included file name.
Chassis List Files

The /etc/sysconfig/iba/chassis and /etc/sysconfig/iba/esm_chassis files are used to specify the QLogic and SilverStorm IB chassis that Fast Fabric will operate against for many operations. If desired alternate filenames may be specified in fastfabric.conf, via environment variables or on the command line. Refer to Fast Fabric Command Line Interface Reference Guide for more information. Below is a sample chassis file:
# this is a comment 192.168.0.5# chassis IP address edge1# chassis resolvable TCP/IP name include /etc/sysconfig/iba/corechassis# included file
Each line of the chassis list file may specify a single chassis, a comment or another chassis list file to include. Chassis may be specified by chassis management network IP address or a resolvable TCP/IP name. Typically, names are used for readability. Files to be included may be specified via an include directive followed by a file name. File names specified should generally be absolute path names. If relative path names are used, they will be searched for within the current directory then /etc/sysconfig/iba. Comments may be placed on any line by using a # to precede the comment. On lines with chassis or include directives, the # must be white-space separated from any preceding name, IP address or included filename.
D000084-002 Rev A
B-13
Fast Fabric Configuration Files Selection of slots within a chassis
Selection of slots within a chassis

Normally, operations are performed against the management card in the chassis. For operations such as cmdall, the command is executed against the management interface for the given chassis. For more sophisticated operations, such as firmware update, a directory with firmware for each chassis card type can be supplied and all cards in the chassis will be updated with the appropriate firmware from that directory. However, in some cases it may be desirable to perform operations against a specific subset of cards within the chassis. In this case the chassis IP address, name within a chassis list or a chassis file can be augmented with a list of slot numbers to operate on. This is done in the form:
chassis:slot1,slot2,
For example:
i9k229:0 i9k229:0,1,5 192.168.0.5:0,1,5
NOTE: There must be no spaces within the chassis name and/or slot list. This format is used by cmdall and chassis firmware update. This format may be used anywhere a chassis name or IP address is valid, such as the -H option, the CHASSIS environment variable or chassis list files. The slot number specified is ignored on some operations (such as pingall). Only slots containing management cards, EVICs and FVICs may be specified with this format. For all 9000 series chassis, slot 0 is always an alias for the presently active management card for the chassis. For the remainder of slot usages in the chassis, the chassisQuery command can be executed against a given chassis to identify which slots have management, EVIC or FVIC cards. NOTE: For any operation, care should be taken that a given chassis is listed only once with all relevant slots as part of that single specification. This is important so that parallel operations do not cause conflicting concurrent operations against a given chassis.
B-14
D000084-002 Rev A
A
Switch List Files
Fast Fabric Configuration Files Switch List Files
The /etc/sysconfig/iba/ibnodes file is used to specify the externally-managed QLogic and SilverStorm IB switches that Fast Fabric will operate against for many operations. If desired alternate file names may be specified in fastfabric.conf, via environment variables or on the command line. Refer to Fast Fabric Command Line Interface Reference Guide for more information. Below is a sample switch list file:
# this is a comment 0x00066a00d9000138,i9k138# Node GUID with desired Name 0x00066a00d9000139,i9k139# Node GUID with desired Name 0x00066a00d9000140:1:2,i9k140 # Node GUID with port and Name include /etc/sysconfig/iba/moreswitches# included file
Each line of the switch list file may specify a single switch, a comment or another switch list file to include. Switches can be specified by node GUID optionally followed by a colon hca:port and optionally followed by a coma and the IB Node Description (i.e., the name) to be assigned to the switch. The GUID will be used to select the switch and on firmware update operations, the node description will be written to the switch such that other Fast Fabric tools (such as iba_aquery and iba_report) can provide a more easily readable name for the switch. The hca:port may be used to specify which local port (subnet) to use to access the switch. If this is omitted, all local ports specified will be checked for the switch and the first port found to be able to access the switch will be used to access it. See the Fast Fabric Command Line Interface Reference Guide for more information about how to specify and hca:port value. Files to be included may be specified via an include directive followed by a file name. File names specified should generally be absolute path names. If relative path names are used, they will be searched for within the current directory then /etc/sysconfig/iba. Comments may be placed on any line by using a # to precede the comment. On lines with chassis or include directives, the # must be white-space separated from any preceding GUID, name or included file name. It is recommended that a unique node description be specified for each switch. This name should follow typical naming rules and use the characters a-z, A-Z, 0-9, and underscore. No spaces are allowed in the node description. Additionally, names should not start with a digit.
D000084-002 Rev A
B-15
Fast Fabric Configuration Files Port List Files
For externally-managed switches, the node GUID can be found on a label on the bottom of the switch. Alternately the node GUIDs for switches in the fabric can be found using a command such as:
iba_saquery -t sw -o nodeguid
NOTE: The above command will report all switch node GUIDs, including those of internally-managed chassis such as the QLogic 12000 and SilverStorm 9120. GUIDs for internally-managed chassis cannot be specified for use in the ibnodes file.
Port List Files

The /etc/sysconfig/iba/ports file is used to specify the local HCA ports (i.e., subnets) that Fast Fabric will use for assorted commands (such as iba_reports, fabric_info, iba_switch_admin, fabric_analysis, all_analysis) for fabric access. If desired, alternate filenames may be specified in fastfabric.conf, via environment variables or on the command line. Refer to Fast Fabric Command Line Interface Reference Guide for more information. Below is a sample port list file:
# this is a comment 1:1# first port on 1st HCA 1:2# second port on 1st HCA 2:1# first port on 2nd HCA 3:0# first active port on 3rd HCA include /etc/sysconfig/iba/ports-plain2# included file
Each line of the port list file may specify a single port, a comment or another port list file to include. Ports are specified as hca:port. No spaces are permitted. The first HCA is 1 and the first port is 1. The special value 0 for HCA or port has special meaning. The allowed formats are:
0:0 = 1st active port in system 0:y = port y within system x:0 = 1st active port on HCA x x:y = HCA x, port y
B-16
D000084-002 Rev A
Fast Fabric Configuration Files topology.0:0.xml
Files to be included may be specified via an include directive followed by a file name. File names specified should generally be absolute path names. If relative path names are used, they will be searched for within the current directory then /etc/sysconfig/iba. Comments may be placed on any line. By using a # to precede the comment. On lines with a port or include directive, the # must be white-space separated from any preceding port or included file name.
topology.0:0.xml
The /etc/sysconfig/iba/topology.0:0.xml file is used to specify the expected fabric topology and augmented fabric information (such as cable labels, types, lengths, SM details, node details, link details, etc). If present this file will be used by assorted Fast Fabric commands (such as iba_reports, fabric_analysis, all_analysis). Refer to the Fast Fabric Command Line Interface Reference Guide for more information on how to create a topology file describing the fabric If desired, alternate filenames may be specified in fastfabric.conf, via environment variables or on the command line. Refer to Fast Fabric Command Line Interface Reference Guide for more information.
D000084-002 Rev A
B-17
The XML format of topology input can appear as follows (the example below is purposely brief and omits many links, nodes, and SMs):
<?xml version="1.0" encoding="utf-8" ?> <Report> <LinkSummary> <Link> <Rate>20g</Rate> <MTU>2048</MTU> <Internal>0</Internal> <LinkDetails>Bender to Switch</LinkDetails> <Cable> <CableLength>11m</CableLength> <CableLabel>S4567</CableLabel> <CableDetails>gore cable model 456</CableDetails> </Cable> <Port> <NodeGUID>0x0002c9020020e004</NodeGUID> <PortGUID>0x0002c9020020e005</PortGUID> <PortNum>1</PortNum> <NodeType>CA</NodeType> <NodeDesc>bender HCA-1</NodeDesc> <PortDetails>bender primary port</PortDetails> </Port> <Port> <NodeGUID>0x00066a0007000df6</NodeGUID> <PortNum>1</PortNum> <NodeType>SW</NodeType> <NodeDesc>i9k159 Leaf 4, Chip A</NodeDesc> </Port> </Link> <Link> <Rate>20g</Rate> <MTU>2048</MTU> <Internal>0</Internal> <Port> <NodeGUID>0x0002c9020025a678</NodeGUID> <PortGUID>0x0002c9020025a679</PortGUID> <PortNum>1</PortNum>
B-18
D000084-002 Rev A
A
<NodeType>CA</NodeType> </Port> <Port>
<NodeDesc>mindy2 HCA-1</NodeDesc>
<NodeGUID>0x00066a0007000e6d</NodeGUID> <PortNum>4</PortNum> <NodeType>SW</NodeType> <NodeDesc>i9k159 Leaf 5, Chip A</NodeDesc> </Port> </Link> </LinkSummary> <Nodes> <CAs> <Node id="0x0002c9020025a678"> <NodeGUID>0x0002c9020025a678</NodeGUID> <NodeDesc>mindy2 HCA-1</NodeDesc> <NodeDetails>mindy2 only HCA</NodeDetails> </Node> </CAs> <Switches> <Node id="0x00066a000600025a"> <NodeGUID>0x00066a000600025a</NodeGUID> <NodeDesc>i9k159 Spine 1, Chip A</NodeDesc> <NodeDetails>core switch</NodeDetails> </Node> </Switches> <SMs> <SM id="0x0002c9020025a678:1"> <NodeGUID>0x0002c9020025a678</NodeGUID> <NodeDesc>mindy2 HCA-1</NodeDesc> <PortNum>1</PortNum> <PortGUID>0x0002c9020025a679</PortGUID> <NodeType>CA</NodeType> <NodeType_Int>1</NodeType_Int> <SMDetails>mindy2 SM</SMDetails> </SM> </SMs> </Nodes> </Report>
D000084-002 Rev A
B-19
B-20
D000084-002 Rev A
Multi-Subnet Fabrics
Fast Fabric supports management of both single-subnet fabric and multi-subnet fabrics. When operating a multi-subnet fabric, an subnet manager (SM) is required for each subnet. An SM may be run within switches within each subnet, or a host-based SM may be run. A host-based SM can manage multiple subnets (assuming the host server is connected to more than one subnet). For multi-subnet fabrics a number of combinations are possible: 1. All subnets are completely independent (except for any interconnecting routers): If a separate Fast Fabric node is being used per subnet and servers are not installed in more than one subnet, the individual subnets can be treated completely separately. In this case, follow all the previous Fast Fabric instructions for each fabric. The subnets are primarily independent: If the only IB components common to more than one subnet are the Fast Fabric node (and possibly SM nodes) and no servers are installed in more than one subnet, consult the instructions below for Primarily Independent Subnets. The subnets are overlapping: If multiple IB components are common to more than one subnet, such as Fast Fabric node(s), servers, etc., consult the instructions below for Overlapping Subnets.
2.
3.
Primarily Independent Subnets

If the Fast Fabric node (and possible SM nodes) is the only common IB server between subnets, Fast Fabric may be used to assist in server installation and fabric operation. Follow the installation instructions outlined in Install the Fabric Software with the following adjustments: From Design the Fabric on page 2-1, design the cabling such that the Fast Fabric node will be connected to each IB subnet it will manage. The Fast Fabric node must also have a management network path to all the nodes in all the subnets that it will manage. As part of the design consider where routes between subnets are desired between IB routers, IPoIB routers, etc.
D000084-002 Rev A
C-1
Multi-Subnet Fabrics Primarily Independent Subnets
Design the Fabric on page 2-1 can be performed as per the instructions. When installing the Fabric Access Software on the IB management node, IPoIB will need to be configured such that each IB subnet is an independent IPoIB network interface, typically with different IP subnets. Consult the QuickSilver Fabric Access Software Users Guide for more information on configuring IPoIB. NOTE: When managing a cluster where the IPoIB settings on the compute nodes are incompatible with the IB Management node (e.g., when a 4K MTU is used on the compute nodes and a 2K MTU is used on the IB Management Node), it is recommended not to run IPoIB on the IB management node(s). Configure QLogic and SilverStorm IB Chassis on page 2-13 can be performed as per the instructions. When creating the chassis file, list all QLogic 12000 and SilverStorm 9000 series internally-managed IB switches in all subnets. If desired, additional files may also be created per subnet that list only the QLogic and SilverStorm chassis in each subnet. When editing the ports file, list all the IB Management Node IB ports which access the managed fabrics. If desired, additional files may also be created per subnet that list only the IB Management Node IB port connected to the given managed fabric. Install and Configure the Fabric Manager on page 2-23 can be performed as per the instructions. At least one subnet manager is required per IB subnet. Consult the QLogic Fabric Manager Users Guide for more information on how to configure a host SM node to manage more than one IB subnet. Configure QLogic and SilverStorm IB Chassis on page 2-13 can be performed as per the instructions. When editing the ports file, list all the IB Management Node IB ports which access the managed fabrics. If desired, additional files may also be created per subnet that list only the IB Management Node IB port connected to the given managed fabric. If desired the ibnodes file may specify a hca:port per switch. However, if hca:port is not specified, all the hca:port specified in the ports file will be searched to locate the given IB Switchs Node Guid. Install InfiniBand on the Remaining Servers on page 2-29 can be performed as per the instructions. When creating the hosts file, list all of the hosts in all subnets except the IB management node where Fast Fabric is being run. If desired, additional files may also be created per subnet that list the hosts in each subnet (except the IB management node). Verifying InfiniBand on the Remaining Servers on page 2-33 has the following adjustments from the instructions.
C-2
D000084-002 Rev A
A
Multi-Subnet Fabrics Primarily Independent Subnets
(All): Create the allhosts file as per the instructions. Next, create additional files per subnet that list all the hosts in each subnet including the IB management node. When editing the ports file, list all the IB Management Node IB ports which access the managed fabrics. If desired, additional files may also be created per subnet that list only the IB Management Node IB port connected to the given managed fabric. (All): Verify Hosts via Ethernet ping on page 2-31 can be performed as per the instructions. (All): Summary of Fabric Components on page 2-35 can be performed as per the instructions. (All): Check Status of IB Ports on page 2-36 can be performed as per the instructions. (Host): Verify Hosts see each other on page 2-36 can be run for each subnet by using the allhosts files specific to each subnet (i.e., those listing only hosts in a single subnet). (Host): Verify Hosts ping via IPoIB on page 2-36 and (Linux): Refresh ssh Known Hosts on page 2-37 may be run per the instructions. (Host): Check MPI Performance on page 2-37 can be run for each subnet by using the allhosts files specific to each subnet (i.e., those listing only the hosts in a single subnet). This is currently not available on OFED.

Complete Installation of additional IB Management Nodes on page 2-38 can be performed as per the instructions. When copying Fast Fabric configuration files to the additional IB management nodes, be sure to also copy the additional hosts, chassis and allhosts files that were created per subnet. NOTE: In asymmetrical configurations where the IB management nodes are not all connected to the same set of subnets, the files copied to each management node may need to be slightly different. For example configuration files for fabric_analysis may indicate different port numbers or host files used for Fast Fabric and MPI may need to list different hosts. Configure and Initialize Health Check Tools on page 2-40 can be performed as per the instructions. Additionally, make sure the /etc/sysconfig/iba/ports file lists each of the IB management node local HCAs and ports that are connected to a unique subnet. When running iba_reports, fabric_info, fabric_analysis, or all_analysis, the default will be to use the ports file. If desired, the -p and -t options or the PORTS/PORTS_FILE environment variables may be used to specify all the HCAs and ports on the IB management node such that all subnets are checked. Similarly, the esm_chasssis and chassis files used should list all relevant QLogic and SilverStorm IB chassis in all subnets.
D000084-002 Rev A
C-3
Multi-Subnet Fabrics Overlapping Subnets
Running High Performance Linpack on page 2-41 can be run for each subnet by creating mpi_hosts files specific to each subnet (i.e., only listing hosts in a single subnet). Upgrade the Management Node on page 3-1 can be performed as per the instructions.
Overlapping Subnets
If multiple IB components are common between subnets (in addition to the IB management nodes), Fast Fabric may be used to assist in server installation and fabric operation. Follow the installation instructions outlined in Install the Fabric Software on page 2-1 with the following adjustments: From Design the Fabric on page 2-1, design the cabling such that the Fast Fabric node will be connected to each IB subnet it will manage. The Fast Fabric node must also have a management network path to all the nodes in all the subnets it will manage. As part of the design consider where routes between subnets are desired, between IB routers, IPoIB routers, etc. Set Up the Fabric on page 2-2 can be performed as per the instructions. When installing the Fabric Access software on the IB Management node, IPoIB will need to be configured such that each IB subnet is an independent IPoIB network interface, typically with different IP subnets. Consult the QuickSilver Fabric Access Software Users Guide for more information on configuring IPoIB. NOTE: When managing a cluster where the IPoIB settings on the compute nodes are incompatible with the IB management node (e.g., when a 4K MTU is used on the compute nodes and a 2K MTU is used on the management nodes), it is recommended not to run IPoIB on the IB management node(s). Configure QLogic and SilverStorm IB Chassis on page 2-13 can be performed as per the instructions. When creating the chassis file, list all QLogic 12000 and SilverStorm 9000 series internally-managed IB switches in all subnets. If desired, additional files may also be created per subnet that list only the QLogic and SilverStorm chassis in each subnet. When editing the ports file, list all the IB Management Node IB ports which access the managed fabrics. If desired, additional files may also be created per subnet that list only the IB Management Node IB port connected to the given managed fabric. Install and Configure the Fabric Manager on page 2-23 can be performed as per the instructions. At least one subnet manager is required per IB subnet. Consult the Fabric Manager Users Guide for more information on how to configure a host-based SM node to manage more than one IB subnet.
C-4
D000084-002 Rev A
Configure QLogic and SilverStorm IB Chassis on page 2-13 can be performed as per the instructions. When editing the ports file, list all the IB Management Node IB ports which access the managed fabrics. If desired, additional files may also be created per subnet that list only the IB Management Node IB port connected to the given managed fabric. If desired the ibnodes file may specify a hca:port per switch. However, if hca:port is not specified, all the hca:port specified in the ports file will be searched to locate the given IB Switchs Node Guid. Install InfiniBand on the Remaining Servers on page 2-29 can be performed as per the instructions. When creating the hosts file, list all the hosts in all subnets except the IB management node where Fast Fabric is being run. If desired, additional files may also be created per subnet that list the hosts in each subnet (except the IB management node). For hosts that are connected to more than one IB subnet, IPoIB will need to be configured such that each IB subnet is an independent IPoIB network interface, typically with different IP subnets. Consult the QuickSilver Fabric Access Software Users Guide for more information on configuring IPoIB. Verifying InfiniBand on the Remaining Servers on page 2-33 has the following adjustments from the instructions. (All): Create the allhosts file per the instructions. Next, create additional files per subnet that list all the hosts in each subnet including the IB management node. When editing the ports file, list all the IB Management Node IB ports which access the managed fabrics. If desired, additional files may also be created per subnet that list only the IB Management Node IB port connected to the given managed fabric. (All): Verify Hosts via Ethernet ping on page 2-31 can be performed per the instructions. (All): Summary of Fabric Components on page 2-35 can be performed as per the instructions. (All): Check Status of IB Ports on page 2-36 can be performed as per the instructions. (Host): Verify Hosts see each other on page 2-36 can be run for each subnet by using the allhosts files specific to each subnet (i.e., those only listing hosts in a single subnet). (Host): Verify Hosts ping via IPoIB on page 2-36 and (Linux): Refresh ssh Known Hosts on page 2-37 may be run per the instructions. (Host): Check MPI Performance on page 2-37 can be run for each subnet by using the allhosts files specific to each subnet (i.e., those listing only the hosts in a single subnet). This is currently not available for OFED.

D000084-002 Rev A
C-5
Complete Installation of additional IB Management Nodes on page 2-38 can be performed as per the instructions. When copying Fast Fabric configuration files to the additional IB management nodes, be sure to also copy the additional hosts, chassis and allhosts files created per subnet. NOTE: In asymmetrical configurations where the IB management nodes are not all connected to the same set of subnets, the files copied to each management node may need to be slightly different. For example, configuration files for fabric_analysis indicating different port numbers or host files used for Fast Fabric and MPI may need to list different hosts. Configure and Initialize Health Check Tools on page 2-40 can be performed per the instructions. In addition, make sure the /etc/sysconfig/iba/ports file lists the IB management node local HCAs and ports that are connected to a unique subnet. When running iba_reports, fabric_info, fabric_analysis, or all_analysis, the default is to use the ports file. If desired, the -p and -t options or the PORTS/PORTS_FILE environment variable may be used to specify all the HCAs and ports on the IB management node such that all subnets are checked. Similarly, the esm_chasssis and chassis files used should list all relevant QLogic and SilverStorm IB chassis in all subnets. Running High Performance Linpack on page 2-41 can be run for each subnet by creating mpi_hosts files specific to each subnet (i.e., only listing hosts in a single subnet). Upgrade the Management Node on page 3-1 can be performed per the instructions.
C-6
D000084-002 Rev A
Install a Previous Version of Software

If the need exists to install a previous version of the IFS software use the following procedure. 1. Uninstall all existing software using the following command:
iba_config -u
2.
Install the older version of the software using the installation procedures provided in the documentation that was released for that specific version of software. Carefully review all configuration files for information that may need to be discarded or edited which are specific to features in the newer release which were not available in the older release Reboot server.
3.
4.
D000084-002 Rev A
D-1
Install a Previous Version of Software
D-2
D000084-002 Rev A
Install QLogicIB-Basic
Install QLogicIB-Basic on Cluster Nodes

On all hosts, use the package file, QLogicIB-Basic.DISTRO.VERSION.tgz. Using the menus install the desired components (at least OFED IB Stack, QLogic IB Tools and QLogic Fast Fabric) (Fast Fabric and QLogic FM ares displayed as not available) as described below (Fast Fabric and QLogic FM are displayed as not available). NOTE: Fast Fabric may be used to install the QLogicIB-Basic package on all hosts in the cluster from the IB Management node.
Unpack the Tar File

1. 2. 3. 4. Login as user root. Copy the tar file QLogicIB-Basic.DISTRO.VERSION.tgz to the /root directory. Type cd /root and press ENTER. Type tar xvfz QLogicIB-Basic.DISTRO.VERSION.tgz and press ENTER. This unpacks the tar file to the QLogicIB-Basic.DISTRO.VERSION directory.
1. 2. Type cd QLogicIB-Basic.DISTRO.VERSION and press ENTER Type ./INSTALL and press ENTER. Displays the QLogic Inc. Infiniband VERSION Software main menu (Figure E-1).
D000084-002 Rev A
E-1
Install QLogicIB-Basic Install QLogicIB-Basic on Cluster Nodes
X) Exit
Figure E-1 QLogic InfiniBand Main Menu (Example)

3. Press 1 Displays screen 1 of 3 of the QLogic Inc. IB Install menu (Figure E-2)
QLogic Inc. IB Install (VERSION release) Menu Please Select Install Action (screen 1 of 3): 0) OFED IB Stack [ Install ][Available] 1) TrueScale HCA Libs [ Install ][Available] 2) QLogic IB Tools [ Install ][Available] 3) OFED IB Development [ Install ][Available] 4) QLogic Fast Fabric [Don't Install][Not Avail] 5) QLogic SRP [ Install ][Available] 6) QLogic Virtual NIC [ Install ][Available] 7) OFED IP over IB [ Install ][Available] 8) OFED SDP [ Install ][Available] 9) OFED uDAPL [ Install ][Available] a) QLogic FM [Don't Install][Not Avail] b) MVAPICH (gcc) [ Install ][Available] c) MVAPICH2 (gcc) [ Install ][Available] d) OpenMPI (gcc) [ Install ][Available] N) P) R) X) Next Screen Perform the selected actions Re-Install All Return to Previous Menu (or ESC)
VERSION VERSION VERSION VERSION VERSION VERSION VERSION VERSION VERSION VERSION VERSION VERSION
Figure E-2 QLogicIB-Basic Install Menu (Example) (Screen 1 of 3)
E-2
D000084-002 Rev A
NOTE: In Figures E-2, E-3, and E-4 below, the TrueScale and PSM components are only available if a QLogic TrueScale HCA is found in the given server. 4. 5. Verify that all items that are selected to be installed in the default menu are correct. Select any items that are required and not selected by default. NOTE: To change the selection of any of the items in the different menus select the number associated with the item. This toggles the selection from Install to Don't Install and back. 6. Press N. Displays screen 2 of 3 of the QLogic Inc. IB Install menu (Figure E-3)
QLogic Inc. IB Install (VERSION release) Menu Please Select Install Action (screen 2 of 3): 0) MVAPICH/PSM (gcc) [ Install ][Available] 1) MVAPICH/PSM (PGI) [ Install ][Available] 2) MVAPICH/PSM (PSc) [ Install ][Available] 3) MVAPICH/PSM (Intel) [ Install ][Available] 4) OpenMPI/PSM (gcc) [ Install ][Available] 5) OpenMPI/PSM (PGI) [ Install ][Available] 6) OpenMPI/PSM (PSc) [ Install ][Available] 7) OpenMPI/PSM (Intel) [ Install ][Available] 8) MPI Source [ Install ][Available] 9) OFED RDS [ Install ][Available] a) OFED SRP [ Install ][Available] b) OFED SRP Target [Don't Install][Available] c) OFED iSER [Don't Install][Available] d) OFED iSER Target [Don't Install][Available] N) P) R) X) Next Screen Perform the selected actions Re-Install All Return to Previous Menu (or ESC)
D000084-002 Rev A
E-3
7. 8. 9.
Verify that all items that are selected to be installed in the default menu are correct. Select any items that are required and not selected by default. Press N. Displays screen 3 of 3 of the QLogic Inc. IB Install menu (Figure E-4)

10. 11. 12. Verify that all items that are selected to be installed in the default menu are correct. Select any items that are required and not selected by default. Press P. Installs all of the software selected from both screens. The following statement displays:
About to Uninstall previous InfiniBand Software Installations... Hit any key to continue...
13.
Press any key. The system proceeds with the installation. During the installation, the following prompts will be displayed. For each prompt select the default by pressing enter.
Rebuild OFED SRPMs (a=all, p=prompt per SRPM, n=only as needed?) [n]: Permit non-root users to query the fabric? (y)
E-4
D000084-002 Rev A
You have memory locking limits entries for IB drivers from an earlier install Do you want to keep //etc/security/limits.conf? [y]: Enable OFED SMI/GSI renice (RENICE_IB_MAD))? [y]: Install MPI with prefix compatible with mpi-selector (/usr/mpi/qlogic) [y]: Enable IPoIB Connected Mode (SET_IPOIB_CM)? [y]: Do you want to keep OFED IP over IB ifcfg files (//etc/sysconfig/network/ifcfg-ib[0-9]*)? [y]:
If
You want to keep OFED IP over IB ifcfg files (//etc/sysconfig/network/ifcfg-ib[0-9]*) You do not want to keep OFED IP over IB ifcfg files (//etc/sysconfig/network/ifcfg-ib[0-9]*) Skip to Step 21
Then
Type N and press ENTER. System continues installing and displays the following prompt:
Configure OFED IP over IB IPV4 addresses now? [n]:
If
IPoIB will be enabled and run on the Cluster Node. IPoIB will not be run on the Cluster Node.
Then
Continue with Step 14 Skip to Step 21
14.
Type y and press ENTER to configure IPV4 addresses. System prompts:

How many OFED IP over IB interfaces would you like to configure? [1]:
15.
Type the number of IB interfaces that need to be configured and press ENTER.
D000084-002 Rev A
E-5
System prompts:
Use interface name ib0? [y]:
16.
Press ENTER. System prompts:

Enter IPV4 address in dot notation (or dhcp) for ib0:
17.
Enter IB address in dot notation format and press ENTER. System prompts:
Is IPV4 address '172.26.16.173' correct? (y/n):
18.
Type y and press ENTER. System prompts:

Enter IPV4 netmask in dot notation for ib0 172.26.16.173 [255.255.0.0]:
19.
Enter IB netmask in dot notation format and press ENTER. System prompts:
Is IPV4 netmask '255.255.240.0' correct? (y/n):
20.
Type y and press ENTER. The system proceeds with the installation. System prompts:
Skip to Step 22 21. Press ENTER to select default (n). The system displays ifcfg file requirements as shown below and proceeds with the installation.
OFED IP over IB requires an ifcfg file for each OFED IP over IB device instance. Manually create files such as '//etc/sysconfig/network-scripts/ifcfg-ib0'
System prompts:
E-6
D000084-002 Rev A
A
22. Press ENTER to select default (n). System prompts: 23. Press ENTER to accept default. System prompts:
Enable OFED IB Stack (openibd) to autostart? [y]:
Enable IB Port Monitor (iba_mon) to autostart? [y]: .
NOTE: If Fast Fabric will be used to monitor fabric health and/or check the fabric for errors, its recommended to answer n to this prompt. 24. Press ENTER to accept default (Press N to adhere to the note above). System prompts:
Enable S20 Port Tuner (s20tune) to autostart? [y]:
25.
Press ENTER to accept default. The following series of prompts will allow selection of which drivers will be started at system boot time. It is recommended to press ENTER for a default answer of yes to each of the following prompts.
Enable QLogic SRP (qlgc_srp) to autostart? [y]: Enable QLogic Virtual NIC (qlgc_vnic) to autostart? [y]: Enable OFED IP over IB (openibd) to autostart? [y]: Enable OFED SDP (openibd) to autostart? [y]: Enable OFED RDS (openibd) to autostart? [y]: Enable OFED SRP (openibd) to autostart? [y]: Hit any key to continue...
26.
Press any key.
D000084-002 Rev A
E-7
System prompts:
The above lines display when using QLogic HCAs. Skip to Step 30 or
The above line displays the number of HCAs found. The following line(s) list each HCA name and firmware, and displays one of the following scenarios: No firmware available: This will display if the HCA is not a Connect-X HCA (Example).
E-8
D000084-002 Rev A
A
If
Then
27.
Select a number corresponding to the HCA that needs to be upgraded. System prompts (Example):
28.
Select the number corresponding to the firmware revision required for the HCA. The firmware is installed on the HCA System prompts:
If
Then
D000084-002 Rev A
E-9
29.
30. 31. 32. 33.
Press any key. The installation completes and displays the main menu: Press X to exit. Reboot the server. Repeat this procedure for each host.
E-10
D000084-002 Rev A
D000084-002 Rev A
E-11
Notes
E-12
D000084-002 Rev A
QLogicIB-Basic to IFS Upgrade

This procedure provides discussion and step-by-step directions to upgrade a IB Management Node from QLogicIB-Basic application to QLogicIB-IFS application. To install the QLogicIB-IFS software in a node with existing QLogicIB-Basic software perform the following steps. Use the package file, QLogicIB-IFS.DISTRO.VERSION.tgz. Using the menus, select to install the desired components (at least OFED IB Stack, QLogic IB Tools and QLogic Fast Fabric) as described below.
Unpack the Tar File

Use the following procedure to unpack the QLogicIB-IFS.DISTRO.VERSION.tgz or QLogicIB-Basic.DISTRO.VERSION.tgz tar file. 1. 2. 3. 4. Login as user root. Copy the tar file QLogicIB-IFS.DISTRO.VERSION.tgz to the /root directory. Type cd /root and press ENTER. Type tar xvfz QLogicIB-IFS.DISTRO.VERSION.tgz and press ENTER. This unpacks the tar file to the QLogicIB-IFS.DISTRO.VERSION directory.
1. 2. Type cd QLogicIB-IFS.DISTRO.VERSION and press ENTER Type ./INSTALL and press ENTER. Displays the QLogic InfiniBand main menu (Figure F-1).
D000084-002 Rev A
F-1
QLogicIB-Basic to IFS Upgrade Install QLogicIB-IFS
X) Exit
Figure F-1 QLogic InfiniBand Main Menu

3. Press 1 Displays screen 1 of 2 of the QLogic InfiniBand Install menu (Figure F-2). The QLogic Fast Fabric and QLogic FM selections are showing Install while the other selections show Up To Date.
QLogic Inc. IB Install (VERSION release) Menu Please Select Install Action (screen 0) OFED IB Stack [ Up To Date 1) TrueScale HCA Libs [ Up To Date 2) QLogic IB Tools [ Up To Date 3) OFED IB Development [ Up To Date 4) QLogic Fast Fabric [ Install 5) QLogic SRP [ Up To Date 6) QLogic Virtual NIC [ Up To Date 7) OFED IP over IB [ Up To Date 8) OFED SDP [ Up To Date 9) OFED uDAPL [ Up To Date a) QLogic FM [ Install b) MVAPICH (gcc) [ Up To Date c) MVAPICH2 (gcc) [ Up To Date d) OpenMPI (gcc) [ Up To Date N) P) R) X) Next Screen Perform the selected actions Re-Install All Return to Previous Menu (or ESC) 1 of 3): ][Available] ][Available] ][Available] ][Available] ][Available] ][Available] ][Available] ][Available] ][Available] ][Available] ][Available] ][Available] ][Available] ][Available]
Figure F-2 QLogicIB-IFS Install Menu (Example) (Screen 1 of 2)

4. Press P. Installs the Fast Fabric and Fabric Manager software selected.
F-2
D000084-002 Rev A
During the installation, the following prompts will be displayed. For each prompt select the default by pressing enter.
Enable OFED IB Stack (openibd) to autostart? [y]: Enable QLogic FM (iview_fm) to autostart? [y]: Enable QLogic FM SNMP Agent (iview_snmpd) to autostart? [y]: Hit any key to continue...
5.
Press any key. The installation completes and displays the main menu on the screen (Figure F-3)
QLogic Inc. InfiniBand VERSION Software 1) 2) 3) 4) 5) Install/Uninstall Software Reconfigure OFED IP over IB Reconfigure Driver Autostart Generate Supporting Information for Problem Report Fast Fabric (Host/Chassis/Switch Setup/Admin)
X) Exit
Figure F-3 Installation Complete

6. 7. 8. Press X to exit. Reboot the server. (All) Select the Host Setup option from the Fast Fabric menu.
D000084-002 Rev A
F-3
F-4
D000084-002 Rev A
./Install Syntax
./INSTALL
NOTE: To access help for this command type ./INSTALL -? and press ENTER. -a Installs all of the software with the default options. -n Installs all of the software with the default options, but does not change the autostart options. -U Upgrades/re-installs all presently installed software with the default options, and does not change the autostart options. -i comp Installs a given component with the default options. This option can appear multiple times on a command line. -f Skips the installation of the HCA firmware upgrade during installation. -F Install the HCA firmware upgrade during installation. -u Uninstalls all of the software with the default options. -s Enables autostart for all installed software. -e comp Uninstalls a given component with the default options. This option can appear multiple times on a command line.
D000084-002 Rev A
G-1
./Install Syntax
S
-E (or -D) Allows for command line control of software startup. Can be used after the software is installed to change the tools that are started or stopped at boot time. -C Shows the list of supported component names. Additional options: -r dir Specify an alternate root directory. The default is /. --no32bit Disable install of 32 bit libraries on 64 bit OSs --32bit Enable install of 32 bit libraries on 64 bit OSs --rebuild Force rebuild of OFED srpms --user_queries Permits non-root users to query the fabric. This is the default. --no_user_queries Non-root users cannot query the fabric. --user_configure_options options Supply additional OFED build options for user space srpms, this also forces a rebuild of all user space OFED srpms --kernel_configure_options options Supply additional OFED build options for kernel driver srpms, this also forces a rebuild of all kernel driver OFED srpms --prefix_dir Specify alternate directory prefix for OFED installation. Default is /usr. This also causes a rebuild of related srpms. --force Force installation even if linux distribution does not match vendor/version this package was built for. beware, use of this option can result in undefined behaviors -v
G-2
D000084-002 Rev A
A
Verbose logging. Logs to the /var/log/iba.log file. -vv
./Install Syntax
Very verbose debug logging. Logs to the /var/log/iba.log file.
D000084-002 Rev A
G-3
./Install Syntax
G-4
D000084-002 Rev A
Corporate Headquarters QLogic Corporation 26650 Aliso Viejo Parkway
Aliso Viejo, CA 92656 949.389.6000
www.qlogic.com
Europe Headquarters QLogic (UK) LTD. Quatro House Lyon Way, Frimley Camberley Surrey, GU16 7ER UK
+44 (0) 1276 804 670
2009 QLogic Corporation. Specifications are subject to change without notice. All rights reserved worldwide. QLogic and the QLogic logo are registered trademarks of QLogic Corporation. All other brand and product names are trademarks or registered trademarks of their respective owners. Information supplied by QLogic Corporation is believed to be accurate and reliable. QLogic Corporation assumes no responsibility for any errors in this brochure. QLogic Corporation reserves the right, without notice, to make changes in product design or specifications.

Installation Guide QLogic Fabric SW v50 RevA

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Installation Guide QLogic Fabric SW v50 RevA

Uploaded by

Copyright:

Available Formats

QLogic Fabric Software Installation Guide

QLogic Fabric Software Installation Guide

Document Revision History

Software Installation Overview

Install the Fabric Software

QLogic Fabric Software Installation Guide

QLogic Fabric Software Installation Guide

Upgrade the Management Node

Upgrade the Fabric

Infiniband Fabric Suite Installation Checklist

Fast Fabric Configuration Files

QLogic Fabric Software Installation Guide

Install a Previous Version of Software Install QLogicIB-Basic

QLogicIB-Basic to IFS Upgrade

./Install Syntax List of Figures

QLogic Fabric Software Installation Guide

Fast Fabric Configuration Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

QLogic Fabric Software Installation Guide

Software Installation Overview

OFED IP over IB OFED SDP

Software Installation Overview Component Overview

Software Installation Overview Component Overview

QLogic InfiniBand Fabric Suite

Software Installation Overview Installation Overview

Software Installation Overview Installation Overview

Software Installation Overview Installation Overview

Install the Fabric Software

Design the Fabric

Install the Fabric Software Set Up the Fabric

Set Up the Fabric

Install the Fabric Software Set Up the Fabric

Install the Fabric Software Install QLogicIB-IFS

Install the Fabric Software Install QLogicIB-IFS

Install QLogicIB-IFS on IB Management Node

Unpack the Tar File

Figure 2-1 QLogic IB Software Main Menu (Example)

Install the Fabric Software Install QLogicIB-IFS

I) Install All U) Uninstall All

Figure 2-2 QLogic IB Install Menu (Example) (Screen 1 of 3)

Install the Fabric Software Install QLogicIB-IFS

Displays screen 2 of 3 of the QLogic Inc. IB Install menu (Figure 2-3)

I) Install All U) Uninstall All

Figure 2-3 QLogic IB Install Menu (Example) (Screen 2 of 3)

Install the Fabric Software Install QLogicIB-IFS

I) Install All U) Uninstall All

Figure 2-4 QLogic IB Install Menu (Example) (Screen 3 of 3)

Install the Fabric Software Install QLogicIB-IFS

Type y and press ENTER to configure IPV4 addresses. System prompts:

Press ENTER. System prompts:

Type y and press ENTER. System prompts:

Install the Fabric Software Install QLogicIB-IFS

Press ENTER to select default (n). System prompts:

Press ENTER to accept default. System prompts:

Press ENTER to accept default.

Install the Fabric Software Install QLogicIB-IFS

Press any key. System prompts:

Install the Fabric Software Install QLogicIB-IFS

Update available (Example):

Firmware is up to date (Example):

0: MT_04A0110002 (MHGH28-XTC/X4/A0) Firmware 2.5.0: Okay.

Proceed with Step 27 Skip to Step 29

Install the Fabric Software Configure QLogic and SilverStorm IB Chassis

30. 31. 32.

Configure QLogic and SilverStorm IB Chassis