Professional Documents
Culture Documents
Iris Zhu
May 2008
Sun Microsystems, Inc.
Copyright © 2008 Sun Microsystems, Inc., 4150 Network Circle, Santa Clara, California 95054, U.S.A. All rights
reserved.
U.S. Government Rights - Commercial software. Government users are subject to the Sun Microsystems, Inc.
standard license agreement and applicable provisions of the FAR and its supplements. Use is subject to license terms.
This distribution may include materials developed by third parties.
Parts of the product may be derived from Berkeley BSD systems, licensed from the University of California. UNIX is
a registered trademark in the U.S. and in other countries, exclusively licensed through X/Open Company, Ltd.
X/Open is a registered trademark of X/Open Company, Ltd.
All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International,
Inc. in the United States and other countries. Products bearing SPARC trademarks are based upon an architecture
developed by Sun Microsystems, Inc.
Sun, Sun Microsystems, the Sun logo, Java, Solaris, Solstice DiskSuite, Sun BluePrints, Sun Cluster, Sun Fire,
SunPlex, and Sun StorEdge are trademarks or registered trademarks of Sun Microsystems, Inc. in the U.S. and other
countries.
This product is covered and controlled by U.S. Export Control laws and may be subject to the export or import laws
in other countries. Nuclear, missile, chemical biological weapons or nuclear maritime end uses or end users, whether
direct or indirect, are strictly prohibited. Export or reexport to countries subject to U.S. embargo or to entities
identified on U.S. export exclusion lists, including, but not limited to, the denied persons and specially designated
nationals lists is strictly prohibited.
Installing and Configuring Sun Cluster 3.1 Software for Oracle Database 10g HA 2
Table of Contents
Introduction................................................................................................................................................4
Background................................................................................................................................................4
What Is a Cluster?.................................................................................................................................4
What Is a High Availability Cluster?....................................................................................................4
What Can Sun Cluster Software Do for HA Applications?..................................................................4
Physical Connections............................................................................................................................6
Cluster Node.....................................................................................................................................7
Cluster Interconnect..........................................................................................................................7
Public Network Interfaces................................................................................................................8
Multihost Devices.............................................................................................................................8
Key Concepts........................................................................................................................................8
Global Devices, Global Namespaces, and Cluster File System.......................................................8
Quorum and Quorum Device............................................................................................................9
Data Service....................................................................................................................................10
Resource, Resource Types, and Resource Groups.........................................................................10
Cluster Topologies..............................................................................................................................11
Installation and Configuration.................................................................................................................11
Installing the Solaris 10 OS for SPARC Platforms.............................................................................12
Configuring Sun StorEdge 3510 FC Array.........................................................................................13
Installing Sun Cluster 3.1 08/05 Software...........................................................................................14
Installing Sun Cluster HA Agent for Oracle Database 10g HA..........................................................14
Installing Oracle Database 10g HA.....................................................................................................15
Configuring Sun Cluster 3.1 08/05 Software......................................................................................15
Verifying Sun Cluster 3.1 08/05 Software..........................................................................................17
Summary..................................................................................................................................................17
Appendix..................................................................................................................................................17
For More Information..............................................................................................................................19
Installing and Configuring Sun Cluster 3.1 Software for Oracle Database 10g HA 3
Introduction
This article is for those who have no experience with Sun™ Cluster software. After introducing some
basic concepts, this article provides a demonstration of how to set up Sun Cluster 3.1 08/05 software
(hereafter called Sun Cluster software) for Oracle® Database 10g HA with the Solaris™ 10 01/06
Operating System.
The procedures in this article were tested on the Solaris 10 1/06 release. However, the concepts in this
article should also work with the Solaris 10 5/08 OS, because the procedures use general functionality
of the OS.
Background
To understand the installation and configuration steps, you need to have some basic knowledge about
Sun Cluster software. This section provides some basic concepts. For more information, refer to the
Sun Cluster Concepts Guide for Solaris OS (http://docs.sun.com/app/docs/doc/819-0421) and
the Sun Cluster Overview for Solaris OS (http://docs.sun.com/app/docs/doc/819-0579).
What Is a Cluster?
A cluster is two or more systems or nodes that work together as a single, continuously available system
to provide applications, system resources, and data to users.
An HA cluster provides a computing environment that offers high levels of uptime. To understand
more about the other types of clusters, refer to the book SUN Cluster 3 Programming: Integrating
Applications into the SunPlex™ Environment by Joseph Bianco, Peter Lees, and Kevin Rabito (see
http://www.amazon.com/gp/product/0130479756/104-0560062-1750305?v=glance&n=283155).
Installing and Configuring Sun Cluster 3.1 Software for Oracle Database 10g HA 4
The next section describes how to enable Oracle10g to be the HA application within the cluster
environment. A cluster environment named oraclecluster is set up with the following features:
Physical Connections
Figure 1 shows a two-node cluster environment that has the following features:
• Two cluster nodes with local disks (unshared) provide the main computing platform of the
cluster.
• The cluster interconnect provides a channel for internode communication.
• Multihost storage provides disks that are shared between the nodes.
• The console access device enables access to the cluster environment from the administrative
console. Usually, the console access device is special hardware called a Terminal Concentrator,
which provides access to a headless cluster for hosted systems.
• The administrative console enables administration of the cluster environment. Sun workstations
on which this cluster administration tool is installed can monitor any cluster.
• The public network interfaces (multipathing group) provide the failover for the interfaces on
each node to connect with the public network.
• Client systems include workstations or other servers that access the cluster over the public
network.
Installing and Configuring Sun Cluster 3.1 Software for Oracle Database 10g HA 5
Figure 1: Cluster Hardware Components
Cluster Node
A cluster node is a machine that runs both the Solaris OS and Sun Cluster software. The Sun Cluster
3.1 08/05 software enables you to have from 2 to 16 nodes in a cluster. Each node has similar
processing, memory, and I/O capability to enable failover to occur without significant degradation in
performance.
oraclecluster has two Sun FireTM V480 servers with the same specification acting as the two
cluster nodes. The Solaris 10 01/06 OS, Sun Cluster 3.1 08/05 software, Sun Cluster HA for Oracle,
and Oracle Database 10g HA application are installed on each.
Cluster Interconnect
The cluster interconnect, shown in Figure 2, is the physical configuration of devices that are used to
transfer cluster-private communications and data service communications between cluster nodes. The
cluster interconnect is used to detect communication, repair communication, and automatically re-
initiate communication after communication is repaired.
Installing and Configuring Sun Cluster 3.1 Software for Oracle Database 10g HA 6
Figure 2: Cluster Interconnect
In the two-node cluster oraclecluster, the two Sun Fire V480 servers are connected directly by
cables between one pair of physical interfaces. Node configurations with more than two nodes
generally require junctions.
Multihost Devices
Disks that can be connected to more than one node at a time are called multihost devices. Multihost
storage makes disks highly available. Sun Cluster software requires multihost storage for two-node
clusters to establish a quorum. Clusters with more than two nodes do not require quorum devices.
In oraclecluster, there are only three NICs on each node, and a Sun StorEdgeTM 3510 FC array is
used as multihost device. Two NICs are used for interconnection between the two Sun Fire V480
servers, and one is used to connect to the public network. If you have two or more NICs connecting to
the public network, refer to System Administration Guide: IP Services
(http://docs.sun.com/app/docs/doc/816-4554).
Key Concepts
Whenever a Sun Cluster environment is mentioned, there are some other important concepts you might
encounter.
Installing and Configuring Sun Cluster 3.1 Software for Oracle Database 10g HA 7
to a global device, the Sun Cluster software automatically discovers another path to the device and
redirects the access to that path.
The Sun Cluster software mechanism that enables global devices is global namespaces, which provides
multiple failover paths to the multihost disks. Each node that is physically connected to multihost disks
provides a path to the storage for any node in the cluster. In the Sun Cluster system, each device node
in the local volume manager namespace is replaced by a symbolic link to a device node in the
/global/.devices/node@nodeID file system, where nodeID is an integer that represents a
node in the cluster.
After you install Sun Cluster software successfully, you see the difference simply by using the mount
command. For example, one raw disk in a Sun StorEdge FC 3510 FC array is described as follows in
/etc/vfstab:
/dev/did/dsk/d2s3 /dev/did/rdsk/d2s3 /global/.device/node@1 ufs 2 no global
The cluster file system enables the same path to locate a file, no matter where it is in the cluster, and is
dependent on global devices. The cluster file system on a global device can be created by using the
newfs or mkfs commands.
For cluster oraclecluster, raw disks in Sun StorEdge 3510 FC array are configured to RAID-5
and used as the global device, which offers better availability over volume manager-based
configurations and the least overhead.
In other words, a cluster can run only if a partition has the majority of votes. The majority number is
quorum. For example, in a six-node cluster, at least four (yes, four is the quorum for this cluster) nodes
should keep healthy providing service, and for a five-node cluster, at least three nodes are required.
In a two-node cluster, a majority is two, so how do you enable the healthy node to gain quorum? In this
case, an external vote by a quorum device is needed, as shown in Figure 3. A quorum device is a disk
shared by two or more nodes that contributes votes that are used to establish a quorum for the cluster to
run.
Quorum devices acquire quorum vote counts based on the number of node connections to the device.
Quorum devices acquire a maximum vote count of N-1, where N is the number of connected votes to
the quorum device.
The Sun StorEdge 3510 FC array is used as the quorum device for the two-node cluster
oraclecluster, which has a quorum count of one (two minus one). If interconnection between
node A and node B fails, only the node that can get the vote from the Sun StorEdge 3510 FC Array
forms a new cluster to keep running.
Installing and Configuring Sun Cluster 3.1 Software for Oracle Database 10g HA 8
Figure 3: Quorum Device for Two-Node Cluster
Data Service
The term data service describes an application that has been configured to run on a cluster rather than
on a single server. Sun Cluster software manages the application with the following actions:
• Start
• Stop
• Monitor and take corrective measures
Figure 4 shows an application that runs on a single application server and the same application running
on a cluster. The latter is more highly available because whenever one instance fails, the other keeps
providing service.
In this document, Oracle10g is configured into cluster oraclecluster, so failover between two
independent instances on each Sun Fire V480 server is executed.
Installing and Configuring Sun Cluster 3.1 Software for Oracle Database 10g HA 9
In cluster oraclecluster, you manage the following resources in the resource group
clustergroup:
• SUNW.oracle_server, for Oracle Server, is preregistered in the Sun Cluster software.
• SUNW.oracle_listener, for Oracle Listener, is preregistered in the Sun Cluster software.
• SUNW.HAStoragePlus, for failover of the cluster file system and to make the network
resource start up first, is also preregistered in the Sun Cluster software.
How about the necessary network resource? It is associated with a logical host name
(oracleserver). When one of the Sun Fire V480 servers fails, all the resources in the resource
group clustergroup migrate to the other node.
Cluster Topologies
A topology is the connection schema that connects the cluster nodes to the storage platforms that are
used in a Sun Cluster environment.
Since you can access the two nodes by connecting to the serial port or by using telnet from other
systems that have a screen, the topology shown in Figure 5, which is much simpler than the topology
in Figure 1 and contains only the core components (without the Administrative Console and the
console access device), is used for your first experience. However, in an actual deployment, you should
have all the components to simplify management.
Figure 5 is the topology used for oraclecluster, with two nodes that can be operated under a
single cluster administrative framework.
Installing and Configuring Sun Cluster 3.1 Software for Oracle Database 10g HA 10
Table 1: Software Installed
Service
Hardware Software
Module
Sun Fire V480 ● Solaris 10 01/06 OS for SPARC® platforms
server ● Sun Cluster 3.1 08/05 software (SPARC)
Database
● Oracle Database 10g Release 2 (10.2.0.1.0) for Solaris
Server Sun Fire V480 Operating System (SPARC) (64-bit)
server ● Sun Cluster HA for Oracle (SPARC)
Sun StorEdge
Storage N/A
3510 FC array
● Solaris 10 01/06 OS for x86 platforms
Test Client Fujitsu LifeBook ● Oracle Database 10g Client Release 1 (10.1.0.3) for
Solaris Operating System (x86)
The Solaris 10 01/06 OS for SPARC platforms is installed on each node. To meet the Sun Cluster
software's needs, use the file-system allocation shown in Table 2 on each Sun Fire V480 server.
0 / (according to This is used for the Solaris OS, Sun Cluster software, data-
your services software, Sun Cluster module agent packages, root file
demand) system, and database and application software.
1 swap 1024Mbytes 512 Mbytes is for the Solaris OS and 512 Mbytes is for the Sun
Cluster software.
2 /globaldevices 512Mbytes The Sun Cluster software later assigns this slice a different
mount point and mounts the slice as a cluster file system.
Installing and Configuring Sun Cluster 3.1 Software for Oracle Database 10g HA 11
Besides assigning IP addresses for the nodes as usual, a logical host name is necessary for the cluster.
Become superuser and confirm the host name determination order in the file
/etc/nsswitch.conf:
# vi /etc/nsswitch.conf
...
hosts: cluster files
...
For more information, refer to the Sun Cluster Software Installation Guide for Solaris OS
(http://docs.sun.com/app/docs/doc/819-0420).
The Sun Cluster software recognizes the NIC that connects to the public network and automatically
configures the other two NICs into an IPMI group, sc_ipmi0.
Configure RAID-5, and then make a file system and mount it within the Sun Cluster global file system.
For example, locate your file system on /dev/rdsk/d4s2. The following operations can be
performed on either node.
#mkdir -p /global/u01
#newfs /dev/rdsk/d4s2
Installing and Configuring Sun Cluster 3.1 Software for Oracle Database 10g HA 12
Installing Sun Cluster 3.1 08/05 Software
Step 1: Download and install.
The Sun Cluster software is contained in the Free Solaris Cluster/Java Availability Suite. You can
download the software from the Previous Release Download section of this web page:
http://www.sun.com/software/solaris/cluster/get.html. Sun Cluster Agents software is
available for free from the Sun Cluster Agents Download section of the same web page.
Copy the packages on each node, unpack them, and execute the installation on the primary node
(v480-1). Become superuser to perform all the commands.
# ./scinstall
Choose the right options during the installation:
...
1)Establish a new cluster using this machine as the first node
...
Option: 1
Cluster name:oraclecluster
Node name:v480-1
Transport junctions: No
first transport adapter:hm0
second transport adapter:hm1
The new file system to use is: /u01
……
Rebooting ...
Unpack the package and execute the installation on the secondary node, v480-2:
#./scinstall
...
• 2) Add this machine as a node in an established cluster
...
Option: 2
Step 2: Add PATH and MANPATH for the Sun Cluster software:
#vi /.profile
...
PATH=$PATH:/usr/cluster/bin
export PATH
MANPATH=$MANPATH:/usr/cluster/man/
export MANPATH
On each node, you can add an entry in /etc/vfstab to mount the cluster file system:
#vi /etc/vfatab
/dev/global/dsk/d4s2 /dev/global/rdsk/d4s2 /u01 ufs 2 no logging
#mountall
Installing and Configuring Sun Cluster 3.1 Software for Oracle Database 10g HA 13
In the previous command, /dev/global/dsk/d4s2 and /dev/global/rdsk/d4s2 stand for
the disks and raw disks acting as the global device on which the cluster file system is located.
Step 3: Become the superuser and confirm the current state of the cluster:
#su – root
#scstat
Step 2: Use the logical host name for Oracle listener and name service.
Become user oracle and add path for Oracle10g in $HOME/.profile.
Installing and Configuring Sun Cluster 3.1 Software for Oracle Database 10g HA 14
$su - oracle
$vi /u01/oraclehome/.profile
PATH=$PATH:/u01/oraclehome/product/10.2.0/Db_1/bin
export PATH
Add a listener, according to your $SID, with the default name LISTENER and change the host name
to oracleserver.
Add clusterdb in the service naming, change the host name to oracleserver, and restart the
Oracle database to load the modified parameters for the listener.
Create a failover resource group clustergroup to hold and manage all the resources. Register the
two nodes into the resource group:
#scrgadm -a -g clustergroup -h v480-1, v480-2
#scrgadm -a -t SUNW.HAStoragePlus
Create the resource oracle-hastp-rs of type SUNW.HAStoragePlus, and enable the highly
available cluster file system:
#scrgadm -a -j oracle-hastp-ra -g clustergroup -t SUNW.HAStoragePlus -x
FilesystemMountPoint=/u01, -x AffinityOn=True
#scswitch -Z -g clustergroup
Installing and Configuring Sun Cluster 3.1 Software for Oracle Database 10g HA 15
Verifying Sun Cluster 3.1 08/05 Software
Step 1: Become the superuser and determine which node is currently master of the Oracle resource:
#scstat -p
Step 4: After a while (depending on the Oracle server starting time), connect remotely again on the
other node:
# su – oracle
$ sqlplus /nolog
SQL> connect / as sysdba
SQL> select * from v$version;
Summary
Sun Cluster software prevents unexpected downtime by failing over and providing highly available
database access in the computing environment. Sun Cluster software can also provide advanced
features, such as scalable services, global access to devices and networking, and cluster file systems.
As you have seen, the Solaris 10 OS also plays big role in the solution by enabling IPMP, global file
system configuration, and Sun Cluster services management within the Service Management Facility
(SMF) framework.
Appendix
Here is the output of the scstat command:
---------------------------------------------------------------------
-- Cluster Nodes --
------------------------------------------------------------------
------------------------------------------------------------------
-- Quorum Summary --
Installing and Configuring Sun Cluster 3.1 Software for Oracle Database 10g HA 16
Quorum votes needed: 2
Quorum votes present: 3
------------------------------------------------------------------
------------------------------------------------------------------
-- Resource Groups --
-- Resources --
Installing and Configuring Sun Cluster 3.1 Software for Oracle Database 10g HA 17
Resource: oracle-server v480-2 Online Online
------------------------------------------------------------------
-- IPMP Groups --
------------------------------------------------------------------
Installing and Configuring Sun Cluster 3.1 Software for Oracle Database 10g HA 18
● Sun wikis at http://wikis.sun.com/dashboard.action, for example:
● Sun Cluster Wiki: http://wikis.sun.com/display/SunCluster/Home
● Sun BluePrints™ Wiki: http://wikis.sun.com/display/BluePrints/Main
● BigAdmin Wiki: http://wikis.sun.com/display/BigAdmin/Home
Licensing Information
Unless otherwise specified, the use of this software is authorized pursuant to the terms of the license
found at http://www.sun.com/bigadmin/common/berkeley_license.html.
Installing and Configuring Sun Cluster 3.1 Software for Oracle Database 10g HA 19