Page 1 of 4

">

RAC and Oracle Clusterware Best Practices and Starter Kit (Solaris) [ID 811280.1]
Modified 12-OCT-2011

Type BULLETIN

Status PUBLISHED

In this Document
Purpose
Scope and Application
RAC and Oracle Clusterware Best Practices and Starter Kit (Solaris)
RAC and Oracle Clusterware Best Practices and Starter Kit (Platform Independent)
RAC Platform Specific Best Practices and Starter Kits

RAC on Solaris Step by Step Installation Instructions

RAC on Solaris Best Practices

Hardware/Vendor Specific Considerations
Oracle Software Considerations
References

Applies to:
Oracle Server - Enterprise Edition - Version: 10.2.0.1 to 11.2.0.1 - Release: 10.2 to 11.2
Oracle Solaris on SPARC (64-bit)

Purpose
The goal of the Oracle Real Application Clusters (RAC) series of Best Practice and Starter Kit notes
provides customers with quick knowledge transfer of generic and platform specific best practices for
implementing, upgrading and maintaining an Oracle RAC system. This document is compiled and
maintained based on Oracle's experience with its global RAC customer base.
This Starter Kit is not meant to replace or supplant the Oracle Documentation set, but rather, it is
meant as a supplement to the same. It is imperative that the Oracle Documentation be read,
understood, and referenced to provide answers to any questions that may not be clearly addressed by
this Starter Kit.
All recommendations should be carefully reviewed by your own operations group and should only be
implemented if the potential gain as measured against the associated risk warrants implementation.
Risk assessments can only be made with a detailed knowledge of the system, application, and
business environment.
As every customer environment is unique, the success of any Oracle Database implementation,
including implementations of Oracle RAC, is predicated on a successful test environment. It is thus
imperative that any recommendations from this Starter Kit are thoroughly tested and validated using a
testing environment that is a replica of the target production environment before being implemented in
the production environment to ensure that there is no negative impact associated with the
recommendations that are made.

Scope and Application

This article applies to all new and existing RAC implementations as well as RAC upgrades.

RAC and Oracle Clusterware Best Practices and Starter Kit (Solaris)
RAC and Oracle Clusterware Best Practices and Starter Kit (Platform Independent)
The following document focuses on RAC and Oracle Clusterware Best Practices that are applicable to
all platforms including a white paper on available RAC System Load Testing Tools and RAC System
Test Plan outlines for 10gR2 & 11gR1 and 11gR2:
Document 810394.1 RAC and Oracle Clusterware Best Practices and Starter Kit (Platform
Independent)

RAC Platform Specific Best Practices and Starter Kits

The following notes contain detailed platform specific best practices including Step-By-Step
installation cookbooks (downloadable in PDF format):

Document 811306.1 RAC and Oracle Clusterware Best Practices and Starter Kit (Linux)
Document 811280.1 RAC and Oracle Clusterware Best Practices and Starter Kit (Solaris)
Document 811271.1 RAC and Oracle Clusterware Best Practices and Starter Kit (Windows)
Document 811293.1 RAC and Oracle Clusterware Best Practices and Starter Kit (AIX)
Document 811303.1 RAC and Oracle Clusterware Best Practices and Starter Kit (HP-UX)

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=BULLET... 12/19/2011

Page 2 of 4

RAC on Solaris Step by Step Installation Instructions

Click here for a Step By Step guide for installing Oracle RAC 10gR2 on Solaris.
Click here for a Step By Step guide for installing Oracle RAC 11gR1 on Solaris.
Click here for a Step By Step guide for installing Oracle RAC 11gR2 on Solaris.

RAC on Solaris Best Practices

The Best Practices in this section are specific to the Solaris Platform. That said, it is essential that
the Platform Independent Best Practices found in Document 810394.1 be reviewed in addition to
the content provided in this Document.

OS Configuration Considerations
Validate your hardware/software configuration against the RAC Technologies Matrix for
Unix.
11gR2 requires Solaris 10 update 6 or greater. Reference Document 971464.1.
Ensure all required OS packages are installed and system prerequisites have been properly
implemented for your particular release of Oracle. This information is documented in
Document 169706.1 as well as the install guides for your particular release.
In pre-11gR2 clusters, system times are to be synchronized across cluster nodes using
NTPD and NTPD should be configured to slew time to prevent false reboots. Configure NTP
client as per Document 759143.1 to take corrective action on this issue.
With 11gR2, Cluster Time Synchronization Daemon (CTSSD) can be used in place of NTPD.
CTSSD will synchronize time with a reference node in the cluster when an NTPD is not
found to be configured. Should you require synchronization from an external time source
you must use NTPD which will cause CTSSD to run in "observer" mode. However, if NTP is
running, then it must to be configured with the slewing option as documented in Document
551704.1.
Ensure SunOS 5.10 kernel patch 127111-03 or higher is installed in order to address a
condition with a deadlock in the Solaris 10 paging algorithm. Reference Document 460424.1.
There is an interoperability issue between Solaris 10's memory management and Oracle's
Automatic PGA Memory Management in a Solaris 10 environment with a large number of
CPUs. The issue causes CPU spikes happening at random intervals and increases system
cpu time. Corrective action can be found in Document 460424.1.
Solaris Local Containers are supported, refer to Supported Virtualization Matrix for details.

Storage Considerations
Skip the first 1Mb when creating raw device to avoid overwriting the disk VTOC. See
Document 367715.1 for details.
When using NFS, ensure that the correct mount options are used. The proper mount options
for NFS are defined in Document 359515.1.
Use Solaris MPxIO or other 3rd party IO multi-pathing to protect against HBA or SAN switch
failures.
Make sure Solaris SAN Foundation Kit (SFK) patches are installed for Solaris 8 and Solaris
9. Reference Document 392639.1

Network Considerations
For 11.2.0.2 (GI and RDBMS) and above, it is highly recommended that Oracle Redundant
Interconnect/HAIP be used for interconnect redundancy. See Document 1210883.1 for
details.
For pre-11.2.0.2 environments use IPMP NIC redundancy for the private interconnect.
Configuration details can be found in the following notes:
For 11.2.0.1 see Document 1069584.1 - Solaris IPMP and Trunking for the cluster
interconnect in Oracle Grid Infrastructure 11g Rel. 2
For pre-11gR2 see Document 368464.1 - How to Setup IPMP as Cluster Interconnect

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=BULLET... 12/19/2011

Page 3 of 4

For all releases use IPMP for public NIC redundancy. Configuration details can be found in
Document 730732.1.
The Solaris OS default setting for udp_recv_hiwat and udp_xmit_hiwat are too low. For
heavy cluster interconnect traffic, Oracle has increased the recommended value of both
kernel parameters to at least 65536 to improve UDP throughput. See the Oracle Grid
Infrastructure Installation Guide 11g Release 2 (11.2) for Solaris Operating System (same
applies to pre-11gR2 installations).
For Solaris versions earlier than Solaris 10, tune the sq_max_size to protect against
dropped packets on the interconnect. See Document 1004755.1 for details.
Add set ce:ce_taskq_disable=1 to /etc/system if the ce network interface is used for
cluster interconnect to prevent frequent node reboots. Reference Document 437420.1
E1000 NIC can cause kernel panic or CSS packet corruption on Solaris 10 and patch 11883330 or later for SPARC Platform is required.
Jumbo frames with MTU 9000 may cause problems with HP NC7170 drivers.

Hardware/Vendor Specific Considerations

Ensure minimum BIOS version 2.35.3.3 is used for SUN V40Z DUAL CORE machines, for
ECC memory checking.
Ensure SUN V40Z 2.6V memory management voltage regulator issues. A SUN CE can
identify if the voltage regulator is beginning to fail. The new VRM (Voltage Regulator
Module) revision board from rev 1.0 to rev 2.0.
T-Series machines on SUN use a threading model to make it look like there are more CPUs
then the machine actually has. On these Cool Threads machines, (the example here is a
machine with 32 cores), it appears to Oracle that there are actually 8 times as many CPUs (8
threads per core) - so we automatically set CPU_COUNT to 256, and then the DB will not
open. The workaround here is to manually set the CPU_COUNT to a reasonable value (i.e.
16, or 32, etc, depending on the # of actual cores). See Document 781763.1 and Document
1181315.1.
Ensure the network cards don't use the shared PCI-X bus slots 2 and 3 on a SUN V40Z.

Oracle Software Considerations

The Software Considerations in this section are specific to the Solaris Platform. That said, it is
highly recommended that the Platform Independent Best Practices found in Document 810394.1
be reviewed.
For 11.2.0.1 where IPMP is used for public and/or cluster interconnect, critical merge Patch
10094017 should be applied to both Grid Infrastructure and RDBMS Oracle homes. See
Document 1069254.1 for details.
If using IPMP for the cluster interconnect in an 11.2.0.2 environment be sure to take
corrective action on Bug 10357258 (Many HAIP created after active NIC fails in IPMP) by
applying Patch 12666373 (includes the 11.2.0.2.2 GI PSU). This issue is fixed in 11.2.0.2.3
(GI PSU3).
For 11.2.0.2 when using Jumbo Frames for the interconnect HAIP may not start with the
proper MTU size, see Document 1290585.1. It is recommended to apply Patch 12666373
(includes the 11.2.0.2.2 GI PSU) which corrects the following issues (in addition to the
11.2.0.2.2 fixes):
Bug 10357258 - [IPMP] HUNDREDS OF DUP IP AFTER INTRA-NODE FAILOVER
Bug 9795321 - MTU SIZE FOR VIP UNDER 11GR2 GRID INFRASTRUCTURE
Note: Bug 10357258 is corrected in GI PSU3 (11.2.0.2.3), Bug 9795321 is not
corrected in 11.2.0.2.3 at the time of this writing (workaround by disabling Jumbo
Frames). Both bugs are to be fixed in the upcoming 11.2.0.2.4 GI PSU.

References

Attachments
RACGuides_Rac10gR2OnSolaris.pdf (23,209.51 KB)
RACGuides_Rac11gR1OnSolaris.pdf (21,805.84 KB)
RACGuides_Rac11gR2OnSolaris.pdf (9,461 KB)

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=BULLET... 12/19/2011

Page 4 of 4

Related

Products
Oracle Database Products > Oracle Database > Oracle Database > Oracle Server Enterprise Edition
Keywords
CONFIGURATION; CRS; KIT; RAC; SOLARIS; VIP
Errors
ENOMEM; RFC-952; ORA-27102; ORA-7445

Back to top
Copyright (c) 2007, 2010, Oracle. All rights reserved. Legal Notices and Terms of Use | Privacy
Statement

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=BULLET... 12/19/2011