Oracle

Oracle

Oracle Clusterware Real Application Clusters

Oracle
Oracle Clusterware RAC

1Windows Windows Oracle
2Linux Linux Oracle
3Oracle

http://www.koorka.com

1
2

3 3

http://www.koorka.com

Versionbeta
www.koorka.com
 Oracle

1 Oracle

Oracle

Oracle

Oracle

Versionbeta
www.koorka.com
 Oracle

1. Oracle

1.1 Oralce

Oracle Oracle

1-1 Oracle
Listener
Oracle RAC
clusterware

RawOracle OCFSASM
NAS

Server 1 Server 2 Server 3 Server N

Listener Listener Listener Listener
Oracle RAC Oracle RAC Oracle RAC Oracle RAC

ClusterWare ClusterWare ClusterWare ClusterWare

OS OS OS OS

Raw / CFS / NAS / ASM

1-1

Oracle Oracle
RAC Oracle ClusterWareOracle clusterware RAC

1-2

Versionbeta
www.koorka.com
 Oracle

Oracle 1-2

1 2

RAC
Oracle Oracle

SPFILESRVCTL
1-2

1-3

Instance A Instance B

SGA SGA


.

1 2

1-3

Versionbeta
www.koorka.com
 Oracle

1.2 Oracle Clusterware

1.2.1 Oracle Clusterware

Oracle Clusterware
Oracle Clusterware

Oracle Clusterware
1voting disk
OCFS RAC

2Oracle Cluster RegistryOCR

Oracle Clusterware Oracle Clusterware

Oracle Clusterware
RAC Oracle Clusterware
RAC
1-4 Oracle
Oracle Clusterware RAC 202.127.124.0/24
Oracle Clusterware RAC
RAC 202.127.124.0/24Oracle Clusterware 10.10.10.0/24

1 2

1-4

Oracle Clusterware
1-5

Versionbeta
www.koorka.com
 Oracle

1 2

1-5

1.2.2 Oracle Clusterware

Oracle Clusterware 1-1 Oracle

Clusterware

1-1 Clusterware

Cluster Synchronization
Services (CSS)

CSS

Cluster Ready Services high availability
(CRS) cluster resourcedatabase,
instance serviceListener IP (VIP)
application process
CRS OCR

crs
Oracle Clusterware RAC crs Oracle
crs
crs 5
Event Management crs
(EVM)

Versionbeta
www.koorka.com
 Oracle

Oracle Notification Fast Application Notification (FAN)

Service (ONS)
Oracle
RACG Clusterware Oracle
FAN
Process Monitor I/O fencing

Daemon (OPROCD)

1-2 Windows

1-2 Clusterware Windows

Linux/Unix Windows Windows
Process Monitor oprocd (r) OraFenceService
Daemon
RACG racgmain, racgimon racgmain.exe
racgimon.exe
Oracle Notification ons ons.exe
Service
Event Manager evmd (r), evmd.bin, OracleEVMService evmlogger.exe,
evmlogger evmd.exe
Cluster Ready crsd.bin (r) OracleCRSService crsd.exe
Cluster init.cssd (r), ocssd (r), OracleCSService ocssd.exe
Synchronization ocssd.bin
Services

Linux/Unix
r root Oracle Oracle

1.3 RAC

1.3.1 RAC

RAC Oracle Oracle Clusterware

Oracle RAC
RAC datafile
control filePFILEredo log file
RAC
RAC Clusterware
RAC
RAC Oracle10g ASM

Versionbeta
www.koorka.com
 Oracle

RAC RAC
1ASM
2Oracle OCFS2windows Linux

RAC
3NFS
4Raw devices

RAC RAC

1redo thread
2undo tablespace

Oracle LAN
RAC Oracle

unique nameload balancefailover

Oracle
Net Services

1.3.2 RAC

RAC RAC

SGA buffer cacheCache Fusion

RAC SGA
SGA 1-6
,

2. 3.

A B
C D

1. 4.

1-6

Versionbeta
www.koorka.com
 Oracle

RAC Global Cache

Service (GCS) the Global Enqueue Service (GES)GCS GES Global Resource
Directory (GRD)GRD
RAC SGA
RAC GRD RAC RAC
1LMSGlobal Cache Service process
2LMDGlobal Enqueue Service Daemon
3LMONGlobal Enqueue Service Monitor
4LCK0Instance Enqueue Process

1.4 LinuxX86 Oracle

1-3 Oracle

1-3

Clusterware 500M
voting disk 20M
OCROracle 100M
Oracle 1.3GB
RAC 1.2GB
Recovery File 2GB

1-4 Oracle

1-4

Clusterware NFS NAS
ext2 NFS
ext3
voting disk OCFS2 Raw deviceNFS NFS
OCR OCFS2 Raw deviceNFS NFS
OCFS2 NFS
NFS
ext2ext3 OCFS2
OCFS2ASMRaw device NFS
NFS
OCFS2 ASMNFS NFS

NAS
NAS Oracle
Oracle Home

Versionbeta
www.koorka.com
 Oracle

OCFS2
Clusterware Oracle 1-5 Clusterware Oracle

1-5 Clusterware Oracle

Clusterware
ASM
OCFS2
Raw Device
NFS
Clusterware voting disk OCR

1-5 OCFS2 NFS

Oracle10g ASM OCFS2
Clusterware Oracle ASM

Linuxx86 Oracle
1 Oracle
Clusterware ASM 1-6
1-6

Clusterware ext2ext3
voting disk Raw device
OCR Raw device
ext2ext3
ASM
ASM

2 OCFS2 Oracle OCFS2

Orache Home 1-7
1-7

Clusterware ext2ext3
voting disk OCFS2
OCR OCFS2
OCFS2
OCFS2
OCFS2

3 Oracle Orache Home

ASM 1-8

Versionbeta
www.koorka.com
 Oracle

1-8

Clusterware ext2ext3
voting disk OCFS2
OCR OCFS2
OCFS2
ASM
ASM

3 I/O

2. Linux Oracle
x86 Linux Oracle RAC
CPU P4
RAM 1024M
SCSI 9.1GB
SCSI 9.1GB
2
Red Hat Enterprise Linux AS 4 Update 2

1-7

SCSI

1-7

Versionbeta
www.koorka.com
 Oracle

IP
211.152.14.110 koorka_rac01 public DNS /etc/hosts
211.152.14.109 koorka_rac02 public DNS /etc/hosts
211.152.14.210 koorka_vip01 Virtual DNS /etc/hosts
211.152.14.209 koorka_vip02 Virtual DNS /etc/hosts
10.10.10.1 koorka_priv01 Private /etc/hosts
10.10.10.2 koorka_priv02 Private /etc/hosts

2.1

2.1.1

x86 Linux Oracle

x86

Red Hat Enterprise Linux AS/ES 3 (Update 3 )

Red Hat Enterprise Linux AS/ES 4 (Update 1 )
SUSE Linux Enterprise Server 9 (Service Pack 2 )
Red Hat Enterprise Linux 3 (Update 4):
2.4.21-27.EL

Red Hat Enterprise Linux 4 (Update 1):

2.6.9-11.EL

SUSE Linux Enterprise Server 9 (Service Pack 2):

2.6.5-7.97
RAM 1024MB
1.5 x RAM
TCP/IP
2 x NICs

Oracle
Red Hat Enterprise Linux 3
(Update 2) Packages
binutils-2.14
compat-db-4.0.14-5
compat-gcc-7.3-2.96.128
compat-gcc-c++-7.3-2.96.128
compat-libstdc++-7.3-2.96.128
compat-libstdc++-devel-7.3-2.96.128

Versionbeta
www.koorka.com
 Oracle

Red Hat Enterprise Linux 4
(Update 1) Packages
gcc-3.2
glibc-2.3.2-95.27
make-3.79
openmotif-2.2.3
setarch-1.3-1
binutils-2.15.92.0.2-10.EL4
compat-db-4.1.25-9
control-center-2.8.0-12
gcc-3.4.3-9.EL4
gcc-c++-3.4.3-9.EL4
glibc-2.3.4-2
glibc-common-2.3.4-2
gnome-libs-1.4.1.2.90-44.1
libstdc++-3.4.3-9.EL4
libstdc++-devel-3.4.3-9.EL4
make-3.80-5

:
Oracle demos openmotif21-2.1.30-11.RHEL4.2

Red Hat Enterprise Linux 4.0 GCC 2.96
SUSE Linux Enterprise gcc-3.3
Server 9 (Service Pack 2) gcc-c++-3.3.3-43
Packages glibc-2.3.3-98.28
libaio-0.3.98-18
libaio-devel-0.3.98-18
make-3.80
openmotif-libs-2.2.2-519.1

Oracle 10201_database_linux32.zip 10201_clusterware_linux32.zip

2.1.2

Oracle 100 MB

400 MB

Versionbeta
www.koorka.com
 Oracle

500 MB


1.7GB



Voting disk 120M
OCR
3.5 GB

2.1.3

IP
4 IP DNS hosts
IPReal IP
Oracle IPVirtual IP

2 IP /etc/hosts

2.2 OCFS2 1

Linux (x86) Oracle RAC

OCFS2 Oracle Home
koorka_rac01 koorka_rac02

CPU p4

1024M
2
1 SCSI9.1GB
1 SCSI9.1GB
Redhat Enterprise AS 4 Update 2

Oracle RAC


Clusterware ext3
voting disk OCFS2
OCR OCFS2
OCFS2
OCFS2

Versionbeta
www.koorka.com
 Oracle

OCFS2
Oracle
Oracle Clusterware
Oracle
1-8

211.152.14.110 211.152.14.109

Oracle Clusterware Oracle Clusterware

Oracle
RAC
Voting disk

OCR

1-8

clusterware /orac/crs
Oracle /orac/orahome
voting diskORCDatafileRecovery File/orac/oradata

2.2.1

2.2.1.1

Oracle

1OSDBA ( dba)
Oracle Oracle
(SYSDBA privilege) dba
Oracle
(dba) Oracle Universal Installer

[root@koorka_rac01 ~]# groupadd g 1001 dba

2OSOPER (oper
Oracle (

Versionbeta
www.koorka.com
 Oracle

SYSOPER privilege)OSDBA
SYSOPER
OSOPER Oracle
dba Oracle Universal Installer
oper
OSOPER OSDBA dba

3
nobody nobody
(extjob)
nobody
[root@koorka_rac01 ~]# id nobody
uid=99(nobody) gid=99(nobody) groups=99(nobody)
nobody

4Oracle Oracle Inventory ( oinstall)

Oracle oinstall
Oracle (Oracle inventory)

[root@koorka_rac01 ~]# groupadd g 1002 oinstall

5Oracle ( oracle)
Oracle
Oracle Inventory (oinstall) OSDBA
OSOPERY

[root@koorka_rac01 ~]# useradd -u 1001 -g oinstall -G dba oracle

6 Oracle

ID ID

2.2.1.2

1 IP
koorka_rac01
[root@koorka_rac01 ~]# ifconfig eth0 211.152.14.110
[root@koorka_rac01 ~]# ifconfig eth1 10.10.10.1

koorka_rac02
[root@koorka_rac02 ~]# ifconfig eth0 211.152.14.109
[root@koorka_rac02 ~]# ifconfig eth1 10.10.10.2

Versionbeta
www.koorka.com
 Oracle

2
/etc/hosts
211.152.14.110 koorka_rac01
211.152.14.210 koorka_vip01
211.152.14.109 koorka_rac02
211.152.14.209 koorka_vip02
10.10.10.1 koorka_priv01
10.10.10.2 koorka_priv02

2.2.1.3 SSH

Oracle RAC oracle secure shell

(SSH)OUI ssh scp
oracle secure shell
1. RSA DSA
1 oracle

2 oracle .ssh
[oracle@koorka_rac01 ~]$ mkdir ~/.ssh
[oracle@koorka_rac01 ~]$ chmod 700 ~/.ssh

3 ssh-keygen 2 SSH RSA

[oracle@koorka_rac01 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_rsa.
Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
17:e4:7c:79:8d:a0:00:3b:d9:f7:7a:56:f3:ac:54:4d oracle@koorka_rac01

keypublic key~/.ssh/
passphrase ssh

4 ssh-keygen 2 SSH DSA 3

[oracle@koorka_rac01 ~]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_dsa.
Versionbeta
www.koorka.com
 Oracle

Your public key has been saved in /home/oracle/.ssh/id_dsa.pub.

The key fingerprint is:
03:77:b5:d6:cd:51:2f:c4:9a:9f:b2:b5:23:5f:a8:52 oracle@koorka_rac01

2. authorized key file

1 Oracle

2 Oracle OUI koorka_rac01

[oracle@koorka_rac01 ~] $ cd ~/.ssh
[oracle@koorka_rac01 .ssh]$ ssh koorka_rac01 cat /home/oracle/.ssh/id_rsa.pub >>
authorized_keys
[oracle@koorka_rac01 .ssh]$ ssh koorka_rac01 cat /home/oracle/.ssh/id_dsa.pub >>
authorized_keys
[oracle@koorka_rac01 .ssh]$ ssh koorka_rac02 cat /home/oracle/.ssh/id_rsa.pub >>
authorized_keys
[oracle@koorka_rac01 .ssh]$ ssh koorka_rac02 cat /home/oracle/.ssh/id_dsa.pub
>>authorized_keys
[oracle@koorka_rac01 .ssh]$ scp authorized_keys koorka_rac02:/home/oracle/.ssh/
[oracle@koorka_rac01 .ssh]$ chmod 600 ~/.ssh/authorized_keys

koorka_rac02
[oracle@koorka_rac02 .ssh] chmod 600 ~/.ssh/authorized_keys

3
koorka_rac01
[oracle@koorka_rac01 ~]$ ssh koorka_rac01 date
[oracle@koorka_rac01 ~]$ ssh koorka_rac02 date

SSH

2.2.1.4

rpm -q binutils compat-db control-center gcc gcc-c++ glibc gnome-libs libstdc++
libstdc++-devel make openmotif21

2.2.1.5

/etc/sysctl.conf
#For Oracle

Versionbeta
www.koorka.com
 Oracle

kernel.sem = 250 32000 100 128

kernel.shmmni = 4096
kernel.shmall = 2097152
kernel.shmmax = 2147483648
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default = 1048576
net.core.rmem_max = 1048576
net.core.wmem_default = 262144
net.core.wmem_max = 262144

sysctl p

2.2.1.6 Shell Oracle

/etc/security/limits.conf
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536

/etc/pam.d/login
session required /lib/security/pam_limits.so

/etc/profile
if [ $USER = "oracle" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi

2.2.1.7 OCFS2

OCFS2 Oracle Oracle 10.2

OCFS2 Oracle Home

1. OCFS2

Versionbeta
www.koorka.com
 Oracle

1
unamr r
[root@koorka_rac01 ~]# uname -r
2.6.9-22.ELsmp
Redhat Enterprise AS 4 Update 22.6.9-22.ELsmp
OCFS2kernel module
http://oss.oracle.com/projects/ocfs2/dist/files/RedHat/RHEL4/i386/1.2.3-1/2.6.9-22.EL/ocfs2
-2.6.9-22.ELsmp-1.2.3-1.i686.rpm
OCFS2 tools OCFS2
http://oss.oracle.com/projects/ocfs2-tools/dist/files/RedHat/RHEL4/i386/1.2.1-1/ocfs2-tools-
1.2.1-1.i386.rpm
http://oss.oracle.com/projects/ocfs2-tools/dist/files/RedHat/RHEL4/i386/1.2.1-1/ocfs2consol
e-1.2.1-1.i386.rpm

2 rpm
[root@koorka_rac01 ~]# rpm ivh ocfs2-tools-1.2.1-1.i386.rpm
[root@koorka_rac01 ~]# rpm ivh ocfs2-2.6.9-22.ELsmp-1.2.3-1.i686.rpm
[root@koorka_rac01 ~]# rpm ivh ocfs2console-1.2.1-1.i386.rpm

2. OCFS2
1 koorka_rac01
Oracle 3000M
Oracle 4000M
[root@koorka_rac01 ~]# fdisk -l /dev/sdb

Disk /dev/sdb: 9318 MB, 9318694912 bytes

255 heads, 63 sectors/track, 1132 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

/dev/sdb1 1 487 3911796 83 Linux
/dev/sdb2 488 1132 5180962+ 83 Linux

2 koorka_rac01
koorka_rac01 root

[root@koorka_rac01 ~]# ocfs2console

ocfs2 1-9

Versionbeta
www.koorka.com
 Oracle

-19

3 koorka_rac01
OCFS2 TasksFormat OCFS2
1-10

1-10

Available devices
/dev/sdb1 Oracle
orahome 1-11
Oracle Home
/dev/sdb2 oradata

Versionbeta
www.koorka.com
 Oracle

1-11

1-12

1-12

4 koorka_rac01
OCFS2 Console ClusterConfigure Nodes
OCFS2 1-13

Versionbeta
www.koorka.com
 Oracle

1-13

Node ConfigurationAdd 1-14

1-14

1-15 Apply
close

1-15

/etc/ocfs2/cluster.conf
node:
ip_port = 7777
ip_address = 211.152.14.110
number = 0
name = koorka_rac01
cluster = ocfs2

node:
ip_port = 7777

Versionbeta
www.koorka.com
 Oracle

ip_address = 211.152.14.109
number = 1
name = koorka_rac02
cluster = ocfs2

cluster:
node_count = 2
name = ocfs2

5 koorka_rac01
OCFS2 Console ClusterPropagate Configuration 1-16
/etc/ocfs2/cluster.conf

1-16

1-17 koorka_rac02 root

Finishedclose

1-17

6 o2cb
o2cb OCFS2

/etc/init.d/o2cb configure

Configuring the O2CB driver.

This will configure the on-boot properties of the O2CB driver.

The following questions will determine whether the driver is loaded on
boot. The current values will be shown in brackets ('[]'). Hitting
<ENTER> without typing an answer will keep that current value. Ctrl-C

Versionbeta
www.koorka.com
 Oracle

will abort.

Load O2CB driver on boot (y/n) [y]: y

Cluster to start on boot (Enter "none" to clear) [ocfs2]:
Writing O2CB configuration: OK
Cluster ocfs2 already online

2
ocfs2

7

/etc/init.d/o2cb status

Module "configfs": Not loaded
Filesystem "configfs": Not mounted
Module "ocfs2_nodemanager": Not loaded
Module "ocfs2_dlm": Not loaded
Module "ocfs2_dlmfs": Not loaded
Filesystem "ocfs2_dlmfs": Not mounted

/etc/init.d/o2cb status

/etc/init.d/o2cb status
Module "configfs": Loaded
Filesystem "configfs": Mounted
Module "ocfs2_nodemanager": Loaded
Module "ocfs2_dlm": Loaded
Module "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking cluster ocfs2: Online
Checking heartbeat: Not active

8 OCFS2
/dev/sdb1 /orac/orahome/dev/sdb2 /orac/oradata

mkdir p /orac/orahome
mkdir p /orac/oradata

mount -t ocfs2 /dev/sdb1 /orac/orahome

mount -t ocfs2 -o datavolume,nointr /dev/sdb2 /orac/oradata
data files, Voting disk Oracle OCFS2
datavolume nointr

Versionbeta
www.koorka.com
 Oracle

/dev/sdb1 /dev/sdb2/etc/fstab
/dev/sdb2 /orac/oradata ocfs2 _netdev, datavolume, nointr 0 0
/dev/sdb1 /orac/orahome ocfs2 _netdev 0 0

OCFS2 mounted.ocfs2 f

Device FS Nodes
/dev/sdb1 ocfs2 koorka_rac02, koorka_rac01
/dev/sdb2 ocfs2 koorka_rac01, koorka_rac02

2.2.2

1. X-windows koorka_rac01
root koorka_rac01

[root@koorka_rac01 ~]#xhost +127.0.0.1

127.0.0.1 X

2.

1/orac/crs Clusterware
chown -R root:oinstall /orac/crs
chmod -R 775 /orac/crs

2/orac/orahome Oracle
chown -R oracle:oinstall /orac/orahome
chmod -R 775 /orac/orahome

3/orac/oradata Oracle
chown -R oracle:oinstall /orac/oradata
chmod -R 775 /orac/oradata

3. koorka_rac01
1 Oracle
[root@koorka_rac01 ~]# su - oracle

2 10201_clusterware_linux32.zip oracle

3~/clusterware/runInstaller

Versionbeta
www.koorka.com
 Oracle

[oracle@koorka_rac01 clusterware]$ ~/clusterware/runInstaller

OUINext

4. ORACLE_HOME
ClusterWare ORACLE_HOME /orac/crs/10.2.0/
Path /orac/crs/10.2.0/ 1-18

1-18

5.
1-19Next

1-19

6.
1 koorka_rac01
koorka_priv01 koorka-vip01 2
koorka_rac02 koorka_priv02 koorka_vip02 1-20

Versionbeta
www.koorka.com
 Oracle

1-20

Edit
AddNext

7.
IP public IP
private 1-21
EditNext

1-21

Versionbeta
www.koorka.com
 Oracle

8.
OCROracle Cluster Register voting disk

/orac/oradata OCR
/orac/oradata/clusterdata/ocrdata voting disk /orac/oradata/clusterdata/votedisk

1 OCR External Redundancy

Specify OCR Location/orac/oradata/clusterdata/ocrdata 1-22

1-22

2 voting disk External Redundancy

Voting Disk Location/orac/oradata/clusterdata/votedisk 1-23

Versionbeta
www.koorka.com
 Oracle

1-23

9.
Install 1-24

1-24

10.
1-25

Versionbeta
www.koorka.com
 Oracle

1-25

root
/home/oracle/oraInventory/orainstRoot.sh
/orac/crs/10.2.0/root.sh
OK

Exit OUI

2.2.3

1.
1 Oracle
[root@koorka_rac01 ~]# su - oracle

2 10201_database_linux32.zip oracle

3~/database/runInstaller
[oracle@koorka_rac01 clusterware]$ ~/database/runInstaller
OUINext

2.
Enterprise Edition
1-26

1-26

Versionbeta
www.koorka.com
 Oracle

3.
OCFS2 /orac/orahome /orac/oradata
/orac/orahome Oracle Home 1-27

1-27

4.
1-28 Next

1-28
5.

Next

6.
install database software only 1-29
OCFS2
Next

Versionbeta
www.koorka.com
 Oracle

1-29
7.
Install
1-30

1-30
8.
1-31 root
/orac/orahome/10.2.0/db_1/root.sh

1-31

Versionbeta
www.koorka.com
 Oracle

OKExit OUI

2.2.4

1. dbca
dbca Oracle Real Application
1-32 Next
Clusters database

1-32
Create DatabaseNext

2.
1-33 Next

1-33

3.
General Purpose 1-34

1-34

4.
SID 1-35 Next

Versionbeta
www.koorka.com
 Oracle

1-35

5.
Confilgure the Database with Enterprise Manager
1-36 Enterprise Manager Next

1-36

6.
1-37Next

Versionbeta
www.koorka.com
 Oracle

1-37
7.
Cluster File SystemNext

8.
Use Common Location for All Database Files
/orac/oradata 1-38 Next

1-38
9.
/orac/oradata/flash_recovery 1-39Next

1-39

Versionbeta
www.koorka.com
 Oracle

10.
Sample Schemas
Next

11. TAF
TAF 1-40 TAF
Next

1-40

12.
Oracle
1-41 Next

1-41

13.
Next

Versionbeta
www.koorka.com
 Oracle

14.
Finish
1-42
yes

1-42

2.3 ASM 2

2.3.1

2.3.2

2.3.3

Versionbeta
www.koorka.com