You are on page 1of 5

How to Add CSS Voting Disk Mirrors on Linux Using Shared Raw  

Partitions [ID 329734.1]

  Modified 12-NOV-2008     Type HOWTO     Status PUBLISHE  
D

In this Document
  Goal
  Solution
     1. Partition the shared disk
     2. Set Partitions to be used as RAW Devices
     3. Create symbolic links to the raw partitions and set ownership to oracle:dba:
     4. To add the voting disk mirror, run 'crsctl add css votedisk' command as root:
     5. Query the status of all voting disks:
     Conclusion
  References

Applies to:

Oracle Server - Enterprise Edition - Version: 10.2.0.1 to 10.2.0.3


Linux x86

Goal

Voting disk mirroring can be implemented on shared raw partitions and on a supported
shared filesystem - either Oracle Cluster File System (OCFS) or Network FileSystem
(NFS). In this article we will show how to implement voting disk mirroring on raw
partitions.

At the time of installation the clusterware in this setup was configured with one voting
disk (/ora10g/app/oracle/oradata/V10/VotingDisk1) on a shared raw partition:
# cd /ora10g/app/oracle/oradata/V10/
# ls -l

total0
lrwxrwxrwx 1 oracle dba 14 Jul 25 14:58 OracleClusterRegistry ->/dev/raw/raw28
lrwxrwxrwx 1 oracle dba 14 Jul 29 11:38 VotingDisk1 -> /dev/raw/raw29

We would like to add two voting disk mirrors.

Solution

To achieve this we need to partition the shared disk, make the partitions available to the
OS as raw devices, optionally create symbolic links to these raw devices and add them
as mirrored voting disks.
1. Partition the shared disk

In this setup the shared disk is seen as /dev/sdb from both nodes in the two node RAC
setup:

# fdisk -l /dev/sdb

Disk /dev/sdb: 80.0 GB, 80026361856 bytes


255heads, 63 sectors/track, 9729 cylinders
Units= cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System

/dev/sdb1 1 1217 9775521 83 Linux


/dev/sdb2 1218 2434 9775552+ 83 Linux
/dev/sdb3 2435 3651 9775552+ 83 Linux
/dev/sdb4 3652 9729 48821535 5 Extended
...
/dev/sdb13 8538 8562 200781 83 Linux
/dev/sdb14 8563 8587 200781 83 Linux
/dev/sdb15 8588 8650 506016 83 Linux

2. Set Partitions to be used as RAW Devices

Partitions /dev/sdb2 and /dev/sdb15 will be used for voting disk mirroring. To set them
up as raw devices, with permissions 600 and ownership assigned to oracle:dba,run the
following as root:

# /usr/bin/raw /dev/raw/raw30 /dev/sdb2


# /bin/chmod 600 /dev/raw/raw30
# /bin/chown oracle:dba /dev/raw/raw30

# /usr/bin/raw /dev/raw/raw31 /dev/sdb15


# /bin/chmod 600 /dev/raw/raw31
# /bin/chown oracle:dba /dev/raw/raw31

3. Create symbolic links to the raw partitions and set ownership to


oracle:dba:

# ln -s /dev/raw/raw30 /ora10g/app/oracle/oradata/V10/VotingDisk2
# /bin/chown oracle:dba /ora10g/app/oracle/oradata/V10/VotingDisk2
# ln -s /dev/raw/raw31 /ora10g/app/oracle/oradata/V10/VotingDisk3
# /bin/chown oracle:dba /ora10g/app/oracle/oradata/V10/VotingDisk3

4. To add the voting disk mirror, run 'crsctl add css votedisk' command as
root:

# crsctl add css votedisk /ora10g/app/oracle/oradata/V10/VotingDisk2

Cluster is not in a ready state for online disk addition

This fails with "Cluster is not in a ready state for online disk addition" error. The problem
is due to an Oracle bug where this operation does not work (as it should) with
clusterware up and running. 

To work around the problem perform the add operation with the -force option while the
clusterware is down:

    4.1. Shut down the crs stack on ALL nodes

# crsctl stop crs

Stopping resources.
Successfully stopped CRS resources
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.

    4.2. Add the voting disks

# crsctl add css votedisk /ora10g/app/oracle/oradata/V10/VotingDisk2 -
force

Now formatting voting disk: /ora10g/app/oracle/oradata/V10/VotingDisk2


successful addition of votedisk
/ora10g/app/oracle/oradata/V10/VotingDisk2.

# crsctl add css votedisk /ora10g/app/oracle/oradata/V10/VotingDisk3


-force
Now formatting voting disk: /ora10g/app/oracle/oradata/V10/VotingDisk3
successful addition of votedisk
/ora10g/app/oracle/oradata/V10/VotingDisk3.

    4.3. Restart the clusterware on all nodes for these changes to take effect:

# crsctl start crs

Attempting to start CRS stack


The CRS stack will be started shortly
...

5. Query the status of all voting disks:

# crsctl query css votedisk

0. 0 /ora10g/app/oracle/oradata/V10/VotingDisk2
1. 0 /ora10g/app/oracle/oradata/V10/VotingDisk1
2. 0 /ora10g/app/oracle/oradata/V10/VotingDisk3

located 3 votedisk(s).

Verify the permissions and ownership:


# cd /ora10g/app/oracle/oradata/V10

# ls -al
...
lrwxrwxrwx 1 oracle dba 14 Jul 25 14:58 OracleClusterRegistry-
>/dev/raw/raw28
lrwxrwxrwx 1 oracle dba 14 Jul 29 11:38 VotingDisk1 ->/dev/raw/raw29
lrwxrwxrwx 1 oracle dba 14 Jul 29 11:38 VotingDisk2 ->/dev/raw/raw30
lrwxrwxrwx 1 oracle dba 14 Jul 29 11:38 VotingDisk3 ->/dev/raw/raw31

Conclusion

Design of the CSS voting disk mirroring allows for large number of voting disk mirrors. 
In practice, to get the benefit of this feature without an impact on performance, three to
five voting disk mirrors should be implemented.  
Keep in mind that the number of voting disks available in a RAC cluster should be an
odd number.

References

NOTE:428681.1 - How to ADD/REMOVE/REPLACE/MOVE Oracle Cluster Registry


(OCR) and Voting Disk
Oracle Database Oracle Clusterware and Oracle Real Application Clusters
Administration and Deployment Guide 10g Release 2 (10.2)

 Related

Products

 Oracle Database Products > Oracle Database > Oracle Database > Oracle Server -
Enterprise Edition

Keywords

CLUSTERWARE; RAW

Back to top

You might also like