VVR cheat sheet

1) Creating a replicated volume on two hosts, hostA and hostB

Before configuring, you need to make sure two scripts have been run from
/etc/rc2.d: S94vxnm-host_infod and S94vxnm-vxnetd. VVR will not work if these
scripts don't get run AFTER VVR licenses have been instralled. So if you install
VVR licenses and don't reboot immediately after, run these scripts to get VVR to
work.
Before the Primary can be set up, the Secondary must be configured.
First, use vxassist to create your datavolumes. Make sure to specify the logtype
as DCM (Data Change Map, which keeps track of data changes if the Storage
Replicator log fills up) if your replicated volumes are asynchronous.
vxassist -g diskgroupB make sampleB 4g layout=log logtype=dcm
Then create the SRL (Storage Replicator Log) for the volume. Carefully decide
how big you want this to be, based on available bandwidth between your hosts and
how fast your writes happen.
See pages 18-25 of the SRVM Configuration Notes for detailed (excruciatingly)
notes on selecting your SRL size.
vxassist -g diskgroupB make srlB 500m
Next make the rlink object:
vxmake -g diskgroupB rlink rlinkB remote_host=hostA remote_dg=diskgroupA
remote_rlink=rlinkA local_host=hostB synchronous=[off|override|fail] srlprot=dcm
Use synchronous=off only if you can stand to lose some data. Otherwise, set
synchronize=override or synchronize=fail. override runs as synchronous (writes
aren't committed until they reach the secondary) until the link dies, then it
switches to asynchronous, storing pending writes to the secondary in the SRL.
When the link comes back, it resyncs the secondary and switches back to
Synchronous mode. synchronize=fail fails new updates to the primary in the case
of a downed link.
In any of the above cases, you'll lose data if the link fails and, before the
secondary can catch up to the primary, there is a failure of the primary data
volume. This is why it's important to have both redundant disks and redundant
network paths.
Now make the RVG, where you put together the datavolume, the SRL, and the rlink:
vxmake -g diskgroupB rvg rvgB rlink=rlinkB datavol=sampleB srl=srlB
primary=false
Attach the rlink to the rvg:
vxrlink -g diskgroupB att rlinkB
Start the RVG on the Secondary:
vxrvg -g diskgroupB start rvgB
Now work begins on the primary. As with the Secondary, make data volumes, an
SRL, and an rlink:
vxassist -g diskgroupA make sampleA 4g layout=log logtype=dcm
vxassist -g diskgroupA make srlA 500m
vxmake -g diskgroupA rlink rlinkA remote_host=hostB remote_dg=diskgroupB
remote_rlink=rlinkB local_host=hostA synchronous=[off|override|fail] srlprot=dcm
Make the RVG for the primary. Only the last option is different:
vxmake -g diskgroupA rvg rvgA rlink=rlinkA datavol=sampleA srl=srlA primary=true
Now go back to the secondary. When we created the secondary, brain-dead Veritas
figured the volume on the Seconday and the Primary would have the same name, but
when we set this up, we wanted to have the Primary datavolume named sampleA and
the Secondary datavolume be sampleB. So we need to tell the Secondary that the
Primary is sampleA:
vxedit -g diskgroupB set primary_datavol=sampleA sampleB
Now you can attach the rlink to the RVG and start the RVG. On the Primary:

vxrlink -g diskgroupA att rlinkA

You should see output like this:
vxvm:vxrlink: INFO: Secondary data volumes detected with rvg rvgB as parent:
vxvm:vxrlink: INFO: sampleB: len=8388608 primary_datavol=sampleA
Finally, start I/O on the Primary:
vxrvg -g diskgroupA start rvgA
2) Removing a VVR volume
First, detach the rlinks on the Primary and then the Secondary:
vxrlink -g diskgroupA det rlinkA
vxrlink -g diskgroupB det rlinkB
Then stop the RVG on the primary and then the secondary:
vxrvg -g diskgroupA stop rvgA
vxrvg -g diskgroupB stop rvgB
On the primary, stop the datavolumes:
vxvol -g disgroupA stop sampleA
If you want to keep the datavolumes, you need to disassociate them from the RVG:
vxvol -g diskgroupA dis sampleA
vxvol -g diskgroupB dis sampleB
Finally, on both the Primary and the Secondary, remove everything:
vxedit -rf rm rvgA
vxedit -rf rm rvgB
3) Growing/Shrinking a Volume or SRL
This is exactly the same as in regular Veritas. However, VVR doesn't sync the
volume changes. To grow a volume, you first need to grow the secondary, then the
primary. To shrink a volume, first the primary and then the secondary. You
always need to make sure the Secondary is larger than or as large as the
Primary, or you will get a configuration error from VVR.
You may need to grow an SRL if your pipe shrinks (more likely if your pipe gets
busier) or the amount of data you are sending increases. See pages 18-25 of the
SRVM Configuration Notes for detailed (excruciatingly) notes on selecting your
SRL size.
To grow an SRL, you must first stop the RVG and disassociate the SRL from the
RVG:
vxrvg stop rvgA
vxrlink det rlinkA
vxvol dis srlA
From this point, you can grow your SRL (which is now just an ordinary volume):
vxassist growto srlA 2gb
Once your SRL has been successfully grown, reassociate it with the RVG, reattach
the RLINK, and start the RVG:
vxvol aslog rvgA srlA
vxrlink -f att rlinkA
vxvg start rvgA
4) Getting info out of VVR once it's set up
You can get get useful stats out of the vxrlink command:
vxrlink [-i interval] stats rlinkA
Output should look similar to netstat. If you run without an interval, you can
see cumulative statistics. With an interval, you get the stats that hit during
the interval.
# vxrlink -i 5 rlinkA
Messages
Errors
Flow Control
-----------------------#
Blocks RT(usec) Timeout Stream Memory Delays
NW Bytes NW Delay
28
0
512
0
0
0
0
5000
1
0
0
512
0
0
0
0
5000
1

0
0
512
0
0
0
0
5000
1
0
0
512
0
0
0
0
5000
1
0
0
512
0
0
0
0
5000
1
0
0
512
0
0
0
0
5000
1
0
0
512
0
0
0
0
5000
1
10
0
2110
0
0
0
0
5000
1
256
0
22766 0
0
0
0
5000
1
468
0
15417 18
0
0
0
5000
4
18
0
7818
0
0
0
0
5000
4
0
0
7818
0
0
0
0
5000
4
0
0
7818
0
0
0
0
5000
4
# is the number of messages transmitted
Blocks is the number of 512-byte blocks transmitted
RT(usec) is the average round-trip per message (the size of the messages
affects the RT) in microseconds
Timeout is the number of timeouts or lost packets; this can be affected by the
time-out value of the RLINK.
Stream is the number of stream errors that occur when the RLINK attemps to
send messages faster than the network can handle.
Memory is the number of memory errors that occur when the secondary has
insufficient buffer space to handle incoming messages. You can tune this by
changing voliomem_max_nmcompool_sz on the secondary
Delays, NW Bytes, and NW Delay are internal flow control parameters that
indicate how fast the RLINK is attempting to send.
The sample output is from a Secondary system, so there are no Blocks
Transmitted.
You can use vxrlink to check the status of an rlink and SRL:
vxrlink -g diskgroupA status rlinkA
Output should look like this:
Rlink rlinkA has 8 outstanding writes, occupying 520 Kbytes (1%) on the SRL
If there are no outstanding writes and the SRL has been fully played, you will
see this message:
Rlink rlinkA is up to date
You can only use this command on the primary. Trying it on the secondary will
just result in Veritas yelling at you.
vxprint can also give you some useful info, and it comes with some shortcuts.
vxprint -Pl
vxprint -Vl
The first command lists all the RLINKs on the system. The second lists all the
RVGs.
You can use this info to check VVR settings and some basic status. Check the
flags for the RLINK to make sure that both systems are connected and consistent,
the IPs and ports are set right, etc.
In the output for an RVG, you can see listed in the flags line if an RVG is set
as the Primary of Secondary node.
5) Changing the Synchronous/Asynchronous setting
to set the synchronous variable for an RLINK, do the following:
vxedit set synchronous=[off|override|fail] rlinkA
6) Failing Over from a Primary
There are two situations where you would have to fail from a primary. The first
is in preparation for an outage of the Primary, in which case you can happily
turn off your app, switch the Primary to a Secondary, switch the Secondary to a
Primary, and start this up again.
The second case is when your Primary goes down in flames and you need to get
your Secondary up as a Primary.
If your primary is still functioning:
First, you'll need to turn off your applications, umount any filesystems on from

your datavolumes, and stop the rvg:

/etc/rc3.d/S99start-app stop
umount /filesysA
vxrvg stop rvgA
If you can't umount your filesystems because of running apps, DON'T go any
further! You'll make your life harder in the future, and you might lose data.
Once you've stopped the RVG, you need to detach the rlink, disassociate the SRL
volume (you can't edit the PRIMARY RVG attribute while an SRL is associated),
change PRIMARY to false, and bring everything back up:
vxrlink det rlinkA
vxvol dis srlA
vxedit set primary=false rvgA
vxvol aslog rvgA srlA
vxrvg start rvgA
vxrlink -f att rlinkA
Now go to work on the Old Secondary to bring it up as the new Primary.
First you need to stop the RVG, detach the rlink, disassociate the SRL, and turn
the PRIMARY attribute on:
vxrvg stop rvgB
vxrlink det rlinkB
vxvol dis srlB
vxedit set primary=true rvgB
Veritas recommends that you use vxedit to reinitialize some values on the RLINK
to make sure you're still cool:
vxedit set remote_host=hostA local_host=hostB remote_dg=diskgroupA
remote_rlink=rlinkA rlinkB
Before you can attach the rlink, you need to change the PRIMARY_DATAVOL
attribute on both hosts to point the the Veritas volume name of the NEW Primary:
On the new primary (e.g. hostB):
vxedit set primary_datavol=sampleB sampleB
On the new secondary (e.g. hostA):
vxedit set primary_datavol=sampleB sampleA
Now that you have that, go back to the new Primary, attach the RLINK, and start
the RVG:
vxrlink -f att rlinkB
vxrvg start rvgB
The vxrlink command should show normal output as described in section 1 above.
If the Primary is down:
First you'll need to bring up the secondary as a primary. If your secondary
datavolume is inconsistent (this is only likely if an SRL overflow occurred and
the secondary was not resynchronized before the Primary went down) you will need
to disassociate the volumes from the RVG, fsck them if they contain filesystems,
and reassociate them with VVR. If your volumes are consistent, the task is much
easier:
On the secondary, first stop the RVG, detach the RLINK, and disassociate the
SRL:
vxrvg stop rvgB
vxrlink det rlinkB
vxvol dis srlB
Make the Secondary the new Primary:
vxedit -g diskgroupB set primary=true rvgB
Now reassociate the SRL and change the primary_datavol:
vxvol aslog rvgB srlB
vxedit set primary_datavol=sampleB sampleB
If the old Primary is still down, all you need to do is start the RVG to be able
to use the datavolumes:

vxrvg start rvgB

This will allow you to keep the volumes in VVR so that once you manage to
resurrect the former Primary, you can make the necessary VVR commands to set it
up as a secondary so it can resynchronize from the backup system. Once it has
resynchronized, you can use the process listed at the beginning of section 6
(above) to fail from the Old Secondary/New Primary back to the original
configuration.
Here's now to resynchronize the old Primary once you bring it back up:
The RVG and RLINK should be stopped and detached. If not, stop and detach:
vxrvg stop rvgA
vxrlink det rlinkA
Disassociate the SRL and make the system a secondary:
vxvol dis srlA
vxedit set primary=false rvgA
Reassociate the SRL, change the primary_datavol attribute:
vxvol aslog rvgA srlA
vxedit set primary_datavol=sampleB sampleA
Attach the RLINK and then start the RVG:
vxrlink -f att rlinkA
vxrvg start rvgA
This won't do much, as the RLINK on hostB (the Primary) should still be
detached, preventing the Secondary from connecting. Now go back to the Primary
to turn the RLINK on:
vxedit set remote_host=hostA local_host=hostB remote_dg=diskgroupA
remote_rlink=rlinkA
vxrlink -a att rlinkB
Giving the -a flag to vxrlink tells it to run in autosync mode. This will
automatically resync the secondary datavolumes from the Primary. If the Primary
is being updated faster than the Secondary can be synced, the Secondary will
never become synced, so this method is only appropriate for certain
implementations.
Once synchronization is complete, follow the instructions above (the beginning
of section 6) to transfer the Primary role back to the original Primary system.