Db2 Sap Splitmirror

Storage Management for SAP and DB2 UDB: Split Mirror Backup / Recovery
with IBMs Enterprise Storage Server (ESS)
Storage Management for SAP and DB2 UDB:

Split Mirror Backup / Recovery With
IBMs Enterprise Storage Server (ESS)
Sanjoy Das, Siegfried Schmidt, Jens Claussen and BalaSanni Godavari
Page 1
The following terms are trademarks of International Business Machines Corporation in

the United States, other countries, or both:
AIX
DB2
DB2 Universal Database
Enterprise Storage Server (ESS)
ESCON
FlashCopy
OS/390
Tivoli
TME 10
The following terms are trademarks of SAP AG in Germany, in the United States, other
countries, or both:
SAP
SAP Logo
mySAP.com
R/3
ABAP
SSQJ
Advanced Technology Group
OSS
SAP R/3 Note
Java and all Java-based trademarks and logos are trademarks or registered
trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.
Legato NetWorker is a trademark of Legato Systems, Inc. in the United States, Other
countries, or both.
Windows is a trademark of Microsoft Corporation in the United States, Other countries,
or both.
Veritas NetBackup is a trademark of Veritas Software in the United States, Other
countries, or both.
Other company, product, and service names may be trademarks or service marks of
others.
Page 2
The information provided in this document is distributed "AS IS" basis without any
warranty either express or implied. IBM AND SAP DISCLAIM ALL EXPRESS AND
IMPLIED WARRANTIES WITH RESPECT TO SUCH INFORMATION, INCLUDING ANY
WARRANTIES OF MERCHANTIBILITY OR FITNESS FOR A PARTICULAR
PURPOSE. The use of this information or the implementation of any of these
techniques is a customer responsibility and depends on the customer's ability to
evaluate and integrate them into their operating environment.
While the information contained in this paper has been reviewed by IBM and SAP for
accuracy, there is no guarantee that the same or similar results will be obtained
elsewhere. Customers attempting to adapt these techniques to their own environments
do so at their own risk. The performance data contained herein was obtained in a
controlled environment based on the use of specific data. Actual results that may be
obtained in other operating environments may vary significantly. These values do not
constitute a guarantee of performance.
References in this document to IBM and SAP products, programs, or services do not
imply that IBM or SAP intend to make such products available in all countries in which
each company operates. Neither IBM nor SAP warrants the others products. Each
companys products are warranted in accordance with the agreements under which they
are provided.
Page 3
Abstract.......................................................................................................................................................... 5
Preamble ........................................................................................................................................................ 6
1
Introduction .......................................................................................................................................... 6
Customer Requirements...................................................................................................................... 7
SAP Requirements ............................................................................................................................... 8
DB2 UDB and ESS Features to Support the Split Mirror Backup / Recovery Solution................. 10
4A DB2 UDB Features ..................................................................................................................... 11
4B FlashCopy - ESS's Advanced Local Copy Function ............................................................... 12
4C Peer-to-Peer-Remote Copy (PPRC) ESSs Advanced Remote Copy Function .................. 13
Split Mirror Backup & Recovery (SMBR) Setup ............................................................................... 16

5A SSQJ: R/3 Load Simulation and Testing Tool ......................................................................... 17
5B SAP/DB2 UDB File System Definition....................................................................................... 18
5C Split Mirror Infrastructure Setup............................................................................................... 21
5D ESS LUN Definition .................................................................................................................... 22
5E Volume Group Assignment... 24
Split Mirror Backup & Recovery Process ......................................................................................... 26

6A Start Situation............................................................................................................................. 26
6B Split Mirror Backup & Recovery (SMBR) Scenarios................................................................ 34
6C Key Observations about the Split Mirror Backup & Recovery Scenarios ............................. 37
Recovery.............................................................................................................................................. 38
Process Automation - Solution Integration in Customer Environments ....................................... 42
Acknowledgements..................................................................................................................................... 43
Appendix ...................................................................................................................................................... 44
A) DB2 UDB Architectural Overview ............................................................................................. 44
B) DB2 UDB Database Growth and Impact on Storage ............................................................... 49
C) DB2 UDB Memory Management................................................................................................ 51
D) Backup and Recovery Management for DB2 UDB .................................................................. 52
E) Work Process Management in SAP R/3 with DB2 UDB and Storage..................................... 55
F) SAP R/3 data layout.................................................................................................................... 57
G) ESS Raid5 Disk Layout Considerations for SAP R/3 Environments ...................................... 59
H) Considerations for DB2 UDB database layout ......................................................................... 63
I)
Sub System Device Driver (SDD) .............................................................................................. 66
J) Volume Layout for DB2 UDB ..................................................................................................... 69
K) The Role of the ESS Specialist .................................................................................................. 71
References ................................................................................................................................................... 72
Page 4
Abstract
Recent new products such as SAPs B2B Procurement, CRM, mySAP.com Portals and
Trading Exchange markets require an ever increasing need for continuous system
availability.
This paper provides information on an essential component of advanced infrastructure
solutions the High Availability Split Mirror Backup/Recovery (SMBR) for Systems of a
mySAP landscape on IBMs DB2 UDB operating in AIX environment. In the following R/3
will be used as a synonym for SAP products.
The solution described in this paper is intended to deliver a backup process with no
impact on the live R/3 system (server less) using the advanced functions of IBMs
Enterprise Storage Server (ESS). This zero downtime for the live R/3 system means
that SAP users typically will not see a disruption of activity while the backup of the live
database takes place. No transactions typically are cancelled during the copy
process/backup process.
Instant availability of consistent copies of the database provides the ability to place an
emergency system at the users disposal while recovering the live database from a
disaster. Beyond Backup / Recovery, a consistent copy of the database may be used for
various purposes, such as creation of a reporting instance, production-fix system or
repository for building Business Warehouse (BW) system.
Page 5
Preamble
This white paper, written from the DBA's perspective, addresses the infrastructure
design, implementation tasks and techniques required for complex Enterprise
Application Integration landscapes for high availability SAP R/3 applications consisting of
a database (DB2 UDB), an operating system (AIX) and a storage subsystem (ESS)
which all interoperate to deliver an easy-to-manage backup & recovery solution for SAP
customers. Backup & Recovery solutions are mission critical activities in todays world of
7 x 24 computing and are a major focus for IT personnel, application management and
DBAs. With the exploding growth in storage requirements for SAP application
environments, this work touches on major elements of each area of technology spanning
critical operating requirements and how this storage-centric solution delivers compelling
value for SAP customers.
Introduction
Service level agreements increasingly have to reflect that in case of planned downtime
such as backup, hardware / software maintenance, R/3 upgrade and unplanned
downtime such as database Data Manipulation Language (DML) error, error analysis
and restores, the system has to be available within minutes.
SAPs Advanced Technology Group has developed scenarios using live databases that
constantly copy or mirror using storage subsystems, allowing business continuation
during the split (and resynchronization) of the mirror. Once the logical database mirrors
are established, additional copies are created for backup and for use by a standby SAP
R/3 System. This solution minimizes the I/O load impact of the live environment and
offloads the backup activity away from the live database server/storage server to a
standby/backup server host. The Split Mirror solution, based on this concept, was
successfully implemented using DB2 UDB database on the AIX platform using the
Enterprise Storage Server. This solution was implemented using SAP R/3 4.6B with DB2
UDB 7.1 on AIX 4.3.3. It can be implemented for either a single or a dual ESS
configuration using the ESSs advanced functions the local copy function FlashCopyTM
Page 6
and the remote synchronous copy function Peer-to-Peer-Remote-Copy (PPRC). The

core R/3 system was loaded using SAP-developed SSQJ tool for OLTP volumes.
This solution in dual ESS configuration is intended to enable customers to implement
remote data vaulting and/or to scale to a larger database size.
A solution similar to the one described in the following pages was implemented on IBMs
Enterprise Storage Server (ESS) with SAP R/3 on DB2 OS390 and first demonstrated in
November 1999 (see reference [9.,10.] for details). The Split Mirror Backup on the
DB2OS390 for both RVA and ESS storage systems used a special write suspend
function created by DB2OS390 especially for executing the Split Mirror Backup/
Recovery solution (SMBR).
This Open Systems solution for SAP R/3 with DB2 UDB on AIX takes advantage of a
similar functional capability created by IBM especially for this DB2 UDB Split Mirror
Backup /Recovery solution.
Customer Requirements
Increasingly, with the rapid trend towards very large databases, accompanied by the
need for high availability in a global computing environment, customers now demand
that production systems be available on a 7 x 24 basis. This also means that in case of a
disaster, the system has to be available within minutes or hardly longer than the time
needed for the physical reload of the database from a secondary or remote storage
media. This high availability requirement also implies that backup and the creation of an
emergency system may not cause any downtime of the live production system and all
procedures to achieve this must be seamlessly automated.
Customers are also very aware of the fact that software or application logical errors and
not hardware failures are the most likely causes for the need for disaster recovery
capabilities. But, for mission critical applications, customers will do everything possible to
optimally protect themselves from a hardware disaster. This means that cost effective,
Page 7
intelligent fault tolerant storage subsystems are an important ingredient of high

availability advanced infrastructure solutions, helping to insure against cost-prohibitive
downtime possibilities.
Along with the demanding requirements of non-stop, web-centric computing, customers
now have to contend with business processes spread over multi-vendor platforms,
databases and file systems, where automated interaction between systems provide the
essence of competitive productivity. In these emerging environments, they need and
demand database availability with current data, fast backup/restore/recovery with no
impact on the main production (OLTP) system, all enabled with seamless automation
through well defined, user friendly management interfaces. With the emerging world of
SAP technologies, customer environments have to meet stringent requirements for
availability, scalability and flexibility that can handle changing customer requirements
based upon SAP instance strategies, data migration requirements and disparate growth
rates of databases.
SAP Requirements
In order to deliver a solution that matches the complex requirements for high availability
backup, recovery and performance, the SAP administrative workload should be taken off
the live production system and performed on a copy of the system. The recommended
environment is depicted in Figure 1.
Page 8
A d m in is tr a tio n
D B C heck
U p d a te S ta tis tic
R /3 R e p o rtin g S y s te m
re m o te c o p y (s p lit)
2 n d m ir ro r lo c a l c o p y (s p lit) 3 rd m irro r
M irro re d D a ta
P ro d u c tio n D a ta
Figure 1: SAP High Availability Advanced Infrastructure

Backup & Recovery Conceptual Model
While a production server is connected to a primary fault tolerant storage controller, a
mirrored remote copy of the production database is created without the computing
support of the production server, in a separate/secondary fault tolerant storage controller
located at a spatially separated site, providing high availability and disaster recovery
capability for business continuance. This copy can be accessed by a standby/backup
server should IT management procedures require its intervention in the event the
primary site experiences business interruption. A local copy function within the
secondary storage controller produces a consistent point-in-time copy of the production
database. This consistent copy, at the option of the user, can be either held inside
secondary storage controller or be transferred to tape for remote vaulting. In the event
that the primary database or its mirror is not available, the production server or the
backup server can be connected to this disk resident copy, helping to dramatically
reducing downtime.
Page 9
The database and SAP Basis administration can be augmented by providing a

homogeneous split mirror-based system copy for the purposes of test upgrades, support
packages and database recovery routines. Intelligent storage controllers need to deliver
this capability to create near-instant, consistent copies of the database.
Thus a number of key benefits can accrue to users from this recommended
environment:
-
Backup & restore time is database size independent
Minimum impact on production environment
Physical and logical disaster prevention
Enhanced database administration
Remote data vaulting
Offloading backup from production database server
Reduction of backup-restore-time to minutes
In order to realize this concept, the database management system needs to cooperate
with storage subsystems to deliver the results. It should support the creation of a
consistent database copy during the application READ/WRITE processing in a manner
that exerts negligible impact on the production system, database or the live production
storage server.
4 DB2 UDB and ESS Features to Support the Split Mirror

Backup / Recovery Solution
This section describes the key DB2 UDB features required to create a consistent backup
database image of the production database using a set of specially developed DB2 UDB
commands. The backup image includes SAP R/3 objects and File definitions for DB2
UDB. This backup process utilizes two advanced functions of the ESS storage sub
system such as Peer-to-Peer Remote Copy (PPRC) and FlashCopy.
Page 10
4A
DB2 UDB Features
For very large DB2 UDB databases in SAP R/3 production environments, the Split Mirror
backup capability is essential for the creation of consistent database backup copies
without stopping the production system. In order to make this possible, DB2 UDB
created a set of special commands that enable temporary suspension of writes to the
database:
DB2 SET WRITE SUSPEND FOR DATABASE: This command suspends the I/O
writes and puts the tablespaces into a new SUSPEND_WRITE state. Writes to the
logs are also suspended by this command. However, read-only transactions are
allowed to continue. Any changes to tablespace information needs to wait till the
writes are subsequently resumed. Once the SUSPEND_WRITE state is in effect
for the database, all files and directories within the database are protected from
updates.
DB2 SET WRITE RESUME FOR DATABASE: This command resumes I/O writing
and removes the SUSPEND_WRITE state and makes the tablespaces available
for updates.
For an overview of the DB2 UDB architecture and other features that take advantage of
ESS storage management capabilities, please refer to Appendix A (DB2 UDB
Architectural Overview), Appendix B (DB2 UDB Database Growth and Impact on
Storage) and Appendix C (DB2 UDB Memory Management).
Page 11
4B
FlashCopy - ESS's Advanced Local Copy Function
The ESS administration tool (ESS Specialist) identifies the Logical Unit Numbers (LUNs)
by their ESS internal serial numbers. FlashCopy, ESSs near-instant local copy
function, can be used for all systems that have volumes or LUNs within the same Logical
Subsystem (LSS) of an ESS. FlashCopy is set up using the Web interface of the
StorWatch ESS Specialist. Then, task selections on the volume pair FlashCopy with
Full or No Copy, and WITHDRAW options can be made. See reference [12] for set up
details on FlashCopy.
The NO COPY option in FlashCopy is useful if the copy operation has to complete with
in a short time so that the source database/application are returned to their normal
usage from a quiesced state. WITHDRAW command enables the removal of source and
target relationships from a previously specified NOCOPY command. The relationship
between source and target will automatically end when the physical copy is completed.
Using the ESS Specialist, FlashCopy tasks - with either No Copy or Physical Copy
options are created. These tasks are initiated from the Unix command line using
Command Line Interface (CLI) rsExecuteTask. The status of those tasks can be queried
using the CLI, rsExecuteQuery. Based on the results of this query, the SMBR
automation process can capture the error code for each of the FlashCopy tasks.
FlashCopy, when set up by the StorWatch ESS Specialist, creates an identical copy of
the source volume onto target volumes when an appropriate task is initiated using CLI.
Volume identification or disk signatures need to be validated with respect to the host that
is connected to the ESS in order for that host to start using the target ESS FlashCopy
volumes. In order for a single host to mount both source and target volumes of
FlashCopy pairs, AIX provides the RECREATEVG command, which is packaged as a
PTF for AIX 4.3.3 in APAR IY10456. It is officially available in:
1. AIX 4.3.3 Recommended Maintenance Level 05 (RML05)
2. AIX 4.3.3 RML06
Page 12
The RECREATEVG command overcomes the problem of duplicated LVM data

structures and identifiers caused by a disk copying process such as FlashCopy. It is
used to recreate an AIX Volume Group (VG) on a set of disks that are copied. Then, the
normal commands to bring the volume online, i.e. mounting can be attempted by the
host operating system.
4C
Peer-to-Peer-Remote Copy (PPRC) ESSs Advanced

Remote Copy Function
PPRC is a hardware-assisted synchronous remote copy or synchronous mirroring

function that can help preserve data integrity. Synchronous mirroring means that an I/O
is only completed until after it is acknowledged from the remote site.
PPRC is set up on a volume or LUN basis in two or more ESS's, which are connected by
ESCON as in Figure 2. Updates to a PPRC volume on the local or primary site (primary
volume) go first into cache and non-volatile storage (NVS) in the primary storage. The
updates are then sent over the ESCON links via larger ESCON frame transmission to
the remote or secondary storage control. When the data is in cache and NVS on the
secondary site, the receipt of the data is acknowledged and the primary storage control
signals to the application the completion of the I/O by a Device End status.
Page 13
Figure 2: PPRC Connectivity using ESCON Links

The enhancements to ESCON protocol, implemented in ESS micro code as an
advanced copy function, allow an extended distance between two ESS's of up to 103
km, when using multi-mode to mono-mode ESCON converters, amplifiers and switches.
PPRC can be implemented over longer distances using channel extenders from third
party suppliers certified on the ESS.
Up to eight ESCON links are supported between two ESS storage subsystems. The
local primary storage control with PPRC source volumes and the remote secondary
storage control are connected via ESCON links. One ESS storage control can act as
primary and secondary at the same time if it has PPRC source and target volumes.
PPRC links are uni-directional, as shown in Figure 3, so that a physical ESCON link can
be used to transmit data from the primary storage control to the secondary.
Page 14
Figure 3: PPRC Links between two ESS Storage Sub Systems
Before PPRC pairs can be established, logical paths must be defined between the
logical control unit images also called as Logical Sub Systems (LSS). The ESS supports
up to 16 logical Fixed Block (FB) control unit images and up to 32 SCSI/Fiber controller
images. Logical paths are established between LSS of the same type over physical
ESCON links that connect suspending and resuming pairs.
During the suspension of a pair, the primary control unit maintains a bitmap in NVS (Non
Volatile Storage) with a flag bit for each track that was changed on the primary volume.
This allows for a later resynchronization (RESYNCH) of the volume pair while allowing
only cylinders flagged in the bitmap table to be copied to the remote site.
Page 15
As with FlashCopy tasks, PPRC tasks are created via the ESS Specialist and then
invoked from the Unix command line using CLI interface. Based on the result of the
query - rsExecuteQuery, the SMBR automation process can capture the error code for
each of the PPRC tasks.
5 Split Mirror Backup & Recovery (SMBR) Setup

The SMBR set up involves the physical database design from the SAP installations
guide. This includes file systems definitions according to sizing (based upon planned
usage statistics, I/O forecasts, numbers of users etc.). The file systems requirements
follow standard SAP installation procedures.
In a full-featured high availability SMBR solution we recommend to use two physically
separated database hosts and two ESS, each containing two copies of the production
database. In our test environment we used two ESS clusters (refer to figure 6, 7) like
separated storage systems, but without reservations this can simply be extended to a
two ESS implementation.
The installation binaries for SAP kernel and DB2 along with the SSQJ (a load testing tool
developed by SAP) file systems are also setup for each of the four copies. SSQJ was
developed with ABAP4 in R/3 for benchmarking based on SQL/ABAP statements and
table manipulation for performance / throughput analysis.
The ESS LUNs design is based on logical addressing of striped physical volumes in a
RAID5 array. The LUN definitions are based on file system requirements. And are
translated into volume groups via IBMs Subsystems Device Driver (SDD) mapping of
virtual paths installed on the AIX host.
Page 16
5A
SSQJ: R/3 Load Simulation and Testing Tool
This Split Mirror Backup / Recovery solution utilized SAP designed SSQJ tool for load
simulation. This tool is a generic test measurement tool that was developed with ABAP/4
in core R/3 and it enables testing and measurement of the following functions in the R/3
Basis system such as runtimes of specific functions and their alternatives, resource
consumption, query plans in the case of DB statements and other functions. SSQJ runs
on all SAP-supported database systems with built in infrastructure for Measurement &
Analysis including environment History of measurements.
A number of test suites are included in the current version of the package:
1) SQL statements
2) ABAP statements
3) Large Tables for performance and throughput analysis
4) Archiving Systems
5) Data Conversion
6) TPC/D-Benchmark and
7) Business Warehouse
SSQJ is currently used for SAP Basis Benchmarking, by SAP Database Porting teams,
SAP Database Partners and other SAP Hardware Partners.
The SSQJ tool was used as the kernel database for the DB2 UDB SMBR testing.
SSQJ requires the creation of two new tablespaces, PSAPSSQJD for data and
PSAPSSQJI for indexes. The size of the SSQJ tablespaces depends on the required
database size. For the initial installation, we require at least 8GB for PSAPSSQJD and
4GB for PSAPSSQJI. The data files for PSAPSSQJD are in /db2/UDB/sapdata3 (total of
8 files of 2GB each), as shown in Figure 5, and the data files for PSAPSSQJI are in
/db2/UDB/sapdata4 (total of 4 files of 2GB each).
Page 17
5B
SAP / DB2 UDB File System Definition
SAP R/3 Backup Objects and File Definitions for DB2 UDB
All objects in the R/3 environment need to be backed up. These objects are:
1. R/3 data objects:
a. R/3 archiving objects
b. R/3 Interfaces
c. SAP Executables
2. Computing center data such as:
a. The Operating System
b. Third Party Programs connected to R/3
3. Database objects
DB2 UDB supports both Online and Offline database backups. Given a properly
implemented database backup cycle, both forms of backup attain the same end goals.
The specific circumstances at each SAP R/3 installation determine which of the two
methods is appropriate. The SAP tools for database backup support both options. Figure
4 depicts the SAP Backup Objects and Figure 5 depicts the UDB File Systems for SAP.
DB2 UDB Backup Objects for SAP

Directory
Meaning
/$INSTDIR/db2_07_XX/<sqllib>
DB2 software (specified during DB2

installation)
/db2/<SAPSID>
DB2 instance directory
/db2/<SAPSID>/sqllib/bin,tsm,doc,..
DB2 binaries, TSM control files,

documentation
/db2/<SAPSID?/sapdata1..N
DB2 directories for containers
/db2/<SAPSID>/log_dir
DB2 online log directory
/db2/<SAPSID>/log_archive
DB2 offline log directory
Figure 4: UDB Backup Objects for SAP

Page 18
DB2 UDB File System Definitions for SAP

Volume
File system Name
Function
Group
Size (in 8MB

Physical
Partitions)
Sapvg
/usr/sap/UDB
Link to /sapmnt/UDB
/sapmnt/UDB
SAP Executables
60
/usr/sap/trans
Transport directory
60
/db2/UDB
DB2 Executables
130
/db2/db2as
Logvg
236
/db2/UDB/log_dir
Online Log files
50
/db2/UDB/sapdatat
Temporary Tablespace
50
/db2/UDB/log_archive
Archive log files
470
/db2/UDB/sapdata1
SAP R/3 data files
3810
/db2/UDB/sapdata2
SAP R/3 data files
3810
/db2/UDB/sapdata3
SAP R/3 data files
3810
/db2/UDB/sapdata4
SAP R/3 data files
3810
/db2/UDB/sapdata5
SAP R/3 data files
3810
/db2/UDB/sapdata6
SAP R/3 data files
3810
LSS16data
Not Assigned
For future data files
Backupvg
/basebackup
SAP backups
2800
/usr/sap/trans/data
SSQJ data load
700
LSS10data
LSS12data
LSS14data
Figure 5: UDB File Systems for SAP
In Figure 5 above, the volume groups LSS10data, LSS12data, LSS14data and

LSS16data are 64GB each. The other volume groups are sapvg and logvg comprising of
1GB LUNs. sapvg is used for the SAP and DB2 UDB executables, and logvg holds the
online and archive logs.
Page 19
SAP delivers binary executables in order to help achieve on-line and offline backups as
a part of the kernel installation. The external data management tools like Tivoli Storage
Manager (TSM) have products that integrate into SAP R/3 as referred in Appendix D. As
show in Figure 16, the SAP R/3 objects DB2 control files, DB2 containers, offline log
files and online log files are required for a consistent database backup and restore /
recovery.
In order to achieve SAP requirements for SMBR, the ESSs advanced copy functions,
namely FlashCopy and Peer-to-Peer-Remote-Copy are incorporated into the solution.
For the initial SAP installation, the file systems are created as shown in Figure 5.
As shown in Section F (SAP R/3 Data layout for UDB) of the Appendix, consideration
must be given to sizing recommendations, SAP landscape, and data layout spread
evenly across storage systems. ESS RAID5 disk layout considerations, as detailed in
Section G (ESS RAID Disk Layout Considerations for SAP R/3 Environments) of the
Appendix, ensure that the "hot spot phenomenon common in non-RAID5 environments
is eliminated by striping all DB2 UDB tablespaces across all devices, thus utilizing the
cumulative throughput of all device adapters. Additional considerations for DB2 UDB
database layout on ESS should incorporate optimal values for PREFETCHSIZE,
DB2_PARALLEL_IO,
DB2_STRIPED_CONTAINERS,
tablespace
EXTENTSIZE,
database page size as outlined in Section H of the Appendix.

In order to complete the file systems layout, first allocate the logical volumes by
choosing the appropriate AIX options. Then create the file system on top of these logical
volumes.
At this point we complete the installation of DB2 UDB 7.1 with fix pack 2 for our AIX host
joyjoy and complete the installation of SAP Version 4.6B. The SAP and DB2 UDB
system/instance names are UDB.
Page 20
5C
Split Mirror Infrastructure Setup
As shown in Figure 6, our live database consists of DBA for the SAP Instance UDB on
the primary ESS attached to host joyjoy. It contains all the DB2 binaries, SAP kernel,
transport directory, DB2 online / archive log directories, DB2 instance directory, DB2
data container file systems and SSQJ file systems.
DBC is the PPRC mirror of the live database DBA. DBB is the Safety FlashCopy of the live
database DBA. This safety copy is created in order to maintain point in time copy of the
live production database should there be any problems encountered with DBA during the
SMBR operation.
Figure 6: R/3 Split Mirror Backup & Recovery and Backup Host
Page 21
In situations where the customers Basis or DBA group would like to mount the file
systems for verification purposes by using the FlashCopy or PPRC target volumthe
In situations where the customers Basis or DBA group would like to mount the file
systems for verification purposes by using the FlashCopy or PPRC target volumes on
the same host, the Physical Volume Identifier (PVIDs) need to be renamed using the AIX
RECREATEVG command mentioned in Section 4B.
5D
ESS LUN Definition
For all references to LUN and volume group sizes, 1GB translates to 1*1000*1000*1000
bytes. All the LUNS for host joyjoy are defined only on cluster 1 for our SMBR solution
scenarios using FlashCopy and PPRC. Each of the 8 ranks on ESS1 (Cluster 1) consists
of twelve 8GB LUNs. In addition to these, two 1GB LUNs are also defined on each of the
ranks. The 8GB LUNs are used for the DB2 UDB data files, while one set of eight 1GB
LUNs is used for db2 log_dir and log archive files. The other set of eight 1GB LUNs is
used for the SAP R/3 and DB2 UDB executables and other temporary storage.
OSPL2 and OSPL6 are the production and disaster recovery ESS's in our setup.
For the initial installation, four 8GB LUNs from each rank are used to layout the SAP R/3
DB2 UDB data files. This leaves the other eight 8GB LUNs for FlashCopy (safety copy)
and PPRC.
Page 22
Figure 7: ESS Logical Sub Systems (LSS)

As shown in Figure 7, there are a total of eight LSS's, four on each cluster of an ESS.
Each LSS on a cluster has two ranks assigned to it. There are two sets of four 8GB
LUNs of an LSS that are assigned to one volume group.
In our case, that translates to four volume groups of 64GB each, one for each LSS. The
volume groups are LSS10data, LSS12data, LSS14data and LSS16data on ESS1
(OSPL2). Similarly, LSS11data, LSS13data, LSS15data and LSS17data are created on
ESS2 (OSPL6).
The other volume groups are sapvg and logvg comprising of 1GB LUNs. sapvg is used
for the SAP and DB2 UDB binaries, and logvg holds the online and archive log files.
Page 23
In an optimal database layout where the distribution of tablespaces follows the basic
principle spread the data over as many physical disk/arrays as possible (refer to [4])
tablespaces should be extended by at least one full stripe set across all arrays.
This ensures that tablespaces remain distributed across a stripe set. Allocation of a
specific number of arrays for a tablespace facilitates distribution and placement of the
data files. The more arrays that are in the allocation of arrays, the better the distribution.
Data file distribution should be based on a round-robin placement across ESS clusters
and arrays and the granularity can be achieved using AIX Logical Volume allocation.
In the Split Mirror Backup and Recovery Solution validation scenarios, the AIX Logical
Volume Manager maps the assigned ESS 8GB LUNs as hdisks. Grouping the LUNS of
a single RAID array can create volume groups. Under this traditional route for database
layouts, AIX file systems are created over AIX logical volumes that reside on a single
array. The data files of a tablespace are then distributed over file systems on different
arrays. The criterion for a data file to be placed in these file systems is still the same the overall I/O activity should be distributed across all available arrays. Creation of
volume groups and logical volumes should be within the constraints imposed by the AIX
Logical Storage Management.
5E
Volume Group Assignment
Once the LUNS are defined on each ESS and assigned to hosts "joyjoy" and "decme",
the LUNS are made available to "joyjoy" by running the cfgmgr command on that host.
The Subsystems Device Driver (SDD) is a high availability automatic I/O Path Failover
Function that provides management of active paths to the LUNs as outlined in Section I
of the Appendix. The SDD software needs to be installed on the hosts before running
cfgmgr.
The LUNs appear as hdisks on joyjoy. If a LUN is assigned to one path, an hdisk, lets
say hdisk5 is defined on joyjoy. As joyjoy has 4 SCSI adapters, the LUNs are configured
Page 24
such that they can be accessed through all four paths. For each additional path, another
hdisk is assigned. So for 4 paths, we have four hdisks all pointing to the same LUN on
the ESS.
When SDD is used, additional data path devices called vpath devices are created. Each
hdisk set (a set is based on the number of paths from the host to the LUN) is assigned a
vpath device. In our example, vpath0 will consist of hdisk5, hdisk45, hdisk89 and
hdisk125.
When a volume group is to be created on joyjoy, two options are available (in smitty)
(1) Add a Volume Group, or
(2) Add a Volume Group with Data Path Devices.
As we are using SDD, the volume groups are created with the second option.
Volume Layout for DB2 UDB

The Volume layout for SAP R/3 system in DB2 UDB for ESS is depicted in Figures 25A
and 25B (Logical Sub System layout of vpath mapping to hdisks) in Section J of the
Appendix.
Page 25
Split Mirror Backup & Recovery Process
6A
Start Situation
The live system is in normal READ/WRITE operation state. The Log Volumes (log_dir,
log_archive) are in a constant synchronous PPRC connection between source DBA and
DBC across ESS1 and ESS2 along with Data Container Volumes as shown Figure 8.
After the prior SMBR backup activity, the DBD instance is in FlashCopy No Copy
relationship with DBC while DBB is in FlashCopy No Copy relationship with DBA.
Figure 8: Split Mirror Backup / Recovery and Standby R/3 System

The nine SMBR process steps depicted in Figure 8 are listed below:
1:
2:
3:
4:
5:
6:
7:
8:
9:
FlashCopy (FC) DBA to DBB (Safety Copy)

Resync DBA to DBc
Withdraw Prior FC DBD
Write Suspend DBA and Freeze PPRC pairs
Create final FC DBD & Write Resume
Resync PPRC
Backup DBD to EBU/TSM
Mount FC Volumes to Backup Host
Start R/3 DB2 UDB on Backup Host
Page 26
Backup Process
The first step in SMBR process would be to collect all the current layout details from
within the DB2 UDB database. Obtain the database structure information through the
query block ("LIST TABLESPACES SHOW DETAIL") that can determine the latest
information on the database structure including the log group, tablespaces, and
container information. During this step, the file structure information and volume group
layout along with device list from the host operating system is obtained.
On the Stand-by side shutdown all SAP and DB2 UDB processes and clean up any
semaphores that pertain to SAP/DB2 DB2 UDB on the target host decme and also on
the Safety FlashCopy target volumes (DBB) in the primary ESS. This step will confirm
that the current users in the SAP instance UDB on the decme host (probably the
Production Fix system users) are logged off and the journaled file systems on decme will
be unmounted. Unmount all the application related file systems except those that pertain
to the AIX host operating system on the decme host.
To be prepared for a new backup, the systems must be set back to the start situation.
When the DB2 suspend I/O command is issued on the primary system to stop data
volume activity, the suspension of writes on data and log volumes should prevent any
partial page writes for the next step. Read-only transactions will continue as long as they
are not requesting any resource held by the suspended I/O process.
While the database is in SUSPEND_WRITE state, any update operation to the
tablespaces, such as ALTER TABLESPACE, QUIESCE TABLESPACE, BACKUP
DATABASE ... FOR TABLESPACE, etc., will wait because they are waiting for the
tablespace write latch.
Page 27
The use of WRITE SUSPEND and WRITE RESUME commands exclusively developed
for the use with Hardware mirroring facilities, allows a consistent point-in-time snapshot
replica of the live database.
In the backup scenario as described above, we create a safety copy on the source
ESS1. This safety copy will enable us to recover the database with a DB2 conditional
restart on the secondary side, while the live system is available for error analysis.
In this SMBR set up, the following tasks were created with task names that are
indicative of the ESS CLI command sets:
FCAllNoBgCpWD:
Withdraw all previously held FlashCopy NoCopy relationship

between FlashCopy source/target pairs
FC10121416NoBgCp: FlashCopy source production volumes to target Safety copy

volumes in LSS10, LSS12, LSS14, LSS16 in NO COPY mode.
EstALL4PATH:
Establish paths and/or ensure that the PPRC paths exist

between ESS1 (LSS10, LSS12, LSS14, LSS16) and ESS2
(LSS11, LSS13, LSS15, LSS17)
ResyncPPRCAll:
Resynchronize PPRC source and target volumes in between

ESS1 & ESS2
Frzall:
Subsystem level withdrawal of paths and dataflow between

PPRC source and target volumes
FC11131517NoBgCp
OSPL2C0
FlashCopy PPRC target volumes ( DBC ) to DBD volumes in

LSS11, LSS13, LSS15, LSS17 in NO COPY mode.
is the Cluster 0 in ESS1
OSPL6C0
is the Cluster 0 in ESS2
rsExecuteTask.sh
is a CLI command to be initiated from AIX host
CLI command option -v provides verbose output
Page 28
Each SMBR step of the backup process is described as follows:

Step 1: Create a Safety FlashCopy of the Production Instance
i. Withdraw previous Safety Copy source/target FlashCopy relationships on the
Primary ESS1:
To Withdraw previous Safety Copy source/target FlashCopy relationships on the
Primary ESS1, as captured in a Task "FCAllNoBgCpWD", the following CLI is
initiated against OSPL2C0 from AIX.
rsExecuteTask.sh -v -s ospl2c0 FCAllNoBgCpWD
Ospl2c0 is the cluster1 of ESS1 that hosts Production Volumes.
FCAllNoBgCpWD is the predefined task set up between source Production volumes
and Target Safety Flash Copy volumes
ii. Quiesce the database write I/O's using the UDB feature
"db2 set write suspend for database"
iii. FlashCopy all DBA volumes to DBB volumes with No Copy option
rsExecuteTask.sh v s ospl2c0 FC10121416NoBgCp
iv. After the completion of the logical NoCopy, bring the database back from quiesce
to normal mode using DB2 UDB feature
db2 set write resume for database.
v. Extract all the system information pertaining to disk storage from within AIX on
joyjoy and remote copy those files to host decme.
Page 29
Step 2: Resynchronize DBA to DBC

i. Ensure that the Paths do exist between PPRC source (DBA) and target volumes
(DBC); establish if they do not already exist
rsExecuteTask.sh v s ospl2c0 EstALL4PATH
ii. Resynchronize all the PPRC source (DBA) and target (DBC) volumes.
rsExecuteTask.sh v s ospl2c0 ResyncPPRCAll
iii. Monitor % of resynch completions.
Step 3: Withdraw Previously held FlashCopy (source / target) Volumes on

ESS2 after stopping secondary R/3 Application and DB2 UDB
Database on the Backup host
In ESS2, initiate a withdrawl of previously held FlashCopy NoCopy relationship between
DBC and DBD volumes from a previous backup run.
RsExecuteTask.sh -v -s ospl6c0 FCAllNoBgCpWD
Step 4: Quiesce the Production database and Suspend all the PPRC source
and target Volumes
i. Quiesce the DBA - Production database write I/O's using the DB2 UDB feature
db2 set write suspend for database
This will enable a freeze of all write activity on DBA so that PPRC targets - DBC can
function as a consistent FlashCopy source volumes for the final Split Mirror on DBD.
Page 30
ii. Split Log and Data Volume pairs

(a) Obtain error code and send an automated message upon successful
suspension. Freeze all the ESS subsystem level PPRC relationships
between ESS1 and ESS2. Suspend all paired data and log volumes.
rsExecuteTask.sh v s ospl2c0 frzall
(b) Verify the status of the task completion using rsExecuteQuery in a query loop.
Step 5: Create final FlashCopy that can be backed up to a

backup utility
i. Verify the error codes from the PPRC Freeze.
ii. Proceed with the Final FlashCopy. Initiate FlashCopy of the source DBC to target
DBD in NoCopy mode.
rsExecuteTask.sh v s ospl6c0 FC11131517NoBgCp
iii. Verify error codes for the FlashCopy and upon error code of zero, bring back
quiesced database using
"db2 set write resume for database" command.
This step completes the tasks required to create a Split Mirror Copy of Productive
environment.
Page 31
Optional requirement for a complete physical copy:

If there is a requirement for a complete physical copy of the final FlashCopy (logical
copy) targets DBD, that are attached to the Backup Host decme, initiate a FlashCopy
with BackGroundCopy option from DBD to DBE (not shown in the Figure 8) in ESS2.
Step 6: Resynchronize PPRC source and target volumes

After successful completion of FlashCopy (logical copy) and DB2 UDB write_resume
on the Production instance, it is time to reinitiate the PPRC links between ESS1 and
ESS2.
i. Initiate PPRC links and establish paths between ospl2 and ospl6
rsExecuteTask.sh -v -s ospl2c0 EstAll4Path
ii. Resynchronize all the delta activity in ESS1 during the period of PPRC freeze
(during the execution of step 5 above).
rsExecuteTask.sh -v -s ospl2c0 ResyncPPRCAll
Step 7: Backup the Final FlashCopy Volumes to Enterprise Backup Utility

(EBU)
Once the physical movement of the data from DBC to DBD is completed, then remote
copy all the file systems, volume groups, and device lists from the source Production
Host.
At this point using Tivoli Storage Manager (TSM), Legato or any compatible EBU,
backup DBD instances SAP / DB2 UDB objects to tape. We will always save all the DBD
volumes to tape media by invoking complete file system backup or by using DB2
BACKUP utility.
Page 32
Step 8: Mount FlashCopy Volumes to Backup Host

Rediscover all the devices that are part of the defined volume groups at the host
operating system level and (varyon) all the volume groups and mount the file systems
required for DB2 UDB & SAP on host decme.
Varying-on and mounting of all the relevant volume groups is done after rediscovering
the physical volumes and remapping those to the virtual paths as seen by the host
operating system via the data path optimizer.
Step 9: Start DB2 UDB on the Backup Host for verification purposes
In order for us to verify that it is a consistent copy of the live database, this step is
optional. Startup DB2 UDB using db2inidb. Startup SAP after checking for db2 error
logs.
Page 33
6B
Split Mirror Backup & Recovery (SMBR) Scenarios
Customer Options and Configurations for Split Mirror Backup & Recovery
Figure 9: SMBR Scenarios
Scenario 1: Off-line Backup with FlashCopy using only one ESS storage
subsystem.
-
Stop Live UDB DBA ; FlashCopy (FC) DBA to DBB
When "Logically Complete", start DBA, move DBB to tape; If customer elects FC
physical copy option for backup and reporting purposes, then wait for FC physical
completion, start DBA.
Move to tape with Tivoli Data Protection (TDP)/Tivoli Storage Manager (TSM).
This Scenario incurs production down time.

Page 34
Scenario 2: Off-line Backup using FlashCopy and PPRC using two ESS
storage subsystems.
- Stop Live UDB DBA; Break/Suspend PPRC links from DBA to DBC
- FC DBA to DBB (safety copy); When "logical complete, FC DBC to DBD
- Start PPRC ReSynch
- When FC DBD "logical complete", start DBA
- Move DBD to tape with TDP/TS
This Scenario, while requiring production downtime for completion of the
FlashCopy command (logical completion), provides the disaster recovery
capability inherent in ESSs fault tolerant architecture.
Scenario 3: On-Line backup using FlashCopy and PPRC i.e. on-line

implementation of Scenario 2 with two ESS subsystems.
-
Execute DB2 UDBs "Write SUSPEND"..Write RESUME " commands for

consistent FC.
FC DBA to DBB (safety copy)
When FC logical complete, Write Resume
Start PPRC ReSynch (logs, control and container files); For high availability
considerations it is advisable to make a FlashCopy Copy (physical copy) prior to
executing PPRC with remote ESS
FC DBC to DBD
Write Resume DBA
Move DBA to tape with TDP/TSM
This scenario is intended to keep the production system available during the
backup while providing the flexibility to transmit a point-in-time copy to a remote
Disaster Recovery site.
Page 35
Scenario 4: On-line, "Zero-downtime" / Serverless Backup & Recovery

This is the scenario described in section 6A.
Primary (ESS1/Cluster1) DBA; FlashCopy (FC) DBB; Secondary Mirror

ESS2/Cluster2) DBC; FC Secondary DBD
FC DBA to DBB (safety copy before start of backup process); Execute DB2 UDBs
"Write SUSPEND", Write RESUME " commands for consistent FC.
ReSynch PPRC (logs + data volumes) pairs to include transactions during safety
copy step and all changed data since last data files resynch process
Monitor % resynch completion; this process creates complete mirror (logs + data) of
ESS1 on ESS2
As soon as 100% Resynch completes, execute UDB commands " Write SUSPEND "
Suspend PPRC (logs + data vols) on ESS1
FC DBC to DBD for on-disk Point-In-Time-Copy (NoBgCp - No Background Copy)

When FC DBD "logical complete", execute database " Write RESUME ",
"ReSynch" PPRC (log volumes) to resume mirroring of Log volumes for high
availability
Move DBD to tape using TDP/TSM
Start SAP on backup server from DBD to prove restartability
Integrate all scripts for automation through backup agents like TSM, Legato
Networker etc.
For this "serverless/zero downtime scenario - backup activity places no load

on the Production Server or Primary Storage. This scenario conforms to SAPs
recommendation for high availability, advanced infrastructure backup & recovery
requirements.
Page 36
6C Key Observations about the Split Mirror Backup & Recovery

scenarios
Scenario 1: In this single configuration implementation of Split Mirror Backup,
customer expects to have downtime as the database is momentarily shut down for Pointin-Time Copy (PITC) using FlashCopy.
Scenario 2: In this configuration using two ESSs, the customer expects to have
downtime while creating Point-in-Time Copy using FlashCopy (physical copy). PPRC is
used to move this FlashCopy (physical copy) on the Primary ESS to the Secondary
ESS. The PPRC targets on the secondary can then be sent to Tape for remote vaulting.
This scenario provides a disaster recovery capability.
Scenario 3: Repeats Scenario 2 keeping DB2 UDB in on-line mode with "Write
Suspend" and "Write Resume" Operations. With DB2 UDB functions, downtime is
avoided. As in scenario two above, PPRC provides disaster recovery and remote
vaulting capabilities.
Scenario 4: In this complete Split Mirror Backup solution using two - ESS's, customer
has the ability to bring up four copies (Production & Mirror [2 copies] + FlashCopy on
Primary ESS + FlashCopy on Secondary ESS) of the production database. DB2 UDB
"write suspend" and "write resume" features, enable consistent image copies of live DB2
UDB database to be created while suspending and resuming I/O.
After a successful backup, perform a test restore of the database to check the physical
correctness of the data transferred, of the tape device, and it's software drivers.
Depending on the number of different databases laid out within the same LSS and
depending on the requirements for a multi-system database point-in-time concurrent split
mirror backup, individual LUN level SUSPEND can be issued with the ESS CLI instead
of the PPRC Freeze command.
Page 37
The backup strategy should include verification of the data that was backed up. Perform
a logical data check using db2dart (during periods of low system activity) to verify the
consistency of the DB2 database. Corrupt DB2 blocks (error SQL2412) can appear in an
R/3 database as a result of operating system or hardware errors and may make the
backup unusable. The existence of these blocks only becomes evident during the next
read access attempt to a table within the database. Since this particular access attempt
may not occur often, and corrupt DB2 blocks are not recognized during a database
backup, corrupt blocks can remain undetected in an R/3 UDB system for a long time.
7 Recovery
The backup process described above will provide a consistent copy of the live database
achieved at the moment of application write suspension. This backup file set will be on
tape media and on the DBD volumes of ESS2 (OSPL6). Recovery process in DB2 can
be achieved in 2 types:
Rollforward recovery is performed when all or parts of the persistent storage (disk)
have been lost or corrupted. The process could be very lengthy and would require
manual intervention in a non-SMBR process.
Restart recovery is performed by DB2 whenever data in the non-persistent memory

(RAM) has been lost, due to a system hardware failure or a power loss. Restart recovery
is done by restarting the DB2 database, which initiates a crash recovery. DB2 reruns all
transactions that were performed previously but have not been recorded in persistent
memory. To enable DB2 to perform automatic restart recovery after a crash, the
database parameter AUTORESTART is to be set to ON by the R/3 installation program.
The "DB2 log file header file" which contains the log sequence number, must be used as
a starting point for this type of recovery. In cases of a disaster that includes loss of this
file, DB2 recovery is done by restarting the database and performing a rollforward
recovery.
Page 38
The database restart brings the database to the latest consistent state based on the disk
image and the online log files. A restart recovery is based on the entries in the log
control file and the online log files. If a disk error occurs and the contents of the disk are
not recoverable, a restart recovery will not work. This applies to the containers of the
database, the online log files, and the log control file. Therefore, as discussed in Section
J of the Appendix, R/3 UDB Volume Layout has to use ESS RAID5's striping, which
allows a highly available disk subsystem that mirrors the data, the log files, the
containers, and the control files.
The contents of DBD and DBC volumes of the SMBR process differ from each other in
that the restart of the DBD SAP UDB instance will roll back all the transactions that were
open at the time of write suspension. Also, because of the RESYNC task in Step 8 of the
SMBR process, DBC and DBA volumes are a pair of synchronous mirrors and differ from
the DBD (Point In Time) volumes.
The DBD volumes can either be left open for multiple purposes like Database validity
checks, Database reorganization dry-run tasks, testing SAP support packages, testing
and applying of OSS notes or for the purposes of serving as a production fix system or
as a reporting instance. To set up the DBD volumes to serve as a standby database:
a) mount the volumes on host "decme"
b) place the DBD in roll-forward pending and then rollforward the mirror.
This is accomplished by running the tool "db2inidb UDB as standby" (UDB is the DB
instance) to place the mirrored database in roll-forward pending state, removing the
suspended write state and rollforward the database to the end of logs.
Page 39
The database recovery is performed in several steps:

1) For a recovery process, the tape backup needs to be first restored on to the DBD
volumes and then the DBD volumes can be FlashCopied on to the PPRC
volumes on ESS2 (OSPL6). The recovery steps further involve executing PPRC
from source ESS2 to the target volumes on ESS1.
2) Depending on the Service Level Agreements and consistent with customer
scenarios and requirements, we can repeat the steps in reverse order of the
backup scenarios for achieving the recovery of a consistent database copy from
a tape media or from ESS2 volumes on DBD to provide a completely recovered
SAP UDB instance.
3) Transaction log processing. Setup a user exit program to retrieve the log files
that are requested in the recovery history file. The file specifies the log files
needed to maintain consistency of an online backup. In the rollforward phase,
use the DB2 Control Center to roll forward the transactions using the log files that
have been retrieved.
The database can be forwarded to a certain point in time or to the end of the logs
using Coordinated Universal Time (CUT or GMT) since the database server
calculates the timestamps based on this time zone. Applications from different
time zones may connect to the database, but internal processing is done in CUT.
However, the timestamp used for backups is based on the local time zone of the
database server.
4) During the rollforward process, all committed transactions are reapplied to the
database. At the end of the rollforward, a list of open transactions exist. These
transactions are still awaiting a commit. For data consistency reasons, open
transactions will now be rolled back, to make sure that changes that are not
committed are not applied to the database. Then the database is instructed to
"rollforward stop". With this, the database is made consistent.
Page 40
Crash Recovery
The time needed to perform a crash recovery depends on the amount of work DB2 has
to redo. DB2 must redo part of the transactions in the log files. That is committed
transactions are reapplied to the database and uncommitted transactions are removed
from the database. The starting point for this reapplying is found in the log control file.
This file contains the lowest log sequence number (LSN) needed for a crash recovery.
All log records that are older than the LSN are discarded because these changes to the
data in the buffer pool have already been written to disk during the soft checkpoint.
SOFTMAX (a DB2 UDB configuration parameter) specifies the percentage of a log file
used before a soft checkpoint will occur. Large databases are configured with
SOFTMAX spanning several log files.
DB2 UDB's Forward Recovery uses a backup image (DBD) and the entire subsequent
log files created after the backup are used to recover the database to a consistent state.
The design of a backup strategy should be based on how much time is to be spent on a
recovery. With ESS SMBR's advantage in providing rapid backup of VLDB's (Very Large
Data Base) made possible by the combination of PPRC and FlashCopy CopyServices
functions, customers can have multiple Point In Time backups which will enable them to
perform a faster recovery since there are fewer log files to redo.
Other Considerations during Recovery:

During the restore process, the db2agent process communicates with, and controls the
operation of the db2med and db2bm processes. During restore operations, the media
controllers transfer data from device into restore buffers. During a restore, DB2 uses
page cleaners to transfer the buffer contents to disk. The number of buffers should be
large enough to keep the media controllers and their devices busy (generally more than
2). The size of the restore buffer should be 1024, or preferably 2048 4k pages (4 MB,
8MB) or a multiple of the backup buffer size.
Page 41
By using the same size for the restore buffers as achieved in a good backup
performance with a certain backup buffer size, excellent restore performance can be
achieved. When a buffer is full, it is transferred to disk on FIFO (First In First Out) basis.
The number of db2med processes depends on the number of restore devices or
sessions. It is always much faster to empty the buffers to disk than to fill them from
tape/TSM. The usage of DB2 UDB parameter, PARALLELISM when reading the backup
image(s) from disk during a restore will be particularly helpful when not using multiple
tape devices or TSM sessions or when not using multiple db2bm processes.
8 Process Automation - Solution Integration in Customer

Environments
In establishing this Split Mirror Backup & Recovery solution for R/3 with DB2 UDB and
ESS, IBM has created end-to-end automated procedures that will enable this solution to
seamlessly integrate into customer environments. While recognizing that each customer
will likely implement this solution uniquely, the procedures incorporate use of platformindependent Java technology, which will execute the customized, set of Split Mirror task
routines set up for a particular backup environment. Inherent benefits of Java technology
implementation will accrue to the customer in terms of flexibility, security, and location
transparency. With this standards-driven approach, customers can integrate this solution
with customer-preferred enterprise solutions management consoles (TME10, OpenView,
BMC etc).
Page 42
Acknowledgements
Our sincerest thanks to Siegfried Schmidt and Dr. Jens Claussen, of SAPs
Advanced Technology Group (ATG) for their valuable guidance and for sharing
SAPs vision and requirements with us to accomplish the work that is reported in
this paper. In particular this white paper draws considerably from the insightful
experiments, observation and publications on DB2 UDB and storage, conducted by
Dr. Claussen at the SAP ATG labs in Walldorf, Germany. Siegfrieds earlier work
with us on the Split Mirror Backup & Recovery solution implementation on the
DB2OS390 platform with IBMs RVA and ESS storage subsystems have paved the
way for this current implementation with DB2 UDB on the AIX platform.
We would also like to thank a number of IBM colleagues with whose thoughtful
guidance, this work is brought forward with the benefit of their applied knowledge in
R/3, DB2 UDB and ESS, in delivering high availability, zero downtime, and easy to
use Split Mirror Backup and Recovery Solution for mission critical SAP customer
environments. Matt Huras, Jason Racicot, Bill Micka, David Short and James Teng
have been most helpful with their knowledge of DB2 UDB, the ESS and its
advanced functions and have endured this journey with us.
Sanjoy Das,
Siegfried Schmidt,
sanjoy@us.ibm.com
siegfried.schmidt@sap.com
Jens Claussen
BalaSanni Godavari
jens.claussen@sap.com
godavari@us.ibm.com
Page 43
Appendix
A) DB2 UDB Architectural Overview
Figure 10: DB2 UDB Architectural Overview

Figure 10 shows the architecture of DB2 UDB structures. It contains data structures on
disk (extents for objects like tables and indices), in memory (buffer pools, log buffers), in
processes (prefetchers, page cleaners, loggers and sub agents) and connectivity for
applications (UDB client library, coordinator agents, for usage of shared memory,
semaphores, named pipes, protocols TCPIP, Named Pipes, SNA, NetBIOS and
IPX/SPX).
Page 44
Below the level of SQL, DB2 UDB manages data objects using extents. An extent is an
adjacent grouping of a fixed number of DB2 pages.
In order to optimize disk I/O and data distribution, tablespaces manage pages by
grouping them into extents per each database object. The optimal R/3 UDB extent size
ranges from 8 to 64 pages. There should be enough free space in the container of a
tablespace to accommodate the additional extents required for database object growth.
The extent size (the number of adjacent pages) is defined on a per-tablespace level. It
cannot be changed, once it has been defined during installation of the R/3 database.
Typical extent size is eight for tablespaces with many small and unused tables, but up to
64 for the tablespace containing ABAP repository tables. All the extents for a data object
are in one tablespace, but can be spread over several OS files or devices. In DB2 these
storage facilities are called containers.
DB2 UDB supports two kinds of tablespaces:
-
Systems Manager Storage (SMS) tablespaces
Database Managed Storage (DMS) tablespaces
SAP only uses DMS table spaces for data and indexes. The recommendation for R/3
Temp Space is on SMS and for BW on DMS for maximum performance. Both types of
tablespaces may be used in the same database. SMS tablespaces are called Systems
Managed because the operating systems file system manages the containers. With
DMS, DB2 itself manages the container space and allocates the space when the
container is created.
There is no set limitation on the number of extents in a tablespace, but a limit to the total
number of pages. This is reflected in the maximum table/tablespace size of 64 GB with
4K pages, 128 GB with 8K pages, 256 GB with 16K pages and 512 GB with 32K pages.
Page 45
The DB2 Extended Enterprise Edition (EEE) enables the distribution of a database
across multiple hosts and to install multiple partitions on a single host.
The limitation of 2
24
pages per tablespace refers to a single partition of the database.
Consequently, if a tablespace is distributed across multiple partitions (nodes) of the EEE

instance either on a single host or on multiple hosts, the limit is raised to n * 2
24
pages
per tablespace (n is the number of partitions on which the table space is stored). SAP,
however, currently supports EEE only for a few mySAP components.
When a DB2 UDB database instance is started, several processes are created:
1) Dedicated Shadow Processes are created when a new SAP work process
establishes a connection to DB2.
2) Shared Processes are required by the DBMS systems to function and perform
various database management tasks.
The UDB database is stored in 4/8/16/32 KB blocks in data files on disk. The container
files, online and offline log files and control files can be stored like in our test
environment in an underlying set of RAID5 disks of a storage sub system. In order to
accelerate read/write access to the data, these data blocks are cached in the database
buffer pool in production CPU main memory.
Any changes or modification to the database data are logged in the online log files. This
procedure ensures security of data. For fail-safe database operations, without using
additional operating system utilities, the control files and the online active/retained log
(log dir/log retain) files of the database system can be like in our test environment in
constant synchronous remote copy mode.
The UDB database management system holds the executable SQL Statements in the
Shared SQL Area which is part of the shared pool, only once for all processes. The DB2
Storage and Memory structures interact as depicted in Figure 11.
Page 46
Figure 11: DB2 UDB Storage and Memory Structure
Each R/3 work process, as shown in Figure 12, does the following:
a. Connects to the database as one database user
b. Handles database requests for the different R/3 users
c. Communicates with a corresponding shadow process on the database
Page 47
Figure 12: SAP R/3 Work Processes and their Interaction with DB2 UDB
The
R/3
naming
convention
for
tablespace
names
is
PSAP<tablespace_name><extension>. The abbreviations in the tablespace name are

part of the container name. Containers are numbered starting with 001.
Depending on the size of each data record, several data records can be stored in a DB2
page. Within a tablespace, the extents belonging to different data objects compete for
storage space. DB2 allows to separate data and index objects into different tablespaces.
In non-RAID5 storage architecture implementations, the emphasis is to distribute data
and index tablespaces to different physical devices, thus improving database

performance.
Page 48
With ESSs robust RAID5 architecture and its use of striping different types of data on all
RAID arrays, the performance hot-spots are eliminated while I/O performance improves
through parallel access to all members of the array.
B) DB2 UDB Database Growth and Impact on Storage

Over time, because of a gradual and continual accumulation of data or repeated
insertion and deletion operations, fragmentation occurs within a data page. Once space
has been allocated, it can be released for re-use only after table reorganization or a
physical deletion operation (DROP). This can cause storage space fragmentation within
a tablespace, which can be resolved by DB2 UDB online tablespace reorgs.
As shown in Figure 13, reorganizing tables is an operation that is performed on an
exceptional basis. R/3 System may stay up and running, with no heavy transaction load
on the tables which are to be reorganized. PSAPTEMP temporary tablespace is used for
the copy of the table while reorging, thus requiring this tablespace to hold all the data of
the copied table. In general, it is recommended to allow for 2 times the physical size of
the table according to transaction DB02. Very large tables should not be reorganized
using the online method.
Page 49
Figure 13: DB2 UDB Usage Patterns and Impact on Storage

In order to accommodate growing tablespace requirements, each time a new container
is added to the tablespace, the rebalancing process is started. When a new container is
added, DB2 rebalances the tablespace:
1. It creates the container, that is, it allocates the specified amount of pages in a file
system.
2. When the container is available, a process or a thread is started to rebalance the
extents over all containers of the tablespace. This helps ensure that extents are
distributed evenly among all the containers, assisting data distribution for existing
objects and proper striping for newly allocated extents.
Page 50
C) DB2 UDB Memory Management

DB2 UDB provides Buffer Pool, Package Cache, and Catalog Cache for memory
management. Figure 14 depicts how R/3 work processes working through DB2 agents
utilize these three memory management functions to execute logical and physical (disk)
access.
Buffer pool(s) - provide the storage for data and index pages. The buffer pool functions
as an optimized cache for the database to improve database system performance. Since
this cache optimizes its strategy for database use, it is better to use the buffer pool(s)
than to use a large file system cache. The SAP architecture design goal is to minimize
disk accesses (physical reads) and to maximize buffer pool access (logical reads). While
the size of the primary buffer pool is controlled by DB2, buffer pools can also be defined
at tablespace level with at least one buffer pool per database.
Package cache: It stores the DB2 UDB database access plans as part of the dynamic
statement cache function, thus helping to reduce the time/cost per connection to the
database.
Catalog cache: It stores binary and compressed descriptors for tables, views, and
aliases to avoid reading directly from disk.
Page 51
Figure 14: DB2 UDB Memory Management for each SAP work Process
D) Backup and Recovery Management for DB2 UDB

IBM DB2 provides the following tools for performing data backups and restores (Figure
15):
-
DB2 Backup backs up the container files and the control information
DB2 Restore allows data retrieval from the backup image.
The DB2 backup tools store information in the DB2 History file, which is then read by
R/3 tools.
Page 52
Figure 15: Backup & Recovery Tools on UDB for R/3

External data management systems like TSM (Tivoli Storage Manager from IBM) or
Networker (from Legato) enable backup procedures to be integrated with the existing
operations of a computer center. DB2 UDB supports Legato Networker and IBM TDP
R/3 as hierarchical storage management systems for DB2 backup/restore. These
systems improve backup throughput and allow easy management of large number of
tapes. R/3 on DB2 UDB currently supports IBM TDP R/3 as a hierarchical storage
system for log file archiving in brarchive.
SAP delivers several executables to manage database backups and log files archived
offline:
1) db2adutl (Administration utility for TSM, described in SAP R/3 note: 67789)
-
Query, selects backup images in TSM
Extract, extracts backups from TSM to disk
Delete, marks database backup in TSM as inactive, deletion will take place
according to retention policy
Page 53
2) BRARCHIVE (Stores offline retained log files into TSM). BRARCHIVE backs up
the database log files from the offline log directory. BRARCHIVE can be invoked
through the command line, but is usually called from the R/3 DB2 Control Center
Extensions DB2Admin.
3) BRRESTORE (Restores offline archived log files from TSM. BRRESTORE is
used to retrieve log files into a log-retrieval directory (log_retrieve) during
recovery. It is typically used from the DB2 Control Center Extensions.
4) BRDB6QRY (Query for offline archived log files in TSM)
5) BRDBSDSD (Deletes offline archived log files from TSM)
As shown in the following Figure 16, the SAP delivered executables perform backup of
DB2 UDB Control Files, DB2 UDB Containers, Offline Log Files, On Line Log Files and
SAP objects.
Figure 16: SAP R/3 Backup Objects

Page 54
E) Work Process Management in SAP R/3 with DB2 UDB and

Storage
An SAP R/3 work process enables a user to connect to the UDB database. It is only
active for one user during a dialog step i.e. between screen input and screen output.
Different users use the same work process and database connection consecutively
because the work process issues a commit to the database before screen output.
Figure 17: SAP R/3 Work Process Management with DB2 UDB
From the storage subsystems perspective, it is important to note how the SAP R/3
processes, as shown in Figure 17 above, handle database SQL statements and logging:
Page 55
1. A SQL statement is sent from the R/3 work process to its associated agent.
2. The DB2 Optimizer function in the DB2 agent transforms the SQL statement into an
UDB access plan, which is usually cached as part of a UDB function called dynamic
statement cache.
3. DB2 parallel I/O servers obtain information about the required buffer pool pages from
the data containers with the help of db2agents that execute in parallel, performing
the tasks defined in the DB2 access plan. The db2 agents also obtain
UPDATE/INSERT SQL statements from SAP R/3 application.
4. Data changes that occur in the buffer pool and not transferred to disk are written as
log records in the log buffer. The log file header (LFH) is moved forward to the first
log record, which contains data about pages that are changed and not on disk.
Changed pages from buffer pool(s) are copied asynchronously to containers. If the
LFH moves out of a log file, this file is not required for a crash recovery. If the file
sqlogctl.lfh containing the log file header is not readable, crash recovery is not
possible and the database must be restored.
5. A COMMIT / ROLLBACK SQL statement at the completion of the transaction causes
the contents of the log buffer to be synchronously written to the current log file in
log_dir directory.
6. Asynchronous I/O cleaners or page cleaners destage updated pages onto disk when
triggered by the following:
The maximum amount of log space that should be read during crash
recovery has been reached
The maximum percentage of changed pages has been reached
There are no pages available during insert/update
Page 56
F) SAP R/3 data layout
In preparation for SAP R/3 UDB data layout of the ESS, the following sequence of
activities should be considered:
-
Find Number and types of SAP systems to be installed in the system landscape
that typically includes Development, Quality Assurance and Production system
instances.
Start with the Planning of an SAP landscape topology, estimating hardware

requirements for storage, network and server machines. This creates a
hardware requirements list: ESS boxes, number of SCSI/FC connections, server
machines, network connectivity etc.
Sizing (done by HW partners): identify requirements for CPU
Storage space, I/O Access
Network bandwidth
System infrastructure/landscape (inter system communication)
SAP installation guide
Assignment of SAP systems to hosts
Estimations for storage needs of individual database systems / tablespaces
Collect space requirements from SAP applications, their databases

(tablespaces, logging), 3rd party software (bolt-ons)
Consider ESS / Host / DB/ Application restrictions
Map requirements to AIX logical volumes
Group the logical volumes to volume groups
Design ESS configuration and perform configuration steps with ESS Specialist
Create AIX volume groups and logical volumes
Customer experiences have shown that poor database performances at SAP

installations often result from high I/O wait times for database access in non-RAID5
storage. I/O contention can be traced to:
Page 57
- Inefficient application design (expensive, unnecessary, and poorly qualified

statements)
- Data not evenly distributed across many RAID1 disk cylinders
- Disk not fast enough to handle high I/O activity
- Heavily accessed tables or indexes not distributed or striped across many disks
- Incorrect hardware configuration such as many disks on few controllers
- DB2 UDB objects (table spaces and directories) not optimally laid out, thus causing
numerous DB2 threads or processes to access the same disks as shown in the
Figure 18.
Figure 18: SAP R/3 and DB2 UDB Threads Impact on Disk Arrays
In the past, the database layouts have often tried to separate table spaces onto different
devices, e.g., separating data and index table spaces and putting each data table space
on different drives. The consequence was that the database often ended in hotspots on
Page 58
devices that carried a single table space. Manual intervention was necessary in each
case to alleviate the problem. Additional performance degradations occur over time due
to changing application profiles and data growth leading to more hotspots and
restructuring demands.
With advances in storage subsystem technologies and in particular advanced RAID5
implementations, application can now experience significant performance improvements,
high availability and recoverability. In a typical SMBR infrastructure set up, two ESS
storage sub systems would be used. In our SMBR set up, we have two ESS storage sub
systems (OSPL2 and OSPL6) for production and disaster recovery/backup server
access respectively. In OSPL2 we have OSPLC0 and OSPLC1 as high availability
cluster nodes.
G) ESS Raid5 Disk Layout Considerations for SAP R/3

Environments
With proper distribution of all kinds of data across all available devices using state of the
art storage systems like ESS, the whole I/O subsystem is made available to each
individual database object (table space, directory, table, index).
As hotspots tend to occur on a subset of the table spaces or even on a single table
space, by striping these tablespaces across all devices, the cumulative performance of
all devices and device adapters is made available to each individual table space to
improve performance.
ESS RAID5, with maximum width striping of all UDB tablespaces, relieves the DBA from
hotspot identification and tracking which might arise from changing application profiles,
shifting workloads and rapid data growths in non-RAID5 storage environments.
Striping under RAID5 allows further striping of single table space growth while totally
avoiding any additional cost of striping table spaces across all devices.
Page 59
Within the ESS, as show in Figure 19 below, vertical boxes denote RAID5 volumes
(ranks) consisting of several physical disks.
Figure 19: Striping UDB Tablespaces and File Systems on ESS RAID5 Arrays
The ranks are connected via device adapters to the ESS fault tolerant control unit that
contains, e.g., cache and NVRAM as shown in the Figure 20. The control unit attaches
to host systems via SCSI or fiber channel adapters.
Page 60
Figure 20: ESSs Fault Tolerant Architecture and Logical Sub Systems Design
Each UDB tablespace and each file system is striped across all RAID5 volumes. This is
the optimum configuration since the ESSs built-in I/O capabilities are allocated
automatically and dynamically on demand to all the table spaces.
In order to keep the containers balanced, all containers should have the same size.
Since DB2 applies a striping approach on extent basis, the individual containers should
not be striped once again across multiple volumes. Instead, each container should be
Page 61
kept locally on a single RAID volume. A clear one-to-one mapping between containers
and RAID logical volumes (LUNs) also simplifies administration.
A striped container is a DB2 Database Managed table space Storage (DMS) container,
or raw container that is mapped to a device which implements data striping beneath the
database i.e. at disk subsystem level.
Figure 21 shows a tablespace with twelve containers striped across all RAID volumes.
Two containers are mapped to each RAID volume.
Spreading this logical volume across separate hdisks on the same array would provide
no performance benefit, because all of the activity is still going to a single RAID5 array.
However, it should be noted that ESS stripes data equally across all disks in a single
array. The best technique for maximizing throughput of ESS is to balance workload
across as many RAID5 arrays as possible. For some applications, defining AIX logical
volumes across ESS logical disks on separate RAID5 arrays is one way to do this. How
ever, in cases of maintenance requirements to reformat a specific set of volumes
belonging to a single host or of the multiple hosts connected to an ESS, it would require
the reformatting of entire array. Another way would be to spread the database files
across RAID5 arrays.
ESS Configuration for performance may need to be balanced with availability
considerations. If all databases sharing the ESS should be highly available, the
database files of all the databases will be spread over the ESS clusters and DA pairs to
ensure that there are alternate paths to the same data. SDD allows for maintaining these
alternate paths and also can help in load balancing across multiple paths.
Page 62
Figure 21: Distributing File Containers in RAID5 Devices

Finding out on which specific array a set of disks are located can be accomplished by
a) using SDD command "datapath query device or
b) using CopyServices Command Line Interface (CLI) rsList2105s.sh
H) Considerations for DB2 UDB database layout

Considerations for database layout on ESS should also incorporate the following
elements:
-
Ensuring that the PREFETCHSIZE of the tablespace is the RAID stripe size
multiplied by the number of RAID parallel devices. PREFETCHSIZE can be adjusted
to a value that is a product of EXTENTSIZE and number of containers. RAID stripe
size for 18GB Disk Drive Modules (DDM) is 32k and for 36GB DDM the stripe size is
64k.
Page 63
Using the DB2_PARALLEL_IO registry variable as shown in the command set, to

enable parallel I/O for the tablespace:
db2set DB2_PARALLEL_IO= 3,4,5,6 (any one value)
Using the DB2_STRIPED_CONTAINERS registry variable as shown in the

command set below, to ensure extent boundaries are aligned in the table space
during tablespace creation:
db2set DB2_STRIPED_CONTAINERS = ON; db2stop; db2start.
To check if the containers were created with ON option, db2dart command can be
run (as shown in Figure 22).
Making the EXTENTSIZE of the tablespace equal to, or a multiple of, the RAID
stripe size.
Ensuring that the tablespaces are set up well for prefetching and parallel I/O (most
important for DMS raw containers since there is no operating system caching or
prefetching especially when sequential I/O is important (e.g. Business Warehouse
workloads).
Using larger page sizes for sequential row access workloads (e.g. SAP BW) and
smaller page sizes for random row access workloads (e.g. OLTP)
Page 64
Figure 22: UDB data layout considerations in ESS (RAID5)
ESS allows the administrator to define a logical disk that spans the entire RAID5 array,
and also allows for the RAID5 array to be split into many logical disks. For example, a
6+P array of 18.2 GB disks has roughly 104GB of usable space. This can be defined as
one 104GB logical disk, or as 4 x 24GB logical disks and 1 x 8GB logical disk, or 13 x
8GB logical disks.
Page 65
From a performance perspective, one 104GB logical disk should provide roughly the
same throughput as the aggregate of many smaller logical disks on the same array. It is
the array that determines the throughput, not the number of logical disks. There are no
recommended LUN sizes for the ESS. It will vary based on the application and customer
requirements.
Typically, one would allocate 8 or 16 GB LUNs over the available storage and then
distribute the data across the RAID5 arrays to get optimum performance. Further
distribution of I/O is achieved by creating AIX logical volumes using the ESS logical
volumes defined in multiple RAID arrays. Monitoring the I/O access pattern can then be
used for further tuning and re-distributing heavily accessed database files across the
arrays.
I) Sub System Device Driver (SDD)

ESSs SDD, a High Availability Automatic I/O Path Failover Function in the ESS,
enhances data availability. If a failure occurs in the data path between the Host System
and the ESS, SDD automatically switches the I/O to another path. It will also move the
"failed" path back online after a repair is made. It takes advantage of multiple active
paths, distributing the I/O workload and supports more than one path from the host to
the shared LUNs. A single LUN can appear as two or more LUNs. This avoids I/O
bottlenecks that could occur when too many I/O operations are directed to a common
device using the same I/O path.
Figure 23 highlights the necessary steps in preparing the layout for an ESS-based SAP
R/3 UDB system using fixed block (FB) for Open Systems (and CKD for OS390) format:
Page 66
Figure 23: Steps in preparing the ESS for data layout and connectivity to hosts
In AIX, the Logical Volume Manager uses the defined 8GB LUNs by assigning them to
volume groups as hdisks. Each ESS logical disk translates to AIX like an hdisk (or vpath
if using SDD) as shown in the Figure 24.
With each hdisk in AIX, the host sets the
maximum number of concurrent requests that it will send to ESS at 20, which is more
than enough to saturate the 7 disks in an array. This is referred to as queue_depth and
allows for high availability and data distribution across multiple disks in SAP
environments. 'queue_depth' specifies the maximum number of commands that the SSA
disk device driver dispatches for a single disk drive for a hdisk. For high OLTP SAP R/3
installations, ESS provides another parameter, MAX_COALESCE that is the maximum
number of bytes that the SSA disk device driver (SDD) attempts to transfer to or from an
SSA logical disk in one operation. For an N+P array, it is recommended to use
queue_depth=2N and MAX_COALESCE =64K * N during the ESS install procedure.
Page 67
Figure 24: ESS LUNS mapped to AIX Logical Volume Manager
Page 68
J) Volume Layout for DB2 UDB

Tables listed as Figures 25A and 25B show the volume layout for SAP R/3 in DB2 UDB
and ESS environment using the Data Path Optimizers mapping of vpaths to hdisks.
Vpath
device
Serial No
Hdisk mapping
DA
loop
Volume
Group
vpath0
60012005 Hdisk5
Hdisk45
hdisk89
hdisk125 Loop A
LSS16data
vpath1
60112005 Hdisk6
Hdisk46
hdisk90
hdisk126 Loop A
LSS16data
vpath2
60212005 Hdisk7
Hdisk47
hdisk91
hdisk127 Loop A
LSS16data
vpath3
60312005 Hdisk8
Hdisk48
hdisk92
hdisk128 Loop A
LSS16data
vpath4
60C12005 Hdisk9
Hdisk49
hdisk85
hdisk129 Loop B
LSS16data
vpath5
60D12005 Hdisk10
Hdisk50
hdisk86
hdisk130 Loop B
LSS16data
vpath6
60E12005 Hdisk11
Hdisk51
hdisk87
hdisk131 Loop B
LSS16data
vpath7
60F12005 Hdisk12
Hdisk52
hdisk88
hdisk132 Loop B
LSS16data
vpath8
61812005 Hdisk13
Hdisk53
hdisk94
hdisk133 Loop A
Logvg
vpath9
61912005 Hdisk14
Hdisk54
hdisk93
hdisk134 Loop B
Logvg
vpath40
61A12005 Hdisk165 Hdisk173 hdisk181 hdisk189 Loop A
Sapvg
vpath41
61B12005 Hdisk166 Hdisk174 hdisk182 hdisk190 Loop B
Sapvg
vpath10
40012005 Hdisk15
Hdisk55
hdisk95
hdisk135 Loop A
LSS14data
vpath11
40112005 Hdisk16
Hdisk56
hdisk96
hdisk136 Loop A
LSS14data
vpath12
40212005 Hdisk17
Hdisk57
hdisk97
hdisk137 Loop A
LSS14data
vpath1
40312005 Hdisk18
Hdisk58
hdisk98
hdisk138 Loop A
LSS14data
vpath14
40C12005 Hdisk19
Hdisk59
hdisk99
hdisk139 Loop B
LSS14data
vpath15
40D12005 Hdisk20
Hdisk60
hdisk100 hdisk140 Loop B
LSS14data
vpath16
40E12005 Hdisk21
Hdisk61
LSS14data
Vpath17
40F12005 Hdisk22
Hdisk62
LSS14data
Vpath18
41812005 Hdisk23
Hdisk63
hdisk103 hdisk143 Loop A
Logvg
Vpath19
41912005 Hdisk24
Hdisk64
Logvg
Vpath42
41A12005 Hdisk167 Hdisk175 hdisk183 hdisk191 Loop A
Sapvg
Vpath43
41B12005 Hdisk168 Hdisk176 hdisk184 hdisk192 Loop B
Sapvg
Figure 25A: Volume Mapping & Layout
Page 69
Vpath
Device
Serial No
Hdisk mapping
DA
loop
Volume
Group
Vpath20 20112005 Hdisk25
Hdisk65
LSS12data
Hdisk66
LSS12data
Hdisk67
LSS12data
Hdisk68
LSS12data
Vpath24 20C12005 Hdisk29
Hdisk69
LSS12data
Vpath25 20D12005 Hdisk30
Hdisk70
LSS12data
Vpath26 20E12005 Hdisk31
Hdisk71
LSS12data
Vpath27 20F12005 Hdisk32
Hdisk72
LSS12data
Vpath29 21812005 hdisk34
Hdisk73
Logvg
Hdisk74
Logvg
Vpath44 21A12005 hdisk169 Hdisk177 hdisk185 hdisk193 Loop A
Sapvg
Vpath45 21B12005 hdisk170 Hdisk178 hdisk186 hdisk194 Loop B
Sapvg
Hdisk75
LSS10data
Hdisk76
LSS10data
Hdisk77
LSS10data
Hdisk78
LSS10data
Vpath34 00C12005 hdisk39
Hdisk79
LSS10data
Vpath35 00D12005 hdisk40
Hdisk80
LSS10data
Vpath36 00E12005 hdisk41
Hdisk81
LSS10data
Vpath37 00F12005 hdisk42
Hdisk82
LSS10data
Hdisk84
Logvg
Hdisk83
Logvg
Vpath46 01A12005 hdisk171 Hdisk179 hdisk187 hdisk195 Loop A
Sapvg
Vpath47 01B12005 hdisk172 Hdisk180 hdisk188 hdisk196 Loop B
Sapvg
Figure 25B: Volume Mapping & Layout
Page 70
K) The Role of the ESS Specialist

With ESS Specialist, the WebGUI interface for the Enterprise Server, all the tasks
needed for setting up arrays, logical subsystems, logical unit numberings, host
definitions for connectivity via SCSI/Fiber, PPRC path definitions and Copy services
tasks can be administered and managed. This is shown in Figure 26.
Figure 26: ESS Specialist Views and Functions
Page 71
References
1. IBM Enterprise Storage Server - IBM Document Number: SG24-5465-00 (
http://www.redbooks.ibm.com/ )
2. Implementing the Enterprise Storage Server in your Environment IBM
Document Number SG24-5420-00 ( http://www.redbooks.ibm.com/ )
3. IBM Enterprise Storage Server Performance Monitoring and Tuning Guide IBM Document Number: SG24-5656-00
4. ESS Layout Considerations for a Heterogeneous System Landscape, Dr. Jens
Claussen, SAP Advanced Technology Group, December 2000.
5. Database Layout for SAP Installations with DB2 UDB for UNIX and Windows Dr. Jens Claussen, SAP AG, Advanced Technology Group, February 2001,
http://service.sap.com/atg.
6. SSQJ Documentation Version 9.B SAP Advanced Technology Group,
http://service.sap.com/atg ,May 31, 2000.
7. DB2 Internals for Administrators: With V7 Updates May 16 & 18, 2000, Matt
Huras, IBM Toronto Labs.
8. Fundamentals
of
Database
Layout
http://service.sap.com/atg , March, 2000.
SAP
AG,
Version
1.0,
9. SAP R/3 Storage Management - Split Mirror Backup Recovery on IBMs

Enterprise Storage Server (ESS), Siegfried Schmidt, SAP AG, Advanced
or
Technology Group, February 2000,
http://service.sap.com/atg
http://www.storage.ibm.com/hardsoft/products/ess/whitepaper.htm
10. R/3 System: SAP R/3 Storage Management for OS390: Split Mirror Backup
Recovery on IBMs RVA Storage Controller Siegfried Schmidt, SAP AG,
Advanced Technology Group, http://service.sap.com/atg , September 1999 or
http://www.storage.ibm.com/ibmsan/whitepaper.htm
11. R/3 System: R/3 Storage Management for OS390 An Examination of
IBMs RAMAC Virtual Array Turbo 8-path Storage Controller Bruce McNutt,
Siegfried Schmidt, April 1999.
12. Implementing ESS Copy Services on UNIX and Windows NT/2000- IBM
Document Number: SG24-5757-00 ( http://www.redbooks.ibm.com/ )
Page 72

Db2 Sap Splitmirror

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Db2 Sap Splitmirror

Uploaded by

Copyright:

Available Formats

Storage Management for SAP and DB2 UDB: Split Mirror Backup / Recovery

with IBMs Enterprise Storage Server (ESS)

Storage Management for SAP and DB2 UDB:

Sanjoy Das, Siegfried Schmidt, Jens Claussen and BalaSanni Godavari

The following terms are trademarks of International Business Machines Corporation in

SAP Requirements ............................................................................................................................... 8

Split Mirror Backup & Recovery (SMBR) Setup ............................................................................... 16

Split Mirror Backup & Recovery Process ......................................................................................... 26

Process Automation - Solution Integration in Customer Environments ....................................... 42

and the remote synchronous copy function Peer-to-Peer-Remote-Copy (PPRC). The

intelligent fault tolerant storage subsystems are an important ingredient of high

Figure 1: SAP High Availability Advanced Infrastructure

The database and SAP Basis administration can be augmented by providing a

Backup & restore time is database size independent

Minimum impact on production environment

Physical and logical disaster prevention

Enhanced database administration

Remote data vaulting

Offloading backup from production database server

Reduction of backup-restore-time to minutes

4 DB2 UDB and ESS Features to Support the Split Mirror

DB2 UDB Features

FlashCopy - ESS's Advanced Local Copy Function

The RECREATEVG command overcomes the problem of duplicated LVM data

Peer-to-Peer-Remote Copy (PPRC) ESSs Advanced

PPRC is a hardware-assisted synchronous remote copy or synchronous mirroring

Figure 2: PPRC Connectivity using ESCON Links

Figure 3: PPRC Links between two ESS Storage Sub Systems

5 Split Mirror Backup & Recovery (SMBR) Setup

SSQJ: R/3 Load Simulation and Testing Tool

SAP / DB2 UDB File System Definition

DB2 UDB Backup Objects for SAP

DB2 software (specified during DB2

DB2 instance directory

DB2 binaries, TSM control files,

DB2 directories for containers

DB2 online log directory

DB2 offline log directory

Figure 4: UDB Backup Objects for SAP

DB2 UDB File System Definitions for SAP

File system Name

Size (in 8MB

Online Log files

Archive log files

SAP R/3 data files

SAP R/3 data files

SAP R/3 data files

SAP R/3 data files

SAP R/3 data files

SAP R/3 data files

For future data files

SSQJ data load

Figure 5: UDB File Systems for SAP

In Figure 5 above, the volume groups LSS10data, LSS12data, LSS14data and

database page size as outlined in Section H of the Appendix.

Split Mirror Infrastructure Setup

ESS LUN Definition

Figure 7: ESS Logical Sub Systems (LSS)

Volume Group Assignment

Volume Layout for DB2 UDB

Split Mirror Backup & Recovery Process

Figure 8: Split Mirror Backup / Recovery and Standby R/3 System

FlashCopy (FC) DBA to DBB (Safety Copy)

Withdraw all previously held FlashCopy NoCopy relationship