Professional Documents
Culture Documents
Database Replay
Surya Rao
Service Line Architect, IBM Senior Certified IT Specialist
IBM GTS, Service Delivery, Data Management
Table of contents
Abstract..................................................................................................................................... 1
Prerequisites............................................................................................................................. 1
1.0 Executive Summary............................................................................................................ 1
2.0 Introduction ........................................................................................................................ 2
2.1 Licensing Considerations ............................................................................................................. 3
Resources............................................................................................................................... 46
About the author..................................................................................................................... 46
Appendix................................................................................................................................. 47
1.0 Common Errors and Resolutions .................................................................................................47
2.0
Patch level for RAT......................................................................................................47
Abstract
This white paper describes Oracle Real Application Testing product functionality. This is a
tool that captures workload including timing, concurrency and dependency information and
is able to run this captured load into an alternate environment. Real application Testing is
two tools. It is SQL Performance Analyzer where you perform SQL unit testing to identify
SQL regression. Database replay is what is described in this document. The target audience
is installations running Oracle Database on IBM hardware and IBM field specialist that
support those environments.
Prerequisites
The reader of this document is expected to have a general and basic knowledge of the Oracle Database
and IBM servers.
Database upgrades
Database Patches
Storage changes
Network Changes
Hardware upgrades
OS upgrades
Database configuration changes
Scale up testing and consolidation
Database Replay
SQL Performance Analyzer (SPA)
SQL Tuning Sets (STS). This is also part of the Tuning Pack
SPA and STS are not in the scope of this white paper.
2.0 Introduction
Oracle Real Application Testing is a separately licensed option with Oracle
Database Enterprise Edition 11g. The purpose of this product is to allow
extraction of production workloads and simulate running them on preproduction or test databases to replicate a current performance issue for
deeper analysis and resolution on non production environments.
The Database Replay feature of the Real Application Testing suite can be
used to test any substantial changes such as Database upgrades, Storage or
network changes, Hardware or OS upgrades, Database configuration
changes etc.
It can also be used in Change Management related baselines and regression
testing scenarios where the behavior of the system has to be observed
Oracle Real Application Testing
http://www.ibm.com/support/techdocs
Copyright 2012, IBM Corporation
based on a fixed and unchanging workload to ensure that the results are
comparable across test runs.
Note that although this feature was introduced in Oracle Database 11g, it is
possible to capture workloads from 10.2.0.4 databases but it is possible to
replay them only on Oracle Database 11g and upwards. We can also capture
workloads from 9.2.0.8, 10.2.0.2 & 10.2.0.3 provided that the RAT capture
patch is installed. This could be an excellent avenue for testing pre and post
upgrade environments, an area that is not very easy to cover at the
moment, in the absence of Real Application Testing
The Oracle Real Application Testing option includes the following features:
Database Replay
SQL Performance Analyzer (SPA)
SQL Tuning Sets (STS)
1. For versions prior to 11g, i.e. 10.2.0.x, and 9.2.0.8, there are mandatory patches to
be applied before workloads can be captured. These patches (applied via opatch
utility) install the required Real Application Testing functionality in the Pre-11g
releases.
2. For a source 10g R2 database, the minimum version for which capture is possible is
10.2.0.2 for non-Windows operating systems and 10.2.0.3 for Windows operating
systems.
3. Note that the captured workloads can only be replayed on versions 11.1 and above.
4. The entire list of patches for each source (capture) and target (replay) version is
mentioned in the My Oracle Support article 560977.1
5. The following types of client requests are not captured in a workload as
per the Oracle Documentation
* Direct path load of data from external files using utilities such as
SQL*Loader
* Non-PL/SQL based Advanced Queuing (AQ)
* Flashback queries
* Oracle Call Interface (OCI) based object navigations
* Non SQL-based object access
* Distributed transactions (any distributed transactions that are captured
will be replayed as local transactions). Distributed transactions using two
phase commit are not supported and will not be replayed as even local
transactions.
2 oracle
3 oracle
oinstall
oinstall
The schema that is used to run the workload can belong to any user who has
the necessary privileges to perform the required DML, DDL against the
objects referred to in the workload SQL sets.
Instance_Number
User
Module
Program
Action
Service
10
To check the filters defined and their statuses i.e NEW or USED, use the
below step
11
To check for the captures that have been completed on this database or to query
the ones that are currently in progress, follow the step provided below
set lines 200
col status format a15
col name format a15
col dbversion format a12
SELECT ID, NAME, DBID,DBNAME,DBVERSION,
PARALLEL,STATUS, to_char(START_TIME,'dd/mm/yyyy hh24:mi'),
to_char(END_TIME,'dd/mm/yyyy hh24:mi'),DURATION_SECS
FROM DBA_WORKLOAD_CAPTURES;
12
BEGIN
DBMS_WORKLOAD_CAPTURE.FINISH_CAPTURE ();
END;
/
PL/SQL procedure successfully completed.
13
30038
11338
268
171
Oct
Oct
Oct
Oct
07
07
07
07
00:41
00:41
00:40
00:24
wcr_cr.html
wcr_cr.text
wcr_fcapture.wmd
wcr_scapture.wmd
07
07
07
07
07
07
07
07
07
07
07
00:40
00:24
00:24
00:24
00:24
00:24
00:24
00:24
00:24
00:24
00:24
aa
ab
ac
ad
ae
af
ag
ah
ai
aj
NOTE - Once a set of filters are defined and used by starting and finishing a
capture, these filters are marked as USED in dba_workload_filters and can't
be used again.
The same filter names can be redefined and will show up as NEW in the
dba_workload_filters view.
TYPE
ID STATUS NAME
ATTRIBUTE
VALUE
------------------------------------------------------------CAPTURE
1 USED
USER_SCOTT
USER
SCOTT
CAPTURE
1 USED
INSTANCE_FILTER
INSTANCE_NUMBER 1
CAPTURE
0 NEW
USER_SCOTT
USER
SCOTT
CAPTURE
0 NEW
INSTANCE_FILTER
INSTANCE_NUMBER 1
14
15
9.09
2/1,434 [ 0%]
0
5,
9.09
2/1,434 [ 0%]
21,
280
4.55
1/1,434 [ 0%]
24,
28
4.55
1/1,434 [ 0%]
58,
SYS
0
SYS
0
SYS
0
SYS
0
------------------------------------------------------------Workload Filters
DB: ORCL Snaps: 78
# Filter Name
Type
Attribute
Value
--- ------------------------ ------- ------------ -------------------------1 INSTANCE_FILTER
INCLUDE INSTANCE_NUM 1
2 USER_SCOTT
INCLUDE USER
SCOTT
-------------------------------------------------------------
16
The DBA_WORKLOAD_CAPTURES view lists all the workload captures that have been captured in
the current database.
The DBA_WORKLOAD_FILTERS view lists all workload filters for workload captures defined in the
current database.
The DBA_WORKLOAD_REPLAYS view lists all the workload replays that have been replayed in the
current database.
The DBA_WORKLOAD_REPLAY_DIVERGENCE view enables you to view information about
diverged calls, such as the replay identifier, stream identifier, and call counter.
The DBA_WORKLOAD_REPLAY_FILTER_SET view lists all workload filters for workload replays
defined in the current database.
The DBA_WORKLOAD_CONNECTION_MAP view lists the connection mapping information for
workload replay.
The V$WORKLOAD_REPLAY_THREAD view lists information about all sessions from the replay
clients.
The following Oracle supplied PL/SQL API packages contain the subprograms to completely manage and administer the Capture and Replay
processes
package
DBMS_WORKLOAD_REPLAY package
DBMS_WORKLOAD_CAPTURE
17
The preprocessing should always be done on a system which has the same
DB version as the source DB where the workload was captured. This step
converts the captured data into replay files and also creates the necessary
metadata. This step is required to be done only once (for each version of
the target system) after which it can repeatedly be replayed on a target
system.
Preprocess of captured work load is resource intensive hence this step
should be done on a test system separate from the Production system from
where the workload was captured. This system can be the same as the
replay system or a different one. For a RAC configuration, it is required to do
the pre-processing against any one node only.
Workload capture happen into 'c:\app\capture directory in previous section
while for pre-processing we are doing on 'c:\app\capture2 directory.
CREATE OR REPLACE DIRECTORY capture_dir AS 'c:\app\capture2';
Directory created.
BEGIN
DBMS_WORKLOAD_REPLAY.PROCESS_CAPTURE (capture_dir => 'CAPTURE_DIR');
END;
/
PL/SQL procedure successfully completed.
Pre-processed files are created on c:\app\capture2 directory. These preprocessed files will be used by the replay clients in constructing the workload
during the replay stage. The preprocessed files contain the metadata (i.e.
attributes) describing the workload
18
Ensure Oracle Home binary on database replay hosts and replay client
hosts, is at the correct patch level for Real Application Testing replay.
(see appendix ).
Setting up a Database Replay Administrator
The SYS or SYSTEM account is not required to be used for the capture and replay steps
described above. Any user that is granted the below privileges can be used to administer
Capture and Replay
19
20
After the DB has been reverted back to a clean slate as described above,
the workload can be replayed once again. This approach allows us to
make changes to the configuration, code or other variables and replay the
workload any number of times until we achieve a satisfactory
performance.
21
Password:
Workload Characteristics:
Errors
- max in
concurrency:
file :
1 sessions
total
number
of
sessions:
3
ORA-15554: cannot start workload
replay client because the database server
isAssumptions:
not in PREPARE mode
-
22
BEGIN
DBMS_WORKLOAD_REPLAY.INITIALIZE_REPLAY (replay_name => 'PEAK_REPLAY',
replay_dir => 'CAPTURE_DIR');
END;
/
PL/SQL procedure successfully completed.
23
For Oracle Real Application Clusters databases, you can map all connection
strings to a load balancing connection string. This is especially useful if the
number of nodes on the replay system is different from the capture system.
Alternatively, if you
want to direct workload to specific instances, you can use services or
explicitly specify the instance identifier in the remapped connection strings.
7.5 Prepare the Workload Replay and check status
BEGIN
DBMS_WORKLOAD_REPLAY.PREPARE_REPLAY (SYNCHRONIZATION=>TRUE,
--means transactions are replayed in SCN order
CONNECT_TIME_SCALE=>100,
--means same time between connections as in captured workload
THINK_TIME_SCALE=>100,
--means 1.5x the time between
subsequent calls
THINK_TIME_AUTO_CORRECT=>false,
SCALE_UP_MULTIPLIER=>1);
--means how many times to multiply the workload
by during replay
END;
/
PL/SQL procedure successfully completed.
24
Workload replay filters can be added if required similar to the filters defined
during the capture process. If no filters are defined, the entire captured
workload is replayed.
The following replay options can be specified with the
DBMS_WORKLOAD_REPLAY.PREPARE_REPLAY procedure during the
Prepare stage
synchronization
The below query displays one row for every replay client session that is
connected to the database and executing the captured workload.
25
All rights
Password:***********
Wait for the replay to start (19:58:14)
Note: Once a workload replay is started, new replay clients will not be able
to connect to the database. Only replay clients that were started before the
START_REPLAY procedure is executed will be used to replay the captured
workload.
26
Then, we will see the below appearing in the output of the terminals where
the wrc.exe was started
27
Completion Messages
Replay started (20:02:43)
Replay finished (20:04:50)
28
SET SERVEROUTPUT ON
DECLARE
cap_id
NUMBER;
rep_id
NUMBER;
rep_rpt
CLOB;
BEGIN
cap_id := DBMS_WORKLOAD_REPLAY.GET_REPLAY_INFO(dir =>
'CAPTURE_DIR');
/* Get the latest replay for that capture */
SELECT max(id)
INTO
rep_id
FROM
dba_workload_replays
WHERE capture_id = cap_id;
dbms_output.put_line('Max REP_ID is :'||rep_id);
end;
/
Max REP_ID is :1
PL/SQL procedure successfully completed.
29
30
.
.
.
.
.
.
.
.
.
.
exported
exported
exported
exported
exported
31
These exported AWR snapshots can then be imported into another system into a specified and
empty schema. The commands to export AWR reports during Capture and Replay are provided
below
EXEC DBMS_WORKLOAD_CAPTURE.EXPORT_AWR (capture_id => 12);
EXEC DBMS_WORKLOAD_REPLAY.EXPORT_AWR (replay_id => 13);
32
The DBA_WORKLOAD_CAPTURES view lists all the workload captures that have been captured in
the current database.
The DBA_WORKLOAD_FILTERS view lists all workload filters for workload captures defined in the
current database.
The DBA_WORKLOAD_REPLAYS view lists all the workload replays that have been replayed in the
current database.
The DBA_WORKLOAD_REPLAY_DIVERGENCE view enables you to view information about
diverged calls, such as the replay identifier, stream identifier, and call counter.
The DBA_WORKLOAD_REPLAY_FILTER_SET view lists all workload filters for workload replays
defined in the current database.
The DBA_WORKLOAD_CONNECTION_MAP view lists the connection mapping information for
workload replay.
The V$WORKLOAD_REPLAY_THREAD view lists information about all sessions from the replay
clients.
The following Oracle supplied PL/SQL API packages contain the subprograms to completely manage and administer the Capture and Replay
processes
package
DBMS_WORKLOAD_REPLAY package
DBMS_WORKLOAD_CAPTURE
33
Add
begin
DBMS_WORKLOAD_CAPTURE.DELETE_CAPTURE_INFO(<Capture ID>);
aend;
filter on the SCOTT schema
/
34
BEGIN
DBMS_WORKLOAD_CAPTURE.ADD_FILTER (
fname => 'user_scott',
fattribute => 'USER',
fvalue => 'SCOTT');
DBMS_WORKLOAD_CAPTURE.ADD_FILTER (
fname => 'instance_filter',
fattribute => 'INSTANCE_NUMBER',
fvalue => '1' );
END;
/
35
Create a capture directory (if non-existent) and start the capture process
Check the status of the capture process that has been started as
described in the steps below
36
Insert data into employee table to generate work load in the SCOTT
schema.
insert
Insert
insert
insert
into
into
into
into
employee
employee
employee
employee
values
values
values
values
(1,'Akash');
(2,'Kehul');
(3,'Milan');
(4,'Ajay');
hh24:mi'),
ID NAME
DBID DBNAME
DBVERSION
PAR STATUS
TO_CHAR(START_TI TO_CHAR(END_TIME DURATION_SECS
---------- --------------- ---------- ---------- ------------ --- -------------- ---------------- ---------------- ------------57 PEAK_PROCESSING_2 1292646392 ORCL
11.2.0.1.0
NO
COMPLETED
30/10/2011 05:24 30/10/2011 05:27
148
This concludes creation and capture of the workload which will be used to
simulate the Data Divergence in the subsequent steps
Oracle Real Application Testing
http://www.ibm.com/support/techdocs
Copyright 2012, IBM Corporation
37
BEGIN
DBMS_WORKLOAD_REPLAY.INITIALIZE_REPLAY (replay_name =>
'PEAK_REPLAY',
replay_dir => 'CAPTURE_DIR');
END;
/
PL/SQL procedure successfully completed.
38
BEGIN
DBMS_WORKLOAD_REPLAY.PREPARE_REPLAY (SYNCHRONIZATION=>TRUE,
--means transactions are replayed in SCN order
CONNECT_TIME_SCALE=>100,
--means same time between connections as in captured workload
THINK_TIME_SCALE=>100,
--means 1.5x the time between
subsequent calls
THINK_TIME_AUTO_CORRECT=>false,
SCALE_UP_MULTIPLIER=>1);
--means how many times to multiply the workload by
during replay
END;
/
PL/SQL procedure successfully completed.
ID
---------16
1
13
17
34
12
43
21
NAME
--------------PEAK_REPLAY
PEAK_REPLAY
PEAK_REPLAY
PEAK_REPLAY
PEAK_REPLAY
PEAK_REPLAY
PEAK_REPLAY
PEAK_REPLAY
STATUS
-------------------COMPLETED
CANCELLED
CANCELLED
COMPLETED
COMPLETED
COMPLETED
PREPARE
COMPLETED
reserved.
39
Password:***********
Wait for the replay to start (19:58:14)
All rights
BEGIN
DBMS_WORKLOAD_REPLAY.START_REPLAY ();
END;
/
Then, we will see the below appearing in the output of the terminals where
the wrc.exe was started
wrc system mode=replay replaydir=c:\app\capture
Workload Replay Client: Release 11.2.0.1.0 - Production on Sun Oct 30
11:00:18 2011
Copyright (c) 1982, 2009, Oracle and/or its affiliates.
reserved.
All rights
Password:
Wait for the replay to start (11:00:21)
Replay started (11:00:40)
To check what SQLs are being currently replayed by the wrc clients, the
following query can be executed.
40
41
To extract the details of the diverging SQL statement and its associated bind
variables, follow the steps described below. This can then be used to
ascertain if the exclusion of the diverging statements causes an impact to
the workload simulation. If it is important that the diverged statements must
be executed, then the data must be corrected on the target system prior to
re-running the workload replay
<replay_divergence_info>
<sql_id>786cwqgg8m5j7</sql_id>
<sql_text>
insert into employee values (1,'Akash')
</sql_text>
<full_sql_text>
insert into employee values (1,'Akash')
</full_sql_text>
<binds/>
</replay_divergence_info>
SQL>
42
The Database Replay screen contains the links for performing the tasks related to Capturing
Workloads, Preprocessing workloads and Replaying workloads.
43
In the preprocess workload screen, a list of available directory objects are displayed from which
the desired directory, containing the captured data, is to be chosen
The detailed steps for performing Capture and Replay using Enterprise Manager have
intentionally been excluded from this document as the steps and screens are self explanatory
44
Summary
This paper attempted to describe the best practices on deploying Oracle Real Application
Testing. This is a toolset/feature which captures an exact production workload including timing,
concurrency and dependency information and is able to run this captured workload into an
alternate environment. With this load testing, we were able to learn the impact of a prospective
change to a workload. The installation was done on an IBM POWER6 System.
Operating system
AIX 6.1 (6100-06-04-1112) is the operating system used in the tests described in this paper. This test was
pursued in PowerVM VIOS environment. VIOS version 2.2.0.10-FP-24 was used. Virtual SCSI disk
method was used to present the Storage LUNs to the AIX Logical partitions (LPAR).
45
Resources
These Web sites provide useful references to supplement the information contained in this document:
IBM Power Systems Information Center
o http://publib16.boulder.ibm.com/pseries/index.htm
IBM AIX Information Center
o http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp
IBM Publications Center
http://www-05.ibm.com/e-business/linkweb/publications/servlet/pbi.wss?CTY=US
.
IBM Redbooks
o http://www.redbooks.ibm.com/
My Oracle Support articles & references (https://support.oracle.com/CSP/ui/flash.html)
(please note: an userid and password are required to access the above website)
o Real Application Testing Now Available for Earlier Releases [ID 560977.1]
o Real Application Testing (RAT) API Setup and Verification [ID 1083063.1]
o Using the workload capture and replay in 11G [ID 445116.1]
o Database Replay Diagnostic information [ID 1287620.1]
o Database Capture and replay common errors and reasons [ID 463263.1
Oracle Documentation
http://docs.oracle.com/cd/E11882_01/server.112/e16540/rat_intro.htm#RATUG101
46
Appendix
1.0 Common Errors and Resolutions
Please refer to the article DATABASE CAPTURE AND REPLAY COMMON
ERRORS AND REASONS [ID 463263.1] for information on common issues
that could be faced
47
48
respect to any future products. Such commitments are only made in IBM product announcements. The
information is presented here to communicate IBM's current investment and development activities as a
good faith effort to help with our customers' future planning.
Performance is based on measurements and projections using standard IBM benchmarks in a controlled
environment. The actual throughput or performance that any user will experience will vary depending
upon considerations such as the amount of multiprogramming in the user's job stream, the I/O
configuration, the storage configuration, and the workload processed. Therefore, no assurance can be
given that an individual user will achieve throughput or performance improvements equivalent to the
ratios stated here.
Photographs shown are of engineering prototypes. Changes may be incorporated in production models.
Any references in this information to non-IBM Web sites are provided for convenience only and do not in
any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part
of the materials for this IBM product and use of those Web sites is at your own risk.
49