Maximizing Oracle eBusiness Performance

Maximizing the Performance of
the Oracle eBusiness Suite

Ahmed Alomari
Applications Performance Group
Oracle Corporation
June 14, 2005
Agenda
Architecture
Application Services Tier
Forms
Reports
Concurrent Manager
Apache
Mod PL/SQL
Jserv/JVM
Portal
Discoverer
Network Performance
Data Server Tier
Application Tuning
11i10 Performance Improvements
Q&A
Architecture
Portal
Servlet Engine
http
Web Listener
Java Server Pages (JSP)
Real Application
Clusters (RAC)
JDBC
(Thin)
AOLJ
Discoverer
PL/SQL Gateway
Clients
Reports
Forms
Concurrent Manager
Application Tier
TNS
(Net8)
Data Server Tier

Forms
Deploy with socket mode for internal users:

connectMode=socket (appsweb.cfg)
Enable Forms Dead Client Detection

Value specified in minutes
Terminates f60webmx processes for dead
clients.
FORMS60_TIMEOUT=10
Enable Forms Abnormal Termination Handler

Do not set FORMS60_CATCHTERM
Forms
Upgrade to the latest Forms patch set (patch set 16)

Forms patchset 7 (6.0.8.16.X) introduced a regression
which affects scalability (bug 2269913).
Results in Forms generating invalid SQL.
Fixed in Forms patchset 10 (6.0.8.19.X).
Refer to MetaLink document 125767.1 for Forms
patchset upgrades.
Forms Cancel Query
Should not be enabled unless you are on Forms patchset

14 (Forms version 6.0.8.23.x or higher).
Refer to MetaLink Note 138159.1 for instructions on how
to enable and tune Cancel query related parameters.
Cancel Query increases middle-tier CPU as well as DB
CPU.
To Disable Cancel Query
Set the Profile FND: Enable Cancel Query to No.

Forms
Ensure users are optimally utilizing the

professional Forms interfaces.
Avoid Blind queries
Provide selective criteria in Find windows and
LOVs.
Avoid opening and closing forms across
transactions.
Users which need to switch responsibility in
order to complete a business transaction
should instead keep the relevant Forms open.

Forms (Tracing)
If the form is slow, or a specific flow in the Form

is slow, first generate a complete SQL trace
using the Trace menu in Help-Diagnostics.
If the SQL trace does not account for the
majority of the response time experienced by
the user, then generate a Forms Runtime
Diagnostics (FRD) Trace.

Forms (Tracing SQL Trace)

Forms (Tracing FRD Trace)
Logon to Applications 11i with the following

parameters appended to the URL:
play=&record=collect&log=/tmp/oeform.trc
Navigate through the flows which you would like
to trace.
Exit from Oracle Applications completely.
Review the trace file from the application server
(/tmp/oeform.trc).

Reports
Ensure users provide selective parameters to

the standard Reports.
Review the Report output:
Number of rows returned to the report.
Number of report pages.
SQL Trace can be generated at Request

Submission Time (SRS form Debug Options)
or by setting the Enable Trace flag in the
Concurrent Program Definition Form.
Reports (Tracing)
If the SQL trace for the report does not account for the
elapsed time of the request, then generate a Reports trace
by appending tracing parameters the to concurrent
program command line.
The following are the trace options:

TRACE_ALL : log all possible trace information in the trace file. (DEFAULT)
TRACE_APP : log trace information on all the report objects in the trace file.
TRACE_BRK : list breakpoints in the trace file.
TRACE_ERR : list error messages and warnings in the trace file.
TRACE_PLS : log trace information on all the PL/SQL objects in the trace file.
TRACE_PRF : log performance statistics in the trace file.
TRACE_SQL : log trace information on all the SQL in the trace file.
TRACE_TMS : enter a timestamp for each entry in the trace file.
Reports (Tracing)
Enabling Reports Trace (pre 11i10)

Sign-on to Apps with System Administrator responsibility
Navigate to Concurrent -> Program -> Define
Query the concurrent program (report) which you want to trace.
Locate the Options field and enter the following values :
TRACEOPTS=<trace option> TRACEFILE=<path/file>
Save the concurrent program
Run the report and a trace file will be generated.
Remember to clear the trace options after running the report.
Enabling Reports Trace (11i10)

Use the Debug Options page at request submission.
Reports (Trace Output)
Report:/u02/appl/fnd/11.5.0/reports/US/FNDSCURS.rdf
Loggedontoserver:
Username:
........
23:59:57APP...(GenericGraphicalObjectB_SECURITY_GROUP_NAME1
23:59:57APP...)GenericGraphicalObjectB_SECURITY_GROUP_NAME1
23:59:57APP...(GenericGraphicalObjectB_USER_NAME1
........
++
|ReportBuilderProfilerstatistics|
++
TOTALELAPSEDTime:2243.04seconds
ReportsTime:1196.62seconds(53.34%ofTOTAL)
ORACLETime:1046.41seconds(46.65%ofTOTAL)
UPI:1046.00seconds
SQL:0.40seconds
TOTALCPUTimeusedbyprocess:904.71seconds

Concurrent Manager
Avoid enabling an excessive number of standard

or specialized managers.
Use specialization rules and work shifts to bind
specific jobs to specific time windows.
Helps avoid scheduling resource intensive batch
requests during peak activity.
For jobs which spawn parallel workers such as

Auto Invoice or Payroll, set the sleep time of the
Conflict Resolution Manager (CRM) to null (i.e.
10 seconds). The default value is 60 seconds.

Concurrent Manager
Database Resource Manager Integration

Define Resource Consumer Groups and
associate specific concurrent programs with a
resource group.
Allows an administrator to constrain the amount
of CPU resources used by a concurrent program.
Resource Groups can also be defined for online
users using the profile FND: Resource
Consumer Group.
Concurrent Manager
Transaction Managers
Used for Inventory Transactions as well as other synchronous

online processing.
Ensure enough transaction managers exist to service the
request load.
Set the profile Concurrent:Wait for Available TM to 1 (second).
Set TP:INV Transaction processing mode to On-line
processing for small inventory requests from the UI.
Set the sleep time on the transaction managers to a high
number (e.g. 10 minutes).
Avoids constant polls to check for shutdown requests.
Transaction Managers can also be traced.

Concurrent Manager
Enabling Tracing
Program Level
Enable the trace option in the concurrent program
definition form.
Request Level (new in 11i10)
Use the Debug Options button in the Request
Submission Form
The profile Concurrent: Allow Debugging
should be set to Yes.

Concurrent Manager
Enabling Trace at the Program Level

Concurrent Manager
Enabling Trace at the Request Level

Concurrent Manager
Enabling Trace at the Request Level
Apache
On UNIX, Apache is process based (httpd), and mods

such as mod PL/SQL run within the process address
space of the httpd processes.
On Windows, Apache is multi-threaded (Apache.exe).
Tune the number of processes and number of clients
(httpds.conf):
MinSpareServers 5
MaxSpareServers 10
StartServers 5
MaxClients 512
Apache
Minimize levels of logging (httpds.conf):

LogLevel warn
SSLLogLevel warn
Enable Caching of non-HTML resources including images,
style sheets, and Java script.
Caches content in the browser cache.
Reduces network round-trips (non SSL)
Reduces network bandwidth utilization (SSL)
Included in AutoConfig templates in 11.5.8 (or higher).

Apache
Enable Caching of non-HTML resources (httpd.conf

or apps.conf)
<Directory "<physical_path_corresponding_to_the_alias_/OA_HTML/>">
#enable the generation of the Expires header for files under /OA_HTML/
ExpiresActive On
#expire images one month after last client access
ExpiresByType image/gif "access plus 1 month"
#expire stylesheets one week after the last client access
ExpiresByType text/css "access plus 1 weeks"
#expire javascript libraries one day after the last client access
ExpiresByType text/javascript "access plus 1 days
ExpiresByType application/x-javascript "access plus 1 day"
</Directory>

Apache
Review Apache Access Log file to ensure images

are being satisfied from the browser cache.
Http code 200 (request for document)

Http code 304 (request for time stamp)
130.35.127.106 - - [23/Oct/2002:19:00:21 -0700] "GET /OA_MEDIA/FNDINVDT.gif HTTP/1.1" 200 821

130.35.127.106 - - [25/Oct/2002:14:38:23 -0700] "GET /OA_MEDIA/FNDINVDT.gif HTTP/1.1" 304 130.35.127.106 - - [25/Oct/2002:14:38:23 -0700] "GET /OA_MEDIA/FNDWATHS.gif HTTP/1.1" 200 190
130.35.127.106 - - [25/Oct/2002:14:38:23 -0700] "GET /OA_MEDIA/FNDREDPT.gif HTTP/1.1" 200 70
130.35.127.106 - - [25/Oct/2002:14:39:23 -0700] "GET /OA_HTML/OA.jsp?
page=/oracle/apps/icx/por/rcv/pages/ReceivingHomePage&OAHP=ICXP
OR_MENU&OASF=ICXPOR_RCV_HOME_PAGE&dbc=ap107fam_ipdev11i&language_code=US&transactionid=70A5819F04C0F411
HTTP/1.1" 200 14
130.35.127.106 - - [25/Oct/2002:14:39:34 -0700] "GET /servlets/PoolMonitor?jvm HTTP/1.1" 200 7954
130.35.127.106 - - [25/Oct/2002:14:41:03 -0700] "GET /OA_HTML/US/ICXINDEX_ipdev11i.htm HTTP/1.1" 304 130.35.127.106 - - [25/Oct/2002:14:41:03 -0700] "GET /OA_MEDIA/logo.gif HTTP/1.1" 304 130.35.127.106 - - [25/Oct/2002:14:41:03 -0700] "GET /OA_MEDIA/appslogo.gif HTTP/1.1" 304 130.35.127.106 - - [25/Oct/2002:14:41:03 -0700] "GET /OA_MEDIA/FNDJLFRL.gif HTTP/1.1" 304 130.35.127.106 - - [25/Oct/2002:14:41:03 -0700] "GET /OA_MEDIA/FNDINVDT.gif HTTP/1.1" 304 130.35.127.106 - - [25/Oct/2002:14:41:03 -0700] "GET /OA_MEDIA/FNDJLFRR.gif HTTP/1.1" 304 -

Apache Mod PL/SQL
Configure a dedicated mod PL/SQL Listener
Improves performance and scalability

Significantly reduces overall number of
sessions/connections.
Reduces latency of web requests.
Improves cursor sharing.
Documented in Oracle 9i Application Server Using the
PL/SQL Gateway Release 1 (v1.0.2.2)
http://technet.oracle.com/docs/products/ias/doc_library/102
2doc_otn/apps.102/a90099/apptroub.htm#634180

Apache Mod PL/SQL (dedicated listener)
1. For the main Listener running on Port 7000, edit the file
$IAS_HOME/Apache/modplsql/cfg/plsql.conf as follows:
Disable the mod PL/SQL service from the main listener by commenting out the
lines between the two Location parameters as follows:
# <Location /pls>
#
SetHandler pls_handler
Order deny,allow
Allow from all
# </Location>
Comment out the following line as follows:
# LoadModule plsql_module /d1/ias/Apache/modplsql/bin/modplsql.so

Apache Mod PL/SQL (dedicated listener)
2. Configure the main listener to forward all mod_plsql requests to the
dedicated mod PL/SQL listener by adding the following line:
ProxyPass /pls/ http://sechost.us.oracle.com:8888/pls/
For the dedicated mod PL/SQL Listener running on Port 8888, configure each
DAD to override the default CGI environment variables in order to allow
redirects. Edit the file $IAS_HOME/Apache/modplsql/cfg/wdbsvr.app and add
the following line for each DAD:
cgi_env_list=SERVER_NAME=mainhost.us.oracle.com,SERVER_PORT=7000,HOST=mainho
st.us.oracle.com:7000
Apache Jserv / JVM
Minimize Jserv logging

log=true
log.channel.warning=true
log.file=/d1/ias/Apache/Jserv/logs/jserv_7000.log
Review error log files for exceptions.
Disable auto reload in production environments:
autoreload.classes=false
autoreload.file=false
Use Jserv Auto Load Balancing
Configure Multiple Zones.
Provides higher availability and improves scalability.

Apache Jserv / JVM
Upgrade to the latest JDK (1.4.2_08).

Do not disable hotspot or the JIT (i.e.
DCOMPILER=NONE or Xint).
Enable verbose GC in order to optimally tune the
heap sizes based on the GC traffic.
Minor Collections
Major Collections
Elapsed Time of GC

Apache Jserv / JVM
Enable Verbose GC Output

Overload wrapper.bin with a shell wrapper in order
to redirect stdout to a file.
Existing:
wrapper.bin=/apps/jdk/jdk1.4.2/bin/java
Change To:
Wrapper.bin=/apps/scripts/java.sh
java.sh:
=========
/apps/jdk/jdk1.4.2/bin/java -verbosegc $* >> /apps/logs/java.log

Apache Jserv / JVM (Verbose GC Output)
/apps/logs/java.log:
[GC 30460K->1369K(510848K), 0.1135695 secs]
[Full GC 15135K->1686K(510848K), 0.2700469 secs]
[GC 32123K->2131K(510848K), 0.0203634 secs]
[GC 32595K->2130K(510848K), 0.0113639 secs]
[GC 32593K->2171K(510848K), 0.0129179 secs]
[GC 32635K->2419K(510848K), 0.0567306 secs]
[GC 32881K->3157K(510848K), 0.2906981 secs]
[GC 33620K->3197K(510848K), 0.0320023 secs]
[GC 33661K->3218K(510848K), 0.0103013 secs]
[GC 33674K->3309K(510848K), 0.0487887 secs]
[GC 33769K->3532K(510848K), 0.0531514 secs]
[GC 33983K->3784K(510848K), 0.0552549 secs]
[GC 34248K->4056K(510848K), 0.0624969 secs]
[GC 34520K->4404K(510848K), 0.0555575 secs]
[GC 34868K->4828K(510848K), 0.0587044 secs]
[GC 35292K->5242K(510848K), 0.0945290 secs]

Apache Jserv / JVM
Utilize the verbose GC output to tune the JVM

heaps (-Xmx and Xms) accordingly.
Review the frequency of collections, especially
major collections (i.e. Full GC).
Start with:
-Xms256M and Xmx512M
-XX:NewSize=60M -XX:MaxNewSize=120M
Above settings have been incorporated in 11i10.

Apache Jserv / JVM
Process Identification (V$SESSION)

Specify the property DCLIENT_PROCESSID in the
JVM startup shell script.
Set automatically in 11i10.
Existing:
wrapper.bin=/apps/jdk1.4.2/bin/java
Change To:
Wrapper.bin=/apps/scripts/java.sh
java.sh:
=========
#!/bin/sh
/apps/jdk1.4.2/bin/java -verbosegc -DCLIENT_PROCESSID=$$ $* >> /d2/logs/java.log

Apache Jserv / JVM
Process Identification (V$SESSION)

Allows you to map the JDBC session from
v$session to a particular JVM process.
Set automatically in 11i10
SID MACHINE
PROCESS
MODULE
LOGON
---- ------------------------- ------------ ------------------ ----------------41 aptier1.us.oracle.com
28806
JDBC Thin Client
12/02/04 14:26:43
42 aptier2.us.oracle.com
1723
JDBC Thin Client
12/02/04 14:27:01
3201
JDBC Thin Client
12/02/04 14:28:15
28807
JDBC Thin Client
12/02/04 14:29:17
. . . . . . .
. . . . . . .
aptier1{apps_a}-> ps -ef | grep 28806
apps_a 28806 28561 0 14:26:39 pts/20
apps_a 28807 28806 53 14:26:40 pts/20
0:00 /bin/sh ./java.sh

8:55 /jdk1.4.2/bin/../bin/sparc/native_threads/java

Apache Jserv / JVM
JDK 1.4.2 is Certified with Apps.

Refer to MetaLink note 246105.1 for details on
upgrading to the JDK 1.4.2.
~15% performance improvement.
Parallel GC (-XX:+UseParNewGC)
Detail GC Statistics (-XX:+PrintGCDetails)
[GC [DefNew: 2047K->0K(2112K), 0.0012660 secs] 2472K->424K(3520K), 0.0014699 secs]

[GC [DefNew: 0K->0K(2112K), 0.0018657 secs][Tenured: 20227K->227K(21416K), 0.1107343
secs] 20227K->227K(23528K), 0.1387975 secs]
secs] 20227K->227K(23528K), 0.1384847 secs]
secs] 20227K->227K(23528K), 0.1386478 secs]

Apache Jserv / JVM
JDK 1.4 provides new JVM monitoring tool (jvmstat)

Download from
http://developers.sun.com/dev/coolstuff/jvmstat/
Java Server Pages (JSPs)
Precompile the JSPs to avoid dynamic compilation.

Users experience poor performance for the initial page loads.
Potential deadlocks if multiple users attempt to compile the
same JSP.
Potential JVM death due to OutOfMemoryException during
concurrent compilation.
MetaLink Document 215268.1 provides the instructions and the
patch reference for an automated script to perform the
precompilation.
ojspCompile Script
With the latest version of ojspCompile, it takes 15 minutes to
compile all the JSPs.

Java Server Pages (JSPs)
Use a separate JVM to perform the JSP

compilation.
Automatically set in 11i10.
root.properties:
servlet.oracle.jsp.JspServlet.initArgs=translate_params=true,
unsafe_reload=false,
page_repository_root=/appl_top/115/common/html/jsp/pagecache,
alias_translation=true,developer_mode=false,
javaccmd=/usr/jdk142/bin/javac,send_error=true

Portal Tuning
Tune the number of content fetcher threads for

PPE (poolSize). Default is 25.
Adjust the fetch request timeout (requesttime).
Adjust the fetch connection request timeout (stall).
zone.properties:
servlet.page.initArgs=poolSize=50
servlet.page.initArgs=requesttime=200
servlet.page.initArgs=stall=100

Portal Tuning
Ensure that the mod PL/SQL cache is enabled and

sized appropriately:
$APACHE_TOP/modplsql/cfg/cache.cfg:
[PLSQL Cache]
enabled=yes
total_size=100000000
cleanup_size=75000000
cleanup_interval=86400
;
[Cookie Cache]
enabled=yes
total_size=25000000
cleanup_size=15000000
cleanup_interval=86400
max_size = 0
Discoverer 4i Viewer Tuning
Disable Query Prediction by setting QPPEnable=0 in pref.txt.

Set ObjectsAlwaysAccessible=1 to avoid extra workbook
SQL validation, and related dictionary SQL.
Set the Query Governor option Limit retrieved query data to:
to 100 rows.
Set the option After opening a worksheet: to Dont run
query.
Ensure Custom workbooks define mandatory parameters.
Network Performance
Network performance is often neglected.
Networks with poor latency or low bandwidth
impacts user response time.
Can also impact concurrent programs if the
network between the application tier and data
server tier is not optimal.
A direct switch should be used between the
application tier and data server tier
Latency should ideally be ~1-2ms per round-trip.
Network Performance
Use the ping command to determine the latency of requests. Vary

the packet size to determine the average latency. Do not use the
default ping packet size of 32 bytes.
D:\>ping aptierhost.us.oracle.com -l 2048

Pinging aptierhost.us.oracle.com [139.185.128.27] with 2048 bytes of data:
Reply
Reply
Reply
Reply
from
from
from
from
139.185.128.27:
139.185.128.27:
139.185.128.27:
139.185.128.27:
bytes=2048
bytes=2048
bytes=2048
bytes=2048
time=371ms
time=330ms
time=361ms
time=360ms
TTL=252
TTL=252
TTL=252
TTL=252
D:\>ping aptierhost.us.oracle.com -l 512

Pinging aptierhost.us.oracle.com [139.185.128.27] with 512 bytes of data:
Reply
Reply
Reply
Reply
from
from
from
from
139.185.128.27:
139.185.128.27:
139.185.128.27:
139.185.128.27:
bytes=512
bytes=512
bytes=512
bytes=512
time=231ms
time=210ms
time=231ms
time=220ms
TTL=252
TTL=252
TTL=252
TTL=252
Network Performance
Use the ping command to determine the latency between the

application server and the data server. Use a packet size of 2K
since this the default size for SQL*Net traffic.
aptierhost{applmgr}-2-> ping -s dbtierhost 2048

PING
2056
2056
2056
2056
2056
2056
dbtierhost.us.oracle.com: 2048 data

bytes from dbtierhost.us.oracle.com
bytes
(144.25.76.250):
(144.25.76.250):
(144.25.76.250):
(144.25.76.250):
(144.25.76.250):
(144.25.76.250):
---- dbtierhost.us.oracle.com PING Statistics---6 packets transmitted, 6 packets received, 0% packet loss
round-trip (ms) min/avg/max = 1/1/1
icmp_seq=0.
icmp_seq=1.
icmp_seq=2.
icmp_seq=3.
icmp_seq=4.
icmp_seq=5.
time=1.
time=1.
time=1.
time=1.
time=1.
time=1.
ms
ms
ms
ms
ms
Ms
Network Performance
Measure the sustained bandwidth on both the
client network (i.e. desktops) as well as the
application-tiers.
Use traceroute (or tracert) utility to determine
number of hops and impact on latency.
Monitor network statistics including packet
rates, error rates, collisions, etc.. using tools
such as netstat i.
Data Server Tier
Refer to the MetaLink document 216205.1 Database Initialization

Parameters and Configuration for Oracle Applications 11i.
Optimally configure the buffer cache and shared pool as per the
workload and the number of users:
Ensure mandatory parameters are set correctly.
[typical configuration for 2,000 Apps users]
db_block_buffers=800,000
shared_pool_size=2000M
A poorly sized buffer cache results in excessive buffer gets and
physical I/O.
A poorly sized shared pool results in library cache and shared pool
latch contention due to reloads and lack of space.
Data Server Tier
Enable asynchronous I/O

Consider using Quick I/O or equivalent if using file system
based files for the DB files.
Quick I/O improves performance and scalability by simulating raw

devices, and avoiding file system cache traffic for the DB files.
Use locally managed temp files (uniform) for the

temporary tablespace.
Install Stats Pack (spcreate.sql) and use spauto.sql to
automate hourly snapshots.
Stats Pack reports can be used to trend transaction rates
and business flows.
Data Server Tier

Stats Pack Report (Instance Summary)
STATSPACK report for
DB Name
DB Id
Instance
Inst Num Release
Cluster Host
------------ ----------- ------------ -------- ----------- ------- -----------GSIAP
317772662 gsi1ap
1 9.2.0.5.0
YES
agsidbs1
Snap Id
Snap Time
Sessions Curs/Sess Comment
------- ------------------ -------- --------- ------------------Begin Snap: 503400 18-May-05 11:00:01 #######
.0
End Snap: 503405 18-May-05 16:00:04 #######
.0
Elapsed:
300.05 (mins)
Cache Sizes (end)
~~~~~~~~~~~~~~~~~
Buffer Cache:
Shared Pool Size:
. . . . . . . . . . . .
5,313M
2,864M
Std Block Size:

Log Buffer:
8K
1,024K
Data Server Tier

Load Profile
~~~~~~~~~~~~
Redo size:
Logical reads:
Block changes:
Physical reads:
Physical writes:
User calls:
Parses:
Hard parses:
Sorts:
Logons:
Executes:
Transactions:
% Blocks changed per Read:
Rollback per transaction %:
. . . . . . . . . . . .
Per Second
--------------462,767.56
54,537.81
3,237.12
2,403.22
272.19
518.96
223.29
1.26
199.15
1.85
1,566.48
17.62
5.94
3.18
Recursive Call %:
Rows per Sort:
Per Transaction
--------------26,266.16
3,095.50
183.74
136.40
15.45
29.46
12.67
0.07
11.30
0.11
88.91
88.46
102.78
Data Server Tier

Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %:
99.90
Redo NoWait %:
Buffer Hit
%:
96.03
In-memory Sort %:
Library Hit
%:
99.87
Soft Parse %:
Execute to Parse %:
85.75
Latch Hit %:
Parse CPU to Parse Elapsd %:
69.57
% Non-Parse CPU:
Shared Pool Statistics
Memory Usage %:
% SQL with executions>1:
% Memory for SQL w/exec>1:
Begin
-----90.55
66.70
60.86
100.00
99.96
99.44
99.82
97.34
End
-----76.14
75.37
71.08
Top 5 Timed Events

~~~~~~~~~~~~~~~~~~
% Total
Event
Waits
Time (s) Ela Time
-------------------------------------------- ------------ ----------- -------db file sequential read
16,187,270
134,274
31.12
CPU time
110,764
25.67
global cache cr request
21,410,811
66,129
15.33
db file scattered read
3,184,765
23,571
5.46
KJC: Wait for msg sends to complete
10,722,532
20,446
4.74
Data Server Tier

Stats Pack Report (Wait Events)
Total Wait
Event
Waits
Timeouts
Time (s)
---------------------------- ------------ ---------- ---------db file sequential read
16,187,270
0
134,274
global cache cr request
21,410,811
56,241
66,129
db file scattered read
3,184,765
0
23,571
KJC: Wait for msg sends to c
10,722,532
194,069
20,446
row cache lock
74,583
5,842
17,796
wait for unread message on b
73,977
4,624
13,054
SQL*Net more data to client
6,021,483
0
8,873
enqueue
114,204
19,898
7,427
buffer busy global CR
516,382
18
4,735
buffer busy waits
434,054
192
3,857
direct path read
1,239,648
0
3,226
inactive session
1,774
1,774
1,784
latch free
242,171
19,203
1,694
db file parallel read
121,670
0
1,280
direct path write
1,270,008
0
1,110
log file sync
604,002
244
1,097
SQL*Net message from dblink
20,365
0
1,080
Avg
wait
Waits
(ms)
/txn
------ -------8
51.0
3
67.5
7
10.0
2
33.8
239
0.2
176
0.2
1
19.0
65
0.4
9
1.6
9
1.4
3
3.9
1006
0.0
7
0.8
11
0.4
1
4.0
2
1.9
53
0.1
Data Server Tier
Review Top 5 Timed Events section in the Stats Pack

reports to determine the overall instance utilization.
For latch free waits, latch contention is often a symptom
due to a legitimate problem such as non-sharable SQL,
sub-optimal SQL which performs full table or full index
scans, dynamic object creation/removal, etc..
Review the latch Statistics section of the Stats Pack report to

determine the hot latches.
Enable tracing for a few of the sessions waiting on the latch as
well as the holder to determine the actual cause.
Data Server Tier

Stats Pack Report (Expensive SQL)
CPU
Elapsd
Physical Reads Executions Reads per Exec %Total Time (s) Time (s) Hash Value
--------------- ------------ -------------- ------ -------- --------- ---------3,413,015
1
3,413,015.0
7.9 1970.29
6716.03 976039043
Module: ALECDC
SELECT distinct
ooha.order_number , fu.user_name , rac.cu
stomer_name , to_char(wctc.last_update_date, 'DD-MON-YY hh24:mi
') , DECODE(SUBSTR(wctc.comments, 1, 21),
'NOTE: This Order-Line', wctc.process_flag || ': manually scre
ened',
DECODE(wctc.process_flag
,'101','101: host u
2,229,632
1
2,229,632.0
5.2 2115.49 35448.22 1280227044
Module: ALECDC
select distinct
pv.segment1 ,
nvl(pv.vendor_name_a
lt, pv.vendor_name) ,
ai.invoice_num ,
hr.name ,
'ap_gsiap_rpts_us@oracle.com' from
po_vendors pv ,
ap_invoices_all ai ,
ap_holds_all ah ,
hr_organi
zation_units hr where
ai.payment_status_flag||''='Y' and
1,990,002
1
1,990,002.0
4.6 2321.34
Module: OKILOADDR
BEGIN OKI_REFRESH_PUB.REFRESH_ADDRS( :errbuf, :rc ); END;
23934.41 4063073269
Data Server Tier

Monitor CPU and Memory Utilization
Monitor paging
Monitor I/O statistics, including service times.
Review Stats Pack reports and correlate top
SQL to the business flows.
Review expensive custom SQL
Data Server Tier

Gathering Statistics
Do not gather statistics excessively on entire

schemas or the entire database such as nightly or
weekly.
Do not gather statistics on permanent objects
during peak intervals.
Gathering statistics invalidates cursors
Gathering statistics requires dictionary and object
level locks.
Plans are not likely to change if the data distribution
has not changed.
Data Server Tier

Gathering Statistics
For tables which are growing at a rapid rate, gather

statistics only on those tables.
Use only FND_STATS or the Gather Schema and
Gather Table Statistics Concurrent Programs
Do NOT USE the analyze nor dbms_stats command
directly. It is not supported, and results in suboptimal plans.
Review the table and index statistics for the objects

which appear in the top SQL section of Stats Pack.
Data Server Tier
Gathering Statistics Enhancements (11i10)
Gather Schema Statistics

Auto Gather option
Gather Statistics only on tables which have changed
Change threshold is user definable.
Utilizes the Table Monitoring feature. To enable schema
monitoring:
SQL> exec fnd_stats.ENABLE_SCHEMA_MONITORING()
Auto List option

Lists the objects which have changed.
Maintain history of Statistics Collection
No Invalidate Option
Does not invalidate cursors.
Data Server Tier

Gathering Statistics Enhancements (11i10)
Data Server Tier

Verifying the Statistics
SQL> set serveroutput on
SQL> exec apps.fnd_stats.verify_stats ('ONT','OE_ORDER_LINES_ALL');
===================================================================================================
Table
OE_ORDER_LINES_ALL
===================================================================================================
last analyzed
sample_size num_rows
blocks
12-03-2004 22:59 3726829
37268290
3527197
Index name
last analyzed
num_rows LB
DK
LB/key DB/key CF
---------------------------------------------------------------------------------------------------OE_ORDER_LINES_N1
12-03-2004 22:12 36018080 87310
2097282 1
5
11719150
OE_ORDER_LINES_N10
12-03-2004 22:12 26519270 68610
2230418 1
4
10949180
OE_ORDER_LINES_N11
12-03-2004 22:12 16310880 96200
11196380 1
1
7668050
OE_ORDER_LINES_N12
12-03-2004 22:12 50
1
20
1
1
19
OE_ORDER_LINES_N13
12-03-2004 22:12 1363149
3132
138681
1
3
547938
. . .
. . .
OE_ORDER_LINES_N5
12-03-2004 22:12 0
0
0
0
0
0
OE_ORDER_LINES_N6
12-03-2004 22:12 158583
410
158670
1
1
85032
OE_ORDER_LINES_N7
12-03-2004 22:12 103459
238
18873
1
2
48075
OE_ORDER_LINES_N9
12-03-2004 22:12 4276030
11060
3101971 1
1
2444110
OE_ORDER_LINES_U1
12-03-2004 22:12 37510470 91790
37510470 1
1
17574490
---------------------------------------------------------------------------------------------------Histogram Stats
Schema
Table Name
Status
last analyzed
Column Name
---------------------------------------------------------------------------------------------------ONT
OE_ORDER_LINES_ALL
present
03-12-2004 22:59 OPEN_FLAG
Data Server Tier

Verifying the Statistics Column Statistics
SQL> set serveroutput on
SQL> exec apps.fnd_stats.verify_stats ('ONT','OE_ORDER_LINES_ALL',0,TRUE);
===================================================================================================
Table
OE_ORDER_LINES_ALL
===================================================================================================
last analyzed
sample_size num_rows
blocks
12-03-2004 22:59 3726829
37268290
3527197
---------------------------------------------------------------------------------------------------Column Stats
Column name
sample_size num_distinct num_nulls
density
last analyzed
---------------------------------------------------------------------------------------------------ACCOUNTING_RULE_DURATION
0
37219060
0
12-03-2004 22:51
ACCOUNTING_RULE_ID
1882057
72
18398490
.013888888 12-03-2004 22:34
. . .
BOOKED_FLAG
3724692
2
0
.5
12-03-2004 22:51
. . .
CANCELLED_FLAG
3724692
2
0
.5
12-03-2004 22:51
. . .
LINE_ID
3721906
37219060
0
.000000026 12-03-2004 22:34
LINE_NUMBER
3721906
5649
0
.000177022 12-03-2004 22:34
LINE_SET_ID
10420
18873
37114860
.000052985 12-03-2004 22:34
OPTION_NUMBER
1350665
1176
23740270
.00085034
12-03-2004 22:51
ORDERED_ITEM_ID
3210194
50265
5117120
.000019894 12-03-2004 22:34
UPGRADED_FLAG
3125945
2
5987470
.5
12-03-2004 22:51
VEH_CUS_ITEM_CUM_KEY_ID
0
37219060
0
12-03-2004 22:34
VISIBLE_DEMAND_FLAG
668876
2
30558160
.5
12-03-2004 22:51
Data Server Tier
Enable the Auto memory manager
Enable System Managed Undo
No more ORA-1555s!!!
Set undo_retention to the length of the longest running request.
Utilize Auto Segment Tablespace Management
Pga_aggregate_target can be set to tune the PGA for the entire instance.
Automatically tunes hash area and sort area sizes
Returns unused memory to the OS.
No need to manually set Freelist/freelist groups. A bitmap is used to

automatically scale the segment related parameters.
Enable PL/SQL Native Compilation
Improves PL/SQL execution performance.

Improves scalability by removing a lot of stress off of the shared pool.
Data Server Tier
New Tablespace Model for the eBusiness Suite:
Consolidates the number of Apps tablespaces from 400

tablespaces to 10.
Optimizes performance via the wide stripe model.
Ideal for customers migrating to RAC or migrating to raw
devices.
Uniform extents, locally managed
No more fragmentation!
Migration script can be used to migrate a schema at a time.
Data Server Tier
New Tablespace Model for the eBusiness Suite:
Objects are classified by object type and access patterns:
Transaction tables
Transaction indexes
Reference/SEED
Interface
Summary Management
Archive
NoLogging
Queue
Media
Temporary
Undo
System
Total = 12 Tablespaces
Data Server Tier
Applications Tablespace Migration Utility
MetaLink note 248857.1

ARU # is 3381489
Migration timings: (complete migration)
1.4 Terabyte customer DB

24 hours
New size of DB after migration (850 GB).
250 GB customer DB
4 hours
New size of DB after migration (150 GB).
Application Tuning
Review the MetaLink note Recommended
Performance Patches for the Oracle EBusiness Suite, 244040.1.
Recommended performance patches for all the

modules and tech. stack components are
consolidated in this note.
Application Tuning
Workflow
When starting background engines via Concurrent Manager,

set the Process Stuck parameter to No.
Start a separate background engine to handle stuck
processes with a very low frequency such as once or twice a
day.
Use deferred activities to improve online response times and
facilitate asynchronous processing for flows such as
Scheduling, PO Document Approval, etc..
Establish a regular frequency for purging Workflow runtime
data (FNDWFPR - Purge Obsolete Workflow Runtime Data).
Application Tuning
Order Management
OM Pack J contains numerous performance fixes including Pricing

performance fixes.
Ensure that debugging is disabled:
OM: Debug Level, set to 0 for OFF
QP: Debug Mode, set to Request Viewer Off
WSH: Debug Enabled, set to No
WSH: Debug Level, set to 0 for OFF
Use the streamlined workflows.
Removes unnecessary activities and sub-processes.
Improves performance and amount of WF data significantly.
New streamlined flow in 11.5.10
Line Flow Generic: Performance
Refer to MetaLink note 130511.1
Application Tuning
Order Management
For online order entry, use the new Quick Sales Order Form
Provides a checkbox deferred pricing which can be
used to defer price calculation until the order is saved.
Also improves overall speed of order entry from a
functional / navigational perspective.
Enable AutoScheduling
Use deferred activities
Tax Calculation
Credit Check
Application Tuning
Payroll
Tune the number of threads, chunk size and the buffer sizes
(pay_action_parameters):
THREADS=25
CHUNK_SIZE = 20
BAL BUFFER SIZE=550
EE BUFFER SIZE =550
RR BUFFER SIZE =550
COST BUFFER SIZE=550
Set threads to 1.5-2.0 times the number of data server

processors.
Set LOW_VOLUME=N to enable optimal plans.
Application Tuning
OA Framework Applications
Refer to MetaLink note 123456.1 (Recommended Patches for

Applications) and MetaLink note 275880.1 (Framework Roadmap).
If running FWK 5.7, ensure that you are running the latest rollup patch
for 5.7H.
Refer to MetaLink note 258333.1.
Upgrade to the latest 9iR2 JDBC Drivers (MetaLink note

164317.1)
Pool Monitor can be used to monitor the resource utilization

Application Module Pool
Memory Utilization
JVM Properties
Pre 11i10: https://<hostname>/servlet/OAAppModPoolMonitor
11i10: Login->Diagnostics->Show Pool Monitor.
Application Tuning
OA Framework Applications (Pool Monitor)
Application Tuning
OA Framework Applications (Pool Monitor)
Application Tuning (OA)

Tracing Framework Applications
Set profile FND : Diagnostics to Yes at user level

Login to Self Service as the above user
Click on Diagnostics icon at the top of page
Select Set Trace Level and click Go
Select the desired trace level and click Save
Perform the activity that you want to trace
Exit application
Application Tuning (OA)

Tracing Framework Applications
Application Tuning
AOLJ Connection Pool (dbc configuration)
$FND_TOP/secure/<hostname_sid>.dbc
Number of DB connections used by the Java based SelfService Applications.
Start with a reasonable default such as 100 or 200.
FND_JDBC_MAX_CONNECTIONS=200
Tune FND_JDBC_MAX_CONNECTIONS as per the
amount of user concurrency (per JVM).
AOLJ automatically decays idle connections and resizes
the pool.
Application Tuning
Disable sanity checks in production environments

FND_JDBC_USABLE_CHECK=false
FND_JDBC_PLSQL_RESET=false
Reduce the number of OXTA JDBC connections
In jserv.properties file, add the following entries
wrapper.bin.parameters=-DOXTAOutThreads=1
wrapper.bin.parameters=-DOXTAInPoolSize=1
If running multiple JVMs, the OXTA connections can be a
significant portion of the total JDBC connections.
Automatically set in 11i10
Application Tuning
AOLJ Supports the use of Services for RAC load

balancing or failover.
Patch 2566387 allows the use of service names in
the configuration:
New dbc parameter APPS_JDBC_URL
Used in place of DB_HOST, DB_NAME,
DB_PORT
Useful for automatic connection load balancing
APPS_JDBC_URL=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=ON)
(ADDRESS=(PROTOCOL=TCP)(HOST=ap201ops)(PORT=1526))(ADDRESS=(PROTOCOL=TCP)
(HOST=ap202ops)(PORT=1526)))(CONNECT_DATA=(SERVICE_NAME=opsperf)))
Application Tuning
AOLJ Connection Pool Monitoring
Use the JSP (AoljDbcPoolStatus.jsp) to monitor the

connection pool status.
Lists DBC configuration parameter settings.
Connection Statistics
Locked Connections
Available Connections
Closed Connections
Allows drill-downs to determine source of the
connection.
http://apptier.oracle.com:1200/OA_HTML/jsp/fnd/AoljDbcPoolStatus.jsp
Application Tuning
Connection Pool Status (AoljDbcPoolStatus.jsp)
Application Tuning
Purging
Implement a regular purge process including

transient data as well as obsolete transactional
data.
Archive system related entities prior to purge such
as
FND_CONCURRENT_REQUESTS
ICX_SESSIONS
Purge old data before performing upgrades.
Application Tuning
Purging
Product
Concurrent Program Name
FND
Purge Concurrent Request

and/or Manager Data
Purge "Sign on" Audit data
General Ledger
GL Archive and Purge
Assets
Mass Additions Purge Report

Purge accounting tables
Payables
Purge
Receivables
Archive and Purge
Order Management
Order Purge

Order Management
Significant improvements in pricing and order

management.
37,500
40000
35000
30000
25000
20000
19,000
15000
10000
5000
0
HVOP Order Import Throughput
11.5.9
11.5.10

Order Management
New Java Pricing Engine

25,000
25000
20000
15000
10000
9,200
5000
0
Order Throughput with Pricing
11.5.9
11.5.10

Inventory / Shipping
20% improvement in Pick Release.

25% improvement in Interface Trip Stop.
2X improvement in Ship Confirm.

Workflow Improvements
Java Mailer
Parallel worker support
15,000 notifications per hour (per worker)
Workflow engine runtime

Improved Caching
Asset Import (5X)
Order Management (5%)
Integration with GSM
Business Event Listener
Mailer

Payroll Improvements
New Run Balances Architecture (HRMS FP F)
Improves reporting performance significantly (~90%):
TRR 95%+
Payroll Register 90%+
US and Canadian Forms and Reports utilize the new Balance
Reporting Architecture.
Custom reports which utilize the balance user exit also benefit
from the new run balances arch.
Payroll run improves by 15%.
Suppression of zero and null balances provides an
additional 10% performance improvement.

TCA Customer Import
Completely rewritten.
Requires 9iR2 (or higher).
Scales well with parallel workers.
300K customers per hour per worker
1 million customers imported in one hour using 4
workers.

TCA Customer Import
1,000,000
1000000
900000
800000
700000
600000
500000
400000
300000
200000
100000
0
11.5.9
11.5.10
10,000
Customer Import Throughput

11i10 Tracing Improvements
Ability to enable all levels of SQL trace as well as PL/SQL profiler

from Forms, Framework based pages, and Concurrent Programs.
Help->About Oracle Applications shows performance information
including time spent in the Form and time spent in the DB.
For concurrent programs, tracing can be enabled at the program
definition level and at the request level.
Trace file identifier is automatically set to the Applications user name
prod11i_ora_2738_BASMITH.trc
New Report (PL/SQL Profiler)

Generates an HTML report of the PL/SQL profiler data.

11i10 Tracing Improvements (PL/SQL Profiler Report)

11i10 Tracing Improvements (PL/SQL Profiler Report)

11i10 Tracing Improvements (Performance Info.)

FND_STATS Improvements:
No invalidate option can be used to avoid invalidating cursors

while gathering statistics.
Auto Gather has been added to allow statistics to be gathered
only on those objects which have changed.
Change threshold is configurable.
Default is 10%.
Significantly reduces the time to gather statistics.
Improves manageability as FND_STATS automatically
determines changed object list.
Maintain history of all gather stats requests.
Automatically determines degree of parallelism.

Configuration Enhancements
Reviewed and optimized default settings for the entire stack via
AutoConfig templates including:
Forms
JDK
DB
Concurrent Manager
Discoverer
AOLJ DBC
Customer Sequence Cache settings are preserved across

patches/upgrades if current setting exceeds standard default.
AutoConfig support for complex configurations including RAC,
Load Balancing, Firewalls, etc..

Significant reduction in patching downtime
Parallelized large table updates.

FNDLOAD Improvements
15% for small LDTs
Orders of magnitude for large LDTs.
Improvements to distributed AD.
Large indexes are now created in parallel.
APPS_MRC schema has been eliminated.
Optimized PL/SQL recompilation order for invalid objects.
MetaLink References
Note #:
Description:
123456.1
E-Business Suite Recommended Patch List
125767.1
Upgrading Developer 6i with Oracle Applications 11i
138159.1
Canceling Long Running Queries in Oracle Applications 11i
164317.1
Upgrading Oracle JDBC Drivers with Oracle E-Business Suite 11i
215268.1
Implementing and Using the JSP Precompiler
216205.1
Database Initialization Parameters and Configuration for Oracle

Applications 11i
244040.1
Recommended Performance Patches for Oracle E-Business Suite
246105.1
Using J2SE Version 1.4 with Oracle E-Business Suite 11i, Release 11.5.9
or Higher
248857.1
Oracle Applications Tablespace Model Release 11i - Tablespace Migration

Utility
258333.1
About OA Framework Mini-pack 11i.FWK.H
275880.1
Oracle Applications Framework Release 11i Documentation Road Map
Q&
A
QUESTIONS
ANSWERS

Maximizing Oracle eBusiness Performance

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Maximizing Oracle eBusiness Performance

Uploaded by

Copyright:

Available Formats

Maximizing the Performance of

the Oracle eBusiness Suite

Java Server Pages (JSP)

Data Server Tier

Application Services Tier

Deploy with socket mode for internal users:

Enable Forms Dead Client Detection

Enable Forms Abnormal Termination Handler

Application Services Tier

Upgrade to the latest Forms patch set (patch set 16)

Application Services Tier

Forms Cancel Query

Should not be enabled unless you are on Forms patchset

Application Services Tier

Ensure users are optimally utilizing the

Application Services Tier

If the form is slow, or a specific flow in the Form

Application Services Tier

Application Services Tier

Logon to Applications 11i with the following

Application Services Tier

Ensure users provide selective parameters to

SQL Trace can be generated at Request

Application Services Tier

The following are the trace options:

Application Services Tier

Enabling Reports Trace (pre 11i10)

Enabling Reports Trace (11i10)

Application Services Tier

Reports (Trace Output)

Application Services Tier

Avoid enabling an excessive number of standard

For jobs which spawn parallel workers such as

Application Services Tier

Database Resource Manager Integration

Application Services Tier

Used for Inventory Transactions as well as other synchronous

Application Services Tier

Application Services Tier

Enabling Trace at the Program Level

Application Services Tier

Enabling Trace at the Request Level

Application Services Tier

Enabling Trace at the Request Level

Application Services Tier

On UNIX, Apache is process based (httpd), and mods

Application Services Tier

Minimize levels of logging (httpds.conf):

Application Services Tier

Enable Caching of non-HTML resources (httpd.conf

Application Services Tier

Review Apache Access Log file to ensure images

Http code 200 (request for document)

130.35.127.106 - - [23/Oct/2002:19:00:21 -0700] "GET /OA_MEDIA/FNDINVDT.gif HTTP/1.1" 200 821

Application Services Tier

Configure a dedicated mod PL/SQL Listener

Improves performance and scalability

Application Services Tier

Allow from all

Application Services Tier

Application Services Tier

Apache Jserv / JVM

Minimize Jserv logging