Middleware

1 Session #187
D DO OU UB BL LE E T TH HE E P PE ER RF FO OR RM MA AN NC CE E O OF F O OR RA AC CL LE E S SO OA A S SU UI IT TE E 1 11 1G G? ?
A AB BS SO OL LU UT TE EL LY Y! !
Ahmed Aboulnaga, Raastech, Inc.
ABSTRACT
Tuning Oracle SOA Suite 11g is an extremely vast topic that alone can span a book. Some changes are obvious, and others are
quite hidden but can yield drastic performance improvements. One change will result in an immediate 32% performance
improvement of your Oracle SOA Suite 11g infrastructure. Another will result in a further 24% performance improvement.
This paper will cover some of the more impactful tuning recommendations to maximize the performance of your
infrastructure, covering areas surrounding JVM, JTA, auditing, HTTP references, and more.

TARGET AUDIENCE
This paper is intended primarily for Oracle SOA Suite 11g administrators, but will benefit designers and developers as well.

EXECUTIVE SUMMARY
The reader will be able to:
Improve the overall performance of the Oracle SOA Suite 11g infrastructure
Learn about hard-to-find parameters designed to drastically improve the performance Oracle SOA Suite 11g
Gain insight on SCA design-time considerations to alleviate performance issues

BACKGROUND
Out of the box, Oracle SOA Suite 11g performs reasonably well. However, as activity grows and the complexities of your
integrations increase, the default settings will not be sufficient to sustain the load and the infrastructure may not perform
adequately. In this paper, we will present a few key areas that will yield considerable performance improvement; some of them
obvious, and some of them quite unique.
Typically in most Oracle SOA Suite 11g installations, the JVM heap size is rarely a concern as long as it is sized appropriately
and is running PS3 (11.1.1.4) or higher. On the midtier, disk I/O is also rarely a concern. However, Oracle SOA Suite 11g is
very write intensive to the database, so database tuning is an area that should always be looked at by a competent DBA. As
long as your heap and OS memory are sized appropriately, SWAP usage typically remains low. CPU utilization, likewise, is
usually seldom a problem under load.

MONITORING THE INFRASTRUCTURE
Whenever a tuning exercise is conducted, typically there needs to be a way to measure baseline performance. Simply speaking,
instance monitoring can be done by reviewing the actual instance flow through Enterprise Manager Fusion Middleware
Control. Not only is this cumbersome, but the timestamps shown only display in the magnitude of seconds, as shown in the
following screenshot.
Middleware
2 Session #187

Alternatively, running SQL queries against the MDS database schema is a much more preferred way. Not only are more
granular timings providing, but grouping and aggregated statistics can be outputted. Observe the following SQL statement.

SELECT
DOMAIN_NAME PARTITION,
COMPONENT_NAME,
DECODE(STATE,'1','RUNNING','5','COMPLETED','6', 'FAULTED','9','STALE') STATE,
TO_CHAR(AVG((TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE- CREATION_DATE),12,2))*60*60) +
(TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),15,2))*60) +
TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),18,4))),'999990.000') AVG,
TO_CHAR(MIN((TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),12,2))*60*60) +
(TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),15,2))*60) +
TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),18,4))),'999990.000') MIN,
TO_CHAR(MAX((TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),12,2))*60*60) +
(TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),15,2))*60) +
TO_NUMBER(SUBSTR(TO_CHAR(MODIFY_DATE-CREATION_DATE),18,4))),'999990.000') MAX,
COUNT(1) COUNT
FROM
CUBE_INSTANCE
GROUP BY
DOMAIN_NAME, COMPONENT_NAME, STATE
ORDER BY
COMPONENT_NAME, STATE

The query may look complicated, but only to format the output data in a clear and easily read format. The following
screenshot shows the output of this query.
Middleware
3 Session #187

Another tool provided by Oracle SOA Suite 11g is the DMS Spy Servlet. By accessing its URL at
http://<host>:<soaport>/dms/Spy, realtime metrics can be returned. The following screenshot shows the output
of the DMS Spy Servlet.

The WebLogic Server Administration Console provides numerous mechanisms to monitor performance. One of the main
areas includes viewing the JVM runtime heap. By simply navigating to Home > Monitoring Dashboard, heap usage on the
managed servers can be viewed in realtime.

Middleware
4 Session #187
It is important not to neglect other areas such as server disk space, CPU utilization, memory and I/O usage, as well as
regularly reviewing the operating system message logs, open files, and running processes. These are often overlooked and
server issues are one of leading causes of performance degradation.

TUNING CONSIDERATIONS
This section covers some key areas of tuning that should be considered in any Oracle SOA Suite 11g installation. The tuning
options are endless, and this section covers some of the areas that will yield the most considerable performance
improvements. The tuning recommendations documented here are based off of the recommendations from Oracle SOA Suite
11g Administrators Handbook, published in August 2012 by Packt Publishing.

TUNING THE JAVA VIRTUAL MACHINE (JVM)
JRockit is proven to be far superior in performance compared to Sun JDK, resulting in a 32% overall performance
improvement. So consider switching from Sun JDK to JRockit JDK. A 64-bit JVM is recommended to avoid the 4GB
(Linux) or 3GB (Windows) addressing space limitations. Once installed, converting to JRockit JDK is simple:
1. Edit $MW_HOME/wlserver_10.3/common/bin/commEnv.sh:
OLD: JAVA_HOME="/u01/app/oracle/jdk1.6.0_26"
NEW: JAVA_HOME="/u01/app/oracle/jrockit1.6.0_29"
OLD: JAVA_VENDOR=Sun
NEW: JAVA_VENDOR=Oracle
2. Edit $MW_HOME/user_projects/domains/[Domain]/bin/setDomainEnv.sh:
OLD: BEA_JAVA_HOME=""
NEW: BEA_AVA_HOME="/u01/app/oracle/jrockit1.6.0_29"
OLD: SUN_JAVA_HOME="/u01/app/oracle/jdk1.6.0_26"
NEW: SUN_JAVA_HOME=""
3. Edit $MW_HOME/user_projects/domains/[Domain]/bin/setSOADomainEnv.sh:
OLD: PORT_MEM_ARGS="-Xms768m -Xmx1536m"
NEW: PORT_MEM_ARGS="-Xms1536m -Xmx1536m -Xgcprio:throughput
-XXtlasize:min=16k,preferred=128k,wasteLimit=8k"

TUNING THE JAVA TRANSACTION API (JTA)
Ensure that the JTA timeout setting is not set to a value too low so that transactions are not rolled back unnecessarily. The
recommended value is 120-300, but can be set higher as needed. This will reduce the occurrence of the following exception in
the logs:
Unexpected exception while enlisting XAConnection
java.sql.SQLException: Transaction rolled back: Transaction timed out after 301 seconds
For example, if an end to end flow is taking longer than the JTA timeout setting, it will rollback. If retries are involved, the
retry counter is reset and may thus result in an infinite number of retries. To set the JTA:
1. Login to the WebLogic Server Administration Console
2. Click on soa_domain > JTA
3. Set the Timeout Settings to 300

TUNING THE LOG LEVEL
Loggers should typically be set to ERROR in production environments, and set to varying levels of NOTIFICATION or
TRACE on non-production environments to assist in troubleshooting. Logs are typically located under the
Middleware
5 Session #187
$MW_HOME/user_projects/domains/[Domain]/servers/[soa_server]/logs directory. Excessive
logging results in considerably more disk space usage and also impacts performance. To set the logger levels:
1. Login to Enterprise Manager Fusion Middleware Control
2. Right-click on soa-infra, navigate to Logs > Log Configuration
3. Select the dropdown View With Persistent Log Level State
4. Apply the following log configuration settings:
Log Levels - oracle.bpm ERROR:1 (SEVERE)
Log Levels - oracle.integration ERROR:1 (SEVERE)
Log Levels - oracle.sdp ERROR:1 (SEVERE)
Log Levels - oracle.sdpinternal ERROR:1 (SEVERE)
Log Levels - oracle.soa ERROR:1 (SEVERE)
Log Levels - oracle.sysman ERROR:1 (SEVERE)
Log Levels - oracle.wsm ERROR:1 (SEVERE)

TUNING THE AUDIT STORE LOGGING
On all non-development environments, set compositeInstanceStateEnabled to false. The first record of the following screenshot
is shown with the composite instance state disabled while the second one with it enabled. Though keeping it enabled helps
identify the state of the composite (e.g., running, completed), it also results in an overall 24% performance hit! This is because
the dashboard must query the COMPOSITE_INSTANCE, CUBE_INSTANCE, and MEDIATOR_INSTANCE tables to
maintain the actual state.

So consider keeping this setting set to true in development environments and false in all others:
1. Login to Enterprise Manager Fusion Middleware Control
2. Right-click on soa-infra, navigate to SOA Administration > Common Properties
3. Uncheck Capture Composite Instance State
4. Also uncheck Payload Validation in non-development environments

TUNING THE AUDIT LEVEL
Reducing the audit level from Development to Production results in a 46% performance improvement. Changing the audit
level from Production to Off results in a further 61% performance improvement. Thus, it is recommended to keep the audit
levels set to Production in all non-development environments as follows:
1. Login to Enterprise Manager Fusion Middleware Control
2. Right-click on soa-infra, navigate to SOA Administration > Common Properties
3. Set Audit Level to Production

Middleware
6 Session #187
TUNING THE AUDIT CONFIGURATION POLICY
Auditing in SOA Suite 11g has severe performance implications. Perhaps the largest composite instance performance gain can
be attained by setting the audit policy to deferred, which results in all audit data being committed to the database
asynchronously. Therefore, set the audit configuration to Deferred instead of the default Immediate setting:
1. Login to Enterprise Manager Fusion Middleware Control
2. Right-click on soa-infra, navigate to SOA Administration > Common Properties > More SOA Infra Advanced Configuration
Properties
3. Click on Audit Config, and set the following properties as shown in this screenshot:


TUNING THE BPEL SERVICE ENGINE
There are many BPEL Service Engine properties that can be tuned, and the following table highlights some of the keys ones
that will help the most:
Parameter Name Default Recommended
Setting
Description
Audit Trail 50000 20000 Messages with values greater than this threshold are saved in the AUDIT_DETAILS table
instead of AUDIT_TRAIL.
Dispatcher System
Threads
2 10 Cleaning and housekeeping work for the engine.
Dispatcher Engine
Threads
30 100 Responsible for picking and processing instances. Set higher if you have a lot of processes with
many dehydration points.
Dispatcher Invoke
Threads
20 70 Instantiates new instances in the engine. Tune higher value if there are multiple branches/flows
for faster processing. Basically throttles the load setting of your BPEL Engine.
AuditStorePolicy SyncLocal
Transaction
AsyncsyncLocal
Transaction
Will change the audit data storage strategy to save audit data asynchronously, using a separate
local transaction.
SyncMaxWaitTime 45 300 Determines the timeout for request-response operations.

TUNING THE MEDIATOR SERVICE ENGINE
There are many Mediator Service Engine properties that can be tuned, and the following table highlights some of the key
ones. Not only will this improve overall throughput, but, for example, increasing the Mediator worker threads can result in a
30% performance improvement for asynchronous services.
Parameter Name Default Recommended
Setting
Description
Metrics Level Enabled Disabled Disable this unless DMS metrics collection is explicitly required.
Parallel Worker Threads 4 20 Set appropriately; small (4) versus large environments (2).
Parallel Maximum Rows Retrieved 200 600 Set appropriately; small (200) versus large environments (600).
Parallel Locker Thread Sleep 2 1 Bad parameter. Always set to 1.

Middleware
7 Session #187
IMPLEMENTING BPEL IN MEMORY PERSISTENCE
To improve the performance of certain types of BPEL processes, consider using inMemoryOptimization which prevents in-flight
instances from writing to the database. Though this improves performance, the restrictions are that the WAIT and
ONMESSAGE activities cannot be used, long running processes are at risk of being unable to rollback in the event of a
server crash, and limited audit information is displayed on the screen.
To reduce the number of times the BPEL process writes to the dehydration store, perform the following:
1. Add the following to the BPEL component in composite.xml to enable inMemoryOptimization:
<BPELProcess src="YourProcess.bpel" id="YourProcess">
<configurations>
<property name="inMemoryOptimization">true</property>
</configurations>
</BPELProcess>

IMPLEMENTING HTTP TIMEOUTS
It is strongly recommended to set HTTP timeouts at design time. This is to reduce consumption of threads in the event of
poor performing external services. There are two parameters that help control this. HTTP Connection Timeout is based on
whether SOA Suite is able to establish a connection with the remote server. HTTP Read Timeout is based on how long the
actual remote service takes to respond. To set these properties at design time:
1. Add the following two properties in the reference section of composite.xml:
<reference name="Register">
<binding.ws port="http://tempuri.org/#wsdl.endpoint
location="RegisterWS.svc.wsdl">
<property name="oracle.webservices.httpConnTimeout"
type="xs:string" many="false">10000</property>
<property name="oracle.webservices.httpReadTimeout"
type="xs:string" many="false">10000</property>
</binding.ws>
</reference>
Note that these can also be manipulated at runtime through Enterprise Manager Fusion Middleware Control.

USING MEDIATOR SEQUENTIAL ROUTING RULES
Mediator executes routing rules either sequentially or in parallel, as dictated during design time. Sequential routing rules are
superior in performance compared to parallel. This is because of the poor design that Oracle has chosen for avoiding the
starving of threads. The problem with sequential routing rules though is that they are synchronous in nature and fault policies
do not take effect. So consider using them for all end-to-end synchronous Mediator flows:
1. Edit all *.mplan files for your Mediator operations and change the executionType as follows:
OLD: executionType="direct"
NEW: executionType="queued"

Considering this is a design time setting and that sequential routing rules will not provide true asynchronicity, test them out to
understand the behavior first. Implementing sequential routing rules may improve performance of Mediator instances
anywhere from 4% to 509%.

Middleware
8 Session #187
SUMMARY
Tuning Oracle SOA Suite 11g is a very lengthy exercise, as there are a near unlimited number of areas that can be addressed,
even expanding to the WebLogic Server, Oracle Database, and the underlying operating system. The tuning considerations
provided in this paper were meant to highlight core areas of tuning that would yield considerable performance improvements
in most environments.
Some of the key takeaways of this paper include:
Upgrading to PS3 (11.1.1.4) addresses memory instability issues
Moving from Sun JDK to JRockit results in a 32% boost
Optimizing the audit level makes a huge difference, resulting in a 46% to 92% performance improvement
Increasing the Mediator worker threads results in a 30% performance improvement for async services
Mediator sequential routing rules may improve performance from 4% to 509%
Dont forget to tune the BPEL Service Engine
Consider parallel garbage collection, but don't have statistics
The majority of the recommendations here are pulled from Oracle SOA Suite 11g Administrators Handbook, published in August
2012 by Packt Publishing. Chapter 4 of the book, titled Tuning Oracle SOA Suite 11g for Optimum Performance, includes hundreds
of other recommendations.

REFERENCES
Oracle SOA Suite 11g Administrators Handbook, Ahmed Aboulnaga & Arun Pareek (Packt Publishing, 2012)
http://www.packtpub.com/oracle-soa-suite-11g-administrators-handbook/book
Performance tuning Oracle SOA Suite 11g, Ahmed Aboulnaga, April 22, 2011
http://blog.raastech.com/2011/04/performance-tuning-oracle-soa-suite-11g.html