Oracle Troublesoohting Vol 2

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY.
COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Volume II Student Guide
D61523GC20
Edition 2.0
May 2011
D72554
Oracle University and Sentra inversiones y servicios LTDA use only
Oracle WebLogic Server 11g:

Diagnostics and Troubleshooting
Copyright 2011, Oracle and/or it affiliates. All rights reserved.
Bill Bell
Disclaimer
Technical Contributors
and Reviewers
Will Lyons
TJ Palazzolo
Serge Moiseev
Editors
Richard Wallis
Malavika Jinka
Publishers
Jobi Varghese
Shaik Mahaboob Basha
This document contains proprietary

y information and is protected by
y copyright
y g and
other intellectual property laws. You may copy and print this document solely for your
own use in an Oracle training course. The document may not be modified or altered
in any way. Except where your use constitutes "fair use" under copyright law, you
may not use, share, download, upload, copy, print, display, perform, reproduce,
publish, license, post, transmit, or distribute this document in whole or in part without
the express authorization of Oracle.
The information contained in this document is subject to change without notice. If you
find any problems in the document, please report them in writing to: Oracle University,
500 Oracle Parkway, Redwood Shores, California 94065 USA. This document is not
warranted
t d tto be
b error-free.
f
Restricted Rights Notice
If this documentation is delivered to the United States Government or anyone using
the documentation on behalf of the United States Government, the following notice is
applicable:
U.S. GOVERNMENT RIGHTS
The U.S. Governments rights to use, modify, reproduce, release, perform, display, or
disclose these training materials are restricted by the terms of the applicable Oracle
license agreement and/or the applicable U.S. Government contract.
Trademark Notice
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names
may be trademarks of their respective owners.
THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Author
Course Overview
Course Objectives 1-2
Target Audience 1-3
Introductions 1-4
Course Schedule 1-5
Course Appendix 1-7
Course Practices 1-8
Classroom Guidelines 1-9
For More Information 1-10
Related Training 1-11
Oracle by Example (OBE) 1-12
WLST Monitoring
Objectives 2-2
WLS Domains: Review 2-3
Java Management Extension (JMX): Review 2-4
WLS MBean Hierarchies 2-5
WLS MBean Reference Documentation 2-6
Console Monitoring: Review 2-8
WebLogic Scripting Tool (WLST): Review 2-9
WLST MBean Syntax: Review 2-10
Domain Runtime 2-11
Basic Jython Syntax: Review 2-12
Basic WLST Commands 2-13
Variable Declaration 2-14
Password Management 2-15
Error Handling 2-16
File I/O 2-17
Standard Jython Libraries 2-18
WLST Example: Monitor a JMS Server 2-19
Quiz 2-20
Summary 2-23
Practice 2-1 Connecting to the Classroom Grid 2-24
Practice 2-2 Developing a Custom Monitoring Script 2-25
iii
Contents
Guardian
Objectives 3-2
Guardian Capabilities 3-3
Using Guardian 3-4
Guardian Architecture 3-5
Agent Installation 3-6
Collected Data 3-7
Client Installation 3-8
Guardian User Interface 3-9
Activating a Domain 3-10
Creating a Domain Inventory 3-11
Signatures and Bundles 3-12
Updating the Signature Repository 3-13
Signature Annotations 3-14
Evaluating a Domain 3-15
Evaluation Summary 3-16
Generating a Support Request 3-17
Command-Line Interface 3-18
Quiz 3-19
Summary 3-22
Practice 3-1 Using Guardian to Evaluate a Domain 3-23
Diagnostic Framework Essentials

Objectives 4-2
Road Map 4-3
WebLogic Diagnostic Framework (WLDF) 4-4
WLDF Architecture 4-5
WLS Logging: Review 4-6
Log Severity Thresholds 4-7
Application Logging 4-8
Server Logging Bridge 4-9
WLDF Configuration: Overview 4-10
Diagnostic Images 4-11
Capturing a Server Diagnostic Image 4-12
WLST: Downloading Diagnostic Image Files 4-13
Diagnostic Archives 4-14
Configuring Server Diagnostic Archives 4-15
Archive Retirement Policies 4-16
Archive Database Schema 4-17
Viewing Archive Contents 4-18
Creating a Diagnostic Module 4-19
iv
WLDF WLST Examples 4-20

Section Summary 4-22
Road Map 4-23
Harvester Architecture 4-24
Metric Collector Definitions 4-25
Configuring a Metric Collector 4-26
Watches and Notifications 4-28
Configuring a Watch 4-29
Watch Alarms 4-31
Configuring a JMS Notification 4-32
Configuring an Email Notification 4-33
Harvester WLST: Example 4-34
Watch WLST: Example 4-35
WLDF Sample Framework 4-36
Practice 4-1 Harvesting Diagnostic Metrics 4-38
Road Map 4-39
New Monitoring Dashboard 4-40
Viewing the Dashboard 4-41
Monitoring Dashboard Interface 4-42
Views 4-43
Built-In Views 4-44
Creating a Custom View 4-45
Metric Browser 4-46
Anatomy of a Chart 4-47
Chart and Graph Properties 4-48
Chart Styles 4-49
Current and Historical Data 4-50
Practice 4-2 Monitoring Diagnostic Metrics 4-52
Road Map 4-53
Subsystem Debugging 4-54
Console Debug Scopes 4-55
Debug Scopes: Examples 4-56
Debug Logging 4-57
WLST Debugging: Examples 4-58
Quiz 4-60
Summary 4-64
Diagnostic Instrumentation
Objectives 5-2
Road Map 5-3
Instrumentation Scenarios 5-4
Instrumentation Architecture 5-5
Monitor Actions 5-6
Application-Scoped Modules 5-8
WLS Monitor Library 5-9
Deployment Plan Review 5-11
WLDF and Deployment Plans 5-12
WLDF Deployment Plan: Example 5-13
WLDF Hot Swap 5-14
Configuring a System-Scoped Monitor 5-15
Configuring an Application-Scoped Monitor 5-17
Aspect-Oriented Programming (AOP) Concepts 5-18
Custom Monitors 5-19
Instrumentation WLST: Example 5-20
Instrumentation and Request Performance 5-21
Practice 5-1 Configuring and Monitoring Diagnostic Events 5-23
Road Map 5-24
Request Context ID 5-25
Viewing Context IDs 5-26
Request Dying 5-27
Available Dyes 5-28
Configuring a Dye Injection Monitor 5-29
Event Filtering 5-30
Configuring Dye Masks 5-31
Event Throttling 5-32
Configuring Throttle Properties 5-33
Quiz 5-35
Summary 5-38
Practice 5-2 Tracing a Client Request 5-39
JVM Diagnostics
Objectives 6-2
Road Map 6-3
Basic Java Concepts 6-4
Java Virtual Machine (JVM): Review 6-5
Oracle JVM Support 6-6
vi
vii
JVM Recommendations 6-7

JVM Memory 6-8
Garbage Collection 6-9
Sun HotSpot Garbage Collection 6-10
Garbage Collection (GC) Types 6-11
Setting WLS JVM Arguments 6-12
Basic Sun JVM Arguments 6-13
JRockit Garbage Collection 6-14
Basic JRockit JVM Arguments 6-15
Out of Memory 6-16
Out-of-Memory Response 6-17
Memory Leak 6-18
JVM Crash 6-19
JVM Error Log 6-20
Road Map 6-22
JVM Tool Varieties 6-23
Java Stack Trace 6-24
Java Thread Dump: Overview 6-25
Thread Dump Signal 6-26
JVM Crash Actions 6-27
Verbose GC 6-28
Sun JVM Profiler Agent 6-29
Sun JVM Diagnostic Tools: Overview 6-30
Sun Diagnostic Tools: Examples 6-31
JVisualVM 6-33
Using JVisualVM 6-34
Practice 6-1 Troubleshooting a Running JVM 6-37
Road Map 6-38
Console JVM Monitoring 6-39
JVM WLST: Example 6-40
WLS Low Memory Detection 6-41
Configuring Low Memory Detection 6-42
Road Map 6-44
JRockit Diagnostic Tools: Overview 6-45
JRockit Diagnostic Tools: Examples 6-46
Management Communication 6-47
JRockit Mission Control (JRMC) 6-48
JRockit Discovery Protocol (JDP) 6-49
Troubleshooting Java Applications

Objectives 7-2
Java Exception-Handling Concepts 7-3
Exception Chains 7-4
Class Not Found Errors 7-5
Class Cast Errors 7-6
Classpath: Review 7-7
WebLogic Start Script: Review 7-8
Viewing the WLS Classpath 7-9
Manifest Files and the Classpath 7-10
Domain Libraries 7-11
Java Class Loaders 7-12
Searching Class Loaders 7-13
Searching Class Loaders: Example 7-14
Default WLS Class Loader Hierarchy 7-15
Java EE Packaging: Review 7-16
Prefer Web Application Classes 7-17
Prefer Enterprise Application Classes 7-18
Client Library Errors 7-19
Null Pointer Errors 7-20
Stack Overflow Errors 7-21
viii
JVM Browser 6-50

Management Console: Features 6-51
Management Console: General > Overview 6-52
Management Console: Runtime > Threads 6-53
Management Console: MBeans > Triggers 6-54
JRockit Flight Recorder (JFR) 6-55
Integration of JRockit Flight Recorder and WLDF 6-56
Starting the Flight Recorder from JRMC 6-57
Flight Recorder Output 6-58
General > Overview 6-59
Memory: Object Statistics 6-60
Code > Overview 6-61
Memory Leak Detector (Memleak): Features 6-62
Memleak: Trend Tab 6-63
Memleak: Type Graph 6-64
Quiz 6-66
Summary 6-70
Practice 6-2 Troubleshooting Applications on JRockit 6-71
Troubleshooting Servers
Objectives 8-2
Road Map 8-3
WLS Message Catalog: Review 8-4
Server Startup Errors 8-5
Boot Identity Errors 8-6
WLS Native Libraries 8-7
Setting the Native Library Path 8-8
Causes of Unresponsive Servers 8-9
WLS Threading Architecture 8-10
Execute Thread State 8-11
Work Managers 8-12
Work Manager Architecture 8-13
Creating a Work Manager 8-14
Creating and Using a Request Class 8-15
Assigning Work Managers to Applications 8-16
Monitoring a Server Thread Pool 8-17
Monitoring Individual Server Threads 8-18
Server Monitoring: WLST Examples 8-19
Server WLDF Image Contents 8-20
Java Deadlock Concepts 8-21
Thread Analysis 8-22
Lock Chains 8-23
Stuck Thread Detection 8-24
Overload Protection 8-25
Configuring Overload Protection 8-26
Practice 8-1 Investigating Server Problems 8-28
Road Map 8-29
WLS Deployment: Review 8-30
Deployment Errors 8-32
Application Staging 8-33
Deployment Memory Errors 8-34
Shared Library: Review 8-35
Library Errors 8-36
Deployment Debug Flags 8-37
ix
Too Many Open Files Errors 7-22

Quiz 7-23
Summary 7-25
Practice 7-1 Investigating Classpath Problems 7-26
Troubleshooting JDBC
Objectives 9-2
JDBC: Review 9-3
Data Sources: Review 9-4
JDBC Management: WLST Examples 9-5
JDBC Runtime Attributes 9-6
JDBC Monitoring: WLST Examples 9-7
JDBC WLDF Image Contents 9-8
JDBC WLDF Monitor: Review 9-9
Data Source Diagnostic Profiling 9-10
Configuring Diagnostic Profiling 9-11
JDBC Debug Flags 9-12
Other JDBC Debugging Tools 9-13
Common Configuration Errors 9-14
Configuration Error Examples 9-15
Insufficient Connection Errors 9-16
Connection Leaks 9-17
Database Cursor Considerations 9-18
Common Connection Errors 9-19
Statement Timeout 9-20
Data Sources and Database Availability 9-21
Retry Frequency and Login Timeout 9-22
Connection Testing: Review 9-23
Testing Trusted Connections 9-24
Firewall Considerations 9-25
Multi Data Source: Overview 9-26
Multi Data Source: Architecture 9-27
Java Persistence API (JPA): Overview 9-28
JPA Configuration: Overview 9-29
Troubleshooting JPA: Overview 9-30
Quiz 9-31
Summary 9-34
Practice 9-1 Investigating JDBC Problems 9-35
Application Error Handling 8-38

Application Monitoring: Review 8-39
Application Monitoring: WLST Examples 8-40
Quiz 8-42
Summary 8-45
xi
10 Troubleshooting JMS
Objectives 10-2
JMS: Review 10-3
WebLogic JMS Configuration: Review 10-5
JMS Transactions: Review 10-7
JMS Management: Overview 10-8
Console JMS Management 10-9
JMS Management: WLST Examples 10-10
JMS Runtime MBean Hierarchy 10-11
JMS Monitoring: WLST Examples 10-12
JMS Diagnostic Image Contents 10-13
JMS Message Logging 10-14
Configuring JMS Logging 10-15
JMS Debug Flags 10-16
Message Type Considerations 10-17
JMS Client Libraries 10-20
Out-of-Memory Errors and Quotas 10-21
Configuring a JMS Server Quota 10-22
Creating a Destination Quota 10-23
Message Paging 10-24
Too Many Pending Messages 10-25
Quota Blocking Policies 10-26
Thresholds and Flow Control 10-27
Configuring Thresholds 10-28
Tuning Flow Control 10-29
Lost Messages 10-30
Time to Live (TTL) 10-31
Expiration Policies 10-32
Delivery Mode 10-33
Message Redelivery 10-34
Time to Deliver (TTD) 10-35
Durable Subscriber Review 10-36
Monitoring and Managing Subscriptions 10-37
Duplicate Messages 10-38
Poison Messages 10-39
Consumer Acknowledgement Modes 10-40
Messages Out of Sequence 10-41
Unit of Order (UOO): Overview 10-42
Unit of Work (UOW): Overview 10-43
Message-Driven Beans (MDBs): Review 10-44
11 Troubleshooting Security
Objectives 11-2
Road Map 11-3
Secure Sockets Layer (SSL): Review 11-4
SSL Communication: Review 11-5
WebLogic SSL Scenarios 11-6
Proxy Server SSL Scenarios 11-8
Keystore: Review 11-9
Trust Keystores 11-10
Keytool: Review 11-11
WebLogic SSL Support 11-13
SSL Configuration: Review 11-14
Restarting SSL 11-16
SSL Debug Flags 11-17
SSL Handshake Trace 11-18
Other SSL Traces 11-19
Invalid Format or Cipher Errors 11-20
Certificate Validation Errors 11-21
Host Name Verification Errors 11-22
Certificate Chains 11-23
WLS Chain Validation Utility 11-24
Missing Constraint or Policy Errors 11-25
Practice 11-1 Investigating SSL Problems 11-27
Road Map 11-28
Security Realm: Review 11-29
Security Provider Stores 11-30
Some Security Providers 11-31
Embedded LDAP: Review 11-32
Embedded LDAP Backups 11-33
Embedded LDAP Synchronization Issues 11-34
Viewing Embedded LDAP Contents 11-35
LDAP Concepts 11-36
LDAP Structure 11-37
xii
MDB Capabilities 10-45

MDB Runtime Attributes 10-46
MDB Diagnostics and Debugging 10-47
Quiz 10-48
Summary 10-51
Practice 10-1 Investigating JMS Problems 10-52
12 Troubleshooting Node Manager

Objectives 12-2
Node Manager (NM): Review 12-3
Node Manager Types: Review 12-4
Node Manager Configuration: Review 12-5
Basic Java Node Manager Properties 12-6
Java Node Manager Logging 12-7
Java Node Manager Availability 12-8
Basic Script Node Manager Interface 12-9
Node Manager Server Start Parameters 12-11
Configuring Server Start Parameters 12-13
Monitoring Node Managers 12-14
Node Manager: WLST Examples 12-15
Generating Template Properties for Java NM 12-17
Node Manager Authentication 12-18
Configuring Node Manager Credentials 12-19
Node Manager Trusted Domains 12-20
Machine Enrollment 12-21
Server Boot Identity 12-22
Configuring Node Manager SSL 12-23
xiii
LDAP Search Operations 11-38

Resetting Admin Password in Embedded LDAP 11-39
Database Store Cache Synchronization Issues 11-40
Auditing Provider 11-41
Security Audit Events 11-42
Configuring the Auditing Provider 11-43
Realm Debug Flags 11-44
Typical Authentication Trace 11-45
Typical Role Mapping Trace 11-47
Typical Authorization Trace 11-48
LDAP Trace Log 11-49
Authentication Provider Control Flags 11-50
External LDAP Authentication Providers 11-52
LDAP Provider Configuration: Overview 11-53
Common LDAP Issues 11-56
Quiz 11-58
Summary 11-62
Practice 11-2 Investigating Security Realm Problems 11-63
13 Troubleshooting Clusters
Objectives 13-2
Road Map 13-3
Cluster Review 13-4
Proxy Plug-in Review 13-5
Obtaining and Using Plug-Ins 13-6
Oracle HTTP Server (OHS) Review 13-7
Oracle Process Manager and Notification Server (OPMN) Review 13-9
OPMNCTL Examples 13-10
OHS Logs 13-11
Plug-in Configuration Review 13-12
Basic Plug-in Parameters 13-13
Proxy Connection Architecture 13-14
Dynamic Server List 13-16
Connection Parameters 13-17
Common Connectivity Issues 13-18
Proxy SSL Issues 13-19
Proxy Debug Page 13-20
Proxy Debug Log 13-22
Typical Proxy Trace 13-23
Practice 13-1 Investigating Proxy Problems 13-25
Road Map 13-26
Cluster Communication Review 13-27
Unicast Architecture 13-28
Session Management Review 13-29
Session Persistence Review 13-30
In-Memory Replication Review 13-31
Cluster Monitoring WLST Examples 13-32
Session Monitoring WLST Examples 13-33
Session Monitoring Attribute 13-34
Session Instrumentation 13-35
Cluster Debug Flags 13-36
Typical Cluster Heartbeat Trace 13-37
Typical Replication Trace: Primary 13-38
Typical Replication Trace: Secondary 13-39
Common Replication Issues 13-40
xiv
Quiz 12-24
Summary 12-26
Practice 12-1 Investigating Node Manager Problems 12-27
A WebLogic SNMP
Simple Network Management Protocol (SNMP) A-2
SNMP Architecture A-3
Object Identifier (OID) A-4
Management Information Base (MIB) A-5
WLS MIB and OIDs A-6
Common SNMP Message Types A-7
WLS SNMP Architecture A-8
Creating an SNMP Agent A-10
Configuring an SNMP Agent A-11
SNMP Channels A-12
WLS SNMP Notifications A-13
Creating Trap Monitors A-14
Creating Trap Destinations A-15
SNMP Security A-16
Configuring Agent Security A-17
Configuring SNMP V3 Credentials A-18
Configuring Trap Destination Security A-19
WLS SNMP Utility A-20
xv
HttpSession API Overview 13-41

Serialization Overview 13-42
Serialization Debug Messages 13-43
Quiz 13-45
Lesson Summary 13-48
Practice 13-2 Investigating Cluster Replication Problems 13-49
Copyright 2011, Oracle and/or its affiliates. All rights reserved.
Troubleshooting JDBC
After completing this lesson, you should be able to:

Use the console, WLST, and WLDF to monitor and
troubleshoot data sources
Identify common data source configuration issues
Identify connection leaks
Configure a data source to handle database availability
Describe the multi data source architecture
Explain the relationship between JDBC and JPA
Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 9 - 2
Objectives
Java Database Connectivity (JDBC):

Allows Java programs to access databases in a uniform way
Requires the use of a JDBC-compliant driver that supports
your DB vendor and version
Several drivers are included with WLS and are already in

the server classpath.
Most other third-party drivers are also supported.
Application
1.
2.
3.
4.
5.
Create connection.
Create statement.
Execute SQL.
Process results.
Close connection.
JDBC Driver
The JDBC API is the industry standard for database-independent connectivity between the
Java programming language and a wide range of databases. The JDBC API provides a calllevel API for SQL-based database access. The JDBC API makes it possible to do three
things: establish a connection with a database, send SQL statements, and process the
results. With the JDBC API, no configuration is required on the client side. With a driver
written in the Java programming language, all the information needed to make a connection is
defined by the JDBC URL and optional connection properties.
Type 4 JDBC drivers are installed with WebLogic Server in the <WL_HOME>\server\lib
folder, where <WL_HOME> is the directory in which you installed WebLogic Server. Driver
class files are included in the manifest classpath in weblogic.jar, so the drivers are
automatically added to your classpath on the server. If you choose a custom installation,
ensure that the WebLogic JDBC Drivers option is selected (checked). If this option is not
selected, the drivers are not installed.
JDBC: Review
Data sources:
Allow database connectivity to be managed by the
application server
Are obtained by applications from the servers JNDI tree
Use a dynamic pool of reusable database connections
Data Source
App
Connection
App
Connection
Connection
App
1.
2.
3.
4.
JNDI lookup
Reserve connection.
Perform SQL
Release connection.
Connection
JDBC Driver
Oracle WebLogic Server can manage your database connectivity through JDBC Data
Sources and multi data sources. Each data source you configure contains a pool of database
connections that are created when the data source instance is createdwhen it is deployed
or targeted, or at server startup. The connection pool can grow or shrink dynamically to
accommodate demand.
Applications look up a data source on the Java Naming and Directory Interface (JNDI) tree or
in the local application context (java:comp/env), depending on how you configure and
deploy the object, and then request a database connection. When finished with the
connection, the application uses the close operation on the connection, which simply returns
the connection to the connection pool in the data source.
Oracle WebLogic Server data sources allow connection information such as the JDBC driver,
the database location (URL), and the username and password to be managed and
maintained in a single location, without requiring the application to worry about these details.
In addition, limiting the number of connections is important if you have a licensing limitation on
your database or if it can support only a specific capacity.
Data Sources: Review
Data sources support various management tasks via the

console or WLST.
Most data source modifications require that the data
source be restarted.
Restart a data source using the latest configuration settings:

serverRuntime()
ds = getMBean('/JDBCServiceRuntime/ServerC/
JDBCDataSourceRuntimeMBeans/NodeXDataSource')
ds.shutdown()
ds.start()
Recreate all data source connections:
...
ds.reset()
Using the Administration Console or WebLogic Scripting Tool (WLST) scripts, you can
manage the JDBC data sources in your domain for maintenance or troubleshooting purposes.
The JDBCDataSourceRuntimeMBeans operations include:
forceShutdown(): Destroys all open database connections, including those currently
in use by applications. The shutdown() command will fail if there are active
connections.
shrink(): When you shrink a data source, WebLogic Server reduces the number of
connections in the pool to the greater of either the initial capacity or the number of
connections currently in use.
reset(): Closes and re-creates all available database connections in a data source.
This may be necessary, for example, after the DBMS has been restarted. Often when
one connection in a data source has failed, all of the connections in the pool are bad.
suspend(): When you suspend a data source, the data source is marked as disabled
and applications cannot use connections from the pool. Applications that already have a
reserved connection from the data source when it is suspended will get an exception
when trying to use the connection.
clearStatementCache(): When a prepared statement or callable statement is used
on a connection, WebLogic Server caches the statement so that it can be reused. You
can manually clear the statement cache for all connections in a data source.
JDBC Management: WLST Examples
Metric
Description
State
Running, Suspended, Shutdown, Overloaded, or Unhealthy
Capacity
Current number of connections in the pool
Capacity High
Highest number of connections in the pool since server start
Active Connections
Current number of connections reserved by applications
Active Connections
High
Highest number of reserved connections since server start
Active Connections
Average
Average number of reserved connections at any given time
Number Available
Current number of connections not reserved
Number Unavailable
Current number of connections that are either reserved or

being tested
Number Unavailable
High
Highest number of connections that were unavailable since

server start
Failed Reserve
Cumulative count of connection reservation failures

Refer to the JDBCDataSourceRuntimeMBean documentation for a complete list of available

metrics.
If the Connection Reserve Timeout attribute of the data source is not disabled (1),
additional runtime metrics are available that describe how often applications waited for
connections and for how much time, as in the following example:
WaitingForConnectionCurrentCount
A data source can be in one of several states:
Running: The data source is enabled (deployed and not suspended). This is the normal
state of the data source.
Suspended: The data source has been administratively disabled, although connections
may still remain alive.
Shutdown: The data source is shut down and all database connections have been
closed.
Overloaded: All available connections are in use.
Unhealthy: All connections are unavailable (not because they are in use). This state
occurs if the database server is unavailable when the data source is created (creation
retry must be enabled) or if all connections have failed connection tests (on creation, on
reserve, or on periodic testing).
JDBC Runtime Attributes
Get data source runtime statistics:

serverRuntime()
ds = getMBean('/JDBCServiceRuntime/ServerC/
JDBCDataSourceRuntimeMBeans/NodeXDataSource')
print 'Active: ' , ds.getActiveConnectionsCurrentCount()
print 'Unavailable: ' , ds.getNumUnavailable()
Print the status of each connection in a data source:
ds = ...
ds.dumpPool()
Refer to the JDBCDataSourceRuntimeMBean documentation for a list of all available

attributes and operations.
JDBC Monitoring: WLST Examples
Diagnostic images include a file named JDBC.img that

captures the dumpPool() output for all data sources:
Current capacity
Status and general attributes of each connection
Reservation stack trace for each active connection
JDBC.img:
Resource Pool:NodeX_DS:dumpPool Current Capacity = 19
Resource Pool:NodeX_DS:dumpPool available[0] =
autoCommit=true,enabled=true,isXA=true ...
Resource Pool:NodeX_DS:dumpPool reserved[3] = ...
at weblogic.jdbc.common.internal.ConnectionEnv.setup ...
at weblogic.common.resourcepool.ResourcePoolImpl ...
at com.mycompany.payroll.EmployeeManager.getEmployee
...
A diagnostic image is a heavyweight artifact meant to serve as a server-level state dump for
the purpose of diagnosing significant failures. It enables you to capture a significant amount of
important data in a structured format and then to provide that data to support personnel for
analysis. Because it is an artifact intended primarily for internal consumption, the image
contents are not documented in detail and are subject to change.
JDBC WLDF Image Contents
Subsystem
JDBC
Available Code Points

Before/After Connection
Reserve/Release Connection
Before/After Commit
Before/After Rollback
Before/After SQL Statement
Perform a stack dump when a
connection is released from
any data source.
A diagnostic monitor is a dynamically manageable unit of diagnostic code that is inserted into
server or application code at specific locations. WLDF provides a library of predefined
diagnostic monitors and actions. Diagnostic actions perform some type of data collection
intended to help gain insight into the server or application. Each diagnostic action can be
used with only those monitor types with which they are compatible. All actions also capture
general statistics such as the current time, transaction ID, and user ID, if applicable.
A Stack Dump action generates an instrumentation event at the affected location in the
program execution to capture a stack dump. It captures the current Java stack trace as an
event payload.
A Thread Dump action generates an instrumentation event at the affected location in the
program execution to capture a JVM thread dump. It captures the thread dump as event
payload.
When attached to before monitors, the Display Arguments instrumentation event captures
input arguments to the joinpoint (for example, method arguments). When attached to after
monitors, the instrumentation event captures the return value from the joinpoint.
JDBC WLDF Monitor: Review
You can also enable more sophisticated diagnostic data

collection on individual data sources.
Data source connections are profiled periodically and the
results are recorded to the WLDF archive as events.
These event payloads can be viewed only by using WLST.
Event Type
Description
Usage
A thread that is currently using a specific connection
Last Usage
A thread that received an exception while using a connection
Reservation Wait
A thread that is currently waiting for a connection
Reservation
Failed
A thread that received an exception while requesting a

connection
Leak
A missing connection along with the reservation stack trace
Statement Usage
A cached SQL statement that is being used

You can configure any data source to collect profile information to help you pinpoint the
source of a problem. The collected profile information is stored as events in the WLDF event
archive. Each available data source event type has the same attribute names, but they will
contain different information. When configuring your data source for profiling, you must
specify the interval at which profile data is harvested.
Enable connection usage profiling to collect information about threads currently using
connections from the pool of connections in the data source. This profile information can help
determine why applications are unable to get connections from the data source. By default,
enabling connection usage profiling on its own will not provide a stack trace of the threads
using the connections. To obtain this information, you must enable profiling of connection
leaks in addition to this profile type.
Enable connection reservation wait profiling to collect information about threads currently
waiting to reserve a connection from the data source. This profile information can help
determine why applications are unable to get connections from the data source or to wait for
connections.
Enable connection leak profiling to collect information about threads that have reserved a
connection from the data source but have had the connection leaked (the connection was not
properly returned to the pool of connections). This profile information can help determine
which applications are not properly closing JDBC connections.
Data Source Diagnostic Profiling
1
2
Enable or disable
diagnostic event types.
3
How often to profile
connections?
To configure diagnostic profiling for a JDBC data source:

1. Navigate to the data source that you want to modify.
2. Select Configuration > Diagnostics tab.
3. Enable any of the following diagnostic profiling options. Also configure how frequently
the selected profiling options should occur by using the Profile Harvest Frequency
Seconds field (default is five minutes).
Enable connection reservation failure profiling to collect information about threads that
attempt to reserve a connection from the data source but fail to get that connection. This
profile information can help determine why applications are unable to get connections from
the data source even after reserving them.
Enable connection last usage profiling to collect information about the previous thread that
last used the connection. This information is useful when you are debugging problems with
connections infected in pending transactions that cause subsequent XA operations on the
connections to fail.
Enable statement cache entry profiling to collect information for prepared and callable
statements added to the statement cache, and for the threads that originated the cached
statements.
Configuring Diagnostic Profiling
Flag
Description
DebugJDBCConn
Trace all connection reserve and release operations.
DebugJDBCSQL
Trace all JDBC API calls, parameters, and return values.
DebugJDBCInternal
Dump all low-level internal data source activities.
DebugJTAJDBC
Trace transaction management for JDBC resources.
Server log messages with DebugJDBCSQL enabled:

<Debug> <JDBCSQL> <...
LogicalConnection-MedRecGlobalDataSourceXA-5 ...
prepareStatement(SELECT name, address, contact from vendor)>
OraclePreparedStatementWrapper@18] executeQuery()>
OracleResultSetImpl@19] getString(1)>
OracleResultSetImpl@19] getString returns Waverly Supply>
DebugJDBCSQL (scope weblogic.jdbc.sql): Prints information about all JDBC methods

invoked, including their arguments and return values, and thrown exceptions
DebugJDBCConn (scope weblogic.jdbc.connection): Traces all connection reserve
and release operations in data sources as well as all application requests to get or close
connections
DebugJDBCInternal (scope weblogic.jdbc.internal): Low-level debugging related
to the data source, the connection environment, and the data source manager
DebugJDBCRMI (scope weblogic.jdbc.rmi): Similar to JDBCSQL but at the RMI level.
Turning on this flag and JDBCSQL will get two sets of debug messages for each operation
called from a remote JDBC client.
DebugJDBCDriverLogging (scope weblogic.jdbc.driverlogging): Enables JDBC
driver-level logging (this replaces ServerMBean JDBCLoggingEnabled and
getJDBCLogFileName). Note that to get driver-level tracing for Oracle, you need to use
ojdbc14_g.jar instead of ojdbc14.jar.
DebugJTAJDBC (scope weblogic.transaction.jdbc,
weblogic.jdbc.transaction): Traces information about reading and writing JTA records
DebugJTAXAStackTrace (scope weblogic.transaction.stacktrace): Detailed
tracing that prints stack traces at various critical locations
JDBC Debug Flags
JDBC Spy logging connection properties for WLS drivers

Debugging and logging properties for Oracle thin and other
drivers
Third-party JDBC debugging tools such as P6Spy
Data source URL with JDBC Spy logging attributes:

jdbc:weblogic:db2://199.177.1.1:50000;spyAttributes=(log=(file
)db2-spy.out;load=weblogic.jdbc.db2.DB2Driver;timestamp=yes)
JDBC Spy log file:
spy>>
spy>>
spy>>
spy>>
Connection[1].createStatement
OK (Statement[1])
Statement[1].executeQuery(String sql)
sql = select name, job from employee where empno=7369
WebLogic JDBC Spy is a wrapper that wraps a WebLogic Type 4 JDBC driver. It logs detailed
information about JDBC calls issued by an application and then passes the calls to the
wrapped WebLogic Type 4 JDBC driver. You can use the information in the logs to help
troubleshoot problems in your application.
Before you start the server, add <WL_HOME>/server/lib/wlspy.jar to your
CLASSPATH, where <WL_HOME> is the directory in which you installed the WebLogic Server
software. Using WLS administrative tools such as the console or WLST, append the
WebLogic JDBC Spy options to the data source URL. Enclose all JDBC Spy options in one
set of parentheses; separate multiple options with a semicolon. The load parameter
indicates the Java class name of the JDBC driver whose work should be intercepted by JDBC
Spy.
P6Spy is an open source framework to support applications that intercept and optionally
modify database statements. The P6Spy distribution includes the following modules: P6Log
and P6Outage. P6Log intercepts and logs the database statements of any application that
uses JDBC. This application is particularly useful for developers to monitor the SQL
statements produced by EJB servers, enabling the developer to write code that achieves
maximum efficiency on the server. P6Outage detects long-running statements that may be
indicative of a database outage problem; it will log any statement that surpasses the
configurable time boundary during its execution.
Other JDBC Debugging Tools
Typical causes include:

Using invalid connection URL, credentials, and/or
attributes
Missing driver or wrong version in the system classpath
(bundled drivers are included indirectly using MANIFEST
entries in JAR files)
Missing native libraries for older, platform-dependent
drivers
Specifying invalid connection testing parameters
The 11g version of the Oracle thin driver (ojdbc6.jar for JDK 6, and ojdbc5.jar for JDK
5) is installed with Oracle WebLogic Server. In addition to the Oracle thin driver, the mySQL
5.0.x (mysql-connector-java-commercial-5.0.x-bin.jar) JDBC driver is installed
with WebLogic Server.
Drivers are installed in the <WL_HOME>\server\lib folder (where <WL_HOME> is the
folder where WebLogic Server is installed) by using weblogic.jar. The manifest file found
in weblogic.jar lists driver JARs to be loaded when weblogic.jar is loaded (when the
server starts). Therefore, you do not need to add these JDBC drivers to your CLASSPATH. If
you plan to use a third-party JDBC driver that is not installed with WebLogic Server, you must
install the drivers, which includes updating your CLASSPATH with the path to the driver files
and may include updating your PATH with the path to database client files.
If you plan to use a different version of any of the drivers installed with WebLogic Server, you
can replace the driver file in <WL_HOME>\server\lib with an updated version of the file or
add the new file to the front of your CLASSPATH. Copies of the MySQL and Oracle thin
drivers installed with WebLogic Server and other supporting files are installed in
<WL_HOME>\server\ext\jdbc\. There is a subdirectory in this folder for each DBMS. If
you need to revert to the version of the driver installed with WebLogic Server, you can copy
the file from <WL_HOME>\server\ext\jdbc\DBMS to <WL_HOME>\server\lib.
Common Configuration Errors
Invalid URL or missing native libraries:

<Error> <JDBC> <Connection Pool "Node4DS" deployment failed
with the following error:
weblogic.common.ResourceException: Could not create pool
connection. The DBMS driver exception was:
java.sql.SQLException: IO exception: The Network Adapter could
not establish the connection>
Invalid credentials:
<Error> <JDBC> <Connection Pool "Node4DS" deployment failed
with the following error:
Could not connect to 'oracle.jdbc.driver.OracleDriver'.
The returned message is: ORA-01017: invalid username/password;
logon denied. It is likely that the login or password is not
valid.>
In the first example in the slide, the data source activation failed (either during server startup
or later when the data source is first created) due to the fact that no connections could be
established. The message IO Exception likely indicates that the database URL could not be
reached. Therefore, the database is unavailable or the connection properties of the data
source are incorrect. For some type 2 native JDBC drivers, this message can also result if the
required native libraries could not be found by the JVM.
In the second example, the data source activation failed not because a network connection
could not be established, but because the database rejected the connections due to a security
violation. As the error message indicates, the supplied connection credentials were not
granted access.
Configuration Error Examples
If a Maximum Capacity is not indicated, the default is 25

connections (production mode).
If the Reserve Timeout is greater than 0, data source clients
block and wait for a specified time before an error occurs.
If the Reserve Timeout is 1, clients immediately receive an
exception if no connections are available.
When an application requests a connection from a data source, if all connections in the data
source are in use and if the data source has expanded to its maximum capacity, the
application will get a ConnectionUnavailableSQLException. To avoid this, you can configure
the Connection Reserve Timeout value (in seconds) so that connection requests will wait for a
connection to become available. After the Connection Reserve Timeout has expired, if no
connection becomes available, the request will fail and the application will get a
PoolLimitSQLException exception.
If you set Connection Reserve Timeout to 1, a connection request will time out immediately if
there is no connection available. If you set Connection Reserve Timeout to 0, a connection
request will wait indefinitely. The default value is 10 seconds. You may also want to tune this
attribute if you have very high network timeout settings between WLS and your database, and
you want WLS to be more responsive to connectivity issues.
Connection requests that wait for a connection block a thread. If too many connection
requests concurrently wait for a connection and block threads, your system performance can
degrade. To avoid this, you can set the Maximum Waiting for Connection attribute, which
limits the number connection requests that can concurrently wait for a connection. If the
maximum number of requests has been met, a SQLException is thrown when an application
requests a connection.
Insufficient Connection Errors
Poorly implemented applications can starve the data

source of connections.
Cache or hog connections instead of releasing them.
Do not explicitly release connections using close().
To provide a failsafe, WLS can automatically reclaim

leaked connections.
Reclaim connection if not

released after two minutes.
A leaked connection is a connection that was not properly returned to the connection pool in
the data source. To automatically recover leaked connections, you can specify a value for
Inactive Connection Timeout. Find this attribute under the Advanced area of the Connection
Pool tab. WebLogic Server will forcibly return a connection to the data source when there is
no activity on a reserved connection for the number of seconds that you specify. When set to
0 (the default value), this feature is turned off.
Note that the actual timeout could exceed the configured value for Inactive Connection
Timeout. The internal data source maintenance thread runs every five seconds. When it
reaches the Inactive Connection Timeout (for example 30 seconds), it checks for inactive
connections. To avoid timing out a connection that was reserved just before the current check
or just after the previous check, the server gives an inactive connection a second chance.
On the next check, if the connection is still inactive, the server times it out and forcibly returns
it to the data source. On average, there could be a delay of 50% more than the configured
value.
Connection Leaks
Oracle and other databases define a maximum number of

concurrent tasks that it can perform, often called cursors.
A database generates errors when this maximum limit is
exceeded.
Try the following:
Increase the open cursor limit on the database.
Decrease the Maximum Capacity of the data source.
Decrease the Statement Cache Size of the data source.
Sample database cursor error:

java.sql.SQLException:
ORA-01000: maximum open cursors exceeded
Cursors provide a mechanism by which you can iterate over the records in a database. Using
cursors, you can get, put, and delete database records. If a database allows duplicate
records, cursors are the easiest way you can access anything other than the first record for a
given key.
The Statement Cache Size attribute determines the total number of prepared and
callable JDBC statements to cache for each connection in each instance of the data source.
By caching statements, you can increase your system performance. However, you must
consider how your DBMS handles open prepared and callable statements. In many cases, the
DBMS will maintain a cursor for each open statement. This applies to prepared and callable
statements in the statement cache. If you cache too many statements, you may exceed the
limit of open cursors on your database server. For example, if you have a data source with 10
connections deployed on two servers, if you set the Statement Cache Size to 10 (the
default), you may open 200 (10 x 2 x 10) cursors on your database server for the cached
statements.
Database Cursor Considerations
Exception
Description
ConnectionUnavailable
SQLException
Reservation failed because there are no available

connections in the pool.
PoolLimitSQLException
Reservation failed due to some threshold such as

Reserve Timeout or Maximum Waiting.
PoolDisabledSQL
Exception
Reservation failed because data source has been

administratively disabled.
ConnectionDeadSQL
Exception
Reservation failed because connection was tested and

database is unavailable.
ConnectionUnavailableSQLException: Generated when an application request to get a

connection fails because there are currently no connections available in the pool to be
allocated. This is a transient failure and is generated if all connections in the pool are currently
in use. It can also be thrown when connections are unavailable because they are being
tested.
PoolLimitSQLException: Generated when an application request to get a connection fails
due to a configured threshold of the data source, such as HighestNumWaiters,
ConnectionReserveTimeoutSeconds, and so on
PoolDisabledSQLException: Generated when an application request to get a connection
fails because the JDBC Data Source has been administratively disabled
ConnectionDeadSQLException: Generated when an application request to get a
connection fails because the connection test on the reserved connection failed. This typically
happens when the database server is unavailable.
PoolPermissionsSQLException: Generated when an application request to get a
connection fails a security authentication or authorization check. This scenario applies only
when using the Client ID on Connection feature.
Common Connection Errors
Most JDBC drivers support a maximum time limit for SQL

statements that can be set:
By using the data source Statement Timeout attribute
Programmatically by an application
Increase this value if your application requires complex,

long-running database operations.
Refer to your driver

documentation for
supported values.
With the Statement Timeout option on a JDBC data source, you can limit the amount of time
that a statement takes to execute on a database connection reserved from the data source.
When you set a value for Statement Timeout, WebLogic Server passes the time specified to
the JDBC driver using the Statement.setQueryTimeout() method. WebLogic Server will
make the call, and if the driver throws an exception (it is unsupported, for example), the value
will be ignored. In some cases, the driver may silently not support the call or may document
limited support. Oracle recommends that you check the driver documentation to verify the
expected behavior. When Statement Timeout is set to 1, (the default) statements do not time
out. Find the Statement Timeout attribute in the Advanced area of the Connection Pool tab.
Statement Timeout
If a database is not available when the data source is first

initialized:
A pool is created with 0 connections
Data source clients will block and/or receive exceptions
Connections are automatically created after the DB becomes
available
WLS data sources support several database availability

features:
Retry frequency
Connection testing
Multi data sources
Ultimately, even if WebLogic does its best, a connection may fail in the instant after WebLogic
successfully tested it and just before the application uses it. Therefore, every application
should be written to respond appropriately in the case of unexpected exceptions from a dead
connection.
Data Sources and Database Availability
The Retry Frequency data source attribute specifies how

often to attempt creating the initial pool of connections.
Most drivers also support connection properties that limit
how long to wait when creating a connection before
generating an error.
Try to initialize the pool

every five minutes if the
DB is unavailable.
WebLogic JDBC data sources include the Connection Creation Retry Frequency option
(under Advanced) that you can use to specify the number of seconds between attempts to
establish connections to the database. If it is set, and if the database is unavailable when the
data source is created, WebLogic Server attempts to create connections in the pool again
after the number of seconds you specify, and continues to attempt to create the connections
until it succeeds. This option applies to connections created when the data source is created
at server startup, when the data source is deployed, or if the initial capacity is increased. It
does not apply to connections created for pool expansion or to replace a defunct connection
in the pool. By default, Connection Creation Retry Frequency is 0 seconds. When the value is
set to 0, connection creation retries is disabled and data source creation fails if the database
is unavailable. For data sources used by WLS subsystems (persistent store, for example), a
value of 0 can also prevent your server from starting successfully.
When creating database connections in a JDBC data source, if the database is unavailable,
the request may hang until the default system timeout expires. On some systems, this can be
as long as several minutes. The request will hang for each connection in the JDBC data
source. To minimize this hang time, you can specify a login timeout value for the connection.
All WebLogic Type 4 JDBC Drivers support this connection property.
Retry Frequency and Login Timeout
Testing helps avoid scenarios in which an application is given

an unusable connection.
Test before giving a

connection to application.
Test connections
periodically.
Table name or
custom SQL to
test connection
Data sources rely on the Test Reserved Connections feature to know when database
connectivity is lost. Testing reserved connections must be enabled and configured for all of
the data sources within the multi data source. WebLogic Server will test each connection
before giving it to an application. With the failover algorithm, the multi data source uses the
results from connection test to determine when to fail over to the next data source in the multi
data source. After a test failure, the data source attempts to re-create the connection. If that
attempt fails, the multi data source fails over to the next data source.
Test Connections on Reserve: Select this check box to test the database connection
before giving it to your application when your application requests a connection from the
data source.
Test Frequency: Enable periodic background connection testing by entering the
number of seconds between periodic tests.
Test Table Name: Enter the name of a small table to use in a query to test database
connections. The standard query is select count(*) from <table>. Most database
servers optimize this SQL to avoid a full table scan, but it is still a good idea to use the
name of a table that is known to have few rows, or even no rows. If you prefer to use a
different query as a connection test, enter SQL followed by a space and the SQL code
that you want to use to test database connections.
Connection Testing: Review
WLS will trust a connection if it has recently been used by

an application without problems (<10 seconds by default).
For better data source performance, WLS skips the testing
cycle for trusted connections.
For troubleshooting purposes, you can set this attribute to
0 to disable this feature.
Seconds to Trust an Idle Pool Connection: The number of seconds that WebLogic Server
trusts that a connection is still viable and will skip the connection test, either before delivering
it to an application or during the periodic connection testing process, if one or both of these
testing features are enabled. This option is an optimization that minimizes the performance
impact of connection testing, especially during heavy traffic. For multi data sources, this field
is typically set to 0 to ensure higher availability immediately after a database instance
becomes unavailable.
Testing Trusted Connections
Firewalls may close any idle connections between WLS

and the database.
WLS or the database may not immediately know that the
connection has failed, and may try to use it.
Tune the Test Frequency so that tests are performed
before firewall timeouts and the connections remain alive.
Data source
If you have a firewall between the database and WLS, the firewall may close idle connections
after a certain amount of time. When applications then reserve and use this connection, an
error occurs because both the WLS and the database sockets appear to still function
correctly. So either of them may try to write into this socket connection and fail, because it has
been closed by the firewall without notification or error message to either participating parties.
A typical error message in the scenario will be similar to end of file (EOF) reached on
communication channel.
You can use the data source testing functionality to ensure that connections are not closed by
the firewall by periodically executing test SQL. Set the Test Frequency attribute so that
connections are tested at least once during the firewall timeout period. Connections that do
not pass the test will be closed and reopened to re-establish a valid physical database
connection.
Firewall Considerations
To avoid a single point of failure and to achieve greater

scalability, many enterprises employ multiple database
servers.
A multi data source:
Is a pool of data sources
Is used by applications exactly like a standard data source
Transparently provides load balancing or failover across the
member data sources
Requires connection testing
A multi data source is an abstraction of a group of data sources that provides load balancing
or failover processing between the data sources associated with the multi data source. Multi
data sources are bound to the JNDI tree or local application context just like data sources are
bound to the JNDI tree. Applications look up a multi data source on the JNDI tree just like they
do for data sources, and then request a database connection. The multi data source
determines which data source to use to satisfy the request depending on the algorithm
selected in the multi data source configuration: load balancing or failover.
All data sources used by a multi data source to satisfy connection requests must be deployed
on the same servers and clusters as the multi data source. A multi data source always uses a
data source deployed on the same server to satisfy connection requests. Multi data sources
do not route connection requests to other servers in a cluster or in a domain. To deploy a
multi data source to a cluster or server, you select the server or cluster as a deployment
target. When a multi data source is deployed on a server, WebLogic Server creates an
instance of the multi data source on the server. When you deploy a multi data source to a
cluster, WebLogic Server creates an instance of the multi data source on each server in the
cluster.
Multi Data Source: Overview
Data Source A
App
1.
2.
3.
4.
JNDI lookup.
Get connection.
Perform SQL.
Return connection.
Connection
Connection
Connection
App
JDBC Driver
App
Data Source B
Data Source C
A multi data source can be thought of as a pool of data sources. Multi data sources are best
used for failover or load balancing between nodes of a highly available database system,
such as redundant databases or Oracle Real Application Clusters (RAC). Multi data sources
do not provide any synchronization between databases. It is assumed that database
synchronization is handled properly outside of WebLogic Server so that data integrity is
maintained.
You create a multi data source by first creating data sources and then creating the multi data
source using the Administration Console or the WebLogic Scripting Tool (WLST), and then
assigning the data sources to the multi data source.
The data source member list for a multi data source supports dynamic updates. You can
remove a database node and corresponding data sources without redeployment. This
capability provides you the ability to shut down a node for maintenance or shrink a cluster.
Multi Data Source
Synchronize
Multi Data Source: Architecture
Custom Java EE applications often do not directly use data

sources and JDBC.
Instead, the application relies on a persistence engine or
entity manager to automatically synchronize Java data in
memory with a database.
WLS includes a JPA entity manager, but others such as
Oracle TopLink can be used as well.
Application
Application
Entity
manager
Data source
An automatic persistence service such as a JPA entity manager is expected to provide the
following capabilities:
Manage, monitor, and track state changes in entities, including entity relationships
(aggregation/embedded objects), directionality, and cardinality.
Generate the correct vendor-specific SQL dialect.
Provide a general way to map object fields to the table columns in the database.
Support transactions (commit/rollback behavior).
Employ some kind of caching mechanism to aid performance.
Efficiently manage database resources.
JPA does not strictly require the use of a data source. Instead, the entity manager itself can
be configured to perform its own connection pooling by using a supplied JDBC driver.
However, Oracle recommends the use of a data source whenever possible, because it often
provides services (testing, high availability, and so on) beyond what the entity manager can
provide.
Oracle Kodo is the JPA entity manager included with WebLogic Server. However, it is
deprecated and customers are encouraged to consider using Oracle TopLink.
Java Persistence API (JPA): Overview
A deployment descriptor named persistence.xml

indicates the data source to use.
Developers use this file and/or code annotations to map
objects to database entities (tables, columns, and so on).
persistence.xml:
...
<jta-data-source>com.mycompany.jdbc.MyDS</jta-data-source>
UserProfile.java:
@Table(name="USER_PROFILES")
...
@Column(name="USER_FIRST_NAME")
String firstName;
Applications configure entity managers by using deployment descriptors. By default, the JPA
specification uses the file name persistence.xml. This file defines a persistence unit,
which among other things identifies the names of the persistence provider (entity manager)
and data source this application should use. The data source is typically referenced by its
JNDI name.
Annotations are markers or hints to the compiler or virtual machine that a class exhibits a
certain behavior or characteristic. In essence, an annotation is a mechanism for associating
extra metainformation with program elements and allows the compiler or the virtual machine
to extract program behaviors from these annotated elements.
A persistent Java class follows traditional Java semantics. It includes fields and methods. The
class is then annotated as an entity with the appropriate metainformation so that the entity
manager can provide persistence services for it. These annotations identify the parts of the
object to persist. Finally, the object is compiled and then enhanced. The process of byte code
enhancement adds specific code to the object based on the annotations, which can then be
used by the entity manager to detect state changes. In the simplest cases, you can think of
each instance of this class as a row in a database table.
As an alternative to annotations, developers can identify persistent classes and their entity
mappings by using the persistence.xml descriptor.
JPA Configuration: Overview
Each JPA provider includes tools to help monitor and debug its
behavior:
Connection usage
SQL execution
Queries
Caching
These tools may include:
Logs
MBeans
Console applications
Consider the Kodo JPA implementation as an example. Kodo supports various configuration
properties in persistence.xml that control its logging levels and output. Kodo also deploys
a set of MBeans so that your persistence units can be managed and monitored from remote
locations. In addition to using WLST and other JMX tools to work with these MBeans, Kodo
includes its own JMX based graphical console.
Kodos MBeans include:
PoolingDataSource: Has a number of attributes that enable the viewing and editing
of pool settings and the viewing of pool statistics
PreparedStatementCache: For viewing and editing the number of statements to
cache and a number of attributes for viewing cache statistics
QueryCache, DataCache: For viewing and editing the size of the LRU cache and for
clearing the cache
Troubleshooting JPA: Overview
Name three data source runtime attributes.

a. Message Count
b. Average Session Available
c. Active Connections Current Count
d. Current Capacity
e. Number Available
Answer: c, d, e
Quiz
Which data source field is used to help with potential

connection leaks?
a. Statement Timeout
b. Connection Retry Frequency
c. Maximum Session Count
d. Inactive Connection Timeout
e. Statement Cache Size
Answer: d
Quiz
Which data source field may require tuning if a firewall exists

between WLS and your database?
a. Test Frequency
b. Maximum Message Size
c. Profile Connection Usage
d. Capacity Increment
Answer: a
Quiz
In this lesson, you should have learned how to:

Use the console, WLST, and WLDF to monitor and
troubleshoot data sources
Identify common data source configuration issues
Identify connection leaks
Configure a data source to handle database availability
Describe the multi data source architecture
Explain the relationship between JDBC and JPA
Summary
This practice covers the following topics:

Correcting a JDBC configuration error
Troubleshooting a connection leak scenario
Configuring JDBC diagnostic monitors
Configuring data source connection timeouts
Practice 9-1
Investigating JDBC Problems
Troubleshooting JMS

Describe the components of JMS communication
Manage and monitor JMS resources by using the console,
WLST, and WLDF
Investigate JMS issues by using logs and debug flags
Discuss some common JMS configuration issues
Configure JMS to handle large message backlogs
Configure JMS to handle failed and expired messages
Troubleshoot problems with message-driven EJBs
Objectives
Client
App
Server
JNDI
Download a connection factory.
Download a destination.
Create a connection from the factory.
Start a session from the connection.
Create a producer or consumer for destination.
Start a transaction (optional).
Transaction
Manager
Send or receive messages on destination.
JMS server
Commit or rollback the transaction (optional).
1. WebLogic JMS provides two default connection factories that are included as part of any
servers configuration, but administrators can define their own as well. After the
connection factory has been defined, you can look it up by first establishing a JNDI
context with a local or remote server.
2. Look up a destination (queue or topic) by reusing the same JNDI context in the previous
step.
3. Use the connection factory to establish a JMS connection to the server. Depending on
the type of connection factory used, this connection may or may not support distributed
XA transactions. However, all JMS connections support local (JMS-only) transactions.
4. Create one or more sessions for accessing a queue or topic using the connection. A
session and its message producers and consumers can only be accessed by one thread
at a time. Sessions can be transactional or nontransactional, and for consumers the
sessions support various acknowledgement modes. For transactional sessions,
messages are acknowledged when the transaction is committed.
5. Create message producers and/or message consumers for the given destination.
Producers and consumers are also associated with an existing JMS session. Message
consumers can receive messages synchronously (polling) or asynchronously
(notifications).
JMS: Review
Note: When finished, any previously created objects should be closed as soon as possible to
free up client and server resources.
6. JMS clients can participate in a distributed or local transaction.

7. Message producers can send one or more messages to the corresponding destination.
Message consumers can receive one or more messages.
8. For producers, a transaction begins and some operations (such as sending messages)
are performed. If the transaction commits, all the messages are sent to the destination. If
the transaction rolls back, none of the messages arrive at the destination. For
consumers, if the transaction commits, the processed messages are acknowledged and
removed from the destination. If the transaction rolls back, the messages stay in the
destination.
JMS Module
JNDI
Queue
Server
Subdeployment
JMS Server
Subdeployment
JMS Server
Queue
Topic
Connection
Factory
Persistent Store
A JMS destination identifies a queue (point-to-point) or topic (publish/subscribe) that can be

used by applications to communicate asynchronously via messages.
Connection factories are resources that enable JMS clients to create JMS connections.
WebLogic JMS provides preconfigured default connection factories that can be enabled or
disabled on a per-server basis. Otherwise, you can configure one or more connection
factories to create connections with predefined options that better suit your application.
After you create a JMS module, you can configure resources for the module, including standalone queues and topics, distributed queues and topics, connection factories, and JMS
templates.
A JMS servers primary responsibility for its destinations is to maintain information about the
persistent store that is used for any persistent messages that arrive on the destinations and to
maintain the states of the durable subscribers created on the destinations. JMS servers also
manage message paging on destinations. Multiple JMS servers can be targeted to the same
Oracle WebLogic Server instance.
While a JMS module is targeted to one or more servers, all of its resources need not be
targeted homogenously to these servers. Instead, modules can define subdeployments so
that different resources are targeted to different server instances or JMS servers.
WebLogic JMS Configuration: Review
Resource
Description
Destination
(Queue or Topic)
Used by applications to produce and consume messages
Connection
Factory
Used by applications to connect to a JMS provider. Each can

be configured with different default communication settings.
JMS Module
A group of related JMS resources, including destinations and

factories, which is targeted to one or more servers
JMS Server
Settings that configure how messages should be processed

and persisted; targeted to a single server
Subdeployment
A reusable list of target servers or JMS servers that can be

associated with a resource in a JMS module
Persistent Store
A file or database assigned to a JMS server to provide high

availability
WebLogic administrators typically use the Administration Console or the WebLogic Scripting
Tool (WLST) to create and deploy (target) JMS modules and to configure the modules
configuration resources (such as queues) and topic connection factories. JMS modules that
you configure in this way are considered system modules. JMS system modules are owned
by the administrator, who can at any time add, modify, or delete resources. System modules
are globally available for targeting to servers and clusters configured in the domain and
therefore are available to all applications deployed on the same targets and to client
applications.
JMS configuration resources can also be managed as deployable application modules, similar
to standard Java EE descriptor-based modules. JMS application modules can be deployed
with a Java EE application either as a packaged module in which the resources in the module
are optionally made available to only the enclosing application (that is, application-scoped), or
as a stand-alone module that provides global access to the resources defined in that module.
Unlike destinations, connection factories can be targeted to an entire WebLogic Server
instance as well as a JMS server. In this case, the connection factory is available for use with
destinations on all JMS servers on that server instance.
WebLogic JMS Configuration: Review
Messages produced within a transaction are inserted on

the destination only after a commit.
Messages consumed within a transaction are
acknowledged and removed only after a commit.
Transactions can either be:
Local (JMS only)
Distributed using the WebLogic transaction manager
(requires XA connection factories)
Start transaction
Producer
Start transaction
JMS
Commit/rollback
Consumer
Commit/rollback
JMS clients can participate in a distributed or local transaction. There are two scenarios:
On the producer side, a transaction begins and some operations, such as sending
messages, are performed. If the transaction commits, all the messages are sent to the
destination. If the transaction rolls back, none of the messages arrive at the destination.
On the consumer side, a transaction begins and some operations, such as processing
messages, are performed. If the transaction commits, the processed messages are
acknowledged and the server removes them from the destination. If the transaction rolls
back, the messages stay in the destination.
JMS Transactions: Review
Using the console or WLST, administrators have the ability to:

View or delete messages
Publish test messages
Move messages to another destination
Export message contents to a file
Import message contents from a file
Pause and resume message processing
Migrate JMS services to another server (cluster only)
The WebLogic JMS message monitoring and management features allow you to create new
messages, delete selected messages, move messages to another queue, export message
contents to another file, import message contents from another file, or drain all the messages
from the queue.
A JMS-related service can be migrated with its hosting migratable target to another server
member within a cluster. This includes both scheduled migrations as well as manual
migrations in response to a server failure within the cluster. For example, a JMS server, all of
its destinations, and its associated persistent store can migrate with their migratable target to
a healthy server if the current hosting server fails. JMS-related services include JMS servers,
SAF agents, path services, and custom stores. You can migrate all migratable targets at once
or on a target-by-target basis.
JMS Management: Overview
1
2
3
1. In the JMS Modules table, click the JMS module that contains the configured
destination. In the selected JMS modules Summary of Resources table, click the
queue or topic that you want to monitor or manage.
2. Click the Monitoring tab.
3. Select the check box next to the queue or topic and click Show Messages to access the
queues JMS Messages table.
4. Click the ID link for a message in the queue to open the View Contents page, where you
can view the contents of a JMS message. You can optionally filter the list of messages
by using the Message Selector field (not shown). Message selectors are Boolean
expressions based on message header or property values.
The available columns in the JMS Messages table include:
State String: The current state of a message
Type: The JMS message type, such as BytesMessage, TextMessage, StreamMessage,
ObjectMessage, MapMessage, or XMLMessage
Priority: An indicator of the level of importance or urgency of a message, with 0 as the
lowest priority and 9 as the highest
Message Size: The size of a message in bytes
Console JMS Management
Delete all messages from a destination:

domainRuntime()
queue = getMBean('/ServerRuntimes/serverA/JMSRuntime/
serverA.jms/JMSServers/MyJMSServer/Destinations/
OrderJMSModule!OrderQueue')
queue.deleteMessages('')
Pause/resume the sending of messages to consumers for a destination:

queue = ...
queue.pauseConsumption()
...
queue.resumeConsumption()
Refer to the JMSDestinationRuntimeMBean in the documentation. The available

operations include:
deleteMessages()
getMessage()
getMessages()
importMessages()
moveMessages()
pauseConsumption()/pauseInsertion()/pauseProduction()
resumeConsumption()/resumeInsertion()/resumeProduction()
JMS Management: WLST Examples
JMSRuntime
*
JMSServerRuntime
*
JMSConnectionRuntime
JMSDestinationRuntime
JMSSessionRuntime
JMSDurableSubscriberRuntime
JMSProducer
Runtime
*
JMSConsumer
Runtime
In the diagram in the slide, the asterisk (*) indicates a one-to-many relationship. For example,
if multiple JMS servers are deployed to the same server, that server will host multiple
instances of JMSServerRuntimeMBean in its MBean hierarchy.
JMSRuntimeMBean provides runtime attributes for the entire server instance, including the
current and total number of JMS connections. For each connection, an instance of
JMSConnectionRuntimeMBean is available. This MBean in turn has references to one or
more JMSSessionRuntimeMBeans, although in most scenarios a connection will only have
a single session. JMSSessionRuntimeMBean provides attributes such as the number of
pending, received, and sent messages.
Alternatively, to obtain runtime statistics for an entire JMS server or one of its destinations,
access the JMSServerRuntimeMBean and JMSDestinationRuntimeMBean objects in
the MBean hierarchy. These MBeans include attributes such as the number of active,
received, pending, and paged out messages.
JMS Runtime MBean Hierarchy
Get JMS server runtime statistics:

serverRuntime()
jms = getMBean('/JMSRuntime/ServerC.jms/JMSServers/HRJMS')
print 'Messages: ', jms.getMessagesPendingCount()
print 'Bytes: ', jms.getBytesPendingCount()
print 'Transactions: ', len(jms.getPendingTransactions())
Get JMS destination runtime statistics:
serverRuntime()
q =
getMBean('/JMSRuntime/ServerC.jms/JMSServers/HRJMS/Destination
s/BenefitsJMSModule!SyncBenefitsQueue')
print 'Curr Messages: ', jms.getMessagesCurrentCount()
print 'Max Messages: ', jms.getMessagesHighCount()
print 'Consumers: ', jms.getConsumersCurrentCount()
Refer to the documentation for a list of all available attributes and operations. Some additional
JMSServerRuntimeMBean attributes are the following:
BytesCurrentCount
BytesHighCount
BytesPageableCurrentCount
BytesReceivedCount
BytesThresholdTime
ConsumptionPaused
DestinationsCurrentCount
HealthState
InsertionPaused
MessagesReceivedCount
MessagesThresholdTime
ProductionPaused
Transactions
JMS Monitoring: WLST Examples
Diagnostic images include the following XML files:

JMS.img captures statistics from all configuration and
runtime JMS MBeans along with details about each
message currently on the server.
PERSISTENT_STORE.img captures general usage
statistics for the servers file store.
JMS.img
<Queue name="BenefitsJMSModule!SyncBenefitsQueue" ...
<Statistics messageCurrent="10" messagesHigh="120"
messagesLow="0" messagesPending="6" ... />
<Messages currentCount="10">
<MessageElement ... size=976 persistent=true ...
</Queue>
A diagnostic image is a heavyweight artifact meant to serve as a server-level state dump for
the purpose of diagnosing significant failures. It enables you to capture a significant amount of
important data in a structured format and then to provide that data to support personnel for
analysis. Because it is an artifact intended primarily for internal consumption, the image
contents are not documented in detail and are subject to change.
JMS Diagnostic Image Contents
WebLogic can capture all JMS events in a separate server

log file (which, by default, is the following):
<server>/logs/jmsServers/<jmsServer>/jms.messages.log
JMS logging:
Is configured at both the JMS server and destination levels
Is enabled or disabled for individual destinations
Records event types (produced, consumed, expired, and so
on)
Records message headers or payload
Supports file rotation
Message lifecycle logging provides an administrator with easy access to information about the
existence and status of JMS messages from the JMS server viewpoint. A record is added to
the JMS servers message log file each time a message meets the conditions that correspond
to a basic message lifecycle event.
The lifecycle events that trigger a JMS message log entry are the following:
Produced: A message enters a JMS server.
Consumed: A message leaves a JMS server.
Removed: A message is manually deleted from a JMS server.
Expired: A message reaches the expiration time stored on the JMS server. This event
is logged only once per message even though a separate expiration event occurs for
each topic subscriber who received the message.
Retry Exceeded: A message has exceeded its redelivery retry limit. This event may be
logged more than one time per message, as each topic subscriber has its own
redelivery count.
Consumer Created: A JMS consumer is created for a queue, or a JMS durable
subscriber is created for a topic.
Consumer Destroyed: A JMS consumer is closed, or a JMS durable subscriber is
unsubscribed.
JMS Message Logging
Enable logging for the current

destination (queue or topic).
Include standard JMS message headers

in log entries (all or specific ones).
Include WLS message headers in log
entries (all or specific ones).
Include message
payload in log entries.
Each record added to the log includes basic information such as the time stamp, message ID,
and correlation ID for the subject message. You can also configure each JMS destination to
include additional information such as the message type and user properties.
Enable Message Logging: Select whether to log information about the life cycle of
messages on this destination.
All Headers: Include all JMS header fields in the log.
Headers: Include only a subset of JMS header fields in the log by moving them from the
Available column to the Chosen column.
All Properties: Include all WLS properties in the message log, such as JMSXUserID
and JMSXDeliveryCount.
Properties: Include only a subset of WLS properties in the log by moving them from the
Available column to the Chosen column.
User-Defined Properties: Specify any custom message properties to include in the log.
All Body: Include all of the message body in the log entries.
Configuring JMS Logging
Flag
Description
DebugJMSFrontEnd
Trace high-level message production events.
DebugJMSBackEnd
Trace high-level message consumption events.
DebugJMSCommon
Trace some events that are common to both producers

and consumers (such as security).
DebugJMSModule
Trace the life cycle of JMS module resources.
DebugJMSBoot
Trace the JMS server-initialization process.
DebugJMSSAF
Debug the JMS store-and-forward feature.
Server log messages with DebugJMSBackEnd enabled:

<Debug> <JMSBackEnd> <BEA-000000> <Got a stop request on the
consumer>
<Debug> <JMSBackEnd> <BEA-000000> <Removing pending messages
through MAX>
DebugJMSBackEnd (scope weblogic.jms.backend) prints information for debugging the

JMS back end (including some information used for distributed destinations and JMS SAF).
DebugJMSFrontEnd (scope weblogic.jms.frontend) prints information for debugging
the JMS front end (including some information used for multicast).
DebugJMSCommon (scope weblogic.jms.common) prints information for debugging JMS
common methods (including some information from the client JMS producer).
DebugJMSConfig (scope weblogic.jms.config) prints information related to JMS
configuration (backend, distributed destinations and foreign servers).
DebugJMSBoot (scope weblogic.jms.boot) prints some messages at boot time
regarding what store the JMS server is using and its configured destinations.
DebugJMSPauseResume (scope weblogic.jms.pauseresume) prints information about
(backend) pause/resume destination operations.
DebugJMSModule (scope weblogic.jms.module) prints a lot of information about JMS
module operations and message life cycle.
DebugJMSSAF (scope weblogic.jms.saf) prints information about JMS SAF (store-andforward) destinations.
JMS Debug Flags
In addition to simple message types (bytes, text, XML,

map), WebLogic JMS supports messages that contain
arbitrary Java objects.
The same version of the class must be available on both
the producer and consumer to avoid Class Not Found and
Class Cast errors.
You can monitor object messages in WLS if the class
defines a textual representation (toString()).
WebLogic JMS messages support payloads of the following types:

TextMessage: Single text string, which can be unformatted or the contents of some
structured document
BytesMessage: Stream of uninterpreted bytes that must be understood by the sender
and receiver. The access methods for this message type are stream-oriented readers
and writers.
MapMessage: Set of name-value pairs in which the names are strings and the values
are Java primitive types. It is conceptually similar to a database table with a single row.
Pairs can be read sequentially or randomly by specifying a name.
ObjectMessage: Single serializable Java object
StreamMessage: Similar to a BytesMessage. But it also supports reading and writing
Java primitive types to and from the stream.
XMLMessage (WLS only): Unlike a simple TextMessage, the XMLMessage type
includes features to facilitate message filtering based on XML elements and attributes.
Message Type Considerations
WebLogics JMS configuration is very flexible but provides

opportunities for errors.
Typical causes of problems include:
Targeting connection factories to JMS servers using
subdeployments instead of to entire servers
Targeting distributed destinations to entire servers instead
of to JMS servers using subdeployments
Using multiple subdeployments within the same module
Trying to configure JMS migration without a cluster
Trying to convert a single-server domain to a cluster
Configure a JMS server on each WebLogic server. Configure the JMS server to reference a
persistent store (default or custom). Target the JMS server to the same target that was used
for the store. Multiple JMS servers can reference the same store.
Create a system module and target it to a single cluster (if using clusters) or a single
WebLogic Server instance. You must always target the module even when leveraging
subdeployments. It is almost always preferable to use JMS system modules instead of
deployable JMS application modules. Create exactly one subdeployment per module.
Subdeployments are sometimes referred to as advanced targeting on the Administration
Console. A single subdeployment aids simplicityit is much easier for third parties to
understand the targeting, and it reduces the chances of making configuration errors. If a
single subdeployment is not sufficient, create two modules.
Populate the subdeployment only with JMS servers and not with WebLogic servers. Include
only the JMS servers that you want to host destinations. This ensures that when the JMS
resources are configured, they are targeted to the correct JMS servers. For modules that
support nondistributed destinations, the subdeployment must reference only a single JMS
Server. If you have a mix of distributed and nondistributed destinations, use two modules
each with its own subdeployment.
Server
JMS Module
Destination
Destination
Subdeployment
JMS Server
Connection
Factory
Destinations cannot be
targeted to an entire server.
JMS Module
Server
Destination
Destination
Subdeployment
Connection
Factory
Subdeployment
JMS Server
Connection factories do not

require subdeployments.
Avoid default targeting, WebLogic server targeting, and cluster targeting with destinations.
Instead use advanced targeting (subdeployment targeting) for destinations and ensure that
the subdeployment references only JMS servers or SAF agents. This applies to all destination
types, including nondistributed, distributed, and imported.
Create and use custom connection factories instead of using default connection factories.
Default connection factories are not tunable. In most cases, you can use default targeting with
connection factories because default targeting causes the resource to inherit the modules
target. One exception may be factories that are only used by remote clients, which can use a
subdeployment target.
As a best practice, for domain configurations that include JMS resources, do not attempt to
update an existing nonclustered domain to use a cluster. This approach often leads to
configuration mistakes. Instead, create a completely new domain configuration with a cluster
and re-create your JMS resources. Also, keep in mind that the JMS migration feature (through
migratable targets) is only supported with clusters. If you are not using a cluster, you may
want to reconsider and use a cluster of size one. This approach gives you the option of
completely restarting a JMS server on a running server, such as for maintenance or
troubleshooting purposes. This approach also helps future-proof your application, as it
simplifies the process of expanding from a single server to multiple servers. It is
recommended that you always target to migratable targets when available (instead of direct
server targets).
Remote JMS clients require an appropriate JAR file in their

classpaths to support specific JMS features.
WLS client libraries support connections based on a single
URL, a list of addresses to try, a range of ports to try, or a
DNS name.
Client Libraries
Support
wlthint3client.jar
T3 protocol, subset of full client; recommended
wlfullclient.jar
T3 protocol, includes JMS SAF and JMS C

clients; larger footprint
weblogic.jar
Full WLS install; allows server-side operations
wlclient.jar,
wljmsclient.jar
All WLS JMS features; slower performance
The WebLogic thin T3 client jar is a lightweight alternative to the full client. It is a Java RMI
client that uses Oracles proprietary T3 protocol to communicate with WebLogic Server and
therefore has good performance. It provides access to JMX, JNDI, and EJB resources in
WebLogic Server. It supports transactions via JTA. It also features WLS-specific JMS features
such as Unit of Order, XML messages, XML messages, and automatic client reconnect. It is
the recommended option for most remote JMS clients.
The WebLogic full client also uses the T3 protocol. It is the largest JAR file, but it has the most
features. Features added that are not in the thin client include support for JMS SAF clients
and JMS C clients. The full client also provides IIOP support.
If WebLogic Server is installed, a client may be written and access weblogic.jar. Using
this enables server-side operations such as use of the ejbc compiler, application deployment,
and WLST applications that invoke server-side operations.
Other JMS thin clients, wlclient.jar and wljmsclient.jar, provide full WebLogic JMS
functionality, EJB and JNDI access, and transaction capability. They have lower performance
because they do not use T3.
JMS Client Libraries
Large numbers of pending messages can cause the server

to run out of memory and fail.
A JMS quota is a policy that limits the maximum number
and/or size of messages that await delivery.
Servers reject new messages after a quota has been met.
Destinations that do not have an assigned quota share any
quota configured on the host JMS server.
500 messages
maximum
50 MB maximum
A quota is a named configurable JMS module resource, although quota parameters are also
available for entire JMS servers as well. It defines a maximum number of messages and
bytes, and is then associated with one or more destinations and is responsible for enforcing
the defined maximums. Multiple destinations referring to the same quota share available
quota according to the sharing policy for that quota resource.
The Quota parameter of a destination is used to associate it with a quota resource to enforce
a quota for the destination. This value is dynamic, so it can be changed at any time. In some
cases, there will be destinations that do not configure quotas. JMS server quotas allow JMS
servers to limit the resources used by these quota-less destinations. All destinations that do
not explicitly set a value for the Quota attribute share the quota of the JMS server where they
are deployed. However, note that quota resources are not assigned to JMS servers as with
destinations. Instead, quota parameters are defined as part of the JMS server itself.
To begin tuning your quota settings, assume that an average message requires 512 bytes of
storage, and configure the Bytes Maximum attribute to reflect the maximum memory
consumption that your server can tolerate. This value will vary depending on how many other
services and applications are competing for memory on the same server.
You can also help control JMS memory usage by enabling WebLogics message compression
feature.
Out-of-Memory Errors and Quotas
1
2
Quotas for all destinations

running on this JMS server
To set quotas on an entire JMS server:

1. Edit an existing JMS server
2. Click Thresholds and Quotas.
3. Edit the following attributes and then click Save:
- Bytes Maximum: The total number of bytes that can be stored by destinations
that use this JMS server
- Messages Maximum: The total number of messages that can be stored by
destinations that use this JMS server
Configuring a JMS Server Quota
Policy that can be

assigned to destinations
Pool quota if targeted to

multiple destinations
JMS quotas are used to control the allotment of system resources available to destinations.
For example, you can specify the number of bytes or messages that a destination is allowed
to store, or specify whether the quota can be shared be all destinations that refer to it. A value
of 0 means that no messages can be placed on a destination without exceeding the quota. A
value of 1 prevents WebLogic Server from imposing a limit.
To create a JMS quota:
1. In the JMS Modules table, click the name of JMS module in which you want to configure
the quota resource. Then click the New button in the Summary of Resources table.
2. Select Quota from the list of JMS resource types. Then click Next.
3. Edit the following attributes and click OK:
- Name: The name of this quota object
- Bytes Maximum: The total number of bytes that can be stored in a destination
that uses this quota
- Messages Maximum: The total number of messages that can be stored in a
destination that uses this quota
- Shared: Indicates whether this quota is shared by multiple destinations that refer
to it. If this attribute is disabled, the quota object behaves as a template.
Creating a Destination Quota
To avoid running out of memory under peak loads, JMS

servers move in-progress messages to virtual memory.
By default, paging is triggered when messages consume
more than 1/3 of maximum server memory (heap).
Use the Message Buffer Size field to override the default
memory trigger.
JMS Server
Messages in
server memory
Messages in virtual
memory (file or
database)
With the message paging feature, JMS servers automatically attempt to free up virtual
memory during peak message load periods. This feature can greatly benefit applications with
large message spaces. Message paging is always enabled on JMS servers, and so a
message paging directory is automatically created without having to configure one. You can,
however, specify a different directory if you want.
JMS message paging saves memory for persistent messages, as even persistent messages
cache their data in memory. If a JMS server is associated with a file store (either user-defined
or the servers default store), paged persistent messages are generally written to that file
store, while nonpersistent messages are always written to the JMS servers paging directory.
If a JMS server is associated with a JDBC store, then both paged persistent and
nonpersistent messages are always written to the JMS servers paging directory.
However, a paged-out message does not free all the memory that it consumes, because the
message header (with the exception of any user properties that are paged out along with the
message body) remains in memory for use with searching, sorting, and filtering. Queuing
applications that use selectors to select paged messages may show severely degraded
performance because the paged out messages must be paged back in.
Message Paging
A backlog of pending messages may indicate that:

Consumers are failing
Consumers cannot handle the message load
Consumers are not acknowledging messages or
completing transactions
Potential solutions include:
Quota blocking policies
Flow control
Removing abandoned topic subscriptions
Server or application performance tuning
When message senders inject messages faster than consumers, messages accumulate into
a message backlog. Large backlogs can be problematic for a number of reasons. First, this
often indicates that consumers may not be capable of handling the incoming message load,
are failing, or are not properly load balanced across a distributed queue. This scenario can
lead to out-of-memory errors on the server, which in turn prevent the server from doing any
work. Furthermore, this can lead to high garbage collection (GC) overhead. A JVMs GC
overhead is partially proportional to the number of live objects in the JVM.
Most administrative solutions to this problem involve slowing down or stopping message
processing. While this approach may help to some degree, there are several potential
problems. For example, it puts back-pressure on the downstream flow that is calling the
producer. Sometimes the downstream flow cannot handle this back-pressure, and a hard-tohandle backlog develops behind the producer. The location of the backlog depends on what
is calling the producer. For example, if the producer is being called by a servlet, the backlog
might manifest as packets accumulating on the incoming network socket or network card.
Blocking calls on server threads can also lead to thread starvation, too many active threads,
or even deadlocks.
It is highly recommended that you always configure message count quotas. Quotas help
prevent large message backlogs from causing out-of-memory errors, and WebLogic JMS
does not set quotas by default.
Too Many Pending Messages
By default, multiple producers waiting for available quota

are served in the order in which they attempted to send
messages (FIFO: first in, first out).
Alternatively, smaller messages can be serviced before
larger ones as space becomes available.
Quota resources and JMS servers provide a blocking send policy attribute to control how
message producers compete for space on a destination that has exceeded its message
quota. For quota resources, the attributes name is Policy, while on JMS servers you use the
Blocking Send Policy attribute found on the Configuration > Thresholds and Quotas tab.
Select one of the following options:
FIFO: All send requests for the same destination are queued up one behind the other
until space is available. No send request is permitted to complete when another send
request is waiting for space before it.
Preemptive: A send operation can preempt other blocking send operations if space is
available. That is, if there is sufficient space for the current request, that space is used
even if there are previous requests waiting for space.
Quota Blocking Policies
A threshold:
Can be configured on a destination or entire JMS server
Defines a lower and upper boundary based on the number or
size of messages
Can trigger log messages and/or flow control
Flow control:
Is enabled on a connection factory
Is activated when the upper threshold is exceeded and
deactivated for the lower one
Introduces delays to slow down
producers and keep the server from
becoming swamped
High and low thresholds are used to signal periods of load and/or congestion. Administrators
can set thresholds for both messages and bytes, and on both JMS servers and destinations.
Exceeding these thresholds will trigger events, such as generating log messages and starting
message flow control.
With the Flow Control feature, you can direct a JMS server or destination to slow down
message producers when it determines that it is becoming overloaded. Specifically, when
either a JMS server or its destination exceeds its specified byte or message threshold, the
server or destination becomes armed and instructs producers to limit their message flow
(messages per second). Producers will limit their production rate based on a set of flow
control attributes configured for producers via the JMS connection factory.
As producers are slowed down, the threshold condition gradually corrects itself until the
server (or destination) is unarmed. At this point, a producer is allowed to increase its
production rate, but not necessarily to the maximum possible rate. In fact, its message flow
continues to be controlled (even though the server/destination is no longer armed) until it
reaches its prescribed flow maximum, at which point it is no longer flow controlled.
Thresholds and Flow Control
Define thresholds for a

specific destination.
Define thresholds for an

entire JMS server.
For all threshold attributes, the default value is java.lang.Long.MAX_VALUE, which

disables logging and flow control events for the destination or JMS server.
To configure thresholds for a destination:
1. In the JMS Modules table, click the JMS module that contains the configured
destination. In the selected JMS modules Summary of Resources table, click the
queue or topic that you want to configure.
2. Click Configuration > Thresholds and Quotas. The available attributes are Bytes
Threshold High, Bytes Threshold Low, Messages Threshold High, Messages Threshold
Low, and Maximum Message Size.
To configure thresholds for a JMS server:
1. In the JMS Servers table, select a JMS server
2. Click Configuration > Thresholds and Quotas. The available threshold attributes are the
same as for destinations.
Configuring Thresholds
Flow control settings for

a connection factory
Maximum number of
messages per second
Minimum number of messages

that producer can be slowed to
To configure flow control for a JMS connection factory, use its Configuration > Flow Control
tab. Edit the following fields:
Flow Maximum: The maximum number of messages per second allowed for a producer
that is experiencing a threshold condition. When a producer is flow-controlled it will
never be allowed to go faster than this rate. If a producer is not currently limiting its flow
when a threshold condition is reached, the initial flow limit for that producer is set to
Flow Maximum. If a producer is already limiting its flow when a threshold condition is
reached (the flow limit is less than the maximum), the producer will continue at its
current flow limit until the next time the flow is evaluated.
Flow Minimum: The minimum number of messages per second allowed for a producer
that is experiencing a threshold condition. This is the lower boundary of a producers
flow limit. That is, WebLogic will not further slow down a producer whose message flow
limit is at its Flow Minimum.
Flow Interval: The adjustment period of time, in seconds, when a producer adjusts its
flow from the Flow Maximum number of messages to the Flow Minimum amount (or vice
versa)
Flow Steps: The Flow Interval adjustment period is divided into a number of steps. For
example, 60 seconds divided by six steps results in 10 seconds per step.
Tuning Flow Control
Lost messages are successfully added to a destination but

are later removed without being delivered to consumers.
Expired messages
JMS system failure and nonpersistent messages
Insufficient retry settings
Message delay settings
Nondurable topic subscriptions
A persistent message is guaranteed to be delivered once and only once. The message
cannot be lost due to a JMS provider failure, and it must not be delivered twice. It is not
considered sent until it has been safely written to a file or database. WebLogic JMS writes
persistent messages to a WebLogic persistent store (disk-base file or JDBC-accessible
database) that is optionally targeted by each JMS server during configuration.
Nonpersistent messages are not stored. They are guaranteed to be delivered at most once
(unless there is a JMS provider failure, in which case messages may be lost) and must not be
delivered twice. If a connection is closed or recovered, all nonpersistent messages that have
not yet been acknowledged will be redelivered. After a nonpersistent message is
acknowledged, it will not be redelivered.
For durable subscriptions, WebLogic JMS stores a message in a persistent file or database
until the message has been delivered to the subscribers or has expired, even if those
subscribers are not active at the time that the message is delivered. Durable subscriptions are
supported for publish/subscribe (topic) messaging only.
Lost Messages
A TTL is assigned to all new messages in one of the

following ways:
Programmatically
Using connection factory defaults
Using destination overrides
By default, expired messages are removed.

JMS log entries include message expiration events.
When sending a message, you can optionally specify the delivery mode, priority, and time-tolive (in milliseconds) values. If not specified, these attributes are set to the connection factory
configuration attributes. These same attributes can also be administratively overridden by the
target destination.
To set the default TTL on a connection factory, use the Configuration > Default Delivery tab.
To override the current TTL for all messages on a destination, use the destinations
Configuration > Overrides tab.
Messages are not necessarily removed from the system at their expiration time, but they are
removed within a user-defined number of seconds. The smaller the window, the closer the
message removal is to the actual expiration time. Edit a JMS server. Then, using the Scan
Expiration Interval field, enter the amount of time (in seconds) that you want the JMS server to
pause between its cycles of scanning its destinations for expired messages to process. To
disable active scanning, enter a value of 0 seconds. Expired messages are passively
removed from the system as they are discovered.
Time to Live (TTL)
Destinations can perform one of the following actions for

expired and failed messages:
Add an entry to the server log file.
Move or redirect the message to another destination.
1
2
To configure expiration policies for a destination:

1. Edit an existing topic or queue in a JMS module.
2. Click the Configuration > Delivery Failure tab.
3. Edit the following fields.
Expiration Policy: The valid expiration policies are:
- Discard (default): Removes expired messages from the messaging system. The
removal is not logged and the message is not redirected to another location.
- Log: Removes expired messages from the system and writes an entry to the
server log file indicating that the messages have been removed from the system
- Redirect: Moves expired messages from their current location to the Error
Destination indicated. The message retains its body and all of its properties. The
message also retains all of its header fields, but all property overrides associated
with the error destination are applied to the redirected message. In addition,
quotas are ignored when redirecting expired messages.
Expiration Logging Format: Specify any additional information that should be logged,
such as all message headers (%header%) and all message properties
(%properties%).
Expiration Policies
All new messages must indicate whether or not they are

eligible for persistence.
This delivery mode is set in one of the following ways:
Programmatically
After you configure a connection factory, you can define various default message delivery
parameters that also override any values configured by the client. For example, if a client
does not specify certain delivery parameters, the value of those parameters can be controlled
with the delivery parameters on this page:
1. Select the connection factory.
2. Click Configuration > Default Delivery.
3. In Default Delivery Mode, select the delivery mode (Persistent or Non-Persistent)
assigned to all messages sent by a producer using this connection factory.
After you create a queue or topic, you can define message delivery override values that can
override those specified by a message producer.
1. Select the queue or topic.
2. Click Configuration > Overrides.
3. In Delivery Mode Override, select the delivery mode (Persistent, Non-Persistent, or NoDelivery) assigned to all messages that arrive at the queue. A value of No-Delivery
specifies that the producers delivery mode will not be overridden. This attribute is
dynamically configurable, but only incoming messages are affected; stored messages
are not affected.
Delivery Mode
For temporary consumer problems, destinations can retry

failed messages:
After waiting some amount of time
A specific number of times
These redelivery parameters are set programmatically or

through connection factory and destination settings.
JMS log entries include redelivery events.
If JMS immediately redelivers the message, the error condition may not be resolved and the
application may still not be able to handle the message. However, if an application is
configured for a redelivery delay, then when it rolls back or recovers a message, the message
is set aside until the redelivery delay has passed, at which point the messages are made
available for redelivery.
You can specify a limit on the number of times that WebLogic JMS will attempt to redeliver a
message to an application. Once WebLogic JMS fails to redeliver a message to a destination
for a specific number of times, the message can be redirected to an error destination that is
associated with the message destination. If the redelivery limit is configured but no error
destination is configured, persistent and nonpersistent messages are simply deleted when
they reach their redelivery limit.
A session inherits the redelivery delay and limit from its connection factory when the session
is created (Configuration > Default Delivery tab). However, the application that creates the
session can then override these connection factory settings by using WebLogic-specific
extensions to the javax.jms.Session interface. Regardless of the redelivery delay and
limit that are set on the session, the destination where a message is being rolled back or
recovered can override the setting by using the Configuration > Delivery Failure tab.
Message Redelivery
New messages can be delayed before being eligible for

consumption:
A specific amount of time
Until a given time of day
A TTD is assigned to messages in one of the following ways:
Programmatically
JMS messages include a Time to Deliver header field that defines the earliest absolute time
at which the message can be delivered. That is, the message is held by the messaging
system and is not given to any consumers until that time. If the specified Time to Live (TTL)
value is less than or equal to the specified TTD value, message delivery succeeds. However,
the message is then silently expired.
When a producer is created, it inherits its Time to Deliver attribute, expressed in
milliseconds, from the connection factory used to create the connection that the producer is a
part of. Regardless of the value that is set on the producer, the destination to which a
message is being sent or published can override the setting. An administrator can set the
Time-To-Deliver Override attribute on a destination in either a relative or a scheduled string
format.
Examples of scheduled TTD values:
0 0 0,30 * * * * : Exact next nearest half-hour
* * * 9-16 * * * : Between 9 AM and 5 PM (09:00:00 to 16:59:59)
* * * 13-16 * * 0: Between 1 PM and 5 PM on Sunday (13:00:00 to 16:59:59;
day 0 = Sunday)
Time to Deliver (TTD)
By default, inactive or failed topic consumers will miss any

messages produced in their absence.
If consumers register as durable and provide a unique
client ID, messages will be saved across connections.
Topic
Topic
Topic
ID
ID
Consumer
(Durable Subscriber)
Consumer
Consumer
Support for durable subscriptions is a feature that is unique to the publish/subscribe

messaging model, so client IDs are used only with topic connections; queue connections also
contain client IDs, but JMS does not use them.
Nondurable subscriptions only last for the lifetime of their current session with the JMS server.
That is, a client will only see the messages published on a topic while it is active. If the
subscriber is not active, it is potentially missing messages that are published on its topic. By
default, subscribers are nondurable.
A durable subscriber, on the other hand, registers a durable subscription with a unique
identity that is retained by JMS. Subsequent subscriber objects with the same identity resume
the subscription in the state it was left in by the previous subscriber. If there is no active
subscriber for a durable subscription, JMS retains the subscribers messages until they are
received by the subscriber or until they expire.
Durable Subscriber Review
Delete abandoned
subscriptions.
Show saved
message.
Manually create
a subscription.
Is subscriber
connected?
Check WebLogic statistics for topics with high pending counts. This usually indicates that
there are topic subscriptions that are not being serviced. There may be a slow or
unresponsive consumer client that is responsible for processing the messages. Or it is
possible that a durable subscription may no longer be needed and should be deleted, or the
messages may be accumulating due to delayed distributed topic forwarding. You can check
statistics for individual durable subscriptions on the Administration Console. A durable
subscription with a large backlog may have been created by an application but never deleted.
Unused durable subscriptions continue to accumulate topic messages until they are either
administratively destroyed or unsubscribed by a standard JMS client.
1. Select a topic in a JMS module.
2. Click Monitoring > Durable Subscribers to display the topics subscription management
options.
3. In addition to monitoring statistics for each subscriber, you can perform the following:
- Click New to create a new durable subscriber.
- Click Delete to delete specific durable subscribers.
- To manage the saved messages for a subscriber, select the check box next to the
subscribers name, and then click Show Messages to access its message
management page.
Monitoring and Managing Subscriptions
Messages are not typically removed from a destination

until consumers send acknowledgements back to WLS.
Consumers indicate an acknowledgement mode when
connecting to the server.
Acknowledgement and/or transaction issues can lead to a

consumer receiving duplicate messages.
A consumer can check if a message has been delivered
before with the JMSRedelivered field in the header.
If the consumer can handle duplicate messages, but the
message redelivery is happening too quickly for the
consumer, set the Redelivery Delay to slow down
redelivery of the duplicate.
When you create a transacted JMS session, the acknowledge mode is ignored. When an
application commits a transaction, all the messages that the application received during the
transaction are acknowledged by the messaging system and the messages that it sent are
accepted for delivery. If an application rolls back a transaction, the messages that the
application received during the transaction are not acknowledged and the messages that it
sent are discarded.
JMS messages include a standard header field named JMSRedelivered, whose initial value
is false. If WebLogic attempts to redeliver a message due to a missing acknowledgement or
some other condition, this header is set to true. WebLogic Server also adds a custom
message property named JMSXDeliveryCount, which indicates the number of redelivery
attempts the server has made for the current message. The first attempt is 1, the next attempt
is 2, and so on. WebLogic Server makes a best effort to persist the delivery count so that it
does not reset to 1 after a server reboot.
Duplicate Messages
A poison message is any message that the consumer is

unable to process. Perhaps the message is:
Corrupt
Not the message type the consumer expected
A duplicate, but the consumer cannot handle duplicates
The message is not processed, so it is not acknowledged.

That causes it to be sent again (and again). This wastes
resources.
Set attributes to deal with poison messages:
Set Redelivery Limit and Error Destination
When the Redelivery Limit is reached, the message is sent to

the Error Destination rather than to the consumer.
Poison messages are messages that cannot be processed, for whatever reason. Because
they are not processed, they are not acknowledged. Because they are not acknowledged,
they are re-sent. They can tie up resources being sent again and again.
To alleviate this situation, set the Redelivery Limit and an Error Destination. When the
message has been redelivered for the number of times specified in the limit, it is sent to the
Error Destination rather than to the consumer. Later, messages in the Error Destination can
be investigated to see why they could not be processed by the consumer in the usual way.
Poison Messages
Mode
Description
Auto
Acknowledgements are immediately sent after receiving

messages.
Client
Acknowledgements must be sent manually by the application.
Dups OK
Acknowledgements are sent in batches; depending on redelivery

settings, duplicate messages can result.
No (WLS only)
Server do not expect acknowledgement and assume that all

messages were received successfully.
Transactional
When using transactions, mode settings are ignored and

messages are acknowledged as part of a commit.
In a nontransacted session, the application creating the session programmatically selects one
of several available acknowledgement modes:
AUTO_ACKNOWLEDGE: The Session object acknowledges receipt of a message after
the receiving application method has returned from processing it.
CLIENT_ACKNOWLEDGE: The Session object relies on the application to call an
acknowledge() method on a received message. After the method is called, the
session acknowledges all messages received since the last acknowledgement. This
mode allows an application to receive, process, and acknowledge a batch of messages
with one method call.
DUPS_OK_ACKNOWLEDGE: The Session object acknowledges receipt of a message
after the receiving application method has returned from processing it; duplicate
acknowledgements are permitted. This mode is most efficient in terms of resource
usage, but you should avoid using this mode if your application cannot handle duplicate
messages. Duplicate messages may be sent if an initial attempt to deliver a message
fails.
NO_ACKNOWLEDGE: Messages sent to this session are immediately deleted from the
server. Messages received in this mode are not recovered; as a result, messages may
be lost and/or duplicate messages may be delivered if an initial attempt to deliver a
message fails. This mode is supported for applications that do not require this quality of
service.
Consumer Acknowledgement Modes
By default, consumers may not receive messages in the

order in which they were produced due to factors such as:
Message priority, time-to-deliver, and sorting features
Transactional boundaries and rollbacks
Consumer acknowledgement modes
This issue can require complex filtering and collating logic

in consumers, which can affect performance.
While the JMS specification provides an ordered message delivery, it does so in a very strict
sense. It defines the order between a single instance of a producer and a single instance of a
consumer, but does not take into account the following common situations:
Many consumers on one queue
Multiple producers within a single application acting as a single producer
Message recoveries or transaction rollbacks where other messages from the same
producer can be delivered to another consumer for processing
Use of filters and destination sort keys
As a result, consumers that need to guarantee message ordering must employ some complex
and often expensive design patterns, such as:
A dedicated consumer with a unique selector per each subordering
A new destination per subordering, with one consumer per destination
Messages Out of Sequence
With WebLogics Unit of Order feature:

Producers provide a unique ID to the server
Multiple producers can use a common ID and act as a single
producer
The server ensures that messages are always consumed
sequentially in the order in which they were produced
UOO IDs can be set programmatically, configured using a

connection factory, or assigned automatically.
ID
Producer
Destination
Consumer
The WebLogic Server Unit of Order feature enables a message producer or group of
message producers acting together as one to group messages into a single unit that is
processed sequentially in the order in which the messages were created. The message
processing of a single message is complete when a message is acknowledged, committed,
recovered, or rolled back. Until message processing for a message is complete, the remaining
unprocessed messages for that UOO are blocked. UOO is ideal for applications that have
strict message-ordering requirements. UOO simplifies administration and application design
and in most applications improves performance.
Member messages of a UOO are delivered to queue consumers sequentially in the order in
which they were created. The message order within a UOO will not be affected by sort
criteria, priority, or filters. However, messages that are uncommitted, have a Redelivery
Delay, or have an unexpired Time To Deliver will delay messages that arrive after them.
A queue that has several messages from the same UOO must finish processing all of them
before they can be delivered to any queue consumer or before the next message can be
delivered to the queue.
Unit of Order (UOO): Overview
WebLogics Unit of Work feature has the same benefits

as Unit of Order with the following additions:
Allows producers to identify groups of messages that must
be consumed together as a unit
Ensures that a message group is processed only by a single
consumer
Unlike UOO, UOW must be explicitly enabled on individual

destinations.
Producer
UOW IDs
Consumer
Destination
Consumer
Many applications need an even more restricted notion of a group than provided by the
message UOO feature. If this is the case for your applications, WebLogic JMS provides the
Unit of Work (UOW) feature. The UOW enables applications to send JMS messages,
identifying some of them as a group and allowing a single JMS consumer to process them as
such. No message within the UOW will be available to a consumer until all of them are
available on the destination. The group of messages will be delivered to the user without
interruptions. In other words, all messages in the group will be delivered to the consumer
before messages from any other group (or part of no group at all).
For WebLogic JMS to identify a message as part of a UOW, the producer must include
several message properties. First, each UOW must be assigned its own unique ID. Next,
each message that is a component of the UOW must be assigned a sequence number,
starting with 1. Finally, the last message in the UOW must declare itself as such.
A destination will only process UOW message properties when it has been identified as a
terminal destination. This is done using the Unit-of-Work Message Handling Policy field in the
console. Optionally, you can also specify an Expiration Time for Incomplete UOW
Messages. After this amount of time (in milliseconds), any incomplete message groups
automatically expire, regardless of each message's time-to-live setting.
Unit of Work (UOW): Overview
Message-driven EJBs:
Are packaged and deployed as part of EJB applications
Automatically connect to and consume JMS messages
Hide JMS programming from the developer
Are pooled and share JMS client resources for
performance
Conn. Factory
Producers
Destination
MDB A
Conn. Factory
Producers
Destination
MDB B
A message-driven bean implements loosely coupled or asynchronous business logic in which

the response to a request does not need to be immediate. An MDB receives messages from a
JMS queue or topic, and performs business logic based on the message contents. It is an
asynchronous interface between EJBs and JMS. The EJB container interacts directly with an
MDB, creating bean instances and passing JMS messages to those instances as necessary.
The container creates bean instances at deployment time, adding and removing instances
during operations based on message traffic. Unlike other EJB types, clients do not directly
access MDBs through interfaces.
WebLogic Server maintains a free pool where MDB instances that are not currently servicing
requests reside. The number of MDB instances in the free pool is determined by the number
of available threads in the thread pool, but can also be controlled through EJB deployment
descriptors. Each MDB that is deployed to a server instance creates a single JMS connection.
In a queue-based JMS application (point-to-point model), each MDB instance has its own
session. In a topic-based JMS application (the publish/subscribe model), all local instances of
an MDB share a JMS session. A given message is distributed to multiple MDBs, with one
copy to each subscribing MDB. If multiple MDBs are deployed to listen on the same topic,
each MDB receives a copy of every message. A message is processed by one instance of
each MDB that listens to the topic.
Message-Driven Beans (MDBs): Review
MDBs support most standard and WLS-specific JMS

consumer features:
Local and remote JMS connections

Automatic reconnection after JMS server failure
Acknowledgement modes
Transactions and batching
Durable subscriptions for topics
Load balancing and failover for clustered destinations
Custom work managers
Configure MDB settings using a combination of code

annotations, deployment descriptors, and deployment
plans.
An MDB pool processes each message at least once. Potentially, a message can be
processed more than once. A message is redelivered and processed again if an application
fails, a transaction rolls back, or the hosting server instance fails during or after the
onMessage() method completes, but before the message is acknowledged or committed.
Nonpersistent messages are also redelivered in the case of failure, except for the case where
the messages host JMS server shuts down or crashes, in which case the messages are
destroyed. To ensure that a message is processed exactly once, use container-managed
transactions so that failures cause transactional MDB work to roll back and force the message
to be redelivered.
WebLogic Server provides a mechanism for grouping onMessage() calls together as a
single transaction. This mechanism can help increase database performance of an EJB
application by grouping all of the transactions into a single I/O request. Grouping transactions
requires fewer transaction logs.
WebLogic Server supports migration and recovery for clustered MDB applications. In the
event of failure, you can bring a JMS destination and MDBs back online. After an MDB
application migrates to another server, it reconnects to the migrated JMS destination and
begins to receive messages from the JMS destination again. An MDB automatically detects
the JMS server migration target during deployment and uses that as its migratable target. You
must ensure that MDBs are deployed everywhere that a JMS Server is deployed.
MDB Capabilities
MessageDrivenEJBRuntime
Attribute
Description
JMSConnectionAlive Is the MDB currently connected to the destination?

ProcessedMessage
Count
The total number of messages consumed since the EJB

was deployed
1
EJBPoolRuntime
Attribute
Description
PooledBeansCurrent Current MDB pool size

Count
BeansInUseCurrent
Count
Number of MDB instances in the pool that are currently

processing messages
WaiterCurrentCount
Number of threads currently waiting for an available MDB

instance in the pool
Refer to the online documentation for a list of all available attributes and operations for these
MBeans:
EJBRuntimeMBean
MessageDrivenEJBRuntimeMBean (a child of EJBRuntimeMBean)

EJBPoolRuntimeMBean (a child of MessageDrivenEJBRuntimeMBean)
EJBTransactionRuntimeMBean (a child of MessageDrivenEJBRuntimeMBean)
MDB Runtime Attributes
WLDF Monitor
MDB Before/After/Around
Message Received
Debug Flag
Description
Trigger diagnostic actions each time an MDB in
this application processes a message.
Description
DebugEJBPooling
Trace EJB pool utilization.
DebugEJBMDBConnection
Trace JMS (re-)connections from MDB pool.
In addition to MBean attributes, you can also monitor MDBs by using the WebLogic
diagnostics framework or server debug flags. The instrumentation component of the
diagnostics framework includes monitors that can trigger actions whenever a MDB receives a
message. For example, you can automatically capture stack or thread dumps to troubleshoot
MDB problems. WebLogic also includes several debug flags for the EJB subsystem. The two
flags that pertain to message driven EJBs are shown in the slide.
MDB Diagnostics and Debugging
Which of the following is not a technique for monitoring JMS?

a. WLST
b. Message Logs
c. WLDF Images
d. Flow Control
e. Console
Answer: d
Quiz
Which of these is not an override setting available on JMS

destinations?
a. Pool Size
b. Time to Deliver
c. Redelivery Delay
d. Time to Live
e. Delivery Mode
Answer: a
Quiz
Name two techniques to help prevent JMS out-of-memory

scenarios.
a. Time to Deliver
b. Unit or Order
c. Quotas
d. Durable Subscribers
e. Paging
Answer: c, e
Quiz

Describe the components of JMS communication
Manage and monitor JMS resources by using the console,
WLST, and WLDF
Investigate JMS issues by using logs and debug flags
Discuss some common JMS configuration issues
Configure JMS to handle large message backlogs
Configure JMS to handle failed and expired messages
Troubleshoot problems with message-driven EJBs
Summary

Analyzing JMS producer and consumer problems
Monitoring JMS communication by using WLST, logs, and
debug flags
Configuring delivery failure settings
Responding to failed messages
Practice 10-1
Investigating JMS Problems
Troubleshooting Security

Identify several WLS scenarios that involve SSL
Describe the fundamentals of SSL and LDAP
communication
Trace SSL and security realm functionality
List some common causes of SSL errors
Work with Java keystore files
Troubleshoot the WLS embedded LDAP
Objectives
SSL Diagnostics
Keystores and Keytool

Configuration Review
Debug Flags
Host Name Verification
Chain Validation
Security Realm Diagnostics
Road Map
SSL:
Is an enhancement to the TCP protocol that secures a
point-to-point socket connection
Enables the transmission of sensitive information without
the information being read or tampered with by a third party
SSL involves the:
Encryption/decryption of data using a randomly generated
pair of symmetric keys
Distribution of an asymmetric public key to clients by using
a digitally signed certificate
Digital validation of the certificate by using a public key from
a trusted certificate authority who issued the certificate
Secure Sockets Layer (SSL) provides secure connections by allowing two applications
connecting over a network to authenticate each others identity and by encrypting the data
exchanged between the applications. Authentication allows a server and optionally a client to
verify the identity of the application on the other end of a network connection. Encryption
makes data transmitted over the network intelligible only to the intended recipient.
SSL uses public key encryption technology for authentication. With public key encryption, a
public key and a private key are generated for a server. Data encrypted with the public key
can only be decrypted using the corresponding private key and data encrypted with the
private key can only be decrypted using the corresponding public key. The private key is
carefully protected so that only the owner can decrypt messages that were encrypted using
the public key.
The public key is embedded in a digital certificate with additional information describing the
owner of the public key, such as name, street address, and email address. A private key and
digital certificate provide identity for the server. The data embedded in a digital certificate is
verified by a certificate authority and digitally signed with the certificate authoritys digital
certificate. Well-know certificate authorities include Verisign and Entrust.net. The trusted
certificate authority (CA) certificate establishes trust for a certificate.
Secure Sockets Layer (SSL): Review
Client
Greeting, supported SSL standards

Selected SSL standards
3
Obtain public
key from cert
using trust cert
Server
Certificate request
Certificate and response
Response
encrypted with
private key
Decrypt
response with
public key
Generate
symmetric key
and encrypt with
public key
Symmetric key
Decrypt and
use new key
SECURE COMMUNICATION
Trust
Certificates
Server Cert and

Private Key
1. The SSL session begins with a negotiation between the client and the server as to which
cipher suite they will use. This negotiation is known as the SSL handshake. A cipher suite
is a set of cryptographic algorithms and key sizes that a computer can use to encrypt data.
The cipher suite includes information about the public key exchange algorithms or key
agreement algorithms, and cryptographic hash functions. The client tells the server which
cipher suites it has available.
2. The server chooses the best mutually acceptable cipher suite.
3. The client requests the servers certificate, so the server can prove it is legitimate.
4. The server presents its public key certificate to the client. If this certificate is valid, the
client can be sure of the identity of the server.
5. The client and server exchange information that allows them to agree on the same secret
key. With RSA, the client uses the servers public key, obtained from the public key
certificate, to encrypt the secret key information. The client sends the encrypted secret key
information to the server. Only the server can decrypt this message because only it has
the private key.
6. Both the client and the server now have access to the same secret key. With each
message, they use the cryptographic hash function, chosen in the first step of this
process, and shared secret information, to compute a media access control (MAC)
address that they append to the message.
SSL Communication: Review
ONE-WAY INCOMING SSL

Browser
HTTPS
Trust Certs
Other Client
HTTPS or T3S
WebLogic
Server Cert and
Private Key
Server
SSL port
TWO-WAY INCOMING SSL

Trust Certs
Browser
Client Cert and

Private Key
Other Client
HTTPS
WebLogic
Server Cert and
Private Key
HTTPS or T3S
Trust Certs
In one-way SSL authentication, the target or server is required to present a digital certificate
to the initiator or client to prove its identity. The client verifies that the certificate is trusted and
valid. This implies that the certificate was issued by the clients trusted CA and has not
expired.
In two-way SSL authentication, both the client and the server must present digital certificates
before the SSL connection is established. Therefore, in this case, WebLogic Server not only
authenticates itself to the client, which is the minimum requirement for certificate
authentication, it also requires authentication from the requesting client. Two-way SSL
authentication is useful when you need to restrict access of your resources to trusted clients
only.
To configure two-way SSL for a server, update its Two Way Client Cert Behavior attribute.
The following options are available:
Client Certs Not Requested: The default (meaning one-way SSL).
Client Certs Requested But Not Enforced: Requires a client to present a certificate. If
a certificate is not presented, the SSL connection continues.
Client Certs Requested And Enforced: Requires a client to present a certificate. If a
certificate is not presented, the SSL connection is terminated.
WebLogic SSL Scenarios
ONE-WAY OUTGOING SSL

WebLogic
HTTPS or T3S
Trust Certs
WebLogic
Other Server
Server Cert and

Private Key
ADMIN CHANNEL SSL, SERVER STARTUP

Managed Server
Trust Certs
Admin Server
T3S
Server Cert and

Private Key
Admin port
ADMIN CHANNEL SSL, SERVER ADMINISTRATION

WebLogic
Trust Certs
WLST
T3S
WebLogic
Server Cert and
Private Key
You can define an optional administration port for your domain. When configured, the
administration port is used by each managed server in the domain exclusively for
communication with the domains Administration Server. If an administration port is enabled,
WebLogic Server automatically generates an administration channel based on the port
settings upon server instance startup.
The administration port accepts only secure, SSL traffic, and all connections via the port
require authentication. The Administration Server and all managed servers in your domain
must be configured with support for the SSL protocol. Managed servers that do not support
SSL cannot connect with the Administration Server during startup. You will have to disable
the administration port in order to configure them
After enabling the administration port, all Administration Console and WLST traffic must
connect via the administration port.
If you boot managed servers either at the command line or using a start script, specify the
administration port in the Administration Servers URL. The URL must also specify the
HTTPS/T3S protocol rather than HTTP/T3. If you use Node Manager for starting managed
servers, it is not necessary to modify startup settings or arguments for the managed servers.
Node Manager obtains and uses the correct URL to start a managed server.
WebLogic SSL Scenarios
PERIMETER SSL
Browser
HTTPS
Web or Proxy
Server
HTTP
Server Cert and

Private Key
Trust Certs
Two separate
SSL sessions
PASS-THROUGH SSL
Browser
Trust Certs
WebLogic
HTTPS
Web or Proxy
Server
HTTPS
Server Cert and

Private Key
WebLogic
Server Cert and
Private Key
Trust Certs
If using a WebLogic Server proxy plug-in, you can configure it to use SSL for communications
between the proxy server and WebLogic. The available plug-in settings include:
WLProxySSL: Set this parameter to ON to maintain SSL communication between the
plug-in and WebLogic Server when incoming client requests specify the HTTPS protocol
SecureProxy: Set this parameter to ON to enable the use of the SSL protocol for all
communication between the plug-in and WebLogic Server. Remember to configure a
port on the corresponding WebLogic Server for the SSL protocol before defining this
parameter.
TrustedCAFile: Name of the file that contains the digital certificates for the trusted
certificate authorities for the plug-in. This parameter is required if the SecureProxy
parameter is set to ON. The required format of this file and of the included certificates
will vary depending on the proxy vendor.
RequireSSLHostMatch: Determines whether the host name to which the plug-in is
connecting must match the Subject Distinguished Name field in the digital certificate of
the WebLogic Server to which the proxy plug-in is connecting.
WebLogic Server proxy plug-ins do not support two-way SSL. However, the plug-ins can be
set up to require the client certificate and pass it on to WebLogic Server. The steps to
configure this will vary depending on the proxy vendor.
Proxy Server SSL Scenarios
The secure storage of private keys and other SSL artifacts

is critical to the integrity of your organization.
Java applications such as WLS use keystore files (.jks),
which provide multiple layers of password-based
encryption.
Requires keystore
password to access file
Requires alias
password to access
specific certs/keys
Keystore
Alias1
Cert and/or
Private Key
Alias2
Cert and/or
Private Key
A keystore is logically a database of security artifacts that can only be accessed with the
proper credentials. Security artifacts are used for a variety of purposes, including
authentication and data integrity. There are various types of keystores available, including
PKCS12 and Suns Java Keystore (JKS) format. WebLogic supports both of these types, but
JKS is preferred and PKCS12 support is deprecated as of this release.
Generally speaking, keystore information can be grouped into two different categories: key
entries and trusted certificate entries. A key entry consists of an entitys identity and its private
key, and can be used for a variety of cryptographic purposes. In contrast, a trusted certificate
entry only contains a public key in addition to the entitys identity. Thus, a trusted certificate
entry cannot be used where a private key is required. For the JKS type, a keystore may
contain both key entries and trusted certificate entries.
Demonstration keystores with digital certificates, private keys, and trusted CA certificates are
found in the WL_HOME\server\lib directory. These demonstration security artifacts should
be used in a development environment only. Because the digital certificates and trusted CA
certificates in the demonstration keystores are signed by a WebLogic Server demonstration
certificate authority, a WebLogic Server installation using the demonstration keystores will
trust any WebLogic Server installation that also uses the demonstration keystores. You want
to create a secure environment where only your installations trust each other.
Keystore: Review
Web browsers maintain a repository of trust certificates

(public keys) from popular authorities.
Java applications use a separate keystore file to manage
trust certificates, which:
Is bundled with the JDK and is named cacerts
Includes many of the same entries found in Web browsers
Uses the password changeit
By default, WLS uses the JDK trust keystore, but a custom

keystore can be used instead.
A truststore is a keystore that is used when making decisions about what to trust. If you
receive some data from an entity that you already trust, and if you can verify that the entity is
the one it claims to be, then you can assume that the data really came from that entity. An
entry should be added to a truststore only if the user makes a decision to trust that entity. By
either generating a key pair or by importing a certificate, the user has given trust to that entry,
and thus any entry in the keystore is considered a trusted entry.
A file named cacerts resides in the JVM runtimes security properties directory,
jre/lib/security. The cacerts file represents a system-wide JKS keystore with CA
certificates. System administrators can configure and manage that file by using Keytool. The
cacerts keystore file ships with several common root CA certificates, each with an alias and
an X.500 owner distinguished name. The initial password of the cacerts keystore file is
changeit. System administrators should change that password and the default access
permission of that file when installing the SDK.
Because you trust the CAs in the cacerts file as entities for signing and issuing certificates
to other entities, you must manage the cacerts file carefully. The cacerts file should
contain certificates of only those CAs that you trust. It is your responsibility to verify the trusted
root CA certificates bundled in the cacerts file and make your own trust decisions.
Trust Keystores
JDKs include a command-line tool to create, view, and modify

keystore files.
Generate a new self-signed certificate and private key and add it to a store:
keytool genkeypair alias mykey keypass mykeypass
keyalg RSA keysize 512 -dname "CN=payroll.mycompany.com..."
-keystore mykeys.jks storepass mypass
Import a signed certificate from a CA into a store:
keytool importcert file payroll.pem alias mykey keypass
mykeypass -keystore mykeys.jks storepass mypass
Inspect the contents of a store:
keytool list v -keystore mykeys.jks storepass mypass
Keytool is a key and certificate management utility. It allows users to use digital signatures to
administer their own public-private key pairs and associated certificates for use in selfauthentication (where users authenticate themselves to other users or services) or data
integrity and authentication services. It also allows users to cache the public keys (in the form
of certificates) of their communicating peers. Keytool also enables users to administer secret
keys used in symmetric encryption/decryption (for example, DES). Keytool stores the keys
and certificates in a keystore.
Available commands include:
-genkeypair: Generates a key pair (a public key and associated private key). Wraps
the public key into an X.509 v3 self-signed certificate, which is stored as a singleelement certificate chain. This certificate chain and the private key are stored in a new
keystore entry identified by alias. keyalg specifies the algorithm to be used to
generate the key pair, and keysize specifies the size of each key to be generated.
sigalg specifies the algorithm that should be used to sign the self-signed certificate;
this algorithm must be compatible with keyalg. dname specifies the X.500
distinguished name to be associated with alias, and is used as the issuer and subject
fields in the self-signed certificate. If no distinguished name is provided at the command
line, the user will be prompted for one.
Keytool: Review
-importcert: Reads the certificate or certificate chain (in which the latter is supplied in a
PKCS#7 formatted reply or a sequence of X.509 certificates) from the file cert_file and
stores it in the keystore entry identified by alias. If no file is given, the certificate or
certificate chain is read from standard in (standard input or standard input stream).
Keytool can import X.509 v1, v2, and v3 certificates, as well as PKCS#7-formatted
certificate chains consisting of certificates of that type. The data to be imported must be
provided either in binary encoding format or in printable encoding format (also known as
Base64 encoding) as defined by the Internet RFC 1421 standard. In the latter case, the
encoding must be bounded at the beginning by a string that starts with -----BEGIN and
bounded at the end by a string that starts with -----END. If the alias does not point to a
key entry, then Keytool assumes you are adding a trusted certificate entry. In this case, the
alias should not already exist in the keystore. If the alias does already exist, Keytool
outputs an error (because there is already a trusted certificate for that alias) and does not
import the certificate. If the alias points to a key entry, Keytool assumes you are importing
a certificate reply. When importing a certificate reply, the certificate reply is validated using
trusted certificates from the keystore, and optionally using the certificates configured in the
cacerts keystore file (if the -trustcacerts option was specified). If the reply is a
single X.509 certificate, Keytool attempts to establish a trust chain, starting at the
certificate reply and ending at a self-signed certificate (belonging to a root CA). The
certificate reply and the hierarchy of certificates used to authenticate the certificate reply
form the new certificate chain of the alias. If a trust chain cannot be established, the
certificate reply is not imported. In this case, Keytool does not print out the certificate and
prompt the user to verify it, because it is very hard (if not impossible) for a user to
determine the authenticity of the certificate reply.
-list: Prints to standard output the contents of the keystore entry identified by alias. If
no alias is specified, the contents of the entire keystore are printed. This command by
default prints the MD5 fingerprint of a certificate. If the -v option is specified, the certificate
is printed in human-readable format, with additional information such as the owner, issuer,
serial number, and any extensions. If the -rfc option is specified, the certificate contents
are printed using the printable encoding format, as defined by the Internet RFC 1421
standard.
-delete: Deletes from the keystore the entry identified by alias. The user is prompted
for the alias if no alias is provided at the command line.
WebLogic Server supports both the SSL v3.0 and

Transport Layer Security (TLS) v1.0 protocols.
When WebLogic Server acts as an SSL server, it agrees to
use either of the two protocols that the client specifies as
preferred.
When WebLogic Server acts as an SSL client, it specifies
TLS v1.0 as the preferred protocol.
WebLogic Server supports both the SSL V3.0 and TLS V1.0 protocols. When WebLogic
Server is acting as an SSL server, the protocol that the client specifies as preferred in its
client hello message is used. Note that WebLogic Server does not support SSL V2.0. When
WebLogic Server is acting as an SSL client, it specifies TLS1.0 as the preferred protocol in its
SSL V2.0 client hello message. But it can use SSL V3.0 as well if that is the highest version
that the SSL server on the other end supports. The peer must respond with an SSL V3.0 or
TLS V1.0 message; otherwise, the SSL connection is dropped.
Although the SSL V3.0 protocol is acceptable in most cases, some circumstances
(compatibility, SSL performance, and environments with maximum security requirements)
make the TLS V1.0 protocol more desirable. The
weblogic.security.SSL.protocolVersion command-line argument lets you specify
which protocol is used for SSL connections. The SSL V3.0 and TLS V1.0 protocols cannot be
interchanged. Use the TLS V1.0 protocol only if you are certain that all desired SSL clients
are capable of using the protocol.
WebLogic SSL Support
Perform the following for each server:

1. Enable an SSL port (and optionally an admin port).
2. Set identity and trust keystore file locations and
passwords.
3. Set the alias/password of the certificate/key to use in the
identify keystore file.
Before you configure WebLogic Server for SSL, you must do the following:
Obtain private keys and digital certificates from a reputable certificate authority such as
Verisign, Inc., or Entrust.net.
Create identity and trust keystore files by using tools like Keytool.
Load the private keys and trusted CAs into the keystores.
Register your keystores with WLS. Click the name of the server for which you want to
configure the identity and trust keystores. Select Configuration > Keystores. Next to the
Keystores information, click the Change button. When the Keystores drop-down list is shown,
select the method for storing and managing private keys/digital certificate pairs and trusted
CA certificates. Then click Save. The options for the Keystores field are:
Demo Identity and Demo Trust: The demonstration identity and trust keystores,
located in the MIDDLEWARE_HOME\server\lib directory and the JDK cacerts
keystore, are configured by default. Use for development only.
Custom Identity and Java Standard Trust: A keystore that you create and the trusted
CAs defined in the cacerts file in the JAVA_HOME\jre\lib\security directory
Custom Identity and Custom Trust: Identity and trust keystores that you create
Custom Identity and Command Line Trust: An identity keystore that you create and
command-line arguments that specify the location of the trust keystore
SSL Configuration: Review
To configure SSL settings for a server, use its Configuration > SSL tab. The available fields
include:
Private Key Alias: The keystore attribute that defines the string alias used to store and
retrieve the servers private key
Private Key Passphrase: The keystore attribute that defines the passphrase used to
retrieve the servers private key
Export Key Lifespan: Indicates the number of times WebLogic Server can use an
exportable key between a domestic server and an exportable client before generating a
new key. The more secure you want WebLogic Server to be, the fewer times the key
should be used before generating a new key.
Use Server Certs: Sets whether WLS should use the server certificates/key as the
client identity when initiating a two-way outgoing connection over SSL
Two Way Client Cert Behavior: By default, WebLogic Server is configured to use oneway SSL (implied by the Client Certs Not Requested value). Selecting one of the other
available options enables two-way SSL.
Other fields are only for backward compatibility when not using keystores (for example,
Private Key File Name, Server Certificate File Name, and Trusted CA File Name).
SSL Configuration: Review
WLS allows you to dynamically stop and restart the SSL

subsystem and port after configuration changes.
1
Restart SSL using WLST:

serverRuntime()
cmo.restartSSLChannels()
All the server SSL attributes are dynamic; when modified via the console or WLST, they
cause the corresponding SSL server or channel SSL server to restart and use the new
settings for new connections. Old connections will continue to run with the old configuration.
To ensure that all the SSL connections exist according to the specified configuration, you
must reboot WebLogic Server.
Using the console, select a specific server and then select Control > Start/Stop tab. Then click
the Restart SSL button found at the bottom of the page.
The corresponding MBean operation is ServerRuntime.restartSSLChannels(), which
can be invoked using WLST.
Restarting SSL
Flag
Description
DebugSecuritySSL
Trace initialization of Java SSL library and SSL

channels.
View available SSL ciphers.
Trace loading of identity and trust keystores.
View metadata for each keystore entry.
Trace SSL handshake and data communications.
DebugSecuritySSLEaten
View certain benign Java SSL exceptions that are

otherwise ignored by WLS.
DebugSecurityKeyStore
View the values of WLS keystore configuration

parameters.
Server log messages with DebugSecuritySSL enabled:

<Debug> <SecuritySSL> <BEA-000000> <Loaded public identity
certificate chain:>
<Debug> <SecuritySSL> <BEA-000000> <Subject:
CN=payroll.mycompany.com, ...>
SSL debugging provides more detailed information about the SSL events that occurred during
an SSL handshake. The SSL debug trace displays information about trusted certificate
authorities, SSL server configuration information, server identity keystores, the encryption
strength that is allowed, the enabled ciphers, the SSL communication during a handshake,
and additional information about any SSL failures detected by WebLogic Server.
WebLogic Server allows SSL sessions to be cached. Those sessions live for the life of the
server. Clients that use SSL sockets directly can control the SSL session cache behavior. The
SSL session cache is specific to each SSL context. All SSL sockets created by SSL socket
factory instances returned by a particular SSL context can share the SSL sessions. Clients
default to resuming sessions at the same IP address and port. Multiple SSL sockets that use
the same host and port share SSL sessions by default, assuming the SSL sockets are using
the same underlying SSL context.
SSL sessions exist for the lifetime of the SSL context; they are not controlled by the lifetime of
the SSL socket. Therefore, creating a new SSL socket and connecting to the same host and
port used by a previous session can resume a previous session as long as you create the
SSL socket by using an SSL socket factory from the SSL context that has the SSL session in
its cache. By default, clients that use HTTPS URLs get a new SSL session for each URL
because each URL uses a different SSL context and, therefore, SSL sessions cannot be
shared or reused.
SSL Debug Flags
Normal SSL handshake log messages with DebugSecuritySSL enabled:

Messages include an
<23634469 SSL3/TLS MAC>
internal SSL session ID.
<23634469 received HANDSHAKE>
<HANDSHAKEMESSAGE: ClientHello>
<write HANDSHAKE, offset = 0, length = 58>
Greeting
...
<23634469 SSL3/TLS MAC>
Certificate
exchange
<HANDSHAKEMESSAGE: ClientKeyExchange RSA>
<Using JCE Cipher: SunJCE version 1.6 for algorithm RSA>
...
Protocol
<23634469 SSL3/TLS MAC>
negotiation
<23634469 received CHANGE_CIPHER_SPEC>
<Using JCE Cipher: SunJCE version 1.6 for algorithm RC4>
...
<23634469 SSL3/TLS MAC>
Secure session
established
<HANDSHAKEMESSAGE: Finished>
1. The client sends the server information, including the highest version of SSL that it
supports and a list of the cipher suites that it supports. (TLS 1.0 is indicated as SSL 3.1.)
The cipher suite information includes cryptographic algorithms and key sizes.
2. The server chooses the highest version of SSL and the best cipher suite that both the
client and server support, and sends this information to the client.
3. The server sends the client a certificate or a certificate chain. A certificate chain typically
begins with the servers public key certificate and ends with the certificate authoritys
root certificate. If the server needs to authenticate the client (two-way SSL), it sends the
client a certificate request.
4. The server tells the client that it is finished with its initial negotiation messages.
5. The client generates information used to create a key to use for symmetric encryption.
For RSA, the client then encrypts this key information with the servers public key and
sends it to the server.
6. The client sends a message telling the server to change to encrypted mode.
7. The client tells the server that it is ready for secure data communication to begin.
8. The server tells the client that it is ready for secure data communication to begin. This is
the end of the SSL handshake.
SSL Handshake Trace
Normal SSL data transfer log messages with DebugSecuritySSL enabled:

<23634469 SSL3/TLS MAC>
<23634469 received APPLICATION_DATA: databufferLen 0,
<23634460 read databufferLen 578>
Requests and
<23634460 read A returns 578>
responses after
<23634460 read(offset=578, length=3502)>
handshake
...
<write APPLICATION_DATA, offset = 0, length = 297>
...
Problem log messages with DebugSecuritySSL enabled:

<NEW ALERT with Severity: FATAL, Type: 42 ...exception...>
When sending encrypted data, SSL typically uses a cryptographic hash function to ensure
data integrity. A cryptographic hash function is similar to a checksum. The main difference is
that while a checksum is designed to detect accidental alterations in data, a cryptographic
hash function is designed to detect deliberate alterations. When data is processed by a
cryptographic hash function, a small string of bits, known as a hash, is generated. The
slightest change to the message typically makes a large change in the resulting hash. A
cryptographic hash function does not require a cryptographic key. Two hash functions often
used with SSL are Message Digest 5 (MD5) and Secure Hash Algorithm (SHA). A message
authentication code (MAC) is similar to a cryptographic hash, except that it is based on a
secret key.
SSL debugging dumps a stack trace whenever an ALERT is created in the SSL process. The
types and severity of the ALERTS are defined by the Transport Layer Security (TLS)
specification. The stack trace dumps information into the log file where the ALERT originated.
Therefore, when tracking an SSL problem, you may need to enable debugging on both sides
of the SSL connection (on both the SSL client or the SSL server). The log file contains
detailed information about where the failure occurred. To determine where the ALERT
occurred, confirm whether there is a trace message after the ALERT. An ALERT received
after the trace message indicates that the failure occurred on the peer. Note that severity 1
type 0 is a normal close ALERT and not a problem.
Other SSL Traces
WLS supports only the default JDK cryptography provider

or those available from nCipher.
Your certificates must use a signature algorithm or cipher
that is supported by the cryptography provider.
Oracle recommends the use of:
X.509-compliant certificates
The Privacy-Enhanced Mail (PEM) format for keys and
certificates
RSA algorithms (DSA is not supported.)
OpenSSL and other tools are available to help convert

between various formats.
The PEM (Privacy Enhanced Mail) format is the preferred format for private keys, digital
certificates, and trusted certificate authorities (CAs). A .pem-format file begins with this line:
---BEGIN CERTIFICATE--- . A .pem-format file supports multiple digital certificates (for
example, a certificate chain can be included). The order of certificates within the file is
important. The servers digital certificate should be the first digital certificate in the file,
followed by the issuer certificate, and so on. Each certificate in the chain is followed by its
issuer certificate. If the last certificate in the chain is the self-signed (self-issued) root
certificate of the chain, the chain is considered complete. Note that the chain does not have to
be complete.
By default, WLS uses the Java Cryptography Extension (JCE) SSL implementation or
provider that is bundled with the JDK used to start the server. However, heavy SSL traffic
can cause bottlenecks that affect the performance of Web servers. JCE providers (such as
nCipher) that use a hardware card for encryption will offload SSL processing from Web
servers, which frees the servers to process more transactions. Install and configure the
nCipher JCE provider according to the nCipher documentation. Then edit the Java security
properties file (java.security) to add the nCipher JCE provider to the list of approved JCE
providers for WebLogic Server. The Java security properties file is located in
JAVA_HOME/jre/lib/security/java.security.
Invalid Format or Cipher Errors
For outgoing and two-way SSL communication, WLS

performs several validation routines on received
certificates.
Typical causes of validation problems include:
Trust cannot be established using available CA certificates

Certificate is expired
Host name does not match
Certificate is missing certain constraints or policies
Certificate is not present in the security realm registry, if
enabled
WebLogic Server may receive digital certificates as part of incoming Web Services requests,
incoming two-way SSL connections, or other outgoing SSL connections. To validate these
certificates, WebLogic Server includes a certificate lookup and validation framework, whose
function is to look up and validate X.509 certificate chains.
A public key certificate contains several fields, including a period of validity, which essentially
acts like an expiration date. Consumers of certificates like WebLogic Server must verify that
the certificate has not expired before establishing an SSL connection.
A host name verifier ensures that the host name in the URL to which the client connects
matches the host name in the digital certificate that the server sends back as part of the SSL
connection. A host name verifier is useful when an SSL client (or a WebLogic Server acting
as an SSL client) connects to an application server on a remote host. It helps to prevent manin-the-middle attacks.
The certificate registry is an optional WebLogic security provider that enables you to explicitly
register the list of trusted certificates that are allowed to access WebLogic Server. If you
configure a certificate registry as part of your security realm, then only certificates that are
registered will be considered valid. The certificate registry provides an inexpensive
mechanism for performing revocation checking. By removing a certificate from the registry,
you can invalidate a certificate immediately. The registry is stored in the embedded LDAP
server.
Certificate Validation Errors
By default WLS will accept a certificate only if its Common

Name (CN) attribute matches one of the following:
The host name that the certificate was sent from
The local machine host name (certificate must be sent from
localhost)
Host name verification helps prevent man-in-the-middle

attacks, but it can be disabled if desired.
By default, WebLogic Server has host name verification enabled. As a function of the SSL
handshake, WebLogic Server compares the common name in the SubjectDN in the SSL
servers digital certificate with the host name of the SSL server used to accept the SSL
connection. If these names do not match, the SSL connection is dropped. The SSL client is
the actual party that drops the SSL connection if the names do not match. If anything other
than the default behavior is desired, either turn off host name verification or configure a
custom host name verifier. Turning off host name verification leaves WebLogic Server
vulnerable to man-in-the-middle attacks. Oracle recommends leaving host name verification
on in production environments.
If the host name in the certificate matches the local machines host name, host name
verification passes if the URL specifies localhost, 127.0.01, or the default IP address of the
local machine. The demo identity certificates that are generated during WLS installation use
the default network settings of the host OS and therefore will validate successfully only if the
server is accessed using the same address.
Hostname Verification: Specifies whether to execute the default, custom, or no host
name verifier when this server is acting as a client to another application server
Custom Hostname Verifier: The name of the class that implements the
weblogic.security.SSL.HostnameVerifier interface
Host Name Verification Errors
A server certificate may be signed by a chain of

authorities.
Either the server or one of the successor certificates must
be signed by an authority in the clients trusted list.
Server Cert
Intermediate Cert
Signs
Root Cert
Signs
Client
Server
Trust Certs
Multiple certificates may be linked in a certificate chain. When a certificate chain is used, the
first certificate is always that of the sender. The next is the certificate of the entity that issued
the senders certificate. If there are more certificates in the chain, each is the certificate of the
authority that issued the previous certificate. The final certificate in the chain is the certificate
for a root CA. A root CA is a public certificate authority that is widely trusted.
When WebLogic Server receives a certificate, it must first look up and validate the certificate
chain. If the certificate chain is found and valid, the framework then performs any additional
validation on each certificate in the chain. A chain is valid only if all of the certificates in the
chain have digitally signed each other properly, and if the chain terminates in a certificate that
is one of the servers trusted CAs.
Certificate Chains
WLS includes a command-line tool to validate a specific

certificate and any successors within a keystore file.
Validate a certificate chain offline:
java utils.ValidateCertChain -jks mykey mykeystore mystorepass
Cert[0]: CN=payroll.mycompany.com, ...
CA cert not marked with critical BasicConstraint indicating it
is a CA
Cert[1]: CN=www.mycompany.com, ...
Certificate chain is invalid
Use the WebLogic Server ValidateCertChain command-line utility to confirm whether an

existing certificate chain will be rejected by WebLogic Server. The utility validates certificate
chains from PEM files, PKCS-12 files, PKCS-12 keystores, and JKS keystores. A complete
certificate chain must be used with the utility.
The following is the syntax for the ValidateCertChain command-line utility:
java utils.ValidateCertChain -file pemcertificatefilename
java utils.ValidateCertChain -pem pemcertificatefilename
java utils.ValidateCertChain -pkcs12store pkcs12storefilename
java utils.ValidateCertChain -pkcs12file pkcs12filename password
java utils.ValidateCertChain -jks alias storefilename [storePass]
Another example:
java utils.ValidateCertChain -pem zippychain.pem
Certificate chain appears valid
WLS Chain Validation Utility
By default, WLS requires certificates to include a Basic

Constraint extension, which is used by CAs to specify a
maximum chain length.
For older certificates, you may need to disable this check
by using the JVM argument -Dweblogic.security.SSL.
enforceConstraints.
CAs may also include additional Policy extensions.
Indicate which policies must be present by using
-Dweblogic.security.SSL.
allowedcertificatepolicyids.
By default, WebLogic Server rejects any certificates in a certificate chain that do not have the
Basic Constraint extension defined as CA. However, you may be using certificates that do not
meet this requirement or you may want to increase the level of security to conform to the IETF
RFC 2459 standard.
Use the following option to ensure that the Basic Constraints extension on the CA certificate is
defined as CA and set to critical. This option enforces the IETF RFC 2459 standard:
-Dweblogic.security.SSL.enforceConstraints=strict
Use the following option to turn off checking for the Basic Constraints extension. The rest of
the certificate is still validated:
-Dweblogic.security.SSL.enforceConstraints=off
WebLogic Server offers limited support for Certificate Policy Extensions in X.509 certificates.
Use the weblogic.security.SSL.allowedcertificatepolicyids argument to
provide a comma-separated list of Certificate Policy IDs. When WebLogic Server receives a
certificate with a critical Certificate Policies Extension, it verifies whether any Certificate Policy
is on the list of allowed certificate policies and whether there are any unsupported policy
qualifiers. This release of WebLogic Server supports Certification Practice Statement (CPS)
Policy qualifiers and does not support User Notice qualifiers. A certificate is also accepted if it
contains a special policy anyPolicy with the ID 2.5.29.32.0, which indicates that the CA
does not want to limit the set of policies for this certificate.
Missing Constraint or Policy Errors
In this section, you should have learned how to:

Trace SSL communication on WLS
Identity and correct common SSL issues
Section Summary

Managing and monitoring the SSL subsystem
Analyzing SSL debug messages
Analyzing and updating keystore contents
Troubleshooting invalid and untrusted certificates
Practice 11-1
Investigating SSL Problems
SSL Diagnostics
Security Realm Diagnostics
Embedded LDAP
Password Reset
Database Store
Realm Debug Flags
Auditing Provider
Authentication Control Flags
LDAP Concepts
External LDAP Problems
Road Map
A security realm:
Handles security logic and decisions for a domain
Consists of a series of pluggable providers
Is scoped to all servers in a domain
Administration
Tools
Applications
Default Store
External
Clients
Realm
Authentication
providers
Role Mapping
providers
Authorization
providers
Adjudication
provider
Password
Validation pvds.
Credential Map
providers
Certificate
providers
Auditing
providers
The security service in WebLogic Server simplifies the configuration and management of
security while offering robust capabilities for securing your WebLogic Server deployment.
Security realms act as a scoping mechanism. Each security realm consists of a set of
configured security providers, users, groups, security roles, and security policies. You can
configure multiple security realms in a domain; however, only one can be the active security
realm.
A security policy is an association between a WebLogic resource and one or more users,
groups, or security roles. Security policies protect the WebLogic resource against
unauthorized access. A WebLogic resource has no protection until you create a security
policy for it.
A security provider store contains the users, groups, security roles, security policies, and
credentials used by some types of security providers to provide their services. For example,
an authentication provider requires information about users and groups; an authorization
provider requires information about security policies; a Role Mapping provider requires
information about security roles; a Credential Mapping provider requires information about
credentials to be used for remote applications. These security providers need this information
to be available in a database to function properly.
Identity Assertion providers validate specific tokens found in client requests. Unlike
authenticators, identity asserters must trust that the user has submitted sufficient proof to log
in and not require a password or some other proof material.
Security Realm: Review
A persistent store is assigned to a security realm to persist

assets such as:
Users and groups

Roles
Policies
Credential maps
The default realm store can be the embedded LDAP

server (default) or an external database.
Some providers use the default realm store, whereas
others rely on another external system.
The default auditing and adjudication providers do not use the persistent stores configured for
their parent security realm.
If you have multiple security providers of the same type configured in the same security realm,
these security providers may use the same security provider database. For example, if you
configure two WebLogic authentication providers in the default security realm (called
myrealm), both WebLogic authentication providers will use the same location in the
embedded LDAP server as their security provider database and thus will use the same users
and groups. Furthermore, if you or an administrator add a user or group to one of the
WebLogic authentication providers, you will see that user or group appear for the other
WebLogic authentication provider as well.
Security Provider Stores
Provider
Brief Description
Default Authenticator
Authenticates users against the default store
LDAP Authenticator
Authenticates users against an external LDAP server
Database Authenticator
Authenticates users against a custom database schema
Windows Authenticator
Authenticates users against the local Windows realm
SAML Authenticator
Generates a SAML token for authenticated users
Default Identity Asserter
Assigns a users identity by using incoming X509

certificates
LDAP X509 Identity

Asserter
Looks up a users identity from an external LDAP, by using

the DN from incoming X509 certificates
SAML Identity Asserter
Assigns a users identity by using incoming SAML tokens
XACML Role Mapper
Assigns logical identities to a user with XML rules defined

in the default store
XACML Authorizer
Authorizes a users access with XML policies defined in the

default store
Default Adjudicator
Requires all authorization providers to grant access
LDAP authentication providers access external LDAP stores. WebLogic Server provides
LDAP authentication providers that access Oracle Internet Directory, Oracle Virtual Directory,
Open LDAP, Netscape iPlanet, Microsoft Active Directory, and Novell NDS stores.
An RDBMS authentication provider is a username/passwordbased authentication provider
that uses a relational database (rather than an LDAP directory) as its data store for user,
password, and group information.
The SAML authentication provider may be used with the SAML 1.1 or SAML 2.0 Identity
Assertion provider to allow virtual users to log in via SAML. This provider creates an
authenticated subject by using the user name and groups retrieved from a SAML assertion.
The Windows NT authentication provider uses account information defined for a Windows NT
domain to authenticate users and groups and to permit Windows NT users and groups to be
listed in the Administration Console.
The default WebLogic identity asserter can validate X509 certificates provided by clients. The
LDAP X509 Identity Assertion provider receives an X509 certificate, looks up the LDAP object
for the user associated with that certificate, ensures that the certificate in the LDAP object
matches the presented certificate, and then retrieves the name of the user from the LDAP
object. A correlation must exist between the Subject DN in the certificate and the location of
the object for that user in the LDAP server.
Some Security Providers
A simple LDAP server runs on each server in a domain

(servers/<server>/data/ldap).
The admin server tracks changes to the LDAP and pushes

them out to managed servers when available.
Replica
Admin
Server
Replica
Master LDAP
The WebLogic Server embedded LDAP server for a domain consists of a master LDAP server
that is maintained in the domains Administration Server, as well as a replicated LDAP server
that is maintained in each Managed Server in the domain. When changes are made using a
Managed Server, updates are sent to the embedded LDAP server on the Administration
Server. The embedded LDAP server on the Administration Server maintains a log of all
changes. The embedded LDAP server on the Administration Server also maintains a list of
Managed Servers and the current change status for each one. The embedded LDAP server
on the Administration Server sends appropriate changes to each Managed Server and
updates the change status for each server. This process occurs when an update is made to
the embedded LDAP server on the Administration Server. However, depending on the
number of updates, it may take several seconds or more for the change to be replicated to the
Managed Server.
Embedded LDAP: Review
By default, every server records a backup copy of its LDAP

data once every 24 hours to
<server>/data/ldap/backup.
As part of troubleshooting security problems, try:
Restoring the LDAP to a prior version

Comparing the contents of two versions
1
2
To update the embedded LDAP settings for all servers in a domain:

1. From the Domain Structure panel, click the domains name.
2. Click the Security > Embedded LDAP tab.
3. Edit embedded LDAP options, including:
- Backup Hour, Backup Minute: These two values are used in conjunction to
determine the time of day at which the embedded LDAP server data files are
backed up. At the specified time, WLS suspends writes to the embedded LDAP
server, creates a zip file of the latest LDAP data found in the servers
data/ldap/ldapfiles directory, and records it to the servers
data/ldap/backup directory.
-
Backup Copies: The maximum number of backup copies that should be made for
the embedded LDAP server. This limits the number of zip files that are created. A
value of 0 disables the backup feature.
Embedded LDAP Backups
If a managed servers security behavior does not seem to

correspond to the latest changes, try the following:
Force an LDAP refresh at startup.
Delete the LDAP replica from the file system.
The most common LDAP synchronization problem occurs when you change the domains
administrative account while a managed server is not running. Consequently, when the
managed server later boots and cannot locate the correct credentials in its LDAP replica, it
fails to start.
You can configure the replication behavior of the embedded LDAP server. Select the name of
the domain in the Administration Console. Then click the Security > Embedded LDAP Server
tab and use these attributes:
Refresh Replica At Startup: Specifies whether the embedded LDAP server in a
Managed Server should refresh all replicated data at boot time. This setting is useful if
you have made many changes when the Managed Server was not active, and you want
to download the entire replica instead of having the Administration Server push each
change to the Managed Server.
Master First: Specifies whether a Managed Server should always connect to the
embedded LDAP server on the Administration Server, instead of connecting to the local
replicated LDAP server
Embedded LDAP Synchronization Issues
1. Update the default generated LDAP password.

2. Obtain an LDAP browsing/editing tool.
3. Connect to the servers default port (or SSL port).
a. Set the base distinguished name (DN) to your domain name.
b. Set the users DN to Admin.
When you delete a WebLogic Security provider, the security data in the embedded LDAP
server is not automatically deleted. The security data remains in the embedded LDAP server
in case you want to use the provider again. Use an external LDAP browser to delete the
security data from the embedded LDAP server.
To view the contents of the embedded LDAP server through an LDAP browser:
1. In the WLS console, change the Credential for the embedded LDAP server. Select your
domain name and then click the Security > Embedded LDAP tab. Reboot WebLogic
Server. Note that changing the credential can affect the operation of the domain. Do not
perform this step on a production server.
2. Download and install an external LDAP browser. You can find one LDAP browser at the
following location: http://www.openldap.org/.
3. Launch the LDAP browser and supply the host name and port of your WebLogic Server.
Set the base DN to dc=<mydomain>, where <mydomain> is the name of the WebLogic
Server domain that you are using. Set the user DN to cn=Admin. Set the password to
the credential you specified earlier.
4. Use the LDAP browser to navigate the hierarchy of the embedded LDAP server,
including users, groups, roles, and policies.
Viewing Embedded LDAP Contents
LDAP:
Is a TCP/IP protocol
Provides a hierarchical lookup and search service
Models information as a tree of entries, whose attributes are
defined by a schema or object class
Defines default schemas for common entities such as people
and groups
Supports SSL
Entries:
Identify their locations in the tree by using a distinguished
name (DN)
Can be referrals that link to other LDAP servers
The Lightweight Directory Access Protocol (LDAP) is based on the X.500 standard but is
significantly simpler and more readily adapted to meet custom needs. Unlike X.500, LDAP
supports TCP/IP, which is necessary for Internet access. The core LDAP specifications are all
defined in Request for Comments (RFCs).
LDAP is a protocol that provides access to a compliant directory via TCP/IP. The strengths of
LDAP-compliant directories include speed, simplicity, and the ability to be replicated and
distributed across several servers. An LDAP directory can be used to store a great deal of
information, from user login credentials to company telephone directories.
Unlike databases that are designed for processing hundreds or thousands of changes per
minute, LDAP directories are heavily optimized for read performance. LDAP is intentionally
designed for environments where search operations are much more common than modify
operations.
LDAP Version 3 implements a referral mechanism that allows servers to return references to
other servers as a result of a directory query. This makes it possible to distribute directories
globally by partitioning a directory information tree (DIT) across multiple LDAP servers.
LDAP Concepts
Domain or
organization
myldap.com
Organizational
unit
Employees
Mike
Mark
Contractors
Mary
Mimi
Mike
Person
DN: uid=mike, ou=Contractors, o=myldap.com
Like a computers file system, directories are viewed as a tree. Each entry in a directory is
called an object. These objects are of two types: containers and leaves. A container is like a
folder; it contains other containers or leaves. A leaf is simply an object at the end of a tree. A
tree cannot contain an arbitrary set of containers and leaves. It must match the schema
defined for the directory.
The top level of the LDAP directory tree is the base, referred to as the base DN. A base DN
can be one of several forms. Here are some examples:
A domain name, broken into components (dc=Acme,dc=com)
An organization name (o=Acme Corp)
An organization name along with a country (o=Acme Corp,c=India)
Organizational units are standard LDAP object classes that act as containers for other entries.
The identifying attribute for an organizational unit is ou. The standard LDAP schema also
defines a person class and a group class, which is a collection of people.
The person type also includes such other attributes as Common Name (cn), Unique Identifier
(uid), Last Name (sn), and Password (userpassword).
LDAP Structure
Searching for LDAP entries involves:

1. The base DN from which to start searching
2. A search filter that specifies the:
Search criteria in terms of attribute values
The type or object class of the desired entries
3. An indication whether or not the search should include any

child entities
An LDAP search filter that finds all people whose user ID begins with m,
while ignoring those whose name is Mike:
(&(uid=m*)(!cn=Mike*)((objectclass=person))
The & represents a logical and when combining multiple expressions that have been
grouped together in parentheses. Similarly, the | represents a logical or, and a !
represents a logical not.
Search filters can specify one or more object classes. Here is an example:
(&(&(objectClass=person)(objectClass=organizationalPerson))(objectCl
ass=user))
LDAP Search Operations
1.
2.
3.
4.
Shut down the domain.

Back up the master LDAP data.
Run the AdminAccount command-line tool.
Remove the admin servers
data/ldap/DefaultAuthenticator<realm>Init.initia
lized file.
5. Restart the Administration Server and enter the new

password (remove any boot.properties file temporarily).
6. Set the original password by using the console or WLST.
java weblogic.security.utils.AdminAccount weblogic
temppass1 /home/oracle/domains/mydomain/security
WebLogic Server provides a method of resetting the administrative password if the password
becomes lost. This tool re-creates part of the domains LDAP structure. However, it is only
effective on the default administrative user assigned to the Admin role.
To preserve additional LDAP customizations, perform an export of the realms providers prior
to these steps by using the console or WLST. After these steps are complete, use the console
or WLST to import the data back into the reset realm.
Resetting Admin Password in Embedded LDAP
Each server in a domain caches security data retrieved

from the realms database store.
If configured, the admin server can use JMS to broadcast
notifications to managed servers:
When security data changes
To force them to refresh their caches
If notifications are not configured, managed servers may

temporarily use old security data.
Cache
Admin
Server
Cache
JMS
Managed
Server
JMS notifications enable the security data (which is contained in the RDBMS security store
and managed by security providers in the realm) to be synchronized among all server
instances in the domain. If you do not configure a JMS topic that can be used by the RDBMS
security store when configured in a multi-server or clustered domain, care should be taken
when making security policy or security configuration updates. If no JMS topic is configured, it
may be necessary to reboot the domain to ensure that all server instances function
consistently with those security updates.
To configure JMS notifications:
1. Create a new JMS server, JMS module, and JMS topic.
2. Edit your security realm in the console.
3. Click the Configuration > RDBMS Security Store tab.
4. Edit the fields in the section labeled Server Synchronization Configuration. These fields
include JNDI User Name, JNDI Password, JMS Topic, JMS Topic Connection Factory,
and Notification Properties.
Database Store Cache Synchronization Issues
The WebLogic auditing provider:

Creates a detailed record of all security changes and
decisions within a domain (DefaultAuditRecorder.log)
Can also create a record of all domain configuration
changes
Is not enabled by default
Security Events
Configuration Events
Auditing
Provider
Audit Log
Auditing is the process whereby information about operating requests and the outcome of
those requests are collected, stored, and distributed for the purposes of nonrepudiation. In
WebLogic Server, an auditing provider implements this electronic trail of computer activity.
WebLogic Server includes a sample auditing provider, but, by default, it is not activated for a
new security realm. The WebLogic auditing provider records information from a number of
security requests, which are determined internally by the WebLogic Security Framework. The
WebLogic auditing provider also records the event data associated with these security
requests and the outcome of the requests.
You can also configure the Administration Server to emit audit messages that enable tracking
of configuration changes in a domain. This provides a record of changes made to the
configuration of any resource within a domain, as well as invocations of management
operations on any resource within a domain. Configuration audit records can be saved to a
log file, sent to an auditing provider in the security realm, or both.
All auditing information recorded by the WebLogic auditing provider is saved in
<domain>/servers/<server>/logs/DefaultAuditRecorder.log by default.
Although an auditing provider is configured for a security realm, each server writes auditing
data to its own log file in the server directory.
Auditing Provider
Typical security events include:

An authentication or identity assertion attempt
A new role or policy
A locked/unlocked user account
Security events have the following characteristics:

Name
Severity (Warning, Error, Success, and so on)
Zero or more context attributes:
Protocol, port, address

HTTP headers
EJB method parameters
SAML tokens
In addition to the events listed in the slide, the default WebLogic auditing provider records the
following types of security events:
The lock on a user account expires.
A security policy is used and an authorization decision is made.
A role definition is used.
A role or policy is removed or undeployed.
The WebLogic auditing provider audits security events of the specified severity, as well as all
events with a higher numeric severity rank. For example, if you set the severity level to
ERROR, the WebLogic auditing provider audits security events of severity level ERROR,
SUCCESS, and FAILURE. You can also set the severity level to CUSTOM, and then enable
the specific severity levels that you want to audit, such as ERROR and FAILURE events only.
An audit event includes a context object that can hold a variety of different attributes,
depending on the type of event. When you configure an auditing provider, you specify which
context attributes are recorded for each event. By default, no context attributes are audited.
Security Audit Events
1
3
2
Context attributes
to record
Minimum severity
to record
1. In the left pane, select Security Realms and click the name of the realm you are
configuring (for example, myrealm). Next, click the Providers tab in the right pane.
Then click the tab for auditing providers and click New.
2. Select the new provider and click the Configuration > Provider-Specific tab.
3. Update the following fields if desired:
- Active Context Handler Entries: Specifies which context attributes are recorded
by the auditing provider, if present within an event. Use the arrow buttons to move
the available entries to the Chosen list.
- Rotation Minutes: Specifies how many minutes to wait before creating a new
audit log file. After this time has elapsed, the audit file is closed and a new one is
created.
- Severity: The minimum severity level of an event to record
Configuring the Auditing Provider
Flag
Description
DebugSecurityRealm
Trace the initialization of the realms providers and

the loading of initial data from the default store.
DebugSecurityAtn
Trace the authentication and management of users

and groups.
DebugSecurityRoleMap
Trace role policy evaluations and results.
DebugSecurityAtz
Trace authorization policy evaluations and access

decisions.
DebugSecurityAdjudicator
Trace final authorization decisions.
DebugSecurityUserLockout
Trace the locking/unlocking of user accounts based

on the number of invalid login attempts.
DebugSecuritySAML*
Trace the processing and/or generation of SAML

tokens.
Realm Debug Flags
Normal authentication log messages with DebugSecurityAtn enabled:

<... LoginModuleClassName=weblogic.security.providers.
authentication.LDAPAtnLoginModuleImpl, ControlFlag=...
required>
...
Provider and control
<LDAP ATN LoginModule initialized>
flag being used
...
<LDAP Atn Login>
...
<getConnection return conn:LDAPConnection ...>
<getDNForUser search(...), base DN & below)>
Search for
...
user in LDAP.
<LDAP Atn Authenticated User myuser123>
<List groups that member: myuser123 belongs to>
Search for
...
groups in LDAP.
<search(...), base DN & below)>
<LDAP Atn added group Payroll to user myuser123>
...
In WebLogic Server, authentication providers are used to prove the identity of users or system
processes. Authentication providers also remember, transport, and make identity information
available to various components of a system (via subjects) when needed. During the
authentication process, a principal validation provider provides additional security protections
for the principals (users and groups) contained within the subject by signing and verifying the
authenticity of those principals.
Whether the client is an application, applet, Enterprise JavaBean (EJB), or servlet that
requires authentication, WebLogic Server uses the Java Authentication and Authorization
Service (JAAS) classes to reliably and securely authenticate to the client. JAAS implements a
Java version of the Pluggable Authentication Module (PAM) framework, which permits
applications to remain independent from underlying authentication technologies. Therefore,
the PAM framework allows the use of new or updated authentication technologies without
requiring modifications to your application.
A LoginModule does much of the work of authentication. All LoginModules are responsible for
authenticating users within the security realm and for populating a subject with the necessary
principals (users/groups). LoginModules that are not used for perimeter authentication also
verify the proof material submitted (for example, a users password).
Typical Authentication Trace
Continued:
Create internal user identity

data structure (subject).
<LDAP Atn Principals Added>

...
<...JAASLoginServiceImpl.login subject=Subject:
Principal: myuser123
Principal: Payroll
>
...
<Generated signature and signed WLS principal myuser123>
...
<Generated signature and signed WLS principal Payroll>
...
<...PrincipalValidationServiceImpl.validate(Principals)
validated all principals>
Digitally sign identities to validate

during subsequent requests.
Authentication providers rely on principal validation providers to sign and verify the
authenticity of principals (users and groups) contained within a subject. Such verification
provides an additional level of trust and may reduce the likelihood of malicious principal
tampering. Verification of the subjects principals takes place when accepting incoming
requests from a previously authenticated RMI client. The authenticity of the subjects
principals is also verified when making authorization decisions.
As part of a successful authentication, principals are signed and stored in a subject for future
use. A principal validation provider signs principals, and an authentication providers
LoginModule actually stores the principals in the subject. Later, when a caller attempts to
access a principal stored within a subject, a principal validation provider verifies that the
principal has not been altered since it was signed, and the principal is returned to the caller
(assuming all other security conditions are met).
A principal validation provider is a special type of security provider that primarily acts as a
helper to an authentication provider. The main function of a principal validation provider is to
prevent malicious individuals from tampering with the principals stored in a subject.
Typical Authentication Trace
Normal log messages with DebugSecurityRoleMap:

<...RoleMappingServiceImpl.getRoles Resource=type=<url>,
application=hrapp, contextPath=/hrweb, uri=/index.jsp,
httpMethod=GET>
<XACML RoleMapper getRoles(): input arguments:>
Resource being
<Subject: 3
accessed
Principal = ...WLSUserImpl("myuser")
Principal = ...WLSGroupImpl("Benefits")
Subject passed
Principal = ...WLSGroupImpl("Payroll")
to role mapper
>
...
<Evaluate ...string-is-in(Payroll,[everyone, users, Benefits,
Payroll]) -> true>
<primary-rule evaluates to Permit>
<XACML RoleMapper: accessing role PayrollUsers: GRANTED>
... authorization ...
Evaluate conditions for

all roles in scope.
Role mapping is the process whereby principals (users or groups) are dynamically mapped to
security roles at run time. In WebLogic Server, a Role Mapping provider determines what
security roles apply to the principals stored in a subject when the subject is attempting to
perform an operation on a WebLogic resource. Because this operation usually involves
gaining access to the WebLogic resource, Role Mapping providers are typically used with
authorization providers.
The WebLogic Security Framework calls each Role Mapping provider that is configured for a
security realm as part of an authorization decision. The result of the dynamic security role
computation (performed by the Role Mapping providers) is a set of security roles that apply to
the principals stored in a subject at a given moment. These security roles can then be used to
make authorization decisions for protected WebLogic resources, as well as for resource
container and application code.
The role security policies are represented as a set of expressions or rules that are evaluated
to determine whether a given security role is to be granted. These rules may require the Role
Mapping provider to substitute the value of context information obtained as parameters into
the expression. In addition, the rules may also require the identity of a user or group principal
as the value of an expression parameter.
Typical Role Mapping Trace
Normal log messages with DebugSecurityAtz:
Type of authorization
policy (role-based)
<...AccessDecisionServiceImpl.isAccessAllowed Roles=[
"PayrollUsers" ]>
<... AccessDecisionServiceImpl.isAccessAllowed
Resource=type=<url>, application=hrapp, contextPath=/hrweb,
uri=/index.jsp, httpMethod=GET>
...
Resource being
<Evaluate ...string-isaccessed
in(PayrollUsers,[PayrollUsers,Anonymous]) -> true>
<primary-rule evaluates to Permit>
Evaluate policy
...
conditions and
<...AccessDecisionServiceImpl.isAccessAllowed
return
decision.
AccessDecision returned PERMIT>
Authorization is the process whereby the interactions between users and WebLogic resources
are controlled, based on user identity or other information. In other words, authorization
answers the question, What can you access? In WebLogic Server, an authorization provider
is used to limit the interactions between users and WebLogic resources to ensure integrity,
confidentiality, and availability.
Like LoginModules for authentication providers, an access decision is the component of an
authorization provider that actually answers the question, Is access allowed? Specifically, an
access decision is asked whether a subject has permission to perform a given operation on a
WebLogic resource with specific parameters in an application. Given this information, the
access decision responds with a result of PERMIT, DENY, or ABSTAIN.
The WebLogic Security Framework delegates the job of reconciling any discrepancies among
the results rendered by the configured authorization providers access decisions to the
adjudication provider. The adjudication provider determines the ultimate outcome of the
authorization decision. The adjudication provider returns either a TRUE or FALSE verdict,
which is forwarded to the resource container through the WebLogic Security Framework.
Typical Authorization Trace
When authentication debugging is enabled, any LDAP

providers also generate an ldap_trace.logATN file.
This log includes details about all communication between

the server and external LDAP systems.
Connected to ldaps://ldap.mycompany.com:389
BindRequest {version=3, name=cn=LDAPAdmin, authentication=**}
BindResponse {resultCode=0}
...
SearchRequest {baseObject=dc=mycompany,dc=com, scope=2,
derefAliases=0,sizeLimit=0, timeLimit=0, attrsOnly=false,
filter=(&(uniquemember=uid=weblogic,dc=mycompany,dc=com)
(objectclass=groupofuniquenames)), attributes=cn}
SearchResult {resultCode=0}
SearchResponse {entry='cn=Operators,dc=mycompany,dc=com',
attributes='LDAPAttribute {type='cn', values='Operators'}'}
This internal log file is undocumented and therefore its availability, location, and contents are
subject to change in all future WebLogic Server revisions. Currently this log file is generated
at your domains root directory.
LDAP Trace Log
Multiple providers of the same type are executed in the

order in which they are defined.
Control flags also determine the processing logic as each
provider is executed.
Flag
Success Action
Failure Action
REQUIRED
Execute next provider.
Execute next provider, but

outcome is still failure.
REQUISITE
Do not execute next provider.
SUFFICIENT
Do not execute next

provider.
OPTIONAL
Each security realm must have at least one authentication provider configured. The WebLogic
Security Framework supports multiple authentication providers for multipart authentication.
Therefore, you can use multiple authentication providers as well as multiple types of
authentication providers in a security realm.
The order in which WebLogic Server calls multiple authentication providers can affect the
overall outcome of the authentication process. The authentication providers table lists the
authentication providers in the order in which they will be called. By default, authentication
providers are called in the order in which they were configured. Use the Reorder button on the
Security Realms > Providers > Authentication page in the Administration Console to change
the order in which authentication providers are called by WebLogic Server and listed in the
console.
When you configure multiple authentication providers, you also use the control flag for each
provider to control how the authentication providers are used in the login sequence. When
additional authentication providers are added to an existing security realm, by default the
control flag is set to Optional. If necessary, change the setting of the control flag and the order
of authentication providers so that each authentication provider works properly in the
authentication sequence.
Authentication Provider Control Flags
The overall authentication succeeds only if all Required and Requisite providers succeed. If a
Sufficient provider is configured and succeeds, then only the Required and Requisite providers
prior to that Sufficient provider need to have succeeded for the overall authentication to
succeed. If no Required or Requisite providers are configured for an application, then at least
one Sufficient or Optional provider must succeed.
WebLogic Server includes:

A base LDAP Authenticator that can be configured to
support any compliant vendor
Vendor-specific LDAP authenticators, whose attributes are
set to vendor-specific defaults for convenience
LDAP Authenticator
OpenLDAP
Authenticator
Oracle Internet Directory

Authenticator
Active Directory
Authenticator
Each LDAP authentication provider stores user and group information in an external LDAP
server. The providers differ primarily in how they are configured by default to match typical
directory schemas for their corresponding LDAP server. For example, the generic
authenticator is configured to use a persons common name (cn) as a user ID, while by
default Oracle Internet Directory uses the uid attribute for this purpose. Similarly, the names
of object classes used to represent people or groups may vary from vendor to vendor. For
example, the generic authenticator is configured to use the object class
groupofuniquenames, while by default Oracle Internet Directory uses the object class
groupofnames.
WebLogic Server does not support or certify any particular LDAP servers. Any LDAP v2- or
v3-compliant LDAP server should work with WebLogic Server.
If an LDAP authentication provider is the only configured authentication provider for a security
realm, you must have the Admin role to boot WebLogic Server and use a user or group in the
LDAP directory. If the LDAP user who boots WebLogic Server is not properly added to a
group that is assigned to the Admin role, and if the LDAP authentication provider is the only
authentication provider with which the security realm is configured, WebLogic Server cannot
be booted.
External LDAP Authentication Providers
Specify the LDAP host and connection credentials.

Indicate whether or not SSL should be used.
Optionally, specify connection retry and failover behaviors.
Supply the following for users and for groups:
Base DN to start searching from

Object class name
Attribute name
Search filter expression for all entries
Search filter expression for a specific entry
For groups, additionally specify:

Membership attribute
Whether membership is static or dynamic
Host: A single host name or IP address, or a comma-separated list of host names and
ports to try
Port: The port number of the LDAP server
Principal: The distinguished name (DN) of the LDAP user that WebLogic Server should
use to connect to the LDAP server
Credential: The credential (usually a password) used to connect to the LDAP server
SSL Enabled: Specifies whether the SSL protocol should be used when connecting to
the LDAP server. For a more secure deployment, Oracle recommends using the SSL
protocol to protect communications between the LDAP server and WebLogic Server.
Connect Timeout: Specifies the maximum number of seconds to wait for the
connection to the LDAP server to be established. If it is set to 0, there is no maximum
time limit, and WebLogic Server waits until the TCP/IP layer times out to return a
connection failure.
Dynamic Member URL Attribute: The attribute of the dynamic LDAP group object that
specifies the URLs of the members of the dynamic group
LDAP Provider Configuration: Overview
User Dynamic Group DN Attribute: A user attribute indicating its dynamic group
membership. If such an attribute does not exist, the provider determines if a user is a
member of a group by evaluating the URLs on the dynamic group. If a group contains
other groups, WebLogic Server evaluates the URLs on any of the descendants
(subgroups) as well.
Location in tree to
start searching from
How to retrieve
all groups?
How to retrieve
a group given
its name?
Entity attribute that
contains group name
Schema used to
model a group
Entity attribute that
contains members
Many LDAP servers have a concept of dynamic (or virtual) groups. These are groups that,
rather than consisting of a list of users and groups, contain some policy statements, queries,
or code that define the set of users that belong to the group. Even if a group is marked
dynamic, users must log out and log back in before any changes in their group memberships
take effect. The term dynamic describes the means of defining the group and not any runtime
semantics of the group within WebLogic Server.
Group Base DN: The base distinguished name (DN) of the tree in the LDAP directory
that contains group definitions
All Groups Filter: An LDAP filter expression for finding all groups beneath the base
group distinguished name (DN). If a filter is not specified, a simple default search filter is
created based on the group object class.
Group From Name Filter: An LDAP filter expression for finding a group given the name
of the group. If a filter is not specified, a simple default search filter is created based on
the group schema.
Static Group Name Attribute: The attribute of a group object that specifies the name of
the group
Static Member DN Attribute: The attribute of a group object that specifies the
distinguished names (DNs) of the members of the group
.
LDAP Provider Configuration: Overview

Connection credentials do not have all the necessary
permissions
Wrong base DN, object class, or attribute for users or
groups
Configured search filter is valid but fails to retrieve any
users or groups
Insufficient maximum level for nested group memberships
(only some members are retrieved)
WLS does not trust the LDAP servers SSL certificate:
CA not in the trust keystore
Host name verification fails.
After the user is authenticated, a search on the groups occurs to get the list of groups to which
this user belongs, and this used to be able to do the role mapping between groups and roles.
If the user does not belong to any group or if the search criteria are not valid, you will see
debug messages similar to the following:
<SecurityDebug> <search(...), base DN & below)>
<SecurityDebug> <Result has more elements: false>
The Max Group Membership Search Level provider field specifies how many levels of group
membership can be searched. This setting is valid only if Group Membership Searching is set
to limited. A value of 0 indicates that only direct groups will be found. That is, when
searching for membership in Group A, only direct members of Group A will be found. If Group
B is a member of Group A, the members will not be found by this search. Any nonzero value
indicates the number of levels to search. For example, if this attribute is set to 1, a search for
membership in Group A will return direct members of Group A. If Group B is a member of
Group A, the members of Group B will also be found by this search. However, if Group C is a
member of Group B, the members of Group C will not be found by this search.
Common LDAP Issues

Enable the security realm auditing provider
Define basic LDAP terminology
Identify and troubleshoot common embedded and external
LDAP issues
Work with authentication providers
Trace security realm actions and decisions
Section Summary
Which of the following is not a typical step in establishing an

SSL session?
a. Exchange certificate chains.
b. View a keystore by using Keytool.
c. Check that certificates are trusted.
d. Negotiate and select a protocol.
e. Generate and share a unique secret key.
Answer: b
Quiz
Name three ways in which WLS establishes trust for a supplied

certificate.
a. Use public keys of trusted authorities.
b. Perform host name verification.
c. Perform role mapping.
d. Synchronize security realm data.
e. Check whether it has expired.
Answer: a, b, e
Quiz
What domain attribute affects how embedded LDAP servers

are synchronized?
a. Private Key Alias
b. Static Group Object Class
c. Refresh Replica at Startup
d. Hostname Verification
e. User Base DN
Answer: c
Quiz
Which of the following is not an available authentication

provider control flag?
a. SUFFICIENT
b. REQUISITE
c. OPTIONAL
d. REQUIRED
e. ALWAYS
Answer: e
Quiz

Identify several WLS scenarios that involve SSL
Describe the fundamentals of SSL and LDAP
communication
Trace SSL and security realm functionality
List some common causes of SSL errors
Troubleshoot the WLS embedded LDAP
Summary

Diagnosing a server that fails to start
Working with authentication control flags
Debugging the WLS security realm
Analyzing LDAP user and group data
Tuning LDAP authentication settings
Practice 11-2
Investigating Security Realm Problems
Troubleshooting Node Manager

Compare the architectures of the Java-based Node
Manager and the script-based Node Manager
Configure a Node Manager to use server start and stop
scripts
Describe some typical Node Manager problems
Initialize a Node Managers security files by using WLST
Tune the SSL settings of the Java Node Manager
Objectives
Node Manager:
Is a process that accepts remote commands to start, stop,
or suspend servers on the same machine
Monitors server availability and restarts failed processes
Periodically monitors server health and kills/restarts
unhealthy processes
Can manage servers for multiple domains
Can use SSL
WLST
Admin Server
Node
Manager
Start/stop
Restart
Monitor
Server
Server
Server
Server instances in a WebLogic Server production environment are often distributed across
multiple domains, machines, and geographic locations. Node Manager is a WebLogic Server
utility that enables you to start, shut down, and restart Administration Server and Managed
Server instances from a remote location. Although Node Manager is optional, it is
recommended if your WebLogic Server environment hosts applications with high-availability
requirements.
A Node Manager process is associated not with a specific WebLogic domain but with a
machine. You can use the same Node Manager process to control server instances in any
WebLogic Server domain, as long as the server instances reside on the same machine as the
Node Manager process.
If a server instance that was started using Node Manager fails, Node Manager automatically
restarts it. If Node Manager fails or is explicitly shut down, on restart it determines the server
instances that were under its control when it exited. Node Manager can restart any failed
server instances as needed.
The Administration Console can be used to issue commands to Node Managers running on
remote machines. WLST (in offline mode) also serves as a Node Manager command-line
interface that can run in the absence of a running Administration Server. You can use WLST
commands to start, stop, and monitor a server instance without connecting to an
Administration Server.
Node Manager (NM): Review
Implementation
Java NM
A Java application that
accepts requests over a
configured port
Script NM
A shell script that relies on
the local SSH server
process to accept remote
communication
Platform Support Windows and UNIX
UNIX
Configuration
nodemanager.properties
Script command-line
arguments
Security
Supports user/password
and/or one-way SSL via
Java keystores
Relies on SSH security,

which is based on simple
symmetric key encryption
Logging
Supports log file rotation
Prints to standard output
weblogic.
NodeManager
wlscontrol.sh
The Java-based Node Manager runs within a Java Virtual Machine (JVM) process.
Configuring SSL for the Java Node Manager involves obtaining identity and trust for the Node
Manager and each Administration and Managed Server with which the Node Manager will be
communicating and then configuring the Node Manager, the Administration Server, and
Managed Servers with the proper identity and trust. In addition, the use of host name
verification must be taken into consideration.
For UNIX and Linux systems, WebLogic Server provides a script-based version of Node
Manager. This script is based on UNIX shell scripts but uses SSH for increased security. If
you are installing WebLogic Server on a Windows system, you must use the Java version of
Node Manager. The scripted version of Node Manager is not supported on Windows.
The script (SSH) version does not provide as much security as the Java-based version.
However, the advantage of the script-based Node Manager is that it can remotely manage
servers over a network that has been configured to use SSH. SSH uses user IDbased
security along with secret key (symmetric) encryption.
The Java version of Node Manager can be used in conjunction with inetd on supported
UNIX systems. inetd allows Node Manager to be automatically restarted on receiving a
request on the configured port. Refer to the documentation for an example configuration.
Criteria
SSH
Node Manager Types: Review
1. Configure nodemanager.properties (Java type only).

2. Create Machine definitions in the console so that the
admin server knows the following about each Node
Manager:
Type
Location (host/port)
The security level (SSL, SSH, none)
3. Define server startup arguments and/or startup scripts.

4. Configure security, such as:
Node Manager username and password

Trusted domain list
SSL between Node Managers and clients (Administration
Server and/or WLST)
A WebLogic Server machine resource associates a particular machine with the server
instances that it hosts, and it specifies the connection attributes for the Node Manager
process on that system. Configure a machine definition for each machine that runs a Node
Manager process and then edit the settings on the machines Node Manager tab in the
Administration Console. In the Listen Address field, enter the DNS name or IP address on
which Node Manager listens. If you set the Type field to SSH or RSH, you should specify
values in the Node Manager Home and Shell Command fields.
Node Manager Configuration: Review
Property
Description
NodeManagerHome
Location of NM property and log files
ListenAddress,
ListenPort
Address and port from which NM will accept

commands
Authentication
Enabled
Require a username/password to use this NM.
StartScriptEnabled
Start servers by using local scripts.
StopScriptEnabled
Stop servers by using local scripts.
StartScriptName
Name of the script used to start local servers

(<domain>/bin/startWebLogic.sh by
default)
StopScriptName
Name of the script used to stop local servers

(<domain>/bin/stopWebLogic.sh by
default)
CrashRecovery
Enabled
Automatically restart servers after machine

restart.
Node Manager properties define a variety of configuration settings for a Java-based Node
Manager process. In many environments, the SSL-related properties in
nodemanager.properties may be the only Node Manager properties that you must
explicitly define. However, nodemanager.properties also contains non-SSL properties
that you might need to specify, depending on your environment and preferences. Any values
supplied on the command line override the values in nodemanager.properties.
By default, the StartScriptEnabled, StopScriptEnabled, and
CrashRecoveryEnabled properties are false. The default value for
AuthenticationEnabled is true.
You can also use nodemanager.properties to specify the default restart parameters for
servers launched from this Node Manager instance:
RestartInterval: The amount of time Node Manager will spend attempting to restart a
failed server. By default, Node Manager will attempt to restart a server indefinitely until the
FAILED_NOT_RESTARTABLE state is reached.
RestartMax: The number of times Node Manager will attempt to restart a failed server within
the interval defined by RestartInterval. RestartMax is recognized only if
RestartInterval is defined.
Basic Java Node Manager Properties
Property
Description
LogFile
Location of NM log (default is nodemanager.log)
LogLimit
Maximum size of NM log before rotation
LogCount
Maximum number of NM log files
LogAppend
Continue to use current log file if NM restarted.
LogLevel
Log severity threshold (FINEST, FINE, INFO,

WARNING, ERROR, ...)
Node Manager log excerpts:

<MyDomain> <Server1> <Rotated server output log to ...>
<MyDomain> <Server1> <Starting WebLogic server with command
line .../startWebLogic.sh>
...
<MyDomain> <Server1> <Server failed so attempting to restart
(restart count = 1)>
Node Manager creates a log file located in NodeManagerHome/nodemanager.log. This

log file stores data about all of the domains administered by Node Manager. Log output is
appended to the current nodemanager.log. Log rotation is disabled by default but can be
enabled by setting LogCount in nodemanager.properties.
For each server instance that it controls, Node Manager can also maintain a log file that
contains stdout and stderr messages generated by the server instance. Use the
LogToStdout property in nodemanager.properties. Node Manager creates the server
output log for a server instance in the server instances logs directory, with the name
<server>.out. If the debug property is enabled as a remote start property for the server
instance, or if the Node Manager debug property is enabled, Node Manager will include
additional debut information in the server output log information. You cannot limit the size of
these log files.
Java Node Manager Logging
Ideally, the Node Manager Java process should always be

running.
On Windows, use installNodeMgrSvr.cmd to help
install the Node Manager as an automatic service.
On UNIX, develop a custom init script that:
Responds to the start argument sent by the init process
Navigates to your NM home directory
Calls <WL_HOME>/common/bin/commEnv.sh
Launches a JVM process for the application
weblogic.NodeManager
WebLogic Server does not provide command scripts for uninstalling and reinstalling the Node
Manager as a UNIX daemon process. Refer to your operating system documentation for
specific instructions. For example, here is one approach that is applicable to most Linux
flavors:
1. Add a new script file (or create a symbolic link to a file at another location) named
S99wlsnmd to /etc/rc.d/rc5.d.
2. Within the script, check for an argument named start. If it is set, launch the Java Node
Manager by using a local JVM.
On Windows, the Node Manager service management tools can be found at
WL_HOME/server/bin. Perform the following steps:
1. Edit uninstallNodeMgrSvc.cmd and installNodeMgrSvc.cmd to specify your
Node Managers listen address and listen port.
2. Run uninstallNodeMgrSvc.cmd to delete any existing Node Manager service.
3. Run installNodeMgrSvc.cmd to reinstall Node Manager as a service, listening on
the updated address and port.
Java Node Manager Availability
Argument
Description
-d
Domain name
-s
Server name
-c
Start/stop servers using local scripts.
-f
Name of the script used to start local servers

(<domain>/bin/startWebLogic.sh by default)
-p
Name of the script used to stop local servers

(<domain>/bin/stopWebLogic.sh by default)
-x
Print debug messages to standard output.
A typical invocation of the script Node Manager from the admin server:
ssh l myuser o PasswordAuthentication=no p 22 192.168.1.1
/u01/mw/wlserver/common/bin/wlscontrol.sh d MyDomain s
ServerA c START
When you use the script Node Manager and you issue commands from your domains
Administration Server or WLST, these clients internally create a SSH or RSH session with the
Node Managers host machine. As part of creating this remote session, the Node Manager
client also indicates the script to run (wlscontrol.sh) along with the necessary arguments.
For troubleshooting purposes, it can be helpful to know the details of this interface. For
example, if you experience issues with the script Node Manager, try to establish the remote
session from the command line without the Administration Server or WLST. The results help
determine whether the issue is related to the configuration of your domain, Node Manager, or
SSH/RSH service.
Node Manager clients use a template to send commands using SSH/RSH. This template
includes variables for which the client (such as the Administration Server) provides values,
including:
%H: Host name of the machine to connect to
%N: Remote Node Managers home directory
%P: Port number of the machine to connect to
%S: Name of the WebLogic Server to start or stop
Basic Script Node Manager Interface
%D: The domain of which this Administration Server and the remote managed server are
members
%C: The name of the command to execute: START, KILL, STAT, GETLOG, or VERSION
The default template resembles the following:

ssh -o PasswordAuthentication=no %H %P wlscontrol.sh -d %D -r %R -s
%S -x -c -f sample_custom_startscript.sh %C
To override this template for a domain, either include the JVM argument -Dweblogic.
nodemanager.ShellCommand when starting the Administration Server, or update the Shell
Command field in the machines configuration by using the Administration Console.
NM starts servers by using parameters provided by clients:

Health Monitoring server attributes in the console
Server Start attributes in the console (if not using scripts)
WLST command arguments
Prior to starting a server, NM records these start settings

at servers/<server>/data/nodemanager/
startup.properties.
If not using start scripts, JAVA_HOME, CLASSPATH, and
other variables are either:
Set using Server Start attributes
Inherited from the NM process environment
To minimize potential issues, Oracle recommends that NM

use standard scripts to start servers.
Node Manager uses the startup.properties file to record the initial configuration when
starting a server as well as when automatically restarting the server later. In general, the
values of these properties are sent by the Administration Server to Node Manager based on
the domains configuration settings. These include:
RestartMax: The number of times Node Manager can attempt to restart the server
RestartDelaySeconds: The number of seconds Node Manager should wait before
attempting to restart the server
AutoRestart: Specifies whether Node Manager can automatically restart this server if
it fails
AutoKillIfFailed: Specifies whether Node Manager should automatically kill the
server if its health status is failed
AdminURL: The URL of the Administration Server
The following additional properties may also be set if you are not using a server start script:
JavaHome: Defines the Java home directory used when starting the server
Node Manager Server Start Parameters
Arguments: Other JVM arguments used when starting the server

ClassPath: The classpath to use when starting a server
SecurityPolicyFile: Specifies the security policy file to use when starting this
server
Admin credentials this

server will boot with
Restart server
automatically?
In the Administration Console, on the Configuration > Server Start tab for a managed server,
specify the startup arguments that Node Manager will use to start this server. If you do not
specify startup arguments for the server and do not indicate that Node Manager should use a
start script, Node Manager uses its own default settings to start the server. Although these
defaults are sufficient to boot a server, ensure a consistent and reliable boot process by
explicitly specifying the scripts or arguments to use. The console arguments are used for
starting managed servers only. They are not used if Node Manager starts an Administration
Server.
A server can monitor key aspects of its subsystems and report when a subsystem is not
functioning properly. If the server is running under a Node Manager, the Node Manager can
automatically restart a server with an unhealthy subsystem. Select a server in the console and
edit its Configuration > Health Monitoring tab.
You can also use the nmGenBootStartupProps WLST online command to test the
generation of the startup.properties file for a given server, without actually requesting
Node Manager to start that server. This command requires that you are connected to an
Administration Server and takes a single argumentthe name of the managed server.
Configuring Server Start Parameters
Use the console or WLST to remotely monitor the status and

logs of all registered Node Managers.
The Administration Console gives you the ability to download and view the log files for all
Node Managers in use by the current domain. The Node Manager must be running in order to
accept this command. Simply select a machine and click the Monitoring > Node Manager Log
tab. If this tab does not display any log messages, use the Monitoring > Node Manager Status
tab to determine whether this machines Node Manager is currently available or unreachable.
The same functionality is also available from WLST.
You can also download the output messages for a server being managed by a Node
Manager, if the Node Manager has been configured to direct server output to a file. Select a
specific server in the console and click the Control > Remote Start Output tab.
Monitoring Node Managers
Connect to a Node Manager and manage servers on a machine:

nmConnect('myuser','mypassword','myhost',5556,'MyDomain')
nmStart('ServerA')
...
nmKill('ServerC')
Connect to a Node Manager and download its log:

nmLog('ServerB')
Connect to a Node Manager and download a server log:

nmServerLog('ServerB')
WLST offline serves as a Node Manager command-line interface that can run in the absence
of a running Administration Server. You can use WLST commands to start, stop, and monitor
a server instance without connecting to an Administration Server. However, if an
Administration Server is available, the preferred approach is to connect to this server and use
it to issue commands to Node Manager.
nm(): Determines whether WLST is currently connected to Node Manager
nmConnect(): Connects WLST to Node Manager to establish a session. After
connecting to Node Manager, you can invoke any Node Manager commands via WLST.
Once connected, the WLST prompt displays as follows, where <domainName>
indicates the name of the WebLogic domain that is being managed:
wls:/nm/<domainName>. A command named nmDisconnect is also available.
nmStart(): Starts the specified server in the current WebLogic domain by using Node
Manager
nmKill(): Kills the specified server instance that was started with Node Manager
nmServerStatus(): Returns the status of the specified server that was started with
Node Manager
Node Manager: WLST Examples
Problem Description
Invalid nodemanager.properties
Java NM
Script NM
Invalid SSH script template
Invalid Node Manager authentication setup
Missing server startup credentials
Target domain is not trusted
Invalid SSL identity or trust configuration
If you misspell the name of a property in nodemanager.properties, the Java Node

Manager may not print a warning message and may simply use the default value for that
property. This mistake can significantly affect Node Manager behavior.
To override this template for a domain, either include the JVM argument -Dweblogic.
nodemanager.ShellCommand when starting the Administration Server, or update the Shell
Command field in the machines configuration using the Administration Console. If you
experience issues with the script Node Manager, try to establish the remote session from the
command line without the Administration Server or WLST. The results will help determine
whether the issue is related to the configuration of your domain or Node Manager, or whether
the issue is related to the actual SSH/RSH command being used.
When communication between the Administration Server and a Node Manager uses SSL, the
administrator server becomes, in effect, an SSL client and performs all the same handshake
and validation tasks that any SSL client performs, such as a Web browser.
Start NM without using a nodemanager.properties

file, and NM will create a template file with default values.
Compare the generated template to your version to
troubleshoot configuration issues.
nodemanager.properties:
DomainsFile=nodemanager.domains
LogLimit=0
PropertiesVersion=10.3
AuthenticationEnabled=true
LogLevel=Info
...
The first time Node Manager starts, it checks the path specified by NodeManagerHome for
the existence of the nodemanager.properties file. If one is not found, a new file is
created. When Node Manager is starting and the home path is unknown, the current directory
is used instead. However, as a best practice, you should always specify a home path.
Generating Template Properties for Java NM
Any credentials required by NM to manage a specific

domain are stored in the encrypted file <domain>/
config/nodemanager/nm_password.properties.
Similar to boot.properties, any changes made to this file
are automatically encrypted again when NM is restarted.
Client
user/password
Node
Manager
Domain
nm_password.properties
The nm_password.properties file contains the Node Manager username and password.
These are used to authenticate the connection between a client (for example, the
Administration Server) and Node Manager. These credentials are independent from the WLS
administrative account that is used to boot servers. The nm_password.properties file
must exist on each physical machine that runs Node Manager. However, the Node Manager
username and password do not have to be identical for every domain running on the same
machine. If you edit nm_password.properties manually, you must restart Node Manager
for the changes to take effect.
Node Manager Authentication
You can use the console to update the credentials used by the
admin server to access Node Managers.
When you create a new domain, it automatically generates a random username and
password that the Administration Server will use to access Node Managers. Select the name
of your domain in the Administration Console, click the Security tab, and then click Advanced.
If these credentials do not match those indicated in the nm_password.properties file on a
Node Managers host machine, access is denied by the Node Manager. As a best practice, do
not edit these generated credentials. Instead, use the nmEnroll WLST command to
synchronize the nm_password.properties file with these credentials.
Configuring Node Manager Credentials
The nodemanager.domains file specifies:

A list of domains that can be managed from this NM
The local file system location of each trusted domain
nodemanager.domains:
MyDomain1=/u01/domains/MyDomain1
MyDomain2=/u01/domains/MyDomain2
...
The nodemanager.domains file specifies the domains that a Node Manager instance
controls, as well as their locations on the file system. Stand-alone clients thus do not need to
specify the domain directory explicitly. This file provides additional security by restricting Node
Manager client access to the domains listed in this file.
After the system is restarted, Node Manager checks each managed domain specified in the
nodemanager.domains file to determine whether there are any server instances that were
not cleanly shut down. This is determined by the presence of any lock files which are created
by Node Manager when a WebLogic Server process is created. This lock file contains the
process identifier for the WebLogic Server startup script. If the lock file exists but the process
ID is not running, Node Manager will attempt to automatically restart the server if configured
to do so.
If you are using the Java Node Manager, you can disable the use of this file for security
purposes by using the DomainsFileEnabled property in nodemanager.properties. If
you simply want to change the default name of this file, use the DomainsFile property.
Node Manager Trusted Domains
For convenience, you can also use WLST to initialize NM

so that it accepts commands from a specific domain.
The nmEnroll() command:
Generates nm_password.properties and initializes it
with the NM credentials configured for the domain to which
you are currently connected
Adds the domain to the nodemanager.domains file
Establish trust between a Node Manager and a running admin server:

connect('mydomainuser','mypassword','myadminhost:7001')
nmEnroll('/u01/domains/MyDomain1','/u01/nodemanager')
Not required if running
WLST from this location
Use the nmEnroll() command to create and update the necessary Node Manager
configurations files after creating a domain. You should run nmEnroll() on each machine
that is running a managed server. Additionally, you should run nmEnroll() for each domain
directory on each machine. WLST must be connected to an Administration Server to run this
command. WLST does not need to be connected to Node Manager.
The following files are created or updated:
nm_password.properties
SerializedSystemIni.dat (used to encrypt/decrypt credentials)
nodemanager.domains
Specify the path of the root domain directory to which you want to save
nm_password.properties and SerializedSystemIni.dat. This argument defaults to
the directory in which WLST was started.
Also specify the path of your Node Manager home and the nodemanager.domains file. This
argument defaults to <WL_HOME>/common/nodemanager, where <WL_HOME> refers to the
top-level installation directory for WebLogic Server.
Machine Enrollment
Due to the nature of NM, server startup credentials must

be supplied in boot.properties files.
If the client sends startup credentials to NM, it

automatically generates the
<server>/data/nodemanager/boot.properties
file for the server to boot against.
Node Manager output:

<INFO> <MyDomain> <Server1> <Boot identity properties saved to
".../servers/Server1/data/nodemanager/boot.properties">
<INFO> <MyDomain> <Server1> <Startup configuration properties
saved to
".../servers/Server1/data/nodemanager/startup.properties">
...
Any server instance that is started by Node Manager encrypts and saves the credentials with
which it started in a server-specific boot.properties file, for use in automatic restarts.
You can also use the WLST online command nmGenBootStartupProps to test the
generation of the boot.properties and startup.properties files for a given server
without actually requesting the Node Manager to start that server. This command requires you
to be connected to an Administration Server.
When you invoke Node Manager to start an Administration Server, the boot identity can be
provided on the command line or obtained from the Administration Servers
boot.properties file. For convenience, the WebLogic Server Configuration Wizard
initializes the boot.properties file and the startup.properties file for an
Administration Server when you create the domain.
Server Boot Identity
Property
Description
SecureListener
Require SSL.
Keystores
Use the demo keystore or a

custom one.
CustomIdentityKeyStoreFileName
Node Managers identity keystore
CustomIdentityKeyStorePassPhrase
CustomIdentityAlias
CustomIdentityPrivateKeyPass
Phrase
WLST or Admin
Server
Trust Certs
T3S
Node Manager
Server Cert and
Private Key
Node Manager supports one-way SSL and keystore-based SSL artifacts. For backward
compatibility, additional non-keystore and two-way SSL properties are supported by Node
Manager but should not be used on new WebLogic Server installations.
By default, the Keystores property is set to the value DemoIdentityAndDemoTrust,
meaning that it will use the sample certificate and associated private key found in your
WebLogic installation at server/lib/DemoIdentity.jks. For production environments,
change this value to CustomIdentityAndJavaStandardTrust and use the remaining
properties to specify the location and credentials for your keystore file and its contents.
As with all SSL communication, the use of host name verification must be taken into
consideration. By default, the host name present in the Node Managers identity certificate
must match the host name used by the Administration Server or WLST to access this Node
Manager.
An additional property, named CipherSuite, is available to indicate which protocol and
encryption algorithms are preferred. The available options depend on your Java SSL
implementation. Enable SSL debugging on a server to view a list of the available ciphers in
your WebLogic Server installation. Currently, the default value used by Node Manager is
TLS_RSA_EXPORT_WITH_RC4_40_MD5.
Configuring Node Manager SSL
Which is not an available configuration parameter for the Java

Node Manager?
a. StopScriptEnabled
b. LogCount
c. AuthenticationEnabled
d. StartScriptName
e. SSHEnabled
Answer: e
Quiz
Name three components of the Java Node Managers security

infrastructure.
a. nodemanager.domains
b.
c.
d.
e.
Keystore
Role Mapping
Username/Password
nodemanager.keys
Answer: a, b, d
Quiz

Compare the architectures of the Java-based Node
Manager and the script-based Node Manager
Configure a Node Manager to use server start and stop
scripts
Describe some typical Node Manager problems
Initialize a Node Managers security files by using WLST
Tune the SSL settings of the Java Node Manager
Summary

Debugging Node Manager client communication
Enrolling Node Manager with a domain
Debugging remote server startup issues
Configuring Node Manager to use standard and custom
scripts
Practice 12-1
Investigating Node Manager Problems
Troubleshooting Clusters

Describe how a proxy plug-in creates connections and
performs failover
Explain the details of heartbeat and replication
communication within a cluster
Monitor and debug proxy plug-ins
Perform some basic tasks with Oracle HTTP Server
Monitor cluster members and session utilization
Identify several common proxy and cluster issues
Objectives
Proxy Diagnostics
OHS Log
Plug-in Parameters
Connection Architecture
Proxy Debug Page
Proxy Debug Log
Replication Diagnostics
Road Map
A WLS cluster supports additional features:

To provide high availability for applications and services
To perform load balancing and failover
That are transparent to both applications and clients
Hardware or
software
Cluster
Server 1
Web client
Proxy
Server 2
EJB client
JMS client
Stub
Server n
Clustering is configuring a group of Oracle WebLogic Servers to work together to provide

client access to the services offered by the servers in the cluster. The cluster appears to a
client as one instance, whether the client is a Web client or a Java application. By replicating
the services provided by one instance, an enterprise system achieves a fail-safe and scalable
environment. Scalability is achieved by balancing the load of incoming requests across the
servers in the cluster.
High availability is achieved through the replication of services, so that when one service fails,
another service can resume where the first service left off. A cluster uses the redundancy of
multiple servers to insulate clients from failures.
Oracle WebLogic Server provides clustering support for Web applications by replicating the
HTTP session state of clients. You can balance the Web application load across a cluster by
using an Oracle WebLogic Server proxy plug-in or the external load-balancing hardware.
Failover for Enterprise JavaBeans (EJBs) objects is accomplished using the objects replicaaware stub. When a client makes a call through a replica-aware stub to a service that fails,
the stub detects the failure and retries the call on another replica.
Cluster Review
Oracle provides plug-ins to popular Web servers that:

Dynamically load-balance requests across a cluster of
WebLogic Servers using round robin
Transparently failover requests if a server is unavailable
Support sticky sessions using cookies
Cluster
Client
Client
Web Server
Plug-In
Server
Server
Client
Server
Client
Plug-ins enable WebLogic Server to communicate with applications deployed on Oracle

HTTP Server, Apache HTTP Server, Sun Java System Web Server, or Microsofts Internet
Information Server. Typically, WebLogic Server handles the application requests that require
dynamic functionality, the requests that can best be served with dynamic HTML pages or
JSPs (Java Server Pages). The Web server then hosts static content and also in the case of a
cluster, performs load balancing and failover. As always, refer to the documentation to make
sure your that your specific Web server version and host OS are supported by Oracle.
The plug-in does a simple round-robin between all available servers in a cluster. The server
list specified in this property is a starting point for the dynamic server list that the server and
plug-in maintain. WebLogic Server and the plug-in work together to update the server list
automatically with new, failed, and recovered cluster members.
The plug-in also supports sticky session behavior, similar to hardware load balancers. This
means that the plug-in directs HTTP requests containing a session cookie (or URL-encoded
session ID or a session ID stored in the POST data) to the same server in the cluster that
originally created the cookie. In other words, the client is pinned to a specific server in the
cluster. This behavior ensures that users do not lose their session data between requests.
Proxy Plug-in Review
Plug-ins are:
Bundled with WLS and Oracle HTTP Server
Also available online to download separately
Released on a separate cycle than that for the server itself,
so always confirm that a newer version is not available
Although all plug-ins share the same major capabilities, the

installation and configuration steps vary slightly by vendor.
The WebLogic Sun Java System Web Server plug-in module is distributed as a shared object
(.so) on UNIX platforms and as a dynamic-link library (.dll) on Windows. These files are
located in the WL_HOME/server/plugin/OperatingSystem/Architecture directory
of your WebLogic Server distribution. WL_HOME represents the top level installation directory
for your WebLogic platform. The server directory contains installation files for WebLogic
Server. OperatingSystem refers to the operating system, such as UNIX or Windows. Add
the following lines to the beginning of the magnus.conf file. These lines instruct Sun Java
System Web Server to load the native library (the .so or .dll file) as a module:
Init fn="load-modules" funcs="wl_proxy,wl_init"\
shlib=/usr/local/netscape/plugins/(.so or .dll file) Init
fn="wl_init"
To install the Apache HTTP Server Plug-In as a dynamic shared object, locate the shared
object directory for your specific platform, such as linux/i686, solaris/sparc, or
win/32. Then copy the mod_wl_20.so file to the APACHE_HOME/modules directory and
add the following line to your APACHE_HOME/conf/httpd.conf file:
LoadModule weblogic_module modules/mod_wl_20.so
Obtaining and Using Plug-Ins
OHS is:
A component of the Oracle Web Tier suite
Based on Apache HTTP Server (httpd.conf)
Installed with a WLS plug-in module (mod_wl_ohs)
Managed and monitored using command-line tools
Configurable through the Fusion Middleware Control
application, if installed on WLS
The OHS plug-in must be configured manually using the

mod_wl_ohs.conf file.
Oracle HTTP Server 11g, Release 1 (11.1.1.2.0) is based on Apache 2.2.13 infrastructure,
and includes modules developed specifically by Oracle. The features of single sign-on,
clustered deployment, and high availability enhance the operation of the Oracle HTTP Server.
Oracle HTTP Server can also be a proxy server, both forward and reverse. A reverse proxy
enables content served by different servers to appear as if coming from one server.
Configuration for Oracle HTTP Server are specified through directives in configuration files in
the exact same manner as Apache HTTP Server configuration files.
The mod_wl_ohs module provides the same functionality as the Oracle WebLogic Server
Plug-in for Apache HTTP Server (mod_weblogic) except for some minor differences, as
follows:
Uses Oracles security layer (NZ) to provide SSL support for the module. A new
directive, WlSSlWallet, has been added to Oracle HTTP Server through the
mod_wl_ohs module that allows the use of Oracle Wallets.
Supports two-way SSL between Oracle HTTP Server and Oracle WebLogic Server.
Supports IPv6 for communication with WebLogic Server.
Oracle HTTP Server (OHS) Review
Similar to WLS domains:

A single OHS product installation can support multiple
processes or components.
Processes that are managed together are called an
instance.
Instances have a root directory under which process
configuration and log files are stored.
Client
Component1
Component2
Config, Logs
Config, Logs
Client
Instance
OHS Installation
Oracle HTTP Server directories are divided between the Oracle home and the Oracle
instance. The Oracle home directories are read-only, and contain the Oracle Fusion
Middleware binaries. The Oracle instance directories contain the modules, applications, and
logs for Oracle HTTP Server.
Each OHS component has a root configuration directory found at
<instance>/config/OHS/<component>, which includes the WLS plug-in configuration
file, mod_wl_ohs.conf. Similarly, each components log files are found at
<instance>/diagnostics/logs/OHS/<component>.
When Oracle HTTP Server starts up, it writes the process ID (PID) of the parent httpd process
to the httpd.pid file located, by default, in the following directory:
<instance>/diagnostics/logs/OHS/<component>. The process ID can be used by
the administrator when restarting and terminating the daemon. If a process stops abnormally,
it is necessary to stop the httpd child processes by using the kill command. The PidFile
directive in httpd.conf specifies the location of the PID file.
Oracle HTTP Server (OHS) Review
OPMN is:
A process that is used to start, manage, and monitor other
Oracle Web Tier and Identity Management processes
Similar conceptually to the WLS node manager
Accessed using the opmnctl command-line tool or Oracle
Web-based tools
Start
Stop
OPMNCTL
OPMN
Status
Restart
Virtual
Directory
OHS 1
OHS 2
Web Cache
Oracle Fusion Middleware provides a high availability infrastructure integration with Oracle
Process Manager and Notification Server (OPMN), for process management, failure
detection, and failover for Oracle HTTP Server processes.
Oracle Fusion Middleware components that are managed by OPMN should never be started
or stopped manually. Do not use command-line scripts or utilities from previous versions of
Oracle Fusion Middleware for starting and stopping system components. OPMN must be the
last service turned off whenever you restart or turn off your computer.
Oracle Process Manager (PM) is the centralized process management mechanism used to
manage system processes. The PM is responsible for starting, restarting, stopping, and
monitoring every process it manages. The PM handles all requests sent to OPMN associated
with controlling a process or obtaining status about a process. The PM is also responsible for
performing death-detection and automatic restart of the processes it manages. The system
processes that PM is configured to manage are specified in the opmn.xml file.
Oracle Notification Server (ONS) is the transport mechanism for failure, recovery, startup, and
other related notifications between components in Oracle Fusion Middleware. It operates
according to a publish-subscribe model: a system component receives a notification of a
certain type for each subscription to ONS. When such a notification is published, ONS sends
it to the appropriate subscribers.
Oracle Process Manager and Notification Server

(OPMN) Review
Start OPMN and all managed processes, if not already started:

opmnctl startall
Get the name, status, memory usage, and port number of processes:
opmnctl status -l
Restart a specific OHS process:
opmnctl restartproc ias-component=PayrollWeb1
Stop all OHS processes:
opmnctl stopproc process-type=OHS
The opmnctl program provides a centralized way to control and monitor system components
from the command line. The available commands include:
start: Start the OPMN server for a local Oracle instance without starting system
processes.
startall: Start OPMN as well as the system processes for a local Oracle instance.
startall is equivalent to start followed by startproc without arguments.
stopall: Shut down the OPMN server as well as the system processes for the local
Oracle instance. This request operates synchronously; it waits for the operation to
complete before returning.
startproc, restartproc, stopproc: Use these commands to start, restart, or stop
system processes. The OPMN server must be up and running.
The following attributes are supported. Any of these attributes may be omitted, and treated as
a wild card:
ias-component: The name of a specific managed process, as defined in opmn.xml.
process-type: The type of managed process to command, as defined in opmn.xml.
process-set: The name of a custom process group defined in opmn.xml.
OPMNCTL Examples
Error and access logs are found at

<instance>/diagnostics/logs/OHS/<component> by
default.
Error logs can use Apache or Oracle (ODL) formats.
Use httpd.conf to configure ODL format logging:

OraLogMode odl-text
OraLogSeverity NOTIFICATION
OraLogRotationParams S 10:70
Also ERROR,
WARNING, or TRACE
Rotate log when
size (S) is 10 MB.
Typical error log output when plug-in enabled:

[NOTIFICATION:16] [mod_weblogic.c] Oracle WebLogic plugin
build date/time: ...
[NOTIFICATION:32] [mod_weblogic.c] mod_weblogic: Testing
Debug=ON LogFile=...
There are two types of logs for Oracle HTTP Server. Error logs record server problems by
default, but can also be configured to record other server events. Access logs record which
components and applications are being accessed and by whom. You can view Oracle Fusion
Middleware log files using either Fusion Middleware Control or a text editor. The default name
of the error log file is <ohs_name>.log.
Oracle HTTP Server enables you to choose the format in which you want to generate log
messages. You can choose to generate log messages in the legacy Apache message format,
or use Oracle Diagnostic Logging (ODL) to generate log messages in text or XML-formatted
logs, which complies with Oracle standards for generating error log messages. By default,
Oracle HTTP Server error logs use ODL for generating diagnostic messages.
OHS supports two types of log rotation policies: size-based and time-based. You can
configure both error log and access log to use either one of these two rotation polices. When
rotation type is set to S (sized-based), set the policy to
<maxFileSize>:<maxAllFilesSize> (in MB).
The specified message severity is interpreted as the lowest desired severity, and all
messages of that severity level and higher are logged. Additional, you can also supply an
integer in the range of 132, where 1 is the most severe, and 32 is the least severe. Using
level 1 will result in fewer messages than using level 32.
OHS Logs
Plug-ins route to servers or clusters based on the incoming

URL path, requested file type, or virtual host definition.
The plug-in configuration file name and format vary based
on proxy type.
Use mod_wl_ohs.conf to proxy to a cluster:
Default parameters
for all locations
<IfModule weblogic_module>
WebLogicCluster nodea.xyz.com:7001,nodeb.xyz.com:7001,
nodec.xyz.com:7001
</IfModule>
Proxy based
<Location /hrWeb>
on URL path
SetHandler weblogic-handler
Debug OFF
Parameters for this
</Location>
specific location
For Apache HTTP Server, edit the httpd.conf file, or alternatively create a separate
configuration file and use the Include directive in httpd.conf. This technique is already
done for you if using Oracle HTTP Server. The separate configuration file is named
mod_wls_ohs.conf.
Oracle recommends that you use an Apache IfModule block to define parameters only
when the plug-in module is found. If you choose to not use the IfModule, you can instead
directly place the WebLogic properties inside Location or VirtualHost blocks. If you use an
Apache HTTP Server VirtualHost block, you must include all configuration parameters
(MatchExpression, for example) for the virtual host within the VirtualHost block.
If you are proxying requests to a cluster of WebLogic Servers, use the WebLogicCluster
parameter instead of the WebLogicHost and WebLogicPort parameters.
To proxy requests by path, use the Location block and the SetHandler statement.
SetHandler specifies the handler for the Apache HTTP Server Plug-In module. To proxy
requests by MIME type, add a MatchExpression line to the IfModule block. Note that if
both MIME type and proxying by path are enabled, proxying by path takes precedence over
proxying by MIME type. You can also use multiple MatchExpressions.
Plug-in Configuration Review
Parameter
Description
WebLogicHost,
WebLogicPort
Proxy to a single server
WebLogicCluster
Proxy to this initial list of clustered servers
MatchExpression
Proxy requests for files of this MIME type
PathTrim
Remove this text from the incoming URL path before

forwarding a request.
PathPrepend
Add this text to the incoming URL path before forwarding a

request.
ErrorPage
URL to direct users to if all servers are unavailable
WLExcludePathOrM
imeType
Do not proxy for this specific URL path or MIME type.
WLProxySSL
Set to ON to establish an SSL connection to WebLogic if the

incoming request also uses HTTPS.
TrustedCAFile
Location of PEM file that contains trusted CA certificates
WebLogicCluster: List of WebLogic Servers that can be used for load balancing. The
server or cluster list is a list of host:port entries. The plug-in does a simple round-robin
between all available servers. The server list specified in this property is a starting point
for the dynamic server list that the server and plug-in maintain. WebLogic Server and
the plug-in work together to update the server list automatically with new, failed, and
recovered cluster members.
WLDNSRefreshInterval: If defined in the proxy configuration, specifies the number of
seconds interval at which WebLogic Server refreshes DNS name to IP mapping for a
server. This can be used in the event that a WebLogic Server instance is migrated to a
different IP address, but the DNS name for that servers IP remains the same. In this
case, at the specified refresh interval the DNS<->IP mapping will be updated.
ErrorPage: Create your own local error page that is displayed when your Web server
is unable to forward requests to WebLogic Server.
MaxPostSize: Maximum allowable size of POST data, in bytes. If the content-length
exceeds this value, the plug-in returns an error message. If set to -1, the size of POST
data is not checked. This is useful for preventing denial-of-service attacks that attempt to
overload the server.
Basic Plug-in Parameters
By default, plug-ins:
Pool and reuse connections to servers using the keep
alive feature of the HTTP protocol
Fail over to the next server if the initial connection times
out
Continue to retry connections for a certain amount of time
Fail over to the next server after waiting too long for a
response to an individual request
Eventually retry failed servers after a certain amount of
time
When the Apache HTTP Server Plug-In attempts to connect to WebLogic Server, the plug-in
uses several configuration parameters to determine how long to wait for connections to the
WebLogic Server host and, after a connection is established, how long the plug-in waits for a
response. If the plug-in cannot connect or does not receive a response, the plug-in attempts
to connect and send the request to other WebLogic Server instances in the cluster. If the
connection fails or there is no response from any WebLogic Server in the cluster, an error
message is sent.
Proxy Connection Architecture
1
Client
request
Connection
refused or
timeout
Create
connection if
none available
in pool.
FAILOVER
Select next
server.
Check whether
time left to retry.
Send request
to WLS and
wait for
response.
4
Client
response
Request timeout
Client error
1. The plug-in receives a request from the Web server. If a WebLogic session ID is found
in the request, the plug-in connects to the primary server defined in the cookie.
Otherwise, the plug-in connects to the next server in the round-robin cycle.
2. If a previously created connection to the selected server is not available, the proxy
attempts to establish a connection. If the connection is unsuccessful because the server
refused it or it timed out, the plug-in sleeps for the time defined by the
ConnectionRetrySecs parameter and then tries the next server in the list. If at any
time ConnectTimeoutSecs is exceeded, an HTTP error is sent to the client or the
page identified by the ErrorPage parameter is shown.
3. After a connection is successful, the request is sent and the server waits for the
response for the time specified by the WLIOTimeoutSecs parameter. If the plug-in
does not receive a response from the server, the plug-in checks the value of the
Idempotent parameter. If the parameter is enabled (default), the server is marked as
bad. Then the process starts again from step 2 above, until the request succeeds. If
the Idempotent parameter is disabled or all servers have been exhausted, an HTTP
error is sent to the client or the page identified by the ErrorPage parameter is shown.
4. If the server responds successfully to the request, the response is forwarded back to the
original client that sent the request.
Proxy Connection Architecture
The WebLogicCluster parameter simply specifies an

initial list of servers for the proxy to try.
Cluster members use heartbeat messages to determine
the availability of current members and the presence of
new ones.
Responses from a cluster member to a proxy include an
updated list of servers to try.
Server A
Proxy
A,B,C
Server B
A,B
Server C
When you use the WebLogicCluster parameter in your configuration file to specify a list of
WebLogic Servers, the plug-in uses that list as a starting point for load balancing among the
members of the cluster. After the first request is routed to one of these servers, a dynamic
server list is returned containing an updated list of servers in the cluster. The updated list
adds any new servers in the cluster and deletes any that are no longer part of the cluster or
that have failed to respond to requests. This list is updated automatically with the HTTP
response when a change in the cluster occurs.
Dynamic Server List
Parameter
Description
KeepAliveEnabled
Enable/disable connection pooling and reuse.
KeepAliveSecs
Maximum time that a connection can be idle and remain

open, before a new connection must be created
ConnectTimeoutSecs
Maximum total wait time to establish a connection
WLSocketTimeoutSecs
Wait time for a connection attempt
ConnectRetrySecs
Pause time between retrying failed connections
WLIOTimeoutSecs
Maximum wait time to receive a response for a request

sent over a connection
Idempotent
Enable/disable failover after WLIOTimeoutSecs.
MaxSkipTime
Wait time before retrying a server marked as failed
DynamicServerList
Enable/disable dynamic server list feature.
KeepAliveSecs: The length of time after which an inactive connection between the
plug-in and WebLogic Server is closed. You must set KeepAliveEnabled to true (ON
when using the Apache plug-in) for this parameter to be effective. The default is 20
seconds.
WLSocketTimeoutSecs: Set the timeout for the socket while connecting, in seconds.
The value must be greater than 0 and the default is 2 seconds.
WLIOTimeoutSecs: Defines the amount of time the plug-in waits for a response to a
request from WebLogic Server. The plug-in waits for the server to respond and then
declares that server dead, and fails over to the next server. The minimum value is 10
seconds and the default is 300 seconds.
ConnectRetrySecs: Interval in seconds that the plug-in should sleep between
attempts to connect to the WebLogic Server host (or all of the servers in a cluster). The
number of times the plug-in tries to connect before returning an HTTP 503/Service
Unavailable response to the client is calculated by dividing ConnectTimeoutSecs by
ConnectRetrySecs. To specify no retries, set the parameters to equal values.
However, the plug-in always attempts to connect at least twice.
MaxSkipTime: The amount of time after which the plug-in will retry the server marked
as bad. The plug-in attempts to connect to a new server in the list each time a unique
request is received (that is, a request without a session cookie). The default is 10
seconds.
Connection Parameters
Connectivity problems can cause:

Unnecessary failover conditions
Unexpected HTTP errors sent to the client
Causes of unexpected connection failures include:

WebLogicCluster has typos.
ConnectTimeoutSecs is set too low.
ConnectRetrySecs is set higher than ConnectTimeoutSecs.
Causes of unexpected request failures include:

WLIOTimeoutSecs is set too low.
MaxPostSize is set too low on either the proxy or WLS.
If the proxy is slow to use a restarted cluster member,

MaxSkipTime may be set too high.
The value of the KeepAliveSecs parameter must be less than or equal to the value of the
Duration field set using a WebLogic Servers HTTP tab in the administration console (the
KeepAliveSecs attribute of the MBean). Similarly, if the MaxPostSize parameter is greater
than or equal to the same WLS setting, it will have no effect.
The WLIOTimeoutSecs parameter should typically be set to a large value (default is 5
minutes). If the value is less than the time your application takes to process a request, then
you may see unexpected results.
Common Connectivity Issues
Refer to your vendors documentation on configuring and

troubleshooting SSL between the client and the proxy.
When SSL is enabled on the plug-in, it behaves exactly as
if it were a standard WLS client:
Validate WLS certificate using trusted CAs.
Verify WLS certificate host name.
Check for required constraint in WLS certificate.
Parameter
Description
SecureProxy
If enabled, always use SSL between proxy and WLS.
RequireSSLHostMatch
Enable/disable host name verification.
SSLHostMatchOID
Attribute in certificates distinguished name that

contains the host name (CN by default)
EnforceBasicConstraint
The level of validation for certificate constraints
You can use the SSL protocol to protect the connection between the Apache HTTP Server
Plug-In and WebLogic Server. The SSL protocol provides confidentiality and integrity to the
data passed between the Apache HTTP Server Plug-In and WebLogic Server.
RequireSSLHostMatch: Determines whether the host name to which the plug-in is
connecting must match the Subject Distinguished Name field in the digital certificate of the
WebLogic Server to which the proxy plug-in is connecting. Depending on your network
configuration, the host name in the certificate should be the same as the servers Listen
Address or External DNS Name attributes.
SSLHostMatchOID: The field in the Subject Distinguished Name of the peer digital certificate
is to be used to perform the host match comparison. Supported values are 23 (Surname), 22
(Common Name), 13 (Email), 30 (Organizational Unit), 29 (Organization) and 26 (Locality).
EnforceBasicConstraint: This parameter closes a security hole that existed with SSL
certificate validation where certificate chains with invalid V3 CA certificates would not be
properly rejected. This allowed certificate chains with invalid intermediate CA certificates,
rooted with a valid CA certificate to be trusted. X509 V3 CA certificates are required to contain
the BasicConstraints extension, marked as being a CA, and marked as a critical
extension. This checking protects against non-CA certificates masquerading as intermediate
CA certificates. Supported values are STRONG (default), STRICT, and OFF.
Proxy SSL Issues
When enabled, plug-ins:

Check for a special query string parameter
Display the current status of each server in the initial and
dynamic lists
Display various configuration settings and communication
statistics
...
DebugConfigInfo ON
</IfModule>
http://proxyhost:port/proxypath/?__WebLogicBridgeConfig
DebugConfigInfo: Enables the special query parameter __WebLogicBridgeConfig.

Use it to get details about configuration parameters from the plug-in. The plug-in gathers the
configuration information and runtime statistics and returns the information to the browser.
The plug-in does not connect to WebLogic Server in this case. This parameter is strictly for
debugging and the format of the output message can change with releases. For security
purposes, keep this parameter turned OFF in production systems.
Proxy Debug Page
General Server List:

1. Host: 'nodea' Port: 7001 SecurePort: 7001 Status: OK
2. Host: 'nodeb' Port: 7001 SecurePort: 7001 Status: OK
3. Host: 'nodec' Port: 7001 SecurePort: 7001 Status: OK
ConnectRetrySecs: '2'
ConnectTimeoutSecs: '10'
...
Configuration
parameters
Status of cluster
members
Runtime statistics:
requests: 524
Runtime
successful requests: 518
statistics
Exception objects created: 6
...
CONNECTION_REFUSED exceptions: 0
CONNECTION_TIMEOUT exceptions: 0
READ_ERROR_FROM_SERVER exceptions: 6
Additional configuration parameters are displayed. For example:

WLCookieName: JSESSIONID
Debug: 'ALL'
DebugConfigInfo: 'ON'
DefaultFileName: ''
DynamicServerList: 'ON'
ErrorPage: ''
FileCaching: ON
Idempotent: ON
KeepAliveEnabled: ON
KeepAliveSecs: 20
MaxPostSize: '0'
MaxSkipTime: '10'
PathPrepend: '
Proxy Debug Page
Plug-ins can generate a log file for troubleshooting purposes.

Configure the data that is logged for each request.
...
Debug ERR
WLLogFile /tmp/wlproxy.log
</IfModule>
Debug Value
What Is Logged
ON
Only error and informational messages
ERR
Only error messages
ALL
All messages and HTTP headers
HFC
ON + HTTP headers sent from client
HFW
ON + HTTP headers sent back from WLS

The debugging information is written to the /tmp/wlproxy.log file on UNIX systems and
c:\TEMP\wlproxy.log on Windows systems. Override this location and file name by
setting the WLLogFile parameter to a different directory and file. The WLTempDir parameter
provides an additional way to change this location. Ensure that the directory has write
permission assigned to the user who owns the proxy process.
Use the Debug parameter to set any of the listed options. The default is OFF. HFC, HTW, HFW,
and HTC options may be set in combination by entering them separated by commas, for
example HFC,HTW. Additional options include:
HTW: The plug-in logs headers sent to WebLogic Server, and informational and error
messages.
HTC: The plug-in logs headers sent to the client, informational messages, and error
messages.
Proxy Debug Log
<2199012609093463> Using Uri /hrWeb/updateProfile.action

...
<2199012609093463> Found cookie from cookie header ...
...
<2199012609093463> attempt #0 out of a max of 5
...
<2199012609093463> getPooledConn: No more connections in the
pool for Host[nodea.xyz.com] Port[7001] SecurePort[7001]
...
<2199012609093463> general list: created a new connection ...
...
<2199012609093463> Hdrs from clnt:[Cookie2]=[USER="tom123"]
...
<2199012609093463> Method is POST
<2199012609093463> URL::sendHeaders(): meth='POST'
file='/hrWeb/updateProfile.action' protocol='HTTP/1.1'
...
<2199012609093463> Reader::fill(): first=0 last=0 toRead=4096
Each incoming client request is assigned a unique ID, which can be used to correlate log
messages. The above example traces the following proxy activities:
1. Accept an incoming request from a client.
2. Check for session ID cookie in the request and if present map it to its primary server.
3. Start timer to process the request. If the configured maximum time is exceeded, an
HTTP error is sent to the client. The number of attempts is calculated by dividing the
configured try time from the maximum time.
4. Check whether a connection for the selected server is available in the connection pool.
5. Create a new connection to the server if needed.
6. Print incoming request headers, including cookies.
7. Forward the request to the server.
8. Wait for the server response and forward it back to the client.
Typical Proxy Trace

Perform basic administrative tasks on OHS
List several plug-in parameters
Describe the plug-in connection and failover processes
Configure and interpret the plug-in debug page
Analyze the contents of a plug-in log file
Section Summary

Monitoring cluster utilization under load
Monitoring runtime statistics for the proxy plug-in
Analyzing proxy plug-in logs
Tuning proxy connection settings
Practice 13-1
Investigating Proxy Problems
Proxy Diagnostics
Replication Diagnostics
Unicast Communication
Runtime Monitoring
Debug Flags
Serialization
Road Map
Clusters use a unicast communication model to distribute

messages intended for all members:
Heartbeats so that all members know each others
availability in real time
JNDI tree synchronization, such as during JMS migration
Clusters use standard IP communication for simple peerto-peer interactions:

Session replication and failover for Web or EJB applications
Distributing JMS messages to member destinations
Oracle does not recommend the use of IP multicast

communication and supports it only for backwards
compatibility.
WebLogic Server provides an alternative to using multicast to handle cluster messaging and
communications. WebLogics unicast (leader distribution) model tends to be simpler to
configure and manage because it does not require the cross network configuration that
multicast requires. Additionally, it reduces potential network errors that can occur from
multicast address conflicts.
Each WebLogic Server instance in a cluster uses this unicast model to distribute regular
heartbeat messages that advertise its availability. By monitoring heartbeat messages, server
instances in a cluster determine when a server instance has failed. Clustered server instances
also monitor IP sockets as a more immediate method of determining when a server instance
has failed.
Each instance in a cluster uses unicast to announce the availability of clustered objects that
are deployed or removed locally. Each server instance monitors these announcements and
updates its local JNDI tree to reflect current deployments of clustered objects.
When a JMS message is produced to a distributed destination that has been targeted to a
cluster, the initial server (member destination) that receives the message may choose to
forward it to another server. For example, distributed queue members with active consumers
are preferred over those with zero consumers.
Cluster Communication Review
WLS unicast is a hub and spoke communication model.

As servers join or leave a cluster, members communicate
with each other to dynamically:
Nominate a subset of the members as group leaders
Assign members to groups
Leaders are responsible for relaying broadcast messages

to their group members.
Server
Server
Server
Server
Server
Server
Server
Server
Cluster members communicate to the group leader when they need to send a broadcast
message, which is usually the heartbeat message. When the cluster members detect the
failure of a group leader, the next oldest member becomes the group leader. The frequency of
communication in unicast mode is similar to the frequency of sending messages on multicast
port.
Unicast Architecture
Web and EJB applications use sessions to track

information in server memory for each client (a shopping
cart, for example).
Sessions that are not accessed for a period of time are
invalidated.
WLS periodically scans all sessions and deletes any
invalidated ones.
weblogic.xml:
...
<session-descriptor>
<timeout-secs>3600</timeout-secs>
<invalidation-interval-secs>60
</invalidation-interval-secs>
</session-descriptor>
Session tracking enables you to track a users progress over multiple servlets or HTML
pages, which, by nature, are stateless. A session is defined as a series of related browser
requests that come from the same client during a certain time period. Session tracking ties
together a series of browser requests that may have some meaning as a whole, such as a
shopping cart application.
The cookies that WebLogic Server uses to track sessions are set as transient by default and
do not outlive the session. When a user quits the browser, the cookies are lost and the
session ends. This behavior is in the spirit of session usage and it is recommended that you
use sessions in this way.
You can specify a period of time after which HTTP sessions expire if inactive. When a session
expires, all data stored in the session is discarded. You can set the timeout period in either
web.xml or weblogic.xml. You also use weblogic.xml to tune the time, in seconds, that
WebLogic Server waits between doing house-cleaning checks for timed-out and invalid
sessions, and deleting the old sessions and freeing up memory.
Session Management Review
By default, when a client fails over to another server in the

cluster, its session information is lost.
WebLogic Server supports several session persistence
strategies to recover sessions when clients are redirected
from failed servers, including:
In-memory replication
JDBC persistence
File persistence
Cookie persistence
Session persistence
Can be done synchronously or asynchronously
Is configured in the weblogic.xml descriptor
Web application components, such as servlets and JavaServer Pages (JSPs), maintain data
on behalf of clients using an HttpSession instance that is available on a per-client basis. To
provide high availability of Web applications, shared access to one HttpSession object
must be provided. HttpSession objects can be replicated within WebLogic Server by
storing their data using in-memory replication or file system persistence, or by storing it in a
database.
For clusters that use a supported hardware load-balancing solution, the load-balancing
hardware redirects client requests to any available server in the WebLogic Server cluster. The
cluster itself obtains the replica of the clients HTTP session state from a secondary server in
the cluster.
In clusters that use Web servers with WebLogic proxy plug-ins, the proxy plug-in handles
failover transparently to the client. If a server fails, the plug-in locates the replicated HTTP
session state on a secondary server and redirects the clients request accordingly.
With synchronous replication, the secondary is updated at the end of the current request and
before the response is sent back to the client. This helps to guarantee reliability and data
integrity. Alternatively, in order to achieve greater performance, replication can instead be
performed asynchronously. With this approach, session updates are placed on an internal
queue, and when the queue is full, all changes are flushed to the secondary.
Session Persistence Review
Cookie
Client
Session
stickiness
Primary = A
Client
Proxy may fail
over to any server
Proxy
Proxy
Cluster
ServerA
ServerB
2
Primary
Secondary
Primary = C
ServerC
Cluster
ServerA
ServerB
ServerC
5
Secondary
Primary
The graphic in the slide depicts a client accessing a Web application hosted in a cluster. All
client requests are forwarded to the WebLogic Server cluster via a Web server plug-in:
1. The proxy load balances a clients initial request to Server A. The Web application
creates a new session for this user.
2. To provide failover services for the Web application, the primary server replicates the
clients session state to a secondary WebLogic Server in the cluster. This ensures that a
replica of the session state exists even if the primary server fails.
3. The HTTP response back from Server A includes a cookie indicating that it is the
primary server and Server B is the secondary.
4. Suppose at some point in the future the primary server for the user fails.
5. Server B detects the failure and becomes the primary server hosting this users session
state. The server then selects another server and a new secondary is created.
6. On the next client request, the proxy transparently fails over to some other server in the
cluster.
7. Server C promotes itself as the primary for this users session and it updates the clients
cookie to reflect the new primary and secondary servers.
In-Memory Replication Review
Monitor a cluster from a specific member:

serverRuntime()
cluster = getMBean('/ClusterRuntime/MyCluster')
print 'Alive Count: ', cluster.getAliveServerCount()
print 'Alive Names: ', cluster.getServerNames()
print 'Unicast Sent: ', cluster.getFragmentsSent()
print 'Unicast Received: ', cluster.getFragmentsReceived()
print 'Primary Sessions: ', cluster.getPrimaryCount()
print 'Secondary Sessions: ', cluster.getSecondaryCount()
Monitor cluster unicast communication from a specific member:
serverRuntime()
unicast = getMBean('/ClusterRuntime/MyCluster/
UnicastMessaging/UnicastMessagingRuntime')
print 'Total Groups: ', unicast.getTotalGroupCount()
print 'Group Leaders: ', unicast.getDiscoveredGroupLeaders()
print 'My Leader: ', cluster.getLocalGroupLeaderName()
Refer to the documentation for the ClusterRuntimeMBean (which implements the interface
ReplicationRuntimeMBean) and UnicastMessagingRuntimeMBean for a list of all
available attributes and operations.
Cluster Monitoring WLST Examples
Monitor sessions for a Web module within an enterprise application:

serverRuntime()
web = getMBean('/ApplicationRuntimes/HRApp/ComponentRuntimes/
ServerC_/payrollWeb')
print 'Current: ', web.getOpenSessionsCurrentCount()
print 'High: ', web.getOpenSessionsHighCount()
Destroy all sessions for a Web module:

web = ...
sessionIDs = web.getServletSessionsMonitoringIds()
for sid in sessionIDs:
web.invalidateServletSession(sid)
Refer to the documentation for the WebAppComponentRuntimeMBean for a list of all

available attributes and operations.
Session Monitoring WLST Examples
For security purposes, WLS does not expose the real

session IDs sent back to clients via JMX.
To aid monitoring and troubleshooting, WLS can instead
track a custom session attribute such as a username.
weblogic.xml:
...
<session-descriptor>
<monitoring-attribute-name>user_profile
</monitoring-attribute-name>
</session-descriptor>
Use WLST to view sessions with custom monitoring IDs:

...
sessionIDs = web.getServletSessionsMonitoringIds()
print 'Session User 1' + sessionIDs[0]
HTTP Sessions are identified with a monitoring ID. By default, the monitoring ID for a given
HTTP session is a random string (not the same as a session ID for security reasons). This
monitoring ID can be configured by setting the monitoring-attribute-name element in the
weblogic.xml deployment descriptor. The toString() of the session attribute value will
then be used as a monitoring ID.
The monitoring-attribute-name element is useful for tagging session runtime
information for different sessions. For example, you can set it to username, if you have a
username attribute that is unique.
The WebAppComponentRuntimeBean.getServletSessionMonitoringIds() method
returns an array of session attribute values with this name. If it is not set, it returns an array of
randomly generated unique strings.
Session Monitoring Attribute
WLDF includes application-scoped monitors that support

standard diagnostic actions:
Servlet_Before_Session
Servlet_After_Session
Servlet_Around_Session
The additional monitor HttpSessionDebug has a built-in

action that records a sessions current size any time it is
updated.
Servlet_Before_Session/Servlet_After_Session: Runs the specified

diagnostic actions before or after calls to HttpSession methods getAttribute(),
setAttribute(), removeAttribute(), and invalidate().
HttpSessionDebug: Inspects the target HTTP session before and after calls to
HttpSession methods getAttribute(), setAttribute(), and
removeAttribute(). At inspection points, the approximate session size is computed
and stored as the payload of a generated event. The size is computed by flattening the
session to a byte array. If an error is encountered while flattening the session, a
negative size is reported.
Session Instrumentation
Servers generate info messages to trace the addition and

removal of members to and from the cluster.
Use debug flags for more detailed log messages.
Flag
Description
DebugCluster
View very low level details of cluster communications.
DebugClusterHeartbeats
View how often heartbeats are sent from this server.
DebugCluster
Announcements
Trace all heartbeats sent from and received by this

server.
DebugLeaderElection
Trace unicast group initialization and updates.
DebugHttpSessions
Trace the creation, updating, invalidation, persistence,

replication, and size of all HTTP sessions.
DebugReplication
More detailed trace of session replication activities
DebugReplicationDetails
View some additional details on session replication

decisions.
The following is an example of a cluster informational log message:

<Info> <Cluster> <BEA-000111> <Adding MyServer2 with ID
1234:199.177.1.2:[7001,7001,-1,-1,-1,-1,-1]:MyDomain:MyServer2 to
cluster: MyCluster view.>
The flags DebugCluster, DebugClusterHeartbeats, DebugClusterAnnouncements,
DebugLeaderElection, DebugReplication, and DebugReplicationDetails are
found in the weblogic.core.cluster scope. The weblogic.cluster scope contains
flags for the cluster-leasing feature, which is used to support WebLogic Servers service
migration and whole server migration capabilities. The DebugHttpSessions flag is found in
the weblogic.servlet.internal.session scope.
Some of these cluster debug flags, particularly those involved with heartbeat communications,
generate a huge amount of log information in a short period of time. Consequently, you may
also need to increase the log file size as well as the maximum number of log files.
Cluster Debug Flags
Normal log messages with DebugClusterHeartbeats and

DebugClusterAnnouncements enabled:
<Sending local attributes MyServer2
jvmid:1114449976969291583S::MyDomain:MyServer2>
...
<Sending Announcement numItems:26>
...
<Sent Heartbeat with 3 items>
...
<Received Announcement numItems:26 from ...
199.177.1.1:[7001,7001,-1,-1,-1,-1,-1]:MyDomain:MyServer1>
...
<Received Announcement numItems:26 from ...
199.177.1.3:[7001,7001,-1,-1,-1,-1,-1]:MyDomain:MyServer3>
...
<Sent Heartbeat with 3 items>
...
When a cluster member becomes unavailable and other members stop receiving heartbeat
messages, an informational log message is generated:
<Info> <Cluster> <BEA-000129> <Removing
4567:199.177.1.5:[7001,7001,-1,-1,-1,-1,-1]:MyDomain:MyServer5 from
the cluster.>
The debug trace above depicts the following activities:
1. As part of starting up, the current server prepares to broadcast its local attributes,
including its internal JVM ID, to other cluster members. In this example, the current
server is MyServer2.
2. The initial announcement with this servers attributes is sent.
3. The current server begins broadcasting heartbeat messages to other cluster members.
4. The current server begins receiving announcements and heartbeat messages from
other cluster members. In this example, other running servers in the cluster include
MyServer1 and MyServer3.
5. Heartbeats continue to be sent and received.
Typical Cluster Heartbeat Trace
Each server in a cluster is assigned a unique internal

JVMID.
Each replication task is assigned a unique internal ID.
DebugHttpSessions and DebugReplication enabled on primary server:

<Creating new session with ID: XYZ for Web application:
/hrWeb.>
<1234: Creating primary 1234 for application key /hrWeb>
<The current server is becoming primary for replicated session
ID: XYZ.>
<1234: Created secondary on ... 199.177.1.1:[7001,7001,-1,-1,1,-1,-1]:MyDomain:MyServer1>
...
<Going ahead with replication for 5678>
<5678: Secondary server ... 199.177.1.1:[7001,7001,-1,-1,-1,1,-1]:MyDomain:MyServer1>

1. The application requests a new session during the current request. The server creates a
new session and assigns it a unique session ID, which includes the JVM ID of the
current server.
2. Because in-memory replication is enabled for this application, the current server assigns
itself as the primary for the new session. An initial replication ID is also generated.
3. Another member of the cluster is selected as a secondary server for this session. The
current server sends the initial session data to the secondary. This request completes
and the response is sent back to the client along with the session ID.
4. A subsequent request updates the session. A new replication ID is generated.
5. The secondary server assigned to this session is sent the latest session changes.
Typical Replication Trace: Primary
DebugHttpSessions and DebugReplication enabled on secondary server:

<1234 Creating secondary 1234 for application key /hrWeb>
<The current server is becoming the secondary for replicated
session ID: XYZ.>
<Updated local secondary with version 1 from
...:199.177.1.2:[7001,7001,-1,-1,-1,-1,-1]:MyDomain:MyServer2
for replication object 1234, ...>
...
<Received Change ...>
<Updated local secondary with version 2 from
...:199.177.1.2:[7001,7001,-1,-1,-1,-1,-1]:MyDomain:MyServer2
for replication object 5678, ...>

1. The current server receives a request from another cluster member to initialize a
secondary copy of a session. The request includes the replication ID that was generated
on the primary server.
2. The server confirms that it is now the secondary for this session.
3. The server receives the initial session data for the secondary copy.
4. Some time later the server receives a message from the primary server with session
modifications. Once again, the message includes a replication ID for tracking purposes.
5. The server updates the secondary copy of the session.
Typical Replication Trace: Secondary
Replication problems often result in the loss of session

data (requiring the user to log in again, for example).
Typical culprits include:
Cluster configuration/deployment is not uniform.

Cluster members are not using the same release of WLS.
Network or firewall issues that result in missed heartbeats
Invalid session persistence settings
Session and/or cookie timeout settings are too low.
Proxy and WLS session cookie names do not match.
Application is not using HttpSession API appropriately.
Application is storing non-serializable objects in the session.
All servers in the cluster must have the same major version number, but can have different
minor version numbers and service packs. The clusters administration server is typically not
configured as a cluster member, but it should run the same major version of WebLogic Server
used on the managed servers. Also make sure the value of CLASSPATH is the same on all
managed servers in the cluster. CLASSPATH is set by the setDomainEnv (and therefore
commEnv as well) script, which is used by other server startup scripts.
Network issues can keep cluster group leaders from properly distributing heartbeat messages
throughout the cluster. This scenario can lead to unnecessary work concerning WebLogics
high availability services. For example, a proxy plug-ins dynamic server list may be missing
candidate servers. Or servers may replicate HTTP data excessively. To help uncover the
culprit in your network topology, use the console or WLST to monitor the general and unicast
runtime data of every cluster member.
Common Replication Issues
Web applications or their underlying application

frameworks (JSF, for example) must use the correct APIs
to support session persistence.
HttpSession Method
WebLogic Behavior
setAttribute(name,value)
Persist or replicate new or updated attribute.
removeAttribute(name)
Delete attribute from secondary or persisted

copy.
invalidate()
Mark session for deletion along with its

secondary or persisted copy.
An HttpSession object is created if one does not already exist for that client when the
request.getSession(true) method is called with the argument true. The session object
lives on WebLogic Server for the lifetime of the session, during which the session object
accumulates information related to that client. Your servlet adds or removes information from
the session object as necessary. A session is associated with a particular client. Each time
the client visits your servlet, the same associated HttpSession object is retrieved when the
getSession() method is called.
You can store data in an HttpSession object using name=value pairs. Data stored in a
session is available through the session. To add or overwrite a named attribute, use the
setAttribute() method. To remove a named attribute altogether, use the
removeAttribute() method.
If your application deals with sensitive information, consider offering the ability to log out of the
session. This is a common feature when using shopping carts and Internet email accounts.
When the same browser returns to the service, the user must log back in to the system. Use
the invalidate() method to mark the current session for deletion.
Do not use session persistence for storing long-term data between sessions. In other words,
do not rely on a session still being active when a client returns to a site at some later date.
Instead, your application should record long-term or important information in a database.
HttpSession API Overview
WebLogics session persistence capabilities require Java

objects to be sent over I/O channels (network, file,
database, and so on).
JVMs allow a Java object to be sent over I/O only if its type
is declared as Serializable; otherwise, the JVM throws
an error.
Most standard Java types can be serialized; exceptions
are the I/O objects themselves (connections, files,
sockets).
JVM
JVM
Socket
File
To serialize an object means to convert its state to a byte stream so that the byte stream can
be reverted back into a copy of the object. A Java object is serializable if its class or any of its
superclasses implements either the java.io.Serializable interface or its subinterface,
java.io.Externalizable. Deserialization is the process of converting the serialized form
of an object back into a copy of the object.
You can add any Java descendant of Object as a session attribute and associate it with a
name. However, if you are using session persistence, your attribute value objects must
implement java.io.Serializable.
Serialization Overview
WebLogic logs error and debug messages when it detects

that a session attribute cannot be persisted or replicated.
The debug messages include the attribute name along
with the original Java exception and stack trace.
Log messages on the primary server during replication:

<Debug> <HttpSessions> <HTTPSession attribute: "cmDoc" is not
serializable.
java.io.NotSerializableException: com.xyz.content.Document
at java.io.ObjectOutputStream.writeObject(...)
at ...
<Debug> <HttpSessions> <Session attribute with name:cmDoc
class:com.xyz.content.Document is not serializable and will
not be replicated or persisted>
If you add your own serializable classes to a persistent session, make sure that each instance
variable of your class is also serializable. Otherwise, you can declare a non-serializable
variable in the class as transient, and JVM with not attempt to serialize it.
Serialization Debug Messages

Explain the different types of cluster communication
Monitor cluster member availability and communication
Monitor HTTP sessions and in-memory replication
Analyze server logs to troubleshoot session management
and replication issues
Describe some common replication problems
Section Summary
Name three concepts associated with Oracle HTTP Server.

a. Domain
b. Instance
c. Process Manager
d. Component
e. Unicast
Answer: b, c, d
Quiz
Which of the following is not an available plug-in parameter?

a. WLLogFile
b. DebugConfigInfo
c. ReplicationType
d. DynamicServerList
e. WebLogicCluster
Answer: c
Quiz
Name three cluster debug flags.

a. DebugLeaderElection
b. DebugClusterAnnouncements
c. DebugClusterSSL
d. DebugReplication
e. DebugOHSList
Answer: a, b, d
Quiz

Describe how a proxy plug-in creates connections and
performs failover
Explain the details of heartbeat and replication
communication within a cluster
Monitor and debug proxy plug-ins
Perform some basic tasks with Oracle HTTP Server
Monitor cluster members and session utilization
Identify several common proxy and cluster issues
Lesson Summary

Configuring HTTP session debugging
Troubleshooting a session stickiness problem
Tracing session replication across a cluster
Monitoring HTTP sessions by using the console and WLST
Troubleshooting a session serialization error
Practice 13-2
Investigating Cluster Replication Problems
Appendix
WebLogic SNMP
SNMP:
Is a simple protocol for managing and monitoring distributed,
heterogeneous devices or agents
Allows enterprises to have a centralized management
infrastructure
Is well supported throughout the industry
Can be run over UDP (default) or TCP
WLS provides an SNMP agent:

For monitoring a servers configuration and runtime state
That supports SNMP v1, v2, and v3
SNMP is used in network management systems to monitor network-attached devices for

conditions that warrant administrative attention. SNMP exposes management data in the form
of variables on the managed systems, which describe the system configuration. These
variables can then be queried (and sometimes set) by managing applications.
In typical SNMP use, there are a number of systems to be managed, and one or more
systems managing them. A software component called an agent (see below) runs on each
managed system and reports information via SNMP to the managing systems.
With SNMP, a manager sends a request for information about managed resources to an
agent. The agent gathers the requested data and returns a response. You can also configure
agents to issue unsolicited reports (notifications) to managers when they detect predefined
thresholds or conditions on a managed resource.
In practice, SNMP implementations often support multiple versions. SNMP version 1
(SNMPv1) is the initial implementation of the SNMP protocol. SNMPv1 operates over
protocols such as User Datagram Protocol (UDP). SNMPv2 revises version 1 and includes
improvements in the areas of performance, security, confidentiality, and manager-to-manager
communications. SNMPv3 provides three important enhancements: authentication, privacy,
and access control.
Oracle WebLogic Server 11g: Diagnostics and Troubleshooting A - 2
Simple Network Management Protocol (SNMP)
Port 161, by default

SNMP Console
MIB
Request
Response
Device
SNMP
Agent
MIB
Notifications
SNMP Console
Device
MIB
Trap
SNMP
Agent
MIB
Port 162, by default
A managed device is a network node that contains an SNMP agent and that resides on a
managed network. Managed devices collect and store management information and make
this information available to management systems or consoles using SNMP. Managed
devices, sometimes called network elements, can be any type of device including, but not
limited to, routers, access servers, switches, bridges, hubs, IP telephones, computer hosts,
and printers.
An agent is a network-management software module that resides in a managed device. An
agent has local knowledge of management information and translates that information into a
form compatible with SNMP.
Typically, SNMP uses UDP ports 161 for the agent and 162 for the manager. The manager
may send synchronous polling requests to port 161 to retrieve values for specific variables.
However, agents may also publish asynchronous notifications or traps to port 162 on a
periodic basis or when variables meet certain conditions.
Although SNMP supports requests that update variables with new values, WebLogic Servers
agent gives SNMP managers only read access to its management system.
SNMP Architecture
Similar to JMX MBeans, an SNMP devices manageable

objects are organized using a hierarchy.
Each object is identified using a unique OID, which is:
Represented as a series of dot-separated integers
Supplied as part of an SNMP poll or trap message
Multiple instances of the same object type are

represented as tables.
1
.1.3.6.1
WebLogic Server SNMP agents query the WebLogic Server management system and
communicate the results to managers over the SNMP protocol. The WebLogic Server
management system exposes management data through a collection of managed beans
(MBeans). When a WebLogic Server SNMP agent receives a request from a manager, it
determines which MBean corresponds to the SNMP object identifier (OID) in the managers
request. Then it retrieves the data and wraps it in an SNMP response. Similarly, trap
messages generated in response to some criteria include an OID that maps to the
corresponding MBean being evaluated.
Each OID consists of a left-to-right sequence of integers. This sequence defines the location
of the object in the hierarchy and specifies a unique path through the tree to the object. Each
node in the path has both a number and a name associated with it. The path .1.3.6.1.4.1
defines the private.enterprises OID and each number beneath that node on the tree
represents the branches in the tree reserved for a particular vendor, for example, Oracle.
Object Identifier (OID)
MIB modules:
Are files that utilize the ASN.1 format
Are registered with an SNMP console
Identify a set of available SNMP objects for a device
Map numeric OIDs to descriptive names
MIB
1.3.6.4.1.10
NetworkSettings
1.3.6.4.1.10.1
PortNumber
1.3.6.4.1.14.7
PacketCount
In telecommunications and computer networking, Abstract Syntax Notation One (ASN.1) is a

standard and flexible notation that describes data structures for representing, encoding,
transmitting, and decoding data. It provides a set of formal rules for describing the structure of
objects that are independent of machine-specific encoding techniques and is a precise, formal
notation that removes ambiguities. An adapted subset of ASN.1, Structure of Management
Information (SMI), is specified in SNMP to define sets of related objects; these sets are
termed Management Information Base (MIB) modules.
Two types of managed objects exist. Scalar objects define a single object instance. Tabular
objects define multiple related object instances that are grouped in MIB tables. Tables are
composed of zero or more rows, which are indexed in a way that allows SNMP to retrieve or
alter an entire row with a single command.
Management Information Base (MIB)
The WLS SNMP agent supports most standard MBeans

and their attributes, including:
Domains, servers, JVM, and work managers
Applications
JDBC and JMS services
WLS includes two versions of its MIB module:

An ASN.1 file found at <WL_HOME>/server/lib
An XML file found in weblogic.jar
All WLS OIDs begin with .1.3.6.1.4.1.140.625.
The documentation includes an interactive application for

browsing and searching the MIB hierarchy.
WebLogic Server exposes a large number of data points in its management system. To
organize this data, it provides a hierarchical data model that reflects the collection of services
and resources that are available in a domain. The WebLogic Server MIB module reflects this
hierarchy. For example, a WebLogic Server domain describes its overall configuration in a
tabular managed object called domainTable. This tabular object refers to (contains) a
collection of scalar objects, each of which describes some attribute of the domain. For
example, domainTable contains a domainServers scalar object that names all servers in
the domain. The serverTable object contains a serverDeployments scalar object, which
describes all applications currently deployed on a server.
Tabular objects never directly contain object instances (MIB variables). Instead, tabular
objects contain scalar objects, and the scalar objects contain variables. For example, if you
created two managed servers in a domain named MS1 and MS2, the MIB contains one
serverTable object, which in turn contains a serverName object. The serverName object
contains two variables that contain the value MS1 and MS2.
WLS MIB and OIDs
Type
Description
Versions
Supported
GET
Gets one or more objects
All
GETNEXT
Gets the next sibling object after the given one
All
GETBULK
Gets all objects found in a given ID range
V2,V3
WALK
Gets an object and all of its children
V2,V3
TRAP
Notification object sent from device
All
INFORM
TRAP message with acknowledgement
V2,V3
SNMPv2 introduced GETBULK in order to retrieve large amounts of management data in a

single request. In SNMPv1, this could only be accomplished using a series of GETNEXT
requests.
An SNMP agent that uses the SNMPv2 or SNMPv3 protocol can send one of two types of
notifications when a monitored attribute crosses a defined threshold. The agent sends a
TRAP notification once and assumes that the SNMP manager received the message. The
agent sends an INFORM notification and waits for a response from the SNMP manager that
indicates the manager has received the message. If the manager does not respond, the agent
resends the notification. By default, a WebLogic Server SNMP agent sends TRAP
notifications.
Common SNMP Message Types
Internally, WLS agents use JMX to monitor servers.

WLS supports two SNMP models:
Centralized
Decentralized
In a centralized architecture, all SNMP communications:

Use a single, domain-wide agent
Are directed through the administration server
A new domain is automatically configured with a domainwide SNMP V1 agent, but it is disabled by default.
In each WebLogic Server domain, you can create multiple SNMP agents and organize them
into a decentralized or centralized model for SNMP monitoring and communication. In a
decentralized model, you create SNMP agents on each managed server. SNMP managers
communicate with the agents on individual managed servers.
In a centralized model, you create an SNMP agent only on the Administration Server. SNMP
managers communicate only with the SNMP agent on the Administration Server and the
agent gathers monitoring data from all managed servers in the domain. This model is
convenient and enables a single request to retrieve data for the entire domain, but there are
some potential drawbacks. If the Administration Server is unavailable, you cannot monitor the
domain through SNMP. The model also introduces performance overhead. Because the
Domain Runtime MBean server communicates with all managed servers in the domain, it is
subject to network latency and increases the amount of memory that the Administration
Server uses.
The domain-scoped agent is automatically overridden if you target a server SNMP agent to
the Administration Server.
WLS SNMP Architecture
Centralized
Admin
Server
Console
SNMP
JMX
Agent
Server
Server
Decentralized
Server
Agent
Console
SNMP
Server
Agent
A WebLogic Server SNMP agent can always communicate with managers by using the
SNMPv3 protocol. You can configure whether the agent also supports the SNMPv1 and
SNMPv2 protocols. While you cannot prevent an agent from receiving SNMPv3 requests, an
agent processes only requests from known users that you configure through the WebLogic
Server security realm.
If you plan to target the SNMP agent that contains a trap monitor to the Administration Server,
specify which servers in the domain you want the monitor to observe. If you target SNMP
agents to individual managed servers, you do not need to specify which servers you want to
monitor. An SNMP agent on a managed server monitors only its host managed server.
Although not shown here, WebLogic SNMP agents can also function as master agents that
forward (proxy) requests to other SNMP agents. To use the master agent functionality of a
WebLogic Server agent, you assign branches of the registration tree (OID tree) as the
responsibility of other SNMP agents. When an SNMP manager sends a request to a
WebLogic SNMP agent, if the OID of the requested object is under the branch of the OID tree
assigned to a proxied agent, the WebLogic SNMP agent forwards the request to the proxied
agent.
WLS SNMP Architecture
1
Domainwide agent
Create agents for

individual servers.
1. Under Domain Structure, expand Diagnostics and select SNMP.

2. Click the Agents tab if not already selected.
3. In the Server SNMP Agents table, click the New button. The SNMP page also displays a
Domain SNMP Agent table, which provides access to the default, domain-scoped
SNMP agent.
4. Enter a name for the SNMP agent and click the Finish button.
5. Select your new agent and configure it as desired.
Creating an SNMP Agent
Target to one or
more servers.
Use Inform messages

instead of traps?
How long to wait for

acknowledgement?
What protocol version

to use for traps?
SNMP UDP Port: The port on which you want this SNMP agent to listen for incoming
requests from SNMP managers that use the UDP protocol. If you target this SNMP
agent to multiple server instances, and if two or more servers are running on the same
computer, WebLogic Server will automatically increment this UDP port value by 1 for
each agent. WebLogic Server never assigns port 162, because it is the default port that
an agent uses to send notifications. In addition, if any port is already in use, WebLogic
Server skips the port and assigns the next available port.
Trap Version: The SNMP notification version that this SNMP agent generates.
Send Automatic Traps Enabled: Specifies whether this SNMP agent sends
automatically generated notifications to SNMP managers when servers are started or
stopped.
Engine ID: An identifier for this SNMP agent that is unique among all other SNMP
agents in the current WebLogic Server domain. If you use SNMPv3 to send messages
to this SNMP agent, you must specify the SNMP engine ID when you configure the
SNMP manager.
Inform Enabled: Configures this SNMP agent to send notifications as an INFORM
instead of a TRAP. Requires you to specify the agents SNMPTrapVersion as SNMPv2
or SNMPv3.
Inform Retry Interval: The number of milliseconds that this SNMP agent will wait for a
response to an INFORM notification before resending.
Configuring an SNMP Agent
To support SNMP over the TCP transport, create a separate

network channel.
An SNMP agent communicates through a port that accepts UDP traffic and another port that
accepts TCP traffic. By default, all TCP traffic uses the host servers listen port. For example,
if you target this agent to a server named ManagedServer1, and ManagedServer1 listens for
requests on port 7001, then the SNMP agent listens for TCP requests on port 7001. When
communicating through a TCP port, WebLogic Server protects SNMP communication from
denial of service (DOS) attacks. If you want to separate SNMP TCP traffic from business
traffic, you can create a custom network channel.
Create the SNMP channel as you would create any network channel for a server (under the
Protocols tab). Any SNMP agent that you target to this WebLogic Server instance will then
automatically use the SNMP network channel that you created.
SNMP Channels
By default, agents automatically send traps for server

startup and shutdown.
Agents also support custom traps that monitor log
messages or MBean attributes.
Monitor
Type
Change
Description
Sends a notification every time a configuration attribute has a new
value
String
Sends a notification if a runtime attribute is equal to or different from a

specified value
Gauge
Sends a notification if a runtime attribute is above a high threshold or

below a low threshold
Counter
Sends a notification if a runtime attribute exceeds some threshold
You can configure a WebLogic Server SNMP agent to detect certain thresholds or conditions
within a managed resource and send a report (notification) to one or more SNMP managers.
WebLogic Server SNMP agents will automatically generate the following notifications without
any additional configuration:
coldStart: The WebLogic Server instance that hosts the SNMP agent starts.
serverStart: A WebLogic Server instance that was down is now up. Contains two
name-value pairs to identify server start time and the server name.
serverShutDown: A server that was up is now down. Contains two name-value pairs
to identify server down time and the server name.
Each server instance saves messages in a local log file and then broadcasts them as JMX
notifications. You can set up a WebLogic Server SNMP agent to listen for all of these JMX
notifications or you can set up a filter based on criteria such as the severity level, subsystem,
or the message text. Log message notifications include these variable bindings: trapTime,
trapServerName, trapLogSubsystem, trapLogSeverity, and trapLogMessage.
JMX monitors poll WebLogic Server MBeans at a specified interval and send notifications to
an WebLogic SNMP agent when an event that you specify occurs, such as the crossing of a
threshold. The SNMP agent generates a notification and sends it to the SNMP managers.
Gauge and counter monitors are only applicable to numeric MBean attributes.
WLS SNMP Notifications
3
MBean attribute
to monitor
MBean instance
name (default = all)
2
A counter monitor periodically polls an attribute whose value is an integer and compares the
attribute value with a high threshold that you define. It generates a notification if the attribute
value equals or exceeds the high threshold. You can also configure an offset or modulus,
which increases or decreases the threshold each time the threshold is crossed.
1. Click the name of an existing SNMP agent.
2. Click Configuration > Counter Monitors. Then click New.
3. Enter the following information:
- Name: The name of the monitor
- Monitored MBean Type: The runtime MBean type that defines the attribute you
want to monitor
- Monitored MBean Name: The name of the MBean instance that you want to
monitor. If you leave the name undefined, WebLogic Server monitors all instances
of the MBean type that you specify in Monitored MBean Type. Alternatively, use
the User Entered MBean name field to enter a name directly.
- Monitored Attribute Name: The name of an MBean attribute to monitor. This
attribute must be in the WebLogic Server MIB.
- Polling Interval: The frequency (in seconds) that the server checks the attribute
value
Creating Trap Monitors
2
3
Location of
management console
Trap destinations specify the SNMP management station to which a WebLogic Server SNMP
agent sends notifications, both automatically generated ones and those generated by custom
monitors (counters, gauges, and so on).
1. Click the name of an existing SNMP agent.
2. Click Configuration > Trap Destinations. Then click New.
3. Enter the following information:
- Name: The name of this trap destination. This value is for your identification
purposes only.
- Community: The password that a WebLogic Server SNMP agent sends to the
SNMP manager when the agent generates SNMPv1 or SNMPv2 notifications. The
community name that you enter in this trap destination must match the name that
the SNMP manager defines. For SNMPv3, use the Security Name and Security
Level fields instead.
- Host: The DNS name or IP address of the computer on which the SNMP manager
is running
- Port: The UDP port on which the SNMP manager is listening
Creating Trap Destinations
SNMP V1 and V2 simply support clear text passwords

called communities.
SNMP V3 supports:
Authentication based on a username/password
MD5 or SHA hashing of authentication data to prevent
message tampering
DES or AES encryption based on a privacy password
The security features that are available for SNMP depend on which SNMP protocol an agent
uses to communicate with managers. To ensure that an SNMP manager requesting data from
the WebLogic SNMP agent has permission to obtain the data, and to verify that the agent has
permission to send notifications to a target manager, SNMPv1 and SNMPv2 use clear-text
passwords called community names.
In the SNMPv3 protocol, both SNMP agent and manager must encode identical credentials in
their PDUs for the communication to succeed. The credentials include several tokens: a
username, an SNMP engine ID, an authorization protocol, and an optional privacy password,
all of which are encrypted before being transported over the network.
SNMP Security
Use V1,V2
community strings?
Use V3
authentication?
Use V3
encryption?
Because SNMPv1 and SNMPv2 use clear-text passwords, the level of security is weak. If you
can use SNMPv3 to communicate with managers, consider disabling SNMPv1 and SNMPv2
by disabling Community Based Access Enabled for each SNMP agent.
Community Prefix: The password that you want this SNMP agent to use to secure
SNMPv1 or v2 communication with SNMP managers. SNMPv3 does not use community
names, so this field is ignored.
Authentication Protocol: The protocol that this SNMP agent uses to ensure that only
authorized users can request or receive information about your WebLogic Server
domain. Applicable only with SNMPv3. The protocol also ensures message integrity and
prevents masquerading and reordered, delayed, or replayed messages. If you do not
choose an authentication protocol, the SNMP agent does not authenticate incoming
SNMPv3 requests; anyone can use SNMPv3 to retrieve information about your
WebLogic Server domain.
Privacy Protocol: The protocol that this SNMP agent uses to encrypt and unencrypt
messages. If you do not choose a privacy protocol, communication between this agent
and managers can be viewed (but not altered) by unauthorized users.
Configuring Agent Security
Register username/passwords that will be:

Granted access to the SNMP agent
Included when sending notifications
Used as privacy passwords for encryption
Usernames must map to existing WLS users that have the

appropriate monitoring privileges.
In WebLogic Server, SNMPv3 agents work with the domains security realm to secure
communication. The SNMP agent decodes SNMP credentials in requests and passes the
SNMP username to the security realm. The security realm maps the SNMP username to a
WebLogic Server user, authenticates the user, and authorizes access to monitoring data in
the domain. To map the SNMP credentials to a user in a WebLogic Server security realm, you
create a credential map.
Under Domain Structure, click SNMP. Then click the Security tab. After clicking the New
button, select Authentication from the Credential Mapping Type list. In User Name, enter the
name of the WebLogic Server user. In SNMP Password, enter the authentication password
that SNMP managers will send in their requests. Finally, repeat these steps to create a
Privacy credential mapping and enter the privacy password that SNMP managers will send
in their requests.
To optimize performance, an SNMPv3 agent caches the credential maps that correlate
WebLogic Server users with SNMP credentials. To make sure that the cache contains the
latest set of SNMP credentials, an agent periodically invalidates its cache. After the cache is
invalidated, the next time the agent requests credentials, it regenerates its cache.
Configuring SNMP V3 Credentials
Use V1,V2
community strings?
Username for V3
authentication
Password for user stored

in credential map
Use V3 security?
If the SNMP agent sends SNMPv1 or v2 notifications and the management console requires a
specific community to accept traps, use the Community field in the corresponding trap
destination in WebLogic Server.
To use a specific SNMPv3 authentication and or privacy protocol when sending responses or
notifications, you must also configure the security level of your trap destinations. In Security
Name, enter the username on whose behalf the WebLogic SNMP agent sends notifications.
The username must be the name of an existing WebLogic Server user for whom you have
created an SNMP credential map. When the WebLogic SNMP agent prepares a notification, it
uses the credential map to look up and encode SNMP credentials.
In the Security Level list, select a security level that is equal to or higher than the security level
that is configured for receiving requests from SNMP managers. For example, if the WebLogic
SNMP agent requires incoming SNMPv3 requests to use the authentication protocol, the
security level for this trap destination must either require authentication or both authentication
and privacy.
Configuring Trap Destination Security
WLS includes a command-line SNMP utility that supports:

V1, V2, or V3 messages and all message types
Polling and capturing traps (UDP or TCP)
MIB modules (XML format)
V3 security features
Print Help for a specific command:
java weblogic.diagnostics.snmp.cmdline.Manager SnmpWalk -?
Poll a management object by using the WLS MIB:

java weblogic.diagnostics.snmp.cmdline.Manager SnmpWalk
-M /weblogic/diagnostics/snmp/mib -m BEA-WEBLOGIC-MIB
-h localhost -p 7090 O -u myuser -A mypassword
-e myEngineID safAgentRuntimeMessagesPendingCount
WebLogic Server provides a command line utility that offers many of the same features as an
SNMP manager. You can use this utility to test and troubleshoot the configuration of your
SNMP agents in a WebLogic Server domain. Like most WLS tools, first open a command
prompt (shell) and invoke the following script: <WL_HOME>\server\bin\setWLSEnv.sh
(or setWLSEnv.cmd on Windows). The script adds a supported JDK to the shells PATH
environment variable and adds WebLogic Server classes to the CLASSPATH variable.
Available commands include:
SnmpGet: Retrieves the value of one or more MIB variables. This command does not
accept OIDs for managed objects. You can specify an optional interval at which this
command repeatedly retrieves the value of the specified variable.
SnmpInform: Constructs a test INFORM notification and distributes it to an SNMP
manager or trap monitor.
SnmpGetBulk: Returns a collection of MIB variables by repeatedly invoking
SnmpGetNext in a pattern that you specify. Use the -Bn and -Bm arguments and one
or more OIDs to specify the pattern.
SnmpTrapMonitor: Starts a process that listens for notifications. Prints each
notification that it receives to standard out.
WLS SNMP Utility

Oracle Troublesoohting Vol 2

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Oracle Troublesoohting Vol 2

Uploaded by

Copyright:

Available Formats

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY.

COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Volume II Student Guide

Oracle University and Sentra inversiones y servicios LTDA use only

Oracle WebLogic Server 11g:

Copyright 2011, Oracle and/or it affiliates. All rights reserved.

This document contains proprietary

Oracle University and Sentra inversiones y servicios LTDA use only

Oracle University and Sentra inversiones y servicios LTDA use only

Diagnostic Framework Essentials

Oracle University and Sentra inversiones y servicios LTDA use only

Oracle University and Sentra inversiones y servicios LTDA use only

WLDF WLST Examples 4-20

Oracle University and Sentra inversiones y servicios LTDA use only

Oracle University and Sentra inversiones y servicios LTDA use only

JVM Recommendations 6-7

Troubleshooting Java Applications

Oracle University and Sentra inversiones y servicios LTDA use only

JVM Browser 6-50

Oracle University and Sentra inversiones y servicios LTDA use only

Too Many Open Files Errors 7-22

Oracle University and Sentra inversiones y servicios LTDA use only

Application Error Handling 8-38

Oracle University and Sentra inversiones y servicios LTDA use only

Oracle University and Sentra inversiones y servicios LTDA use only

MDB Capabilities 10-45

12 Troubleshooting Node Manager

Oracle University and Sentra inversiones y servicios LTDA use only

LDAP Search Operations 11-38

Oracle University and Sentra inversiones y servicios LTDA use only

Oracle University and Sentra inversiones y servicios LTDA use only

HttpSession API Overview 13-41

Oracle University and Sentra inversiones y servicios LTDA use only

Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Oracle University and Sentra inversiones y servicios LTDA use only

After completing this lesson, you should be able to:

Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 9 - 2

Oracle University and Sentra inversiones y servicios LTDA use only

Java Database Connectivity (JDBC):

Several drivers are included with WLS and are already in

Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 9 - 3

Oracle University and Sentra inversiones y servicios LTDA use only

Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 9 - 4

Oracle University and Sentra inversiones y servicios LTDA use only

Data Sources: Review

Data sources support various management tasks via the

Restart a data source using the latest configuration settings:

Copyright 2011, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 11g: Diagnostics and Troubleshooting 9 - 5

Oracle University and Sentra inversiones y servicios LTDA use only

JDBC Management: WLST Examples

Running, Suspended, Shutdown, Overloaded, or Unhealthy

Current number of connections in the pool

Highest number of connections in the pool since server start

Current number of connections reserved by applications

Highest number of reserved connections since server start

Average number of reserved connections at any given time

Current number of connections not reserved

Current number of connections that are either reserved or

Highest number of connections that were unavailable since