Maximum Availability Architecture(MAA):

Oracle E-Business Suite Release 12

Lyn Pratt, Richard Exley Metin Yilmaz
CMTS, MAA Group Senior Principal Support Engineer,
Oracle Server Technologies Oracle Support Services
The following is intended to outline our general
product direction. It is intended for information
purposes only, and may not be incorporated into any
contract. It is not a commitment to deliver any
material, code, or functionality, and should not be
relied upon in making purchasing decisions.
The development, release, and timing of any
features or functionality described for Oracle’s
products remains at the sole discretion of Oracle.
 Agenda

• Maximum Availability Architecture (MAA) <Insert Picture Here>

• E-Business Suite MAA

• E-Business Suite Disaster Recovery
• E-Business Suite MAA Demonstration
• New Database Features and E-Business Suite
 <Insert Picture Here>

Maximum Availability
Architecture (MAA)
 Maximum Availability Architecture (MAA)
Maximum Availability = Unbreakable Architecture + Best Practices

y Oracle's best practices blueprint based on proven Oracle high

availability technologies and recommendations
y Technology + Configuration + Operational Practices
y Applications, Enterprise Manager, Application Server, Collaboration
Suite and Database
• Constantly validated and enhanced as new products and features
become available
• Focused on reducing unplanned and planned downtime
• Focused on making our customers successful
• Papers published to the Oracle Technology Network (OTN)
• http://www.oracle.com/technology/deploy/availability/htdocs/maa.htm
 Presentation Focus

• This presentation is on Oracle E-Business Suite

Release 12 MAA
• Please see the MAA home page for detailed papers
covering Oracle E-Business Suite 11i and R12 MAA
• http://www.oracle.com/technology/deploy/availability/htdocs/maa.htm
 <Insert Picture Here>

E-Business Suite MAA

E-Business Suite MAA
Target Architecture

Primary Site Disaster Recovery Site

Application Tier

Oracle
Database Tier Oracle
RAC and RAC and
ASM ASM
Oracle Data Guard

Oracle Oracle
HA Storage Standby
Database
Database
 E-Business Suite MAA
Oracle Database MAA
Real Application Clusters
Data Guard
& Clusterware Fully Active
Fault Tolerant Failover Replica
Server Scale -Out

Primary Site Disaster Recovery Site

Database Servers Database Servers

Storage Storage

Automatic Storage Recovery Manager &

Flashback
Management Correct Errors by
Oracle Secure Backup
Fault Tolerant Low Cost High Performance
Moving Back in Time
Storage Scale -Out Data Protection and Archival
 E-Business Suite MAA
Transitioning to RAC and ASM

• Migrate Oracle Applications Release 12 to a Real Application

Clusters (RAC) environment running Oracle database server 10g
Release 2 with Automatic Storage Management (ASM)
• https://metalink.oracle.com/metalink/plsql/ml2_documents.showDoc
ument?p_database_id=NOT&p_id=388577.1

Oracle
RAC and
ASM

Oracle Oracle
Database Database
E-Business Suite MAA
Application HA Features
Hardware Load Balancers
Redundant Configuration

Multiple Web Servers

Application Tier Load Balanced

Database Tier
Multiple Forms Servers
Load Balanced

Parallel Concurrent Processing

Fault tolerant batch processing
Database or Application Tier
 E-Business Suite MAA
Load Balancing Configuration

• Load balancing distributes client requests across

multiple application tier nodes, providing additional
scalability and fault tolerance
• Implementation requires adding and configuring
application tier nodes and configuring the load
balancer
• Metalink note 380489.1 describes the application
configuration options in detail:
• https://metalink.oracle.com/metalink/plsql/ml2_documents.showD
ocument?p_database_id=NOT&p_id=380489.1
• Load balancer configuration is vendor specific
• Persistence through cookies is required
 E-Business Suite MAA
Parallel Concurrent Processing

• Parallel concurrent processing allows concurrent

managers to execute on multiple server nodes at the
same time, providing scalability and fault tolerance
• The feature is detailed in the Oracle Applications
System Administrator's Guide, Release 12:
• http://download.oracle.com/docs/cd/B34956_01/current/acrobat/120
sacg.pdf
• Metalink note 388577.1 explains additional
considerations when using PCP with RAC:
• https://metalink.oracle.com/metalink/plsql/ml2_documents.showDoc
ument?p_database_id=NOT&p_id=388577.1
E-Business Suite MAA
Target Architecture

Primary Site Disaster Recovery Site

Application Tier

Oracle
Database Tier Oracle
RAC and RAC and
ASM ASM
Oracle Data Guard

Oracle Oracle
HA Storage Standby
Database
Database
 <Insert Picture Here>

E-Business Suite
Disaster Recovery
 E-Business Suite Disaster Recovery

• In this section we describe how to setup and operate

a E-Business Suite disaster recovery site
• Detailed papers describing this process for 11i and
R12 are published on the MAA home page
• http://www.oracle.com/technology/deploy/availability/htdocs/maa.htm

• The process is based on Rapid Clone; see Metalink

note 559518.1 for how to clone an EBS environment:
• https://metalink.oracle.com/metalink/plsql/ml2_documents.showDoc
ument?p_database_id=NOT&p_id=559518.1
E-Business Suite Disaster Recovery
Creating the Standby Site
Standby Site Creation Process
Standby Primary Apps
Node

Clone Apps
Software
Apps Nodes

Establish Standby Prepare Apps

Hardware and Software for
System Software DR Database
Database
Primary

Clone
Nodes

Prepare for Data Backup

Database
Guard Operations Database
Software
Database
Standby

Establish Standby Establish Oracle Prepare New Establish

Nodes

Hardware and Clusterware and Database Standby

System Software ASM Instance Database
 E-Business Suite Disaster Recovery
Creating the Standby Site

• Establish Standby Hardware and System Software

• Install the standby site middle tier and database hardware,
networks, etc.
• Think carefully about the performance and HA
requirements - best to mimic production?
• Plan also for all required external services
• Install System Software

• Establish Oracle Clusterware

• Required for ASM and RAC

• Establish ASM
 E-Business Suite Disaster Recovery
Preparation on the Primary

• Prepare for Data Guard operation

• Enable archive log mode
• Enable forced logging
• Create database password files
• Configure SQL*NET communication between sites
• Set database parameters for Data Guard
• Create standby redo logs
 E-Business Suite Disaster Recovery
Clone and Configure Database Software

• Clone database software and prepare the standby instances

• Run the E-Business Suite pre-clone utility on a primary database
node
• Copy the database home to the standby database servers
• [RAC Only] Run adclonectx.pl on each standby database server
• Edit the context file to adjust the cluster_database_instances,
instance_number, instance_thread, and undotablespace
parameter values appropriately
• Run adcfgclone.pl on each standby database server
• Configure Data Guard database parameters on standby site
• Configure SQL*NET Communication on standby site
• [RAC Only] Restart the listeners
 E-Business Suite Disaster Recovery
Clone and Configure Application Software

• Clone the application tier to standby

• Run the pre-clone utility on a primary applications tier node
• Copy the software to the standby
• Run adcfgclone.pl on each standby middle tier server
• Use atTechStack option and run AutoConfig with
“run=INSTE8” option as there is no database connection
• [RAC Only] Edit the context file to point Tools OH
TWO_TASK, iAS OH TWO_TASK, and Apps JDBC Connect
Alias to the appropriate load balancing services
• (Optional) Establish log and out file synchronization
• Use your favorite tool for this, for example rsync
 E-Business Suite Disaster Recovery
Create the Physical Standby Database

• Establish the standby database

• Back up the production database including archive logs, and the
production control file for standby
• Restore the database to the DR site using (one of) the configured
instance(s)
• Start managed recovery
• Check out MAA papers for tuning best practices:
• http://www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10gR
2_RecoveryBestPractices.pdf
• http://www.oracle.com/technology/deploy/availability/pdf/MAA_WP_10gR
2_DataGuardNetworkBestPractices.pdf

• [RAC Only] Register the standby database with Oracle

Clusterware
 E-Business Suite Disaster Recovery
Operational Procedures

• Switchover
• Be sure you are up to date with redo apply
• Have procedures in place to also be up to date with file
system synchronization
• Shut down the app and all but one database instance on each
site
• Prepare the primary database for standby, switch the standby
to primary, open, and start other instances
• Remove the applications topology from the database
• Run AutoConfig on database tier (twice for RAC) then middle
tiers to configure the application for DR operation
• Start the Apps
• Convert the old primary database to a Data Guard standby
 E-Business Suite Disaster Recovery
Operational Procedures

• Failover (assuming you lost the primary database)

• Convert the standby to primary operation, open, and start
other instances
• Remove the application topology from the database
• Run AutoConfig on database tier (twice for RAC) and then on
middle tier to configure the application for the DR site
• Start the Apps
• If the primary site becomes available, flash the old primary
database back to before the failure and convert it to a
physical standby
• If the primary site is permanently lost, create a new standby
environment
 E-Business Suite Disaster Recovery
Operational Procedures

• Standby testing using flashback

• Cancel media recovery on the standby database
• Create a guaranteed restore point on the standby database
• Switch the redo log on the primary database
• Activate the standby database and configure EBS
• Take care with external interfaces such as Workflow -
disable them before starting EBS
• [Perform testing at the standby site]
• Flash the database back to the restore point
• Remove file system changes?
• Drop the restore point
• Convert the database back to a physical standby and resume
standby operation
 E-Business Suite Disaster Recovery
Automating Switchover and Failover

• Configure Data Guard Broker to automate Data

Guard operation and the database failover and
switchover steps
• Configure Data Guard Fast Start Failover to
automatically detect a database failure and initiate
failover
• Create a script to automate the application
configuration and startup
• Create DB_ROLE_CHANGE trigger that will execute
the script on switchover and failover
 <Insert Picture Here>

E-Business Suite MAA

Demonstration
 E-Business Suite MAA Demonstration
Hardware Vendor Partners

• Part of MAA means proving and testing our best

practices
• We work closely with our hardware vendor partners to
check that our joint solutions work well together
 E-Business Suite MAA Testing
Demo Configuration

• The demonstrations were developed and executed using HP

Systems hardware and software and F5 load balancers
• Hardware
• F5 BIG-IP Local Traffic Manager v9 - 3400 Series
• HP DL145 G2 running HP Load Runner v8.1 for load generation
• HP Integrity rx2620 servers for the application tier
• HP Integrity rx4640 servers for the database tier
• HP StorageWorks EVA4000 for database storage
• Software
• HP-UX 11i v3 (11.31) for application and database Servers
• HP Serviceguard extensions for RAC (SGeRAC) Version A.11.18
• Oracle Enterprise Manager Grid Control was used to monitor the
systems
E-Business Suite MAA Demonstration
Lab Configuration
Primary Site Disaster Recovery Site
Load Runner
HP DL145 G2

F5 BIG-IP LTM 3400 F5 BIG-IP LTM 3400

Application Tier

HP RX2620 HP RX2620 HP RX2620

Oracle Oracle
RAC and
Database Tier RAC and
ASM ASM

HP RX4640 HP RX4640 HP RX4640 HP RX4640

Oracle Data Guard

Oracle Oracle
HP EVA Database HA Storage HP EVA Database
4000 4000
 E-Business Suite MAA Testing
Current Test Configuration

• Testing for R12 was done with HP Systems hardware

and software and F5 load balancers
• Hardware
• F5 BIG-IP Local Traffic Manager v9 - 3400 Series
• HP DL145 G2 running HP Load Runner v8.1 for load
generation
• HP BL 460c servers for the application and database tiers
• HP StorageWorks EVA4400 for database storage
• Software
• Oracle Enterprise Linux 4.6 for application and database
Servers
• Oracle Enterprise Manager Grid Control was used to monitor
the systems
E-Business Suite MAA Testing
Current Configuration
Primary Site Secondary Site
Load Runner
HP DL145 G2

F5 BIG-IP LTM 3400 F5 BIG-IP LTM 3400

Application Tier

HP BL460c HP BL460c HP BL460c HP BL460c

Oracle Oracle
RAC and Database Tier RAC and
ASM ASM

HP BL460c HP BL460c HP BL460c HP BL460c

Oracle Data Guard

Oracle Oracle
HP EVA Database HA Storage HP EVA Database
4400 4400
 <Insert Picture Here>

E-Business Suite MAA

Demonstration
 <Insert Picture Here>

New Features and the

E-Business Suite
 New Database Features

• Secure File System (SFS) entering beta

• Streams Extended Datatype Support (EDS) being
made available this week
 Secure File System (SFS)

• File system in the database

• Conceptually like NFS client
• User mode filesystem, no changes to Linux kernel
• Standards-based POSIX compatible
• Transparent to file system applications
• Mount SFS at a Linux mount point with a normal Unix path
 SFS – Potential Opportunities for
E-Business Suite
• Concurrent Manager log and out files
• Shared NFS mount for all PCP processes
• Keep in EBS database. Failover to disaster recovery site will
be at virtually the same point in time as the OLTP
transactions
• Application software – shared NFS install for local
efficiency, DR consistency
• Address single point of failure
 Extended Datatype Support (EDS)

• Adds Streams support for:

• Object columns with simple or nested objects
• Object tables
• XMLType (CLOB, Binary, ObjectRelational)
• Varrays
• Partial Spatial types (SDO_GEOMETRY)
• The EDS package generates workarounds for moving
the above data
 Extended Datatype Support –
Generated Workaround
• Streams configuration
• Pre-instantiation setup
• Post-instantiation setup
• Clean-up
• Workaround to move unsupported data
• Log table for source and target databases, capable of holding
all the fields in the original table, using only supported data
types
• Trigger on base table on source database, to write inserts,
updates, deletes to log table
• DML handler on log table on target database, to replicate
original DML on the base table
 EDS – Potential Opportunities for
E-Business Suite
• Reporting database
• … if enough columns are now supported for the customer’s
reporting requirements
• Still up to the customer to determine viability / mechanics
• Database-level maintenance is being researched:
• Rolling database upgrade
• Cross-platform migration
 HA Sessions, Labs, Demos From Oracle Development
Mon, Sep 22
• 2:30 pm - Database 11g: Next-Gen HA, Moscone South 103
Tue, Sep 23
• 9:00 am - Active-Active Data Centers, Moscone South 103
• 11:30 am - Sharding with Oracle, Moscone South 302
• 11:30 am - HA with Oracle VM, Moscone West 3024
• 1:00 pm - Active Data Guard, Moscone South 104
Wed, Sep 24
• 9:00 am - Fusion Middleware Grid HA, Marriott Nob Hill AB
• 11:30 am - RMAN Best Practices, Moscone South 103
• 5:00 pm - Data Guard & Real Application Testing, Moscone 102
• 5:00 pm - EM in Secure MAA, Moscone West 2001
Hands On Labs - Thu, Sep 25
• 10:30 - 11:30 am, 12:00 - 1:00 pm - Active Data
Guard, Marriott Golden Gate A3
Wed, Sep 24 (contd.)
• 5:00 pm - E-Business Suite HA, Moscone West 2002/04
Thu, Sep 25
• 9:00 am - Oracle Secure Backup, Moscone South 102
• 10:30 am - Streams Replication, Moscone South 102
• 12:00 pm - Rolling Database Upgrades, Moscone South 103
• 1:30 pm - Streams Performance, Moscone South 102
• 3:00 pm - Oracle Grid Computing, Moscone South 303
• 3:00 pm - E-Business Suite R12 MAA, Moscone West 2007
• 3:00 pm - Siebel MAA, Moscone South 308
• 3:00 pm - Fusion SOA HA & Scalability, Marriott Salon 14/15
DEMOgrounds, Mon-Thu
• Active Data Guard, Streams, Oracle Secure
Backup, RMAN/Flashback, MAA
For More Information

search.oracle.com
Maximum availability architecture

or
oracle.com