Oug Ireland 2012 Rittman Odi PDF

Inside Oracle Data Integrator 11g
Mark Rittman, Technical Director, Rittman Mead

OUG Ireland Conference 2012
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
Mark Rittman
Mark Rittman, Co-Founder of Rittman Mead
Oracle ACE Director, specialising in Oracle BI&DW
14 Years Experience with Oracle Technology
Regular columnist for Oracle Magazine
Author of forthcoming Oracle Press book on OBIEE 11g
Writer for Rittman Mead Blog :
http://www.rittmanmead.com/blog
Email : mark.rittman@rittmanmead.com
Twitter : @markrittman
Monday, 19 March 12
About Rittman Mead

Oracle BI and DW platinum partner
World leading specialist partner for technical excellence, solutions delivery and innovation in
Oracle BI
Approximately 50 consultants worldwide
All expert in Oracle BI and DW
Offices in US (Atlanta), Europe, Australia and India
Skills in broad range of supporting Oracle tools:
OBIEE
OBIA
ODIEE
Essbase, Oracle OLAP
GoldenGate
Exadata
Endeca
Monday, 19 March 12
Oracle Data Integrator 11g

Data Integration tool within Oracle Fusion Middleware 11g
Used for database, web service, file, XML etc integration
Declarative design, separation of
business and technical integration
Uses ELT approach to leverage
target platform capabilities
Licensed together with OWB as
Oracle Data Integrator EE
Strategic tool within Oracle for
data integration
Monday, 19 March 12
Oracle Data Integrator Key Features

Same philosophy as OWB and Oracle RDBMS Use the Database as the ETL engine
ODI supports heterogeneous databases, not just Oracle
Built for SOA environments
Support for Web Services, EII etc
Supports batch, event-based and real-time integration
Data Integrity Controls create a data firewall
Reduces data prep time by not
processing erroneous data
Extensible through Knowledge Modules
Change Data Capture
Slowly Changing Dimensions
Bulk load
Java client application
with server elements
Monday, 19 March 12
Oracle Data Integrator 11g Key Components

ODI Studio (the main development IDE)
Designer Navigator
Operator Navigator
Topology Navigator
Security Navigator
ODI Agents (used for executing mappings and packages)
ODI Console (web-based console for running published packages)
ODI Repositories (contain definitions of models, interfaces and other objects)
Monday, 19 March 12
ODI Studio Navigators

Modules within a single IDE used for performing certain tasks
Each module may be used by different category of ODI user (usage can be scoped)
Designer
Operator
Model data sources

Define projects
Create interfaces
Release scenarios
View running
interfaces and
packages
Operate production
Topology
Create data sources
Define agents
Manage infrastructure
Repository
Monday, 19 March 12
Security
Manage users and
priviledges
Designer Navigator
Used for designing ODI data, transformation and mapping objects
Import (reverse) metadata on data objects from source systems
Define variables, sequences, functions
Interfaces and packages
Use knowledge modules to
move data from source to target
Monday, 19 March 12
Topology Navigator
Defines the physical and logical architecture for the ODI system
Connections through to databases, OLAP servers, applications, files etc
Logical aliases for these connections
Contexts (environments)
Standalone and JEE agents
Monday, 19 March 12
Operator Navigator
Monitor the execution of interfaces, static controls and packages
Import and run compiled packages
Drill-into the actual code executed at each KM step
View errors, warnings and
execution metrics
Monday, 19 March 12
Agents
Java processes (applications) that are used for executing interfaces and packages
ODI Studio comes with an embedded agent,
or you can install them standalone on
Designer
Operator
one or more servers
Model data sources
View running
Define projects
interfaces and
ODI 11g introduces option to install into
Create interfaces
packages
Release scenarios
Operate production
WebLogic server (more resilent,
clustering etc)
Standalone
Agent
Orchestrate
interfaces
and scenarios
Information Systems
(Databases, files, applications)
Monday, 19 March 12
Repository
Master and Work Repositories

ODI systems connects to a single Master Repository, and one or more Work Repositories
Master Repository contains topology, security and version management details for all of an
organizations resources (generally only one per organization)
Work Repository contains data models, projects, interfaces etc
- Can have more than one work repository;
subdivide by project, department etc
Security
Work Repositories can either be
Topology
Development or Execution work repositories
Versioning
- Development work repositories are full-function,
Master Repository
and can be developed in
- Execution work repositories can only import
compiled ETL routines (scenarios) and
Models
execute them
Projects
Execution
Execution
Work Repository
(Development)
Monday, 19 March 12
Execution Repository
(Production)
ODI 11g Overall Logical Architecture, incl. Java EE Components
Monday, 19 March 12
ODI Heterogenous Connectivity

Extract and load to many data sources
Structured Data
Oracle, SQL Server, IBM DB/2 etc
Multi-dimensional Data
Essbase, Oracle OLAP, MS AS
Packaged Applications
Legacy
CICS, ISM, VSAM etc
B2B
EDIFACT, X12, RosettaNet
Financial Management
Hyperion, XBRL
Broad range of protocols
JCA, JDBC, ODBC, FTP, HTTP etc
T : +44 (0) 8446 697 995 E : enquiries@rittmanmead.com W: www.rittmanmead.com
Monday, 19 March 12
Business Rules and Technical Details

ODI splits data mappings in to business rules, and
technical implementation
Business rules define what goes where,
and using which transformation rules
Technical implementation defines
how data is moved
Changed Data Capture
SQL to SQL
File to SQL
Slowly Changing Dimension
Allows you to split mapping role into
business analyst and technical specialist
Monday, 19 March 12
Extensible Knowledge Modules

Provided out of the box, and are user extensible
Written for specific source and target platforms
User created example : Oracle 10g Data Pump
Monday, 19 March 12
ODI Within Oracle Fusion Middleware 11g

ODI is the data integration technology within Oracle Fusion Middleware 11g
Leverages key FMW technologies such as
WebLogic Server
Oracle Platform Security Services
(optional)
Web Services and SOA
Clustering and HA
Monday, 19 March 12
Oracle Data Integrator 11g New Features

New Fusion IDE (Same as JDeveloper, SQL Developer)
New Mapping Elements J2EE Deployment Option (using WebLogic Server)
Enterprise Scheduler (ESS) Integration
Enterprise Manager Integration
New ODI Console thin-client (replaces Metadata Navigator)
Java SDK APIs for embedding in applications
New SAP knowledge modules
New Golden Gate knowledge modules
New OBIEE knowledge module
Monday, 19 March 12
New Fusion User Interface - ODI Studio

New JDeveloper-based UI, same as SQL Developer, OWB 11gR2
Now called ODI Studio, replaces
separate Designer, Topology,
Security etc tools
Benefits from Fusion IDE
Framework features multi-tabs, log viewer panel etc
Monday, 19 March 12
New Interface Elements

Set operators for combining datasets (MINUS, INTERSECT etc)
Temporary interface as derived table
Complex file handling (multiple record files)
Technology updates and editions
Many more
Monday, 19 March 12
Load Plans
Alternative to packages for sequencing and orchestrating interfaces and procedures (scenarios)
Allows conditional logic for mapping exection
Define resume / restart settings for steps or groups of steps
Explicitely define parallel / serial execution
Monday, 19 March 12
Use of Repository Creation Utility

ODI Repository can now be created using RCU, as well as direct through ODI
No additional functionality, however aligns repository creation with rest of FMW11g
Monday, 19 March 12
Use of Oracle WebLogic Server

Agent can now be deployed within WebLogic Server domain
Allows agent to take advantage of connection pooling, HA etc
Deploy using a WebLogic Domain template
Monday, 19 March 12
Oracle Data Integrator Console

Evolution of ODI Metadata Navigator
View metadata information, run compiled packages
JEE thin-client application, runs in WebLogic Server
Monday, 19 March 12
Oracle Fusion Middleware Control Integration

Leverages Oracle Enterprise Manager Fusion Middleware Control
View status of agenda
View status of packages, interfaces
Manage clustered agents
View ODI metrics
Monday, 19 March 12
ETL Integration with OBIEE 11g

Connect the metadata in ODI and Oracle Business Intelligence
View metadata from OBIEE catalog object through to ODI source
OBIEE metadata extracted and loaded into ODI work repository, then reported on using
OBIEE dashboards
Monday, 19 March 12
Demonstration
Oracle Data Integrator 11g
Monday, 19 March 12
Some Typical ODI Development Tasks

Creating an interface (mapping)
Using some of the new 11g interface features - set operators, temporary table as sub-select
Creating a package and use of scenarios
And where load plans come in
Monday, 19 March 12
Interfaces within ODI

Interfaces define the mappings between sources and targets
Uses an E-L-T approach
Extract from sources; Load onto target; and Transform
Defined using the Designer navigator
Sequenced using packages
Monday, 19 March 12
Business Rules and Technical Details

ODI splits data interfaces in to business rules (mappings), and
technical implementation (flows)
Business rules define what goes where,
and using which transformation rules
Technical implementation defines
how data is moved
Changed Data Capture
SQL to SQL
File to SQL
Slowly Changing Dimension
Allows you to split mapping role into
business analyst and technical specialist
Monday, 19 March 12
Interface Designer within ODI Studio
Monday, 19 March 12
Available Sources & Targets
Drag and drop objects onto
mapping canvas
Access files, database tables, XML
files & other datastores
Monday, 19 March 12
Monday, 19 March 12
Sources Canvas
Data sources for the interface

Add joins, filters
Additional data sets through
set operations (union, intersect..)
Monday, 19 March 12
Monday, 19 March 12
Target Datastore
Datastore that will be loaded
by the interface
Columns can be transformed
Can be defined as temporary
interface (input to other interface)
Monday, 19 March 12
Monday, 19 March 12
Property Inspector
Define keys to be used

Expressions for columns
Other properties of the datastore
Monday, 19 March 12
Monday, 19 March 12
Components of an Interface
An interface is made up of three elements
The Mapping : the set of logical expressions mapping sources to target
The Flow : the choice of KMs plus their run-time settings
Controls (optional) : data-checking KMs that enforce data rules
Monday, 19 March 12
Monday, 19 March 12
Monday, 19 March 12
Creating an Interface Step 1 : Create & Name Interface

Right-click on Interfaces section in Project panel, select New Interface
Name the interface, provide description
Monday, 19 March 12
Creating an Interface Step 2 : Select Sources

Drag and drop source datastores from Models panel onto sources area
Monday, 19 March 12
Creating Joins Between Multiple Source Datastores

More than one datastore can be dropped onto the sources canvas
Join datastores by dragging and dropping key columns between datastores
Set join condition in the properties panel if join is non-equijoin
Monday, 19 March 12
Creating Filters
Individual datastores can be filtered
Drag and drop the column from the datastore onto the canvas
Set filter condition in the properties inspector
Monday, 19 March 12
Combining DataSets using Set Operations

DataSets with same column definition can be combined using set operations
Union, Union All, Intersect, Minus etc
Press Add/Remove Dataset button, choose set operation
Monday, 19 March 12

Monday, 19 March 12

Monday, 19 March 12
Creating an Interface Step 3 : Select Target Datastore

Interfaces have one target datastore
Drag and drop from the Models panel into the Target Datastore area
Monday, 19 March 12
Creating an Interface Step 4 : Define Mappings - Part I

Columns from the source area can either be dragged and dropped directly onto the
target datastore
Or they can be created or transformed using SQL expressions
SQL syntax is based on where the transformation is executed
Source, staging or target
Monday, 19 March 12
Creating an Interface Step 4 : Define Mappings - Part II

Execute on property specifies where the mapping expression will get substituted in the
query generated by KM task
Source - substituted in query to load data from source datastores into C$ table for
LKM and I$ table for IKM
Staging - substituted in query to load data from C$ table into I$ table for LKM or
source datastores into I$ table for IKM
Target - substituted in query to load data into target datastore either from I$ table or
source datastores
Monday, 19 March 12
Temporary Datastores (Yellow Interfaces)

Interfaces can become inputs into other interfaces
Designated by yellow interface icon in Project panel
Enter details in Temporary Target Properties panel to designate in this way
Interface icon becomes yellow, can be dropped onto other interfaces as source
Monday, 19 March 12
Creating an Interface Step 5 : Define Flows (KMs)

Second part of the interface, defines the technologies used for the mapping
Mapping is broken down into execution units
Select the appropriate KM for each unit
Select options presented by the KM (truncate table, check controls etc)
Monday, 19 March 12
Creating an Interface Step 6 : Optionally, Use Check KM

When enabled, runs all the checks defined by the constraints on the tables
Moves error rows to E$ error tables
Can be recycled later on if errors fixed
Generally good practice to include CKM in interface if constraints defined
Monday, 19 March 12
Creating an Interface Step 7 : Execute and Test Interface

Press Execute to run interface
View progress and outcome using the Operator navigator
Monday, 19 March 12
Demonstration
Defining an Interface
Monday, 19 March 12
What are Modular Interfaces?

Complex Integration Task - Requirement is broken down into set of regular and
temporary (yellow) interfaces
Pipeline - Interfaces are linked together to generate complex code or execute in an
orderly fashion
Monday, 19 March 12
What are Temporary Interfaces?

Modular - Interfaces designed to generate intermediate SQL, part of a complex query
or populate a temporary datastore
Re-usable - can be used as a source in any regular or temporary interface
Monday, 19 March 12
Create Modular Interface Step 1 - Create a Temporary Interface

Provide a Name and Description
On the Overview tab, select the right logical schema from the drop down menu
Logical schema forces ODI to generate code specific to target technology
Monday, 19 March 12
Create Modular Interface Step 2 - Set the Temporary Datastore

Provide a temporary datastore name under Temporary Target Properties
Under Target Datastore, create target columns and specify mappings
Interface icon becomes yellow on saving
Monday, 19 March 12
Create Modular Interface Step 3 - Set as a Source

Drag and Drop temporary interface as a source
Used like any other source to carry transformations building integration pipeline
Monday, 19 March 12
Create Modular Interface Step 4 - Sub-Select Property

Enable Sub-Select Property
Code generated from temporary interface acts as a derived table
Temporary interface execution not required
Disable Sub-Select Property
Temporary interface execution required
Temporary datastore used as a source
Monday, 19 March 12
Code Generation - Sub-Select Enabled

Use simulation to generate
complex SQL
Simulation can be performed at
any linked interface
Verify the SQL
Monday, 19 March 12
Demonstration
Temporary Interfaces, Sub-Select Tables and Interface Pipelines
Monday, 19 March 12
Sequencing Interfaces using Packages and Scenarios

ODI package is an organized sequence of steps
Each step performs a small integration task
They make up a workflow for data integration / data warehousing loads
Provides execution control like looping, branching , serial/parallel execution of steps, etc
Monday, 19 March 12
ODI Package Editor within ODI Studio
Monday, 19 March 12
Diagram Palette
Organizes sequence of steps to be
executed
Provides visual look to the workflow
Monday, 19 March 12
Monday, 19 March 12
Properties Panel
Each Step has different properties

Sets value for properties of a step
Monday, 19 March 12
Monday, 19 March 12
Toolbar
Icons to reorganize, delete, duplicate steps

Icons to draw transitions between steps to
control executions
Monday, 19 March 12
Monday, 19 March 12
Toolbox
Provides list of all the available ODI
tools
ODI tools can be added as step within
the package
Monday, 19 March 12
Monday, 19 March 12
Scenarios (Compiled Code) In ODI

Interfaces and other project objects can be added to packages in their regular,
uncompiled state
Useful during development
Or they can be compiled into Scenarios
Fixes the code at a certain state
Allows interfaces to be run in parallel
Required when promoting into production
Monday, 19 March 12
ODI Package - Controlling Execution

Each step can have two possible next steps
Next step upon success, transitioned using green ok line
Next step upon failure, transitioned using red ko line
Package stops If no next step is specified
Every package should have a first step, indicated by green arrow icon
First step is where the execution begins
No first step, package does not compile
Execution can branch
On success or failure of a step
Using evaluate variable as a step
Monday, 19 March 12
Creating a Package Step 1 : Create and Name Package

Right-click on the Packages entry under the required project, select New Package
Name the package and fill the Description
Monday, 19 March 12
Creating a Package Step 2 : Add Steps

Switch to the Diagram tab, drag and drop the required ODI objects (scenarios,
variables, etc) as steps from the project
ODI tool as step
To add click the ODI tool under Toolbox and then click under diagram palette
To stop adding ODI tool, click the arrow icon on the toolbar
Monday, 19 March 12
Creating a Package Step 3 : Set Properties for Steps

Select the step, under Properties panel set values for required properties
Runtime substitution - ODI variables\substitution apis
Static substitution - Hardcode values
Monday, 19 March 12
Creating a Package Step 4 : Add Transitions

Click the ok or ko transition icon on the toolbar and draw lines between steps
ok (green line) to next step on success
ko (red line) to next step on failure
Monday, 19 March 12
Creating a Package Step 5 : Save and Execute

Save ODI Package once all steps are added
Test execution to confirm functionality
Monday, 19 March 12
Demonstration
Compiling Interfaces into Scenarios, and Sequencing using a Package
Monday, 19 March 12
New in ODI 11.1.1.5+ : Load Plans

Alternative to packages for sequencing interfaces and other steps
Helps organize an optimal execution schedule for a batch
Advanced sequencing capabilities
Parallel or Serial
Conditional branching
Exception handling
Complements Scenarios and Packages, does not replace them
Can be created and edited in both Development and Execution work repositories
Monday, 19 March 12
Load Plans
Built up of a hierarchy of steps
Serial and Parallel run the steps within them
either sequentially, or in parallel
- Parallelize work to reduce net runtime
- for example, dimensions can often be
loaded independently, and thus
parallelized.
- Serialise work where dependencies exist
- eg. all dimensions must be loaded before
the facts can be
- eg. load base fact tables before
aggregates
Run Scenario executes a named Scenario
- Doesnt have to exist yet, can just be a
placeholder
Monday, 19 March 12
Conditional Branching
Variables can be included in a Load Plan, and used for conditional
execution of packages or steps
Also where required by
Case steps define a conditional branch in the Load Plan, based on
the evaluation of a variable
When and Else steps define the condition on the variable
Flexible list of operators
Only the first matching condition is executed
Monday, 19 March 12
Exception Handling
If a load plan step fails, it can call an Execution Step
It can either continue running, or raise the exception
If the exception is raised, then the exception handler for the parent step is invoked
Granular control of error handling, allowing groups of steps to fail whilst protecting the
overall batch
Serial/Parallel tasks can be configured to raise an exception after x failures of child
steps, or a fixed timeout
Monday, 19 March 12
Exception Steps
Exception steps can be defined and attached to one or many Load Plan Steps
Exception steps can invoke serial/parallel steps, scenarios
Conditional branching can be used within exception steps
Monday, 19 March 12
Restarting Failed Load Plans

The behavior of each step and scenario in a load plan on restart can be configured
Serial steps:
- Restart all children steps (failed or not)
- Restart from the failing child step
Parallel steps:
- Restart all children steps (failed or not)
- Restart just the children step(s) that failed
For Run Scenario steps:
Restart session, session step, or session task
Monday, 19 March 12
Running Load Plans

Load Plans can be executed in several ways:
On schedule, via the ODI scheduler
Command Line
Web Service
ODI Studio (i.e. manually)
A Load Plan must run on a JEE agent or standalone agent
it cannot run on a Local Agent
When a Load Plan runs, a Load Plan Instance is created
This Load Plan Instance will have one or many Load Plan Runs associated with it
Unlike restarting a session, which overwrites the previous one, a restarted Load Plan
will create a new Load Plan Run within the Load Plan Instance
Monday, 19 March 12
Running Load Plans

When a Load Plan is running, it can be monitored through Operator
The view is the same as when editing, i.e. the steps and flow are shown in context
Session IDs are hyperlinked to the session detail
Monday, 19 March 12
Demonstration
Load Plans
Monday, 19 March 12
Summary, and More Information

ODI is a fully-featured ETL and data integration tool, and available now on OTN
New features in ODI 11g to support complex mappings, enterprise deployments
More information can be found at http://www.rittmanmead.com
Contact us at info@rittmanmead.com or mark.rittman@rittmanmead.com
Follow-us on Twitter (@rittmanmead) or Facebook (facebook.com/rittmanmead)
Monday, 19 March 12
Inside Oracle Data Integrator 11g

Mark Rittman, Technical Director, Rittman Mead
OUG Ireland Conference 2012
Monday, 19 March 12

Oug Ireland 2012 Rittman Odi PDF

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Oug Ireland 2012 Rittman Odi PDF

Uploaded by

Copyright:

Available Formats

Inside Oracle Data Integrator 11g

Mark Rittman, Technical Director, Rittman Mead

About Rittman Mead

Oracle Data Integrator 11g

Oracle Data Integrator Key Features

Oracle Data Integrator 11g Key Components

ODI Studio Navigators

Model data sources

Master and Work Repositories

ODI 11g Overall Logical Architecture, incl. Java EE Components

ODI Heterogenous Connectivity

T : +44 (0) 8446 697 995 E : enquiries@rittmanmead.com W: www.rittmanmead.com

Business Rules and Technical Details

T : +44 (0) 8446 697 995 E : enquiries@rittmanmead.com W: www.rittmanmead.com

Extensible Knowledge Modules

T : +44 (0) 8446 697 995 E : enquiries@rittmanmead.com W: www.rittmanmead.com

ODI Within Oracle Fusion Middleware 11g

Oracle Data Integrator 11g New Features

New Fusion User Interface - ODI Studio

New Interface Elements

Use of Repository Creation Utility

Use of Oracle WebLogic Server

Oracle Data Integrator Console

Oracle Fusion Middleware Control Integration

ETL Integration with OBIEE 11g

Some Typical ODI Development Tasks

Interfaces within ODI

Business Rules and Technical Details

Interface Designer within ODI Studio

Interface Designer within ODI Studio

Available Sources & Targets

Drag and drop objects onto

Interface Designer within ODI Studio

Interface Designer within ODI Studio

Data sources for the interface

set operations (union, intersect..)

Interface Designer within ODI Studio

Interface Designer within ODI Studio

Datastore that will be loaded

Interface Designer within ODI Studio

Interface Designer within ODI Studio

Define keys to be used

Interface Designer within ODI Studio

Creating an Interface Step 1 : Create & Name Interface

Creating an Interface Step 2 : Select Sources

Creating Joins Between Multiple Source Datastores

Combining DataSets using Set Operations

Combining DataSets using Set Operations

Combining DataSets using Set Operations

Creating an Interface Step 3 : Select Target Datastore

Creating an Interface Step 4 : Define Mappings - Part I

Creating an Interface Step 4 : Define Mappings - Part II

Temporary Datastores (Yellow Interfaces)

Creating an Interface Step 5 : Define Flows (KMs)

Creating an Interface Step 6 : Optionally, Use Check KM

Creating an Interface Step 7 : Execute and Test Interface

What are Modular Interfaces?

What are Temporary Interfaces?

Create Modular Interface Step 1 - Create a Temporary Interface

Create Modular Interface Step 2 - Set the Temporary Datastore

Create Modular Interface Step 3 - Set as a Source

Create Modular Interface Step 4 - Sub-Select Property