<Insert Picture Here>

Oracle Data Integrator

Technical Session 3-4:
Defining The Logical Architecture. Setting up a new project
Designing Interfaces and Packages

Agenda
ODI Repositories
ODI Data environments
Reverse and enhance metadata
Design your transformations
Create packages workflow

<Insert Picture Here>

<Insert Picture Here>

ODI Repositories:
scalable metadata

Example of a Repository Set-Up

Security
Topology
Versioning
Create and archive
versions of models,
projects and
scenarios

Master
Repository

Import released
versions of models,
projects and
scenarios for testing

Import released and

tested versions of
scenarios for
production

Models
Projects

Models
Execution

Execution

Projects

Work Repository
(Development)

Execution

Execution Repository
(Production)

Work Repository
(Test & QA)

Development Test Production Cycle

<Insert Picture Here>

ODI Data environments:

scale from development to production

Logical vs. Physical Architecture

Physical Architecture
Defines physical resources
Describes the real locations of servers and schemas
How to access databases (Pack. Appl, etc.)
Which data area to use, also as physical staging area

Logical Architecture
Abstract view of the resources
The accounting application schema is defined only once
here. It is one Logical Schema implemented in different
Contexts.

Mapping Logical and Physical

Resources
Datawarehouse
(Logical Schema)

n
sto
Bo

Contexts

Tokyo

De
vel
opm
ent

Logical Architecture

Physical Architecture
Windows
MS SQL Server

db_dwh

Windows

Windows
MS SQL Server A
dwh

MS SQL Server

db_dwh
db_purchase

Development site: New York

Production site: Tokyo

Production site: Boston

Our Example Infrastructure With Agents

Development site:
New York
Windows
Oracle

Production site: Boston

Windows

Linux
Oracle 9

MS SQL Server

MS SQL Server

ACCOUNTING
ACCOUNTING

db_dwh

db_dwh

SALES

db_purchase

db_purchase

Oracle 10
SALES

Production site: Tokyo

Windows

Windows

MS SQL Server A

MS SQL Server B

Linux
Oracle
ACCT

dwh
purchase

SAL

<Insert Picture Here>

ODI still on step 1) going to production:

Reverse and enhance metadata

Standard vs. Customized ReverseEngineering

File-specific
reverse-engineering

ODI Repository

Oracle Data Integrator

Fixed format
COBOL copybooks

Delimited format

Model (Metadata)

MS SQL Server

JDBC Driver

Data Model

Standard
Reverse-engineering
System tables

Customized
Reverse-engineering

MDM Enabled: Canonical Format Design

Master Data

Use in conjunction with

packaged MDM solution
Design and Populate Canonical
Format
1. Use existing metadata artifacts
to design MDM application
(entities, fields, relationships)
2. Generate and maintain Master
Data structure
3. Generate and deploy
transformations using metadata
artifacts

Enterprise Service Bus

CRM

SCM

Legacy

ERP

Adding primary Keys

1.
2.
3.
4.
5.
6.

Select the Constraints node

under the datastore.
Right-click, select Insert Key.
Fill in the Name.
Select the Key or Index Type
Go to the Columns tab
Add/remove columns from the
key.

Creating a Reference

1.
2.
3.
4.

Select the Constraints node under

the datastore
Right-click, select Insert Reference
Fill in the Name
Select the reference type

5.

User Reference
Complex Reference

Select a Parent Model and Table

Set the model and table to

<undefined> to manually enter the
catalog, schema and table name.

Creating Simple or Complex Reference

Creating a Condition

1.

2.
3.
4.

Right-click Constraints node,

select
Insert Condition
Fill in the Name.
Select ODI Condition type.
Edit the condition clause

5.

Use the Expression Editor

Type in the error message for the

condition.

<Insert Picture Here>

ODI still on step 2) going to production:

Design your transformations

What Is an Interface?

An Interface is an ODI object that defines the

rules to load one target datastore with data
from one or more source datastores
A declarative design is used to implement the
transformations as mappings.

Using the Expression Editor

1. Click the expression
editor button ( ) in the
mapping window
2. Build your SQL
expressions from the
SQL help at the bottom,
and from the Columns at
the left

The Scenario
About the interface
Filter
- ORDERS.STATUS=CLOSED

Source Sybase

Target Oracle

ORDERS

SALES
LINES

CORRECTIONS
File

Join
- ORDERS.ORDER_ID =
LINES.ORDER_ID

Mapping
- SALES = SUM(LINES.AMOUNT)
+ CORRECTION.VALUE.
- SALES_REP =
ORDERS.SALES_REP_ID

What Defines the Flow?

Three factors:
Where the staging area is located
On the target, on a source or on a third server

How mappings, filters and joins are set up

Execution location: Source, target or staging area
Whether transformations are active

Choice of Knowledge Modules

LKM: Loading Knowledge Module
IKM: Integration Knowledge Module

The Basic Process

Sequence of operations

Source: Sybase
Transform & Integrate

Target: Oracle

ORDERS

1
LINES

Extract/Join/Transform
Join/Transform

CORRECTIONS
File

55

C$_0

22
Extract/Transform

C$_1

33

I$_SALES

SALES

What Is the Staging Area?

Staging Area A separate, dedicated area in

an RDBMS where Sunopsis creates its
temporary objects and executes some of your
transformation rules.
You usually put the staging area on the target
data server.

Case Study: Flow #1

The Staging Area on the target server
Target (Oracle)

Source (Sybase)

Staging Area
Transform & Integrate

ORDERS

1
LINES

Extract/Join/Transform
Join/Transform

CORRECTIONS
File

55

C$_0

22
Extract/Transform

C$_1

33

I$_SALES

SALES

Case Study: Flow #1 in Sunopsis

Staging Area in the Target

Staging Area + Target

Source Sets

Case Study: Flow #2

The Staging Area on a third server
DB2 UDB, Sunopsis Engine, etc.
Source (Sybase)
Staging Area

Transform & Integrate

Target (Oracle)

ORDERS

1
LINES

Extract/Join/Transform
Join/Transform

CORRECTIONS
File

55

C$_0

22
Extract/Transform

C$_1

33

I$_SALES

SALES

Case Study: Flow #2 in Sunopsis

Staging Area is the Sunopsis Memory Engine

Target

Source Sets

Staging Area

Case Study: Flow #3

The Staging Area on the Sybase source server

Source (Sybase)
ORDERS

1
LINES

55

C$_0

Extract/Join/Transform
Join/Transform

C$_1

22
CORRECTIONS
File

Transform & Integrate

Staging Area

Extract/Transform

33

I$_SALES

Target (Oracle)
SALES

Case Study: Flow #3 in Sunopsis

Staging Area in the Source

Target

Source Sets

Staging Area

KM Description
A Knowledge Module is
made of steps.
Each step has a name
and a template for the
code to be generated.
These steps are listed in
the Details tab.
The code that will be
generated by ODI will list
the same step names

Details of the Steps

Details of the steps are
generic: the source and target
tables are not known, only the
technologies are known
Substitution Methods are used
as placeholders for the table
names and column names
Parameters of the substitution
methods let you select which
tables or columns are used in
the KM

Which KMs for What Flow?

Four possible arrangements:
Loading phase

Integration phase

Target

Staging area

Source
Multi-tech LKM
Multi-tech LKM
(No LKM needed)
Staging area on source

(No LKM needed)

Multi-tech IKM
Single-tech IKM
Staging area on target

Multi-tech IKM
Single-tech IKM

Source, staging area and target in same location

Flow #1
Using the Target as the Staging Area
Target (Oracle)

Source (Sybase)

Staging Area

ORDERS

LKM_1
LKM_1
LINES

IKM_1
IKM_1

C$_0

LKM SQL to Oracle

IKM_1
IKM_1

CORRECTIONS
File

LKM_2
LKM_2
LKM File to Oracle (SQLLDR)

C$_1

I$_SALES

IKM Oracle Incremental Update

SALES

IKM Oracle Incremental Update

Flow #2
Using a third server as the Staging Area
Sunopsis Memory Engine
IKM SQL to SQL Append

Source (Sybase)
Staging Area

Target (Oracle)
IKM_1
IKM_1

ORDERS
C$_0

LKM_1
LKM_1
LINES

IKM_1
IKM_1

LKM SQL to SQL

CORRECTIONS
File

LKM_2
LKM_2
LKM File to SQL

C$_1

I$_SALES

IKM SQL to SQL Append

SALES

Flow #3
Using the Source as the Staging Area

Source (Sybase)
IKM SQL to SQL Append
ORDERS

Staging Area
IKM_1
IKM_1

LINES

C$_0

IKM SQL to SQL Append

LKM_1
LKM_1
LKM File to SQL

IKM SQL to SQL Append

IKM_1
IKM_1
C$_1

CORRECTIONS
File

Target (Oracle)

I$_SALES

IKM_1
IKM_1

SALES

Real-time enabled: Message Oriented

Integration

Subscribe

CDC

JMS Provider
(MOM, ESB)

Publish

Connect to Publish and

Subscribe JMS Message
Providers
Ensure messages delivery with
transaction integrity
High-volume bulk
transformations
1. Design complex bulk
transformations mixing Queues,
Databases and Applications
2. Use JMS Queues and topics as
sources or targets

ODI and XML files

Synchronization
or Connection
JDBC

Parser

XML

Integration

JDBC

Workspace for XML JDBC

(OME or RDBMS)

Loading
Driver

XML driver management

Source
Or Target

Agent

XML driver transforms the file

object in a relational model

ODI XML driver can handle also very huge XML files, using Oracle and non Oracle RDBMS as
workspace for the schema.
ODI and OWB can use also the native XML management functions in Oracle RDBMS (xml_load_f)

<Insert Picture Here>

ODI still on step 3) going to production:

Creating packages workflow

The Package Diagram

Toolbar

Diagram

ODI tool step

Toolbox for ODI
tools

Interface step
(selected)

Properties of
selected step

How to Create an ODI Tool Step

1.

2.
3.

In the Toolbox, expand the

group containing the tool you
want to add.
Click the tool.
Click the diagram.

4.
5.
6.

A step named after

the tool appears.

Change the Step Name

in the Properties panel.
Set the tools Properties.
Click Apply to save.

Defining a Variable and using it in a package

Declare Variable step type:

Set Variable step type:

Refresh Variable step type:

Evaluate Variable step type:

Create a New Procedure

1.

Right-click the Procedures node

under a project.
Select Insert Procedure
Fill in the

2.
3.

4.

Optionally, define the default:

5.

Name
Description
Source Technology
Target Technology

Click Apply to save.

How to Create Model, Sub-Model and Datastore Steps

1.

2.
3.
4.

Select the model, sub-model or datastore

which you want to work
with in the Models view.
Drag and drop it into the package.
Select the type of
operation to perform.
Set the Options
for the chosen
operation

A Simple Package
First step

Step on
success

Step on failure

This package executes two interfaces then archives some files.

If one of the three steps fails, an email is sent to the administrator.

How to Create a Loop

Loops need a counter variable

3.
1.
Package Loop

1.
2.
3.
4.

2.
4.

Set the counter to an initial value

Execute the step or steps to be repeated
Increment the counter
Evaluate counter value and loop to step 2 if the goal has been
reached

Executing a Package
1.

Click the Execute button

in the package window

2.

Open Operator

The package is
executed as a session

Each package step is a step

Tool steps appear

with a single task

Interface steps show each

command as a separate task