11 - Odi CDC GG

Oracle Data Integrator 11g Bootcamp
Lesson 11 : ODI Change Data Capture (CDC)
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com
Agenda
Understand the role of CDC
Identify different CDC methods
Understand the role of ODI Journalizing
Overview of ODI and Oracle Golden Gate Integration
Learn how to Implement CDC
Overview of ODI Tools for CDC
Summary
What is CDC (Change Data Capture)?

A common requirement for (trickle-feed) data warehouses is to load just the new
and changed source data into the data warehouse
Less data to move, less data to process
CDC is a process to identify, capture and deliver the changes made to data
sources
Provides ability to extract data in real time
Allows sharing of changed data across users and applications
Coordinates retention/purge of change data
Synchronization and Replication of data can be also be performed
Different CDC Methods

CDC process use different methods to identify changed data
Timestamps on rows
Version numbers on rows
Status indicators on rows
Triggers on tables
Mining of logs on database
- Built-in: Oracle streams, LogMiner, etc
- Third party tools: Oracle Golden Gate, Attunity, etc
ODI provides pre-built journalized knowledge modules (JKM) to support various
CDC methods
All the JKM s integrate with ODI CDC framework
ODI CDC Framework - Journalizing

ODI performs CDC by journalizing a model
Journalizing consists of applying a JKM which creates the infrastructure to
capture change data
Journalizing infrastructure is a set of change tables, views, triggers, etc. to
capture, store and purge the changed data
LKM and IKM process data from this journalized infrastructure
Journalizing consists of two methods
Simple - tracks changes in individual datastores in a model
Consistent Set - tracks changes to a group of the model s datastores,
taking into account referential integrity between them
- Creates consistency window which guarantees consistency of
changed data
ODI CDC Framework - Journalizing Components

Journals - tables holding references to the changed records along with the
type of changes (insert/update/delete)
Capture processes - captures changed data from source datastores
ODI-managed triggers on the source datastore
Database specific programs to retrieve log data from data server log files
Subscribers - entities (applications, integration processes,etc.) that use the
changed data tracked on a datastore or consistent set
At least one subscriber needed to start CDC
Data purged from journals after all subscribers have consumed changed
data
Journalizing views - provides access to changed data, used by IKM s and
LKM s
Oracle Support for CDC

Oracle Database has an built-in CDC functionality that automates change extraction
Synchronous : based on triggers
Asychronous : based on log miner and streams
Works within and across
databases
Requires source RDBMS to be
Oracle 9i+ and in ARCHIVELOG
mode
ODI knowledge modules
support Oracle s built-in CDC
features, including sync/async
CDC
T : +44 (0) 8446 697 995 E : enquiries@rittmanmead.com W: www.rittmanmead.com
ODI CDC - Oracle

ODI leverages the CDC features in the Oracle database
ODI JKM s utilize the CDC features of the database
Generates and deploy all the code
Integrates with ODI CDC framework
Automates the entire CDC process
Standard edition license
Synchronous CDC using triggers
- JKM Oracle Simple
- JKM Oracle Consistent
JKM Oracle Consistent (Update Date)
Enterprise edition license
Asynchronous CDC using oracle streams
- JKM Oracle 11g\10g Consistent (Streams)
Implementing CDC Step 1 : Journalizing a Model

Open a model, select the Journalizing tab
Under the Journalizing KM section, set the Knowledge Module by selecting a JKM
from the drop down menu
Set the Journalizing Mode to Simple or Consistent Set using the radio button
This process will execute the applied JKM across all datastores in the model
Implementing CDC Step 2 : Adding Datastore to CDC

Right click a datastore, select Changed Data Capture > Add to CDC option
This step add the datastore to the CDC list
A yellow clock icon appears on top of the datastore
Implementing CDC Step 3 : Starting Journal

Right click a datastore, select Change Data Capture > Start Journal option
On the Select your Subscriber(s) dialog, select subscribers from the available list
and click the OK button
This step will execute the JKM to create journalizing infrastructure and start the
capture process
Implementing CDC Step 4 : Verifying Capture Process

Use the ODI Operator to make sure JKM executed successfully
A green clock icon should appear on top of datastore
Implementing CDC Step 5 : Journalized Views as source

Select the source datastore in the interface, under Source Properties enable
Journalized Data Only option
ODI will add a filter and force LKM and IKM to use journalized view as source to
process change data only
ODI Tools for CDC

ODI Tools are discussed in detail in the ODI Tools course module
ODI Tools for change data detection and to automate continuous processing
OdiWaitForData - waits for no. of rows in a table or set of tables
OdiWaitForLogData - waits for no. of modifications to occur on a journalized
table or a list of tables
OdiWaitForTable - waits for a table to be created and populated with predefined number of rows
OdiRefreshJournalCount - refreshes for a given journalizing subscriber the
number of rows to consume for given table list or CDC set
OdiRetrieveJournalData - retrieves for a given journalizing subscriber the
journalized events for given table list or CDC set
Oracle GoldenGate (OGG)

Acquired in 2009 from GoldenGate
Best-of-Breed & Easy-to-Deploy product
Change Data Capture engine (CDC)
Replicate and integrate transactional data
Sub-second speed
Multi-enterprise system support
Carry data between
(heterogenous) systems
Oracle databases
DB/2
MSSQL
... and so on
ODI CDC - Oracle Golden Gate

ODI 11g (and ODI 10g) now have knowledge module support for OGG
JKM Oracle to Oracle Consistent (OGG)
Automatic deployment of OGG configuration files
Integration with ODI CDC framework
Uses same design and deployment approach as other ODI CDC KMs
ODI 11g and OGG : Integration Overview

CDC (Journalizing) using ODI 10/11g and GoldenGate requires three schemas
Source (where the data is)
Staging (where GoldenGate copies the changes to, and J$ objects exists)
Target (where the data is finally integrated)
GoldenGate handles transporting changes from source to staging, and populates the J$/JV
$ structures required by ODI JKMs
ODI then takes the J$/JV$ structures in
staging, and uses them to load the target
Benefits of approach include
Uses standard ODI CDC pattern
Auto-setup of GG by ODI
Full use of GG capabilities
Can be swapped in for
existing RBDMS-based JKMs
Hands-On Labs
Lab 11 - ODI Change Data Capture (CDC)
Summary
CDC optimizes the extraction phase in a data warehouse environment

Guarantees transactional consistency for changes across multiple source tables
Acts as a framework for change data management
Oracle Data Integrator 11g Bootcamp

Lesson 11 : ODI Change Data Capture (CDC)

11 - Odi CDC GG

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

11 - Odi CDC GG

Uploaded by

Copyright:

Available Formats

Oracle Data Integrator 11g Bootcamp

Lesson 11 : ODI Change Data Capture (CDC)

What is CDC (Change Data Capture)?

Different CDC Methods

ODI CDC Framework - Journalizing

ODI CDC Framework - Journalizing Components

Oracle Support for CDC

T : +44 (0) 8446 697 995 E : enquiries@rittmanmead.com W: www.rittmanmead.com

ODI CDC - Oracle

Implementing CDC Step 1 : Journalizing a Model

Implementing CDC Step 2 : Adding Datastore to CDC

Implementing CDC Step 3 : Starting Journal

Implementing CDC Step 4 : Verifying Capture Process

Implementing CDC Step 5 : Journalized Views as source

ODI Tools for CDC

Oracle GoldenGate (OGG)

ODI CDC - Oracle Golden Gate

ODI 11g and OGG : Integration Overview

CDC optimizes the extraction phase in a data warehouse environment

Oracle Data Integrator 11g Bootcamp

You might also like