ETL - Oracle Database Features and PL/SQL Techniques

Boyan Boev CNsys

BGOUG - 2005

Agenda
ETL Concepts ETL Process Development Recommendations
Oracle Database Configuration PL/SQL Techniques Database Features

Conclusion
BGOUG - 2005

What is ETL?
ETL Elements ETL Purpose ETL as a Part of Business Intelligence Solution Types of ETL
Using Tools Custom PL/SQL Scripts
BGOUG - 2005

ETL Extract, Transform and Load

Extract
Pull the Data From the Source

Transform
Convert the Input Format to the Target Format Encode any Values

Load
Insert the Transformed Data to the Target Tables
BGOUG - 2005

Agenda
ETL Concepts ETL Process Development Recommendations
Oracle Database Configuration PL/SQL Techniques Database Features

Conclusion
BGOUG - 2005

Oracle database configuration

Dont use ARCHIVE LOG Mode
Reduce the Amount of I/O Activity

Use NOLOGGING DML Statements

ALTER TABLE NOLOGGING; INSERT /*+ nologging*/

Change the stage schema data model

Indexes Constraints Triggers
BGOUG - 2005

Agenda
ETL Concepts ETL Process Development Recommendations
Oracle Database Configuration PL/SQL Techniques Database Features

Conclusion
BGOUG - 2005

ETL recommendations PL/SQL Techniques

Bulk Binding Collection Types Pipelined Table Functions Package Variables Native Compilation
BGOUG - 2005

PL/SQL Binding Types of Binds

IN-Binds OUT-Binds

Bind Options
Single Row Binds Bulk Binds
BGOUG - 2005

Single Row Binds

Cursor For Loop
Example:
DECLARE CURSORexample_cursor( p_id_in NUMBER ) IS SELECT * FROM customer_action WHERE cust_id = p_id_in; v_cust_id NUMBER := 1681; BEGIN FOR rec INexample_cursor( v_cust_id ) LOOP INSERT INTO cust_action_hist() VALUES( ); END LOOP; END; /

BGOUG - 2005

Context Switching

BGOUG - 2005

Bulk Binding
IN-Binds

- An Array of Values is passed to SQL Engine

- SQL Engine populates a PL/SQL Bind Array

OUT-Binds

Context Switching once per Batch Instead of Once Per Row

Performance Increase to Up to 15 Times
BGOUG - 2005

Bulk Bind Operators

Bulk Collect
Specifying of the LIMIT clause Be Careful to handle last batch

OPEN cursor LOOP FETCH cursor BULK COLLECT INTO LIMIT 100; EXIT WHEN cursor%NOTFOUND; END LOOP;
BGOUG - 2005

Bulk Bind Operators

FORALL
Bulk DML Operator Not Looping Construct PL/SQL table is Referenced in the statement Handling and Reporting Exceptions VALUES OF , INDICES OF ( New in Oracle 10g )

FORALL i IN 1..20 INSERT INTO emp2 VALUES (enums(i), names(i), median(sals), ...);
BGOUG - 2005

Collection Types
Associative Arrays( PL/SQL Tables )
PL/SQL type only - not a SQL Type Easy to Use

Nested Tables
Shared Type

BGOUG - 2005

Associative Arrays( PL/SQL Tables )

automatic element allocation no need to initialize

DECLARE TYPE EmpTabTyp IS TABLE OF employees%ROWTYPE INDEX BY PLS_INTEGER; emp_tab EmpTabTyp; BEGIN SELECT * BULK COLLECT INTO emp_tab FROM employees WHERE employee_id = 100; END; /
BGOUG - 2005

Nested Tables
PL/SQL
Example

DECLARE TYPE nest_tab_t IS TABLE OF NUMBER; nt nest_tab_t := nest_tab_t(); BEGIN FOR i IN 1..100 LOOP nt.EXTEND; nt(i) := i; END LOOP; END; /
BGOUG - 2005

Nested Tables
Global
Example

CREATE OR REPLACE TYPE email_demo_obj_t AS OBJECT ( email_id NUMBER, demo_code NUMBER, value VARCHAR2(30) ); / CREATE OR REPLACE TYPE email_demo_nt_t AS TABLE OF email_demo_obj_t; /
BGOUG - 2005

Nested Tables
SQL-Defined Nested Tables
Sorted Aggregated Ised in Dynamic In-Lists Joined with SQL Tables Joined with other PL/SQL nested tables

Nested Tables Enable Table Functions

TABLE Operator
o tell Oracle to treat the Variable like SQL Table

CAST Operator
o Explicitly Tells Oracle the Data Type to be used to Handle the Operation

BGOUG - 2005

Nested Tables
Global Nested tables example:
DECLARE eml_dmo_nt email_demo_nt_t := email_demo_nt_t(); BEGIN -- Some logic that populates the nested table eml_dmo_nt.EXTEND(3); eml_dmo_nt(1) := email_demo_obj_t(45, 3, '23'); eml_dmo_nt(2) := email_demo_obj_t(22, 3, '41'); eml_dmo_nt(3) := email_demo_obj_t(18, 7, 'over_100k'); -- Process the data in assending order of email id. FOR r IN (SELECT * FROM TABLE(CAST(eml_dmo_nt AS email_demo_nt_t)) ORDER BY 1) LOOP dbms_output.put_line(r.email_id || ' ' || r.demo_id); END LOOP; END; /

BGOUG - 2005

Table Functions
Data is Buffered in the Local Variables During Function Execution Pipelined Table Functions
SELECT * FROM plsql_function;

BGOUG - 2005

Pipelined Functions - example

FUNCTION pipelined_demo( data SYS_REFCURSOR ) RETURN demo_nt_t PIPELINED IS CURSOR demo_cursor IS SELECT demo_obj_t(emp_id, value) FROM emp; emp_nt demo_nt_t; BEGIN OPEN email_demo_cur; LOOP EXIT WHEN email_demo_cur%NOTFOUND; FETCH email_demo_cur BULK COLLECT INTO eml_dmo_nt LIMIT 1000; FOR i IN 1..eml_dmo_nt.COUNT LOOP /* Apply some business logic on the object here, and return a row. */ PIPE ROW (eml_dmo_nt(i)); END LOOP; END LOOP; RETURN; END;

BGOUG - 2005

Package Variables
What are Package Variables
Globally Declared Variables in Package, outside of a Procedure or Function Definition Values set by Package Initialization Code Enable Data Caching

BGOUG - 2005

Native Compilation
How it works?
PL/SQL First is Compiled Down To P-Code C Source Code is Generated from P-Code Native Compiler is invoked Creating a 'C' Shared Object Library Subsequent Calls to PL/SQL Object are Run by the 'C' Library

NComp and Performance

BGOUG - 2005

Agenda
ETL Concepts ETL Process Development Recommendations
Oracle Database Configuration PL/SQL Techniques Database Features

Conclusion
BGOUG - 2005

ETL recommendations
Database Features and Utilities
External Tables( 9i ) Transportable Tablespaces DML Exception Handling( 10g R2 ) Regular Expressions Support( 10g R1 ) SQL*Loader Direct Path INSERT Parallel DML Statements
BGOUG - 2005

Database Features
External tables
File can be Queried as if it is a Real Database Table Can Sort, Aggregate, Filter Rows, etc. External File Can be Queried in Parallel

Oracle 9i - Read-Only using SQL*Loader Engine

Only read-only access to external files

Oracle 10g - Read-Write using Data Pump Engine

Can do read/write operations with external files

Transportable Tablespaces
Oracle 10g - transport across different platforms

BGOUG - 2005

Database Features
DML Exception Handling ( New in 10g R2 ) Catch the Error and Move On: Error Logging Clause
Enabling: SQL> exec dbms_errlog.CREATE_ERROR_LOG ('ACCOUNTS','ERR_ACCOUNTS'); SQL> insert into accounts select * from accounts_ny log errors into err_accounts reject limit 200; SQL> select ORA_ERR_NUMBER$, ORA_ERR_MESG$, ACC_NO from err_accounts;
BGOUG - 2005

Database Features
Regular Expression Support ( New in 10g R1 ) SQL*Loader Direct Path Insert
SQL*Loader CREATE TABLE AS SELECT INSERT /*+ APPEND*/ SELECT

Parallel DML Statements

BGOUG - 2005

Agenda
ETL Concepts ETL Process Development Recommendations
Oracle Database Configuration PL/SQL Techniques Database Features

Conclusion
BGOUG - 2005

Example
create package with PIPELINED Function
....

FUNCTION transform (new_data SYS_REFCURSOR) RETURN email_demo_nt_t PIPELINED PARALLEL_ENABLE ( PARTITION new_data BY ANY ) IS
....

Transformation is Just a Simple INSERT as SELECT

Elegant solution to Parallel , Transactional Co-Processing

INSERT /*+ append nologging*/ INTO ... SELECT /*+ parallel(a,4)*/ FROM .. TABLE( CAST( package.pipe_func_name( select * from table_3 ) as TYPE ) a);
BGOUG - 2005

Conclusion
High Perofmance ETL Solution

BGOUG - 2005

Recommended Readings Sites

http://www.oracle.com/technology//index.h tml http://rittman.net http://www.intelligententerprise.com/ http://asktom.oracle.com http://www.dbazine.com/datawarehouse http://dbasupport.com/

BGOUG - 2005

Recommended Readings Books

http://www.amazon.com/exec/obidos/tg/detail//0471200247/qid=1129731960/sr=2-1/ref=pd_bbs_b_2_1/1049260048-7507956?v=glance&s=books
BGOUG - 2005

Recommended Readings Books

http://www.amazon.com/exec/obidos/tg/detail//0764567578/ref=pd_sim_b_1/104-92600487507956?%5Fencoding=UTF8&v=glance
BGOUG - 2005

Recommended Readings Books

http://www.amazon.com/exec/obidos/tg/detail//1555583350/qid=1129732390/sr=22/ref=pd_bbs_b_2_2/104-92600487507956?v=glance&s=books
BGOUG - 2005

Questions or Comments
Boyan Boev
b.boev@cnsys.com

BGOUG - 2005