You are on page 1of 47

PL/SQL Webinars for Oracle Education

Collect Yourself: Optimize PL/SQL Code with Collections


Steven Feuerstein
PL/SQL Evangelist, Quest Software steven.feuerstein@quest.com www.ToadWorld.com/SF

Copyright 2000-2009 Steven Feuerstein - Page 1

How to benefit most from this session


Watch, listen, focus on concepts and principles. Download and use any of my the training materials:

PL/SQL Obsession

http://www.ToadWorld.com/SF

Download and use any of my scripts (examples, performance scripts, reusable code) from the same location: the demo.zip file. filename_from_demo_zip.sql You have my permission to use all these materials to do internal trainings and build your own applications.
But remember: they are not production ready. Modify them to fit your needs and then test them!
Copyright 2000-2006 Steven Feuerstein - Page 2

What we will cover on collections


Review of "foundation" features Indexing collections by strings Working with collections of collections MULTISET operators for nested tables Then we will apply collections (overview):
Data caching Bulk processing with FORALL and BULK COLLECT Table functions and pipelined functions
Copyright 2000-2008 Steven Feuerstein - Page 3

What is a collection?
1 abc 2 def 3 sf 4 q

...

22 rrr

23 swq

A collection is an "ordered group of elements, all of the same type." (PL/SQL User Guide and Reference)
That's a very general definition; from collections, you can build queues, stacks, lists, sets, arrays. Collections are single-dimensional and homogeneous, but you can emulate multi-dimensional structures.

Collections are a critical feature in many of the newest and most important features of PL/SQL.
Yet they are greatly underutilized by PL/SQL developers.
Copyright 2000-2006 Steven Feuerstein - Page 4

Why use collections?


Generally, to manipulate in-program-memory lists of information.
Much faster than working through SQL.

Serve up complex datasets of information to nonPL/SQL host environments using table functions. Dramatically improve multi-row querying, inserting, updating and deleting the contents of tables.
Combined with BULK COLLECT and FORALL....

Emulate bi-directional, random access cursors. Avoid mutating table trigger errors.
Copyright 2000-2006 Steven Feuerstein - Page 5

Three Types of Collections


Associative arrays (aka index-by tables)
Can be used only in PL/SQL blocks. Similar to hash tables in other languages, allows you to access elements via arbitrary subscript values.

Nested tables and Varrays


Can be used in PL/SQL blocks, but also can be the datatype of a column in a relational table. Part of the object model in PL/SQL. Required for some features, such as table functions With Varrays, you specify a maximum number of elements in the collection, at time of definition.
Copyright 2000-2006 Steven Feuerstein - Page 6

About Associative Arrays


Unbounded, practically speaking.
Valid row numbers range from -2,147,483,647 to 2,147,483,647.
This range allows you to employ the row number as an intelligent key, such as the primary key or unique index value, because AAs also are:

Sparse
Data does not have to be stored in consecutive rows, as is required in traditional 3GL arrays and VARRAYs.

Index values can be integers or strings (Oracle9i R2 and above). assoc_array_example.sql


Copyright 2000-2006 Steven Feuerstein - Page 7 collection_of_records.sql

About Nested Tables


No pre-defined limit on a nested table.
Valid row numbers range from 1 to 2,147,483,647.

Part of object model, requiring initialization. Is always dense initially, but can become sparse after deletes.

Can be defined as a schema level type and used as a relational table column type.
Copyright 2000-2006 Steven Feuerstein - Page 8 nested_table_example.sql

About Varrays
Has a maximum size, associated with its type.
Can adjust the size at runtime in Oracle10g R2.

Part of object model, requiring initialization.

Is always dense; you can only remove elements from the end of a varray.
Can be defined as a schema level type and used as a relational table column type.
varray_example.sql

Copyright 2000-2006 Steven Feuerstein - Page 9

How to choose your collection type


Use associative arrays when you need to...
Work within PL/SQL code only Sparsely fill and manipulate the collection Take advantage of negative index values and string indexing

Use nested tables when you need to...


Access the collection inside SQL (table functions, columns in tables) Want or need to perform high level set operations

Use varrays when you need to...


If you need to specify a maximum size to your collection Access the collection inside SQL (table functions, columns in tables).
Copyright 2000-2006 Steven Feuerstein - Page 10

Handy Collection Methods


Obtain information about the collection
COUNT returns number of rows currently defined in collection. EXISTS returns TRUE if the specified row is defined. FIRST/LAST return lowest/highest numbers of defined rows. NEXT/PRIOR return the closest defined row after/before the specified row. LIMIT tells you the max. number of elements allowed in a VARRAY.

Modify the contents of the collection


DELETE deletes one or more rows from collection. EXTEND adds rows to a nested table or VARRAY. TRIM removes rows from a VARRAY.
Copyright 2000-2006 Steven Feuerstein - Page 11

Useful reminders for PL/SQL collections


Memory for collections comes out of the PGA (Process Global Area) or UGA (User Global Area)
One per session, so a program using collections can consume a large amount of memory.

Use the NOCOPY hint to reduce overhead of passing collections in and out of program units. Encapsulate or hide details of collection management. Don't always fill collections sequentially. Think about how you need to manipulate the contents. Try to read a row that doesn't exist, and Oracle raises NO_DATA_FOUND.
nocopy*.* Copyright 2000-2006 Steven Feuerstein - Page 12

PL/SQL in Shared Memory


System Global Area (SGA) of RDBMS Instance
Shared Pool
Shared SQL Reserved Pool Pre-parsed
Select * from emp

Library cache
Update emp Set sal=...

Large Pool

calc_totals

show_emps

upd_salaries

Session 1

emp_rec emp%rowtype; tot_tab tottabtype;

emp_rec emp%rowtype; tot_tab tottabtype;

Session 1 memory (PGA/UGA)


Copyright 2000-2006 Steven Feuerstein - Page 13

Session 2 memory (PGA/UGA)

Session 2
plsql_memory*.*

How PL/SQL uses the SGA, PGA and UGA


The SGA contains information that can be shared across schemas connected to the instance.
From the PL/SQL perspective, this is limited to package static constants.
PACKAGE Pkg is Nonstatic_Constant CONSTANT PLS_INTEGER := My_Sequence.Nextval; Static_Constant CONSTANT PLS_INTEGER := 42; END Pkg;

The User Global Area contains session-specific data that persists across server call boundaries
Package-level data

The Process Global Area contains session-specific data that is released when the current server call terminates. Local data plsql_memory.pkg
plsql_memory_demo.sql
Copyright 2000-2006 Steven Feuerstein - Page 14

Expanded indexing capabilities for associative arrays


Prior to Oracle9iR2, you could only index by BINARY_INTEGER. You can now define the index on your associative array to be:
Any sub-type derived from BINARY_INTEGER VARCHAR2(n), where n is between 1 and 32767 %TYPE against a database column that is consistent with the above rules A SUBTYPE against any of the above.

This means that you can now index on string values! (and concatenated indexes and...)
Copyright 2000-2006 Steven Feuerstein - Page 15

Examples of New TYPE Variants


All of the following are now valid TYPE declarations in Oracle9i Release 2
You cannot use %TYPE against an INTEGER column, because INTEGER is not a subtype of BINARY_INTEGER.
DECLARE TYPE TYPE TYPE TYPE TYPE TYPE TYPE
INDEX BY BINARY_INTEGER; INDEX BY PLS_INTEGER; INDEX BY POSITIVE; INDEX BY NATURAL; INDEX BY VARCHAR2(64); INDEX BY VARCHAR2(32767); INDEX BY employee.last_name%TYPE; TYPE array_t8 IS TABLE OF NUMBER INDEX BY types_pkg.subtype_t; array_t1 array_t2 array_t3 array_t4 array_t5 array_t6 array_t7 IS IS IS IS IS IS IS TABLE TABLE TABLE TABLE TABLE TABLE TABLE OF OF OF OF OF OF OF NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER NUMBER

Copyright 2000-2006 Steven Feuerstein - Page 16

Working with string-indexed collections


The syntax is exactly the same, but you should keep this in mind:
The datatype returned by FIRST, LAST, NEXT and PRIOR methods is VARCHAR2. The longer the string values, the more time it takes Oracle to "hash" that string to the integer that is actually used as the index value.

If you are indexing by integer and find that your values are getting close to the limits (2**31 - 1 or -2**31 + 1), convert to a string assoc_array*.sql index.
Copyright 2000-2006 Steven Feuerstein - Page 17

assoc_array_perf.tst int_to_string_indexing.sql genaa.sql

Practical application for string indexing


I need to keep track of names used in my program.
Specifically, in Quest Code Tester, we generate test code and declare variables. So I need to make sure that I do not declare the same variable more than once.

There are lots of ways to do this, but string-indexed collections make it really easy!
FOR indx IN 1 .. l_variables.COUNT LOOP If varname_already_used THEN -- DO NOTHING ELSE

add_variable_declaration; mark_varname_as_used; END IF;


END LOOP; Copyright 2000-2006 Steven Feuerstein - Page 18

Without string indexing: string_tracker0.*

The String Tracker package (V1)


First iteration: I only need to maintain one list of names.
CREATE OR REPLACE PACKAGE BODY string_tracker IS TYPE used_aat IS TABLE OF BOOLEAN INDEX BY maxvarchar2_t; g_names_used used_aat; FUNCTION string_in_use ( value_in IN maxvarchar2_t ) RETURN BOOLEAN IS BEGIN RETURN g_names_used.EXISTS ( value_in ); END string_in_use;

PROCEDURE mark_as_used (value_in IN maxvarchar2_t) IS BEGIN g_names_used ( value_in ) := TRUE; END mark_as_used; END string_tracker;

string_tracker1.* Copyright 2000-2006 Steven Feuerstein - Page 19

Oracle9i

Multi-level Collections

Prior to Oracle9i, you could have collections of records or objects, but only if all fields were scalars.
A collection containing another collection was not allowed.

Now you can create collections that contain other collections and complex types.
Applies to all three types of collections.

The syntax is non-intuitive and resulting code can be quite complex.


Copyright 2000-2006 Steven Feuerstein - Page 20

String Tracker Version 2


The problem with String Tracker V1 is that it only supports a single list of strings.
What if I need to track multiple lists simultaneously or nested?

Let's extend the first version to support multiple lists by using a string-indexed, multilevel collection.
A list of lists....

Copyright 2000-2006 Steven Feuerstein - Page 21

The String Tracker package (V2)


CREATE OR REPLACE PACKAGE BODY string_tracker IS TYPE used_aat IS TABLE OF BOOLEAN INDEX BY maxvarchar2_t; TYPE list_of_lists_aat IS TABLE OF used_aat INDEX BY maxvarchar2_t; g_list_of_lists list_of_lists_aat; PROCEDURE mark_as_used ( list_in IN maxvarchar2_t , value_in IN maxvarchar2_t , case_sensitive_in IN BOOLEAN DEFAULT FALSE ) IS l_name maxvarchar2_t := CASE case_sensitive_in WHEN TRUE THEN value_in ELSE UPPER ( value_in ) END; BEGIN g_list_of_lists ( list_in ) ( l_name) := TRUE; END mark_as_used; END string_tracker;

string_tracker3.* Copyright 2000-2006 Steven Feuerstein - Page 22

Other multi-level collection examples


Multi-level collections with intermediate records and objects. multilevel_collections.sql Emulation of multi-dimensional arrays
No native support, but can creates nested collections to get much the same effect. Use the UTL_NLA package (10gR2) for complex matrix manipulation. multdim*.*

Four-level nested collection used to track arguments for a program unit.


Automatically analyze ambiguous overloading.
ambig_overloading.sql OTN: OverloadCheck
Copyright 2000-2006 Steven Feuerstein - Page 23

Encapsulate these complex structures!


When working with multi-level collections, you can easily and rapidly arrive at completely unreadable and un-maintainable code. What' s a developer to do?
Hide complexity -- and all data structures -- behind small modules. Work with and through functions to retrieve contents and procedures to set contents.
cc_smartargs.pkb: cc_smartargs.next_overloading cc_smartargs.add_new_parameter Copyright 2000-2006 Steven Feuerstein - Page 24

Oracle10g

Nested Tables unveil their MULTISET-edness

Oracle10g introduces high-level set operations on nested tables (only).


Nested tables are multisets, meaning that there is no inherent order to their elements and duplicates are significant.

You can now


Check for equality and inequality Perform UNION, INTERSECT and MINUS operations Check for and remove duplicates

Works with nested tables of scalars, records, objects.


Copyright 2000-2008 Steven Feuerstein - Page 25

Oracle10g

Check for equality and inequality

Just use the basic operators.and NULLs have the usual disruptive impact.
DECLARE TYPE clientele IS TABLE OF VARCHAR2 (64); group1 clientele := clientele ('Customer 1', 'Customer 2'); group2 clientele := clientele ('Customer 1', 'Customer 3'); group3 clientele := clientele ('Customer 3', 'Customer 1'); BEGIN IF group1 = group2 THEN DBMS_OUTPUT.put_line ('Group 1 = Group 2'); ELSE DBMS_OUTPUT.put_line ('Group 1 != Group 2'); END IF; IF group2 != group3 THEN DBMS_OUTPUT.put_line ('Group 2 != Group 3'); ELSE DBMS_OUTPUT.put_line ('Group 2 = Group 3'); END IF; END; Copyright 2000-2008 Steven Feuerstein - Page 26

10g_compare.sql 10g_compare2.sql 10g_compare_old.sql

Oracle10g

UNION, INTERSECT, MINUS

Straightforward, with the MULTISET keyword.


BEGIN our_favorites := my_favorites MULTISET UNION dad_favorites;
our_favorites := my_favorites MULTISET UNION DISTINCT dad_favorites; our_favorites := my_favorites MULTISET INTERSECT dad_favorites; our_favorites := dad_favorites MULTISET EXCEPT my_favorites; END;
SQL: UNION ALL

SQL: UNION

SQL: INTERSECT

SQL: MINUS

Copyright 2000-2008 Steven Feuerstein - Page 27

10g_setops.sql 10g_string_nt.sql 10g_favorites.sql 10g*union*.sql

Oracle10g

Distinct sets of values

Use the SET operator to work with distinct values, and determine if you have a set of distinct values.
DECLARE keep_it_simple strings_nt := strings_nt (); BEGIN keep_it_simple := SET (favorites_pkg.my_favorites); favorites_pkg.show_favorites ('FULL SET', favorites_pkg.my_favorites); p.l (favorites_pkg.my_favorites IS A SET, 'My favorites distinct?'); p.l (favorites_pkg.my_favorites IS NOT A SET, 'My favorites NOT distinct?'); favorites_pkg.show_favorites ( 'DISTINCT SET', keep_it_simple); p.l (keep_it_simple IS A SET, 'Keep_it_simple distinct?'); p.l (keep_it_simple IS NOT A SET, 'Keep_it_simple NOT distinct?'); END; 10g_set.sql 10g_favorites.pkg

Copyright 2000-2008 Steven Feuerstein - Page 28

Collections vs. Global Temporary Tables


Global temporary tables cut down on the overhead of working with persistent tables.
And you can use the full power of SQL, which is their main advantage over collections.

GTTs still require interaction with the SGA. So collections will still be faster, but they will use more memory.
GTTs consume SGA memory.
global_temp_tab_vs_coll.sql Copyright 2000-2008 Steven Feuerstein - Page 29

Applying Collections
Data caching using packaged data Turbo-charged SQL with BULK COLLECT and FORALL Table functions I offer light coverage of these topics, simply to ensure that you know what is possible.

Copyright 2000-2006 Steven Feuerstein - Page 30

Data Caching Options


Why cache data?
Because it is static and therefore you want to avoid the performance overhead of retrieving that data over and over again.

Options for caching data:


The SGA: Oracle does lots of caching for us, but it is not always the most efficient means. Package data structures: PGA memory has less access overhead than SGA. Oracle11g Function Result Cache Deterministic functions
Page 31

Packaged collection caching


Prior to Oracle 11g, the best caching option for PL/SQL programs involves declaring a package-level data structure.
It persists for the entire session. Usually a collection, to store multiple rows of data.

Why query information from the database (SGA) if that data does not change during your session?
Trivial example: the USER function More interesting: static tables

Instead, load it up in a package variable!


Page 32 Very simple example: thisuser.*

Data Caching with PL/SQL Tables


First access
Database
Not in cache; Request data from database Pass Data to Cache Data retrieved from cache Data returned to application

Application
PGA
Function
Application Requests Data

Subsequent accesses
Data found in cache. Database is not needed.

Data retrieved from cache

Data returned to application

Database

Application
PGA
Function
Application Requests Data

Copyright 2000-2006 Steven Feuerstein - Page 33

emplu.pkg emplu.tst 11g_emplu*.*

Turbo-charge SQL with bulk processing statements


Improve the performance of multi-row SQL operations by an order of magnitude or more with bulk/array processing in PL/SQL!
CREATE OR REPLACE PROCEDURE upd_for_dept ( dept_in IN employee.department_id%TYPE ,newsal_in IN employee.salary%TYPE) IS CURSOR emp_cur IS SELECT employee_id,salary,hire_date FROM employee WHERE department_id = dept_in; BEGIN FOR rec IN emp_cur LOOP adjust_compensation (rec, newsal_in); UPDATE employee SET salary = rec.salary WHERE employee_id = rec.employee_id; END LOOP; END upd_for_dept; Copyright 2000-2006 Steven Feuerstein - Page 34

Row by row processing: elegant but inefficient

Use the FORALL Bulk Bind Statement


Instead of executing repetitive, individual DML statements, you can write your code like this:
PROCEDURE upd_for_dept (...) IS BEGIN FORALL indx IN list_of_emps.FIRST .. list_of_emps.LAST UPDATE employee SET salary = newsal_in WHERE employee_id = list_of_emps (indx); END;

Things to be aware of with FORALL:


You MUST know how to use collections to use this feature! Only a single DML statement is allowed per FORALL. New cursor attributes: SQL%BULK_ROWCOUNT returns number of rows affected by each row in array. SQL%BULK_EXCEPTIONS... Prior to Oracle10g, the binding array must be sequentially filled. Use SAVE EXCEPTIONS to continue past errors. bulktiming.sql
Copyright 2000-2006 Steven Feuerstein - Page 35

bulk_rowcount.sql bulkexc.sql

Use BULK COLLECT INTO for Queries


DECLARE TYPE employees_aat IS TABLE OF employees%ROWTYPE INDEX BY BINARY_INTEGER; l_employees employees_aat; BEGIN SELECT * BULK COLLECT INTO l_employees FROM employees; FOR indx IN 1 .. l_employees.COUNT LOOP process_employee (l_employees(indx)); END LOOP; END;

Declare a collection of records to hold the queried data.


Fetch all rows into collection sequentially, starting with 1.

bulkcoll.sql

WARNING! BULK COLLECT will not raise NO_DATA_FOUND if no rows are found. Always check contents of collection to confirm that something was retrieved. Copyright 2000-2006 Steven Feuerstein - Page 36

Iterate through the collection contents with a loop.

Limit the number of rows returned by BULK COLLECT


CREATE OR REPLACE PROCEDURE bulk_with_limit (deptno_in IN dept.deptno%TYPE) IS CURSOR emps_in_dept_cur IS SELECT * FROM emp WHERE deptno = deptno_in;

Use the LIMIT clause with the INTO to manage the amount of memory used with the BULK COLLECT operation.

TYPE emp_tt IS TABLE OF emps_in_dept_cur%ROWTYPE; emps emp_tt; BEGIN OPEN emps_in_dept_cur; LOOP FETCH emps_in_dept_cur BULK COLLECT INTO emps LIMIT 100;
EXIT WHEN emps.COUNT = 0; process_emps (emps); END LOOP; END bulk_with_limit; Copyright 2000-2006 Steven Feuerstein - Page 37

WARNING! BULK COLLECT will not raise NO_DATA_FOUND if no rows are found. Best to check contents of collection to confirm that something was retrieved.

bulklimit.sql

The Wonder Of Table Functions


A table function is a function that you can call in the FROM clause of a query, and have it be treated as if it were a relational table. Table functions allow you to perform arbitrarily complex transformations of data and then make that data available through a query.
Not everything can be done in SQL.

Combined with REF CURSORs, you can now more easily transfer data from within PL/SQL to host environments.
Java, for example, works very smoothly with cursor variables
Copyright 2000-2006 Steven Feuerstein - Page 38

Simple table function example Return a list of names as a nested table, and then call that function in the FROM clause.
CREATE OR REPLACE FUNCTION lotsa_names ( base_name_in IN VARCHAR2, count_in IN INTEGER ) RETURN names_nt IS retval names_nt := names_nt (); BEGIN retval.EXTEND (count_in); FOR indx IN 1 .. count_in LOOP retval (indx) := base_name_in || ' ' || indx; END LOOP; RETURN retval; END lotsa_names; Copyright 2000-2006 Steven Feuerstein - Page 39

SELECT column_value FROM TABLE ( lotsa_names ('Steven' , 100)) names; COLUMN_VALUE -----------Steven 1 ... Steven 100

tabfunc_scalar.sql

Streaming data with table functions


You can use table functions to "stream" data through several stages within a single SQL statement.
Example: transform one row in the stocktable to two rows in the tickertable. CREATE TABLE stocktable (
ticker VARCHAR2(20), trade_date DATE, open_price NUMBER, close_price NUMBER
) / CREATE TABLE tickertable ( ticker VARCHAR2(20), pricedate DATE, pricetype VARCHAR2(1), price NUMBER) /

tabfunc_streaming.sql

Copyright 2000-2006 Steven Feuerstein - Page 40

Streaming data with table functions - 2


In this example, transform each row of the stocktable into two rows in the tickertable.
CREATE OR REPLACE PACKAGE refcur_pkg IS TYPE refcur_t IS REF CURSOR RETURN stocktable%ROWTYPE; END refcur_pkg; / CREATE OR REPLACE FUNCTION stockpivot (dataset refcur_pkg.refcur_t) RETURN tickertypeset ...

BEGIN INSERT INTO tickertable SELECT * FROM TABLE (stockpivot (CURSOR (SELECT * FROM stocktable))); END; /
Copyright 2000-2006 Steven Feuerstein - Page 41

tabfunc_streaming.sql

Use pipelined functions to enhance performance.


CREATE FUNCTION StockPivot (p refcur_pkg.refcur_t) RETURN TickerTypeSet PIPELINED

Pipelined functions allow you to return data iteratively, asynchronous to termination of the function.
As data is produced within the function, it is passed back to the calling process/query.

Pipelined functions can be defined to support parallel execution.


Iterative data processing allows multiple processes to work on that data simultaneously.
Copyright 2000-2006 Steven Feuerstein - Page 42

Applications for pipelined functions


Execution functions in parallel.
In Oracle9i Database Release 2 and above, use the PARALLEL_ENABLE clause to allow your pipelined function to participate fully in a parallelized query. Critical in data warehouse applications.

Improve speed of delivery of data to web pages.


Use a pipelined function to "serve up" data to the webpage and allow users to being viewing and browsing, even before the function has finished retrieving all of the data.
Copyright 2000-2006 Steven Feuerstein - Page 43

Piping rows out from a pipelined function


Add PIPELINED keyword to header CREATE FUNCTION stockpivot (p refcur_pkg.refcur_t) RETURN tickertypeset PIPELINED IS out_rec tickertype := tickertype (NULL, NULL, NULL); in_rec p%ROWTYPE; BEGIN LOOP FETCH p INTO in_rec; EXIT WHEN p%NOTFOUND; out_rec.ticker := in_rec.ticker; out_rec.pricetype := 'O'; out_rec.price := in_rec.openprice; PIPE ROW (out_rec); END LOOP; CLOSE p; RETURN; END; tabfunc_setup.sql tabfunc_pipelined.sql

Pipe a row of data back to calling block or query

RETURN...nothing at all!

Copyright 2000-2006 Steven Feuerstein - Page 44

Table functions Summary


Table functions offer significant new flexibility for PL/SQL developers. Consider using them when you...
Need to pass back complex result sets of data through the SQL layer (a query); Want to call a user defined function inside a query and execute it as part of a parallel query.

Copyright 2000-2006 Steven Feuerstein - Page 45

Collections don't start coding without them.


It is impossible to write efficient, high quality PL/SQL code, taking full advantage of new features, unless you use collections.
From array processing to table functions, collections are required.

Today I offer this challenge: learn collections thoroughly and apply them throughout your backend code.
Your code will get faster and in many cases much simpler than it might have been (though not always!).
Copyright 2000-2009 Steven Feuerstein - Page 46

Some Useful URLs...


E-Attendance link http://education.oracle.com/eattendance.html (Class id : 2347771) Course Evaluation link https://eval.oracle.com Instructor Feedback link : https://ougbsapex.us.oracle.com/pls/ougbsapex/f?p=120:17

Copyright 2000-2009 Steven Feuerstein - Page 47

You might also like