Dynamic Paging of Data for Java and .

Net
LewisC(Database Architect) Posted 5/17/2006
Comments (15) | Trackbacks (0)

The last couple of weeks have been really head's down for me. I'm learning a new application,
a new database (including a data warehouse) and learning how .Net can integrate with Oracle
as a robust architecture for application development.

Anyway, a common need for Java and .Net is the need to page through result sets. Paging in
.Net is particularly important when running on 32 bit windows. The executable that runs the
server can only address 2GB of memory. If you have 100 users caching 100,000 records each,
you have a real problem.

In my case, I have some very large tables being accessed by very complex queries. The
response time does not lend itself to the normal methods of paging. I'm actually caching the
queries to intermediate tables and then returning dynamic paging sets from that.

Today I'm going to show you how to dynamically return a result set. In the near future, I will
show how to dynamically convert that result set into XML. Because it's on the middle tier, I
think XML is an obvious choice for the interface. If there is interest, I can show how to cache
large queries using intermediate tables also.

To start with, the requirement is to pass in a query, a start row and an end row. I want the
procedure to return the range of rows provided from the query. The query is generated by the
application and will provide it's own sorting (ORDER BY). I could have said send in the start row
and the number of rows to return but that's really just semantics. The procedure would work
fine either way.

I decided I wanted XML as my input parameter. That makes it very dynamic and the signature
can be changed over time with minimal impacts. So at this point, I know I will have XML as
input and a ref cursor as output. My package spec will look like this:
CREATE OR REPLACE PACKAGE paging AS
PROCEDURE get_data( p_input IN XMLTYPE,
p_output OUT SYS_REFCURSOR );
END;

If you aren't familiar with REF CURSORs, they are just instantiated cursors (technically, it's like
a pointer in C). Think of a regular PL/SQL cursor that is normally accessed by OPEN, FETCH,
CLOSE. A ref cursor is opened and you can pass it around like a variable. Who ever FETCHes,
gets the next row. The PL/SQL user guide on OTN has a good write up on REF CURSORs.

The input XML has to provide our parameters so it can look like this:

<DATA>
<QUERY></QUERY>
<BEGIN></BEGIN>
<END></END>
</DATA>
So, we have the spec and the XML, it's simple to put together:

CREATE OR REPLACE PACKAGE BODY paging AS

PROCEDURE get_data( p_input IN XMLTYPE,

p_output OUT SYS_REFCURSOR ) IS
BEGIN
OPEN p_output
FOR (
'SELECT a.* ' ||
'FROM ( ' ||
'SELECT rownum rn, b.* ' ||
'FROM (' ||
p_input.extract('/DATA/QUERY/text()').getSt
ringVal() ||
') b ' ||
'WHERE rownum <= :row_stop ' ||
') a ' ||
'WHERE a.rn >= :row_start '
)
USING p_input.extract('/DATA/END/text()').getStringVal(),
p_input.extract('/DATA/BEGIN/text()').getStringVal()
;

END;

END;

It doesn't get much simpler than that. The get_data procedure is a single PL/SQL command. It's
a dynamic select wrapped inside a couple of other selects to restrict the number of rows
returned.

It will make more sense once we see a sample call. Here is a PL/SQL script to call this
procedure:

DECLARE
v_xml_in XMLType; -- Item 1
v_ref_cursor sys_refcursor; -- Item 2

-- Item 3
v_LOCATION_ID NUMBER(4);
v_STREET_ADDRESS VARCHAR2(40);
v_POSTAL_CODE VARCHAR2(12);
v_CITY VARCHAR2(30);
v_STATE_PROVINCE VARCHAR2(25);
v_COUNTRY_ID CHAR(2);
v_rn NUMBER;
BEGIN

-- Item 4
SELECT XMLElement( "DATA",
 XMLForest(
'1' as "BEGIN"
, '10' as "END"
, 'select * from locations order by location_id' as
"QUERY"
)
)
INTO v_xml_in
FROM DUAL;

-- Item 5
DBMS_OUTPUT.PUT_LINE( v_xml_in.getStringVal() );

-- Item 6
paging.get_data( p_input=>v_xml_in, p_output=>v_ref_cursor );

-- Item 7
FOR i IN 1..10 LOOP
FETCH v_ref_cursor INTO v_rn,
v_location_id, v_street_address,
v_postal_code,
v_city, v_state_province, v_country_id;
dbms_output.put_line('Row Number: ' || to_char(v_rn) ||
', Location ID: ' || to_char(v_location_id) ||
', Address: ' || v_street_address );
END LOOP;

-- Item 8
CLOSE v_ref_cursor;

END;

I've labeled the sections as Items 1-8. Let's look at those one at a time.

1. v_xml_in is the XMLType variable that will hold our input parameters.
2. v_ref_cursor is the REF CURSOR with our result set. Based on the query we passed in,
the REF CUROSR is executing:

3. SELECT a.*
4. FROM (
5. SELECT rownum rn, b.*
6. FROM (
7. select *
8. from locations
9. order by location_id
10. ) b
11. WHERE rownum <= :row_stop
12. ) a
13. WHERE a.rn >= :row_start
14. These variables are discrete values that will be returned by the REF CURSOR. Normally,
when you call a REF CURSOR, you can pass a ROWTYPE variable. Because my dynamic
call is adding the ROWNUM (rn), the records don't match. I can either pass in discrete
 variables, or create my own record type. For simplicity sake, I just used discrete types
here.
15. On Item 4, I a executing a select against the DUAL table to create my input XML. I
could have concatenated strings together but I like to let the database do as much
work as possible, and this way I know it's going to be valid XML.
16. This DBMS_OUTPUT is just to show you what our newly created XML document looks
like,
17. Item 6 is where we call our stored procedure, passing in XML and getting back the REF
CURSOR.
18. Item 7 loops through 10 records and displays some of the data.
19. And finally, we close our cursor.

Does all of this makes sense? My real preference would be to have static SQL in my stored
procedures but the application I have inherited has the application generating all of the SQL
dynamically. In the future, I plan to push for moving all SQL calls to the database.

That's it for me today. Time to put my head back down and come up with some more solutions!
I'm really enjoying my new job. It involves some real technical challenges and I get to stretch
my mental muscles quite a bit.

Dynamic Paging of Data, Retrieve XML instead of REF CURSOR

LewisC(Database Architect) Posted 5/18/2006
Comments (2) | Trackbacks (0)

A couple of days ago I wrote about paging through data using a REF CURSOR and dynamic SQL.
Today I'm going to expand on that. Instead of using a REF CURSOR, it would be nice to pull back
an XML document with just the subset of records the REF CURSOR would normally provide.

We're going to add a new procedure to our package. Actually, we're going to overload the
existing procedure. Here's our new spec:

CREATE OR REPLACE PACKAGE paging AS

PROCEDURE get_data( p_input IN XMLTYPE,

p_output OUT SYS_REFCURSOR );

PROCEDURE get_data( p_input IN XMLTYPE,

p_output OUT XMLTYPE );
END;

Notice that our p_output parameter is now an XMLType. Pass in an XMLTYPE and a REF CURSOR
and you get your data as a result set. Pass in two XMLTypes and get your data back as an XML
document.

XMLType is supported in most languages. I know they're supported in Java, .Net, C and C++.
Post a comment if you would like to see sample calls from one of those languages.
Rather than write something from scratch to create an XML document, I'm going to reuse the
existing query and create an XML document from the REF CURSOR. The best part about this is
how little code it really requires.

Add this new procedure to the bottom of our existing package body:

PROCEDURE get_data( p_input IN XMLTYPE,

p_output OUT XMLTYPE ) IS

v_data sys_refcursor;
v_handle dbms_xmlgen.ctxHandle;

BEGIN

-- Retrieve the data

get_data( p_input, v_data );

-- If the ref cursor is not empty

IF v_data IS NOT NULL THEN

-- Get a handle to the XML document

v_handle := dbms_xmlgen.newcontext( v_data );

-- Replace the default ROWSET tag with my own

dbms_xmlgen.SetRowSetTag(v_handle, 'DocumentLevel');

-- Remove the default ROW level tag with my own

-- If I passed in text instead of NULL I would just change the
element name
-- I can leave it off altogther and get the default, canonical
format
dbms_xmlgen.SetRowTag(v_handle, NULL);

-- Display null tags for empty columns

dbms_xmlgen.SetNullHandling(v_handle, dbms_xmlgen.EMPTY_TAG);

-- Create an XML document out of the ref cursor

p_output := dbms_xmlgen.getXMLType( v_handle );
END IF;

END;

That's very little code that we've needed to add. And by overloading our procedure, we can
pick and choose when to use a REF CURSOR or XML. Actually, you could switch back and forth.
Get the first 10 rows as a REF CURSOR and the second set as XML. Your choice.

Here's a sample script to call it:

DECLARE
v_xml_in XMLType; -- Declare the input XML
v_xml_out XMLType; -- Declare our output XML

-- Notice we don't need to declare the column or record type

BEGIN

-- This is the exact same query from the ref cursor example
-- It generates the input XML
SELECT XMLElement( "DATA",
XMLForest(
'1' as "BEGIN"
, '10' as "END"
, 'select * from locations order by location_id' as
"QUERY"
)
)
INTO v_xml_in
FROM DUAL;

-- Display the input XML

DBMS_OUTPUT.PUT_LINE( v_xml_in.getStringVal() );

-- Call our new overloaded procedure

paging.get_data( p_input=>v_xml_in, p_output=>v_xml_out );

-- Display the result set XML

DBMS_OUTPUT.PUT_LINE( v_xml_out.getStringVal() );

END;

That's it. This script is easier to read and contains less code than our first example.

In this example, we've used both the SQLX support that Oracle provides as well as the XML API.

Hope you find it useful!

Dynamic Paging, Calling XML From Java

LewisC(Database Architect) Posted 5/23/2006
Comments (6) | Trackbacks (0)

Last week I wrote about how to dynamically get paged results using either a ref cursor or an
XML document. Rakesh wrote and asked to see an example of a call using Java. So that's what I
wrote.

Now, I want to point out that while I do a little bit of Java coding here and there, I'm no guru.
This code works but if you know how to improve it or have a nice way to expand the example,
please post it.

I think the code is pretty much self-explanatory. I used jDeveloper to compile this. It's a
console app that connects to the HR account of an XE database and returns some employee last
names.
If you don't have XE on your local machine, you'll have to modify the connect string.

// Just cause I like packages

package callxmltype;

// Standard Stuff
import java.sql.SQLException;

// SQL Related
import java.sql.Connection;
import java.sql.DriverManager;
import oracle.jdbc.OracleCallableStatement;

// Type Related
import oracle.jdbc.OracleTypes;
import oracle.xdb.XMLType;

// What a unique and interesting class name I have!

public class Class1 {

public static void main(String[] args) throws SQLException {

// What am I connecting to?

DriverManager.registerDriver(
new oracle.jdbc.driver.OracleDriver());

// Connect to it!
Connection conn =
DriverManager.getConnection(
"jdbc:oracle:thin:@localhost:1521:xe",
"hr", "hr");

// What will my input XML look like?

String queryText =
"<DATA><BEGIN>1</BEGIN><END>5</END><QUERY>" +
"select last_name from employees</QUERY></DATA>";

// Declare my XMLTypes
XMLType xt_in = XMLType.createXML(conn, queryText);
XMLType xt_out = null;

// Call my stored procedure

// Can also callit like
// OracleCallableStatement call =
// (OracleCallableStatement)conn.prepareCall(
// "{call paging.get_data( ?, ? ) |");
// I just like the Oracle syntax better
 OracleCallableStatement call =
(OracleCallableStatement)conn.prepareCall(
"begin paging.get_data( :1, :2 ); end;");

// Set the input parameter using setObject

call.setObject(1, xt_in);

// Register the output parameter as an OPAQUE

// Opaque is kind of like a generic, binary holder data type
// The first paramater is the index (which call parameter
// is it)
// The second parameter is the type, OPAQUE
// The third parameter is what the OPAQUE will map to, in
// this case XMLType
call.registerOutParameter(2,
OracleTypes.OPAQUE,
"SYS.XMLTYPE");

// Call the procedure

call.execute();

// Convert out out parameter to the XMLType variable

xt_out = XMLType.createXML(call.getOPAQUE(2));

// Here is where you can covert the XML, process it,

// send it to a web service, bind it to a grid, etc
// We'll just display the XML
System.out.println(xt_out.getStringVal());

// Close our connection

call.close();
}