High Performance JDBC

Jorg Janke
Consultant and Coach
jorg.janke@compiere.org

Agenda

What I do
Persistence
Performance
JDBC

Connection
Query
Result

Timing Statistics

Shameless Self-Promotion

Consultant & Coach (available)

I talk business

Trained Accountant, MBA

VP Product Marketing

I talk tech

Started programming PL/I, SmallTalk,

Oracle5
Director in Oracle Apps Development
Certified Oracle DBA, Sun Java Pr/Dev/Arch

Compiere

The Open Source Business Solution

Targeting Small-Medium Enterprise

More than 240.000 downloads

Usually among Top 10 in SourceForce
ASP Solution
Accounting, Inventory,
Worldwide (Language, Law, )

J2EE compliant Application

Swing or HTML interface

Agenda -- Persistence

What I do
Persistence
Performance
JDBC

Connection
Query
Result

Persistence

Object meets Database

Put a square

in a round

JDBC is not a Persistence Framework

Object-relational Mapping

Container Managed Persistence

Java Data Objects

Persistence Entity Bean

CMP
<enterprise-beans>
<entity>
<description>Models a music CD</description>
<ejb-name>CDBean</ejb-name>
<home>com.web_tomorrow.cd.CDHome</home>
<remote>com.web_tomorrow.cd.CD</remote>
<ejb-class>com.web_tomorrow.cd.CDBean</ejb-class>
<persistence-type>Container</persistence-type>
<prim-key-class>java.lang.String</prim-key-class>
<reentrant>False</reentrant>
<cmp-field><field-name>id</field-name></cmp-field>
<cmp-field><field-name>title</field-name></cmp-field>
<cmp-field><field-name>artist</field-name></cmp-field>
<primkey-field>id</primkey-field>
</entity>
</entity-beans>

Persistence JDO

Java Data Objects

Transparent database access

Code that accesses the underlying data
store, without using database-specific code

Use of JDO in J2EE

Persistent helper classes for Session Beans

Delegate classes for Bean or Container
Managed Persistence Entity Beans

JDBC Based Frameworks

Everyone ( and their Grandmother) writes

JDBC based Persistence Libraries

Free or Commercial
Often with IDE integration
Examples

Borland
Oracle
Compiere (no IDE)

30 % of
typical IT project:
DB programming &
persistence mgt

Agenda -- Performance

What I do
Persistence
Performance
JDBC

Connection
Query
Result

Performance

Is a side effect of good OO design

Schema should be 3rd Normal Form

That is want Optimizers recognize

For high-end databases (e.g. Oracle)

there is no need for de-normalization
Performance Rules

#1 Clean design
#2 Tweak performance

Dont Design for

Performance

Performance Rules

Tune the Design

Tune the Application
Tune the OS
Tune Memory Structure
Tune Disk I/O Usage
Detect and Eliminate Resource
Contention

Performance Design
Options

Data Intensive Routines in DB Server

Within DB Engine - Low Switching Cost

No Communication Overhead
PL/SQL, Java Stored Procedures, C++,

Automatic data consistency

(Helps the Optimizer)

Explicit Constraints
Trigger (in PL/SQL, Java, )

DB Programming Options

Set Processing outperforms everything else

Not your beginner SQL

PL/SQL

Complex UPDATE statements

Dependency Management (*)

Data intensive manipulations

Java Stored Procedures

Special VMs Upload, Debugging

Special JDBC Driver always connected

DB Programming Options

SQLJ

Java Language SQL Pre-compiler

High level SQLJ-Language
Converted to normal JDBC code
Supported by IBM, Oracle,

Question:
Is SQL a Standard?

DB Independence

Lowest common denominator

Use Access Functionality with Oracle ?

You have to design & program

differently for the database / camps

Oracle, DB2, (PostgreSQL),

Microsoft SQL Server, Sybase,
MS Access, mySQL,

DB Incompatibilities (1)

SQL join

FROM
FROM
FROM
FROM

a JOIN b ON (a.id=b.id);
a LEFT OUTER JOIN b ON (a.id=b.id);
a, b WHERE a.id=b.id(+);
a, b WHERE a.id=*b.id;

SELECT .. FROM a, (SELECT .. FROM b) AS b WHERE

Pseudo-Columns

..
..
..
..

Sub-Query

SELECT
SELECT
SELECT
SELECT

ROWNUM, SysDate
LIMIT

Reserved Words Optional Syntax Alias

DB Incompatibilities (2)

Data Types

PL/SQL - Syntax/Functionality for

Packages, Functions, Procedures, Triggers,

Not just naming (NUMBER vs. DECIMAL)

BLOB, CLOB,

Commit Strategy
Exception (raise & catch)

Data Dictionary
OLAP Support

Materialized Views,

DB Independence cont.

DB Management Layer

Lowest Common SQL

Denominator Alternative
Application Dependent

Uses Database specific Functionality

Examples:

SAPs DB Access Layer

(CMP Products)
Compieres dbPort

I am
talking
to
Oracle

Application
dbPort
Oracle

PG

DB/2 ?

Performance Reality

You cant have everything

High functionality & Ability to fine-tune

Database independence

Automatic Object Mapping

You need skill-mix

(Traditional) Database Design

Database Programming Environment
Object Design
Java Programming Environment

The Test Environment

Windows 2000 SP 2 Server

RedHat Linux 7.2

Dual 450MHz CPU

Oracle 8.1.7 (parallel option not enabled)
JDBC driver for 9i
Single 500 MHz CPU
PostgreSQL 7.1.3
JDBC driver for 7.2

Testing DB
Driver

Java 1.4.0
SELECT * FROM AD_Column;

4,800 Rows - 38 Columns ca. 1,2 MB

Agenda -- JDBC

What I do
Persistence
Performance
JDBC

Connection
Query
Result

Connecting

Connection Options

java.sql.Driver &
java.sql.DriverManager
javax.sql.DataSource
javax.sql.PooledConnection
Connection Cache

JDBC Driver Types

Type 1

Type 2

Type 4
Pure
Java

Type 3

Connecting .. Driver

Type 1..4 Driver

Type 4: Pure Java Driver

DB/2 does not provide Type 4 driver

Interface java.sql.Driver

Explicitly

Class.forName(org.postgresql.Driver");
new oracle.jdbc.OracleDriver();

System Properties

jdbc.drivers=
oracle.jdbc.OracleDriver:org.postgresql.Driver

Connecting DriverManager

Basic service for managing a set of JDBC

drivers
JDBC Drivers register with the DriverManager
Source

Connection c = DriverManager.getConnection
("jdbc:oracle:thin:@dev:1521:dev1");

Connection Strings

jdbc:oracle:oci8:@
-- (#1) Bequeath
jdbc:oracle:oci8:@ora
-- (#3) Net8
jdbc:oracle:thin:@dev:1521:ora
-- (#2) Type 4
jdbc:postgresql://dev:5432/compiere

Connecting .. Data Source

Interface javax.sql.DataSource

Server Source

Alternative to DriverManager
Based on JNDI
DataSource ds = new OracleDataSource();
Context ctx = new Initial Context();
ctx.bind (jdbc/myDB, ds);

Client Source

Context ctx = new Initial Context();

DataSource ds =
(DataSource)ctx.lookup(jdbc/myDB);
Connection c = ds.getConnection();

Connection .. Pool

Optional extension
One Physical Connection
Multiple Logical Connection

Connection.close() makes it available

Eliminates overhead of creating physical

connection:

Connection c = pool.getConnection();
Statement stmt = c.createStatement(..);
stmt.close();
c.close();

Connection .. Cache

Optional Extension no standard

(!)
Multiple Physical Connections
We wrote own Connection Cache

Connection not closed in code

Multi-Thread

Most JDBC implementations

synchronize on Connection

You need multiple connections for

multi-threaded applications

Compiere uses

2 read-only never closed Cached

Connections (w. health check)
1 update never closed Connection

Test Results (10 Threads)

Test Results (10 Threads)

Multiple Pre-Created Connections

No xxx.getConnection()
Yield surprise

while (rs.next)
yield();
(High) switching costs
Database has result in buffer

just transmitting

Connection Cache

Optimum Cache Size 2-4

Test Results

Oracle

Use Thin Driver

Fastest Driver Bequeath

jdbc:oracle:thin:@dev:1521:ora
"jdbc:oracle:oci8:@
20-40% faster Only works if only one instance
installed
For Server Connections

Slowest Driver Net8

jdbc:oracle:oci8:@ora
10-20% slower
Net8 conversion Overhead

Connection Trx Level

Transaction Isolation Level

TRANSACTION_NONE

TRANSACTION_READ_UNCOMMITTED

No: dirty reads (uncommitted data)

Get: non-repeatable, phantom reads

TRANSACTION_REPEATABLE_READ

Get: dirty, non-repeatable, phantom reads

TRANSACTION_READ_COMMITTED

Not supported

No: dirty, non-repeatable (data changed) reads

Get: phantom reads

TRANSACTION_SERIALIZABLE

No: dirty reads, non-repeatable, phantom (WHERE

changed) reads

Connection

You probably want to get the latest data

conn.setTransactionIsolation
(Connection.TRANSACTION_READ_COMMITTED
);
Data updated by others & Triggers
Set explicitly

Connection is Interface
Default set by Vendor Implementation

Consider

Read Only Hint to Driver/Database

conn.setReadOnly(true);

Querying

Statement conn.createStatement()

ResultSet.TYPE_

ResultSet.CONCUR_

Database dependent statement caching

UPDATE EMPLOYEES SET SALARY = ? WHERE ID = ?
pstmt.setDataType (pos, value);

Callable Statement conn.prepareCall()

READ_ONLY or _UPDATABLE

Prepared Statement conn.prepareStatement()

FORWARD_ONLY, SCROLL_INSENSITIVE, SCROLL_SENSITIVE

{?= call myProcedure (?,?)}

Batch [implementation not mandatory]

stmt.addBatch(Insert ..);
stmt.executeBatch();

setDataType Conversion

Result - ResultSet

Created by stmt.executeQuery()
Fetch Size

Non updateable RS

rs.setFetchSize(int);
stmt.setFetchSize(int);
VendorConnection.setFetchSize(int);
Driver hint
Forward only & read just once

Data Retrieval

getDataType (int pos);

getDataType (String columnName);
wasNull();

No way to
Cancel a
Query

getDataSet Conversion

ResultSet

Navigate

Update

moveToInsertRow();
updateDataType
insertRow(); or cancelRowUpdates();
moveToCurrentRow();

Delete

updateDataType (pos, data);

updateDataType (columnName, data);
updateRow(); or cancelRowUpdates();

Insert

absolute(int), relative(int) first(), last(), next(),

previous()

deleteRow();

Refresh

refreshRow();

Result - RowSet

JavaBean - Extends ResultSet

Types Implementation Optional

Cached serialized - disconnected - X-JVM Synchronize

Client
JDBC
RMI no
Web

Code Example

JDBC

RowSet rowset = new OracleCachedRowSet();

rowset.setUrl (CONNECTION);
rowset.setUsername (UID);
rowset.setPassword (PWD);
rowset.setFetchSize (fetchSize);
rowset.setCommand (SQL_STATEMENT);
rowset.execute ();

Meta Data Interfaces

DatabaseMetaData

ParameterMetaData

Driver JDBC compatibility

Data Dictionary Queries
Prepared Statement

ResultSetMetaData

Get Column Name, Data Type of

SELECT * query

Connection - Query Retrieve

First Connection, (Query) & Retrieval took significantly longer 4,800

Fetch Size

Compiere JDBC Use

Async worker to load data

Select

Prepared Statement - Read-Only, Forward Only

Statement & ResultSet (cursor) closed after read

Insert
ResultSet from Prepared Statement (WHERE 1=2)

rs.moveToInsertRow(); rs.insertRow();

Re-read from Database using PK (trigger, ..) deterministic,

Update
optimistic

Re-read using ROWID or OID (for update)

locking

Check, if data* not changed

rs.updateXX(); rs.updateRow();
or rs.cancelRowUpdates();

Re-read from Database using PK

Performance Hints

Qualify Columns in SELECT Clause

Use position of Column not Name

WHERE clause
Scrollable/Updateable/

Use Locks carefully

rs.getString(1);

Give as much info as possible

Gets only data needed

and release them explicitly

Close Statements & ResultSets closes DB cursors

rs.close(); stmt.close();

Performance Hints (2)

Use Batches where possible

Connection.setAutoCommit (false)
Database Meta-Data Queries are slow

Consider Use of

Quantify arguments, search patterns

Statement.setFetchSize (int no)
Statement.setMaxRows (int no)
Statement.setQueryTimeout (int seconds)
Statement.setMaxFieldSize (int byte)
Hidden unique ID: RowID, OID
RowSet for complex mass update

Use Prepared Statements (*)

Resources (Selection)

JDBC

http://java.sun.com/products/jdbc

Several courses and introductions

http://www.jguru.com/faq/JDBC
http://www.javaskyline.com/learnjdbc.
html

JDO

http://access1.sun.com/jdo
http://www.jdocentral.com

Thanks

You can download the presentation

If you have questions

http://www.compiere.org/download/
jorg.janke@compiere.org
(203) 445-8182

Have a look

http://www.compiere.com/consulting.h
tml