Open-ESB tutorials

Schema and Encoding:

Import a CSV file
Paul Perez / Bruno Sinkovic

Copyright Pymma Consulting (2009)

Contents
Preamble ........................................................................................................................................................................3
Objectives ......................................................................................................................................................................3
Create Country Table .....................................................................................................................................................3
Create WSDL with DB Binding .......................................................................................................................................4
Encoding step 1: Create a schema to import CSV file ....................................................................................................7
A look on the legacy data ..............................................................................................................................................7
Encoding step 2 Create an encoding for one record only ..............................................................................................8
Preamble ....................................................................................................................................................................8
Create a file with one record......................................................................................................................................8
Add encoding in the schema ......................................................................................................................................8
Encoding step 3 Test the encoding ..............................................................................................................................11
Create a WSDL to import the DATA .............................................................................................................................12
Design the BPEL ...........................................................................................................................................................13
Mapping design .......................................................................................................................................................15
Design A Composite Application ..................................................................................................................................16
Add the encoding in the Binding component ...........................................................................................................19
Deployment and test ...................................................................................................................................................21
Application test ........................................................................................................................................................22
Conclusion ....................................................................................................................................................................23

2 Pymma Consulting Open-ESB tutorial. Schema and Encoding

Preamble
This is the version 2 of the document. Further to the Open-ESB community feedback, we changed some parts of the
paper, add some remarks and comment. Thanks your feedback and especially to Lixin Tang from Sun Microsystems
(Lixin.Tang@Sun.COM).

Objectives
The objective of this paper is to show you a simple way to import in Open-ESB non XML data. To do it we use a very
powerful feature of the schema editor: the Encoding. The case in this paper is simple but gives a good idea of the
process to use to import legacy data.

We hope you will enjoy this exercise.

Create Country Table

Before starting the main project, we want to import a country list found in a CSV file and put the list in a database.

To do it, we have to create the table Countries in a database.

First open the windows “Services“ and expand the database node

Select the database sample, right click then select connect.

After a while, the connection icon changes, expand the travel DB node Tables.

Pymma Consulting Open-ESB tutorial. Schema and Encoding 3

Click right on the Tables node and select the menu item “Create a Table”. Create the table and the column as
follows.

Click Ok and check if the table has been created in the DB.

Create WSDL with DB Binding

Create a new BPEL project and name it “ImportCountries”

Create a new WSDL in the process folder. Name it “CountriesDBServices”.

Select the screen option as follows

4 Pymma Consulting Open-ESB tutorial. Schema and Encoding

At Pymma, we are used to promote abstract instead concrete WSDL documents. Unfortunately, creating an
abstract database binding at the CASA level generates some bugs. It is the reason why we create a concrete WSDL
at this level.

Click on the Next Button

In the next screen select the sample database

In the next screen select the table “COUNTRIES”

Pymma Consulting Open-ESB tutorial. Schema and Encoding 5

Click on the Next button twice then on the Finish button

Note on the project window 2 elements have been added: the WSDL you create and the XSD document COUNTRIES
as well.

In the WSDL document, many operations have been created in the port type. These are the classical operations of
maintenance for a database.

6 Pymma Consulting Open-ESB tutorial. Schema and Encoding

Encoding step 1: Create a schema to import CSV file
Create in the process folder a new schema and name it “CountryCSV”
Create a Schema as follow: Please understand the XSD code and create the document with the graphical editor.

<?xml version="1.0" encoding="UTF-8"?>

<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
targetNamespace="http://xml.netbeans.org/schema/CountryCSV"
xmlns:tns="http://xml.netbeans.org/schema/CountryCSV"
elementFormDefault="qualified">
<xsd:complexType name="CountryCSVComplexType">
<xsd:sequence>
<xsd:element name="ISO3166_2" type="xsd:string"></xsd:element>
<xsd:element name="ISO3166_3" type="xsd:string"></xsd:element>
<xsd:element name="countryName" type="xsd:string"></xsd:element>
</xsd:sequence>
</xsd:complexType>
<xsd:element name="CountryCSV" type="tns:CountryCSVComplexType"></xsd:element>
</xsd:schema>

A look on the legacy data

Before starting to design the encoding download the file countries.csv on the web site pymma, let’s have a look on
the CSV file that contains the country list.

http://www.pymma.com/eng/IT-Tech-Papers/Pdf-folder/CSV-countries

The best way to see countries list details is to use a powerful text editor like Textpad or NotePad++. Don’t use
NotePad that is not able to display hidden characters like LF CR TAB…

List of countries

Pymma Consulting Open-ESB tutorial. Schema and Encoding 7

Remarks on the list

Header data must be removed since they will not be inserted in the database
Data are enclosed with quotes. Quote must be deleted before inserting data in the database.
Records are limited with CR LF (windows format)

Encoding step 2 Create an encoding for one record only

Preamble
In this step, we define an encoding for one record. However, it is possible to define an encoding for the entire file as
well. What is the difference between the two options?

First: when defining a encoding for a entire file, your encoding has to take into account the delimiter between the
countries (\r\n in our case) then you have additional level in your encoding. It makes easier to design and
understand.

Second (the most important): If you define in your application a global schema for the entire file, Open-ESB
considers the entire file as one record. As a result, at the runtime, it load all the document in memory and you will
not be able to benefit from the parallel processing provided by the BCs (Further detail on http://wiki.open-
esb.java.net/Wiki.jsp?page=Throttling%20Configuration )

Create a file with one record

Copy the file and named the copy “CountriesOneRecord.csv”. Delete all the line except the line 2. In the new CSV file
delete CD and LF as well. Copying the file is better than create a new file since you keep the original format (Unix,
Windows, Mac…)

The new file appears as follow:

To make your life easier, copy the file in c:\temp if possible.

Add encoding in the schema

Encoding is use to convert no-XML document in XML document. Since JBI understand XML message only, encoding
is very useful to communicate with legacy systems. For more detail on encoding, please have a look on Open-ESB
web site.

In order to add encoding in your XSD document, select the node “CountryCSV.xsd”, then click right and select the
item”EncodingAdd Custom Encoding”

8 Pymma Consulting Open-ESB tutorial. Schema and Encoding

Click on the process button if a popup windows appears.

At the first glance, you will not see any difference in your XSD document. But if you expand the element countryCSV
you can note that new nodes appear (annotation & encoding).

Right click on the node “encoding” and select properties.

Pymma Consulting Open-ESB tutorial. Schema and Encoding 9

Fill properties form as follows

For more detail on encoding properties, have a look on the document “Design CustomEncoder.pdf”.

Double click on the field “Delimiter List”.

When the “Delimiter List Editor” Windows appears, click twice on the Add Delimiter button.

In the first delimiter line add the character “,” in the “Bytes” column.

In the second delimiter line select quot-esc in the “Type Colum” (Thanks Lixin) and add the character “ in the byte
column. The encoding will suppress the quotes that delimit your data. (further information on http://wiki.open-
esb.java.net/Wiki.jsp?page=Delimiters). (Tip: Type Enter to confirm the Bytes delimiter)

Click on OK Button

10 Pymma Consulting Open-ESB tutorial. Schema and Encoding

Properties have been changed. The delimiter list has 1 level.

Click Close button now and save the schema (CTRL+SHIFT +S)

Encoding step 3 Test the encoding

Right click on the Schema and select “EncodingTest”

Pymma Consulting Open-ESB tutorial. Schema and Encoding 11

Fill the form as follows

Then Click on Process.

CONGRATULATION, you generate a simple XML file.

Note, that the data are not enclosed with quotes anymore

Create a WSDL to import the DATA

We create an accurate schema in order to transform CSV data in XML, now we have to design a simple WSDL to
define an “ImportCSV” service.

In the project “ImportCountries” create a new WSDL in the process folder. Name it ImportCSV and let it abstract.

12 Pymma Consulting Open-ESB tutorial. Schema and Encoding

Click next button

Set the Operation type to “One way operation” and choose the CountryCSV element for the part 1. Click on the
button finish.

Design the BPEL

Open the BPEL document named “importCountries.bpel”. If it does not exist in your project, create it.

Pymma Consulting Open-ESB tutorial. Schema and Encoding 13

 Drag “ImportCSV.wsdl” in the left side of the BPEL editor, rename it “Import”.
Drag “CountriesDBServices.wsdl” on the right side of the BPEL editor and rename it “Database”.
Add 3 BPEL activities (Receive, Assign, Invoke) as follows.

14 Pymma Consulting Open-ESB tutorial. Schema and Encoding

 Mapping design
Remember we have to suppress the quotes to insert the countries in the database. We use the Mapper to do it.

For the element “countryName” use first “SubString After” function then “Substring Before” to suppress the quotes.

Validate and save the BPEL. Note you have warning messages.

Pymma Consulting Open-ESB tutorial. Schema and Encoding 15

These messages are not important. They indicate that you don’t use the message sent back by the Database service
in your process and the type definitions don’t match perfectly.

Clean and Build the project.

Design A Composite Application

Create a new Composite Application project and name it “importCountryCompApp”.

16 Pymma Consulting Open-ESB tutorial. Schema and Encoding

Drag the project “importCountry” and drop it on the JBI Module Lane then click on the build icon.

Open the Palette windows. Drag the File Icon and drop it on the WSDL port Lane. Connect the file port and the port
type import. A popup window appears. Fill it as follows

Pymma Consulting Open-ESB tutorial. Schema and Encoding 17

Click OK.

The Composite application is now

Save your composite application (Crtl+shift+S)

18 Pymma Consulting Open-ESB tutorial. Schema and Encoding

 Add the encoding in the Binding component
Go back to the project window. Select in the composite the WSDL document and open it.

In the WSDL select the file:message node and open the properties editor

Fill the form as follows

Pymma Consulting Open-ESB tutorial. Schema and Encoding 19

In the previous chapter, we design the filter for one record only. To take into account multiple Records per file, we
don’t change the encoding but we have to set the property “multiplerecordsPerFile”to true. Complete the
recordDelimiter with the value “\r\n” = CR LF. For more detail about the file message properties, please have a look
on the File Binding Documentation.

Close the Properties editor and save the project.

Note that the directory where the file countries.csv will be stored, is a property of the Service and not of the binding

20 Pymma Consulting Open-ESB tutorial. Schema and Encoding

Deployment and test
First check if the application server is started.

Then deploy the composite application.

When your build is successful, have a look on the Service assembly node in the service window.

Pymma Consulting Open-ESB tutorial. Schema and Encoding 21

3 modules have been deployed in your application server. They are grouped in a service assembly.

Application test
Open the file countries.csv and delete the first line (The CSV header)

Then save the file in the c:\temp directory. After one second, countries.csv file disappears because it is consumed
by your application.

Now have a look on the database and the table countries

22 Pymma Consulting Open-ESB tutorial. Schema and Encoding

CONGRATULATION you success

Conclusion
In this tutorial, you learn how to integrate a CSV file in your ESB application. We used the encoding capabilities of
open-ESB.

We define here a simple and efficient process:

Create the schema that define the format of the XML data you want to get after the import
Add and design the encoding for one record only
In the concrete WDSL set the encoding of the binding component.

Keep in mind that the example provided in this document is simple. In the field, when working with legacy flat file,
you can face very tricky legacy files. Then, designing an accurate encoding can take more time than plan by the
project management.

At Pymma, we found the encoding tools not obvious and the documentation not helpful. However it is an essential
tool we have to use and to be conversant with.

We hope this tutorial will be helpful. Feel free to send your feedback at: contact@pymma.com

Pymma Consulting Open-ESB tutorial. Schema and Encoding 23