Professional Documents
Culture Documents
The Author Herv is an IT professional with 14 years industry experience over 11 years with Oracle products and solutions. Herv is working as a Technical Manager for ASSIST and has been working for major companies and projects throughout Europe, Hong Kong and Australia. He has extensive experience with Oracle and the Oracle Applications product range together with significant involvement in the management of the development of custom applications. Herv has been involved in all aspects of technical development including systems analysis, systems design, programming, quality assurance, documentation and project management. Herv has been extensively involved with relational database systems over the past 11 years, focussing on the Oracle RBDMS and interfacing other software technologies with relational databases. Herv has worked with Oracle Applications for the last 5 years. ASSIST Pty Ltd ASSIST is the premier Oracle applications implementer in Australia and one of the most experienced consultancies providing functional and technical support for Oracle products. ASSIST designs, develops, integrates and implements sophisticated business process solutions incorporating best business practice and methods. ASSIST consultants are highly motivated professionals whose primary objective is to ensure the most effective implementation of Oracle applications and Oracle technologies. They have extensive experience in project management, system implementation, applications development, DBA services as well as customised training for financial and general business systems. The technical and business knowledge within ASSIST is constantly updated to enable us to meet the dynamics of the changing business environment. ASSIST is an authorised Oracle Reseller and an Oracle Certified Solutions Partner. Abstract What is the impact on your existing interfaces when you upgrade your system? How much time have you got to transfer all your data into your new system? Have you got any correction facilities when your interfaces and migrations are failing? A generic approach for interfaces and data migrations is probably what you need to improve productivity and save cost. This paper discusses how ASSIST implements these complex processes.
Page 1
INTRODUCTION Interfacing is an important part of Oracle Applications, whether for converting legacy data or integrating information on a regular basis from other business IT systems. To ensure data integrity and validation, Oracle has provided numerous Open Interfaces and standard Application Programming Interfaces (APIs). There are many ways to implementing an interface, but a standard, consistent and flexible approach and one that is easy to maintain and use is desirable. This paper will discuss an automated approach to implementing an interface for Oracle Applications. Interfaces are used to import external data into another system. They can be used to provide a continuous interface between two systems, or are used to convert legacy data into the new system (i.e. Data Conversion). Oracle provides a suite of interface applications called Open Interfaces that should be used to import external system data into the Oracle Applications. These Open Interfaces are flexible and powerful tools that allow the capture of data from another system, define format conversions and direct this data into Oracle Applications. They provide validation of data, data integrity checking and reporting of results, error messages and archiving. Oracle does not support the import of data directly to Applications and anyone planning to implement an interface or planning to import legacy data should use these Open Interfaces.
Page 2
TYPICAL METHODS FOR INTERFACING TO ORACLE APPLICATIONS Although the Interface can vary, the following approach is typical to Oracle Application: 1. Extract data from the external system and output to a flat file. 2. Transfer the file to the new platform. There are many transfer tools. FTP is a common tool used across platforms. 3. Load the flat file into a temporary Oracle table using the tool SQL*Loader. (You can skip this step and load directly into the Oracle Interface table, however this method provides some advantages, such as better error handling procedures and better control of the data loaded into the Interface tables.) 4. Run a data verification on the temporary table. Even though the open interface produces an error report, it is often advisable to do your own checking. Some interfaces reject the whole batch when one piece of information is wrong, and for time critical systems the organisation may prefer to process what is correct and pass the incorrect records back to the external system with error messages for further refining. This step is optional. Usually SQL*Plus or PL/SQL is used to validate the records. 5. Load the Temporary table to the standard Oracle Interface Tables. Usually SQL*Plus or PL/SQL is used to load the records to the interface tables. Additional data manipulation can take place in this step, for example assigning the unique Ids or translating date formats, or using mathematics to calculate some fields, or joining tables to populate certain interface fields. 6. Initiate the Oracle Import, for example, run the Oracle AUTOINVOICE for the import to Oracle Accounts Receivables or Journal Import into General Ledger.
Page 3
Exec: ARCAAA
AutoInvoice
Host
ARCAAA.prog
Check Files, Check Environment, SQL*Loader, Validations, Log Messages, Transfer to Oracle
ARCAAA.ctl
Report A
ARCAAA.pls
ConcPrg: ARCBBB
Exec: ARCBBB
LockBox
Host
ARCBBB.prog
Check Files, Check Environment, SQL*Loader, Validations, Log Messages, Transfer to Oracle
ARCBBB.ctl
ARCBBB.pls
Report B
ConcPrg: CECCCC
Exec: CECCCC
SQL*Loader
CECCCC.ctl
Oracle Applications
Log D
ConcPrg: GLCDDD
Exec: GLCDDD
GL Interface
Host
GLCDDD.prog
GLCDDD.ctl
ConcPrg: APCEEE
Exec: APCEEE
Suppliers Load
Host
APCEEE.prog
APCEEE.ctl
APCEEE.pls
Report E
Log F
Log E
ConcPrg: BATCH
Exec: BATCH
Batch Process
SQL
BATCH.sql
Generic Modules
Page 4
The main problems or issues with this typical model of interfacing are:
No reusable code, instead a lot of duplication; Different styles of programming; A lot of different documentation and AOL set up instructions; Installations and migration into different environments is laborious; Support and maintenance difficult; Too many custom objects, your DBA will be worried! Not flexible and inconsistent design; Probably a lot of hard coded values; Effort required to test and re-test the same functions and Effort required to implement common requests.
Page 5
AUTOMATING THE WHOLE PROCESS GENERIC MODULES All interfaces and data migration processes follow the same logic and always require the same type of validations. The only difference is which Open Interface you have to populate or if no Open Interface is available, which API or application tables you have to maintain. What are the common tasks required for an interface? Receive and check existence of the input data file (name and directory); Check Unix environment (i.e. Unix directories and permissions); What to do with input data file? (Delete, keep, compress, archive); Where do you want to archive your data file? Display the input parameters and steps information in the log file; Perform data validations (field lengths and formats, derive ids, check existing of suppliers, customers, GL accounts, and so on); Produce a report of invalid records with meaningful messages; Transfer validated data into Oracle Open Interface tables; Run Oracle standard Open Interface process.
And finally
The 3 main tasks can be summarised in the three following steps: Load data file Perform Validations Transfer Data SQL*Loader PL/SQL PL/SQL
How do I move from the typical model? So, you already have interfaces implemented, but you would like to make them generic. So how do you go about this in theory? Identify all possible steps at the lowest level; Create all corresponding functions; Create a master program that calls all the required functions based on a set of parameters.
How do I implement the new model? And what are the practical steps involved in actually implementing the new model? Identical Concurrent Programs for all interfaces, only with different parameter values; Only one Concurrent Executable; Your parameters decide what function/program to call and what needs to be displayed in your log files.
The following diagrams provide a high level of a common interface tasks. This is followed by a detailed architecture diagram of the generic interface, highlighting all of the common modules that can be re-used.
OAUG Conference Sydney 2001 Page 6
CONTEXT
Data File
OPTIONAL
Header Table
Lines Table
API
Open Interfaces
Page 7
MODULAR APPROACH
All duplicated Concurrent Programs
ConcPrg: ARCAAA
AutoInvoice (CSV File)
Log Parameters
Check Files
Check Environment
Generic Modules
ConcPrg: ARCBBB
LockBox (CSV File)
Call Archive
ConcPrg: CECCCC
Bank Stmt (FL file)
Exec: XXINT_SHELL
Host
XXINT_SHELL.prog
ConcPrg: GLCDDD
GL Interface (CSV File)
Call SQL*Loader
Call PL/SQL
Call SQL
ConcPrg: APCEEE
Suppliers Load (CSV File)
CECCCC.ctl
CECCCC.pls
BATCH.sql
ConcPrg: BATCH
Batch Process
Oracle Applications
Page 8
GENERIC INTERFACE COMPONENTS The broad components of the Generic Interface solution are: Master Script and Generic Functions; Common Custom Interface Tables; Unique Error Report; Concurrent Program.
All of these components are generic, in that they do not change irrespective of the interface being developed. The parts that require changing relate to specific validations unique to the nature of each interface.
Page 9
The Master Script and Generic Functions One Generic Interface Master Script is used to control all interfaces. This Master Script does not need to be changed at all. By implementing your interfaces using this approach, you will gain the following advantages: A consistent look and feel throughout all interfaces; Standardisation will reduce training time; Standardisation will make maintenance easier; You will have better error handling features; You can feedback logic throughout the controlling interface logic; All of the identified issues and problems from the typical model are addressed.
Make it simple, consistent and flexible. Change the content of output files and log files only with different parameters values (calling generic functions) and not by changing your code. Using the generic model, all the infrastructure for new interfaces will already exists, just concentrate on the important business rules and validations (your PL/SQL programs). You can then just call you custom procedures from your template. Using this generic solution and with automatic installation scripts, you can implement a fully tested solution that will load your file, perform some pre-defined validations and produce an error report in less than 1 hour. This allows you to concentrate on the important business rules and requirements that you need to implement in your interface or data migration processes.
Page 10
The Generic Custom Tables The Generic Interface is implemented using 3 custom tables, as follows: Batch Header information contains summary of interface results; Batch Lines information contains all records from data file; Batch Exceptions contains all messages and errors from validations.
You can use the same custom tables for all your interfaces and data migrations. You dont have to change the table definitions at all.
Batch Headers
Batch Exceptions
Batch Lines
Page 11
Concurrent Programs Main Parameters Only one concurrent program definition is required for any interface. The concurrent program definition deals with standard issues such as, file name and location, archiving, etc. It will also cater for unique issues such as parameters that are specific to a particular interface. Scenario Example for interface (output log file example 1 and 2) Questions Do you want to log system and user parameters? What is the file name? Where is your data file? Do you want to compress and archive your file? What is your archive function? Where do you want to archive this file? Which rollback segment do you want to use? Do you want to see all the details from SQL*Loader? What are your 3 main steps? 1 - SQL*Loader 2 - PL/SQL Validation 3 - PL/SQL Transfer Set Script1 to xxcall_loader Set Program1to XXTEMPL_CSV.ctl Set Script2 to xxcall_plsql Set Program2 to XXTEMPL_PLSQL.pls Set Script3 to xxcall_plsql Set Program3 to XXTRANSFER.pls What parameter to set? Set Log Header to xxcall_header Set Log User Header to xxcall_header_user Set Parameter File Name to: HD.csv Set Parameter File Directory to /tmp Set Compress and Archive Flag parameters to Yes Set Archive Script to xxcall_archive Set the Archive Directory parameter to /usr/interface/archive Set the Rollback Segment parameter to BIG_RB Set Log/Discard/Bad flags to Yes
Scenario example for PL/SQL Batch Process (output log file example 3) Questions What parameter to set?
Do you only want to run a PL/SQL Set Script1 to xxcall_plsql process with no additional information in Set Program1 to BATCH.pls your log file? Note: This is only a subset of the parameters available, additional parameters are for internal use and are not specified here. These parameters insure that no hard coded values will be used for specific sites.
Page 12
WHAT CAN BE ADDED TO MAKE IT MORE GENERIC? Data Validation and Mapping Form The base Generic Interface model can be enhanced to include a number of value added features. To streamline the validation and mapping processes when setting up or changing an interface, a form can be used to facilitate the process. The form would define and store mapping and validations. Generic PL/SQL packages can then control the application of this mapping and validation rules.
Page 13
Pre-defined spreadsheets A pre-defined spreadsheet can be used to format and load data as part of the overall interface or data migration process. This gives users the ability to create and load interface files without the need for programming support.
Page 14
Correction Facility One you have loaded your file and you realise that numerous records are incorrect and rejected. What usually happens is you go back to you external system and require a corrected file and run your interface again. How long is this going to take? Why not use a correction facility that will allow you to correct your invalid records on-line and then restart your interface? Because, based on a common set of custom tables, one single custom form can be defined allowing users to change the loaded data. This would mainly be applicable for one off data migration and it is not always applicable for on-going interfaces. Let users change their data, manually apply some data cleansing instead of trying to fix the data from their legacy systems.
Page 15
WHAT ELSE COULD BE ADDED? This generic solution and templates can also be re-used for other batch type processes like running overnight PL/SQL processes, producing CSV or XML files, and can be used by Unix Administrators and/or DBAs to perform system checks or tasks. System Administration functions Simple SQL script producing statistics against your Concurrent Managers. Create function to purge/delete data loaded into the custom Batch Lines table. This is just a simple PL/SQL process that you can call from your generic Concurrent Program. Unix and DBA Administration utilities Disk Space, Database free space, ... Development Utilities List of custom programs and their SCCS versions from your custom applications ... Data Warehouse extract files Data extract into CSV file or XML format for your Data Warehouse system ... And much more!
Page 16
CONCLUSION Creating a Shell script that controls each interface process takes a lot of development time. The cost of developing an Interface in this way is far greater than writing a few programs for loading and validating the external data and then managing the Import through the supplied Oracle Open Interface methods. However, the time spent developing a Generic Interface is worthwhile. Error handling will be more efficient, training time will be shorter and a standard method of running interfaces will be achieved. You can move from a complex implementation that is inflexible and inconsistent to a very simple and flexible model. This will not only cater for your interfaces and data migration but will also allow you to define additional functions or utilities still using the same design and model. If you improve or change one library, all interfaces/batch processes can benefit from these immediately. Make a change once and dont spent time changing all your programs for the same request. The benefits and features of a Generic Interface can be summarise as follows: Make it simple, flexible and consistent; Re-usable modules, write it once; Design valid for Interfaces, Data Migrations and any other type of processes; Improve your support and maintenance; Your technical team is trained and familiar with a common approach; Only concentrate on the important business rules of your processes; Simplify your documentation, installations and migrations; No Hard Coded values, work in different environments with different set up; Developed, tested and implemented in Production for 10.7, R11 and R11i sites; Everything can be done remotely; Save time, cost and improve quality.
Page 17
Verifying Run Environment ------------------------Working environment validated successfully. Check Data File --------------Data file verified successfully. SQL*Loader Processing ---------------------
xxcall_check_file function
xxcall_loader function
SQL*Loader: Release 8.0.6.1.0 - Production on Wed Oct 3 11:51:36 2001 (c) Copyright 1999 Oracle Corporation. All rights reserved.
Commit point reached - logical record count 2 Commit point reached - logical record count 3 SQL*Loader completed successfully. PLSQL Validation Processing --------------------------PL/SQL Validation completed successfully. Process Status -------------Process completed successfully with no Errors. xxcall_plsql function
xxcall_footer function
+---------------------------------------------------------------------------+ Executing request completion options... Printing output file. Request ID : 630276 Number of copies : 0 Printer : flint Finished executing request completion options. +---------------------------------------------------------------------------+ Concurrent request completed successfully Current system time is 03-OCT-2001 11:51:37 +---------------------------------------------------------------------------+
OAUG Conference Sydney 2001 Page 18
User Parameters --------------File Name File Directory Compress File Archive Flag Archive Directory Archive Script Step 1 Step 2 Delete Flag Bad File Flag Discard Flag
Verifying Run Environment ------------------------Working environment validated successfully. Check Data File --------------Data file verified successfully. SQL*Loader Processing ---------------------
xxcall_check_file function
xxcall_loader function
SQL*Loader: Release 8.0.6.1.0 - Production on Tue Oct 9 10:10:39 2001 (c) Copyright 1999 Oracle Corporation. All rights reserved.
Commit point reached - logical record count 3 SQL*LOADER LOG FILE SQL*Loader: Release 8.0.6.1.0 - Production on Tue Oct 9 10:10:39 2001 (c) Copyright 1999 Oracle Corporation. All rights reserved.
Control File: /apps/VIS/cust/1.0.0/bin/XXTEMPL_LOADER.631230 Data File: /apps/VIS/cust/1.0.0/data/HD.dat Bad File: /apps/VIS/cust/1.0.0/log/XXTEMPL_LOADER.bad Discard File: none specified (Allow all discards) Number to load: Number to skip: Errors allowed: Bind array: Continuation: Path used: ALL 0 50 64 rows, maximum of 65536 bytes none specified Conventional
Table "CUST"."XXINT2_BATCH_LINES", loaded from every logical record. Insert option in effect for this table: APPEND TRAILING NULLCOLS option in effect Column Name Position Len Term Encl Datatype ------------------------------ ---------- ----- ---- ---- --------------------PVAL001 FIRST * , O(") CHARACTER PVAL002 NEXT * , O(") CHARACTER PVAL003 NEXT * , O(") CHARACTER PVAL004 NEXT * , O(") CHARACTER PVAL005 NEXT * , O(") CHARACTER PVAL006 NEXT * , O(") CHARACTER PVAL007 NEXT * , O(") CHARACTER PVAL008 NEXT * , O(") CHARACTER PVAL009 NEXT * , O(") CHARACTER PVAL010 NEXT * , O(") CHARACTER PVAL011 NEXT * , O(") CHARACTER PVAL012 NEXT * , O(") CHARACTER PVAL013 NEXT * , O(") CHARACTER PVAL014 NEXT * , O(") CHARACTER PVAL015 NEXT * , O(") CHARACTER PVAL016 NEXT * , O(") CHARACTER PVAL017 NEXT * , O(") CHARACTER PVAL018 NEXT * , O(") CHARACTER PVAL019 NEXT * , O(") CHARACTER PVAL020 NEXT * , O(") CHARACTER PVAL021 NEXT * , O(") CHARACTER PVAL022 NEXT * , O(") CHARACTER PVAL023 NEXT * , O(") CHARACTER PVAL024 NEXT * , O(") CHARACTER PVAL025 NEXT * , O(") CHARACTER BATCH_NAME REQUEST_ID LINE_STATUS LAST_UPDATE_LOGIN LAST_UPDATED_BY CREATED_BY RECORD_NUMBER LAST_UPDATE_DATE CREATION_DATE CONSTANT CONSTANT CONSTANT CONSTANT CONSTANT CONSTANT RECNUM SYSDATE SYSDATE 'BATCH NAME' '631230' 'V' '1001250' '1001250' '1001250'
Table "CUST"."XXINT2_BATCH_LINES":
OAUG Conference Sydney 2001 Page 20
3 0 0 0
successfully loaded. not loaded due to data errors. not loaded because all WHEN clauses were failed. not loaded because all fields were null. 51734 bytes(2 rows) 0 bytes
Space allocated for bind array: Space allocated for memory besides bind array: Total Total Total Total logical logical logical logical records records records records skipped: read: rejected: discarded: 0 3 0 0
Run began on Tue Oct 09 10:10:39 2001 Run ended on Tue Oct 09 10:10:40 2001 Elapsed time was: CPU time was: 00:00:00.67 00:00:00.07
SQL*Loader completed successfully. PLSQL Validation Processing --------------------------PL/SQL Validation completed successfully. Data File Archiving/Clean Up ---------------------------Data file archiving/clean up completed successfully. Process Status -------------Process completed successfully with no Errors. xxcall_plsql function
xxcall_archive function
xxcall_footer function
+---------------------------------------------------------------------------+ Executing request completion options... Printing output file. Request ID : 631230 Number of copies : 0 Printer : flint Finished executing request completion options. +---------------------------------------------------------------------------+ Concurrent request completed successfully Current system time is 09-OCT-2001 10:10:41 +---------------------------------------------------------------------------+
Page 21
+---------------------------------------------------------------------------+ Executing request completion options... Printing output file. Request ID : 630278 Number of copies : 0 Printer : flint
+---------------------------------------------------------------------------+ Concurrent request completed successfully Current system time is 03-OCT-2001 12:03:39 +---------------------------------------------------------------------------+
Page 22
3 3 3 14 3 26 25 27
15:07
XXINT_SHELL
->
25 25 25 25 25 25
CUSTOM PROGRAM SCCS VERSIONS ---------------------------Executable files in bin ----------------------/apps/VIS/cust/1.0.0/bin/XXINT_SHELL.prog: SCCS: (#) XXINT_SHELL.prog 1.22@(#) /apps/VIS/cust/1.0.0/bin/XXTEMPL_LOADER.ctl: SCCS: (#) XXTEMPL_LOADER.ctl 1.23@(#) +---------------------------------------------------------------------------+ Executing request completion options... Printing output file. Request ID : 630279 Number of copies : 0 Printer : flint Finished executing request completion options. +---------------------------------------------------------------------------+ Concurrent request completed successfully Current system time is 03-OCT-2001 12:07:40 +---------------------------------------------------------------------------+
Page 23
Page 24
+---------------------------------------------------------------------------+ Executing request completion options... Printing output file. Request ID : 630279 Number of copies : 0 Printer : flint Finished executing request completion options. +---------------------------------------------------------------------------+ Concurrent request completed successfully Current system time is 03-OCT-2001 12:07:40 +---------------------------------------------------------------------------+
Page 25