Professional Documents
Culture Documents
Table of Contents
Document History....................................................................................................................................... 4
1. Introduction......................................................................................................................................... 5
2. COB / EOD issues.............................................................................................................................. 5
2.1 Some basic points on COB / EOD.................................................................................................... 5
Single Threaded /Multi-threaded job................................................................................................... 5
PROCESS.STATUS or JOB.STATUS................................................................................................ 6
Flow chart of a multi-threaded job....................................................................................................... 7
2.2 Basic Information required in case of COB/EOD Crash...................................................................8
Irrespective of T24 release.................................................................................................................. 8
Information Specific to T24/Globus release:.......................................................................................8
3. LOCKS & Process information.......................................................................................................... 11
3.1 LOCKS..................................................................................................................................... 12
SHOW-ITEM-LOCKS........................................................................................................................ 12
jRLA.................................................................................................................................................. 12
jDLS.................................................................................................................................................. 15
LOCKS in UNIVERSE....................................................................................................................... 16
3.2 Process Information................................................................................................................. 16
WHERE............................................................................................................................................ 16
MW42 (My WHERE)......................................................................................................................... 17
ps -ef................................................................................................................................................. 18
PORT.STATUS.................................................................................................................................. 18
3.3 Unlock or kill a process................................................................................................................... 18
4. File corruption and Maintenance....................................................................................................... 19
4.1 File Corruption in Universe environment (fixtool)............................................................................19
4.2 File Resize in Universe environment (RESIZE)..............................................................................20
4.3 File Corruption in jBASE environment (jcheck)...............................................................................21
4.4 File Resize in jBASE environment (jRF).........................................................................................23
4.5 File corruption - JR files.................................................................................................................. 24
4.6 File corruption of Sensitive files – New file creation........................................................................25
Procedure to Create New file for STMT.ENTRY:...............................................................................26
Document History
Change Description /
S.N0 Version Issue Date Issued By Approved By
Comments
1.Introduction
This document is a technical guide for ASG, contains the details on the procedure to be followed
in various scenario of handling the crashes/online issues.
ASG handles all production issues which cause a system down situation for the client. It mainly
consist of
2- SYSTEM WIDE System wide processes (Interest & Charges, Revaluation etc.)
4- START of DAY Change date (Standing orders, split month end events, cash flow
maintenance etc.)
5- ON-LINE Any non-critical reports and processes which can be run after the
system has returned to on-line mode.
The stage of a batch is determined based on the field BATCH.STAGE in the BATCH
record
A COB job may be single threaded or multi-threaded job. Most of the jobs in COBs
are now multi-threaded. A COB job can be identified as a single thread or Multi-threaded by
checking the PGM.FILE (field BATCH.JOB) of the job.
Multi-threaded
A multi-threaded job will either have the value @BATCH.JOB.CONTROL in the field
BATCH.JOB or contains blank.
PROGRAM IC.COB
------------------------------------------------------------------------------
1 TYPE.............. B
4. 1 BATCH.JOB...... @BATCH.JOB.CONTROL
5 PRODUCT........... IC
7. 1 DESCRIPTION.... Performs entire IC processing
7. 2 DESCRIPTION.... for single account from GROUP.
7. 3 DESCRIPTION.... ACCOUNT
Single threaded
A single-threaded job will have the value @JOB.NAME in the field BATCH.JOB as shown
below.
.SELECT =”processing”
BATCH.BUILD.LIST
PROCESS.LIST
CONTROL.LIST = processing
PROCESS.PORTION
PROCESS.RECORD
Calling record routine
No If Keys
processed
eq 0
BASIC.LISTING
iv. If BASIC.LISTING is not available then get the output of the below commands from
the client.
The batch that had been crashed can be obtained from the above set of commands
where the PROCESS.STATUS is set to 3 or 4.
v. If the EOD is run from classic then fetch the full como of the batch where the EOD
had crashed. This can be obtained from the universe/jbase prompt:
CT BAT.<<Comp.mne>>_BATCH.NAME
Eg: BAT.BNK_LD.END.OF.DAY
vi. If the EOD is run in desktop check how many sessions are used. It can be obtained
from the field BATCH.SESSIONS in the SPF file.
27 HELD.RPT.RETENTION 5
28 CACHE.SIZE........ 500
30 PREVIOUS.RELEASE.. G11.1.02
32 BATCH.SESSIONS.... 10
35. 1 NON.2DEC.CCY... BEF
vii. If EOD is run in desktop then run MERGE.GUI.COMO from classic and send the
como. The como id will be in the format :
If need to extract the thread como (desktop) then the id of the como will be in the
form BAT.X (Where X = 1 to Number of sessions)
viii. The above 7 steps hold good for Globus releases G13.2.07 onwards.
In addition to the above steps the following information are also required.
ix. F.BATCH.STATUS record for the crashed job. The id of the F.BATCH.STATUS record
will be of the format
BATCH.NAME-JOB.NAME-JOB.POSITION
Eg: BNK/SYSTEM.END.OF.DAY3-PROC.REVAL.STATIC.CHANGE-1
The BATCH.STATUS of currently running job is obtained using the below command
LIST.ITEM F.BATCH.STATUS WITH F1 LIKE ...processing...
Sample BATCH.STATUS record for a job looks as below while the job is running
Example 1
This is a sample record while the job finishes the selection process and running in
record routine. This job does not have CONTROL.LIST
BNK/PM.EOD2-PM.ACTIVITY.ONITE-1
001ýprocessingý206
Example 2
This is a sample record for a CONTROL.LIST job. This job has
CONTROL.LIST. It has two CONTROL.LIST, 20061113 and
TO.CALL.LD.END.OF.DAY.8.
LD.END.OF.DAY-LD.END.OF.DAY.4-3
001 20061113]processing]202
002 TO.CALL.LD.END.OF.DAY.8
In this example “processing” is concatenated with the date.It means Control list is
built and the job is running for the control list 20061113
SHOULD NOT bypass the job at this stage as it has to process one more control list.
Example 3
Here the job is currently processing the control list “PROTOCOL”
x. Check the job list the job is using. This info is maintained in the F.LOCKING with the
same id format of F.BATCH.STATUS,
BATCH.NAME-JOB.NAME-JOB.POSITION. The format of the job list file name is
F.JOB.LIST.Y( In lower release, the JOB.LIST file is a FIN type, it will be in the format
FXXX.JOB.LIST.Y where XXX is the company mnemonic.
This record exists in G13 release, when a job crashes. This has to be deleted before
proceeding with the job. No need to check this record in above R4 release
LIST F.EB.EOD.ERROR WITH @ID LIKE …<<date when the crash had occurred>>…
xv. Full COMO of the crashed agent. This can be obtained from the file F.TSA.STATUS
in the field COMO.NAME
CURRENT.SERVICE. COB
NEXT.SERVICE.... STOP
LAST.MESSAGE.... _MTA/STARTUP_EB.PRINT_4_04 JUN 2005_12:37:30_Processing...1
COMO.NAME....... tSA_100_20120404_22-32-07
JOB.PROGRESS.... 3
Id format it tSA_100_20120404_22-32-07
100 – Agent Number
20120404 – Date
22-32-07- Time
xvi. The como can be fetched from the jbase prompt as:
Jsh liveJED &COMO& tSA_100_20120404_22-32-07
3.1 LOCKS
SHOW-ITEM-LOCKS
SHOW-ITEM-LOCKS is used to obtain the information on the locks in the system. This is
used for getting the locks information on a single application server. SIL should not be
requested for the clients running on multi-application server, instead we need to request for
jDLS –dVL output (as explained in the below section of this document). Otherwise, for a
multi-application server, we need to request the SHOW-ITEM-LOCKS from all the application
servers.
Below is a sample output of the SHOW-ITEM-LOCKS. The output will display the port id,
process id, file name and record which is locked.
The last column BY-PID, usually will not hold any value for a normal port which locked a
record. If there is a blocking situation, then It will hold the PID of the port which actually
locked the record. In this example, port 48700 is waiting to attain a lock on the record, which
was already locked by the pid 217540.
jRLA
jRLA is a component of jBASE which helps to control locks for T24. jRLA uses shared
memory locking model, all locks will be held in shared memory which can be accessed by
each T24/jBASE processes.
To Stop:
jBTP -k
jPML -k
jRLA -k
To start :
jRLA -ibs13120,2640,30
jPML -ib
jBTP -ib
-->jRLA -dvl
jRLA: $Revision: 3.8.4.4 $
Program at pid: 1757
Program started: Sat Oct 11 23:31:43 2003
Time last ran: Tue Oct 21 10:00:09 2003
Time last tidied up: Tue Oct 21 09:55:57 2003
Lock retries: 130554
Tidy-up time: 5 minutes
Locking mechanism: Memory semaphores based on PROCESS id
Tidy-up operations: 15
Shared memory IDs: 2055 , 8
Semaphore IDs: 23, 24
Memory: 15000 bytes used , 49536 bytes jfree
CPU Usage: 36.95 usr , 50.14 sys
Binary locks: 610 locks maximum in 61 groups of 10 locks/group
0 locks currently in use
0 is max. locks found in one group
Record locks: 3020 locks maximum in 151 groups of 20 locks/group
17 locks currently in use
2 is max. locks found in one group
Please follow the below steps to Increase jRLA locking parameters as follows
1) Logoff all jBASE userS from the server. No foreground and background the process
should be running.
export JBCRELEASEDIR=/usr/jbc
export JBCGLOBALDIR=$JBCRELEASEDIR
export PATH=$JBCRELEASEDIR/bin:$PATH
export SHLIB_PATH=$JBCRELEASEDIR/lib:/usr/ccs/lib:/usr/lib
jsh ->jRLA -K
(Note capital K)
4) Use the ipcs -a command to check whether the daemon has been properly shut down,
jsh->ipcs -a
In the output of the above command you can see lines similar to the ones given below:
Please remove the ids starting with '0x4a' using the command
ipcrm -m1573094
ipcrm -m393437
ipcrm -s2
ipcrm -s3
6) After starting the daemon check the jRLA -dvl whether the new parameters gets into effect
jDLS
jDLS locking is used for Multi-Application server architecture.
For setting up the jDLS locking mechanism, export the below variable in .profile of all
the application servers except the jDLS server
export JDLS=SERVER=10.40.10.71
jDLS -ibs13000,50
Use the next prime number for the number of groups. so in this case the shared lock
table will be 13150 locks in 263 groups with 50 locks per group
jDLS -dVL
Lock monitor at pid: 4685 - Active
Lock listener at pid: 4686 - Active
Program started: Sat Dec 13 14:00:44 2008
Time last ran: Tue Jan 20 10:57:28 2009
Time last tidied up: Tue Jan 20 10:56:18 2009
Lock retries: 11769
Tidy-up time: 5 minutes
Locking mechanism: IPC semaphores based on THREAD id
Tidy-up operations: 14013
Semaphore IDs: 163841, 196610
Memory: 13760 bytes used , 121408 bytes free
CPU Usage: 0.13 usr , 0.16 sys
LOCKS in UNIVERSE
In Universe, the locks can be monitored using the below command
>LIST.READU EVERY
Active Record Locks:
Device.... Inode.... Netnode Userno Lmode Pid Login Id Item-ID.............
3014657 245891 0 29 20 RU 55182 dataop LD010720002220011920
3014657 508504 0 29 41 RU 55182 dataop 0760095044
3014657 245891 0 29 68 RU 55182 dataop LD002830000720012210
3014657 508504 0 29 73 RU 55182 dataop 0760100889
3014657 442505 0 29 82 RU 55182 dataop PDLD0018900011
3014657 508504 0 29 137 RU 55182 dataop 0760061751
3014657 172042 0 29 139 RU 55182 dataop B04/LD.END.OF.DAY
3014657 245891 0 29 140 RU 55182 dataop LD002510000920012190
3014657 245891 0 29 172 RU 55182 dataop LD011550000120012210
3014657 508504 0 29 185 RU 55182 dataop 0760007269
213 vbalaji 999632 54 (50) 7 153 34747 4040 1487 4.74M 4.67 2 BLOCK tSA 3 (BATCH.JOB.CONTROL,1080)
245 vbalaji 405964 54 (49) 8 601 21370 3459 847 4.75M 3.18 2 tSA 3 (S.JOB.RUN,520) <BNK/TEC
262 vbalaji 590212 40 (37) 7 1 5667 7 99 4.76M 0.26 2 tSA 3 (S.JOB.RUN,407) <CO3/REP
269 vbalaji 946178 131 (118) 123 273 36925 2349 2154 5.88M 3.61 2 tSA 3 (BATCH.JOB.CONTROL,1207)
270 tskumar 581846 6 (4) 2 1 24 6 25 2.60M 0.00 1 E jsh -s jsh (jsh.b,111)
277 parthee 438716 7 (4) 8 1 27 3 20 2.97M 0.38 2 E /jbtemprel/R08.000.01.aix52.jbc505
Mw42 also gives subroutine trace which shows which subroutine and which line of code is
currently been executed by jBASE, using this functionality performance issues can be easily
monitored and analysed
It is always advised to get the output using –f option as below, which will avoid truncating the
output
--->mw42 –f
The below command from unix prompt will save the output into a file mw42.out. This will be
helpful in case of analyzing performance issues.
$mw42 -e -f 300>mw42.out
ps -ef
ps –ef command is used from unix to identify the process running in the system.
Syntax:
$ps –ef
A selective display of process can be done using the below command. For example, if we want to
display all the process in which tSA are running, we can execute the below command from unix
prompt
PORT.STATUS
A sample VOC
FBNK.ACCOUNT
001 F
002 ..\bnk.data\ac\FBNK_ACCOUNT
003 ..\bnk.dict\F_ACCOUNT]D
The T24 file name can be obtained from the physical file name using the below command on
VOC
>FILE.STAT F.ENQUIRY.LEVEL
If the file is found to be corrupted, fixtool can be used to recover the file.
$ cd ../mbdemo.data/eb
$ cp F.ENQ000 F.ENQ000.BAK
1. Do a FILE.STAT for the file you want to resize and note down the following
a) Number of records (for ex R)
b) Average number of bytes per record (say A)
c) Separation (say S)
d) Type (say T)
>FILE.STAT <File.Name>
i.e. R*A
N = ----------------------------
(S*512) - 0.1* (S-512)
>PRIME N
The output will give one highest and lowest Prime Number; now consider the Highest one let
that be P.
For the file corruption in jBASE, we can use the jcheck utility to recover the file.
--->jstat -v F.JOURNAL
If the file is corrupt, the output of jstat command would display something similar to the
following:
Follow the below steps to correct the file using jcheck utility.
1. jstat -v F.XYZ
It will tell you that the file is corrupted
Jsh---->jcheck -S ../bnk.data/eb/F.XYZ
It will say " Salvaged n records."
5. Go to the ../bnk.data/eb/F.XYZ path there you can see the file SLVG_F.XYZ
mv SVLG_F.XYZ F.XYZ
$ mv SLVG_F.JOURNAL F.JOURNAL
There is a utility in jBASE by name jrf which can be used to resize a J4 file. Follow the below
steps to resize a J4 file
1.) You can use jstat to find the physical file path of a file (from jshell prompt):
for example:
jsh-->jstat FBNK.ACCOUNT
File ../mbdemo.data/ac/FBNK.ACCOUNT
2.) Use jrf with -R option to check if a file needs resizing. '-R' option will only report and will
not resize the file.
for example:
Then do:
3.) But if jrf -Rv shows something like 'Resize from xx to nn.
where 'xx' and 'nn' are numeric value.
Then do:
This will resize the file. This can be checked with a subsequent jrf -RV.
JR files may corrupt if there is a corruption to the header of the file. If the header get
corrupts, the file will not be recognized as a JR file and so the SELECT or COUNT will
display as below for the corrupted file.
To correct the header of a JR file, we have a c program. The program should be executed on
the corrupted file, by following the below steps
1. Copy the below contents to a file named test.c. The program should be copied to the
path where the file is corrupted. <filename> should be replaced with actual file name
as below
f d=open64("./FBNK.STMT008.20120405",O_WRONLY,0);
JED test.c
#include<stdio.h>
#include<fcntl.h>
int main(void)
{
int fd=0,rc=0;
char buffer[]="JBC__HSR";
fd=open64("./filename",O_WRONLY,0);
rc=pwrite(fd,buffer,sizeof(buffer),0);
close(fd);
cc -O test test.c
$./test
We have to create a new file and copy the relevant records to it from the corrupted file.
jcheck or fixtool can apply to those file only in case the files couldn’t recover from the backup
file, that is when it fails to copy the records from the corrupted file.
Each file has its own procedure to create and copy the records.
Using Program:
This program can be used both in jBASE and Universe environment. This program
will create a new file and copy the relevant records for processing the system for the day.
Always give priority to run the routine to correct the file. Follow the manual steps only if there
is any issues in running the routine.
CORRECT.STMT.ENT
RY
Please follow the steps in the attached document to create the new file and copy the relevant
records in the event of corruption of the file in jBASE environment
Please follow the steps in the attached document to create the new file and copy the relevant
records in the event of corruption of the file in Universe environment
This program can be used both in jBASE and Universe environment. This program
will create a new file and copy the relevant records for processing the system for the day.
Always give priority to run the routine to correct the file. Follow the manual steps only if there
is any issues in running the routine.
CREATE.NEW.CATE
G.ENTRY.txt
Please follow the steps in the attached document to create the new file and copy the relevant
records in the event of corruption of the file in jBASE environment
Please follow the steps in the attached document to create the new file and copy the relevant
records in the event of corruption of the file in Universe environment
Categ_Entry_File_Cr
eation_Universe.doc
Please follow the steps in the attached document to create the new file and copy the relevant
records in the event of corruption of the file in jBASE environment
RE.CONSOL.SPEC.E
NTRY CORRUPTION PROCEDURE.doc
Please follow the steps in the attached document to create the new file and copy the relevant
records in the event of corruption of the file in jBASE environment
Please follow the steps in the attached document to create the new file and copy the relevant
records in the event of corruption of the file in Universe environment
And compile.
Choose the distribution algorithm as required (details given in the attached document) and
distribute the current FBNK.STMT.ENTRY file.
After doing so, please also copy the contents of the original FBNK.STMT.ENTRY.BAK into
the new file.
DISTRIB.zip
Below are some of the routines or utilities required frequently while handling issues in ASG
Sl.
No. Name Description Routine
For correcting BATCH records in
the event of crash due “Snap shot
PACS.BATCH.CORR.GEN
1 PACS.BATCH.CORR.GEN too old error”
1. Missing Files
2. Corrupted Files
3. Badly Sized Files
5 CHECK.T24.FILES 4. Files reaching 2 GB CHECK.T24.FILES
PROGRAM TO CHECK IF THE
CONSOL KEY IN THE
RE.STAT.LINE.CONT
EXISTS IN THE
CONSOLIDATE.ASST.LIAB TABLE
IF NOT, DELETES THE MULTI-
CONT.CHECK.ASSET.AND.L VALUE SET OF THE
6 IAB.KEYS RE.STAT.LINE.CONT RECORD CONT.CHECK.ASSET.AND.LIAB.KEYS
This is a correction program, for
missing GROUP.DATE records.
It will select
GROUP.DEBIT.INT/GROUP.CREDIT.I
NT records, loop
through each record and call
MAINT.ACC.INT.DATES.
MAINT.ACC.INT.DATES gets called
by GCI/GDI to update
7 CORR.GRP.DATE GROUP.DATE file CORR.GRP.DATE
To display the account ids in
CROSS.CHECK.GROUP.DAT JOB.LIST file, which doesn’t
8 E contains the GROUP.DATE CROSS.CHECK.GROUP.DATE
EXISTS IN THE
CONSOLIDATE.PRFT.LOSS TABLE
IF NOT, DELETES THE MULTI-
VALUE SET OF THE
9 CORRECT.CONT.PFT.KEYS RE.STAT.LINE.CONT RECORD CORRECT.CONT.PFT.KEYS
JOURNAL CORRUPTION.txt
PACS.TRACE.SUBROUTINE
Procedure.doc
19 PACS.TRACE.SUBROUTINE Routine to generate the jimi tracer
This routine is written to update
the field PRINT.STMT of the
ACCOUNT.STATEMENT.CO ACCOUNT.STATEMENT record to
20 RR 'NO' ACCOUNT.STATEMENT.CORR
To rebuild RE.STAT.LINE.CONT
28 Y.REBUILD.CONT from the backup corrupted file Y.REBUILD.CONT.txt
This routine will select the ids
saved under the &SAVEDLISTS&
record name 'INCORR.ID' and
delete from the problematic job
list.
It will also delete the ids from the
29 JOB.LIST.DELETION bulk job list file. JOB.LIST.DELETION
6.Browser Issues
Attached the document briefly explain how to analyse Browser issues. It contains
procedure to enable relevant logs. The document also contains some frequently
occurring issues and their solution.
A GUIDE FOR
ANALYZING GENERIC BROWSER ERRORS.doc
7.Oracle related.
1. Oracle errors will be logged in ORA / XMLdriver.log. We can tail the driver.log using
the below command to identify the latest error logged.
>tail –500f ORAdriver.log (The log name varies based on the database, This is upto R09
release)
>tail –500f XMLdriver.log (From R10, the name of the log is generalized as
XMLdriver.log)
2. We can have either Stubfile and Stubless Architecture. In both, the VOC will differ.
Stubfiles are in bnk.data. bnk.data and VOC should be NFS mounted to all application
server.
‘Stub’ are pointers file that hold the following details
jEDI Driver that need to be initiated/used (XMLORACLEInit /XMLDB2Init /XMLMSSQLInit
/jediInitRemote)
TABLE Name
TABLE Type (CLOB or BLOB)
VOC in Stubfile
0001 F
0002 ../bnk.data/ac/FBNK.ACCOUNT
0003 ../bnk.dict/F.ACCOUNT]D
In Stubless, VOC can be as a Table, placed in DB. Both VOC and bnk.data no
longer exists in App server.
In a ‘Stubless’ architecture, the ‘stub’ file details are moved to VOC and hence this
eliminates the NFS mount of ‘bnk.data’. Also the VOC is to table and the data resides
in DB.
VOC in Stubless
FBNK.ACCOUNT
001 TABLE
002 XMLORACLEInit
003 D_FBNK.ACC001
004 ACFBNK_ACC001
3. Jstat output on a filename will gives the underlying table name corresponds to the
file.
jstat -v F.JOB.LIST.1
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
With the Partitioning, OLAP and Data Mining options
File = default_F_JOB_LIST_1
Type = XMLORACLE, Version R09.1.91658, Table = F_JOB_LIST_1
Schema = COREORA1, Tablespace = COREORA1DATA
NLS Characterset = AL32UTF8
From sql prompt, we can execute the DESC command to check the
existence of the table in Oracle
>DESC F_JOB_LIST_1;
By the above command, index will be created for the file FBNK.ACCOUNT for
the field CUSTOMER.
We can use the below command to check if index exists for a table
5. config-XMLORACLE
Using this utility, we can see the user setting , database connection etc
jsh config-XMLORACLE
** Version 3.3.0.60149 *
* Main menu
*
*
*
****************************************************************
1. Database settings.
2. User settings.
3. Test connection
4. XDB Setup
5. Save and exit.
eg: A package SQL will be run before using the Oracle database if it is not run then this error might
occur and if the user who is trying to access the view does not have the permission.
** Oracle Error **: ORA-00942: table or view does not exist
ORA-01400 – Cannot insert NULL values eg: this error occurs when a record is written to a file
without the ID part.
ORA-01658: unable to create INITIAL extent for segment in tablespace T24DATA eg: if a data file is
not created with an AUTOEXTEND option
Error - ORA-01658: unable to create INITIAL extent for segment in tablespace T24DATA
3.2.3.56077 - 19683 - (select.b,7) - Wed Mar 26 18:03:05 - F.BATCH - EBF_BATCH - ** Error ** READ:
Record key: SYSTEM.END.OF.DAY1.UBSSNÀü0
3.2.3.56077 - 19722 - (select.b,7) - Wed Mar 26 18:03:05 - F.BATCH - EBF_BATCH - ** Error **
readLob(): Unable to read LOB descriptor
3.2.3.56077 - 19722 - (select.b,7) - Wed Mar 26 18:03:05 - ** Oracle Error **: ORA-01555: snapshot
too old: rollback segment number with name "" too small
ORA-0955: name is already used by an existing OBJECT eg: If you are trying to use an object name
which was used previously but deleted and it exists in the recycle bin, then this error occurs.
3.2.3.156450 - 229874 - (BATCH.JOB.CONTROL,437) - Thu Jun 19 13:00:15 - ** Oracle Error **: ORA-
00001: unique constraint (T24USR.F_JOB_LIST_1_PK) violated
3.2.3.156450 - 262448 - (BATCH.JOB.CONTROL,437) - Thu Jun 19 13:29:50 - F.JOB.LIST.1 -
F_JOB_LIST_1 - ** Error ** writeRecordAsBLOB: Unable to write key 1 record SingleThread
3.2.3.156450 - 262448 - (BATCH.JOB.CONTROL,437) - Thu Jun 19 13:29:50 - ** Oracle Error **: ORA-
00001: unique constraint (T24USR.F_JOB_LIST_1_PK) violated
Annotation:The above problem ORA-00001: seems to be accuring since the JOB.LIST files accesing
the single data file. Multiple process tries to write into same file with same id (No locks before
write)
3.2.3.156450 - 209078 - (S.JOB.RUN,535) - Thu Jun 19 20:48:35 - ** Oracle Error **: ORA-00060:
deadlock detected while waiting for resource
Annotation:We presume that the deadlock error is related to ORA-00001 problem.Also please send
us the Alertlog files for this problem.
3.2.3.156450 - 270590 - (jql_listcolumnar.b,115) - Sat Jun 21 14:49:26 - ** Oracle Error **: ORA-
01013: user requested cancel of current operation
3.2.3.156450 - 270590 - (jql_listcolumnar.b,115) - Sat Jun 21 14:49:26 - FAHO.STMT.ENTRY -
FAHO_STMT_ENTRY - ** Error ** READ: Record key: 147810300111740.05000164101
Annotation:This problem is due to occurs when the user try to terminate the process using "Control
+ C" Or serious network failure problems.
3.2.3.156450 - 242060 - (tSA,117) - Sun Jun 22 13:37:40 - ** Oracle Error **: ORA-12545: Connect
failed because target host or object does not exist
- 242060 - (tSA,117) - Sun Jun 22 13:37:40 - ** Error ** OPEN: Unable to initialise driver data
Annotation : This problem due to appears whenever there is a database down and the users tries to
enter into T24.
ORA-25408: can not safely replay call. Cause: The connection was lost while doing this call. It may
not be safe to replay it after failover
Annotation:It seems when the oracle was shutdown abruptly EB.PHANTOM was not able to replay
to the RDBMS.However please open a new cshd request if you want a further investigation on this.
ORA-01756: quoted string not properly terminated. Cause: A quoted string must be terminated with
a single quote mark ('). ...
3.2.3.156450 - 168230 - (select.b,7) - Fri Jun 27 02:30:45 - ** Oracle Error **: ORA-01756: quoted
string not properly terminated
3.2.3.156450 - 168230 - (select.b,7) - Fri Jun 27 02:30:45 - F.BATCH - F_BATCH - ** Error ** SELECT:
Unable to allocate statement handle
3.2.3.156450 - 168230 - (select.b,7) - Fri Jun 27 02:30:45 - ** Oracle Error **: ORA-01756: quoted
string not properly terminated
3.2.3.156450 - 168230 - (select.b,7) - Fri Jun 27 02:30:45 - F.BATCH - F_BATCH - ** Error **
SCANFILE: Failed to process statement SELECT t.RECID FROM F_BATCH t WHERE RECID LIKE
'SYSTEM.END.OF.DAY1.UB'%'
Annotations:This problem appears whenever a wild character pattern did not meet the standard
selection pattern when LIKE involved.
This error may occur if the record is corrupted.Above error message suggests that the record
LD072500332500 is corrupted on the file LMM.ACCOUNT.BALANCES.
** Error ** READ: Record key with length 563 exceeds maximum of 255 bytes
ORA-01461: can bind a LONG value only for insert into a LONG column
column alignment for Non English characters are exceeded Or not in a sync.
Due to timeout because of long running select. Solution is to index the fields.
ORA-01653 merely denotes that whenever the tablespace is unable to extend <Failed to allocate an
extent for table segment in tablespace> Or there is not enough space left in a datafiles.
ORA-01653 merely denotes that whenever the tablespace is unable to extend <Failed to
allocate an extent for table segment in tablespace> Or there is not enough space left in
a datafiles.
2. You'll have to check the size of the datafiles attached to the tablespace and check
whether they can autoextend or not..
SQL>select file_name, bytes, autoextensible, maxbytesfrom dba_data_fileswhere
tablespace_name='TABLESPACE_NAME'
Either add more datafiles to the tablespace, set the autoextensible flag or enlarge the
datafile(s).
7.4 INITRANS
In oracle, we can increase the initrans value of some frequently used tables to
improve the system performance. The tables that is advised to increase the initrans
are
F.OS.TOKEN
F.OS.TOKEN.USE
PROTOCOL
ENQUIRY.LEVEL
OS.XML.CACHE
TSA.STATUS
The advised value of initrans for the frequently used table is 100
>jstat -v F.OS.TOKEN.USE
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
File = ../bnk.data/eb/F_OS_TOKEN
Type = XMLORACLE, Version 3.4.1.72556, Table = F_OS_TOKEN
Schema = MBLT24LIVE, Tablespace = MBLT24LIVEDATAXML
NLS Characterset = AL32UTF8
Partitioned = NO
jsh mblt24 ~ -->
INI_TRANS MAX_TRANS
-------- ----------
1 255
If the INIT_TRANS value is set to 1 the ask the client to increase it to 100.
INI_TRANS MAX_TRANS
-------- ----------
100 255
Then Increase the SELBUF value which can be get from the CONFIG ALL value.
>CONFIG ALL
Configuration data for license number 406005:
User limit = 100
Expiry date= 1/1/2500
-UCI package is installed.
User limit = 100. Expiry date = 1/1/2500
-UVADM package is installed.
User limit = 100. Expiry date = 1/1/2500
Current tunable parameter settings:
MFILES = 52
T30FILE = 200
OPENCHK = 1
WIDE0 = 0x3eb00000
UVSPOOL = /usr/spool/uv
UVTEMP = /tmp
SCRMIN = 3
SCRMAX = 5
SCRSIZE = 512
QDEPTH = 16
HISTSTK = 99
QSRUNSZ = 2000
QSBRNCH = 4
QSDEPTH = 8
QSMXKEY = 32
LOGSYCNT = 0
LOGSYINT = 0
TXMEM = 32
OPTMEM = 64
SELBUF = 4
ULIMIT = 128000
FSEMNUM = 23
GSEMNUM = 97
PSEMNUM = 64
FLTABSZ = 11
GLTABSZ = 75
RLTABSZ = 75
RLOWNER = 300
PAKTIME = 300
NETTIME = 5
QBREAK = 1
VDIVDEF = 1
UVSYNC = 1
BLKMAX = 8192
PICKNULL = 0
SYNCALOC = 1
MAXRLOCK = 100
ISOMODE = 1
PKRJUST = 0
PROCACMD = 0
PROCRCMD = 0
PROCPRMT = 0
ALLOWNFS = 1
CSHDISPATCH = /usr/bin/csh
SHDISPATCH = /usr/bin/sh
DOSDISPATCH = NOT_SUPPORTED
LAYERSEL = 0
OCVDATE = 0
MODFPTRS = 1
THDR512 = 0
UDRMODE = 0
UDRBLKS = 0
MAXERRLOGENT = 100
JOINBUF = 4095
64BIT_FILES = 0
TSTIMEOUT = 60
To do the changes
Take a backup of the UVCONFIG file of UNIVERSE present in the UNIVERSE home
directory.
From Unix:
1. cd UniVerse Home Directory i.e. cd /usr/uv
2. ls –la .uvconfig To test that you are in the correct directory
3. cp .uvconfig .uvconfig.orig Copy the .uvconfig file
4. cp uvconfig uvconfig.orig Copy the uvconfig file
Change the values of the following parameters in the UVCONFIG file to the ones
specified below.
From Unix:
1. use either VI or ED to edit the uvconfig ascii file.
cd /usr/uv - From UniVerse home directory
ls –l uvconfig - To check that the correct file exists.
vi uvconfig - Edit the uvconfig file.
Ulimit values must be changed when we encounter the following type of Memory errors.
On UNIX systems, the ulimit command controls the limits on system resource, such as
process data size, process virtual memory, and process file size.
Each user can either inherit resource limits from the root user or have specific limits defined.
On Solaris systems, by default, the root user has unlimited access to these resources (for
example, unlimited).
On AIX, some limits might apply to the root user.
In order to find out the current values of the ulimit,following command must be executed.
Values can be changed to "unlimited" by including the following lines in the .profile as below.
ulimit -d unlimited
jsh->export LDR_CNTRL=MAXDATA=0x40000000
jsh->export LDR_CNTRL=MAXDATA=0x60000000
jsh->export LDR_CNTRL=MAXDATA=0x80000000
8.4 OS Tools
OS tools are a valuable tool in determining the type of limitation that is being applied, and
from there
The main tool for AIX is nmon. Before a test is run the following command should be run:
e.g.
The above command would create a file in /tmp call eod1_nmon_file and would give the test
a title of eod1. This command would record stats every 30 seconds for 300 minutes. Once
the test is complete (you do not have to wait for the time period to finish) the output needs
sorting. Still in AIX use the following command:
This command sorts the data into a .csv file. This file should then be ftp’d to a windows
machine, and the “nmon analyzer” spreadsheet used to analyse the data.
NB the “nmon analyzer” is an IBM tool, though it is unsupported. The T & R department can
supply copies of this spreadsheet if required.
Below are some examples of the data that is presented by this tool:
CPU usage:
Disk activity:
Network utilisation:
9 Performance issues
9.1 SEAT
The below document briefs on how to analyse performance issues using SEAT
SEAT Analysis.doc
9.2 TEC
The below document details on how to make use of TEC to analyse performance
issues.
T24EnterpriseConsol
e.doc
10 Installation Procedure
T24 Upgrades.pdf
11 Miscellaneous
jdiag –dv
Found : '/usr/jbc/jbase4.1/lib/libjbase.so'
Found : '/usr/jbc/jbase4.1/lib/libjbaseutil.so'
Found : '/usr/jbc/jbase4.1/lib/libjcon.so'
Found : '/usr/jbc/jbase4.1/lib/libjsub.so'
Found : '/usr/jbc/jbase4.1/lib/libjlic.so'
Object path (JBCOBJECTLIST) :
'/globus/dev/bnk/bnk.run/globuspatchlib:/globus/dev/bnk/bnk.run/globuslib:/globus/dev/bnk/bnk.run/lib'
WARNING: Cannot access Object path '/globus/dev/bnk/bnk.run/globuspatchlib', error 2
Program dir (JBCDEV_BIN) : '/globus/dev/bnk/bnk.run/bin'
U:\>c:
C:\>ftp
ftp> exit
Invalid command.
ftp> quit
221 Goodbye.
C:\>ftp
ftp> open 10.92.5.5
Connected to 10.92.5.5.
220 ibmrs6000 FTP server (Version 4.1 Fri Apr 6 03:07:22 CDT 2001) ready.
User (10.92.5.5:(none)): intstj1
331 Password required for intstj1.
Password:
230 User intstj1 logged in.
ftp> cd RICK.BP
250 CWD command successful.
ftp> BI
200 Type set to I.
ftp> put F.JOURNAL
200 PORT command successful.
150 Opening data connection for F.JOURNAL.
226 Transfer complete.
ftp: 2187264 bytes sent in 0.28Seconds 7783.86Kbytes/sec.
ftp> put D_F.JOURNAL
200 PORT command successful.
150 Opening data connection for D_F.JOURNAL.
226 Transfer complete.
ftp: 4096 bytes sent in 0.00Seconds 4096000.00Kbytes/sec.
ftp> QUIT
221 Goodbye.
In G130 & G131 release, to rerun a job, we need to insert a dummy select for the job. The
procedure for dummy select is a below. Please make sure to decatalog the dummy select
immediately after completing the job.
In Universe environment
> ED BP AC.STATIC.UPDATE.SELECT
------:I
0002 : RETURN
0003 : END
------ : FI
>
d) Catalog the program AC.STATIC.UPDATE.SELECT back to its original voc before continuing the next process.
In JBase environment:
4. Decatalog the routine from the directory after completing the job
jsh patch ~ --> DECATALOG TEST.BP AC.STATIC.UPDATE.SELECT
11.4 ERRMSG
The command can be used to identify the description of the error number
Example :
The error number varies based on the OS, so it is always advisable to get the output
from the client system where the error happened.
$df -k
Tips_ByPass_Rerun_
G13.doc
The attached document contains how to input a single sided Data Capture to
Balance a system in case of imbalance in the system
DC.BATCH.CONTRO
L.doc
(or)
Procedure in jBASE
Catalog
_object_code_Jbase.doc
Procedure in Universe
Catalaog_Object
Code_Universe.doc
Segmentation Violation.zip
Univ_Cmds.doc
12 Conclusion
We had included most of the procedures and routines in this document which can be used
as a guide to handle critical issues in ASG.. Please contact the ASG team Lead for any queries
or suggestions related to document.