IBM Ireland

Lotus Notes Java Crash Analysis

How to identify a Notes java crash?:
What Data you need to collect?. Why is javacore*.txt and *.dmp not created in in the workspace\logs directory and how to fix it. How to isolate a manual java dump versus automated java dump.?

What to look for in a java core file?

Resources for further reading.

2011 IBM Corporation

IBM Ireland

How to identify a Notes java crash

Unique stack always
OSWCTPanicShutdown calls Panic(), which runs NSD and brings down the Notes Client. Its is a generic stack, not useful for troubleshooting. Do not use it in SPRs.

2011 IBM Corporation

IBM Ireland

What to look for in a core file

Check callstack in THREADS section for Current thread:

> > > > > > > > > > > > > > > > > 1XMCURTHDINFO Current thread NULL ---------------------3XMTHREADINFO "main" J9VMThread:0x001EFD00, j9thread_t:0x003854B4, java/lang/Thread:0x009A3458, state:R, prio=6 3XMTHREADINFO1 (native thread ID:0x8EC, native priority:0x6, native policy:UNKNOWN) 3XMTHREADINFO3 Java callstack: 4XESTACKTRACE at org/eclipse/swt/internal/win32/OS.VtblCall(Native Method) 4XESTACKTRACE at org/eclipse/swt/internal/ole/win32/IUnknown.AddRef(Bytecode PC:5) 4XESTACKTRACE at com/ibm/rcp/swt/dnd/FileDescriptorTransfer.nativeToJava(Bytecode PC:846) 4XESTACKTRACE at org/eclipse/swt/dnd/DropTarget.Drop(Bytecode PC:320) 3XMTHREADINFO3 Native callstack: 4XENATIVESTACK (0x00000000) 4XENATIVESTACK _Java_org_eclipse_swt_internal_win32_OS_VtblCall__II@16+0xe (0x302C09F6 [swt-win32-3557+0x109f6]) 4XENATIVESTACK (0x7FF06F1F [j9vm24+0x16f1f]) 4XENATIVESTACK StgGetIFillLockBytesOnFile+0x87eb (0x775F976E [ole32+0x11976e]) 4XENATIVESTACK StgGetIFillLockBytesOnFile+0x8a2d (0x775F99B0 [ole32+0x1199b0]) 4XENATIVESTACK CreateDataCache+0x3cfb (0x775D0464 [ole32+0xf0464])

2011 IBM Corporation

IBM Ireland

Why is javacore*.txt and *.dmp not created in in the workspace\logs directory and how to fix it.

Java files will always be created under: C:\Program Files\IBM\lotus\notes\data\workspace\logs\javacore.xxxx.xxxx.txt But is Configurable in the rcplauncher.properties filercp.data=<workspace path> We collecting data most customer just send in the nsds , we should always advice to us ethe ADC tool as it collect all the files related for debugging, java cores and only applicable to the notes2.exe process Starting in 8.5.2, NSD will detect if the FATAL stack has JVM libraries loaded and dump a full system core dump Java frame found in fatal thread stack. Setting up core flags for use with IBM JVM diagnostic tools. To disable set DisableAutoJavaCore=1 in the nsd.ini. To customize set AutoJavaCoreFlags in the notes.ini

If a Java core was not found it will also be noted

> WARNING (0): NSD ran because the notes2 process exited unexpectedly (pid=792) with exit code 1. This indicates that the notes2 process was terminated via the Windows task manager. Please inspect the workspace\logs directory under your Lotus Notes data directory for possible jvm dump, heap dump, or system core files. It may have been killed via the task manager or an End Program dialog box

>

2011 IBM Corporation

IBM Ireland

How to Fix javacore files issue?

1. PANIC: Process notes2.exe (pid=####) no longer exists. A JVM fault could be responsible Doc #: 1416038 URL: http://www.ibm.com/support/docview.wss?uid=swg21416038 DISABLE_NOTES2_EXCEPTION_FILTER=1 , On by default in 8.5.2 2. Notes 8.5.2 client installers do not always put down latest version of rcplauncher.exe Doc #: 1472756 URL: http://www.ibm.com/support/docview.wss?uid=swg21472756 3. SPR # DCOY7TJPCB Javacore crashes on the Notes Client should be detected and collected by ADC INI to turn on javacore collection : DEBUG_ADC_JAVA_API=1 INI to turn off javacore collection : DISABLE_JAVACORE_COLLECTION=1

2011 IBM Corporation

IBM Ireland

Java Core sections

> TITLE Section > 1TISIGINFO Reason this java core was generated > Ex: gpf, user, systhrow, etc...
Gpf indicate a crash core User indicate a manual java core dump.

2011 IBM Corporation

IBM Ireland

Analyzing a Java Core for an OutOfMemoryError

In TITLE section, Javacore was triggerred by java/lang/OutOfMemoryError" 1TISIGINFO Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" received

Identify reason for OutOfMemoryError: In ENVINFO section, check minimum and maximum heap size settings: 2CIUSERARG 2CIUSERARG 1STHEAPFREE -Xms48m -Xmx256m

In MEMINFO section, check Heap Space Free and Heap Space Allocated Bytes of Heap Space Free: 28CB58 1STHEAPALLOC Bytes of Heap Space Allocated: 10000000

If heap space allocated is close to max heap size, the java heap was filled to capacity If the heap has not reached maximum size, the problem could be lack of native memory, such as when creating a thread. java.lang.OutOfMemoryError: Failed to create a thread.

2011 IBM Corporation

IBM Ireland

Analyzing a Java Core for a Hang

The standard client has one Java UI thread and one NLNOTES UI thread per Notes instance NLNOTES's UI thread Responsible for Notes UI Can run code via Hotspot Button, Action menu, Form's Post-open, etc... as well. Model dialogs can block this thread. Java UI thread Responsible for rendering Workbench If the Java UI thread or the NLNOTES UI thread is stuck doing the same work in its call stacks between the 2 samples is a good indication of a hang or a delay. Run NSD twice or more to collect multiple sets of java core files for comparison In 8.5.2 there is a added a Windows start menu icon to launch NSD -hang.

Start->All Programs->Lotus Applications-> Support->Collect Lotus Notes diagnostics data and leave Notes running.
This will collects two rounds of calls stack in the NSD and 2 java cores 15 seconds between each collection On the command line: nsd -hang -repeat=<number of collections> -delay=<milliseconds between collections> See nsd -help for additional details on the -hang switch. Starting in 8.5.2 nsd can find the notes.ini of the current user automatically

2011 IBM Corporation

IBM Ireland

Hangs
keys: 1. Compare the two NSD logs, see if the UI thread for NLNOTES is stuck in same stack. 2. In javacore THREADS section, check the stack and state for java UI thread: "main" TID:0x00140200, j9thread_t:0x000166F4, state:B, prio=6 Java core dump , search for state:B, BLOCKED 3. Since the UI thread main in in B (blocked) state, go to LOCKS section in core dump to find out what blocked it In this example, javacore LOCKS section shows main thread is waiting for a monitor held by Worker-1: 2LKMONINUSE sys_mon_t:0x059C2514 infl_mon_t: 0x059C254C:

3LKMONOBJECT com/ibm/rcp/accounts/internal/AccountsManagerImpl@114BEBF8/114BEC04: Flat locked by "Worker-1" (0x05C03500), entry count 1 3LKWAITERQ 3LKWAITER 3LKWAITER 3LKWAITER Waiting to enter: "main" (0x00140200) "Worker-3" (0x05F0E900) "Worker-6" (0x05F40D00)

2011 IBM Corporation

IBM Ireland

Problem Determination Checklist by development

> What is the problem? > If this is a problem or set of problems happening to many people and manifesting in NSD/Javacores, does the customer use ADC for data collection? > If so, get all of the ADC data and run an analysis and grouping on it > If a single issue, do we have an ISA from when/near the problem occurred? > Is the type of problem going to manifest itself in ISA data > A core crash, look for FATAL in the NSD > A "java crash", look for a GPF in the javacore > A "java out of memory", look for a javacore with systhrow > Was it a hang? > Was a "double NSD/Javacore" collected? > For each of these types of problems, irrespective of reproducibility, was an SPR match found? > Beyond the list above, is the problem reproducible?

10

2011 IBM Corporation

IBM Ireland

Good References
(1) Expeditor - Troubleshooting trace logs http://w3.tap.ibm.com/medialibrary/media_view?id=95984 (2) Javacore Analysis (Nik Brauer) http://w3.tap.ibm.com/medialibrary/media_view?id=98850 (covers some XPD log analysis)

11

2011 IBM Corporation