Cluster Alert Log Files:

++++++++++++++++++++++++++++++
2:2016-12-30 23:27:14.550 [OCSSD(9044002)]CRS-1612: Network communication with n
ode dexdbs1 (1) missing for 50% of timeout interval. Removal of this node from
cluster in 14.704 seconds
3:2016-12-30 23:27:22.602 [OCSSD(9044002)]CRS-1611: Network communication with
node dexdbs1 (1) missing for 75% of timeout interval. Removal of this node from
cluster in 6.652 seconds
4:2016-12-30 23:27:26.605 [OCSSD(9044002)]CRS-1610: Network communication with
node dexdbs1 (1) missing for 90% of timeout interval. Removal of this node from
cluster in 2.648 seconds
5:2016-12-30 23:27:29.257 [OCSSD(9044002)]CRS-1632: Node dexdbs1 is being remov
ed from the cluster in cluster incarnation 376414176
11:2016-12-30 23:27:43.603 [ORAROOTAGENT(17629242)]CRS-5017: The resource action
"sap.DEX.DVEBMGS00.instvip start" encountered the following error:
2016-12-30 23:27:43.603+CRS-5005: IP Address: 10.72.2.211 is already in use in
the network
. For details refer to "(:CLSN00107:)" in "/oracle/diag/crs/dexdbs2/crs/trace/c
rsd_orarootagent_root.trc".
14:2016-12-30 23:27:44.134 [ORAROOTAGENT(17629242)]CRS-5017: The resource actio
n "sap.DJ0.javavip start" encountered the following error:
2016-12-30 23:27:44.134+CRS-5005: IP Address: 10.72.2.192 is already in use in
the network
. For details refer to "(:CLSN00107:)" in "/oracle/diag/crs/dexdbs2/crs/trace/c
rsd_orarootagent_root.trc".
17:2016-12-30 23:27:44.607 [ORAROOTAGENT(17629242)]CRS-5017: The resource actio
n "sap.DEX.abapvip start" encountered the following error:
2016-12-30 23:27:44.607+CRS-5005: IP Address: 10.72.2.211 is already in use in
the network
. For details refer to "(:CLSN00107:)" in "/oracle/diag/crs/dexdbs2/crs/trace/c
rsd_orarootagent_root.trc".
20:2016-12-30 23:27:48.523 [ORAAGENT(18022444)]CRS-5011: Check of resource "_mg
mtdb" failed: details at "(:CLSN00007:)" in
OS LOGS:
+++++++++++++++++++++++++
Failure Causes
UNEXPECTED SYSTEM HALT
Recommended Actions
PERFORM PROBLEM DETERMINATION PROCEDURES
Detail Data
DUMP DEVICE
/dev/lg_dumplv
DUMP SIZE
6441798144
TIME
Fri Dec 30 23:27:30 2016
DUMP TYPE (1 = PRIMARY, 2 = SECONDARY)
1
DUMP STATUS
-2
ERROR CODE
0000 0000 0000 0000
DUMP INTEGRITY

Compressed dump - Run "/usr/lib/ras/dmprtns/dmpfmt" with -c flag

on dump after uncompressing.
FILE NAME
--------------------------------------------------------------------------LABEL:
FWADUMP_DUMPERR
IDENTIFIER:
6D13FA41
Date/Time:
Sequence Number:
Machine Id:
Node Id:
Class:
Type:
WPAR:
Resource Name:

Fri Dec 30 23:38:41 PET 2016

46079
00F8D3F44C00
dexdbs1
S
UNKN
Global
fwadump

Description
Firmware-assisted system dump failure
Detail Data
Error code
EEEE 0000 9C22 0424
Error text
Internal error
--------------------------------------------------------------------------LABEL:
SCAN_ERROR_CHRP
IDENTIFIER:
BFE4C025
Date/Time:
Sequence Number:
Machine Id:
Node Id:
Class:
Type:
WPAR:
Resource Name:
Resource Class:
Resource Type:
Location:

Fri Dec 30 23:35:34 PET 2016

46077
00F8D3F44C00
dexdbs1
H
PERM
Global
sysplanar0
planar
sysplanar_rspc

Description
UNDETERMINED ERROR
Failure Causes
UNDETERMINED
Recommended Actions
RUN SYSTEM DIAGNOSTICS
Detail Data
PROBLEM DATA
0644 00E0 0000
2016 1231 0424
8200 07C6 532C
5053 0068 0100
0000 0000 0000
3520 2020 2020
5356 4344 4F43
0000 0000 414C
3033 4434 0000

0160
0895
4B12
4552
0000
2020
5300
3733
0000

C600
2016
5548
0201
0000
2020
4548
305F
0000

8E00
1231
0018
0002
0000
2020
0068
3134
0000

0000
0424
0100
0000
0000
2020
0100
3600
0000

0000
0998
4552
0060
0000
2020
4552
0000
0000

0000
4800
9101
0340
4232
C000
3832
0000
0000

0000
0106
4000
0000
3030
0006
3333
0000
001C

4942
0000
0000
0000
4131
1428
2D45
3030
4232

4D00
0000
0000
0000
3031
4D04
3842
3034
3030

5048
0000
0000
0000
204C
0000
3036
3037
4131

0030
0000
A000
0000
503D
0000
4433
3033
3031

0100
0000
0000
0000
3030
4944
4634
3034
5F30

494F
0000
0000
0000
3030
0CC8
5420
3630
3334

3030 3030 305F 4950 4C43 4F4D 5000 0000 4D54 001C 0100 4552 3832 3333 2D45 3842
3036 4433 4634 5420 0000 0000 4C50 001C 0100 4552 0005 0801 0000 0000 4445 5844
--------------------------------------------------------------------------LABEL:
KERNEL_PANIC
IDENTIFIER:
225E3B63
Detail Data
ASSERT STRING
PANIC STRING
Unexpected ODLM enabled state !
--------------------------------------------------------------------------LABEL:
ERRLOG_ON
IDENTIFIER:
9DBCFDEE
Date/Time:
Sequence Number:
Machine Id:
Node Id:
Class:
Type:
WPAR:
Resource Name:

Fri Dec 30 23:36:12 PET 2016

46074
00F8D3F44C00
dexdbs1
O
TEMP
Global
errdemon

Description
ERROR LOGGING TURNED ON
Probable Causes
ERRDEMON STARTED AUTOMATICALLY
User Causes
/USR/LIB/ERRDEMON COMMAND
Recommended Actions
NONE