ACME Checks and Learnings - 2 PDF

ACME Checks and Learning
24th January 2016

Table of contents
 ACME Test Scenario
 Network Architecture
 VOLTE Call Flow
 Multiband Settings and Learnings
 Neighbours Deletion
 eNode B Prechecks
 True Call Based Analysis
 CSL Based X2 Analysis
 GTP Loss
 X2 Learnings
 S1AP – Critical for Uptime
 S1AP Preservation
 S5 Interface Learnings
 Core Network Learnings
ACME TEST Scenarios
V to V
•VOLTE to VOLTE
V to C
•VOLTE to CS
C to V
•CS to VOLTE
V to P
•VOLTE to PSTN
 Around 300 to 400 Calls for each scenario

 PASS CRITERIA AS MINIMUM 98% SUCCESS
 Call Set Up and Call Drops added together to be less then 2%
2016 © Samsung Electronics 3

ENB – Interface and Protocol
The Protocols that run between UE and eNB
are called Access Stratum Protocols or AS
Protocols
The Protocols that run between UE and Core
Network are called as NAS Protocols or Non
Access Stratum Protocols

MME – Interface and Protocol

MME – Interface and Protocol

SAE GW – Interface and Protocol

SAE GW – Interface and Protocol
• Ip Packet is tunneled between P GW and UE using GPRS Tunneling Protocol GTP

• Each Bearer has associated QCI
• Each QCI has GBR/ Non GBR, Priority, packet delay , Packet Error Rate defined
• PCRF decides the Bearer Type
• P GW performs the DHCP functionality = allocates IP addresses to each UE
VoLTE Call Flow
9
VoLTE – Logical Architecture
ENUM – Electronic Number Mapping System
To translate a telephone number into IP address

VoLTE – Logical Architecture

VoLTE - UE Attachment and IMS Registration message
CCR Credit Control Request Part 1 of 2

CCA Credit Control Answer

VoLTE - UE Attachment and IMS Registration message
Part 2 of 2

VoLTE - UE to UE Call establishment (Originating Side)

VoLTE - UE to UE Call establishment (Terminating Side)

VoLTE - UE to UE Call clearing (Initiated message)

VoLTE - UE to UE Call clearing (Received message )

VoLTE - VoLTE UE to CS call establishment (Originating Side )

VoLTE - VoLTE UE to CS call establishment (Originating Side )

VoLTE - VoLTE UE to CS call establishment (Terminating Side )

VoLTE - VoLTE UE to CS call Clearing (Initiated )

VoLTE - VoLTE UE to CS call Clearing (Received )

Multiband Settings & Learnings

Idle Mode (2300 and 1800)
Service Parameter 2.3G 1.8G
Target 2.3G 1.8G 1.8G 2.3G
Idle Mode Priority 7 6
64 64
QrxLevMin
-128 -128
12 12
S_NonIntraSearch
-104 dBm -104 dBm
12 12
Thresh_Serving Low
-104 dBm -104 dBm
12
ThrehX_Low
-104 dBm
14
ThrehX_High
-100 dBm
(-Q_Rxlevmin + 2 * Thresh_Serv_Low) dBm
1. Equal priority Band and higher priority Bands, are always measured
2. UE on 1.8 G will always measure 2.3G
3. The UE starts measuring other lower priority band neighbors when Serving Cell < S_nonintrasearch
4. UE on 2.3G measures 1.8G when 2.3G is lower than -104 dBm
5. UE does cell reselection from 2.3G to 1.8G when 2.3G is lower than -104 dBm and 1.8G is better
than -104 dBm
6. UE returns to 2.3G when 2.3G is better than -100 dBm
Idle Mode Parameters Behavior
-75dBm -80dBm -85dBm -90dBm -95dBm -100dBm -105dBm -110dBm -115dBm
Band 40
(2300MHz)

Band 3
(1800MHz)
• The UE starts measuring other lower 1. For Cell Reselection from 2300 to 1800 happens when
priority band neighbors when Serving 2300 is weaker than Thresh_Serving_Low and 1800 is
Cell < S_nonintrasearch. stronger than Thresh_Xlow (higher to lower priority)
2. For Cell Reselection from 1800 to 2300 happens when
• Equal priority band And higher priority 2300 is stronger than ThreshX_High (lower to higher
Bands, are always measured. priority)
Example
Band 3
Idle Mode (1800MHz)
Band 40
(2300MHz)
2300 has higher priority than 1800

-104 dBm
Handover : Event in Active Mode
 Event Type
 Intra RAT (Intra FA / Inter FA) : Event A

—Event A1 (Serving becomes better than threshold)
—Event A2 (Serving becomes worse than threshold)
—Event A3 (Neighbor becomes offset better than serving)
—Event A4 (Neighbor becomes better than threshold)
—Event A5 (Serving becomes worse than threshold1 and neighbor
becomes better than threshold2)

LTE Handover Events
 Event A1
Serving becomes better
than threshold
RSRP
TimetoTrigger
Thresh
Hys
Hys
RSRP
 Event A2
Serving Cell
Serving Cell
Time
Leave A1
Condition Enter A1
Enter A1 Condition Hys
Condition
Event A1 Thresh
Hys TimetoTrigger
Serving becomes worse than threshold

Time
Enter A2 Enter A2
Condition Condition
Leave A2
2016 © Samsung Electronics Condition Event A2 27
LTE Handover Events
 Event A3
 Neighbor becomes offset better than serving
RSRP/RSRQ
Serving Cell
Neighbor Cell
TimetoTrigger
A3 Offset
Hys
Hys
Hys
Time
Enter A3 Condition
When Ofn, Ocn, Hys,
Event A3
Ofs, Ocs, Off are all zero
Leaving A3 Condition Enter A3 Condition
When Ofn, Ocn, Hys, When Ofn, Ocn, Hys,
Ofs, Ocs, Off are all zero Ofs, Ocs, Off are all zero

LTE Handover Events
 Event A5
 Serving becomes worse than threshold1 and neighbor becomes better
than threshold2
RSRP/RSRQ Neighbour Cell
TimetoTrigger
Serving Cell
Stronger than
Threshold 2
Hys
Thresh2
Hys
Thresh1 Hys
Hys
Time
Leave Condition1 Serving Cell weaker
Enter Condition1 Enter Condition2
Event A5
than Threshold 1
Enter Condition1

-140 dBm + A2
Active Mode 2300-1800
Service Parameter 2.3G 1.8G
Target 2.3G 1.8G 1.8G 2.3G
GBR Service (VoLTE) 35 35
QCI based A2 Threshold
-105 dBm -105 dBm
QCI based A1 threshold (Meas
-90 dBm -90 dBm
Gap Deactivation)
6 6 6 6
QCI based A3 threshold
3 dB 3 dB 3 dB 3 dB
Non-GBR Service (FTP
27 80
DL) A2 Threshold
-113 dBm -60 dBm
A1 threshold(Meas Gap
-100 dBm -50 dBm
Deactivation)
6 6
A3 Threshold
3 dB 3 dB
A5 Threshold (Th1) -113 dBm -60 dBm
A5 Threshold (Th2) -110 dBm -110 dBm
1. For GBR for transition from 2.3G to 1.8G, as 2.3G becomes than weaker than -105 for
GBR, UE starts measuring other band and transitions when 1.8G is stronger than
2.3G by A3 Offset which is 3 dBm
2. For Non GBR - A5 for transition from 1.8G to 2.3G (as 2.3G would be weaker than
1.8G in most cases ) i.e. when serving cell in 1.8G is weaker than -60 dBm and the
target cell in 2.3G is better than -110 dBm
3. A2 and A3 are used for GBR for intra & inter band handover and for intra band
handover for non GBR
4. A5 is used for inter band handover for non GBR
Active Mode Parameters Behavior
Band 40
(2300MHz)

Band 3
(1800MHz)
For Non GBR - A2 and A3 for transition intra
For GBR - For transition from 2300 to 1800, band
as 2300 becomes than weaker than A2 A5 for transition inter band – especially from
threshold, UE starts measuring other band 1800 to 2300 (as 2300 would be weaker than
and transitions when 1800 is stronger than 1800 in most cases )
2300 by A3 Offset (which would always
happen except for Micro cells)
-113 dBm
Example
Data Mode Band 40

(2300MHz)
Band 3
VoLTE Mode (1800MHz)
2016 © Samsung Electronics

-105 dBm 31
Expected Relative Behavior in Connected Mode
Band 40
(2.3Ghz)
-80Bm -85dBm -90dBm -95dBm -100dBm -103dBm -110dBm -115dBm -120dBm

Band 3
(1.8GHz)
Call Data Service

Continues Data Start
Services Data Service
Data Services to VoLTE

Before -113dBm Data
Data Data Service
VoLTE Call Data Service
Service Start Service
VoLTE call end
VoLTE
Before –110 dBm
Call END

BEFORE -113dBm
Data VoLTE Call Data Service
Service Start
VoLTE call end VoLTE Data
After –110 dBm Call END Service

After -113dBm
Data Data
Service Service
VoLTE call end Data VoLTE Call VoLTE Data Service
After –110 dBm Service Start Call END
Bhopal Learning : Why it Happened ?
Overview: A1 should always be higher than A2
 Parameter Template 1: Configuration with Multi-band Implementation
 Parameter Template 2: Default Parameter implementation during Golden Parameter audit & corrections
 Audit & corrections carried out on 24-Oct over-ride A2 value earlier implemented as a part of Multi-band Parameter list
Technical Analysis:
Default Parameter List vs Multi-band Services Implementation
(Used in Work Stream 1) (Used in Work Stream 2) Remarks
A1: Not Set A1: -100dBm ( value: 40 ) If RSRP is better than A1, it
will stop all other measurements
A2: -90dBm ( value: 50 ) A2: -113dBm ( value: 27 ) If RSRP is lower than A2, UE has
to start measurement of neighbor band
Result:
 A2 got configured as -90dBm in place of -113dBm, with A1 as -100dBm
 When UE RSRP goes less than -90dBm (A2), inter-band measurements will trigger.
 However since RSRP is better than A1 ( -100dBm ), UE will be instructed to stop the inter-band measurements.
 Thus conflicting settings for A1 & A2, leads to high messaging & thus eNB going into Racing condition.
Learnings – from incorrect EARFCN settings
CELL_NUM FA_INDEX DUPLEX_TYPE STATUS EARFCN_UL EARFCN_DL PRIORITY

0 0 TDD ( TDD ) EQUIP ( EQUIP ) 38800 ( 38800 ) 38800 ( 38800 ) 7(7)
0 1 FDD ( TDD ) EQUIP ( 19294 ( 38800 ) 1294 ( 38800 ) 6(7)
N_EQUIP )
1 1 FDD ( TDD ) EQUIP ( 19294 ( 38800 ) 1294 ( 38800 ) 6(7)
N_EQUIP )
2 1 FDD ( TDD ) EQUIP ( 19294 ( 38800 ) 1294 ( 38800 ) 6(7)
N_EQUIP )
3 0 FDD ( FDD ) EQUIP ( EQUIP ) 19294 ( 19294 ) 1294 ( 1294 ) 6(6)
3 1 TDD ( FDD ) EQUIP ( 38800 ( 19356 ) 38800 ( 1356 ) 7(7)
N_EQUIP )
4 1 TDD ( FDD ) EQUIP ( 38800 ( 19356 ) 38800 ( 1356 ) 7(7)
N_EQUIP )
5 1 TDD ( FDD ) EQUIP ( 38800 ( 19356 ) 38800 ( 1356 ) 7(7)
N_EQUIP )
EARFCN settings for eNode B and for Multiband should be the same
Else Handovers would get impacted

Active & Idle Mode – All Three Bands
Service Parameter 2.3G 1.8G 850M
Target 2.3G 1.8G 850 1.8G 2.3G 850 850 2.3G 1.8G
GBR Service (VoLTE) 35 35 35
QCI based A2 Threshold
-105 dBm -105 dBm -105 dBm
QCI based A1 threshold (Meas Gap

-90 dBm -90 dBm -90 dBm
Deactivation)
6 6 6 6 6 6 6 6 6
QCI based A3 threshold
3 dB 3 dB 15+5 dB 3 dB 3 dB 3 dB 3 dB 3 dB 3 dB
Non-GBR Service (FTP DL) 27 80 80

A2 Threshold
-113 dBm -60 dBm -60 dBm
A1 threshold(Meas Gap Deactivation) -100 dBm -50 dBm -50 dBm
6 6 6
A3 Threshold
3 dB 3 dB 3 dB
A5 Threshold (Th1) -113 dBm -113 dBm -60 dBm -113 dBm -60 dBm -113 dBm
A5 Threshold (Th2) -110 dBm -93 dBm -110 dBm -110 dBm -110 dBm -110 dBm
Service Parameter 2.3G 1.8G 850M

Target 2.3G 1.8G 850 1.8G 2.3G 850 850 2.3G 1.8G
Idle Mode Priority 7 6 5
64 64 64
QrxLevMin
-128 -128 -128
12 12 12
S_NonIntraSearch -104 -104
-104 dBm
dBm dBm
12 12 12
Thresh_Serving Low -104 -104
-104 dBm
dBm dBm
12 12 14
ThrehX_Low -104 -104 -104
dBm dBm dBm
14 14 14
ThrehX_High -100 -100 -100
2016 © Samsung Electronics dBm dBm dBm 35
Neighbor Deletion Analysis
36
Neighbor Deletion Based on HO Success
 Neighbor Table
 Auto build by LSMR and UE based . Max NBR in table are 256
 Key parameter are Neighbor cell PCI (Unique/Band), Cell ID , TAC, eNB ID
 Wrong Neighbor impacting HO performance degradation

HO Performance Improvement
 Radio Interface Failure are mainly happened by Poor RF after/Before HO Failures.
 HO Failure Cases and Resolution

 Wrong Neighbor Relation
2.UE Failed HO due to
wrong PRACH Sequence
1.HO Preparation done due to
UE wrong neighbor
Real Target Cell Wrong target
Source cell
PCI : “A” Very far and signal is not detected cell, PCI : “A”
— Needs to turn on HO Success Rate Based NBR Deletion

— The feature deletes Neighbor that shows HO failure count greater than “Th1” if total HO Attempt is greather than “Th2”
 PCI Confusion
Neighbor Cell Neighbor Cell PCI Reallocation
Case1 Source cell
PCI : “A” PCI : “A”  SON PCI needs to be ON
Neighbor Cell Neighbor Cell

Case2 Source cell PCI : “A”
PCI : “A” Very far but signal is detected
Overshooting, RF Optimization
needed
 HO Command Reception Failure due to Poor RF
— Too late HO, Needs to optimize RF condition

Neighbor Relation Table - Optimization
 Methods for Wrong Neighbor deletions
 MR Based This feature removes Neighbor Relation (NR) which cannot receive Measurement Report
(MR) messages among the NRs included when the network was initially created, so that only valid NRs
could be included in the Neighbor Table.
Deployed at Ahemdabad, Ujjain and Hyderabad
 Neighbor Deletion based on HO Success This feature removes Neighbor Relation based on HO Success
Rate
Deployed in network
 Neighbor Deletion based on NBRDEL-CAUSE This feature removes Neighbor Relation based on 2 cause
reported Cell not available & Unknown Target ID
Deployed in network
 Manual Neighbor Deletion This is based on troubleshooting HO call failure observed in CSL/Truecall.
Not Applicable

Neighbor Deletion based on MR: Ahmedabad Plan
Day eNB Configuration Parameter Objective KPI & Performance Indicator Checks
nrDelFlag = True (Enable PCI Conflict Notification

Feature)
NBR Count
Total NBR
MR Based thNumMrNrDel = 3 (Count Relation to RRC Connection Setup Success Rate, ERAB Success
Day 1 Neighbor of MR) reduce after Rate, Attach Success Rate
Deletion Aggressive MR Call Drop Rate, RRC Connection Re-establishment
based Deletion Attempts and Success Rate
thTimeNrDel = 1 (Days)
Handover Preparation Success Rate, Handover
Completion Success Rate
PCI Conflict Notification
thNumMrNrDel = 5 (Count NBR Count
of MR) RRC Connection Setup Success Rate, ERAB Success
Change in MR
MR Based Rate, Attach Success Rate
Thresold to
Day 3 Neighbor
reduce NBR
Deletion Call Drop Rate, RRC Connection Re-establishment
deletion count.
Attempts and Success Rate
thTimeNrDel = 3 (Days)
Handover Preparation Success Rate, Handover
Completion Success Rate

Neighbor Deletion based on MR: Ahmedabad
 MR Based Deletion
 Enabled MR based NR deletion on 13th Jan . Before enabling this feature the total NRs in Ahmedabad city are 407192.
on 14th Jan the MR based deletion algorithm deleted the most of the NBRs and the count on 15th Dec is 44K.
Date NBR Addition NBR deletion

1/12/2016 970 61
1/13/2016 1152 84
1/14/2016 28169 379783
1/15/2016 1187 12665
Handover Statistics

UE Based ANR
 Overall Call Flow
#UE reports unknown PCI
#eNB orders UE to report eCGI by sending

“RRCConnectionReconfig”.
Point 1) Samsung eNB does not trigger
report CGI process for those UE with GBR
bearer.  VoLTE UE does not report eCGI
#In order for UE to get eCGI information, UE

has to go to Idle mode with help of DRX mode
or Meausurement GAP (only applicable for
inter FA) to get eCGI information in SIB1 of the
target cell.
#Data Scheduling has higher priority than DRX

mode by 3GPP Standard.
Point2) If UE continuously receives or transmits
traffic, UE cannot go to DRX mode.
UE-based-ANR is triggered before LSM-based-ANR

MR Based Neighbor Deletion
 Feature Description
This function removes Neighbor Relation (NR) which cannot receive Measurement Report (MR) messages among
the NRs included when the network was initially created, so that only valid NRs could be included in the Neighbor Table.
 Feature Configuration
nrDelFlag = True
thNumMrNrDel = 5 in Ujjain (Range 0 to 1000K)
thTimeNrDel = 1 in Ujjain ( Range 1 to 7 days)
 Ujjain Observations
Total eNB 84 , Neighbors 40,000
18Dec - Enabled MR based NR deletion
20Dec - MR based deletion algorithm deleted the most of the Neighbors.
24Dec – 3413 Neighbor created mostly by UE based ANR.
InterX2 HO Attempts
Ujjain Analysis – MR Based Neighbor Deletion HO Performance Stats InterX2 HO Failure
100.00%
45000 40000 15
26 18
40000 99.80% 24
35000
70
InterX2HOSuccRate
91
Number Of NBRs
99.60% 103
30000
25000 99.40%
20000
99.20%
15000
10000 99.00%
2100 3413 3083
5000 400 1200
98.80%
0
7557 5627 7107 12110 12880 9315 11888
98.60%
22-Dec-15 23-Dec-15 24-Dec-15 25-Dec-15 26-Dec-15 27-Dec-15 28-Dec-15

Neighbour Deletion based on HO Success Pre-Post Analysis 6 cities
 In 6 cities analysis , 3K to 6K Neighbors having HO Attempt greater than 10 and very few neighbors having HO
Success less than equal to 5.
Neighbor Count HO Attempt Neighbors Count Attempt GT
Total Neighbors
City Greater than 10 10 & HO Success less than 5
Daily Average
Daily Average Daily Average
Ahmedabad 167225 6600 (3.95%) 3
Bangalore 380499 6000 (1.58%) 4
Hyderabad 311600 5500 (1.77%) 2
Chennai 191142 3200 (1.67%) 1.5
Kolkata 435667 3700 (0.80%) 1.5
Bhopal 92736 1250 (1.35%) 1
 Neighbor Deletion threshold of HO Attempt GT 10 and HO Success less than 5 are not met in most of the cases
 1 to 4% of neighbors having HO Attempt greater than 10 indicates less users.
 Next Step – Review the HO Attempts and HO Success every month for any optimization of threshold.

eNode B Prechecks
46
Pre-requisites/ Checklist (RAN) …..Contd.
1. SCFT: 100% completed and all sites to be radiating.
• Run command “RTRV-CELL-STS” in all eNBs.
• Cells with Status = DISABLE needs to be checked.
2. No Service affecting alarm in eNB
• Run command “RTRV-ALM-LIST” in all eNBs.
• Service impacting alarms needs to be rectified.
3. eNB on 5.0.0. – all patches to be loaded , SW & FW Audits
• Go to Configuration -> Software -> Verify. Select all eNBs and verify the Software and Firmware.
• eNBs with “NOK” status, should be noted and packges should be upgraded in planned event.
• Please refer Software/Firmware Verification in attached pre-requisite file.
4. OCNS to be removed
• Run command “MON-TEST” in all eNBs.
• eNBs with OCNS running should be noted. Run command “TERM-TEST” to remove OCNS, in all eNBs.
• Please refer OCNS Check section for procedure in attached pre-requisite file.
5. Multiband/Golden Parameter Settings
• Go to Configuration -> Parameter Manager -> Select appropriate parameter planner.
 Golden Parameter: pkg_500new
 Multiband Parameter : Multi_Carrier_23n18_CircleName (For eNBs having 2300 MHz and 1800 MHz bands)
Multi_Carrier_3band_CircleName (For eNBs having 2300 MHz and 1800 MHz bands)
• Select all eNBs and compare the parameter manger.
• eNBs with “NOK” status, should be noted and parameter planner should be applied in planned event.
• Please refer Golden Parameter/Multiband Parameter Settings section for procedure in attached pre-requisite file.
6. PCI SON Notifications to be resolved
• Go to SON -> Log Dump Management. Select the time period and get the dump.
• From dump, check source and target eNBs and solve the PCI collision if both neighbors are required.
• Please refer PCI SON Notifications section for procedure in attached pre-requisite file.

7. Inactivity Timer 30 Sec, All Sites on “Same Power Option”.
— Run command “RTRV-INACT-TIMER” for all eNBs. Internal USER_INACTIVITY_TIMER should be 30 for all QCI.
— Run command :CHG-INACT-TIMER” for changing the USER_INACTIVITY_TIMER.
8. Inter LSMR Neighbors to be added manually or using UE based ANR.

— Inter LSMR required neighbor list would be provided by RF team.
— Run command “RTRV-NBR-EUTRA” for provided cells.
— In case, neighbor does not exist, it needs to be added using GUI.
— Please refer Neighbour Addition/Deletion section for procedure in attached pre-requisite file.
9. Inter Band Neighbors to be checked for 2300, 1800 & 850.

— Run command “RTRV-NBR-EUTRA” for all cells and if inter band neighbor exists.
— In case ,intra eNB inter band neighbor does not exist, neighbors need to be added using GUI.
10. PCI repetition within 2 km to be removed for all cases, Additional Audit for 4 km and first tier /second tier neighbor -
all PCI conflict cases to be plotted on map info and corrective actions identified.
— Go to SON -> Log Dump Management. Select the time period and get the dump.
— This dump needs to be provided to RF team, RF team will map conflict cases on map .
— RF will recommend to change PCI of some sites.
— PCI of sites can be changed using GUI.
— Please refer PCI/EARFCN Change section for procedure in attached pre-requisite file.

11. Small Cells: Neighbor Definitions to be checked within Small cells (if in a cluster).
— List neighboring sites of small cells and run “RTRV_NBR_EUTRA” on all cells.
— If neighbors are found missing, add the neighbors using GUI.
12. RRH Log Analysis to ensure all sites radiating .

— Run “RTRV-RRH-STS” for all eNBs.
— Check if RRH are in operational state and NO OF FA =1.
13. EARFCN definitions to be audited for multiband operation for all elements (Macro & Small cells).
— Run command “RTRV-CELL-IDLE” in all eNBs.
— Check UL and DL EARFCN of all eNBs with EARFCN allocated to respective circle.
— Please refer PCI/EARFCN Change section for procedure in attached pre-requisite file.

TrueCall Based Analysis
50
True Call Report – Cell Wise RLF & X2 Fail (Sample)
eNB ID Cell Name
Total ACME Zone
RLF
5 16 17 18 35 36 37 48 49
1076 13 14 56 112 22 20 104 12 353 New Mumbai JNPT
4147 109 1 11 96 3 4 76 39 339 kalyan
2915 147 80 8 66 2 10 313 New Mumbai JNPT
2166 78 104 35 24 26 16 283 New Mumbai JNPT
2790 17 4 19 124 41 1 61 1 268 New Mumbai JNPT
936 13 24 21 86 12 48 13 15 3 235 New Mumbai JNPT
2918 11 40 56 53 29 13 17 14 233 New Mumbai JNPT
318 38 70 2 1 114 225 New Mumbai JNPT
1556 98 19 8 28 28 23 9 213 kalyan
2919 5 79 61 22 7 174 New Mumbai JNPT
4092 2 15 125 4 146 thane central
4242 123 4 4 2 1 134 south
1337 5 21 30 24 9 7 1 21 9 127 New Mumbai JNPT
1418 13 3 4 27 3 55 20 125 kalyan
1348 1 25 5 7 50 1 17 18 124 west
eNB ID Cell Name

Total ACME Zone
X2 Fail 5 16 17 18 35 36 37 48 49
1055 1 1175 1 328 17 1 1523 New Mumbai JNPT
1386 23 3 168 1108 4 6 22 2 1336 thane central
2166 14 76 18 980 11 25 1124 New Mumbai JNPT
2176 7 447 67 19 5 37 582 New Mumbai JNPT
2791 3 3 5 26 2 3 20 469 1 532 kalyan
1266 266 43 50 32 21 3 4 1 420 New Mumbai JNPT

Abnormal Release Causes
Abnormal Release Cause
GTP Pa th Fa i l ure
Abnormal Release Cause
RLC - Inva l i d Pa ra meter
eNB - Ra di o Li nk Fa i l ure eNB - Rel ea s e Due To eNB Genera ted Rea s on
S1AP - Reduce Loa d In Servi ng Cel l MAC - Inva l i d Cel l Id

eNB - Rees tabl i s hment Fa i l ure Due to Inva l i d State
RRM - UE S-TMSI Dupl i ca te
S1AP - Ra di o Network Uns peci fi ed
eNB Ti meout - Inter X2 Ha ndover Comma nd Compl ete
eNB Ti meout - X2 SN Status Tra ns fer
eNB Ti meout - RRC Connection Setup Compl ete Mes s a ge Not Recei ved
eNB Ti meout - Inter S1 Ha ndover Comma nd Compl ete
eNB Ti meout - RRC Connection Reconfi gura tion Compl ete Mes s a ge Not Recei ved
RRM - MME Overl oa d
eNB Ti meout - Intra Ha ndover Comma nd Compl ete
eNB Ti meout - S1Rel ocOvera l l Expi ry
eNB Ti meout - X2 Rel ocOvera l l Expi ry eNB - Recei ved X2 Res et Reques t
eNB Ti meout - RRC Connection Rees tabl i s hment Compl ete Mes s a ge Not Recei ved eNB Ti meout - Interna l Rees tabl i s h Control
eNB - ARQ Ma xi mum Retra ns mi s s i on eNB Ti meout - RRC Securi ty Mode Compl ete Mes s a ge Not Recei ved
eNB - Ra di o Li nk Fa i l ure by RRC Connection Rees tabl i s hment RLC - Ca l l Is Not Active
eNB - DSP Audi t RLC MAC Ca l l Rel ea s e S1AP - Ca us e NAS Uns peci fi ed
S1AP - Authentica tion Fa i l ure S1AP - HO Fa i l ure In Ta rget EPC eNB Or Ta rget Sys tem
eNB - ARQ Ma xi mum Retra ns mi s s i on by RRC Connection Rees tabl i s hment eNB Ti meout - Interna l Da ta Tra ns fer Start
eNB - S1 SCTP Out of Servi ce eNB Ti meout - S1 MME Status Tra ns fer Not Recei ved
MAC - Inva l i d Pa ra meter
X2AP - Ca us e Mi s c Uns peci fi ed
RRM - MME Not Ava i l a bl e for PLMN
eNB Ti meout - RRC UE Ca pa bi l i ty Informa tion Mes s a ge Not Recei ved
eNB Ti meout - Interna l Res ource Setup
eNB Ti meout - S1 Ini tia l Context Setup Reques t
GTP Setup Fa i l ure
eNB Ti meout - S1 Pa th Swi tch Reques t Ack Not Recei ved
PDCP - Inva l i d RB Id
eNB - Recei ved Cel l Rel ea s e Indi ca tion from eNB
X2AP - Cel l Not Ava i l a bl e
S1AP - Ra di o Connection Wi th UE Los t
S1AP - Loa d Ba l a nci ng TAU Requi red
S1AP - CS Fa l l ba ck Tri ggered
eNB - Recei ved Res et Reques t From eNB
RRM - CQI Res ource Fa i l ure
GTP Recei ve Error Indi ca tion From GW
X2AP - Trel ocprep Expi ry

True Call – Sample Report Abnormal Release
Abnormal Release Cause Mumbai Kolkata Delhi
eNB - Ra di o Li nk Fa i l ure 12474 21.57 8276 25.47 6756 16.88

S1AP - Reduce Loa d In Servi ng Cel l 12151 21.01 2235 6.88 1242 3.10
RRM - UE S-TMSI Dupl i ca te 8547 14.78 5274 16.23 8753 21.86
eNB Ti meout - Inter X2 Ha ndover Comma nd Compl ete 5796 10.02 4117 12.67 5809 14.51
eNB Ti meout - RRC Connection Setup Compl ete Mes s a ge Not Recei ved 4367 7.55 2274 7.00 4141 10.34
eNB Ti meout - RRC Connection Reconfi gura tion Compl ete Mes s a ge Not Recei ved 4214 7.29 1408 4.33 2053 5.13
eNB Ti meout - Intra Ha ndover Comma nd Compl ete 3020 5.22 2771 8.53 2955 7.38
eNB Ti meout - X2 Rel ocOvera l l Expi ry 2613 4.52 1248 3.84 2728 6.81
eNB Ti meout - RRC Connection Rees tabl i s hment Compl ete Mes s a ge Not Recei ved1625 2.81 2072 6.37 2499 6.24
eNB - ARQ Ma xi mum Retra ns mi s s i on 729 1.26 786 2.42 1050 2.62
eNB - Ra di o Li nk Fa i l ure by RRC Connection Rees tabl i s hment 628 1.09 308 0.95 283 0.71
eNB - DSP Audi t RLC MAC Ca l l Rel ea s e 477 0.83 358 1.10 440 1.10
S1AP - Authentica tion Fa i l ure 215 0.37 18 0.05 55 0.14
eNB - ARQ Ma xi mum Retra ns mi s s i on by RRC Connection Rees tabl i s hment 196 0.34 127 0.39 162 0.41
eNB - S1 SCTP Out of Servi ce 187 0.32 26 0.08 417 1.04
X2AP - Ca us e Mi s c Uns peci fi ed 164 0.28 12 0.04 77 0.19
eNB Ti meout - RRC UE Ca pa bi l i ty Informa tion Mes s a ge Not Recei ved 99 0.17 33 0.10 43 0.11
eNB Ti meout - S1 Ini tia l Context Setup Reques t 63 0.11 8 0.03 114 0.28
eNB Ti meout - S1 Pa th Swi tch Reques t Ack Not Recei ved 36 0.06 18 0.05 65 0.16
eNB - Recei ved Cel l Rel ea s e Indi ca tion from eNB 32 0.06 22 0.07 32 0.08
GTP Pa th Fa i l ure 30 0.05 987 3.04 127 0.32

Abnormal Release Trend
True Call Implemented
Total Abnormal Release (%) (PAN India) Only in Gujrat, Mumbai, Delhi, AP,
Karnataka, Kolkata, MP
5.00 60000000
51931704 53543901
4.80 4.50
44683001 50000000
Release Count
% Abnormal Failure
4.60
39154166 4.24 4.22
4.40 40000000
4.20 3.98
4.00 30000000
3.80
3.60 20000000
3.40 10000000
3.20 1843142 1980235 2289216 2218316
3.00 0
Dec Week3 Dec Week4 Jan Week1 Jan Week2
Total Release Abnormal Release % Failure
800000
700000
600000
Absolute Failures
500000
Dec Week3
400000
Dec Week4
300000
Jan Week1
200000 Jan Week2
100000
0
Gujrat Mumbai Delhi AP Karnataka Kolkata MP
Circles With True Call Implemented

Abnormal Release Breakup (4th~10th Jan)
Sr No. Failure Cause Mumbai AP Delhi Gujrat Karnataka Kolkata MP
1 eNB - Radio Link Failure 21.59% 25.98% 16.90% 31.32% 20.15% 25.48% 27.68%
2 S1AP - Reduce Load In Serving Cell 21.03% 2.89% 3.11% 5.45% 6.74% 6.88% 5.94%
3 RRM - UE S-TMSI Duplicate 14.80% 15.81% 21.90% 17.40% 17.09% 16.24% 18.63%
4 eNB Timeout - Inter X2 Handover Command Complete 10.03% 9.71% 14.53% 6.77% 9.35% 12.68% 8.09%
eNB Timeout - RRC Connection Setup Complete Message

5 7.56% 10.90% 10.36% 7.04% 13.29% 7.00% 8.24%
Not Received
eNB Timeout - RRC Connection Reconfiguration Complet

6 7.30% 4.40% 5.14% 4.79% 5.14% 4.33% 4.83%
e Message Not Received
7 eNB Timeout - Intra Handover Command Complete 5.23% 3.76% 7.39% 4.04% 5.27% 8.53% 4.90%
8 eNB Timeout - X2 RelocOverall Expiry 4.52% 5.38% 6.83% 5.09% 5.07% 3.84% 6.26%
eNB Timeout - RRC Connection Reestablishment Complet

9 2.81% 5.33% 6.25% 8.83% 7.14% 6.38% 6.83%
e Message Not Received
10 eNB - ARQ Maximum Retransmission 1.26% 1.81% 2.63% 2.00% 1.80% 2.42% 1.96%
11 X2AP - Cause Misc Unspecified 0.28% 7.93% 0.19% 0.66% 4.20% 0.04% 1.89%
12 GTP Path Failure 0.05% 1.41% 0.32% 0.17% 1.16% 3.04% 0.30%

Detail Analysis for Mumbai (7th Jan)
Total Release Abnormal Release % Fail

1781047 79516 4.46
 Out of 79516 Abnormal release more than 95% failures contributed by below shown 10 Cause codes.
25.00
20.99 20.87
20.00
% Contribution to Total Failures
15.00 14.02
11.92
10.00
7.14 6.63
4.56 4.56
5.00
2.79
1.61
0.00
S1AP - Reduce eNB - Radio Link RRM - UE S-TMSI eNB Timeout - eNB Timeout - RRC eNB Timeout - RRC eNB Timeout - X2 eNB Timeout - eNB Timeout - RRC S1AP -
Load In Serving Failure Duplicate Inter X2 Handover Connection Connection Setup RelocOverall Intra Handover Connection Authentication
Cell Command Reconfiguration Complete Message Expiry Command Reestablishment Failure
Complete Complete Message Not Received Complete Complete Message
Not Received Not Received

Analysis on Reduce load in serving cell
 Specific device model is contributing to this release cause
 Sony, Motorola and Unknown device types
Make Model Total Failure
MOTOROLA MOBILITY LLC A LENOVO COMPANY MOTOROLA LX12506245 19701
SONY MOBILE COMMUNICATIONS TBD 7296
MOTOROLA MOBILITY LLC A LENOVO COMPANY INDR001345 4414
MOTOROLA MOBILITY LLC A LENOVO COMPANY INDR000945 3177
N/A TAC: 35234207 2915
N/A TAC: 35236007 1856
MOTOROLA MOBILITY LLC A LENOVO COMPANY MOTO E (2ND GEN) WITH 4G LTE ST12424645 1818
N/A TAC: 35548407 1265
N/A TAC: 35235907 902
SONY MOBILE COMMUNICATIONS XPERIA E4G DUAL 859
MOTOROLA MOBILITY LLC A LENOVO COMPANY CA12459845 730
N/A TAC: 35386307 488
N/A TAC: 35235607 186
N/A TAC: 35627407 117
N/A TAC: 35627307 103

Analysis on Reduce load in serving cell
 Device Capability Issue
 If device does not support “TDD-FDD HO” or “Inter FA HO”, eNB sends the UE to other carrier with
“Redirection” with release cause of “Reduce Load in Serving Cell” in current eNB Pkg, instead of
“Handover”
 Needs to check Feature Group Indicator 13, 25 and 30 in UE Capability Information for all devices
 Call Flow
UE eNB 862 MME
MR EVENT A2(LteHo)
RRC Conn ReConfiguration
RRC Conn Reconfiguration Comp
MR EVENT A5(IntraLteHandover)
As UE does not support Inter frequency HO or
RRC Connection Release (with redirectedCarrierInfo) FDD TDD handover then eNB will initiate redirection
UE Context Release Request(Reduce load in serving cell(19))
UE Context Release Command
UE Context Release Complete

Radio Link Failure Analysis
 Out of 16596 RLF observed in entire Mumbai, 11.6% fails happened in 11
Cell ID Total Fail
cells.
4294.16 473
 Top 3 Cell Analysis 2761.50 217
 4294.16 is IBS of TC22, and supporting data for further analysis not available
2257.16 201
from CSL for 05/01/2016.
 For 2761.5 & 2257.16,finding given below. 4294.17 171

 In both cell, specific single IMSI is contributing to most of the fails 2257.17 154
Total 820.37 143

eNB Id IMSI Detail Remarks
Failure
Outskirt facing cell having no site within 4142.37 129

405874000046676 212
2-3 km area. Failures from far location
2761.50 405874000025017 4 suspected. Last CQI Reported is of Value 1076.18 114
I-MU-KLYN-ENB-6008 5 (SINR < 0).
405874000047743 1 Also fails observed in one specific IMSI 2918.37 110
only.
405874000026883 199
No site upto 1 km and very high rise 75.35 108
clutter observed. Failures from far and
2257.16
high-rise indoor location suspected. Last 1076.37 104
I-MU-MUMB-ENB-7334 405874000062987 1
CQI Reported is Of Value 3 (SINR < 0).
Also fails observed in one specific IMSI
405874000016480 1
only. Total 1924

Radio Link Failure (2761.50 - I-MU-KLYN-ENB-6008)
 Outskirt facing cell having no site within 2-3 km area. Failures from far location suspected.
 Also fails observed in one specific IMSI only.
2761.50
> 1 km

Radio Link Failure (2217.56 - I-MU-MUMB-ENB-7334)
 No site upto 1 km and very high rise clutter observed. Failures from far and high-rise indoor location suspected.
 Also fails observed in one specific IMSI only.
Non-Integrated Site
> 1 km
2217.56

Inter X2 HO Fail Analysis
 Out of 9481 fails observed in entire Mumbai, 21% fails happened in 1 cells only.
Cell ID Total Fail
1350.18 1953
 Top 1 Cell Analysis

 When checked IMSI wise fails, only one IMSI contributing to 77% of the fails.
 Need to trace that IMSI and do the further testing.
 For remaining fails in this cell, wrong neighbor deletion/PCI conflict to be checked.
eNB Id IMSI Detail Total Failure

405874000050068 1498
1350.18 405874000051815 61
I-MU-MUMB-ENB-1327 405874000064118 53
Other IMSI 308

[Issue-1] Missing Events
 TrueCall is not capturing all the events.
 As seen below, in TrueCall Logs, there are no events between 01/06/16 1:26:56 to 1:28:41, whereas, as per CSL logs, 3
events occurred during same time.
TrueCall Logs
CSL Logs
Call Release FreqBandI cgi.PLMN

eNBNAME Call Attempt Time Call Setup Time cgi.CI PCI Call Release Time PCI cgi.PLMNID cgi.CI IMSI
Cause ndicator ID
eNB_1484 0x00000FFF 2016-01-06 10:55:48:650 10:55:48:744 40 405 874 379921 76 2016-01-06 10:56:17:923 76 405 874 379921 405874000017389
eNB_2166 0x00000FFF 2016-01-06 11:19:49:869 11:19:49:962 3 405 874 554532 208 2016-01-06 11:20:21:337 208 405 874 554532 405874000017389
eNB_4241 0x00000FFF 2016-01-06 12:04:18:839 12:04:18:944 40 405 874 1E+06 332 2016-01-06 12:05:08:922 332 405 874 1085714 405874000017389
eNB_4103 0x00000FFF 2016-01-06 13:12:01:035 13:12:01:190 40 405 874 1E+06 501 2016-01-06 13:12:27:062 501 405 874 1050384 405874000017389
eNB_3417 0x00000308 2016-01-06 13:27:25:845 255:255:255:65535 40 405 874 874768 252 2016-01-06 13:27:28:855 252 405 874 874768 405874000017389
eNB_964 0x00000324 2016-01-06 13:26:57:707 13:26:57:801 40 405 874 246802 236 2016-01-06 13:27:30:859 236 405 874 246802 405874000017389
eNB_4091 0x00000236 2016-01-06 13:28:15:081 255:255:255:65535 40 405 874 1E+06 501 2016-01-06 13:28:16:620 501 405 874 1047313 405874000017389
eNB_1662 0x00000343 2016-01-06 13:28:28:082 13:28:28:188 40 405 874 425488 3 2016-01-06 13:28:41:234 3 405 874 425488 405874000017389
eNB_2686 0x00000308 2016-01-06 13:30:44:682 255:255:255:65535 40 405 874 687633 181 2016-01-06 13:30:47:693 181 405 874 687633 405874000017389

CSL Based
X2 Failure Analysis
For Wrong Neighbour
64
[0x0308]EccTmout_InterX2HandoverCmdComplete – Case Study
 [0x0308] Tags When A call is released when the target eNB cannot receive the RRC Connection Reconfiguration Complete message
during the X2 handover.
 As Per UE Logs UE has sent MR For 118 PCI which It Found better so A3 Event Triggered. But After that UE has Started PRACH
Procedure but failed as done on wrong Cell.
Source cell Wrong Target cell Actual Target

UE
PCI 53 PCI 118 PCI 118
A3 Event Triggered
MR HO Preparation
HO Command
PRACH FAIL MSG2

3000ms
EccTmout_InterX2HandoverCmdComplete
How To Identify Wrong Nbr Based Upon CSL – (1/2)
Target CI Source CI
Call Release
Call Attempt Time Call Setup Time cgi.CI[0] cgi.CI[1] IMSI PCI
Cause
0x00000308 2016-01-04 00:05:04:044 255:255:255:65535 689445 89125 405874000000000 203

0x00000308 2016-01-04 00:05:45:940 255:255:255:65535 354851 287762 405874000000000 147
0x00000308 2016-01-04 00:05:46:505 255:255:255:65535 542225 936466 405874000000000 31
0x00000308 2016-01-04 00:06:48:003 255:255:255:65535 887077 735781 405874000000000 323
0x00000308 2016-01-04 00:06:50:607 255:255:255:65535 947728 869155 405874000000000 45
0x00000308 2016-01-04 00:07:17:781 255:255:255:65535 557073 239633 405874000000000 4
0x00000308 2016-01-04 00:07:20:331 255:255:255:65535 212259 541475 405874000000000 105
 By CSL we get Cause Code/ Cell Id From UE History, so that we can Find Source & Target eNB .
 Please Check Call Release Cause “0x0000 308” is for A call is released when the target eNB cannot receive the RRC
Connection Reconfiguration Complete message during the X2 handover
 By True call Please check Column Name “Last HO Source Cell ID” & “Last HO Target Cell ID”, For Call
Release Casue “eNB Timeout – eNB Timeout - Inter X2 Handover Command Complete
How To Identify Wrong Nbr Based Upon CSL – (2/2)
eNB275
Wrong Id
eNB1231
Correct Id
eNB2155
eNB2155
Wrong Total
Source Id Distance (Km) Source PCI Target PCI Nearst eNB Nearst Distance Remarks
Target Id Failure
2155.37 275.17 5.07 172 53 118 1231 0.76 DELETE
 Ue has sent measurement report for eNB1231 PCI 118 but UE Does RACH With eNB275, Due to Wrong eNB
Defined in eNB2155 PCI 53.
 Hence Handover Fail So we can Check Nearby Same PCI & Delete Wrong One To Reduce Drop Rate by Handover Fail
S1 AP Links – Uptime Very Critical
21-Jan-2016
68
UE Temp ID’s – S1AP, GTP-C, GTP-U

SCTP Link Fluctuation –MT UE in IDLE Mode
 MT UE was connected to eNB-1 and moved to IDLE mode due to User Inactivity.
 Incoming call received for the UE.
 As the UE was in IDLE mode, MME has sent Paging Request to eNB-2.
 As the SCTP link with the eNB-1 where UE is currently present, UE will not be able to recei
ve Paging Request message

SCTP Link Fluctuation –MO Call
 MO UE attached to MME-1 and GW-1 via eNB – 1 and VoLTE call is on-going
 SCTP link to MME-1 with eNB-1 and eNB-2 are down.
 UE sent Service Request to MME-2 via eNB-2(as eNB2 -> MME-1 link is down)
 MME-2 will send service Reject which results in UE sending Fresh Attach Request.
 As UE sent new Attach, the previous call gets dropped

Network issue related to Backhaul
Introduction
 GTP Loss
 GTP Loss
 Root Cause of GTP Loss
 Statistic & Troubleshooting
 MME&IND Communication Fail

 MME&IND communication
 Root Cause of MME&IND Communication
 X2 Communication Fail
 X2 communication Fail
 Root Cause of X2 Communication Fail

GTP Loss -
 GPRS Tunneling Protocol (GTP) is a group of IP-based communications protocols
used to carry general packet radio service (GPRS) within LTE networks
 GTP Loss means packet drops between SAEGW and eNB. It is only calculated on
bearer down link(SAEGW --> eNB direction).
 GTP Loss can happen only by user traffic but cannot occur by ping or traceroute
between SAEGW and eNB.
 In case of Samsung SGW & PGW both are at same location (called SAEGW). So we
will always check GTP loss between SGW and eNB.

Root Cause of GTP Loss-
 There are two reasons for GTP loss (packet drop) in backhaul-
 Buffer Overflow (Backhaul Bandwidth & Buffer size defined)
 Physical drops
 1- Buffer Overflow-
When network element send data packets with high speed and receiving node
cannot forward it with the same speed due to bottleneck in network, And has small buffer t
o store packets so it will start discarding excess packets once its buffer gets
full. This situation is called Buffer overflow.
 2- Physical drops-
 The next common issue that can lead to drops would be a physical component
that is malfunctioning. Physical issues like SFP module problem, cable problem,
loosed connection, line card problem, duplex mismatch, etc. are the most
common reasons for packet drop in network.
 These physical drops and backhaul media Buffer overflow results in low
throughput.

GTP Loss- Trouble-shooting
 Check GTP loss counter(GtpSnEnbDeltaLossRate) in LSMR and if GTP loss more than 0.01%
observed then check for below 3 cases
1. Physical drops (drop rate requirement is less than 0.01%).
2. Small buffer in the bottleneck (Buffer size < 512KBytes and bottleneck bandwidth <= 25
0Mbps).
3. Low bandwidth in the bottleneck (bottleneck bandwidth < 100Mbps).
Even though low throughput is observed in the field, if the above 3 cases are
not in backhaul, definitely there's no backhaul issue causing low throughput.
 Cisco team can only identify case # 1 in the above 3 cases. So Cisco team
should not declare that backhaul is clean without buffer size and rate limitation
or bottleneck bandwidth confirmation. Only by ping test, buffer size issue and
wrong rate limitation issue cannot be identified.

76 76
MME&IND Communication Fail -
 MME is responsible for idle mode UE (User Equipment) paging and tagging
procedure including retransmissions. It is involved in the bearer activation/
deactivation process and is also responsible for choosing the SGW for a UE at
the initial attach and at time of intra-LTE handover involving Core Network (CN)
node relocation
 MME communicates with eNB using S1-AP/S1-MME interface.
 Initially SCTP connection is established between eNB and MME by exchanging SCTP messag
e and once SCTP connection is established then it is used for communication using S1-APset
up as shown below-

MME&IND Communication Fail
 The trend of IND Fail with the worst sample circle (GJ, MH, MP, MU)
MME ID 11-Dec. 13-Dec. 15-Dec. 16-Dec. 21-Dec. 23-Dec. 04-Jen. 06-Jan. 15-Jan. 18-Jan. 20-Jan.
MME[4] 144 159 144 144 145 159 143 143 141 141 141
GJ
MME[5] 1 2 1 1 1 1 7 7 2
MME[0] 279 295 295 148 146 150 151 160 20 21 23
MME[3] 50 64 72 65 46 67 57 73 10 17 11
MH
MME[4] 4502 4546 4552 2344 2330 2360 2370 2374 50 57 44 ▪ In case of MH and MP Circle,
MME[5] 40 21 20 26 22 25 19 21 5 11 5
MME[3] 2 1 2 1 4 22 6 7 1 1 the IND Fail were cleared after CISCO do
MP MME[4] 4060 4502 4501 4504 4499 4510 4522 4518 55 109 3 in the SAR.
MME[5] 2 3 3 3 4 9 3 4 2 2 2
MME[1] 237 253 234 242 205 222 24 18 10 8 46
MU MME[2] 228 247 227 237 208 267 114 86 82 51 95
MME[3] 223 243 221 226 203 260 118 83 80 51 94
 ‘IND Communication Fail’ – Post MME Pool split

(Based on Site Count)
Circle AP AS BR DL GJ HR JK KA KO MH MP MU PB RJ TN UE UW WB Total
Count 12 3 21 12 150 20 64 11 13 76 8 185 326 3 10 35 31 137 1,117
 Intensive inspections and improvements required
(GJ, MU, PB, WB)
1. Missing MME Interface for 1100 eNB added.
2. MME Interface not required in MME Pool deleted as Database cleanup activity.
3. eNB grown before MME Pool activity having Long time IND MME alarms since they were grown with old MME Pool.
Next Step – Above step to continue for 1 week to clean IND communication alarm

Root Cause MME&IND Communication Fail-
 There are two reasons for MME Communication Fail in backhaul-
 SCTP Connection failure
 S1-AP setup failure
Troubleshooting SCTP connection failure-

-IPv6 configuration(“LTE_Signal_S1” should be true for 102 Vlan only)
-eNB to MME ping check.
Troubleshooting S1-AP setup failure-

-MME configuration(MME status should be “EQUIP” and Administrative state should be “Unlock
ed”).
-eNB must be defined in MME .
-Parameters related to eNB like MCC, MNC, Enb_ID, S1-AP_IP etc are defined correctly
in the MME.

X2 Communication Fail -
 X2 handover is performed between a source eNB and a target eNB through the X2
interface.
 In LTE networks allow eNBs to directly exchange status information with each other via the
X2 interface, and to independently perform handovers without any
intervention by EPC nodes.

X2 Link Status
 The Trend of X2-Link disabled.
 Observation from Samsung

- Missed configuration at eNB.
- Wrong IP in Neighbor list.
- Parameter set with ‘neighbor
Unlocked’
 Observation from CISCO

- Missed configuration at AG1
- MTU Size different between eNB and
 The count of X2-Link (Two-Way) disabled wit Circle basis. BH
- Duplicate IP assigned
- Routing issues at CSS
- Physical Link issues
 Observation from Juniper

- Neighbor discovery protocol issues.
- Routing issues at CSS
- Physical Link issues

Root Cause X2 Communication Fail-
 There are two reasons for X2 Communication Fail in backhaul-
 SCTP Connection failure
 X2-AP setup failure
Troubleshooting SCTP connection failure-
-IPv6 configuration(“LTE_Signal_X2” should be true for 101 Vlan only)
- Source eNB to Neighbour eNB ping check for Vlan 101.
Troubleshooting X2-AP setup failure-
Samsung-
-eNB IP Correction in eNB neighbor list.
-eNB Administrative State in both neighbors to be "Unlocked"
- "NO_X2State" should be "False" instead of "True" in both the neighbors.
Cisco-
-MTU Configuration Check & Correction .
-Two IP addresses for single eNB, removing the multiple IP's.
-AG1 Configuration change for boundary Cases
-Correction of routing issue at CSS and other nodes.
Juniper-
-Correction of routing issue at CSS.
GTP LOSS Troubleshooting
GTP Loss Resolution

Table of content
What is GTP loss

Why GTP loss happens
Where & How to see GTP loss
Troubleshooting steps
Case Study
Background
 TCP Congestion : Packet loss has generated by network congestion as well as other things.
 Speed of packets sending > network capacity  Congestion
 The packet loss happened intermittently, even if packets sending under network capacity.
 TCP congestion control
HSS
S6a
GTP-U
GTP-C
GTP-U MPLS Network
GTP-U
FTP

TCP Packet loss
 Intermittently, the packet loss occurs by Rate limitation in the normal network
environment.
 Even if the packet loss happened with an unspecified that it does not stop the DT test.
 If constantly experiencing ‘packet loss’ or find out the ‘Rate limitation’

then stop the DT test and have the action in order to fix it.
— The MOP that is BH checking process already shared by RAN TF
 If you find out the Low Throughput somewhere during test in the field.
 As you well known, Please check the RF Condition including eNB status (Is it down or not ?)
 Please check the current alarm (Related BH, Related RU, Related RF, Etc.)
 Check the Rate limitation using shared MOP
 If you do not find out the reason, then call the RAN BH TF

What is GTP Loss
 GPRS Tunneling protocol is an important IP based protocol used in GSM, UMTS and LTE core networks. It
is used to encapsulate user data when passing through core network and also carries bearer specific si
gnaling traffic between various core network entities.
 It provides mobility. When UE is mobile, the IP address remains same and packets are still forwarded
since tunneling is provided between PGW and eNB via SGW.
 Multiple tunnels (bearers)can be used by same UE to obtain different network QoS
 Main IP remains hidden so it provides security as well.

What is GTP Loss (Contd)
Note-
 GTP Loss means packet drops between SAEGW and eNB.
 It is only calculated on bearer down link(SAEGW --> eNB direction).
 GTP Loss can happen only by user traffic but cannot occur by ping or traceroute betwee
n SAEGW and eNB.
 GTP tunnel is formed only between SAE-GW and eNB so GTP loss has no relation with UE
RF conditions.

Detail picture showing data flow inside GTP tunnel-

90 90
GTP OOS
 When UE tries to download any data then TCP session is established between UE and server.
 When this data comes to SAE-GW from server then It encapsulates this data using GTP, UD
P, IP, L2 and L1 headers and forwards this data to eNB through GTP tunnel.
 GTP header consists of Sequence no. & tunnel end point Id (TE ID) which is assigned to each
packet.
 This encapsulated data transfers through GTP tunnel with Sequence no. 1,2,3,4,5….. and so
on and reaches to eNB.
 If after 2nd packet eNB receives 4th packet instead of 3rd packet then this event would be cou
nted as loss in GTP loss counter but after receiving 3rd packet this event would be counted as
Out of Sequence packet(OOS) as shown below-

91 91
GTP Delta Loss
 That means this packet was not lost but came after words (Out of sequence packet) so this will
not contribute to GTP loss. This can be seen in GtpSnEnbDeltaLoss column which is calculated as-

92 92
Why GTP Loss happens
There are basically two reasons for GTP loss (packet drop) in backhaul-
— Buffer Overflow
— Physical drops
 Buffer Overflow-
What is Buffer in a network device?

-Small amount of memory used for temporary storage of data, usually to compensate for differe
nces in between receiving and transmitting speeds on a network device. It serves as a reservoir i
n which the higher speed ingress data to be dumped which is then 'trickled' to the slower one.
What is Buffer overflow?

-When server is sending data packets with very high speed and receiving node cannot forwar
d it with the same speed due to bottleneck in network and it has small buffer to store packets
so it will start discarding excess packets once its buffer gets full . This situation is cal
led Buffer overflow.

93 93
Buffer Overflow
 Reason for Buffer overflow-
Rate Mismatch-
Case 1 Case2

94 94
Physical Drops
 Physical drops-
The next common issue that can lead to drops would be a physical component that is malfunct
ioning.
-Physical issues like SFP module problem, cable problem, loosed connection, line card proble
m, duplex mismatch, etc. are the most common reasons for packet drop in a network.
-If hardware is not working properly, it will usually lead to error messages being seen on the con
sole of the device or within system logs but there are other software issues also which can lead t
o packet drop and main concern is there is no alarm indication mechanism for these issu
es in the system

95 95
GTP Loss As an Indication
 GTP loss can’t be avoided in commercial mode when multiple TCP session
establishes to access data ,making congestion in network bottleneck point.
 GTP Loss cannot be used as any index or indicator which means backhaul normal
ity because of speed mismatch in backhaul environment (10G -> 1G on AG1 to CSS
ring, 1G -> 470, 250, 150, 100, 45 on Microwave or Leased line), GTP Loss by bo
ttleneck buffer overflow cannot be avoidable in commercial mode where
Multi TCP sessions keep trying to increase TCP window size so congestion happ
ens in a bottleneck point and so buffer overflow occurs.

96 96
Where and how to see GTP Loss
 GTP loss can be seen in LSMR in performance statistics or On demand monitoring.
In GTP statistics, there are two items of GTP Loss and GTP OOS. The real packet loss can be calcul
ated by (GTP Loss Count - GTP OOS count) and it's shown in GtpSnEnbDeltaLoss. So GtpSnEnbDe
ltaLossRate shows the real loss rate.
 How to check GTP loss ?

-Generate the traffic on the UE and check the GTP loss counter (GtpSnEnbDeltaLossRate) in on d
emand monitoring.
— Login to LSMR.
— Go to PERFORMANCE-On Demand.
— Select target.
— Register-In Statistics family, go to GTP & select GTP sequence number per eNB-period (30sec)- duration (
as per testing requirement).Then click on register.
— After this click on site which is registered then Monitor option will be highlighted. Click on monitor and select
GtpSnEnbDeltaLoss(count) and GtpSnEnbDeltaLossRate(%)-click Apply and then click
Start.
NOTE-GTP Loss must be less than equal to 0.01% for proper throughput. (>50Mbps)

97 97
Example Report
Example-
Note- Please note that the GTP loss counter is only valid for downlink throughput
analysis.

98 98
Troubleshooting Steps
 Check GTP loss from LSMR as explained above and if GTP loss occurs perform IPERF test be
tween SGW and eNB to check
— Physical drop
— Backhaul Bandwidth
— Buffer size
Put iperf.octeon64 software in LENA and in eNB, use below command for Iperf test in Downlin
k.
 eNB Side:
iperf.octeon64 -s -V -B eNB_Bearer_IP -u -l 8000 -i 1 -p 8001
 SGW(LENA) :
iperf.octeon64 -c eNB_Bearer_IP -u -V -i 1 -b 1m -l 8000 -t 9999 -p 8001
Note – Change the mode of iperf file using command “chmod777 iperf.octeon64” be
fore executing commands.
Where,
s->server, V->IP version6, B->binds eNB IP, u->UDP traffic, l 8000->length of datagram size(bytes),
p->port, c->client, t->time, b->B.W. to be pumped.
First step is to check if any physical drop is happening.

99 99
Identification
 How to identify Physical issue ?
- Perform IPERF test between SGW and eNB on lower B.W.(less than equal to 10Mbps)
- If packet drop is observed while pumping lower B.W from SGW to eNB then that means there is
some physical issue in backhaul.
-Ask cisco team to check backhaul.
Note- Please note that on Lower B.W. (<=10Mbps) packet loss should be less the 0.01%.
If no drop observed on lower B.W, proceed further to check if any bottleneck present in backha
ul by performing iperf test on higher B.W.
 How to find Backhaul bandwidth ?
- Initiate ping to eNB from SGW to check latency.

- Start sending traffic from SGW to eNB using Iperf. (Make server side ready before sendin
g traffic i.e. command at eNB should be executed first in DL).

100 100
Examples
SAEGW IPERF O/P:
SGW is sending 100Mbps dummy traffic to eNB as shown below.
Now check at eNB how much traffic is being received by eNB .
eNB IPERF O/P:

101 101
Examples
-eNB is receiving 100Mbps traffic as shown above. If packet drop greater than 0.01% observe
d then decrease B.W. (dummy traffic) and perform test or else increase the traffic and check the s
ame.
-Keep monitoring at eNB end, if continuous packet drop is observed on each instant.
-If continuous drop is observed then decrease B.W. (which is being sent from SGWeNB) or else
increase B.W. and monitor at eNB.
-Keep monitoring ping latency from SGWeNB if it is increasing. (Increasing latency indicate
s buffer is getting full)
-Keep doing this process until you get a bottleneck point (B.W.) after which you get CONTIN
UOUS packet drop at eNB.
This point would be your rate limit of backhaul.
Example-
Suppose while pumping 237Mbps traffic from SGW to eNB you are getting 0% packet drop on
each instant and while pumping 238Mbps traffic you are getting continuous packet drop at e
ach instant and ping latency is also increasing at 238Mbps then you can say that 237Mbps is bac
khaul BW or rate limit of backhaul.
NOTE-While performing iperf test you may find a situation where intermittent huge packet
drop is observed. Even though packet drop is more than 0.01% but this point is not rate limit p
oint because loss is not continuous.
102 102
Examples
 How to find Bottleneck Buffer size ?
-Once you start getting continuous packet drop at eNB, you will observe that your ping (SGW
eNB ping) latency is increasing.
-Keep observing it until you observe constant latency.
-Once you observe constant latency, stop (Ctrl+c) all three commands (traffic sending, traffic r
eceiving and ping) in all three windows.
-Now in ping window you will observe min/avg/max RTT as shown below-

103 103
Buffer Calculation
-Once you calculated the RTT, find Buffer size using Buffer estimation chart shown below-

104 104
Buffer Size
Example-
Suppose you got rate limit (Bottleneck B.W.) as 237Mbps and RTT as 0.271 sec then in above c
hart search for RTT equal to or nearer to 0.271 in the row of 250Mbps and look corresponding
Buffer size vertically. Here Buffer size is 8192KBytes i.e. around 8MB.

105 105
Iperf Commands
 TCP throughput can also be affected by uplink issue as TCP depends on Ackno
wledgements.
So we need to perform uplink iperf to check the same.
 Commands for Uplink iperf-
eNB Side :
Iperf.octeon64 -c SGW_IP -V -B eNB_bearer_IP -u -l 8000 -i 1 -t 999 -b 10m -p 8001
SGW(LENA) :
iperf.octeon64 -s -V -u -l 8000 -i 1 -p 8001
If no loss found till 10Mbps in uplink then we can say that uplink path is clean.
NOTE-
You can use any port (“-p 5001”) for iperf test as per your wish but port number must be same
at both client and server end.

106 106
Iperf Command
 These things can also be found using 103 VLAN i.e. by performing Iperf test between LSMR
and eNB in the same way as mentioned above.
Put iperf.octeon64 software in eNB and iperf software in LSMR, use below command for Iperf te
st.
 DOWNLINK UDP (from LSMR to eNB)-
eNB side :
iperf.octeon64 -s -V -B eNB_OAM_IP -u -l 8000 -i 1 -p 5001
LSMR side :
./iperf -c eNB_OAM_IP -V -u -l 8000 -i 1 -b 10m -t 999 -p 5001
 UPLINK UDP (from eNB to LSMR)-
eNB side :
iperf.octeon64 -c LSMR RS_IP -V -B eNB_OAM_IP -u -l 8000 -i 1 -t 999 -b 10m -p 5001
LSMR side :
./iperf -s -V -u -l 8000 -i 1 -p 5001
Note-
-LSMR to eNB iperf test is valid only if LSMR is at same location where SGW is located.
-No loss between LSMR to eNB (OAM VLAN) doesn’t indicate that Bearer VLAN (SGW to eNB)
is also clean.
107 107
Iperf Commands
 We can also perform Iperf test between FTP server and UE for troubleshooting purpos
e.
Traffic comes from FTP server to UE using bearer (101) VLAN so we can also check GTP loss in
LSMR during this test.
(FTP---- > SAEGW ---- > MPLS network ---- > eNB ---- > UE)
-Put iperf software in FTP server and download iperf tool in UE from playstore to perform test.
DOWNLINK UDP (FTP server to UE)-
FTP server side:
iperf -c IP_of_UE -u -i 1 -t 90 -b 50m -p 5002
UE side:
iperf -s -u -i 1 -p 5002
UPLINK UDP (UE to FTP serve)-
UE side:
iperf -c Service_IP_of_FTP_server -u -i 1 -t 90 -b 10m -p 5002
FTP server side:
iperf -s -u -i 1 -p 5002

108 108
Iperf Commands
 TCP IPERF test can also be performed between FTP and UE using below comm
ands-
DOWNLINK TCP (FTP server to UE)-

FTP server side:
iperf -c IP_ of_UE -i 1 -t 90 -w 1m -p 8002
UE side:
iperf -s -i 1 -p 8002
UPLINK TCP (UE to FTP server)-

UE side:
iperf -c Service_IP_of_FTP_server -i 1 -t 90 -w 1m -p 8002
FTP server side :
iperf -s -i 1 -p 8002

109 109
Summary-
-Even though low throughput is observed in the field, if the below 3 cases are not in backh
aul, definitely there's no backhaul issue causing low throughput.
-Physical drops (drop rate requirement is less than 0.01%)

-Small buffer in the bottleneck Buffer size < 512KBytes & bottleneck B.W. <= 250Mbps).
-Low bandwidth in the bottleneck (bottleneck bandwidth < 100Mbps).
- Cisco team can only identify case # 1 in the above 3 cases. So Cisco team should not declar
e that backhaul is clean without buffer size and rate limitation or bottleneck bandwidth co
nfirmation. Only by ping test, buffer size issue and wrong rate limitation issue cannot be iden
tified. Cisco team needs to find out a method to improve their backhaul check.

110 110
CASE STUDY
PHYSICAL ISSUE on I-DL-FDBD-ENB-2685
 Initial analysis using iperf-
Observed packet drop in iperf test and asked cisco team to check backhaul.

111 111
Case Study
 Cisco Analysis

112 112
Case Study
Effect of Low power at AG1 & AG2 –

113 113
Case Study
IPERF logs after changes from Cisco end-
NOTE-
Packet loss got cleared after clearing physical issue in backhaul by cisco circle team.

114 114
X2 Study & Learnings
115 115
X2 Disabled Status Summary
Nexus
SAR
AG3
AG2 AG2
X2 Request to Target eNB
AG1 AG1
CSS CSS
UE
Source eNB X2 Target eNB
 X2 Disabled cases decreased significantly due to X2 routing done through SAR instead of AG2 Router.
 Out of 12,11,000 X2 links, only 4892 links (0.40%) are in disabled state.
 These are primarily in Tamilnadu , UP(East), Karnataka which are deployed with juniper routers and Delhi,
Maharashtra with Cisco routers, Which we are jointly troubleshooting
X2 disabled cases is currently reduced to ~1%. Will further troubleshoot and resolve to make it nil
Confidential
2016 © Samsung Electronics Reliance Jio Infocomm Limited 116 116
Pre-checks for X2 Interface
 From Cisco Side –
 MTU Configuration Check & Correction .
 Two IP addresses for single eNB, removing the multiple IP's.
 AG1 Config change for Boundary Cases
 Cisco ASR 901 ( Old Cisco Router) Image Up-grade to handle Neighbor Discovery Protocol Issue.
 Correction of routing issue at CSS and other nodes.
 From Juniper Side –
 Juniper Router (ACX2200) Image Up-grade to handle Neighbor Discovery Protocol Issue.
 Correction of routing issue at CSS
 From eNB Side
 "LTE_SIGNAL_X2 " needs to be true only for VLAN 101.
 eNB IP Correction in eNB Neighbor list.
 eNB Administrative State in both Neighbors to be "Unlocked"
 "NO_X2State" should be "False" instead of "True" in both the neighbors.
 eMBMS IP change in order to make subnet mask "/126" from "/64" as it was falling under same n/w
of neighbor eNB making wrong entries in eNB routing table.
Confidential
2016 © Samsung Electronics Reliance Jio Infocomm Limited 117 117
S1 AP Bearer Preservation
118
S1 AP Preservation – To improve VoLTE performance
Call Drop Analysis of ACME drive Test
Sr. No. Time Samsung NEO RF/eNB S1AP Cause code Time taken to
recover (sec)
1 15:39:38.259 A call is released when the target eNB cannot receive the RRC radioNetwork: ho-failure-in-target-EPC- 0.22
Connection Reconfiguration Complete message during the S1 eNB-or-target-system (6)
handover.
2 19:08:59.258 RLF at MT due to poor SINR. MT send measurement report radioNetwork: ho-failure-in-target-EPC- 2.10
continuously but no HO command received from eNB eNB-or-target-system (6)
3 15:34:25.566 RLF happened due to HO failure at MO, poor SINR radioNetwork: failure-in-radio-interface- 0.30
procedure (26)
4 16:27:08.406 MO logs are incomplete and no drop at MT found. radioNetwork: ho-failure-in-target-EPC- 0.10
eNB-or-target-system (6)
5 16:37:09.774 No RF issue. No uplink RSSI issue found. Call is released when ARQ radioNetwork: unspecified (0) 2.90
ACK is not received from the UE after the eNB RLC performs
transmission as many as the Max Retransmission to the UE. UE issue
suspected.
6 16:55:17.871 HO Failure due to poor SINR at MT. A call is released when the target radioNetwork: ho-failure-in-target-EPC- 0.30
cell cannot receive the RRC Connection Reconfiguration Complete eNB-or-target-system (6)
message during the intra-handover
7 18:21:42.528 RLF happened due to poor SINR at MT radioNetwork: failure-in-radio-interface- 0.70
procedure (26)
8 18:32:21.596 Call dropped because of HO failure. High DL BLER observed No NW logs available 0.10
9 18:22:00.177 UE lost UL synch and trying to access eNB again. But RACH fails and radioNetwork: failure-in-radio-interface- 0.50
RLF happened at MO. Poor SINR. procedure (26)
Average 800 msec

Average 317 msec (Excluding 2 & 5)

S1 AP Preservation –To improve VoLTE performance
 Observation
• ACME drive reported 9 out of 600 VoLTE call were dropped after HO failure and RLF.
 Analysis
• MME deleting the QCI-1 bearer after receiving UE Context Release request with a cause of HO failure
and UE lost from eNB.
 Solution
• Since HO failure and UE lost due to RLF is being observed in the field and it is also observed that the
recovery process is also happening is less than 1 sec.
• It is suggested to preserve the QCI-1 bearer in case of UEContextRelease with HO failure MME
deleting the QCI-1 bearer after receiving UE Context Release request with a cause of HO failure and
UE lost from eNB
• Preservation ON is implemented in IDC on 1st Sep. Performance will be monitored for ACME .

Other Actions Taken to Avoid Call Drop

 All eNBs upgraded to PKG 5.0.0
 Multi-Band Implementation
 Forward Hand Over enabled

 Equal SAEGW Traffic distribution
 TA/TAL/BTAL Configuration Audited
 GBR Bearer preservation set to “ON” (i.e. In Test Area Serving 7 MMEs Only )
S1AP - Cause
Cause Description Old Value New Value Benefit
Code Value
Handover Failure In Target EPC/eNB Or Target
06 OFF ON
System
Failure in the Radio Interface Reduction in Call Drops due to RLF
26 OFF ON for fractions of seconds
Procedure
21 UE Connection Lost OFF ON
 Function Flag #9 (NOT_USE_PRSV_GBR_FOR_ACT_S1) to “ON”

 To the preserve status of S1AP CAUSE (tx2 relocation timer expiry)

 Clear Test Area Service Impacting Alarms
 Check Test Area Availability Status
S1AP preservation option in MME
 When eNB detects any RAN layer problems, eNB can send ‘UE Context Release Request’
message to MME to release the UE-associated S1 connection
 As per 3GPP standard, MME should release the UE context as per the request
eNB MME
UE Context Release Request (Cause)
UE Context Release Command (Cause)
UE Context Release Complete
 However, eNB’s local decision may be wrong in case of some causes, for example,
 UE gets lost from an eNB, but the UE can re-attach to some other eNB
 During HO, UE is not responding and X2 timer expires, but the UE can re-attach to some other eNB
 For some cases, it can be better to preserve S1 connection rather than releasing UE context
 Samsung MME provides such options (On/Off) for every S1AP cause code
Ref: S1AP cause code
 3GPP TS 36.413 defines the following cause codes
Code Cause Code Cause
0 Unspecified 20 User Inactivity
1 TX2RELOCOverall Expiry 21 Radio Connection With UE Lost
2 Successful Handover 22 Load Balancing TAU Required
3 Release due to E-UTRAN generated reason 23 CS Fallback triggered
4 Handover Cancelled 24 UE Not Available for PS Service
5 Partial Handover 25 Radio resources not available
6 Handover Failure In Target EPC/eNB Or Target System 26 Failure in the Radio Interface Procedure
7 Handover Target not allowed 27 Invalid QoS combination
8 TS1RELOCoverall Expiry 28 Inter-RAT redirection
9 TS1RELOCprep Expiry 29 Interaction with other procedure
10 Cell not available 30 Unknown E-RAB ID
11 Unknown Target ID 31 Multiple E-RAB ID instances
12 No radio resources available in target cell 32 Encryption and/or integrity protection algorithms not supported
13 Unknown or already allocated MME UE S1AP ID 33 S 1 intra system Handover triggered
14 Unknown or already allocated eNB UE S1AP ID 34 S1 inter system Handover triggered
15 Unknown or inconsistent pair of UE S1AP ID 35 X2 Handover triggered
16 Handover Desirable for Radio Reasons 36 Redirection towards 1xRTT
17 Time Critical Handover 37 Not supported QCI Value
18 Resource Optimization Handover 38 Invalid CSG Id
19 Reduce Load in Serving Cell
Preservation option case study
 RJio network has the following S1AP preservation options = ON
 The S1AP preservation options are compared with KT case in Korea
ID DESC KT RJio
1 Tx2relocoverallExpiry ON ON
3 ReleaseDueToEutranGeneratedRsn ON -
6 HoFailInTargetEpcEnborTargetSy ON ON
8 TS1relocoverallExpiry ON -
16 HoDesirableForRadioReason ON -
20 UserInactivity ON -
21 RadioConnectionWithUeLost - ON
26 FailureInTheRadioInterfaceProcedure - ON
28 InterRatRedirection ON -

S 5 Interface – Learning from Delhi
125
X2 HO - SGW change Between Noida/Vikaspuri
• Possible Impacted Case:
For Single PLMN, two Sites
distributing traffic with S5.
• Applicable Sites - Mumbai,
Delhi and Chennai.
• Scenario:
• On Registration Co-located
SAEGW get selected
• Initial PGW provides IP to
user
• Once HO happens
between SGW Group,
New SGW will get selected
but PGW remains same.
• Signaling & Traffic moves
from new SGW to old PGW
on S5 interface.
In Delhi Case – S5 GTP-C was ok hence bearer

was Modified correctly but S5 GTP-U was
down hence Voice/Data was not working.
X2 HO with SGW change in Delhi
S T

S5 Interface GTP-U Packet Drop
 GTPU packets on the S5 Interface getting dropped due to missing IP Route
 Uplink packets are getting dropped in between SGW (Noida) -> PGW (Vikaspuri)
 Downlink packets are getting dropped in between PGW(Vikaspuri) -> SGW(Noida)

MME Pool Breakdown - Update
Nagpur
Ambala Delhi
Mumbai Nagpur Delhi
Ludhiana Lucknow
Ahmedabad Bhopal Jaipur
Shimla Ambala
Mumbai Lucknow Jammu

Ahmedabad Agra Srinagar
Chennai Bhubaneshwar Kolkata

Kochi Patna Asansol
Chennai
Hyderabad
Hyderabad Guwahati
Kolkata
Bangalore Shilong
Bangalore
Guwahati
MME pool break down activity completed.

-Balance re-arrangement after readiness of Jammu & Shilong

Core Network Learnings
130
ACME Related Issue Analysis Summary
Test Scope
• Test Drive Location - Bhopal , Ahmedabad, Kolkata, Analysis Summary
Mumbai, Delhi
• Test Duration %Avg. Call Drop 1.5%

• Bhopal - 12th to 15th Nov
• Ahmedabad – 15th to 17th Nov Failure Setup Fail 1.6%
• Kolkata – 24th Nov to 4th Dec
• Mumbai – 3rd Dec to 8th Dec
• Delhi – 7th Dec to Ongoing DNS DRA eNB EPC IMS IP
• Test Devices
MGCF OCS PCRF RF UE
• Bhopal - ACME
• Ahmedabad – Core Prime / ACME
• Kolkata – Core Prime / ACME 1%
• Mumbai – ACME 3% 0%
11%
• Delhi - ACME 6%
• Total Calls Originated
• Bhopal – 1713
• Ahmedabad – 2000 18% 24%
• Kolkata – 2000+
• Mumbai –1260
2%
• Delhi – 3000+ 2%
• Call Scenario
15%
• VoLTE to VoLTE , VoLTE to CS, 18%
• CS to VoLTE, VoLTE to PSTN

Key Failures and Action Items (1/2)
Sr.
Key Failures Action Taken Status Owner Team
No.
1 MGCF Issue resulting in SIP "480" MGCF Faulty Card replaced. Close MCGF
Other
2 MGCF Issue resulting in SIP “500" MGCF getting release from Other Operator POI Close
Operator
A2 parameter Configuration Corrected PAN India to avoid

3 eNB ERAB Setup Failures Close eNB
ERAB Setup Failure in RACE condition
PDCP layer Sequence number differences has caused

4 RTP Time Out while Intra eNB HO Close eNB
RTCP packet drops while Intra eNB HO.
OCS Delayed response post X2 HO -
5 Solution applied by OCS to respond within 1 sec. Close OCS
SIP 503
IMS to DNS interworking Failure - DNS Configuration updated for missing IMS Node's
6 Close DNS
SIP "500" FQDNs.
TAS/CSCF not forwarding 200 (OK) for PRACK and SIP

7 IMS internal Failure - SIP "500" Close IMS
Invite. Patch applied to fix the same.
8 IMS Failure SIP "480" Patch applied to fix the same. Close IMS

Key Failures and Action Items (2/2) Conitnues..
Sr. Key Failures Action Taken Status Owner Team
1. PCSCF sends fragmented SIP Update message for packet

9 IMS Failure SIP “480” >1518 size on UDP instead of sending on TCP --- Issue is Open IMS
with Nokia R&D team
IMS NOT forwarding SIP INVITE, 183
Patch need to be applied to fix the multiple message
10 Session Progress, SIP UPDATE, Open IMS
forwarding issue at TAS & SCSCF.
(200OK) for INVITE
UE find RLF during X2 Handover process due to X2
connectivity fails/X2 link not available.
X2 HO failure procedure fails due to
These causes QCI-1 bearer deactivation and hence call gets
11 “Tx2 timer expiry” or due to radio- Close MME
dropped.
procedure failure
Preservation mode is enabled in MME for SA1P cause code#
1, 6, 21,26 to make sure
Post successful X2 HO procedure, While requesting for S1U
Source change from old eNB to new eNB, (“path switch
Post Successful X2 HO , Call drops
request”) timer expires at eNB due to delay in response OCS/PCRF/
12 due to Patch Switch request timer Close
( i.e. >3 sec) from OCS or PCRF or DRA to respond to CCR-U DRA
expiry
message which is triggered after Modify bearer request
from MME to SAEGW.
While X2-HO between two eNB served by different MMEs,
S5 Interface issue - Packet loss btw SGW relocation get triggered while PGW remains same as
S-GW and P-GW due to SGW earlier.
13 relocation process while X2-HO - Because there was no IP routing configuration between Close Cisco
between two eNBs served by SGW and PGW each located at different sites, GTPU packets
different MMEs on the S5-interface were dropped.
- After IP routing configuration problem resolved.
[3] MO received SIP-503 “service unavailable” eNB
 Issue Description: QCI-1 bearer setup procedure failed in eNB
 Resolution: Issue got fixed after changing A2 Parameter configuration change in eNB which was
generating race condition within eNB and were not responding to MME for ERAB Setup requests.

[5] MO Received SIP – 503 – Issue at OCS OCS
 Issue Description: Delay from OCS for CCA response resulted in VoLTE Call setup Failure with ASR error
“Session released - service based local policy function aborted session".
 Resolution: Temporarily OCS has been bypassed for ACME testing.

[7] Received 500 Server Internal Error for INVITE IMS
Issue Description: IMS is not forwarding 200 Ok for Invite towards MO UE.
Resolution: IMS patch applied at TAS.

[13] X2 HO - SGW change Btwn Noida/Vikaspuri
• Possible Impacted Case: For S
6 Pair of SAEG
Ws
ingle PLMN, two Sites distribu
G (i.e. 36 combina ting traffic with S5.
tions to check)
• Applicable Sites - Mumbai, D
6 6
elhi and Chennai.
• Scenario:
• On Registration Co-located SA
G •
EGW get selected
Initial PGW provides IP to use
r
• Once HO happens between S
GW Group, New SGW will get
selected but PGW remains sa
me.
• Signaling & Traffic moves fro
m new SGW to old PGW on S
5 interface.
In Delhi Case – S5 GTP-C was ok hence bearer was

Modified correctly but S5 GTP-U was down hence
Voice/Data was not working.
[13] S5 Failure (X2 HO with SGW change) (1/2)
Cisco
S T

[13] S5 Failure (X2 HO with SGW change) (2/2)
 GTPU packets on the S5 Interface getting dropped due to missing IP Route
 Uplink packets are getting dropped in between SGW(Noida) -> PGW(Vikaspuri)
 Downlink packets are getting dropped in between PGW(Vikaspuri) -> SGW(Noida)

No Response for SIP Bye(Backbone) Issue-1
 Previous call was not completed. SIP Bye was sent for which response was not received.
 Even after multiple retransmission(till 12:30:32), even TCP Ack was not received.
 No SIP Invite seen at 12:30:14 in the network logs
City – Mumbai
Date – 06-Jan-2016
Similar issue is observe

d @ 13:10:02 also

UE Issue - No Response for SIP Invite
 MT UE was in connected mode
 MT UE was not responding the SIP Invite message sent.
 Even after multiple retransmissions, there was no response from MT UE.
City – Mumbai
Date – 06-Jan-2016

Call Setup failure due to User Inactivity(30 sec)
 MT UE has not answered the call within 30 sec.
 As User Inactivity timer is 30 sec, eNB has sent UE Context Release with reason USER-INAC
TIVITY.
 So MME deleted the QCI-1 bearer which resulted in VoLTE Call Setup failure.
City – Mumbai
Date – 07-Jan-2016

ACME Checks and Learnings - 2 PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

ACME Checks and Learnings - 2 PDF

Uploaded by

Copyright:

Available Formats

ACME Checks and Learning

24th January 2016

 Around 300 to 400 Calls for each scenario

2016 © Samsung Electronics 3

2016 © Samsung Electronics 4

2016 © Samsung Electronics 5

2016 © Samsung Electronics 6

2016 © Samsung Electronics 7

• Ip Packet is tunneled between P GW and UE using GPRS Tunneling Protocol GTP

2016 © Samsung Electronics 10

2016 © Samsung Electronics 11

CCR Credit Control Request Part 1 of 2

2016 © Samsung Electronics 12

2016 © Samsung Electronics 13

2016 © Samsung Electronics 14

2016 © Samsung Electronics 15

2016 © Samsung Electronics 16

2016 © Samsung Electronics 17

2016 © Samsung Electronics 18

2016 © Samsung Electronics 19

2016 © Samsung Electronics 20

2016 © Samsung Electronics 21

2016 © Samsung Electronics 22

2016 © Samsung Electronics 23

(-Q_Rxlevmin + 2 * Thresh_Serv_Low) dBm

-65dBm -70dBm -75dBm -80dBm -85dBm -90dBm -95dBm -100dBm -105dBm

2300 has higher priority than 1800

 Intra RAT (Intra FA / Inter FA) : Event A

2016 © Samsung Electronics 26

Serving becomes worse than threshold

2016 © Samsung Electronics 28

2016 © Samsung Electronics 29

-65dBm -70dBm -75dBm -80dBm -85dBm -90dBm -95dBm -100dBm -105dBm

Data Mode Band 40

2016 © Samsung Electronics

-80Bm -85dBm -90dBm -95dBm -100dBm -103dBm -110dBm -115dBm -120dBm

Call Data Service

Data Services to VoLTE

Data Services to VoLTE

Data Services to VoLTE

 Parameter Template 1: Configuration with Multi-band Implementation

will stop all other measurements

CELL_NUM FA_INDEX DUPLEX_TYPE STATUS EARFCN_UL EARFCN_DL PRIORITY

2016 © Samsung Electronics 34

QCI based A1 threshold (Meas Gap

Non-GBR Service (FTP DL) 27 80 80

A1 threshold(Meas Gap Deactivation) -100 dBm -50 dBm -50 dBm

Service Parameter 2.3G 1.8G 850M

 Wrong Neighbor impacting HO performance degradation

2016 © Samsung Electronics 37

 HO Failure Cases and Resolution

— Needs to turn on HO Success Rate Based NBR Deletion

Neighbor Cell Neighbor Cell

2016 © Samsung Electronics 38

Deployed at Ahemdabad, Ujjain and Hyderabad

2016 © Samsung Electronics 39

nrDelFlag = True (Enable PCI Conflict Notification

2016 © Samsung Electronics 40

Date NBR Addition NBR deletion

2016 © Samsung Electronics 42

#UE reports unknown PCI

#eNB orders UE to report eCGI by sending

#In order for UE to get eCGI information, UE