Guide To Troubleshooting Paging Failures-20130110-V1.0

Guide to Troubleshooting UPSS V100R003C00 Paging Failures
Issue Date
1.0 2012-12-05
HUAWEI TECHNOLOGIES CO., LTD.
Copyright Huawei Technologies Co., Ltd. 2012. All rights reserved.

No part of this document may be reproduced or transmitted in any form or by any means without prior written consent of Huawei Technologies Co., Ltd.
Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd. All other trademarks and trade names mentioned in this document are the property of their respective holders.
Notice
The purchased products, services and features are stipulated by the contract made between Huawei and the customer. All or part of the products, services and features described in this document may not be within the purchase scope or the usage scope. Unless otherwise specified in the contract, all statements, information, and recommendations in this document are provided "AS IS" without warranties, guarantees or representations of any kind, either express or implied. The information in this document is subject to change without notice. Every effort has been made in the preparation of this document to ensure accuracy of the contents, but all statements, information, and recommendations in this document do not constitute a warranty of any kind, express or implied.
Huawei Technologies Co., Ltd.

Address: Huawei Industrial Base Bantian, Longgang Shenzhen 518129 People's Republic of China Website: Email: http://www.huawei.com support@huawei.com
Issue 01 (2012-12-05)
Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd.
About This Document
About This Document

Author
prepared By Reviewed By UMTS Maintenance Department Zheng Jianhua (employee ID: 00042104) Qian Jin (employee ID: 00064231) Zhu Peng (employee ID: 00058079) Approved By Fang Ming (employee ID: 00061579) Dat e 2012-10-27 Dat e Dat e 2012-10-8 2012-10-25
Change History
Date 2012-10-25 2012-10-30 Revision version 0.9 1.0 change Description Completed the initial draft Completed the Modification Author Zeng Wei (employee ID: 00161296) Zeng Wei (employee ID: 00161296)
Keywords
Paging, PCH congestion, paging delay
Issue 01 (2012-12-05)
ii
About This Document
Abstract
This document describes how to troubleshoot paging failure problems by checking and analyzing all rounds in different network operation scenarios, guides frontline personnel to analyze networks, solve paging failure problems, and separate work of network optimization personnel, technical service personnel, and Research & Development (R&D) personnel from each other, and provides fixed actions to discover factors affecting paging success rates. The primary goal of this document is to guide frontline personnel to quickly handle simple problems and effectively feed back difficult problems to R&D personnel, and improve the troubleshooting efficiency.
Issue 01 (2012-12-05)
iii
Contents
Contents
About This Document.......................................................................ii
Issue 01 (2012-12-05)
iv
4 Description of Troubleshooting Procedures for Paging Failures
Overview
This document focuses on analyzing and resolving paging failures during the initial optimization of new and swapped networks and guides frontline personnel to rapidly analyze and optimize networks to ensure that paging problems can be rapidly located and solved, improving efficiency. Differences may come with tools, traffic statistics, parameters, and versions. However, analysis methods described in this document are applicable to various base station controller (BSC) versions.
Issue 01 (2012-12-05)
Clarification of Paging Problems

Paging success rate, one of the network performance indicators (KPIs), indicates paging failures. The evaluation for paging success rates is based on paging success rates on the core network (CN). Paging success rates on the UTRAN side are auxiliary counters for locating problems.
2.1 Definitions of Paging KPIs in Traffic Statistics

2.1.1 KPIs of Paging Success Rates on the CN Side
3G Paging Success Rate (%) = (Number of First Paging Responses from Iu Interface + Number of Repeated Paging Responses from Iu Interface) / Number of First Paging to Iu Interface Paging success rates on the CN side is mainly measured in LAC areas. The preceding formula shows that paging success rate KPIS on the CN side presented to customers are not First Paging Success Rate KPIs.
2.1.2 KPIs of Paging Success Rates on the RNC Side

VS.RANAP.Paging.Succ.IdleUE.Rate = VS.RANAP.Paging.Succ.IdleUE/VS.RANAP.Paging.Att.IdleUE On the UTRAN side, only this preceding formula is used to measure paging success rates. Paging KPIs are related to two NEs, therefore, confirm KPIs of the two NEs when resolving paging failures. Action 1: confirming the type of a paging failure procedure.
2.2 Sources and Clarification of Problems

Source: Paging success rates deteriorate due to network swapping, complaints of VIP users, NE upgrade or feature enabling. The following table lists sources of paging failures.
Issue 01 (2012-12-05) Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd. 2
Source of Paging Failures Unqualified paging success rate of swapped or new networks
Data Source Analysis KPI mapping relationships between peer vendors and Huawei before and after swapping Traffic statistics KPIs between peer vendors and Huawei before and after swapping Paging success rates on the CN side before swapping
Criteria 1. Analyze KPI mapping accuracy including denominators and numerators) and check that KPIs are proper. 2. Compare traffic statistics for at least one week before and after swapping and check whether the traffic statistics meet related requirements. 1. Analyze impacts from upgrading and feature enabling scenarios on paging success rates. 2. Determine deterioration by comparing the corresponding days. 3. Compare KPIs three days before and after deterioration and exclude impacts of traffic periods.
Deterioration of paging success rates of inventory networks
Original traffic statistics KPIs before and after deterioration
VIP users complain about calling difficulties or long delay.
Drive test (DT) data and traced signaling
Signaling analysis shows that paging failures occur, which is due to causes on the wireless network rather than the CN or UEs.
Issue 01 (2012-12-05)
3
Dimension Application scenario
Troubleshooting Process for Paging Failures
3.1 Common Scenarios of Paging Failures and Possible Causes

Paging failures can be categorized based on application scenarios and paging procedures, as described in the following table. Table 1.1 Paging failures categorized based on application scenarios Paging Problem Type Unqualified paging success rates on swapped networks Unqualified paging success rates on new networks Deterioration of paging success rates of inventory networks Description Unqualified paging success rates on the CN side peer vendor's swapped networks Unqualified paging success rates in LAC areas of new networks On the inventory networks with stable KPIs, paging success rates suddenly deteriorate. Paging success rate deterioration includes non-operation deterioration and operation deterioration (version upgrades, parameter modification, cutover, and feature activation). Criteria Compare traffic statistics within the latest one week before and after swapping. Observe whether traffic statistics for at least one week meet related requirements. Compare paging success rates of the corresponding days in a week (three days before and after deterioration).
Issue 01 (2012-12-05)
3.1.2 Categorizing Common Causes and Corresponding Actions of Paging Failures by Application Scopes
Paging Success Rates on Swapped Networks Do Not Meet Related Requirements.
Paging success rate KPIs may not be available on the UTRAN side of peer vendors' networks, therefore, take paging success rates on the CN side to verify whether they meet the standard. After swapping, paging success rates are lower than those on the original networks. Common reasons for this problem are as follows: 1. Algorithm/Parameter mapping problems: including incorrect mappings of RNC-level and site-level parameters. In addition, some algorithms and parameters of Huawei devices are not accurately mapped with those of original networks. This may result in changes of traffic models, calling control processes, resource allocation, wireless coverage, and network traffic distribution, affecting paging performance. (action 2) Hardware and transmission faults: alarms of device board exceptions and transmission faults. (action 3) NE versions: NE versions are not properly selected (an early version has many problems) or version defects result in low paging success rates. (action 5) Network planning: neighboring cell inheritance errors, LAC/RAC area planning differences, and insufficient paging channels over air interfaces result in low paging success rates. (action 4) RF channel problems: Antenna system faults, RF connection faults, antenna match, antenna tilt adjustment, and internal and external interference of the antenna system result in low paging success rates. (action 6) Long-term impact: In the scenarios with a long swapping period of time, seasonal changes in traffic models and KPIs take place. In this case, increased traffic volumes result in air interface capacity limitation, causing paging success rate deterioration. (action 9) Impacts of external unexpected factors: For example, during swapping or after swapping, user actions and traffic distribution greatly change due to external factors such as charging adjustments of operators, holidays, activities, and climates. Therefore, paging success rates after swapping are affected. (action 10) Exceptions on the CN side: In a short period of time, paging success rates may deteriorate suddenly or fluctuate dramatically. After the RAN-side impacts are eliminated, this problem may be caused by exceptions on the CN side, including exceptions in operations and terminal servers (for example, faults of iTunes and BlackBerry servers may increase the number of paging users), and IP or port scanning in networks in the case of allocating IP addresses to UEs by the CN. (action 11) Abnormal UEs and users: Poor compatibility of UEs, business users, and malicious users result in paging failures. (action 11)
2. 3. 4.
5.
6.
7.
8.
9.
Paging Success Rates of New Networks Do Not Meet Related Requirements.

Common causes are as follows: 10. Inappropriate network parameter planning: algorithm/parameter planning, network
policy planning, and improper neighboring cell and frequency planning result in low paging success rates. (action 2) 11. Hardware and link faults: Devices, board, and transmission faults result in low paging success rates. (action 3) 12. NE version problems: NE versions are not properly selected (an early version has many problems) or precautions actions are not performed, which results in low paging success rates. (action 4) 13. Network planning optimization problems: Inappropriate site planning, networking inheritance policy planning, LAC/RAC planning, air interface resources, and board resources cause coverage problems, affecting paging success rates. (action 5) 14. RF channel problems: Internal and external inference and RF channel connection faults cause uplink and downlink coverage problems, affect paging success rates. (action 6) 15. Impacts of external unexpected factors: For example, during swapping or after swapping, user actions and traffic distribution greatly change due to external factors such as charging adjustments of operators, holidays, activities, and climates. Therefore, paging success rates on new networks are affected. (action 9) 16. Exceptions on the CN side: In a short period of time, paging success rates may deteriorate suddenly or fluctuate dramatically. After the RAN-side impacts are eliminated, this problem may be caused by exceptions on the CN side, including exceptions in operations and terminal servers (for example, faults of iTunes and BlackBerry servers may increase the number of paging users), and IP or port scanning in networks in the case of allocating IP addresses to UEs by the CN. (action 10) 17. Long-term trend impact: Seasonal changes in traffic models and KPIs affect paging success rates to a certain extent. For example, based on historical experience, the time when new networks are deployed, paging success rates are the lowest throughout the year. (action 8) 18. Abnormal UEs and users: Poor compatibility of UEs, business users, and malicious users result in paging failures. (action 11)
Paging Success Rates of Inventory Networks Deteriorate

Common causes are as follows: 19. Parameter adjustment impacts: Inappropriate algorithms, parameters, network policies, and neighboring adjustment result in paging success rate deterioration. In the long-term network operation, traffic models and distribution gradually change. Original algorithms and parameters do not adapt to the current network status, which results in slow paging success rate deterioration. (action 2) 20. Hardware and link faults: Transmission, device board faults or adjustment result in sudden paging success rate deterioration. (action 3 and action 10) 21. Network planning problems: Inappropriate site planning, network policies affect network structures and coverage, resulting in paging success rate deterioration. In the long-term network operation, traffic models and distribution gradually change and inappropriate site planning, antenna design, and power planning cause coverage and capacity problems, affecting paging success rates. (action 5) 22. RF channel problems: Sudden external interference and internal interference such as antenna system faults and looseness causes sudden paging success rate deterioration. (action 6) 23. Long-term trend impact: Seasonal changes in traffic models and KPIs results in slow fluctuation in paging success rates. 24. Impacts of external unexpected factors: For example, during swapping or after swapping,
user actions and traffic distribution greatly change due to external factors such as charging adjustments of operators, holidays, activities, and climates. Therefore, paging success rates deteriorate after swapping. 25. Exceptions on the CN side: In a short period of time, paging success rates may deteriorate suddenly or fluctuate dramatically. After the UTRAN-side impacts are eliminated, this problem may be caused by exceptions on the CN side, including exceptions in operations and terminal servers (for example, faults of iTunes and BlackBerry servers may increase the number of paging users), and IP or port scanning in networks in the case of allocating IP addresses to UEs by the CN. (action 10) 26. NE and version changes: NE replacement and version upgrades result in sudden paging success rate deterioration.(action 4) 27. Abnormal UEs and users: Poor compatibility of UEs, business users, and malicious users result in paging failures. (action 11)
VIP users Complain about Calling Difficulties or Long Delay.

For the preceding problems, contact frontline personnel to collect related data and feed back these problems to R&D engineers.
3.1.3 Classifying Paging Failures based on Paging Procedures
Issue 01 (2012-12-05)
Type 1: Paging Success Rates Drop Dramatically on the CN Side But Those in Idle Mode on the RNC Side Remain the Same.
Common causes are as follows: Paging messages are discarded over the Iu interface. 1. 2. RRC setup success rates of the called party decrease. RRC is successfully set up but no Initial Direct Transfer message is received on the CN.
Type 2: Paging Success Rates on the CN Side and Those in Idle Mode on the RNC Side Drop Dramatically.
Common causes are as follows: 3. 4. 5. 6. 7. 8. 9. Paging messages are not sent over air interfaces due to internal problems within the RNC. Paging congestions occur because of air interface capability limitation (VS.RRC.Paging1.Loss.PCHCong.Cell). Paging success rate decrease may also result from weak coverage, paging black hole cells, NodeB faults, and cell out-of-service problems. Registered RRC setup success rates decrease. RRC is successfully set up but LAU/RAU procedure fails. Location upgrade of UEs in idle mode or inter-RAT cell reselection is frequent. UEs in CELL-PCH mode frequently reselect cells. Some UEs receive no paging response messages from the air interfaces in shared networks.
10. Some parameters are improperly set, therefore, UEs cannot respond to paging messages in a certain period.
Type 3: Paging Success Rates are Normal on the CN Side But Those on the UTRAN Side Deteriorate.
Common causes are as follows: 11. UEs are not located in areas under the local RNC. 12. The first paging success rate on the CN side decreases but repeated paging success rate increases. 13. The CELL-PCH/FD/EFD feature is enabled on the RNC.
3.2 Analysis Report for Paging Problems

Paging problems are analyzed and handled based on the preceding troubleshooting procedures. Based on the detailed description of required actions for RRC and RAB paging problems, frontline personnel perform analysis and checking required in 11 required actions and provide an analysis report. If frontline personnel experience any trouble in analyzing and handling paging problems, they can turn to PSEs and R&D personnel for help. The report template is as follows:
Issue 01 (2012-12-05)
Anal ysi s Report on Pagi ng Probl em s of XX Proj ect . doc
Provide the analysis report to PSEs and R&D personnel for review and specify analysis conclusion, handling measures and optimization suggestions. Analysis and checking in 11 required actions are described in the following several chapters.
Issue 01 (2012-12-05)
Description of Troubleshooting Procedures for Paging Failures
4.1 Action 1: Confirming the Type of a Paging Failure Procedure

4.1.1 Descriptions
Locate paging failures roughly and confirm handling orientation according to paging success rate KPIs on the CN and UTRAN side.
4.1.2 Data Sources and Processing Tools

Data sources: traffic statistics on the UTRAN and CN side. Tools: OMStar and FMA.
4.1.3 Results
Verify whether the current paging problem is one of the followings: 1. 2. 3. Type 1: If paging success rates on the CN side decrease but hardly change on the UTRAN side, it is an Iu interface paging problem. Type 2: If paging success rates decrease on both the CN and UTRAN side, it is a Uu interface paging problem. Type 3: Paging success rates hardly change on the CN side but decrease dramatically on the UTRAN side, it is an improper parameter setting problem.
4.2 Action 2: Checking Parameters

4.2.1 Checking Parameter Mapping
Descriptions
Scenario: Paging success rates on swapped networks do not meet the requirements. The
problems are preceding type 1 and type 2 problems. Criteria: Check parameter mapping if paging success rates on the CN or UTRAN side after swapping do not meet the requirements.
Data Sources and Processing Tools

Data Source 1 Huawei MML configuration files, peer vendors' MML configuration files, and parameter lists. Processing Tool Tools provided in parameter mapping table
Steps for Analyzing Data

1. Obtain parameters of peer vendors for checking parameter mapping based on Huawei mapping system or provide the review conclusion of initial-phase parameter mapping. For details about the parameter mapping process, see xxx. Related parameter/feature mapping guides are incorporated into High Efficiency UMTS Swap Solution (V2.0), which is available at http://wss.huawei.com/sites/gts/gul/Shared%20Documents/Forms/AllItems.aspx 2. Check the consistency between mapping results and actual configurations based on mapping system and current configurations.
Results
3. 4. Provide parameter mapping review result and list parameters that cannot be accurately mapped. Provide the consistency check result between the parameter mapping review and actually configured parameters and list inconsistent parameters.
Closed Actions
5. 6. Analyze the impact of parameters that cannot be accurately mapped and adjust parameters to observe the effect. Analyze the impact of actual parameters that are not consistent with mapping parameters and adjust parameters to observe the effect.
4.2.2 Checking Reasonableness of Parameter Configurations

Descriptions
Scenario: optimization scenario. The problems are preceding type 1 and type 2 problems. Criteria: Check reasonableness in the paging success rate optimization scenario when the paging success rate slowly deteriorates and optimization is required.
Issue 01 (2012-12-05)
11

Data Source 1 Configuration files Processing Tool FMA

For details, see UPSS V100R003C00 Guide to Full Parameter Check.
Results
Based on the rule for checking parameter reasonableness, provide the check result of pagingrelated parameters and list the current parameter values and values specified in the check rule. After enabling FMA function for checking parameters, a file with a name extension of .csv is created in the target folder. The comparison results between parameter values of live networks and baseline values are listed in column I and comparison results between parameter values of live networks and parameter reasonableness results are listed in column J.
For details, see UPSS V100R003C00 Guide to Full Parameter Check.
Closed Actions
Modify parameters based on check results of improper paging-related parameters in the provided .csv file.
Issue 01 (2012-12-05)
12
4.2.3 Checking Parameter Configuration Modifications (Operation Checking)

Descriptions
Scenario: Paging success rates suddenly deteriorate due to upgrade or operation. The problems are preceding type 1 and type 2 problems. Criteria: Check operations and parameter modifications when the paging success rate suddenly deteriorates.

Data Source 1 2 Configuration files before and after paging success rate deterioration Operation logs Processing Tool CME FMA

1. Based on the time when problems occur, analyze the operation logs on the RNC and NodeB before, during, and after the problems take place, altogether three days. Analyze only logs related to operations such as MOD, RMV, SET, ADD, RST, ACT, DEA, and SWP. All RNC-class operation logs must be analyzed.) Compare the configuration files before and after deterioration to analyze parameter differences and provide all-parameter check differences.
2.
Results
3. 4. Provide a check list of operation logs and list non-query operation and their impacts. Provide comparison results of all-parameter differences. CMD Name OP CMD Name OP Object OpCmd& Message Impact
OP Time 2012-03-22 11:00 2012-03-23 11:00 2012-03-24 11:00 2012-03-25 11:00
Issue 01 (2012-12-05)
13
The following table lists examples: OP Time 2012-10-20 11:05 CMD Name Set the DPU configuration data. OP CMD SET UDPUCF GDATA OP Object RNC OpCmd& Message SET UDPUCFG DATA: MaccPageR epeatTimes =2; Impact Paging messages over air interfaces may increase and paging congestions may occur.
Closed Actions
5. 6. Analyze the impact of operation and parameter differences and provide corresponding solutions. Perform action 3 if no obvious operation causes paging success rate deterioration.
4.3 Action 3: Checking Device and Transmission Faults (Performed by Frontline Engineers)
4.3.1 Checking Alarms
Descriptions
Scenario: Check all scenarios where paging success rates decrease, including deterioration or optimization scenarios. The problems are preceding type 1 and type 2 problems. Criteria: According to the problem type confirmed in action 1, analyze site alarms in LAC areas (including site alarms and site-related RNC alarms) and entire RNC alarms (such as interface board alarms and SPU alarms).

Data Source 1 Alarm logs (RNC and NodeB) Processing Tool FMA

According to the problem scope confirmed in action 1, analyze RNC alarm logs and site alarm logs within the LAC areas. For details about alarm information and check methods, see alarm description in product documents and alarm clearing guides released by the maintenance department. Network
optimization engineers need only to submit device and link alarms to device maintenance engineers.
Results
Provide analysis results of device faults and alarms, including alarm lists, cell lists, and alarm checking results.
Closed Actions
Rectify device faults and clear alarms. Observe KPI recovery status.
4.3.2 Checking Transmission Fault Alarms

Descriptions
Scenario: Check all scenarios where paging success rates decrease, including deterioration or optimization scenarios. The problems are preceding type 1 and type 2 problems. Criteria: 1. 2. The confirmed problem type is type 1, namely, transmission alarms are generated over the Iu interface. The confirmed problem type is type 2 in action 1, namely, transmission alarms are generated on NodeBs in LAC areas.

Data Source 1 2 3 Traffic statistics Configuration files Real-time test data Processing Tool FMA and OMStar FMA and OMStar None

Check transmission link configurations Check transmission parameters of the RNC and NodeB by using the link status checking function of the maintaining SOP. Check the transmission consistency of the paging failure top cells (manual checking is required because no tools temporarily support the checking). Some parameters are not properly configured by checking link configurations of the maintaining SOP link, as shown in the following figure:
Issue 01 (2012-12-05)
15
Confirm configuration problems onsite as soon as possible. Check the transmission link status. Check link status of the control plane by using the link status checking function of the maintenance SOP. Link status of the user plane needs manual checking. The following table lists transmission link status by using the maintenance SOP.
Confirm whether the link status is normal onsite as soon as possible. Check the transmission link load. Evaluate and analyze the link load by using the link load evaluation function of the maintenance SOP. The following table lists the evaluation results of transmission link load.
Issue 01 (2012-12-05)
16
The evaluation result shows that transmission congestion exists in many Iub interfaces.
Confirm the link congestion status onsite as soon as possible. Check transmission link quality The maintenance SOP supports IP transmission quality check and exports corresponding results. If transmission is in IP mode, enable the ping checking function of the IPPATH and evaluate transmission link quality using the IP transmission QoS checking function of the maintenance SOP. For the ATM transmission quality, only Iub interface has the PM function and the quality of the Iu/Iur interface cannot be checked. The following table lists the link load evaluation result of the maintaining SOP link.
Issue 01 (2012-12-05)
17
The following table shows that the IPPATH transmission quality of some NodeBs is poor.
Confirm whether the intermediate transmission network has problems onsite. The following document provides the guide to manually checking transmission. See RAN12 IP QoS Transmission Trouble Shooting Guide.
Results
Provide transmission checking items and results.
Closed Actions
Rectify transmission faults and observe KPIs.
4.4 Action 4: Checking Paging Channel Congestion

Descriptions
Scenario: Version upgrade results in paging KPI deterioration. Criteria: KPIs slowly deteriorate after version upgrading or KPIs suddenly deteriorate within one week following version upgrading.

Data Source 1 3 Traffic statistics MML configuration files Processing Tool OMStar FMA

1. 2. Confirm the value of VS.RRC.Paging1.Loss.PCHCong.Cell based on traffic statistics and the value of IU.Paging.Congestion.Cell.Ratio is greater than 1%. The calculation shows that the modified paging channel utilization in all cells is more than 80% and physical channel utilization is greater than 25%. The formula is as follows: Modified paging channel utilization = (VS.PCH.Utilization.Cell.Rate x 6)/7 Physical channel utilization = VS.UTRAN.AttPaging1/2160000
Issue 01 (2012-12-05)
18
Results
Provide values of preceding counters in all cells.
Closed Actions
Take optimization measures by referring to UMTS Access and Paging Principles + Troubleshooting Methods + Cases + Deliverables. Some measures and actions have both positive and negative gains. Therefore, confirm the impact with R&D personnel before taking actions.
4.5 Action 5: Checking Versions

4.5.1 Analyzing RNC and NodeB Version Difference (Performed by R&D Personnel with the Assistance of Frontline Personnel)
Descriptions
Scenario: Version upgrade results in paging KPI deterioration. Criteria: KPIs slowly deteriorate after version upgrading or KPIs suddenly deteriorate within one week following version upgrading.

Data Source 1 2 Release Notes (RNs) MML files before and after the version upgrade Processing Tool None Beyond Compare and OMSTar

1. Analyze RNs of RNCs/NodeBs with upgrade paging success rate before and after deterioration. List version incorporation problems, algorithm change, new features, and the possible effects. Compare RNC/NodeB configuration files to list the parameter configuration differences, and the possible impact of each difference.
2.
Results
3. 4. Provide impact analysis results of version incorporation problems, algorithm changes, and new features. Provide analysis results of version parameter difference impacts.
Closed Actions
5.
Issue 01 (2012-12-05)
Provide a solution to version problems affecting the paging success rate and observe the
Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd. 19
results after implementing measures. 6. If you cannot directly draw any conclusion from version differences, analyze the specific failure range and cause.
4.5.2 Checking RNC and NodeB Versions for Known Problems (Performed by Frontline Engineers)
Descriptions
Scenario: deterioration and optimization scenarios Criteria: Confirm the problem range and cause and eliminate impacts from parameter changes based on actions 1 and 2. Eliminate the impacts of device and link defaults. Check the known version problems.

Data Source 1 2 3 Configuration files Case database Release Notes (RNs) Processing Tool FMA None None

1. 2. Obtain xx to check whether known version problems occur. Analyze the latest RN to confirm whether there are version defects deteriorating the paging success rate based on problem range, initial problem causes, and network operations. Check the case database to confirm whether there are known problems deteriorating paging success rate based on problem range, initial problem causes, and network operations.
3.
Results
4. 5. Provide the version RN checking result: List the name of the checked RNs and check result. (List the RN defect descriptions if related defects exist.) Provide case database searching results: List keywords for searching and checking result. (List the link and content of cases if there are any related cases.)
Closed Actions
6. 7. Confirm related RN defects, provide and perform corresponding solutions. Record results. Confirm associated cases and perform solutions. Record results.
Issue 01 (2012-12-05)
20
4.6 Action 6: Checking SPU Loads

4.6.1 Descriptions
Scenario: all deterioration or optimization scenarios of paging problems of type 1. Paging problems of type 2 are caused by decreased RRC success rates of registered RRCs. Criteria: Check SPU loads when the VS.Paging.FC.Disc.Num.CPUS counter discarded over the Iu interface has a value.

Data Source 1 2 3 Traffic statistics PCHR DEBUG, CallFault, and CHR Processing Tool OMStar FMA FMA
4.6.3 Steps for Analyzing Data

1. 2. Confirm whether values are available for VS.RANAP.CsPaging.Loss or VS.RANAP.PsPaging.Loss or VS.Paging.FC.Disc.Num.CPUS. Evaluate network loads using CMA scripts of an OMStar and generate a solution. Import traffic statistics in the OMStar to run scripts.
CMA scripts and methods for running the scripts are described in the following attachment:
Capacity Estimation Tool (CMA).rar
4.6.4 Results
Provide evaluation results in Excel using related tools. SPU loads are described in the following figure:
Issue 01 (2012-12-05)
21
Solutions for SPU loads are described in the following figure:
4.6.5 Closed Actions

Solutions generated from tools usually involve swapping sites, changing binding relationships, and expanding capacity. Observe the changes in paging success rates and the following three counters including VS.RANAP.CsPaging.Loss, VS.RANAP.PsPaging.Loss, and VS.Paging.FC.Disc.Num.CPUS.
4.7 Action 7: Checking RRC Setup Success Rates of Called Parties and Registered Users
4.7.1 Descriptions
Scenario: Paging failures of type 1 are due to low RRC success rates of the called parties and failures of type 2 are caused by low RRC success rates of the registered users. Criteria: Traffic statistics show that RRC success rates of the called parties or registered users are low.

See required actions and handling measures for RRC access success rates in Guide to Troubleshooting UPSS V100R003C00 Access Failure Problems.
Issue 01 (2012-12-05)
22
4.8 Action 8: Checking Coverage (Performed by Network Optimization Engineers)

Radio link coverage is a key to network performance. Coverage problems, such as, poor or overlapped coverage would affect network access performance. In this action, try to find out poor network RF coverage, excluding the poor coverage caused by RF channels.
4.8.1 Descriptions
Scenario: all deterioration and optimization scenarios where paging failures of type 2 occur. Criteria: According to action 1, it is a paging problem of type 2. No exceptions are found in previous checking. In respect of poor uplink and downlink coverage, check the coverage after checking parameters, devices, and RF channels.

Data Source 1 2 3 Configuration files Traffic statistics Drive test data Processing Tool FMA OMStar Actixe or corresponding data processing tools

1. See seven coverage-related analyses in MTS Network Planning and Optimization Principles + Troubleshooting Methods + Cases + Deliverables .The detailed analysis is described in the following table. Trigger conditions: mandatory. Check the following items: Networking policy Power Handover setting Parameters inconsistency before and after swapping or engineering adjustment Checking coverage Checking traffic statistics Trigger conditions: mandatory. Check coverage problems caused by insufficient sites and carriers, and poor indoor coverage. Such coverage information can be used to monitor overall coverage. Trigger conditions: mandatory. Check coverage problems caused by high traffic volumes. Traffic statistic can also be used to monitor the overall situation of traffics.
Checking Paramete rs
Issue 01 (2012-12-05)
23
Analyzing basic RF informati on
Trigger conditions: mandatory. Check coverage problems caused by missing neighboring cell configuration and improper parameter setting (delayed handover). Identify areas with pilot pollution and high RTWP. Solve pilot pollution problems by combining parameter checking. For high RTWP problems, check RF quality, external interference, and parameters by combing channel check and capacity check. Trigger conditions: This action is required after network swapping or engineering adjustments or when problems must be precisely located. Coverage problems basically can be identified. The prerequisite for identifying coverage problems is to check RF quality, external interference, and parameters by combing channel check and capacity check. Trigger conditions: This action is required when the MR function is enabled on live networks. Combine checking for insufficient sites and parameter problems (pilot pollution) to identify areas with high traffic volume. Trigger conditions: This action is required in engineering adjustment. Check coverage problems caused by parameter problems, for example, parameter adjustment.
Analyzing drive test data
Analyzing MRs
Emulatin g the system
4.8.4 Results
Provide an analysis report based on the coverage-related analysis in UMTS Network Planning and Optimization Principles + Troubleshooting Methods + Cases + Deliverables.

If coverage changes dramatically in comparison and affects paging success rates, exclude impacts from parameter changes and RF channel changes. Perform adjustment after confirming the coverage changes. Network optimization engineers provide specific measures as required.
4.9 Action 9: Comparing Long-Term Change Trends of Traffic Volumes (Seasonal Changes and Increase in the Number of Users)
Paging success rates may come with similar fluctuation or long-term deterioration throughout the year or in two years. If the deterioration trends of paging success rate are similar for two years, this is related to seasons, grand festivals, gatherings, and traveling seasons, which can be clarified by analysis. If paging success rate deterioration persists for long period of time, analyze whether this is related to persistent increase in the number of users. If yes, provide corresponding measures.
4.9.1 Descriptions
Scenario: Paging success rates slowly deteriorate or fluctuate in a long term.
Criteria: Paging success rates have long-term deterioration or slow deterioration instead of obvious deterioration in a long period.

Data Source 1 Traffic statistics Processing Tool PRS Nastar M2000

Analyze the long-term change trend of the following counters. 1. 2. 3. Number of RRC attempts of the called party VS.RRC.Paging1.Loss.PCHCong.Cell VS.RANAP.CsPaging.Loss and VS.RANAP.PsPaging.Loss
4.9.4 Results
Provide long-term change trend results and an association analysis result of the preceding counters, specifically, provide the relationship between paging success rates and the following information: Number of RRC attempts of the called party VS.RRC.Paging1.Loss.PCHCong.Cell VS.RANAP.CsPaging.Loss VS.RANAP.PsPaging.Loss Season change trend Network adjustment time Increase time in the number of abnormal terminals Networks accessing time of new users

1. 2. If paging success rate changes are caused by seasonal changes in traffic volumes or environment, provide analysis reports and explain this situation to customers. Take the following measures to resolve the paging success rate deterioration that are caused by increase in the number of users. a b Analyze whether radio resource bottlenecks exist. If the bottlenecks exist, expand capacity and balance traffic. Go to Action 12: Analyzing UEs Exceptions and Trace if resource bottlenecks do not exist.
Issue 01 (2012-12-05)
25
4.10 Action 10: Checking Unexpected Factors (Charging, Weather, and Gatherings)
In a short period of time, the paging success rate may deteriorate suddenly or fluctuate obviously. If the paging success rate changes are not caused by network adjustments, the changes may be caused by unexpected factors, such as charging policy adjustment, bad weathers, festivals and gathering activities. Provide measures based on whether these unexpected factors can be expected or controlled on networks.
4.10.1 Descriptions
Scenario: Paging success rate deteriorates suddenly in a short period of time. Criteria: Network adjustment impacts, device faults and channel faults are eliminated in actions 2, 3, and 8 respectively.

Data Source 1 Traffic statistics Processing Tool OMStar

1. 2. 3. Analyze the long-term change trend of paging success rates and traffic volumes, which is applicable to short-term unexpected changes of paging success rates. Analyze operation records and network adjustment records to check whether unexpected paging success rate deterioration is caused by network adjustment. Analyze whether unexpected events, such as, charges, festivals and gathering activities, happen simultaneously or continuously when, before or after the paging success rate changes suddenly if network adjustment is not considered.
4.10.4 Results
Provide relationships between the trend of paging success rates before and after deterioration and time of network adjustment and unexpected changes.

1. If paging success rate changes are caused by unexpected factors such as charging, weather, and gatherings that cannot be controlled, provide analysis reports and explain this situation to customers. Provide capacity expansion and hardware adjustment solutions if the paging success rate changes due to predictable and controllable factors, such as gathering activities, festivals and activities. Make a summary and provide cases.
2.
3.
Issue 01 (2012-12-05)
26
4.11 Action 11: Checking Exceptions on the CN Side

In a short period of time, paging success rates may deteriorate suddenly or fluctuate dramatically. After the RAN-side impacts are eliminated, this problem may be caused by exceptions on the CN side, including exceptions in operations and terminal servers (for example, faults of iTunes and BlackBerry servers may increase the number of paging users), and IP or port scanning in networks in the case of allocating IP addresses to UEs by the CN.
4.11.1 Descriptions
Scenario: Paging success rate deteriorates suddenly in a short period of time. Criteria: Eliminate network adjustment impacts in action 2, and device faults, low RRC success rates, and unexpected factors in actions 3, 7, and 10 respectively.

Data Source 1 2 3 Traffic statistics PCHR Internet information Processing Tool Nastar NPMaster None

1. 2. 3. Analyze the traffic statistics to confirm the time point when the paging success rate deteriorates. Obtain and analyze network operations on the CN side to check whether the CN adjustment and sudden deterioration of paging success rate occur at the same time point. Analyze the PCHR data before and after deterioration to check whether the sudden increase paging failures are caused by a certain terminal. If such a UE exists, obtain Internet information to check whether the terminal server is abnormal. Analyze user-plane data to confirm whether IP addresses of UEs are those of public networks and whether IP or port scanning occurs on public networks.
4.
4.11.4 Results
1. 2. Provide the relationship between the paging success rate change trend before and after deterioration and the time points of network operations on the CN side. Provide distribution results of UEs contributing to sudden increased paging failures If a top UE exists, obtain terminal server information.

1. 2. For paging counters deterioration due to network operations on the CN side, provide an impact analysis report and roll back network operations. If the paging success rate changes are caused by unexpected factors (terminal server exceptions) that cannot be controlled, provide analysis reports and explain this situation
Issue 01 (2012-12-05)
to customers. 3. Make a summary and provide cases.
4.12 Action 11: Analyzing UE Exceptions and Trace

4.12.1 Descriptions
Scenario: Paging success rates do not meet related requirements and deteriorate in scenarios of networking swapping or network sharing policy adjustment. Criteria: Top users or top UEs cause unqualified paging success rates or paging success rate deterioration.

Data Source 1 2 Tracing data on the Iu interface UE logs Processing Tool FMA UE log processing tools, such as Probe and Actixe

1. 2. 3. Analyze tracing data at the Iu interface to identify the IMSIs in paging failures. Check IMEIs of these IMSIs on the CN side to confirm whether most IMSIs belong to the same type of UEs. Collect UE logs to analyze causes of paging failures. Collection guidance for UE logs of different UEs is available at http://3ms.huawei.com/hi/group/2299/wiki_2486839.html a The following describes the methods for obtaining mapping between IMSIs and MSISDNs: Check the mapping in batch by running the LST SUBDATA command on the USCDB of the HLR OMU and export the results in a TXT file. Enter a single IMSI to check the mapping by running the DSP USERINFO command on the OM platform of the MSC and export the results in a TXT file. Check the mapping based on IMSIs by running the MM CONTEXT: DSP MMBYIMSI command on the SGSN LMT. Analyze CHRs of the CN based on the CHR database. Turn on the IMSI reporting switch on the SGSN and collect CHR data. Convert the CHR data and import them into tools. Check the mapping based on the SQL data.
This method is used to check the mapping in a single CHR record on the CN but cannot be associated with IMSIs on the RAN side. Therefore, this method is not recommended.
b c d
Export the data of online users from the GU HLR, and save the data in a board of the DTL. The data file will be automatically uploaded to an FTP or SFPT server.
Issue 01 (2012-12-05)
The exported file is compressed into a .tar.gz packet, saved in /opt/uscdb/MemUserExp, and uploaded to an FTP or SFTP server. Exported table contains the mapping between IMSIs and MSISDN. W_COMMON_SER_DATA (table ID is 12), data is shown in the following table: SID 1 2 3 IMSI 460009900088001 460009900088002 460009900088003 MSISDN 8613590208001 8613590208002 8613590208003
For details, see USCDB V100R002C06 Guide to Exporting Subscriber Data Online. http://support.huawei.com/support/pages/kbcenter/view/product.do? actionFlag=detailProductSimple&web_doc_id=SC0000692445&doc_type=VersionDoc&doc _type=VersionDoc
This operation comes with high risks. Therefore, confirm whether to perform this operation based on specific sites and number of users. For a small number of users, export data through the PGW.
This operation is performed with coordination between R&D personnel and frontline personnel.
4.12.4 Results
Provide records of top user paging analysis.

Paging failures of top UEs are usually caused by incompatibility with special networking. In this case, upgrade software through the IOT. If this upgrade experiences difficulties, adjust the networking mode to avoid paging failures.
Issue 01 (2012-12-05)
29
Cases of Required Actions for Paging Problems
5.1 Cases of Parameter Checking

Problem On August 18 in office I of country M, paging flow control alarms are frequently generated on the RNC, leading to flow control on paging messages and decrease in paging success rates. Analysis process After performing action 1, it is confirmed that paging success rates on the CN side decrease but those on the RAN side do not change. In this case, the problem can be roughly identified as discarded paging messages over the Iu interface. After checking the SPU loads, it is found that loads in some subsystems are higher than those in other subsystems, as shown in the following figure:
Perform action 2 to check whether parameter configurations are proper. The results are shown in the following figure;
Issue 01 (2012-12-05)
30
Closed actions and result evaluation Enable the SPU load sharing function by running the following command. After the function is enabled, flow control on the paging messages is eliminated and paging success rates on the CN side are normal again. SET URRCTRLSWITCH: PROCESSSWITCH=RNC_SHARE_SWITCH-1;
5.2 Cases of Paging Success Rates Caused by UE Exceptions

Problem In office T of country T, paging success rates on both the CN and RNC side decrease dramatically from 94% to 30% after the MOCN feature is enabled. Analysis process According to action 1, the problem is identified to be a paging problem of type 2. No exceptions are found by performing required actions in sequence. Collect tracing logs over the Iu interface and IMISIs in paging failures by performing action 11. By checking the CN, it is found that 90% UEs are of the same type. Frontline personnel confirm that this type of UEs is customized to support the MOCN feature. Customers require the terminal company to identify the root causes. By collecting UE logs, the terminal company finds that UEs receive paging messages but do not respond to these messages. This problem can be solved by upgrading UE software. However, it is difficult to upgrade all UEs because many UEs have been delivered to the market. This solution is not feasible. Check the network to find preventive measures. By analyzing the network, the root causes are as follows: Only one carrier is configured on the RNC. Frontline personnel reconstruct the cell under the RNC to an MOCN cell, and configure the ID of the primary operator to 99 and that of the secondary operator to 00. In areas covered by UMTS networks, operator 99's GSM networks also cover the areas. In these areas, operator 00 has no GSM networks but their GSM customers are allowed to roam to operator 99's GMS networks. In the preceding scenario, when the SIM card of a UE supporting MOCN is provided by operator 00, the UE does not respond to received paging messages. Closed actions and result evaluation This problem is solved by using the following solution: Add a carrier, F2, on the RNC. F2 is exclusively used by operator 00. After the solution is performed, paging success rates are normal again.
5.3 Cases of Paging Success Rate Deterioration Caused by Incorrect Configurations on the CN Side
Problem
In country I, paging success rates on one of the two RNCs are low, only 50%. By checking paging counters on the CN side, it is found that this paging success rate in LAC areas is about 90%. Analysis process According to action 1, the problem is identified to be a paging problem of type 3. This problem may be caused by incorrect configurations on the CN side. Check the paging-related configurations on the MSC. The result shows that the number of configured LACs for the RNC is greater than the number of actual LACs. Therefore, paging objects are not within the LACs under this RNC and local paging messages fail to be received, which results in low paging success rates. Closed actions and result evaluation Solve this problem by using the following solution: Delete LACs not used by the RNC on the MSC. After the invalid LACs are deleted, paging success rates on the RNC side are gradually normal and basically equal to those on the CN side.
Issue 01 (2012-12-05)
32

Guide To Troubleshooting Paging Failures-20130110-V1.0

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Guide To Troubleshooting Paging Failures-20130110-V1.0

Uploaded by

Copyright:

Available Formats

Guide to Troubleshooting UPSS V100R003C00 Paging Failures

HUAWEI TECHNOLOGIES CO., LTD.

Copyright Huawei Technologies Co., Ltd. 2012. All rights reserved.

Trademarks and Permissions

Huawei Technologies Co., Ltd.

Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd.

Guide to Troubleshooting UPSS V100R003C00 Paging Failures

About This Document

About This Document

Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd.

Guide to Troubleshooting UPSS V100R003C00 Paging Failures

About This Document

Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd.

Guide to Troubleshooting UPSS V100R003C00 Paging Failures

Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd.

Guide to Troubleshooting UPSS V100R003C00 Paging Failures

4 Description of Troubleshooting Procedures for Paging Failures

Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd.

Guide to Troubleshooting UPSS V100R003C00 Paging Failures

4 Description of Troubleshooting Procedures for Paging Failures

Clarification of Paging Problems

2.1 Definitions of Paging KPIs in Traffic Statistics

2.1.2 KPIs of Paging Success Rates on the RNC Side

2.2 Sources and Clarification of Problems

Guide to Troubleshooting UPSS V100R003C00 Paging Failures

4 Description of Troubleshooting Procedures for Paging Failures

Deterioration of paging success rates of inventory networks

Original traffic statistics KPIs before and after deterioration

VIP users complain about calling difficulties or long delay.

Drive test (DT) data and traced signaling

Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd.

Guide to Troubleshooting UPSS V100R003C00 Paging Failures

4 Description of Troubleshooting Procedures for Paging Failures

Troubleshooting Process for Paging Failures

3.1 Common Scenarios of Paging Failures and Possible Causes

Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd.

Guide to Troubleshooting UPSS V100R003C00 Paging Failures

4 Description of Troubleshooting Procedures for Paging Failures

Paging Success Rates of New Networks Do Not Meet Related Requirements.

Guide to Troubleshooting UPSS V100R003C00 Paging Failures

4 Description of Troubleshooting Procedures for Paging Failures

Paging Success Rates of Inventory Networks Deteriorate

Guide to Troubleshooting UPSS V100R003C00 Paging Failures

4 Description of Troubleshooting Procedures for Paging Failures

VIP users Complain about Calling Difficulties or Long Delay.

3.1.3 Classifying Paging Failures based on Paging Procedures

Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd.

Guide to Troubleshooting UPSS V100R003C00 Paging Failures

4 Description of Troubleshooting Procedures for Paging Failures

3.2 Analysis Report for Paging Problems

Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd.

Guide to Troubleshooting UPSS V100R003C00 Paging Failures

4 Description of Troubleshooting Procedures for Paging Failures

Anal ysi s Report on Pagi ng Probl em s of XX Proj ect . doc

Huawei Proprietary and Confidential Copyright Huawei Technologies Co., Ltd.

Guide to Troubleshooting UPSS V100R003C00 Paging Failures

4 Description of Troubleshooting Procedures for Paging Failures

Description of Troubleshooting Procedures for Paging Failures

4.1 Action 1: Confirming the Type of a Paging Failure Procedure

4.1.2 Data Sources and Processing Tools

4.2 Action 2: Checking Parameters

Guide to Troubleshooting UPSS V100R003C00 Paging Failures

4 Description of Troubleshooting Procedures for Paging Failures