Mobile Application Testing

by OptimusQA
Guide to managing quality of a mobile application.

OptimusQA is a software testing firm located in Vancouver, Canada. Visit our website at www.OptimusQA.com to get help with mobile testing today!

TABLE OF O!TE!T"
INTRODUCTION.................................................................................................................................................. 3 KEY CHALLENGES............................................................................................................................................... 4 PLATFORMS - VARIETY, DIVERSITY, LIMITATIONS................................................................................................ 4 USER EXPECTATIONS............................................................................................................................................... 4 NATIVE APPS VS. MOBILE WEB APPS........................................................................................................... 5 TEST DEVICES..................................................................................................................................................... 6 SIMULATORS EMULATORS..................................................................................................................................... ! REAL DEVICES......................................................................................................................................................... " REMOTE DEVICE SERVICES................................................................................................................................... #$ CROWDSOURCING.................................................................................................................................................. ## TEST AREAS....................................................................................................................................................... #% FUNCTIONAL TESTING.......................................................................................................................................... #% USER EXPERIENCE................................................................................................................................................. #3 PERFORMANCE AND LOAD TESTING..................................................................................................................... #4 SECURITY TESTING............................................................................................................................................... #5 COMPATIBILITY TESTING..................................................................................................................................... #6 INTERRUPT TESTING............................................................................................................................................ #& LOCALI'ATION TESTING....................................................................................................................................... #! TEST AUTOMATION........................................................................................................................................ #" CONSIDERATIONS FOR SELECTING AN AUTOMATION TOOL................................................................................. #" TESTABILITY......................................................................................................................................................... %$ AFTER RELEASE............................................................................................................................................... %# CHEAT SHEET................................................................................................................................................... %% CONCLUSION..................................................................................................................................................... %3 REFERENCES..................................................................................................................................................... %4

http://www.OptimusQA.com

#!T$O%& T#O!
This guide will provide you with suggestions and strategies for testing a mobile application. It will help you plan your QA resources and e uip your QA team with the !nowledge necessary for testing mobile applications. "ho should read this guide# This guide is primarily for QA managers at independent software vendors tas!ed with creating large or comple$ mobile apps. It should also provide value to QA teams and anyone testing a large or comple$ mobile app.

http://www.OptimusQA.com

'E( )ALLE!GE"
PLATFORMS - VARIETY, DIVERSITY, LIMITATIONS
&obile devices differ in terms of operating system' hardware capability and form factor. This variety' which is great for consumers' leads to comple$ity for testers. &any times' mobile application testers see the same application behave differently on different devices. "ith thousands of different hardware/O( combinations' testing all the functionality of an application on every device is impossible. The best approach is to prioriti)e platform coverage' either based on speci*ic customer needs or based on mar!et share.

USER EXPECTATIONS
+$pectations about mobile application performance and reliability are different than for other types of software. ,sers have access to thousands of applications that are either cheap or free. If they install a new application and e$perience what they perceive to be a serious problem within the *irst few minutes' they will not hesitate to uninstall the app -and potentially leave a negative review in the app mar!etplace.. &obile apps are also used in short bursts' compared to traditional or des!top apps. /ere are some other e$pectations:

0ast download time 0ast start1up time 2ood response times for all common user interactions 3isual appeal (mall application footprint -because of limited storage capacity on mobile devices.

http://www.OptimusQA.com

!AT#*E A++" *". MOB#LE ,EB A++"

The mobile web is growing and mobile web applications may be more suitable for some enterprise uses than native apps. It is worth learning the similarities and differences in testing native apps and testing the mobile web.

(imilarities
Testing different screen si)es and form factors is necessary Testing on real devices is a must. Test automation tools are available -but most test automation tools for automating native apps testing do not support mobile web app test automation and vice versa..

5ifferences
N()*+, A--. Testing app installation' uninstallation and update.is re uired. Testing of*line functionality.is re uired. M/0*1, W,0 A--. 6o app installation' uninstallation and update testing. Testing of*line functionality of apps using /T&78 local storage. +$tensive browser testing for supported browser and O( combinations. 7ess gesture testing as fewer gestures are available to web apps.

&inimal or no browser testing needed.

2esture functionality such as swipe and tilt' needs to be tested. Apps need to be tested against the design guidelines and re uirements set by different mobile platforms.

http://www.OptimusQA.com

TE"T %E*# E"

"ith the incredible diversity of devices available' it is impossible to test an application against all of the devices it will be used on. Therefore' we have to answer two uestions.

9. /ow should we select our test devices so that we are con*ident of the application uality in different devices# 2. Are there any alternatives to real devices#

To answer the *irst uestion' we should consider the following:

9. "hat target platforms and devices does this application target for# Is it iO( only# Is it tablets only# 2. "hat are the popular devices in the mar!et by O( platforms# :y models# :y form factors# %. (ome people recommend bounds testing. 5e*ine an upper and lower limit for hardware constraints' and test
each of those e$tremes. If possible' test a couple popular devices that fall within these constraints. :ounds testing is best for tighter budgets; it assures that there are no outliers in your matri$' but it isn<t as comprehensive as it could be. =9>

To get the latest mar!et share' you can refer to these sources: &obile :rowser ,sage

(tat?ounter: http://gs.statcounter.com/ 6et &ar!et (hare: http://mar!etshare.hitslin!.com/browser1mar!et1share.asp$# prid@AB pcustomd@9 "i!ipedia: http://en.wi!ipedia.org/wi!i/,sageCshareCofCwebCbrowsers

Dhone and Dlatform Dopularity

Android 5evelopers 5ashboards -http://developer.android.com/about/dashboards/inde$.html. collects the data of the active devices that have accessed 2oogle Dlay in a recent 941day period or E1day period' and displays the distributions of (5F versions installed' and the distributions of the screen si)es and densities. App:rain -http://www.appbrain.com/stats/top1android1phones. shows the statistics of Android phones 1 most used phones and most common (5F installed by the App:rain users (tat?ounter -http://gs.statcounter.com/. offers statistics on the popularity of different operating systems.

http://www.OptimusQA.com

The cost of ac uiring all of the test devices is e$pensive' but there are alternatives out there. 7et<s e$plore some of the options.

http://www.OptimusQA.com

SIMULATORS EMULATORS
(imulators and emulators are used generally in the early stage of development. P2/. They are free. +very vendor offers them. C/3. Testing with emulators is far different from using the application in the real world' so it does not guarantee the application will wor! on real devices. ,nable to test the application with hardware capabilities. ,nable to test networ! interoperability 1 networ!1 related events -e.g. calls and te$t messages' etc.. and different networ! technologies -e.g. %2' 7T+' etc... Application performance on the emulators does not re*lect the actual performance.

,sers can choose the si)es and O( versions' or even models to test with. Allows developers and testers to verify certain functionality that is not speci*ic to any device' carrier or operating system. ?onvenient to test with simulators or emulators on laptops.

http://www.OptimusQA.com

REAL DEVICES
Testing on real devices is a must because the feel of the application are hard to be mimic!ed on emulators or remote devices. In addition' you are not able to test the hardware capabilities and networ! without actual devices.

(ome facilities have a device library' for e$ample "ave0ront in 3ancouver. (ome vendors offer loan or rental programs to development partners. Iou can chec! out some of the vendor programs:

(amsung: http://www.samsungmobileb2b.com/partner/programCoverview.do#boardCse @%J 6o!ia: http://www.developer.no!ia.com/5eveloperCDrograms/ :lac!berry: https://partners.blac!berry.com/web/guest/new1program1overview

P2/. Testing on real handsets gives reliable and accurate results

C/3. It is costly to ac uire real devices -continuously because new devices come out fre uently. and maintain them Keal devices may be lost or stolen

Interoperability testing is possible because testing is performed on a live networ! Drovides true user e$perience

Kesetting devices between tests can be time1 consuming or re uires e$pensive hardware.

+asier to e$pose performance defects with real handsets in a realistic environment.

http://www.OptimusQA.com

REMOTE DEVICE SERVICES

(ome third1party companies provide services which allow developers and testers to test the application on various devices and networ!s remotely over the Internet. The most popular service providers:

Feynote 5eviceAnywhere -http://www.!eynotedeviceanywhere.com/. Derfecto &obile -http://www.perfectomobile.com/. (OA(TA TouchTest &obile 7ab -http://www.soasta.com/products/touchtest1mobile1labs/.

(ome vendors provide free access to their remote devices labs' for e$ample:

(amsung -http://developer.samsung.com/remotetestlab/rtl5evice7ist.action. 6o!ia -http://www.developer.no!ia.com/5evices/KemoteCdeviceCaccess/.

P2/. Drovides a huge selection of devices -different models. on various networ! carriers Allows testers to record video and capture images while testing Offers an automation module for writing test scripts &ay support test cases management (ome provide integration with testing systems li!e /D Offers different pac!ages so that users can choose according to their needs

C/3. 7atency occurs -depending on how far the client location is from the device location. ,nable to test applications that re uire peripheral devices The automation module is e$pensive Touch and gestures are not easy to accomplish

9A

http://www.OptimusQA.com

CROWDSOURCING
?rowdsourcing is to outsource testing the application to an unde*ined public group. This approach is generally used as beta1testing. ?rowdsourcing services communities: P2/. 2et access to a broader range of devices on various networ!s Obtain feedbac! and reports from thousands of people -instead of Lust ten people on the QA team. 7ower price than hiring a professional &ob4/ire -http://www.mob4hire.com/. JJTests -http://JJtests.com/. C/3. The QA reports may not be credible and reliable

Ta!es time and efforts to manage a large group of people ?ollecting and analy)ing the results may consume lots of resources

2ood for testing locali)ation

99

http://www.OptimusQA.com

TE"T A$EA"
FUNCTIONAL TESTING
0unctional testing ensures the application is performing as speci*ied in the re uirements. The idea is the same as the functional testing for des!top applications. /ere are some areas that testers should speci*ically pay attention to when testing mobile apps:

2estures functionality: Is the app responding as e$pected upon performing a gesture# 0or e$ample' swipe' tilt' and highlight te$t. Dhone orientation: 5oes your app crash when you rotate the phone# (ocial media integration: Are there problems with integrating with social media apps' such as 0aceboo!' Twitter# Dush noti*ications. Is installation re uesting too many or too few permissions# 6etwor! connectivity: /ow does the app respond when there is no internet# Third1party services: /ow does your app handle non1responsive third1party services or ADIs# 0ailure handling: 5oes your app handle failures gracefully#

92

http://www.OptimusQA.com

USER EXPERIENCE
The 6o.9 rule is to follow the design guidelines speci*ied for the particular O( platform. +ach O( platform is designed to have its uni ue loo! and feel' so the application should match the O( design.

OS P1()4/25 D,.*63 G7*8,1*3,. iO(: http://developer.apple.com/library/ios/Mdocumentation/,ser+$perience/?onceptual/&obile/I2/Introduct ion/Introduction.html Android: http://developer.android.com/design/inde$.html "indows Dhone: http://dev.windowsphone.com/en1us/design

Other than the platform design guidelines' testing user e$perience on mobile applications is not very different from testing user e$perience on des!top applications. (ome user e$perience guidelines that are especially important to mobile applications include

(implicity ?onte$t 2ood navigation Appropriate font si)es and button si)es Dortrait and landscape orientation considerations

These articles provide good insights of improving mobile user e$perience:

http://mobile.smashingmaga)ine.com/2A92/AE/92/elements1mobile1user1e$perience/ http://mobilewebbestpractices.com/user1e$perience/ http://www.ionagroup.com/2A99/A8/2%/understanding1the1differences1between1designing1for1mobile1 and1des!top1media1while1learning1the1best1practices1for1designing1iphone1applications/ http://developer.apple.com/library/ios/Mdocumentation/,ser+$perience/?onceptual/&obile/I2/,+:estDr actices/,+:estDractices.htmlM//appleCref/doc/uid/TD4AAAG88G1?/2A1("9 -A lot of designers recommended these guidelines for iO(.

9%

http://www.OptimusQA.com

PERFORMANCE AND LOAD TESTING

Derformance is another critical factor that contributes to the success of a mobile app. The following areas should be considered: (tart1up time: /ow long does it ta!e your app to launch and load the *irst screen# Kesponse time: 5oes your app respond to user input instantly# 6etwor!: /ow does the app perform on various networ! types' li!e "i10i and 7T+# /ow about in different networ! strengths# &emory consumption: &emory is limited on mobile devices. /ow much memory does your app consume at pea!# /ow does your app perform when there are lots of apps running in bac!ground# :attery consumption: /ow much battery does your app consume# Is this level of consumption acceptable in a conte$t where a number of apps are running in the bac!ground as well#

If your app is built with a server/client model' you may want to consider load testing. Testing the server performance is not very different from how we usually do it' but for mobile apps' you have to simulate different networ! types and strengths during your load testing.

94

http://www.OptimusQA.com

SECURITY TESTING
"ith the number of mobile device users soaring' mobile malware is on the rise too. Therefore' you may be interested in identifying the security vulnerabilities and addressing the ris!s' especially if your app has to process *inancial payments or con*idential information. :eware of the following areas: "hat are the changes in the device *ile system after installation and uninstallation# Is sensitive information encrypted when it is sent over the networ!# If sensitive information is stored on the device' where is it stored# Is it stored in a common storage shared among other apps# Is the stored sensitive data encrypted# If your app provides social networ! integration' what information is shared in the social networ!# If your app uses third1party services or ADIs' are there any security ris!s involved# Is the app e$posing con*idential information in log *iles#

98

http://www.OptimusQA.com

COMPATIBILITY TESTING
?onsider the following e$ample =2>: our client would li!e to test the application on the top "# Android devices. $ut with limited time and budget, you are not able to run all types of tests on all platforms. %herefore, you could suggest a complete test on the top & Android devices followed by a compatibility test on the remaining ' devices.

?ompatibility testing approaches:

Test the critical areas or features of the application. The acceptance test suite is a good start. Issues found in the complete tests can be chec!ed to see if they e$ist on the devices used in the compatibility test. Automated test scripts can ease the wor!load of compatibility testing. /owever' automated testing should not replace manual testing because ,I or usability issues may not be spotted by the scripts.

9G

http://www.OptimusQA.com

INTERRUPT TESTING
Interrupt testing is to test how the application reacts or functions when it is interrupted in different scenarios. The following are some e$amples of interruptions =%>:

Incoming and outgoing (&( and &&( Incoming and outgoing calls Incoming noti*ications (udden or forced shutdown ?able insertion and removal for data transfer 6etwor! outage (witching between networ!s &edia player on/off 5evice power cycle

9E

http://www.OptimusQA.com

LOCALI'ATION TESTING
If your app is going to reach the global mar!et' locali)ing your app is important so that it can adapt to speci*ic mar!ets or countries. /owever' locali)ation testing is a challenge for the QA team since it re uires QA to have !nowledge about speci*ic regions. (upporting multiple languages is a basic re uirement; QA should also pay attention to the following when testing locali)ation: Kegion1speci*ic functionality: Are there any features that are uni ue to a particular region# Input *ields: Are some input *ields active in certain regions and inactive in other regions# 0or e$ample' some regions do not have NID codes' some countries use ODrovinceP and some use O?ityP instead. 0ormats: Are the formats' such as date formats and phone number formats' adapted to the region# 7egal re uirements: Are there any legal re uirements speci*ic to a region# 6etwor!: To test the networ!s in a certain region' do you have a test team there# Or would you li!e to ac uire remote device services' li!e Derfecto and 5eviceAnywhere# ?ultural !nowledge: 5o you have employees who are familiar with the target region and culture -not necessarily have to be a QA.# +ven if the translation is perfect' the app may not be able to fully adapt to the region. 0or e$ample' +nglish spea!ers in 6orth America are used to O(hopping ?artP whereas +nglish spea!ers in :ritain are used to O(hopping :as!etP. Automation tools: /ow does your automation tool identify obLects# 5oes it depend on the language used#

9H

http://www.OptimusQA.com

TE"T A&TOMAT#O!
CONSIDERATIONS FOR SELECTING AN AUTOMATION TOOL
/ere are some aspects we can consider for selecting a test automation tool: (upported platforms and devices

"hich O( platforms and models does the tool support# 5oes it support testing with emulators and/or physical devices# ?an the same test script run on different devices -i.e. resolutions' si)es' phone/ tablet. of the same O(# /ow about devices of different O( versions' e.g. iO( 8.9 and iO( G.A# /ow about running the script on different devices of different O( platforms' e.g. Android and iDhone# 5oes the tool support on1screen gestures' such as swipe' )oom and scroll# 5oes it support outside application testing' such as phone battery and settings# 5oes it provide call and (&(/&&( simulations# "hat obLect identi*ication methods does the tool offer' by the obLect native I5' image recognition' te$t recognition' or "eb /T&78 -5O&.# /ow easy is it to manipulate or access -e.g. loo! up the obLect properties' modify the obLect attributes. the obLects using the tool# "hat !ind of diagnostics does the tool collect and report' e.g. ?D, usage' memory consumption# 5oes the tool integrate into e$isting test environment' e.g. QTD' &(Test' Test?omplete' etc.# 5oes the tool support script parameteri)ation and data1driven#

(cript reusability

0unctionality

ObLect identi*ication methods

Test diagnostics

Test environment integration

Others

9J

http://www.OptimusQA.com

TESTABILITY
In many cases' much of functionality of a mobile application is actually accomplished on the server and e$posed through an ADI. This provides options for testing and automation that don<t always involve going through the ,I. "hen testing the ,I is a focus' many of the usual testability best practices should be followed' such as =4>:

,se uni ue and clear identi*iers for !ey ,I elements. &a!e the names understandable and !eep them consistent across platforms -iO(' Android' "indows Dhone. and releases. Drovide ways to uery the state of the application. ?reate a client interface for testing the code underneath the 2,I because it may be simpler to automate some tests without 2,I operations. 5evelop tools or scripts that can complete the whole end1to1end testing -if the app under test re uires receiving responses from another source' or interacting with other apps..

2A

http://www.OptimusQA.com

AFTE$ $ELEA"E
After your app is released' that does not mean you have nothing to do. To help you and your team plan for the ne$t release and improve the uality of your app' you can consider the following: ,se crash analytics tools to monitor and diagnose app crashes. Available tools include 0lurry' :ug(ense' 0lightDath -developed by Test0light team.' ?rashlytics and 2oogle Analytics. Kead the comments and user feedbac!s from app stores ?hec! out the apps from the competitors

29

http://www.OptimusQA.com

)EAT ")EET
If you don<t have time to read and digest all the information in the above sections' Lust read the following top guidelines and !eep them in mind:

9. Testing on real devices is re uired because testing with emulators is far from the reality. 2. "hen you select the test devices' consider the mar!et shares by O( platforms' by models' and by form factors. %. Test the application under different scenarios' such as "i10i and 7T+' running low on battery and incoming
call or (&(.

4. The loading time of the application or mobile site is critical. The golden rule is 8 seconds' but research shows
that GAQ of mobile users will abandon the application or site if it does not load within % seconds =8>.

8. 5ifferent O( types should have slightly different interface designs which should match the guidelines
speci*ied by the O(.

iO(: http://developer.apple.com/library/ios/Mdocumentation/,ser+$perience/?onceptual/&obile/I2/Introduct ion/Introduction.html Android: http://developer.android.com/design/inde$.html "indows Dhone: http://dev.windowsphone.com/en1us/design

G. Day attention to the application submission re uirements before submitting the application to the store. In
fact' it is better to start early in the QA process.

E. The mobile landscape changes constantly' so !eep yourself updated with the mar!et and research.

22

http://www.OptimusQA.com

O! L&"#O!
&obile devices are becoming more and more important' and mobile applications continue to grow in uality and uantity. QA is a valuable investment to ensure your company and your app remain competitive in the mar!et.

The information in this guide will help you have a great start off on mobile application testing. If you have any uestions' or you need further help' feel free to contact OptimusQA.

2%

http://www.OptimusQA.com

$EFE$E! E"
9. OApp Testing: "hich Android 5evices (hould Iou ?hoose#P. blog.amadeusconsulting.com. Ketrieved 2A921 9912A 2. O&obile App ?ompatibility TestingP. testing4success.com. Ketrieved 2A9219912A %. O&obile application testing' "i!ipediaP. wi!ipedia.com. Ketrieved 2A92199198. 4. ODage 9GE19GH in &obile 5eveloper<s 2uide to the 2ala$yP. scribd.com. Ketrieved 2A9219919G 8. O(ummary of Fey &obile (urvey 0indings by ?ompuwareP. compuware.com. Ketrieved 2A9%1AG129

24

http://www.OptimusQA.com