Professional Documents
Culture Documents
Contents
Introduction.............................................................................................3 Why Testing Mobile Apps Is Different Than Any Other Platform............ 4
Rule #1 - Make sure your mobile app does what the user is expecting it to do. .............................................................................................. 5 Rule #2 - Concentrate on the devices where the majority of your target market will be and camp out there................................................... 5
by SMARTBEAR
TestComplete
Introduction
SoLoMo. You know the mantra. Social, local, mobile. Its the way the Internet is heading. Its the direction that the world is heading. We have been hearing it for years. The difference with that clarion cry is that we are way past the hype phase. No longer do we wonder if this is the Year of Mobile. Mobile has arrived and it is past the point of novelty. Whether it is business or personal use, mobile is getting more and more ingrained in all aspects of our lives each and every day. As a result, users are more savvy than ever before and their patience with poorly designed and delivered mobile apps is very limited. This shift in how the world accesses information and uses data has forced the mobile world to grow up faster than it may have been ready for. No longer can an app be almost there and be released into the market. Today, if your mobile app is even the slightest bit buggy or crashes from time to time or doesnt work on as many platforms as possible, you stand to lose big. Its for this reason alone that the practice of mobile app testing is a critical step in the life cycle of an app. Very few companies are allowed any leeway in how their apps perform. A perfect example of this is Facebook. Its hard to get more ubiquitous than Facebook, yet when the company went public the doubts about their mobile capabilities hampered the stocks performance. Now, if Facebooks mobile presence doesnt meet the high expectations of the marketplace the entire brand suffers. Granted, not all businesses face being put under this kind of harsh interrogation, but that day is not far off if we continue to progress at the current pace. A mobile app that has even the smallest imperfection will be quickly pushed aside in search for another to serve an end users needs.
by SMARTBEAR
TestComplete
That said, it is critical to address the testing aspect of your app development with the utmost concern. In this eBook, we will take a look at the different ways mobile app testers need to test.. Scripted testing alone will not be enough. Exploratory testing alone will not be enough. Automated testing alone will not be enough. Instead, todays testers of mobile apps will be required to find the blend of all three types of testing in order to feel confident that their apps are ready for a rigorous marketplace. We will explore the unique elements of the mobile world and how software testers will need to find the right mix of testing so that, when their app gets to the end user, they wont immediately be cast aside.
by SMARTBEAR
TestComplete
Rule #1 - Make sure your mobile app does what the user is expecting it to do.
Before even getting near the intricacies and complexities of the mobile environment, it has to be determined whether the app itself simply accomplishes what it was originally intended to do. Why is this more difficult in the mobile world? Its quite simple as to why, but its a whole other issue as to what to do about it. The why its hard is due to how end users behave in the mobile space. It is a very new area of exploration for everyone and almost everything that is seen is some variation of first use case, which means that there is little or no historical data or experience to fall back on for validation. With this in mind, before you go through the hard work of putting an app through its paces with a variety of tests, you better make sure it is accomplishing the desired business task. Before you go through the hard work of putting an app through its paces with a variety of tests, you better make sure it is accomplishing the desired business task.
Rule #2 - Concentrate on the devices where the majority of your target market will be and camp out there.
There may not be a more difficult hurdle to overcome in the mobile app world than the issue of device acceptance. iOS, Android, Windows, and BlackBerry are difficult enough to test for on their own. Then, get into the various versions of each mobile OS and you can quickly feel like the better alternative is to simply scream and hope it goes away. But add on to this the many and various operating systems of Android, and the mobile landscape begins to seem infinite so it is rarely enough to only develop for iOS anymore. Certainly the demographics of the iOS mobile device user are more attractive than any other, but even that metric is becoming less dependable with the
by SMARTBEAR
TestComplete
advances that Samsung and HTC have made with their Android models. Many kids these days are walking around with a Galaxy S4 and not an iPhone 5S. Since its difficult to be all things to all people, if you think that most of your users are going to be a combination of iOS and the most popular flavor of Android / device, you may need to be developing and testing just for those targets.
GPS How important are geolocation elements to your app? The mere
fact that you are working in the mobile environment implies that you know the location of the user will be a critical piece of your mobile app puzzle.
Does your app play with the major map platforms? Will your app integrate properly with Google Maps? Apple
Maps?
Who do you use? Whom do you partner with? Who gives the best data and user experience? How are the various APIs? What are the limitations and costs? Data How much data? What data? What are your privacy policies? What are you accessing on their device? What is your end users tolerance for privacy encroachment?
by SMARTBEAR
TestComplete
Network Availability How does your app perform on various carrier networks as
opposed to Wifi?
Will your app be a data hog that causes users to watch their
bandwidth usage as a result?
Gestures Does your app give the user gesture options that are confusing
or that create conflict with other apps?
GUI Lets get down to brass tacks here. Is your app simply butt-ugly? Does your app have a command line feel to it or is it appealing
to even the most non-technical of users (who will be a vast majority of most app audiences anyway)?
Portrait and Landscape Views What happens to your app when your end user turns his or her
device on its side or on its head?
Does it respond quickly? Is it responsive at all? Are there any fields that suddenly become impossible to
navigate because of the limited space due to the keyboard expanding for use?
by SMARTBEAR
TestComplete
Processing Power The last thing you want your app to be associated with is
making your users device so slow because it is a tremendous resource hog. If within those first few hours of downloading your app your customer notices that nothing is working as well on their device as it did before they downloaded your app guess what happens.
Memory Lets be honest. Most mobile device users are pushing their
memory limits to the nth degree on any given day.
How will your app impact device memory? Battery What impact will your app have on the battery life of a mobile
device? If an end user downloads your app then suddenly sees their battery life fall through the floor you may be viewed as a troublemaker. Not good.
Fragmentation of devices We alluded to this earlier, but its the simple truth about the
mobile space. There are an ever growing list of devices, screen sizes, form factors, etc. that make up the mobile marketplace.
Its not cheap to make design alterations for every make and
model. Also, once you make accommodations for one you may end up breaking something with another.
Will you need to pick and choose which devices are going to
represent the largest portion of your target market then go from there? This is a critical decision that must be taken seriously.
by SMARTBEAR
TestComplete
by SMARTBEAR
TestComplete
What are those expectations? Well, with a fair amount of planning, a concentrated effort and a little luck, one should expect that they could cover most of the bases when testing mobile apps. What is probably unrealistic is the expectation of testing for every situation, under every potential scenario and uncovering every little ailment of an app before it gets to market. Especially since the fluid nature of the mobile space changes the rules so often, you can never take care of every angle. That doesnt mean that it cant get close though. The best way to get result that are going to set your efforts apart from your competitions is by utilizing various testing methods in the right mix. As with any software or app testing, the process is as much art as it is science. Flexibility will win the day, so loosen your collar as we delve into the testing options that are available and help you decide what is the right formula for your needs.
Automated Testing
We are going to start off this conversation about automated testing with this caveat. Automated testing in the mobile environment should not be an end unto itself. In other words, the majority of mobile app testing should occur in the manual (scripted) and exploratory space, while automated testing serves as a support system for those two approaches. Now that we have established that, we can look at automated testing of mobile apps in the proper light. Just the mere mention of automation in the mobile space can make many queasy because automation implies process. Process implies repetition. Repetition implies that the mobile experience is something that is rote and repeatable and doesnt necessarily require human intelligence or creativity. Well, its not really true, especially given the strategy required in understanding how and when to use automated testing.
by SMARTBEAR
TestComplete
10
So, what can automated tests actually cover? The best-case scenario is to test aspects that require no user interaction. That might include:
Did this API function return the correct value? Did this function calculate the correct result? Did we successfully send a file over the network with the correct
checksum?
by SMARTBEAR
TestComplete
11
Scripted Testing
In manual testing, scripted tests are the next step up from automation. There is much more flexibility in the testing, but the fact that it is scripted implies limitations. Thats fine. Developers, business stakeholders, and other critical input sources have areas of concern that need to be vettedand scripted tests allow for those areas of concern to be addressed. Using Personas Developing user personas (different profiles of those who the app is really being built for) is a key component in increasing the effectiveness of a mobile app. Hopefully, the various personas that make up the target market of any mobile app were considered in the design and development of the app from the onset. There can be scripted testing that is built around personas. This will help to uncover potential issues that a specific subset of customers might encounter based on:
Who they are How they function What they like Why they would use the app, etc.
The number of personas tested will depend on how many there are for a particular app and what kind of resources are available (read: time and money) to test as many variations of a theme as possible. The use of scripted tests from that personas point of view can do well to uncover potential issues.
by SMARTBEAR
TestComplete
12
Exploratory Testing
Its at this point, however, that we quickly reach the limitations of scripted testing. No matter how much an end user fits the persona, a scripted test will never allow for the inevitable variations on a theme. In other words, human beings rarely, if ever, truly follow a script. With that said, it becomes imperative for those conducting the tests of an app to start to go off script and into an area that might more accurately represent reality. The means to get there is through exploratory testing. Exploratory testing is simply a more free flowing methodology of testing that, through experimentation, builds on itself and allows testers to get into real world situations that may never be accounted for in automated or scripted tests. This mindset becomes even more valuable in the testing of mobile apps because the situations an end user finds him or herself in change constantly due to the fact that they are mobile. They are on the move. Performing a task on a mobile device while sitting at a desk may look very different than performing the same task while walking down a crowded sidewalk. Or at a traffic light. Exploratory testing fits perfectly with mobile app development because the very nature of many mobile apps is exploratory in nature. People are doing things on the move and are often in discovery mode themselves. Each time the app is openeven though the function that is being called upon in the app is the samethe reason for its use and its ultimate utility could be different almost every time. In other words, in the mobile world things that might look like they are repeated events are only repeated at a surface level. The intent and
by SMARTBEAR
Exploratory testing fits perfectly with mobile app development because the very nature of many mobile apps is exploratory in nature. People are doing things on the move and are often in discovery mode themselves.
TestComplete
13
ultimate goal of the user might be completely different each time that particular function is called upon, which may create a different interaction with the same function based on the external variables the end user is experiencing. To test how an app responds to such a dynamic environment requires freedom. Exploratory testing that builds on previous experience or on a new experience can allow the test to flex and move in directions that best mimic that of a real life end user. As a result, the real discovery regarding the strength and weakness of an app can occur. It is critical to make sure that in this exploratory process there is accurate data collection. This is a difficulty of the exploratory model that is compounded when used in the mobile space. How you require testers to report their findings could mean the difference between a jumble of interesting tidbits and a result-set that impacts the performance of an app.
by SMARTBEAR
TestComplete
14
Obviously, this kind of balancing act is best performed by those with experience in these matters. If they are not available, or affordable, this might be the time to consider outside help in the form of products and services designed to get you most of the way there.
Conclusion
In the end, testing mobile apps is something that needs special attention and a renewed approach from traditional software testing. Over time, it is likely that new methodologies will be developed that are more appropriate for the mobile space. Right now though, at this early stage in the game, we need to work with the existing testing types like automated and scripted testing but allow them to be flavored by a sense of adventure, so to speak. The mobile space is new and exciting. It is also relatively unmapped as compared to other places in the digital geography. With that in mind, effective mobile app testing will require a spirit of true exploration. Testing mobile applications so that you may be able to create the best possible experience for your end users is a true work in progress. As you move through it, mixing and matching testing procedures and techniques will be the key to giving your efforts the best chance to truly make an impact in the mobile world. Are you ready? Are you flexible? Are you adventurous?
by SMARTBEAR
TestComplete
15
SmartBear Software, Inc. 100 Cummings Center, Suite 234N Beverly, MA 01915 +1 978.236.7900 www.smartbear.com 2013 by SmartBear Software, Inc. Specifications subject to change. EBOK_TC_20130913_V1_Exploritory-Automated-Testing