Professional Documents
Culture Documents
The answer is that comprehensive testing of mobile apps has proven to be difficult and expensive using conventional
methods. While the testing matrix for mobile apps has grown exponentially (handset maker & model, wireless carrier, OS,
language, location, etc.), the methods for testing mobile apps have not kept up. They are outdated, inefficient, and above all,
impractical for the world of mobile. Thus, despite the obvious risks associated with releasing a buggy mobile app, the less
complicated, less costly alternative has been to test a little bit and hope for the best. Though this scenario paints a dark
picture for mobile app companies, there are other options.
This eBook will identify four of the most popular approaches for testing software (listed below) and explain why they are
not enough for the world of mobile apps. This will be followed up by a quick introduction to crowdsourced mobile testing.
If you are testing a new mobile app (or a new version), these four testing methods are not sufficient on their own:
• In-House: For reasons of cost and coverage, it is inexcusably naïve to expect an in-house team to provide comprehensive
testing coverage in a reasonable amount of time. Learn why the in-house route works in other areas, but not in mobile.
• Outsourcing: The same problems that plague in-house teams affect outsourcing firms ten fold. If quality and coverage are
priorities for your mobile application, then outsourcing will be a major disappointment.
• Emulators/Simulators: There’s literally a world of difference between actual testing, and testing in a simulated lab. While
there are benefits to leveraging emulators and simulators, ignoring real-world usage is a proven recipe for disaster.
• Beta Testers: The era of beta testers a substitute for comprehensive testing is long gone. Users use. Testers test.
The crowdsourced alternative: Learn how community-based testing, aka “crowdsourcing” is perfectly
suited to meet the needs of today’s mobile application developers.
Why the drop-off? For starters: bugs - especially ones that appear within minutes of downloading - are likely to cause users to
abandon the application after a short period of use. With the variety of apps now available, users have an extremely low
tolerance for buggy applications. In a never-ending attempt to appease their user base, developers must therefore test their
products across:
Handset Maker/Model
Did You Know?
Operating System
“According to mobile advertising startup AdMob, there
Browser are some $200 million worth of applications sold in
Apple’s iPhone store every month, or about $2.4
Wireless Carrier billion a year.”
No matter what type of mobile applications you develop – whether they’re chat tools, social networking, games, business apps
or others – you face the same fundamental problems encountered by all mobile app developers.
Let’s now take a quick look at how mobile app developers have attempted to deal with these problems…
To be fair, most companies never intend to create an in-house team like the one
described above. Worse, they spend their money trying to get “what they can”
from the method. That is, they have 3-5 in-house testers, with a few dozen
devices (at most). But under this plan, their testing coverage is confined to their
own office, and the whole point of comprehensive QA is lost.
And as it turns out, even the largest of traditional outsourcing firms don’t have
the resources needed to provide adequate testing coverage required by today’s
mobile environment. Just like their clients, they cannot easily (or affordably)
test an app across dozens of handset makers, models, carriers, languages and
locations. Plus, the testers employed by these offshore firms are unlikely to be
among the target audience for a company’s mobile apps.
But even if you are able to assemble a large beta group, the method still falls
short of providing adequate testing coverage. And the reason for this is simple:
beta testers are more often like users, in that they will try to get your app to
function properly, whereas a real tester would attempt to “break” your app.
While the thought of having a virtual QA team from around the globe might
seem like a complicated free-for-all, the reality is just the opposite. With the
right online platform, the process fits neatly into your existing teams and
processes. Here’s a quick look at how it works:
2. The TM then uploads his test plans, use cases, known bugs and other
documentation
4. Bugs are reported and classified by the testing community in real-time – On Crowdsourcing
along with documentation like screenshots, video capture – and monitored by
the TM (who can approve, reject or request more information from testers) “Peer production is more than just
sitting down and having a nice
5. The company pays only for the test cycles it needs (no downtime or long- conversation. It’s about harnessing
term commitments). Testers are paid for approved bugs, completed test scripts a new model of production to take
or surveys.
innovation and wealth creation to
The net effect is that mobile app companies get instant access to a global new levels.”
community of testers via a platform that enables detailed management of the - Eric Schmidt
entire test cycle (or even connects to their in-house bug tracking system). CEO, Google
This community can serve as a virtual testing team (for small companies and
start-ups), or as an on-demand extension for larger, more established in-house
QA departments.
Faster Time-to-Market: Crowdsourcing enables you to ramp up or down to meet -Jeff Howe
your needs at any given time - eliminating the delays and staffing constraints that
often stifle mobile app companies around peak release times.
Journalist, Wired.com
Higher Quality: Customers expect their mobile apps to be fully functional and
bug-free from day one. Companies that believe “good enough” is good enough
when it comes to quality and usability will learn a valuable lesson (the hard way).
Retaining Users: If your product has problems, or doesn’t meet expectations, it’s
easier than ever for your customers (and prospective customers) to find your
competition.
Either way, the future of mobile applications remains bright. Once reserved for
a tech-savvy niche in the business world, the use of mobile applications is now
firmly entrenched in the mainstream. There’s no turning back when it comes to
mobile applications, and that includes methods for testing them.
For more on how crowdsourcing can complete your mobile testing efforts,
chat with one of our QA gurus by clicking here or by calling 800.445.3914.