FEATURE: MOBILE APPS
What Do
Mobile App
Users Complain
About?
Hammad Khalid, Shopify
Emad Shihab, Concordia University
Meiyappan Nagappan, Rochester Institute of Technology
Ahmed E. Hassan, Queens University
// A study of user reviews from 20 iOS apps
uncovered 12 types of user complaints. The
most frequent complaints were functional
errors, feature requests, and app crashes.
However, complaints about privacy and
ethical issues and hidden app costs
most negatively affected ratings. //
MOBILE APPS POPULARITY con-
tinues to grow rapidly. The Apple iOS
App Store alone contained 1.2 mil-
lion apps as of June 2014 and remains
one of the most competitive app
markets.1 This competition, and the
70 I E E E S O F T WA R E | PUBLI S HED BY THE IEEE COMPUTER SO CIE T Y
growth of apps in critical domains
such as e-commerce, government,
and healthcare, has made app quality
an increasingly important concern.
Most recent research on app
quality has focused on problems
from the developers perspective.2
However, one of the fi rst steps in
understanding the issues affecting
app quality is to determine the chal-
lenges users face.
The App Store lets users review
their downloaded apps. Besides as-
signing star ratings (all of which
are aggregated and displayed on a
version-level and an app-level basis),
users can add comments explaining
their ratings. This data source cap-
tures a unique perspective about us-
ers perception of the apps. Such re-
views, like product reviews in Web
stores, correlate highly with down-
load counts (purchases) and are a
key measure of an apps success. (For
more on previous research on this
topic and other topics related to this
article, see the sidebar.) A good un-
derstanding of these issues will help
developers deal with users concerns
and avoid low ratings. Furthermore,
such an understanding is crucial in
guiding the software engineering
community in tackling high-impact
research problems in software devel-
opments fastest-growing area.
To better understand iOS users
complaints, we examined user re-
views for a selection of iOS apps.
We aimed to determine the types of
complaints and how they affected
the ratings. Our fi ndings can enable
developers to better anticipate pos-
sible complaints and prioritize their
limited quality assurance (QA) re-
sources on the complaints most im-
portant to them.
The Studys Design
Users tend to write reviews when
theyre extremely satisfied or ex-
tremely dissatisfied with a prod-
uct. 3 Poor reviews affect sales more
than good reviews because buyers
are more likely to react to low rat-
ings and complaints. 4 So, to under-
0 74 0 -74 5 9 / 15 / $ 3 1. 0 0 2 0 15 I E E E
 RELATED WORK IN CUSTOMER
PERCEPTION AND MANUAL DATA ANALYSIS
User reviews have become an important source of informa-
tion about customers perceptions. Susan Mudambi and Da-
vid Schuff showed that user ratings and reviews played a key
role in the purchasing decisions of products from Amazon.
com.1 Similarly, Hee-Woong Kim and his colleagues inter-
viewed 30 users who bought apps and found that ratings
were a key determinant in the purchase.2 Mark Harman and
his colleagues mined information from 30,000 BlackBerry
apps and found a strong correlation between an apps star
rating and its number of downloads.3
Rajesh Vasa and his colleagues4 and Leonard Hoon and his
colleagues5 analyzed user reviews of mobile apps. They found
that the depth of feedback and the range of words were higher
when the users gave a low ratinghighlighting such reviews
usefulness.
Other researchers have performed manual analysis to
highlight critical information for developers. Ferdian Thung and
his colleagues manually categorized bugs in machine-learning
systems.6 Similarly, Yuan Tian and her colleagues manually
analyzed the content of software engineering microblogs to
better understand what developers microblogged about.7
Our research (see the main article) complements this previ-
ous research. We manually analyzed user reviews to identify the
most frequent and impactful complaints that led to low ratings.
References
1. S.M. Mudambi and D. Schuff, What Makes a Helpful Online Re-
view? A Study of Customer Reviews on Amazon.com, MIS Q., vol.
34, no. 1, 2010, pp. 185200.
2. H.-W. Kim, H.L. Lee, and J.E. Son, An Exploratory Study on the
Determinants of Smartphone App Purchase, Proc. 11th Intl DSI and
the 16th APDSI Joint Meeting, 2011.
3. M. Harman, Y. Jia, and Y. Zhang, App Store Mining and Analysis:
MSR for App Stores, Proc. 9th IEEE Working Conf. Mining Software
Repositories (MSR 12), 2012, pp. 108111.
4. R. Vasa et al., A Preliminary Analysis of Mobile App User Reviews,
Proc. 24th Australian Computer-Human Interaction Conf., 2012, pp.
241244.
5. L. Hoon et al., A Preliminary Analysis of Vocabulary in Mobile App
User Reviews, Proc. 24th Australian Computer-Human Interaction
Conf., 2012, pp. 245248.
6. F. Thung et al., An Empirical Study of Bugs in Machine Learning
Systems, Proc. 23rd Intl Symp. Software Reliability Eng. (ISSRE
12), 2012, pp. 271280.
7. Y. Tian et al., What Does Software Engineering Community
Microblog About? Proc. 9th IEEE Working Conf. Mining Software
Repositories (MSR 12), 2012, pp. 247250.

stand why users give low ratings to

iOS apps, we focused on reviews
High-
with one- and two-star ratings. We rated
explored two sources of informa- app
list All
tion: the star ratings and their as- App Select Collect the extracted
sociated comments. We manually market the apps reviews reviews
Low-
tagged the comments to uncover rated
common complaints. Such tagging app
is time-consuming, so we focused list
on a subset of apps and tagged a
statistically representative sample Tag New
of their reviews. Figure 1 diagrams Select Statistical complaint type
the Tagged
the reviews sample identified No
our studys design. reviews reviews
with 95%
confidence Yes
Selecting Apps level
We picked the 20 most popular iOS
apps, as defi ned by the iOS App
Store during June 2012 (see Table 1) FIGURE 1. Overview of our study process. Tagging is time-consuming, so we focused
that were free to download. (How- on a subset of iOS apps (20) and tagged a statistically representative sample of their
ever, some of these apps required a reviews (6,390 reviews for the 20 apps).

M AY/J U N E 2 0 1 5 | I E E E S O F T WA R E 71
 FEATURE: MOBILE APPS

Statistics of the studied iOS apps.

TABLE 1

Rating No. of No. of poor reviews No. of sampled

quality App Category stars (1 and 2 stars) reviews

High Adobe Photoshop Express Photo & Video 3.5 1,030 280
(3.5 stars)
CNN News 3.5 1,748 315

ESPN Score Center Sports 3.5 2,630 335

EverNote Productivity 3.5 1,760 315

Facebook Social Networking 4.0 171,618 383

Foursquare Social Networking 4.0 1,990 322

MetalStorm: Wingman Games 4.5 1,666 312

Mint Personal Finance Finance 4.0 1,975 322

Netflix Entertainment 3.5 13,403 373

Yelp Travel 3.5 2,239 328

Low Epicurious Recipes & Shopping List Lifestyle 3.0 940 273
(< 3.5 stars)
FarmVille Games 3.0 10,576 371

Find My iPhone Utilities 3.0 846 264

Gmail Productivity 3.0 1,650 312

Hulu Plus Entertainment 2.0 4,122 351

Kindle Books 3.0 3,188 343

Last.fm Music 3.0 1,418 302

Weight Watchers Mobile Health & Fitness 3.0 1,437 303

Wikipedia Mobile Reference 3.0 1,538 308

Word Lens Travel 2.5 1,009 278

fee for premium features.) We en-
sured that the apps had at least 750
reviews so that a few users didnt
skew the tagged reviews we ana-
lyzed. We also ensured that half of
the apps had an overall high rating
(3.5 or more stars) and that the other
half had an overall low rating (less
than 3.5 stars) because we wanted to
identify the complaints in both the
high- and low-rated apps. The apps
covered 15 of the 23 categories in
the iOS App Store.
Collecting Reviews
The iOS App Store doesnt provide a
public API for automatically retriev-
ing reviews. So, we obtained the re-
views from AppComments (http://
appcomments.com), a Web service
that collects reviews of all iOS apps.
We built a Web crawler that visited
each unique page with a specific iOS
review. We parsed the reviews to ex-
tract data such as the app name, the
review title, the rating, and the com-
ments. We collected all the reviews
for each of the apps during the first
week of June 2012.
Selecting Reviews
The 20 apps had more than 250,000
one- and two-star reviews. As we
mentioned before, we studied a sta-
tistically representative sample of the
reviews. To determine the sample
size, we used Creative Research Sys-
tems Sample Size Calculator (www
. su r veysystem.com /ss c a lc.ht m).
We randomly chose the sample to

72 I E E E S O F T WA R E | W W W. C O M P U T E R . O R G / S O F T W A R E | @ I E E E S O F T WA R E
 Inputs = All reviews (each with a review title and comment) and a list of complaint types (which is initially empty)

For each review:

Manually examine all the text in the review.

If the review matches an existing complaint type:

Tag the review with that type.

Else:
Add a new complaint type to the list of complaint types.
Restart tagging with the new list of complaint types.

Outputs = All reviews (tagged with the appropriate complaint types) and a list of the complaint types

FIGURE 2. The review-tagging procedure. This iterative process helped minimize the threat of human error during tagging.

achieve a 95 percent confidence level
and a 5 percent confidence interval.
This means were 95 percent confi-
dent that each result is within a 5
percent margin of error.
For example, Adobe Photoshop
Express had 1,030 one- and two-star
reviews. The statistically representa-
tive sample for 1,030 reviews, with
a 95 percent confidence level and a
5 percent confidence interval, is 280
reviews. So, we randomly selected
280 of those reviews for manual
examination.
In total, we manually examined
6,390 reviews. We performed our
sampling on a per-app basis because
different apps have varying numbers
of reviews and we wanted to cap-
ture the complaints across the differ-
ent apps. The number of randomly
sampled reviews for each app ranged
from 264 to 383 (see the last column
of Table 1).
Tagging Reviews
To identify the complaint types, we
performed coding, which turns qual-
itative information into quantitative
data.5,6 One of us read each review
to determine the type of complaint it
mentioned.
Figure 2 shows how we tagged
the reviews. Each time we identi-
fied a new complaint type, we went
through all the previously tagged
reviews to see whether to tag them
with the new type. We had to restart
tagging three times after discovering
new types. Sometimes, a reviewer
provided no meaningful comments
(for example, simply saying the app
was bad). We tagged such reviews as
Not Specific. For reviews contain-
ing multiple complaints, we tagged
them with multiple complaint types.
For example, if a review mentioned
a network problem and complained
about the app crashing, we tagged
the review with Network Problem
and App Crashing.
Results
We ended up with 12 complaint
types (see Table 2).
The Frequency of Each Complaint Type
We calculated the frequency of the
complaint types for each app. Then,
we normalized the frequency (the
number of complaints of a specific
type divided by the total number of
sampled reviews for an app) so that
we could compare results across dif-
ferent apps with a varying number
of reviews. Owing to the high devi-
ance of each complaint type between
different apps, we used the median
to summarize the frequency of each
complaint type across the apps.
The first three columns of Table 3
show the complaint type, its rank, and
its median percentage. Three types
accounted for more than 50 percent
of all complaints: Functional Error,
Feature Request, and App Crashing.
To better understand Functional
Error, the most frequent type, we
examined the most frequently used
terms in the related reviews. Then,
we read through the review com-
ments that used these terms. We
found that 4.5 percent of the func-
tional errors were about location is-
sues and that 7.3 percent were about
authentication problems. Heres an
example of a functional-error review
in which a user reported an authenti-
cation problem:
Dont do the update! When I try to
log in, it just keeps refreshing the
screen.
Examining Feature Request, we
found that most requests were app

M AY/J U N E 2 0 1 5 | I E E E S O F T WA R E 73
 FEATURE: MOBILE APPS
TABLE 2
Complaint type
App Crashing
Compatibility
Feature Removal
Feature Request
Functional Error
Hidden Cost
Interface Design
Network Problem
Privacy and Ethics
Resource Heavy
Uninteresting
Content
Unresponsive App
Not Specific
specific. However, 6.12 percent of
the requests were for better notifica-
tion support.
Network Problem, Interface De-
sign, and Feature Removal com-
plaints were also frequent. Another
complaint was Compatibility, which
is an important issue for iOS devices.
This refers to the app not working
correctly on a specific device or OS
version. Surprisingly, complaints
about compatibility, resources, and
app responsiveness werent as fre-
quentwe expected more of them.
We also examined whether the
complaint types varied between the
highest- and lowest-rated apps. We
compared the frequency of each com-
74 I E E E S O F T WA R E | W W W. C O M P U T E R . O R G / S O F T W A R E
Identified complaint types.
Description
The app often crashed.
The app had problems on a specific device or an OS
version.
A disliked feature degraded the user experience.
The app needed additional features.
The problem was app specific.
The full user experience entailed hidden costs.
The user complained about the design, controls, or
visuals.
The app had trouble with the network or responded
slowly.
The app invaded privacy or was unethical.
The app consumed too much energy or memory.
The specific content was unappealing.
The app responded slowly to input or was laggy overall.
The users comment wasnt useful or didnt point out a
problem.
plaint type among the 10 highest-
rated and 10 lowest-rated apps. To
do this, we used a two-tailed Mann-
Whitney U test with < 0.05. We
didnt find any statistically signifi-
cant difference between the highest-
rated and lowest-rated apps.
Our findings highlight the im-
portance of software maintenance
for iOS apps because many of the
frequent complaints were related di-
rectly to developmental issues (for
example, Functional Error, App
Crashing, and Network Problem).
We believe developers can avoid such
low ratings by an increased focus
on QA. Also, low ratings frequently
contain information that can help
| @ I E E E S O F T WA R E
Example review
Crashes immediately after starting.
I cant even see half of the app on my iPod Touch.
This app would be great, but get rid of the ads!
No way to customize alerts.
Not getting notifications unless you actually open
the app.
Great if you werent forced to buy coins for REAL
money.
The design isnt sleek and isnt very intuitive.
New version can never connect to server!
Yet another app that thinks your contacts are fair
game.
Makes GPS stay on all the time. Kills my battery.
It looks great, but the actual gameplay is boring and
weak.
Bring back the old version. Scrolling lags.
Honestly the worst app ever.
developers identify features users
want or really hate.
The Impact of Each Complaint Type
We determined which of the most
common complaints were the most
negatively perceived by users. We
looked at the ratio of one- to two-
star ratings for each complaint type
(across all apps). For example, a ra-
tio of 5 indicated that a type had five
times as many one-star ratings as
two-star ratings.
The last two columns of Table 3
show the rank and ratio for each com-
plaint type. The most negatively per-
ceived complaints differed from the
most frequent complaints. Privacy and
 The most frequent and impactful complaint types.*
TABLE 3

Most frequent Most impactful

Complaint type Rank Median (%) Rank 1:2 star ratio

Functional Error 1 26.68 7 2.10

Feature Request 2 15.13 12 1.28

App Crashing 3 10.51 4 2.85

Network Problem 4 7.39 6 2.25

Interface Design 5 3.44 10 1.50

Feature Removal 6 2.73 3 4.23

Hidden Cost 7 1.54 2 5.63

Compatibility 8 1.39 5 2.44

Privacy and Ethics 9 1.19 1 8.56

Unresponsive App 10 0.73 11 1.40

Uninteresting Content 11 0.29 9 1.50

Resource Heavy 12 0.28 8 2.00

Not Specific 13.28 3.80

* All results are at the 95 percent confidence level.

This column indicates the ratio of one- to two-star ratings across all apps.

Ethics, Hidden Cost, and Feature Re- For example, Hulu Plus is free to Discussion
moval were the three most negatively download but has a monthly sub- For many of the complaints, users
perceived complaints (and were mostly scription cost and ads in streaming reported they had recently updated
in one-star reviews). This means that videos. Because of the monthly sub- their app. So, we wanted to study
users were bothered most by issues re- scription requirement, more than the apparent relationship between
lated to privacy invasion and the app 55 percent of the low ratings for updates and complaints. We also
developers unethical actions (for ex- Hulu Plus were about the hidden examined the relevance of different
ample, unethical business practices costs. On closer examination, we types of complaints to software proj-
or selling the users personal data). found that the low ratings were due ect stakeholders (for example, devel-
To avoid such complaints, developers to the developers poor description opers versus project managers).
should access only the data (for ex- of the app or a misunderstanding
ample, the users contacts or location) by the user. Update-Related Complaints
specified in the apps description. Developers should devote extra We could know whether a complaint
Hidden Cost indicated users dis- attention to App Crashing, Hidden was update related only if the user
satisfaction with the hidden costs Cost, and Feature Removal com- mentioned it in the review; however,
needed for the full experience of an plaints because theyre frequent and other complaints could also have
app. This complaint showed up in 15 users perceive them negatively (see been update related.
of the apps. When an app was free to Table 3). Also, our study results Approximately 11 percent of the
download but not free to use, the us- stress the importance of developers sampled reviews mentioned that
ers were disappointed and often gave establishing trust and expectations a recent update impaired existing
low ratings. with app users. functionality. In 22 percent of these

M AY/J U N E 2 0 1 5 | I E E E S O F T WA R E 75
 FEATURE: MOBILE APPS

reviews, the users mentioned Func-

ABOUT THE AUTHORS

tional Error complaints after updat-

HAMMAD KHALID is a software engineer at Spotify. His re- ing. Most of these complaints were
search examines the link between user feedback and software app specific. For example, one user
quality. Khalid received a masters in computer science from
reported a bug affecting Adobe Pho-
Queens University. Contact him at hammad@cs.queensu.ca;
http://hammad.ca. toshop Expresss function keys:

Useless now: Was very useful till

last update. ... Function keys no
longer appear during editing.

EMAD SHIHAB is an assistant professor in Concordia Univer- Also, 18.8 percent of post-update
sitys Department of Computer Science and Software Engineer- reviews included requests for a new
ing. Hes particularly interested in mining software repositories,
or previously removed feature. In
software quality assurance, software maintenance, empirical
software engineering, and software architecture. He received a addition, 18.2 percent of the post-
Natural Sciences and Engineering Research Council of Canada update reviews complained about
Alexander Graham Bell Canada Graduate Scholarship. He has frequent crashing.
served on the program committees of the International Confer-
Developers often release free apps
ence on Software Maintenance, the Working Conference on
Mining Software Repositories (MSR), the International Confer- in hopes of eventually monetizing
ence on Program Comprehension, and the Working Conference them by transforming free content or
on Reverse Engineering. He has also been an organizer of the features to paid ones. We found that
MSR 2012 challenge and MSR 2013 data showcase and a
6.8 percent of the post-update re-
program chair for the 2013 International Workshop on Empirical
Software Engineering in Practice. Contact him at emad.shihab@ views complained about this hidden
concordia.ca. cost. Another important post-update
complaint dealt with changes in the
interface design; 6.2 percent of the
post-update reviews contained such
MEIYAPPAN NAGAPPAN is an assistant professor in the complaints.
Rochester Institute of Technologys Department of Software On the basis of these fi ndings, we
Engineering. He previously was a postdoctoral fellow in the
recommend that developers pay spe-
Software Analysis and Intelligence Lab at Queens University.
His research centers on using large-scale software engineering cial attention (for example, through
data to address stakeholders concerns. Nagappan received a regression testing and user focus
PhD in computer science from North Carolina State University. groups) to features they might con-
He received a best-paper award at the 2012 International Work-
sider removing, to adding fees, and
ing Conference on Mining Software Repositories. Contact him at
mei@se.rit.edu; mei-nagappan.com. to user interface changes they might
introduce. Even if users have previ-
ously liked an app, a bad update
could be irritating enough to make
AHMED E. HASSAN is the Natural Sciences and Engineering them give the app a low rating.
Research Council of Canada / BlackBerry Software Engineer-
ing Chair at the School of Computing at Queens University.
His research interests include mining software repositories,
Identifying Stakeholders
empirical software engineering, load testing, and log mining. Because users review apps as a
Hassan received a PhD in computer science from the University whole, they often raise issues that
of Waterloo. He spearheaded the creation of the International arent directly the developers re-
Working Conference on Mining Software Repositories and
sponsibility; some complaints are
its research community. Hassan also serves on the editorial
boards of IEEE Transactions on Software Engineering, Empirical directed toward product managers
Software Engineering, and Computing. Contact him at ahmed@ or other team members. To identify
cs.queensu.ca. these stakeholders, we divided the
complaints into three categories.

76 I E E E S O F T WA R E | W W W. C O M P U T E R . O R G / S O F T W A R E | @ I E E E S O F T WA R E
 Development-related complaints
were related directly to developers.
They included App Crashing, Func-
tional Error, Network Problem, Re-
source Heavy, and Unresponsive
App and constituted 45.6 percent of
all the complaints. So, many of the
complaints were directly related to
problems developers could address.
Strategic complaints primar-
ily concerned project managers but
could also partially target devel-
opers. These complaints included
Feature Removal, Feature Request,
Interface Design, and Compatibil-
ity and constituted 22.7 percent of
all complaints. The issues related to
these complaints required greater
knowledge of the project and pri-
orities and usually didnt have a
straightforward solution.
Content complaints concerned
the content or value of the app it-
selfdevelopers had little or no
control over the issues related to
these complaints. These complaints
included Privacy and Ethics, Hid-
den Cost, and Uninteresting Con-
tent. Addressing these complaints
would require rethinking the apps
core strategy (the business model or
the content offered). Although these
complaints accounted for only 3.02
percent of all complaints, Privacy
and Ethics and Hidden Cost had the
most negative impact, as we men-
tioned before.
Subscribe today for the latest in computational science and engineering research, news and analysis,
CSE in education, and emerging technologies in the hard sciences.
Potential Threats
to Validity
Because we performed our study on
only a sample of 20 iOS apps, our re-
sults might not generalize to all iOS
apps. To mitigate this threat, we max-
imized the coverage of complaints by
studying apps that covered most of
the categories in the App Store.
Also, as we mentioned before,
one of us manually tagged the re-
views. During this process, human
error or subjectivity could have led
to incorrect tagging. To address this
threat, the other authors randomly
inspected the reviews and corre-
sponding tags.
U
ser reviews strongly af-
fect developers and orga-
nizations that develop iOS
apps. Low ratings negatively reflect
on their apps quality, thus affect-
ing the apps popularity and even-
tually their revenues. To compete in
an increasingly competitive market,
developers must understand and ad-
dress their users concerns.
Our fi ndings point to new soft-
ware engineering research avenues,
such as how ethics, privacy, and
user-perceived quality affect mo-
bile apps. We plan to expand on this
study by considering more apps and
comparing our fi ndings across other
mobile platforms.
www.computer.org/cise
M AY/J U N E 2 0 1 5
References
1. S. Perez, iTunes App Store Now Has 1.2
Million Apps, Has Seen 75 Billion Down-
loads to Date, Techcrunch, 2 June 2014;
http://techcrunch.com/2014/06/02/itunes
-app-store-now-has-1-2-million-apps-has
-seen-75-billion-downloads-to-date.
2. S. Agarwal et al., Diagnosing Mobile
Applications in the Wild, Proc. 9th ACM
SIGCOMM Workshop Hot Topics in
Networks, 2010, p. 22.
3. N. Hu, P.A. Pavlou, and J. Zhang, Can
Online Reviews Reveal a Products True
Quality? Empirical Findings and Analyti-
cal Modeling of Online Word-of-Mouth
Communication, Proc. 7th ACM Conf.
Electronic Commerce (EC 06), 2006, pp.
324330.
4. J.A. Chevalier and D. Mayzlin, The
Effect of Word of Mouth on Sales: Online
Book Reviews, J. Marketing Research,
vol. 43, no. 3, 2006, pp. 345354.
5. C.B. Seaman et al., Defect Categoriza-
tion: Making Use of a Decade of Widely
Varying Historical Data, Proc. 2nd
ACM-IEEE Intl Symp. Empirical Soft-
ware Eng. and Measurement, 2008, pp.
149157.
6. C.B. Seaman, Qualitative Methods in
Empirical Studies of Software Engineer-
ing, IEEE Trans. Software Eng., vol. 25,
no. 4, 1999, pp. 557572.
Selected CS articles and columns
are also available for free at
http://ComputingNow.computer.org.
| I E E E S O F T WA R E 77