Measuring the User Experience by William Albert and Thomas Tullis - Read Online
Measuring the User Experience
0% of Measuring the User Experience completed



Measuring the User Experience was the first book that focused on how to quantify the user experience. Now in the second edition, the authors include new material on how recent technologies have made it easier and more effective to collect a broader range of data about the user experience.

As more UX and web professionals need to justify their design decisions with solid, reliable data, Measuring the User Experience provides the quantitative analysis training that these professionals need. The second edition presents new metrics such as emotional engagement, personas, keystroke analysis, and net promoter score. It also examines how new technologies coming from neuro-marketing and online market research can refine user experience measurement, helping usability and user experience practitioners make business cases to stakeholders. The book also contains new research and updated examples, including tips on writing online survey questions, six new case studies, and examples using the most recent version of Excel.

Learn which metrics to select for every case, including behavioral, physiological, emotional, aesthetic, gestural, verbal, and physical, as well as more specialized metrics such as eye-tracking and clickstream data Find a vendor-neutral examination of how to measure the user experience with web sites, digital products, and virtually any other type of product or system Discover in-depth global case studies showing how organizations have successfully used metrics and the information they revealed Companion site,, includes articles, tools, spreadsheets, presentations, and other resources to help you effectively measure the user experience
Published: Elsevier Science an imprint of Elsevier Books Reference on
ISBN: 9780124157927
List price: $49.95
Availability for Measuring the User Experience by William Albert and Thoma...
With a 30 day free trial you can read online for free
  1. This book can be read on up to 6 mobile devices.


Book Preview

Measuring the User Experience - William Albert

You've reached the end of this preview. Sign up to read more!
Page 1 of 1




1.1 What is User Experience

1.2 What are User Experience Metrics?

1.3 The Value of UX Metrics

1.4 Metrics for Everyone

1.5 New Technologies in UX Metrics

1.6 Ten Myths about UX Metrics

Myth 1: Metrics Take Too Much Time to Collect

Myth 2: UX Metrics Cost Too Much Money

Myth 3: UX Metrics are not Useful When Focusing on Small Improvements

Myth 4: UX Metrics Don’t Help us Understand Causes

Myth 5: UX Metrics are Too Noisy

Myth 6: You Can Just Trust Your Gut

Myth 7: Metrics Don’t Apply to New Products

Myth 8: No Metrics Exist for the Type of Issues We are Dealing with

Myth 9: Metrics are not Understood or Appreciated by Management

Myth 10: It’s Difficult to Collect Reliable Data with a Small Sample Size

The goal of this book is to show how user experience (UX) metrics can be a powerful tool for evaluating and improving the design of any product. When some people think about user experience metrics, they feel overwhelmed by complicated formulas, contradictory research, and advanced statistical methods. We hope to demystify much of the research and focus on the practical application of UX metrics. We’ll walk you through a step-by-step approach to collecting, analyzing, and presenting UX metrics. We’ll help you choose the right metrics for each situation or application and show you how to use them to produce reliable, actionable results without breaking your budget. We will introduce some new UX metrics that you might want to consider adding to your toolkit. We’ll give you guidelines and tips for analyzing a wide range of metrics and provide many different examples of how to present UX metrics to others in simple and effective ways.

Our intention is to make this book a practical, how-to guide about measuring the user experience of any product. We aren’t going to give you a lot of formulas; in fact, there are very few. The statistics are fairly limited, and the calculations can be done easily in Excel or some other common software package or web application. Our intention is to give you the tools you need to evaluate the user experience of nearly any type of product, without overwhelming you with unnecessary details.

This book is both product and technology neutral. The UX metrics we describe can be used for practically any type of product utilizing nearly any type of technology. This is one of the great features of UX metrics: they aren’t just for websites or any single technology. For example, task success and satisfaction are equally valid whether you evaluate a website, a smartphone, or a microwave oven.

The half-life of UX metrics is much greater than any specific design or technology. Despite all the changes in technology, the metrics essentially stay the same. Some metrics may change with the development of new technologies to measure the user experience, but the underlying phenomena being measured don’t change. Eye tracking is a great example. Many researchers wanted a method for determining where exactly someone is looking at any point in time. Now, with the latest advances in eye-tracking technology, measurement has become much easier and far more accurate. The same can be said for measuring emotional engagement. New technologies in affective computing allow us to measure levels of arousal through very unobtrusive skin conductance monitors as well as facial recognition software. This has offered glimpses into the emotional state of users as they interact with different types of products. These new technologies for measurement are no doubt extremely useful; however, the underlying questions we are all trying to answer don’t change that much at all.

So why did we write this book? There’s certainly no shortage of books on human factors, statistics, experimental design, and usability methods. Some of those books even cover the more common UX metrics. Does a book that focuses entirely on UX metrics even make sense? Obviously, we think so. In our (humble) opinion, this book makes five unique contributions to the realm of user experience research:

• We take a comprehensive look at UX metrics. No other books review so many different metrics. We provide details on collecting, analyzing, and presenting a diverse range of UX metrics.

• This book takes a practical approach. We assume you’re interested in applying UX metrics as part of your job. We don’t waste your time with unnecessary details. We want you to be able to use these metrics easily every day.

• We provide help in making the right decisions about UX metrics. One of the most difficult aspects of a UX professional’s job is deciding whether to collect metrics and, if so, which ones to use. We guide you through the decision process so that you find the right metrics for your situation.

• We provide many examples of how UX metrics have been applied within different organizations and how they have been used to answer specific research questions. We also provide in-depth case studies to help you determine how best to use the information revealed by the UX metrics.

• We present UX metrics that can be used with many different products or technologies. We take a broad view so that these metrics can be helpful throughout your career even as technology evolves and products change.

This book is organized into three main parts. The first part (Chapters 1–3) provides background information needed to get up to speed on UX metrics.

• Chapter 1 provides an overview of user experience and metrics. We define user experience, discuss the value of measuring the user experience, share some of the emerging trends, dispel some of the common myths about UX metrics, and introduce some of the newest concepts in UX measurement.

• Chapter 2 includes background information on UX data and some basic statistical concepts. We also provide a guide for performing common statistical procedures related to different UX methods.

• Chapter 3 focuses on planning a study involving metrics, including defining participant goals and study goals and choosing the right metrics for a wide variety of situations.

The second part (Chapters 4–9) reviews five general types of UX metrics, as well as some special topics that don’t fall neatly into any single type. For each metric, we explain what it is, when to use it, and when not to use it. We show you how to collect data and different ways to analyze and present it. We provide examples of how it has been used in real-world user experience research.

• Chapter 4 covers various types of performance metrics, including task success, time on task, errors, efficiency, and ease of learning. These metrics are grouped under an umbrella of performance because they measure different aspects of the user’s behavior.

• Chapter 5 looks at measuring usability issues. Usability issues can be quantified easily by measuring the frequency, severity, and type of issue. We also discuss some of the debates about appropriate sample sizes and how to capture usability issues reliably.

• Chapter 6 focuses on self-reported metrics, such as satisfaction, expectations, ease-of-use ratings, confidence, usefulness, and awareness. Self-reported metrics are based on what users share about their experiences, not what the UX professional measures about their actual behaviors.

• Chapter 7 is devoted to behavioral and physiological metrics. These metrics include eye tracking, emotional engagement, facial expressions, and various measures of stress. All of these metrics capture something about how the body behaves as a result of the experience of interacting with a user interface.

• Chapter 8 discusses how to combine different types of metrics and derive new metrics. Sometimes it’s helpful to get an overall assessment of the user experience of any product. This global assessment is achieved by combining different types of metrics into a single UX score, summarizing them in a UX scorecard, or comparing them to expert performance.

• Chapter 9 presents special topics that we believe are important but that don’t fit squarely into one of the five general categories. These include A/B testing on a live website, card-sorting data, accessibility data, and return on investment (ROI).

The third part (Chapters 10 and 11) shows how UX metrics are put into practice. In this part, we highlight how UX metrics are actually used within different types of organizations and how to promote the use of metrics within an organization.

• Chapter 10 presents five case studies. Each case study reviews how different types of UX metrics were used, how data were collected and analyzed, and the results. These case studies were drawn from UX professionals in various types of organizations, including consulting, government, industry, and not-for-profit/education.

• Chapter 11 provides 10 steps to help you move forward in using metrics within your organization. We discuss how UX metrics can fit within different types of organizations, practical tips for making metrics work within your organization, and recipes for success.

1.1 What is User Experience

Before we try to measure user experience, we should know what it is and what it isn’t. While many UX professionals have their own ideas of what constitutes a user experience, we believe the user experience includes three main defining characteristics:

• A user is involved

• That user is interacting with a product, system, or really anything with an interface

• The users’ experience is of interest, and observable or measurable

In the absence of a user doing something, we might just be measuring attitudes and preferences, such as in a political poll or survey about your favorite flavor of ice cream. There has to be behavior, or at least potential behavior, to be considered user experience. For example, we might show a screenshot of a website and ask participants what they would click if it were interactive.

You might also note that we never defined any characteristics of the product or system. We believe that any system or product can be evaluated from a user experience perspective, as long as there is some type of interface between the system or product and the user. We are hard-pressed to think of any examples of a product that don’t have some type of human interface. We think that’s a good thing, as it means that we can study almost any product or system from a UX perspective.

Some people distinguish between the terms usability and user experience. Usability is usually considered the ability of the user to use the thing to carry out a task successfully, whereas user experience takes a broader view, looking at the individual’s entire interaction with the thing, as well as the thoughts, feelings, and perceptions that result from that interaction.

In any casual conversation about usability, most people would agree that it’s good to have something that works well and isn’t confusing to use. On the other side of the coin, some companies may intentionally design products to be confusing or frustrating. Fortunately, this is a rare occurrence. For the purposes of this book, we will be somewhat idealistic and make the assumption that both users and designers want products to be easy to use, efficient, and engaging.

User experience can sometimes mean the difference between life and death. For example, the health industry is not immune to poor usability. Usability issues abound in medical devices, procedures, and even diagnostic tools. Jakob Nielsen (2005) cites one study that found 22 separate usability issues that contributed to patients receiving the wrong medicine. Even more troubling is that, on average, 98,000 Americans die every year due to medical error (Kohn et al., 2000). While there are no doubt many factors behind this, some speculate that usability and human factors are at least partially to blame.

In some very compelling research, Anthony Andre looked at the design of automatic external defribulators (AEDs)(2003). An AED is a device used to resuscitate an individual experiencing cardiac arrest. AEDs are found in many public spaces, such as shopping malls, airports, and sporting events. An AED is intended to be used by the general public with no background or experience in life-saving techniques such as CPR. The design of an AED is critical, as most individuals who are actually using an AED are experiencing it for the first time, under a tremendous amount of stress. An AED must have simple and clear instructions, and deliver them in a way that is time sensitive and also mitigates user errors. Andre’s research compared four different AED manufacturers. He was interested in how each of them performed in terms of users being able to deliver a shock successfully within a specified time limit. He was also interested in identifying specific usability issues that were impacting user performance with each of the machines.

In his 2003 study, he assigned 64 participants to one of four different machines. Participants were asked to enter a room and save a victim (a mannequin lying on the floor) with the AED they were assigned. The results he found were shocking (no pun intended!). While two machines performed as expected (0% errors from a sample of 16 participants for each machine), two other machines did not fare so well. For example, 25% of the participants who used one of the AEDs were not able to deliver a shock to the victim successfully. There were many reasons for this outcome. For example, participants were confused by the instructions on how to remove the packaging for the pads that adhere to the bare chest. Also, the instructions on where to place the electrodes were somewhat confusing.

After Andre shared his research findings with his client, they agreed to address these issues as part of a product redesign effort.

Similar situations can arise on a regular basis in the workplace or in the home. Just think of the written instructions for such actions as lighting the pilot light on a furnace, installing a new lighting fixture, or trying to figure out a tax form. An instruction that’s misunderstood or misread can easily result in property damage, personal injury, or even death. User experience plays a much wider role in our lives than most people realize. It’s not just about using the latest technology. User experience impacts everyone, every day. It cuts across cultures, age, gender, and economic class. It also makes for some very funny stories!

Saving lives is, of course, not the only motivation for a good user experience. Championing user experience in a business setting is often geared toward increasing revenues and/or decreasing costs. Stories abound of companies that lost money because of the poor user experience of a new product. Other companies have made ease of use a key differentiator as part of their brand message.

The Bentley University Design and Usability Center had the opportunity to work with a not-for-profit organization on the redesign of their charitable-giving website. They were concerned that visitors to their website would have difficulty finding and making donations to the charitable foundation. Specifically, they were interested in increasing the number of recurring donations, as it was an excellent way to build a more continuous relationship with the donor. Our research included a comprehensive usability evaluation with current and potential donors. We learned a great deal about how to not only improve navigation, but simplify the donation form and highlight the benefits of recurring donations. Soon after the launch of the new website, we learned that the redesign effort was a success. Overall donations had increased by 50%, and recurring donations increased from 2, up to 19 (a 6,715% increase!). This was a true usability success story, and one that also benefits a great cause.

User experience takes on an ever-increasing role in our lives as products become more complex. As technologies evolve and mature, they tend to be used by an increasingly diverse set of users. But this kind of increasing complexity and evolution of technology doesn’t necessarily mean that the technologies are becoming easier to use. In fact, just the opposite is likely to happen unless we pay close attention to the user experience. As the complexity of technology grows, we believe that user experience must be given more attention and importance, and UX metrics will become a critical part of the development process to provide complex technology that’s efficient, easy to use, and engaging.

1.2 What are User Experience Metrics?

A metric is a way of measuring or evaluating a particular phenomenon or thing. We can say something is longer, taller, or faster because we are able to measure or quantify some attribute of it, such as distance, height, or speed. The process requires agreement on how to measure these things, as well as a consistent and reliable way of doing it. An inch is the same length regardless of who is measuring it, and a second lasts for the same amount of time no matter what the time-keeping device is. Standards for such measures are defined by a society as a whole and are based on standard definitions of each measure.

Metrics exist in many areas of our lives. We’re familiar with many metrics, such as time, distance, weight, height, speed, temperature, and volume. Every industry, activity, and culture has its own set of metrics. For example, the auto industry is interested in the horsepower of a car, its gas mileage, and the cost of materials. The computer industry is concerned with processor speed, memory size, and power requirements. At home, we’re interested in similar measurements: how our weight changes when we step on the bathroom scale, where to set our thermostat in the evening, and how to interpret our water bill every month.

The user experience field is no different. We have a set of metrics specific to our profession: task success, user satisfaction, and errors, among others. This book gathers all the UX metrics in one place and explains how to use these metrics to provide maximum benefit to you and your organization.

So what is a UX metric and how does it compare to other types of metrics? Like all other metrics, UX metrics are based on a reliable system of measurement: Using the same set of measurements each time something is measured should result in comparable outcomes. All UX metrics must be observable in some way, either directly or indirectly. This observation might be simply noting that a task was completed successfully or noting the time required to complete the task. All UX metrics must be quantifiable—they have to be turned into a number or counted in some way. All UX metrics also require that the thing being measured represents some aspect of the user experience, presented in a numeric format. For example, a UX metric might reveal that 90% of the users are able to complete a set of tasks in less than 1 minute or 50% of users failed to notice a key element on the interface.

What makes a UX metric different from other metrics? UX metrics reveal something about the user experience—about the personal experience of the human being using a product or system. A UX metric reveals something about the interaction between the user and the product: some aspect of effectiveness (being able to complete a task), efficiency (the amount of effort required to complete the task), or satisfaction (the degree to which the user was happy with his or her experience while performing the task).

Another difference between UX metrics and other metrics is that they measure something about people and their behavior or attitudes. Because people are amazingly diverse and adaptable, we sometimes encounter challenges in our UX metrics. For this reason, we discuss confidence intervals with most of the UX metrics discussed in order to reflect the variability in the data. We will also discuss what metrics we consider relevant (and less relevant) in a UX context.

Certain things are not considered UX metrics, such as overall preferences and attitudes not tied to an actual experience of using something. Think of some standard metrics such as the Presidential Approval Ratings, the Consumer Price Index, or the frequency of purchasing specific products. Although these metrics are all quantifiable and may reflect some type of behavior, they are not based on actually using something in order to reflect the variability in the