Yelp Usability Test Report

CCTP 807: Innovation, Design & Usability Authors: Adam Gerber, Kenneth Pennington, and Amir Zonozi Communication Culture & Technology Georgetown University April 28, 2012

Yelp Usability Test

Page 1

TABLE OF CONTENTS Executive Summary Description of Study Project Description Test Description Methodology User Profile Test Environment & Equipment Test Results and Recommendations Evaluation and Measures (Data Collection) Results and Recommendations Conclusion Test Problems Discussion and Conclusion Appendices Appendix A: Test Protocol Appendix B: Post Test Questionnaire Appendix C: Test Data 13 16 19 11 11 6 6 4 4 5 5 6 3

Yelp Usability Test

Page 2

Executive Summary
To evaluate the usability of Yelp.com, we surveyed eight participants on their usability experience with various components of the sites design. The participants varied from moderate to advanced internet users, all of whom had familiarity with Yelps functionality. Following our data collection, we identified several opportunities for improvement of Yelps usability. In this paper, we will provide suggested guidance for improvements and modifications for a better user experience on Yelp. Our results are based on quantitative and qualitative data from our usability tests and reveal various problems and opportunities with the sites design. Much of our data reflects a shared need to clarify the tools on Yelp and provide a more obvious and direct set of tools for new users. A summary of problems addressed and recommendations provided is listed below. A more complete review of problems and recommendations is available in the Results and Recommendations section of this paper: Problem Statement 1. Can the user correctly enter search criteria into the proper search fields? Recommendation Move the two search bars to separate rows with one larger search button spanning the height of the two rows. Add clearer frames and titles to the page components can help guide eyes toward the most useful parts of the page first. Provide a more absolute price range for main courses rather than the abstract dollar signs. Provide the listing details in the right column and dedicate the majority of the main column to photos. The map should be the central feature of the page, moved to a more prominent, central place in the site architecture.

2.

When faced with a page with search results, can the user identify additional search criteria to drill down into the results to find targeted results?

3.

When faced with unsatisfactory results, can the user edit their criteria and sorting selections to retarget their results? When a proper search result is found, can the user make use of critical information within the individual businesses Yelp page?

4.

5.

Does Yelps model of business filtering and sorting add value to the process of searching for local businesses, compared to the users alternative methods?

Yelp strives to be an authoritative source for peer reviews of local business and to serve as the premiere online engine for discovering local businesses. Yelp has already succeeded at generating a large user base, but with some changes, Yelp can build user-confidence in the sites usability and, thus, further expand its competitive advantage.

Yelp Usability Test

Page 3

DESCRIPTION OF STUDY Project Description

Yelp is an engine for discovering and peer-reviewing local businesses. It provides opportunities for local business to list their services and for users to rate those services. The majority of users of Yelp, however, are neither reviewers nor business owners but rather causal browsers looking for new businesses to patronize. A common use case scenario for Yelp is a local user exploring restaurants in a certain neighborhood. We focused on this scenario to highlight the general, non-expert users experience because Yelps business model relies on mass readership of reviews and business listings. Yelps usability under this use case scenario provides several challenges to the casual user. Our usability study explores how these casual users experience certain core functions of the site. By executing usability tests among a range of participants, we identified some problems with and limitations of Yelps general usability. In order for Yelp to expand its base of causal browsers, it is imperative that it address these usability limitations embedded in the sites design as revealed and expounded in this usability study.

Test Description
To best serve Yelps business needs any user should be able to immediately and intuitively find the business they are looking for. By presenting users with multiple search criteria and filtration options on both the home page and search result page, Yelp is choosing the path of an unguided search. This is different than a guided search approach that takes the user through a series of ordered sorting and filtering options with limited options per page. Therefore, our test objective is to measure the following: 1. Can the user correctly enter search criteria into the proper search fields? 2. When faced with a page with search results, can the user identify additional search criteria to drill down into the results to find targeted results? 3. When faced with unsatisfactory results, can the user edit their criteria and sorting selections to retarget their results? 4. When a proper search result is found, can the user make use of key information within the individual businesses page, including the ability to scan reviews and learn about the businesss details? 5. Does Yelps model of filtering and sorting add value to the process of searching for local businesses compared to alternative methods? The test scenarios and post-test questionnaire provide quantitative and qualitative insights into how comfortable users are with Yelps tools. The usability tests were held the week of April 22 with eight participants in total. Each participant was provided with a pre-test questionnaire that clarified the roles, responsibilities, and rights of the participants.

Yelp Usability Test

Page 4

Methodology
The usability test consists of an introduction, a series of tasks, a short interview, and a post-test questionnaire. We instructed the participants to Think Aloud and share their thoughts with our recorders throughout the test. Following the task series, we gathered the users responses to various usability components of Yelp and some additional demographic questions as well. The test concluded with an opportunity for additional, open ended comments on the Yelps usability and the administered test, which provided additional opportunities for qualitative insight. The usability test took users approximately 20 25 minutes. The variation in testing time was often a result in changes to the language (but not content) of the task list and questionnaire design as well as the speed of the proctors speech. Introduction: 2 Minutes The introduction explained the rights, roles, and responsibilities of the various test parties as well as a short series of warm up questions to build a rapport between the participants and testers. Tasks: 15 Minutes The task list guided our participants through a particular use case scenario of selecting a new, local restaurant for that evenings dinner. We collected data on time to complete the tasks, but the primary data gleaned from this portion was in the think aloud comments provided as feedback to each task and new internal Yelp page. Post-Test Questionnaire: 5-8 Minutes Following the tasks, we provided our participants with a post-test questionnaire that asked for a number of responses to the usability and experience of using Yelp. Additionally we asked participants about their likelihood to use Yelp again, recommend it to friends, and to provide any additional positive and negative feedback on the sites design and usability. Please see Appendix A for the test protocol (including the introduction and tasks) and Appendix B for the post-test questionnaire.

User Profile
We recruited 8 participants for our usability study. We had one female participant and seven male participants. Six of our participants were between the ages 23-29, one was between 30-39, and one was between 40-49. Three of our participants used Yelp often, four had used it in the past but not often, and one was familiar with Yelp but had not used it before. None of our participants were unfamiliar with Yelp. Additionally, all of our participants were friends or associates with the designers of the test, all of whom are experienced web professionals. This did impact the neutrality of the survey population as it drew from a generally moderate to advanced internet user audience as opposed to Yelps more general, moderate internet user audience.

Yelp Usability Test

Page 5

We gathered additional user profile data on the participants general familiarity with other popular internet tools. For that additional user profile data, please see Appendix D.

Test Environment & Equipment

In order to perform the usability test, the following equipment and test environment were provided: an internet connected laptop (Mac or PC, based on the participants preference), a table with three chairs, note pads for the test team, the printed scripts, access to a power source, print outs of the user questionnaire and pencils.

Yelp Usability Test

Page 6

TEST RESULTS AND RECOMMENDATIONS Evaluation and Measures (Data Collection)

We evaluated our usability test using the following measures: Qualitative Measures: 1. Comments captured from each participants think aloud responses to the various tasks, 2. Observations of participants actions in completing each task, 3. Observations of participants responses to prompts and new pages on Yelp, 4. Observations of participants errors in completing tasks, and 5. Additional comments and questions provided by the participants. Quantitative Measures: 1. Time to complete tasks, 2. Demographic user data, and 3. Data collected from the post-test questionnaire on 7-point ratings of various usability and design components of Yelps site.

Results and Recommendations

Through the quantitative and qualitative data collected in the usability test, we captured the following key findings and recommendations for each of our stated objectives: 1. Can the user correctly enter search criteria into the proper search fields? Problem: Generally, all of our users immediately identified the proper use of the search fields and used them correctly. However, our least experienced internet user could not initially identify the presence of two separate search bar fields and entered both the location and restaurant theme into one bar, resulting in an initially incorrect search result.

Recommendation: The cause of this user error was the participant not realizing that there were in fact two distinct search boxes with separate purposes. This is an understandable and expected consequence of the similar and consistently horizontal look of the search fields. A simple solution to this user error would be to move the two search bars to separate rows with one larger search button spanning the height of the two rows. This would draw attention to the separateness of the fields and remind the user to fill in both separately.

Yelp Usability Test

Page 7

2. When faced with a page with search results, can the user identify additional search criteria to drill down into the results to find targeted results? Problem: Users had little difficulty identifying the search filters and using them correctly. However, some users reported a sense of crowdedness and overwhelming data on the search results page.

Recommendation: Adding clearer frames and titles to the page components can help guide eyes toward the most useful parts of the page first. 3. When faced with unsatisfactory results, can the user edit their criteria and sorting selections to retarget their results? Problem: We asked users to filter by price, which requires the user to select the highest available $$$$ option listed under the Price filter. The majority of participants were able to perform this without any trouble, but a few expressed concerns about the relativeness of the dollar signs. Participants were unsure if the dollar signs referred to overall cost or the cost of the most expensive item on the menu, as well as whether or not the measure was trustworthy. Other participants searched for a sort option for price, not a filter option. (See above image for filter options.)

Yelp Usability Test

Page 8

Recommendation: In addition to the number of dollar signs as a representation of price, a more absolute price range for main courses would be appropriate. Allowing a user to select an option such as Main Course $10-$15 would remove ambiguity over how relatively expensive a restaurant is. 4. When a proper search result is found, can the user make use of critical information within the individual businesses Yelp page? This includes the ability to scan reviews and learn about the businesss details. Problem: Nearly all of our users expressed a concern over the listing detail pages abundance of information and lack of information hierarchy. The page provides a great deal of detail, but it is unattractive and overwhelming, especially for less experienced users.

Recommendation: It is important to segment these listing detail pages by the most important information first. A potential solution could be to provide the listing details in the right column and dedicate the majority of the main column to photos, which was a common request from our participants and key component of major Yelp competitors.

Yelp Usability Test

Page 9

5. Does Yelps model of business filtering and sorting add value to the process of searching for local businesses, compared to the users alternative methods? Problem: Yelp provides two ways to examine restaurants: a list and a map. Some participants were not clear on how the two were related and expressed concern over focusing on one and not the other. Recommendation: The map features is Yelps distinct feature that helps users find business that are truly local. The map should be the central feature of the page, moved to a more prominent, central place in the site architecture.

Yelp Usability Test

Page 10

CONCLUSION Test Problems

While administering the test, we uncovered a problem with our test design. Our test protocol was highly descriptive and proved to be very easy for our participants. One of our quantitative measurement criteria was time to complete the various task. We captured time to complete task for our participants, however most of the tasks were completed immediately with little variation. In fact, most time variation was related to the time it took the test proctor to read the task. Our test protocol had very explicit and clearly outlined task prompts. We have learned that a more abstract task description provides a more natural test environment with greater potential for usability errors.

Discussion and Conclusion

In addition to our recommendations listed above, we have two primary findings regarding the usability of Yelp.com. First, Yelp is generally considered to have an outdated design that does not feel as contemporary and usable as its competitors. Second, Yelp is still the primary online resource for local business information despite its usability and design shortcomings. All of our participants remarked on the general sense of confusion and lack of clarity brought about by the homepage. Additionally, the participants with more advanced web experience were able to identify and express particular frustrations with the sites abundance of tools and lack of direction for new users.

Yelp Usability Test

Page 11

From these graphs we can glean that no one strongly agrees that Yelp is easy to use. The majority of our participants are neutral on Yelps intuitiveness. The majority of our participants are neutral to negative on Yelps attractiveness. Nevertheless, there is major consensus that these participants will still use Yelp again. This implies that there is a measure of familiarity and comfort around Yelps usability quirks that has allowed it to remain the internet leader in local business search despite its usability flaws. Additionally, 100% of our participants expressed that they would recommend Yelp to a friend.

The conclusion here is that Yelp has set the standard for what is expected of an online local business directory and people are satisfied with its current usability enough that it remains the clear leader. However, increased competition could capitalize on Yelps design shortcomings and provide a more intuitive interface, a more photo-centric design, and a more up-to-date look and feel. With the recommendations that we outlined above, Yelp can close the gap on those usability shortcomings and retain its competitive advantage as the leading online local business directory.

Yelp Usability Test

Page 12

APPENDIX A - PROTOCOL
ORIENTATION Welcome & Purpose Welcome and thank you for agreeing to participate in our usability test of Yelp.com. Our goal is to determine how easy or hard it is for you to find information about local businesses on Yelp's website. Subject Rights & Privacy Your participation is entirely voluntary and you can leave at any time. All results will be kept completely confidential. Facilitator and Participant Roles I will ask you to complete a set of tasks online and I will record your responses. It is very important that you think out loud or tell me what is going on in your mind as much as possible. We may ask you at times to explain why you said or did something. * Required Do you consent to participate in this usability study as outlined in the description above? * Yes No Do you have any additional questions? *If yes, please enter it in the 'other' box No Other: INTRODUCTION What is your favorite local restaurant to go to for special occasions? If you were looking for a new restaurant to try in your neighborhood, how would you go about it? Do you use any online tools for investigating new restaurants in your area? If so, which ones? PROTOCOL (TASKS) Today we are going to use Yelp.com, an online tool for finding and learning about local businesses in your area. Since you identify yourself who does not use Yelp regularly, we will ask you to complete a set of tasks on the site to better understand how you experience the website. Please try to think out loud as much as possible and share with us what is going on through your mind. TASK 1:

Yelp Usability Test

Page 13

Please enter www.yelp.com in the address bar. Welcome to Yelp.com. What are your initial responses to this web page? TASK 2: Lets assume that you are planning to meet some friends for dinner tonight. Your friends requested Thai food, but you are not too familiar with Thai restaurants in your neighborhood. A friend recommends using Yelp.com to discover local Thai options. Let's pull up a list of Thai restaurants in your neighborhood. The user should enter "Thai" under "search for" and the name of their neighborhood (or address) under "near". How did they do? Perfectly Other: What is your initial response to this search result page? TASK 3: We are now looking at a list of search results, a map, and a series of other options. We know that our visiting friends have quite sophisticated taste in Thai food. So, lets sort our list of responses to show the highest rated search result first. The user should select "highest rated" under "sort by". How did they do? Perfectly Other: What do you think the difference is between sorting by "highest rated" and "best match"If there answer is anything other than the obvious answer, please enter it below TASK 4: Now that we have a list of highly rated Thai restaurants in our neighborhood, we want to consider how expensive these restaurants are. We know that your friends are very generous and are willing to visit expensive restaurants. Lets FILTER our search results by only the most expensive category of restaurants. The user should select "$$$$" (or highest available) under "price". How did they do? Perfectly Other: TASK 5: Lets go back to seeing the list of all local Thai restaurants, regardless of price. We can see on the map all of the options we have available to us. Lets use the map to zoom in and take a look at Thai restaurants located within just a few blocks of our home. Yelp Usability Test Page 14

The user should de-select "$$$$" under "price", then use the map to zoom to their own neighborhood. 'Redo search when map moved' should be selected. How did they do? Perfectly Other TASK 6: Let's select the highest rated Thai restaurant in the neighborhood. What is your response to this page? TASK 7: We know that one of our visitors is a vegetarian, but you are uncertain how great the menu is for nonmeat eaters. Let's scan through the reviews to see if anyone has anything to say about the vegetarian options. The user should select a restaurant. It the new page, the user should either search for the search bar to "search reviews" or use their browsers search function to search 'vegetarian'. How did they do? Perfectly Other: TASK 8: Let's return to the search results page. We just learned that our friends prefer Mexican food, and not Thai. Let's change our search to find local Mexican food. The user should return to the results page and change the primary search criteria from 'Thai' to 'Mexican' Perfectly Other:

Yelp Usability Test

Page 15

APPENDIX B POST TEST QUESTIONNAIRE

Great! Thank you for your help today. We will now ask you a series of questions to get a better sense of your experience and skills online. Please answer these honestly and candidly as we are not going to grade you based on how advanced of an internet user you are. What is your age? * 18-22 23-29 30-39 40-49 50-59 60+ How would you describe your familiarity and comfort with the following online tools? Not Familiar Very Familiar Email o Facebook o Twitter o Open Table o Google Maps o Word Press o GitHub o o o o o o o o o o o o o o o o o o o

Please select which best describes your experiences with Yelp.com I use Yelp.com often I have used Yelp in the past but not often I am familiar with Yelp, but I have never used it I have heard of Yelp, but I do not know what it is used for I have not heard of Yelp and I do not know what it is used for Other: Bottom of Form Yelp Usability Test Page 16

Would you recommend Yelp to a friend? Yes No Other:

Now that you have experienced some of Yelp's key functionality, how do you view Yelp? Please check the most appropriate response Strongly Disagree Strongly Agree Yelp is Useful o

Yelp is Easy to Use o o Yelp is Trustworthy o o Yelp is Intuitive o o Yelp is Attractive o o I Will Use Yelp Again o o

What do you think about the way Yelp is designed? Strongly Unfavorable Neutral Overall Welcome Page o o o The Yelp Search Bar o o Search Filter Options o o Search Results, List o o Search Results, Map o o Yelp Usability Test

Strongly Favorable

o Page 17

Listing Detail Page o o

Did you notice any advertisements on Yelp? Yes No Other:

What was your impression of the advertisements? Neutral Helpful Unhelpful Other:

Did anything in particular stand out that you liked about Yelp? Did anything in particular stand out that you did NOT like about Yelp? Is there anything else you would like us to know? Thank you for your time and help today!

Yelp Usability Test

Page 18