Professional Documents
Culture Documents
Aubrie Brault
Professor Gulbrandsen
ENL 620
Final Project
12/11/15
Brault 2
Brault 3
Table of Contents
Pre-report material
Table of contents
Report
Introductionbackground on the project (why, how, etc.) as
well as description of report structure, including information
contained in appendices
Description of users
10
Description of scenarios
11
12
18
36
43
46
Appendices
Brault 4
Heuristic evaluation
48
Personas
78
Test plan
Screening questionnaire completed for each participant
82
92
Moderator script
94
98
100
Brault 5
Brault 6
Executive Summary
Synopsis of study:
My focus within this study analyzes the official AirBnb website through the lenses of
content and community. Questions this the study considers are:
How well does the site engage with the audience?
How well does the company communicate who they are and what they do
(through the website)?
How effectively does AirBnb provide their service through the website?
My general findings to these questions have proved that although AirBnb connotes
effectiveness in these areas through design, they lack in genuinely interacting with their
users. It would appear they allow their unique concept to speak louder than their own
expertise in the business; meaning that the platform is an advanced design but that the
content and general usability practices need to better consider their users. From this, an
improvement in content management is recommended.
Generally speaking this studys findings offer the following suggestions:
1. AirBnb should expand their online presence
2. AirBnb should create new pathways of communication between company and
users, and amongst the users themselves.
3. AirBnb should offer more thorough investigations of parties involved and valid
proof thereof; in other words more guaranteed security.
4. AirBnb should stress focus on their users and conduct an official, broad but indepth, audience analysis.
Brault 7
Introduction
Why AirBnb?
I chose to conduct a usability test on the official AirBnb website for the following
reasons:
a) They are a relatively young company and therefore likely have more room for
improvement.
b) AirBnb appears pretty popular among some circles and non-existent among
others, implying that some further audience analysis would likely produce some
interesting results.
c) I personally enjoy travel and technology thus anticipated this site to be an ideal
opportunity to learn more about both.
Brault 8
Brault 9
Brault 10
Description of Users
Being able to ask the most likely users for AirBnb stems from about two months worth
of research. During this time, surveys and secondary research allowed me to create two
personas through which two likely demographics and their personality traits were
presented. The persona, Aylara Kovir, presented to be the most likely accurate lens
through which to approach this heuristic evaluation.
The user test subjects are described below:
Kerri-Lee
Melanie
Caty
Age:
21
25
25
Occupation:
Full-Time Student;
Marketing Specialist
for Renewal by
Anderson
Graduate Teaching
Fellow; Office Clerk
for UMass
Dartmouths Office
of Graduate Studies
Full-Time Engineer
at Lockheed Martin
Description: *All users match with general description of Aylaras qualities in some
capacity. Most accordingly, all users are in the age range, enjoy travelling/ word like to
travel more, do well for themselves with generating their own incomes, and are earning,
or already have, a Bachelors Degree.
Brault 11
Description of Scenarios
Traditional usability concerns, as well those that were highlighted in my HE, were both
influences in how I created the scenarios for this test. This combination allowed for a
more natural progression of scenarios to be produced. This set began with basic
processes and concluded with specific, complex ones. Realistic goals that average users
would have in coming to this site, were also kept in mind. Maintaining the natural
progression, the first few scenarios catered to more general needs.
It is scenario three in which my research questions starts becoming evident.
My research question for this project on AirBnb is:
How well does AirBnb foster communication and community?
Scenarios 3-5A highlight different facets of this questions, trying to answer it. The last
two scenarios, 4 & 5, both have follow-up scenarios designed to more naturally take the
user through the potential problems that could occur. For this reason, I intentionally
added thorough detail to these latter scenarios.
Brault 12
Overview
From the heuristic evaluation conducted beforehand, I had been lead to a certain set of
possible usability issues. With these core issues in mind, and the influence of other
tested heuristics, I generated the following heuristics.
Categories:
-
*Using the Bottom-Up approach I then clustered these individual findings into groups.
Ultimately I was able to put them into the following categories:
Brault 13
Kerri-Lee
Mel
Caty
Task 1:
Get a 1
Impression
st
Brault 14
Task 2:
Make an
Account
Task 2.1
What captures
your attention
while on Profile
platform/
account page?
Found easily
Preferred signing up
through FB quickest
Ignored other offers/ info
Found easily
Preferred signing up with
email/ also annoyed that she
had to confirm email
Ignored other offers/ info
Found easily
Preferred Email sign up
Likes its not asking her for
her credit card immediately
Appreciated coupons and
inspiration be sent to her
email also
Brault 15
Task 3:
Establishing
Credibility:
Brault 16
Task 4:
Choose and
Compare 2
Listing/ Evaluate
Satisfactory
Listing
Information
Brault 17
Task 4.1
Two ways of
getting direct
info about the
site you chose
Task 4.2
Desiring more
info on site that
is not reviews or
host
Task 5:
Cancelling a
Stay
Brault 18
Task 5.1
When
cancellation &
money back are
needed but the
Host is not
responding
Brault 19
Top-Findings
Reminder: the categories that the top-findings were broken into are Efficiency EffectiveEngagement and Accessibility.
* Note that for each section is first and overview table. Below each table, in each
corresponding section there are labeled screen shots with finding reiterated, then
followed by the recommendation.
Brault 20
Category A: Efficiency
Finding
Severity
Code
Corresponding
Theme of Issue
Recommendations
Minor
Issue
Comfortability
Brault 21
A2) All users wanted a better Minor
way to see where their
Issue
location choice fell on the
map that was provided. This
was desired to understand
surroundings better.
Below is figure A2
Comfortability
Brault 22
A3) 2 out of 3 users explicitly
stated they desired a better
way to compare chosen
possibilities (locations)
Below is figure A3
Cosmetic
Issue
Comfortability
Have a different
framework for
narrowing down
options. For
example, when in
the wishlist folder
that is created,
generate a viewing
option that allows
users to compare
their narrowed
choices by whichever
lens they choose. At
a minimum should
have ALL pertinent
pieces of info.
Brault 23
4) All users felt that, when
Major
they were looking for specific Issue
information, they were put
through an unnecessary
amount of windows/
loopholes
Credibility/
Comfortability
Brault 24
Severity
Code
Corresponding
Theme of Issue
Recommendations
Positive
Comfortability
Brault 25
Below is figure B1
Cosmetic
Issue
Community
Collect more
specified data and
use for areas on map
so that users can
interact with it to find
out more. Could
perhaps even have
fun facts or pop up
images in places they
hover over.
Brault 26
Below is figure B2
Brault 27
B3) All users expected
illustrations on how it
works page to be links
(portals) to another page or
at least move when clicking
them
Below is figure B3
Cosmetic
Issue
Community
Either consolidate
this information to
another, appropriate
page, or allow for
users to interact with
illustrations by
making them live.
Being that they are
icons of people,
perhaps each cartoon
person could speak
on behalf of the
community that icon
is meant to represent
and offer new
information to the
user.
Brault 28
B4) 2 out of 3 users would
have liked to see a mission
statement anywhere most
specified on the first/ home
page
Below is figure B4
Minor
Issue
Credibility
Provide at least a
succinct mission
statement and
welcome with
purpose of the site
somewhere on the
first page users come
to. (Although a basic
one liner is seen
below Welcome
Home not one user
recognized it; so at
least create emphasis
around this text.
Brault 29
B5) All users, at one point,
desired a feature that
allowed them to talk with
other travellers, specifically
with those who had been to
a site they were looking at
already.
Minor
Issue
Community
Have an instant
message chat with
any other user. Allow
users to turn these
notifications on and
off; but do allow
email messages to
be sent when
messenger is turned
off communication
framework like
Facebook.
**Note that, in the
screenshot below,
messenger is an
available application
but one only allowing
users to
communicate with
potential host (not
other users).
Below is figure B5
Brault 30
B6) All users commented on Cosmetic
the incentivizing banner that Issue
pops up on home page for
you, a new user, to invite
friends to join and take a
trip. Majority felt this was
misplaced and, or
distracting; in other words
they did not like it there.
Below is figure B6
Credibility/
Comfortability
Brault 31
7) 2 out of 3 users were
Major
primarily disconcerted with Issue
the fact they could not reach
AirBnb directly. In aftercommentary the third user
reflected, she too, ultimately
felt this was not good.
Below is figure B7
Credibility/
Community
Should have a
personal presence on
site. Should clearly
communicate how a
real person can be
reached (corporate).
They would also do
well to have a live
chat with an official
representative.
Brault 32
Category C: Accessibility
Finding
Severity
Code
Corresponding
Theme of Issue
Recommendations
Collect a variety of
both positive and
negative reviews
post to home page (a
place that is clearly
and immediately
visible) switch them
out every month and
make sure they have
not been altered by
company in any way.
Below is Figure C2
Brault 33
C3) Some listings users desired Major Issue Credibility
to rent were not in English at all
and Google Translate widget did
not work. (*Note this may have
occurred if such an app does not
recognize the variety of dialects
within a language.)
Below is Figure C3
Brault 34
C4) All users appreciated
information on CEOs but
wanted more than that to learn
about the company
Cosmetic
Issue
Credibility/
Community
Feature certain
employees every
month as welcome
representatives on
home page not only
provides welcoming
face but gives
background on
someone who works
for this company.
(Employee
appreciation implies a
positive work
environment).
Could also have a list
of departments and
what they do with
accompanying
pictures; would be
cool to have visual
directory. If you
wanted to get really
interactive you could
have a visual directory
in which workers use
video clips to
introduce themselves
and explain why they
decided to work at
AirBnb. Additionally
they could offer a
suggestion of a place
they would travel to
and why.
Brault 35
Below is figure C4
Brault 36
C5) All users felt the About Us Minor Issue Credibility
section had no substantial
information; it fell flat in serving
its real purpose; they left
wanting more
Below is Figure C5
AirBnb could do a
survey on what their
average customer
would like to see in
this section; better yet
I would suggest they
research and analyze
what some of the most
successful companies
put in these sections
(study the genre
constraints); they
should also more
deeply consider and
develop what kind of
persona they would
like to present here
Brault 37
C6) All users did find the policies Positive
on host contracts for staying
and thought terms agreeable
Below is Figure C5
Credibility/
Comfortability
Brault 38
Pre-Test Analysis
* For this section, the questions will be restated before the analysis of each response
provided.
1) What are your expectations in using/ coming to a new website?
From this response it was explicitly mentioned by each user that, when
approaching a new website they expected it to be easy to navigate as well as
clear, direct. These were the two kinds of responses that thematically ran
through each response. Integrated, these two desirables make for a category of
efficient navigation. More specific responses, however, were also given on an
individual basis. For example, one user further expanded on a navigational
element, specifying she also expected the site to be straightforward.
Meanwhile, other users were more concerned with design, purpose, and
credibility. For example, Melanie stressed all three in her response with stating,
professional engaging design was necessary, as well as having, the purpose
of the website clearly communicated (via ethos/ pathos/ logos). Lastly she
stressed some form of evident credibility.
It was interesting to see that one of three users was only concerned with explicit
credibility when arriving at a new site. It would seem the others might be
judging this same quality but more so through the lens of design, navigation,
and feel that they assess in their first impression. That being said, I did expect
most would be concerned with how easy it should be to go through the site.
2) In thinking of a travelling company, what characteristics/ qualities would you
expect them to embody?
From this response all users expected an enticing site that was aesthetically
pleasing, using appeals to pathos to their advantage. For example all expected
some form of colorful, bright, fun, exciting. Such adjectives are seen
next to their similar expectation of there being [quality] photos.
Additionally all did expect, in some way, credibility to be shown. Kerri-Lee
seemed to have the least concern for this topic with only implying the subject in
her desire for both positive and negative reviews. Meanwhile, Mel and Caty
provided more explicit responses. Mel used adjectives corresponding to this
trait such as, Professionalism, Trustworthy, Experienced, and
Brault 39
Knowledgeable. Similarly, Caty explained how she would expect them to be
well known and well advertised. Although Caty did not list specific qualities or
traits her explanation can be seen as implying those akin to what Mel had
pointed out. I was surprised the users did not have more to say in this response
to this question.
3) What is your first impression of AirBnb? And what does their company name mean
to you when you hear it?
From this response two out of three users had no prior knowledge of, or
association with, AirBnb. These two had interesting speculations on what it
might be. Kerri-Lee generally thought it to be either a travel agency or a science
company. Caty, on the other hand, was it bit closer in thinking AirBnb might be
bed and breakfast on an airplane.
Mel, although she did have prior knowledge of the company, only knew its
general concept and that there seemed to be good (low) rates for staying
somewhere.
Together the responses might suggest however that name AirBnb has no clear
connotations. Doubting that they would want to change their name, as it would
lose some artistic flare (just to be more simple and clear), be a difficult process,
and take away from the credibility they have already established among their
current customer base, AirBnb might want to think about a better branding
campaign. What I mean by branding campaign is for them to find a way of
broadening the audience the generally reach. This might mean that they just
expand efforts and reinforce more consistently (i.e. more ads in magazines per
year or in more places at once), or this might mean that they re-approach their
marketing design in how creative it is and what sort of impression it leaves. In
other words they may want to change how they are engaging their audiences
and to what depth.
4) What do you think of their idea/ product?
The responses for this question were definitely more sparse then others. I think
this was primarily because, even though one of the users had heard of it, none
of them actually ever used it. Kerr-Lee had no answer while Caty explained, that
if it were what she predicted in the last question (bed and breakfast on an
airplane) than she would be all for it. Mel was the most skeptical in saying that,
although it sounds good in theory, it actually could be a misadventure waiting to
go wrong.
Brault 40
It was interesting to see this here. The other two users, Kerri-Lee and Caty,
seemed to consider this aspect less. That being said, I do think Mel brings up a
reasonable concern. Safety should be a guarantee as much as possible.
5) If you were to use AirBnb for international travelling to someone elses home, what
3 things would you want to know most about and why?
The responses for this question varied more largely than others. Most common
on average concerned themes of safety and quality. Users Mel and Kerri-Lee
were most concerned with safety. Kerri-Lee would like to be notified about any
on-going conflicts (violence, wars, or crises) in the country she is going to. This
broader, more general, consideration suggests she is willing to take the average
risk that comes with traveling anywhere new but would not like to put herself
unnecessarily in a place of heightened risk. Addressing more of the exact
location, Mel on the other hand, would like to know about background checks
on the on the residence and host(s) as well as a little information on the level of
safety in that chosen neighborhood. Mels consideration suggests a much more
in depth look should be taken into who can become a part of AirBnb. Having
such background information on specific places and individuals would seem
necessary considering the amount of a traveller invests in the situation as well
as the literal fact that they are putting themselves directly into a closer vicinity
of unknown variables. Although the two, Kerri-Lee and Mel, are dealing with
different levels of safety, they both highlight the important ends of the
spectrum.
Kerri-Lee and Mel also agreed on another point, one considering quality of the
experience. Both mentioned they would like to have some way of knowing
about the surrounding of the place they have chosen to vacation at. This
indicates, that although they are open to trying new things, they would like to
be educated about what situation they could be walking into, and perhaps in
between the lines, what kind of culture they should expect to experience.
Interestingly, Kerri-Lee was the only one who considered what sort of
preparation she might need for a different country if any. For this reason she
would like to be aware of items needed for travelling to or in that country.
The most practical out of the three in this response, Caty, simply wanted to
know about costs, level of quality customer service, and coupons. Her response
suggests she would like to know more immediate, tangible factors before
considering potential risks. Seeming a methodical person, I believe she would
eventually weigh the latter information but only after she was sure of how
realistic a trip was for her.
Brault 41
Post-Test Analysis
1) How would you rate your experience with being able to choose a desirable
location?
On a scale from 1-10, all users rated this experience with an 8. Accompanying
comments were both positive and negative. They read, Not enough on the
site or, Very easy to locate places. Pretty pics.
This rating suggests that the site does enough to help even new users look
through and locate desirable locations. However, they could work on improving
this process itself. In other words, the issue here is internal; the final product is
useful to the users but the process is not as useful (or fulfilling) as it could be. I
believe a deeper audience analysis, by AirBnb, would be helpful in allowing
travelers to make better educated decisions and ultimately feel more connected
with, and trusting of, AirBnb.
2) How would you rate the general processing of booking such a location?
This response received different ratings from each of the users with some
commentary. Kerri-Lee rated this as 9, stating it was pretty easy. Mel rated
this as an 8, while Caty responded with a 7 and stating it was relatively
easy.
Thus the average rate general processing of locations received was an 8. This
number indicates, similar to the last, that improvements are being asked for but
the process is navigable enough.
3) How would you rate the information available to you, as a user, in the scenarios?
Here the average response is 9, again with the 3rd user (Caty) pointing out she
enjoyed the feature of reviews.
This number indicates that, generally speaking, the listings were satisfactory.
The numbers seemed to depend on what the user considered their biggest key
influences. Those who were less concerned with safety and mores so with
practical thoughts, gave this section a higher rating. But for those requiring
specific details, especially of safety concerns, when such information was
absent, not surprisingly gave this section the lowest rating (8).
Brault 42
4) Is there satisfactory customer service?
The average response to this question is 4. One user focused on the simple fact
that there was no contact information for the company period, and thus
indicated it was a 0 in her book for this question. Another implied it did not
matter initially but if a problem arose then that would be troublesome.
Interestingly, the user with the highest security concerns, rated this issue with
an 8.
In retrospect it seems that each of these numbers is reflective of the individuals
way of processing a problem. For those who were more passive, they did not
mind going through any, lengthy but available steps through the website and so
provided this question with a higher rating. Whereas for those more direct, and/
or who preferred instant gratification, the rating to this question was low.
For this range of users it could be concluded that the best approach for AirBnb is
to cater to both audiences (passive and aggressive).
5) Considering the scenarios, do you think anything on the website needs to be added
or altered? If so, what and how?
One theme emerging from the responses here was a request for direct contact
information.
Specifics put forth on an individual basis included areas of content inclusion, site
tools, and policy practices. In terms of content inclusion, Kerri-Lee put forth that
she would like to see a better method of seeing the exact location as well as an
actual list (or view) or the areas surrounding interests. This kind of information
would provide users with a clearer understanding of what their daily view would
look like, and more importantly, what daily activities and restaurants they might
have access to. In terms of site tools, Caty would have like to see a more
efficient method of direct comparison. In essence she would have liked a tool
that brought possible choices she liked together and compare the most
important details of each side by side. She pointed out however that this was
likely directly influenced with her work in the engineering field, as this is how
they typically make choices when buying products with/ for their company. This
note was included to point out how directly outside experiences can influence
potentially frequent users.
Lastly, in terms of policies, Mel stressed the importance of implementing
methods for which the company makes background checks on hosts. She
further brought up a point of liking to know whether or not this business was
certified by the BBB (Better Business Bureau). Though specific to her, these
needs highly influenced her interaction with the site throughout the user test;
Brault 43
and although it was not as much a focus for other users, they too all
commented, at some point, that they would like more background information.
For these reasons Mels thoughts are important for AirBnb to consider. There is
a constant thread of wanting more security and information on that security.
6) Would you trust this site? Would you use it?
Despite some of the high rates each user gave this site in the first questions the
majority answered this question with a consistent level of skepticism. The first
user, Kerri-Lee admitted, Im a little on the fence. This response sounds like
she wants to believe in the company because she likes the idea of what they
offer but might not have enough for her to comfortably use them for
international travel. On the other hand User 3, Caty, clearly said, Yes, it
appears official. Taking a moment to analyze the language though, it could be
argued that she is operating on first impressions and giving the benefit of the
doubt. The word appears indicates she is being influenced by visual but
subconsciously wonders how accurately those visuals represent their intended
message of success and credibility. If she did believe they were truly official
why then, did she not say something akin to, Yes. Their official website proves
to me I can trust them?
Interestingly, in the middle of both opinions, stands Mel. She admits the site
looks credible but remains unsure if she would pursue the experience they are
potentially offering.
7) Would you recommend it to a family member?
Two out of three users responded, yes. One of three users responded,
maybe.
It is interesting some may not use this themselves, or are at least very cautious
in doing so, but would still openly recommend to a family member. These
responses might suggest that the unique adventure the company offers is just
enough to override skepticism, or questions left unanswered.
8) Do you feel you are leaving this testing with any valuable information?
All users responded yes. Two our of three users commented on the point of
learning about a cool, new, and cheaper way to travel.
It appears that their first interaction with the site may have not been enough to
have them sign up for a trip right away but it was enough to leave them with a
positive impression. More importantly however, did it get them adjusting to the
concept. As the two who commented indicate, they like the idea of AirBnb.
Brault 44
Maybe this first visit on the site was enough to bring them back for
reconsideration and attempts to find more info.
9) What did you enjoy, or not enjoy, about this experience?
Two out of three users specifically enjoyed learning about something new. This
information suggests a good number of likely users for this site are openminded. (This was predicted by persona information.) It also suggests that the
content was interesting enough for them to walk away feeling excited and/ or at
least positive.
The other 1/3 user responded to this question about how aesthetically
appealing the site was for her. This note implies, that although she was not
certain if she would actually use their service, she at least felt engaged by the
visual presence on their website.
Brault 45
(Note: Original guidelines asked for short and long term changes or improvements. Not
understanding why there would be recommended changes for only short-term use, I
have here substituted with words more appropriate for my recommendations. The
substituted subjects will be quick fixes and complex fixes. In essence, the former will
take a short-term amount of time and the latter a long-term as the corresponding
processes will take longer to construct and implement.)
Quick Fixes
Complex Fixes
Place the search bar for finding locations at Work with a software developer on
the top of every page.
building an interactive map for under each
listing. Ideally this map can change in range
of scope - highlighting areas from, the city
as whole, all the way down to individual
buildings. (Google Earth might be the best
application for this.)
Brault 46
AirBnbs How It Works page could find a
better way to represent the information
currently on it. For its current state it
should either consolidate its information to
another, more appropriate location (page)
that contends with similar information. If
not, this page should at least clearly show
that there is more information available in
these areas; they can do this by
highlighting the links with a different color
and font size (creating emphasis) or,
perhaps better yet, they could have the
icons above the small paragraphs of
information interact with the user and be a
portal to the corresponding information.
Brault 47
Portray a positive work environment
through highlighting employee
appreciation; AirBnb could feature certain
employees every month as welcome
representatives on home page not only
provides welcoming face but gives
background on someone who works for
this company.
Or AirBnb could also have a list of
departments employees belong to and
what they do (accompanying pictures could
be provided with each name); this might
imply a visual directory. If they wanted to
get really interactive they could have a
visual directory in which workers use video
clips to introduce themselves and explain
why they decided to work at AirBnb.
Additionally they could offer a suggestion
of a place they would travel to and why.
AirBnb could do a survey on what their
average customer would like to see in their
section, About Us; for this they might
research and analyze what some of the
most successful companies put in these
sections (study the genre constraints); they
should also more deeply consider and
develop what kind of persona they would
like to present here.
Brault 48
Call To Action
What has this projects series of tests indicated? The HE and User-Test have supported
that further attention should be paid by AirBnb to consider their audiences. This
consideration breaks into multiple areas as I have demonstrated. That being said more
research should be conducted on the level of authentic interaction AirBnb generates. I
believe that my project has demonstrated that even when sites can be well-designed,
aesthetically appealing, work easily and are even easy to navigate, does not guarantee
that the content is structured in the best way possible. Additionally passing such basic
usability standards does not guarantee the quality of content. Within this quality of
content, the tests done on AirBnbs website have proven that a companys persona is
reflected in the quality of their content and is noticeable to the user in some capacity.
Thus, we are brought to a somewhat newer area in usability, one that extends beyond
the scope of simply asking how well the technology works. What I mean by this is we are
now considering an area that draws connections between well-done mechanics and
represented content - as the former highly influences the latter. Usability must do more
research in considering the bridge being built between technological platforms and
quality of content. I must further qualify that this quality of content ideally maintains
authentic access to all forms of even potentially pertinent information.
Appendices
Brault 49
1.
2.
3.
4.
5.
6.
7.
Brault 50
APPENDIX ITEM: 1
AirBnb :
A Heuristic Evaluation
By Aubrie Brault
Professor Gulbrandsen
ENL 620
11/3/15
Brault 51
Table of Contents
i.
ii.
iii.
iv.
v.
vi.
vii.
Introduction
Heuristics
Ranking System
Persona
Scenario
Tasks
Conclusion
p. 3
p. 3
p. 5
p. 6
p. 7
p. 7
p. 33
Brault 52
Introduction:
There are a variety of methods for evaluating user experience of websites and
their products. Often, before the opportunity of live testing with subjects on site
occurs, or if there are a little to no resources, then another useful approach can
be enacted to predict and find specific user problems likely most important to
the intended users. Heuristic Evaluations can be such resourceful, preventative
measures.
In my case, as a student, it is my lack of resources that most notably influenced
me to conduct a heuristic evaluation. I chose to take a unique heurisitic
approach to the company AirBnb. AirBnb was selected first and foremost for
my interest in how they are approaching travel and the unique solution they
provide. In my research however, I also became concerned with their ethics of
operation which seems to have broken down into how authentically, creatively,
and simply they may or may not be dealing with customers. It is for this latter
reason that I decided to integrate two different sources of heuristic principles.
Heuristics
For this research a somewhat new set of heuristics have been created to
appropriately and effectively evaluate the website of the international company,
AirBnb. The following consists of some of the 5 Es as adapted from the IUG
Evaluation Report (Example) provided to our class ENL 620. Integrated, for more
specific evaluation, are also heuristics from Grice (2013). Any heuristic principles
or details ending with *, are marked such so as to imply altercation. For
example, heuristic 3: Engaging has kept the title and its definition as seen in
IUGs report but, its more defining characteristics, have been pulled from Grice
to add a deeper context to meaning of that meaning.
Effective: How completely and accurately the work or experience is completed or
goals reached
quality of the user assistance
presentation of choices in a way that is clearly understandable
good interface terminology
redundant navigation
Efficient: How quickly this work can be completed
number of clicks or keystrokes required or the total time on task
navigation design elements
Brault 53
keyboard shortcuts, menus, links and other buttons
well-designed with clearly expressed actions
Engaging: How well the interface draws the user into the interaction and how
pleasant and satisfying it is to use
makes user feel welcome*
Use Visuals to draw users in*
Use sound to enhance the experience*
Make a Connection (Engaging extended): How well the website creates meaning
for the use*
Engage people in what is going on
Understand potential barriers and offer users identifiable ways to
overcome them
Use well-crafted story telling to immerse users in the encounter
Share Control: How well the site provides resources and encouragement for user
involvement.*
Follows standard usability guidelines
Provide users with resources to construct something
Provide a selection of professional-quality components for users
Make the process of interpretation participatory
Ensure user actions will not have bad or irreversible consequences
Brault 54
Ranking System:
To ensure I could get the most level of detail and thoughtful consideration of
what my researched persona, Aylara, finds troubling I have used the ranking
system as seen in the Hufflepuff Heuristic Evaluation Report. Rather than making
a table separate and away from the tasks they are referring to, these rankings
have been marked within the primary tables with the corresponding tasks. They
are color coded as you see here; this ranking system consists of the following
levels:
f) Positive Results in a beneficial effect on the users ability to perform
their given task
g) Cosmetic Issue: Affects the appearance and should be fixed only if time
permits
h) Minor Issue: Hinders the users ability to navigate and should be fixed
when possible
i) Major Issue: Frustrates or confuses users and requires repairs as soon as
possible
j) Catastrophic Issue: Prohibits users from performing their given task and
requires and immediate modification.
Brault 55
Personas
Restaurant Assistant Manager, The Brooklyn Grange
Photo:
Name:
Aylara Kovir
Restaurant Assistant Manager, The Brooklyn Grange
Job title/
major
responsibilities:
25 years old
Single
Demographics:
No children yet
Aunt to her older sisters son
Has her MBA and a minor in womens studies
Brault 56
Aylara feels she has more than earned her right to travel in the last few years of rigorou
work. For this reason, along with the fact the she seems to be the only one of her friend
who isnt making time for travelling, Aylara is ready to get out of the country. Hearing g
things about AirBnb from family members and such friends, she wishes to become fam
with the site, make an account, book a stay, and become a continuous user. Naturally
ambitious, liberal, and a risk taker Aylara believes this is the site for her. She likes their
seemingly authentic approach to how experiences should be made
Environment:
Aylara will be finding and securing a place abroad for she and her friends. She is most
comfortable experiencing new sites in either her NY studio with Wes her pup, or, at her
favorite local coffee shop. Being of generation X she is comfortable with the idea of the
as well as the technology it is likely to be built on and influenced by.
Quote:
Be sure that your service is more than satisfactory, aim for exceptional.
Scenario:
25 year old Aylara is preparing for her annual trip with three of her best friends
from her alma mater, NYU. The trip is in 6 weeks. She and her friends are looking
to spend a week in Amsterdam. Aylara is starting to prep now. She will be
making all of the booking arrangements for herself and friends, who are of
course, all staying together. Aylara has a had a few family members use AirBnb
with success so she is choosing them as well. That being said, she has never used
the site the before. Heuristics 4 and 5, Make a Connection and Share Control
respectively, are whole headers and defining characteristics as pulled from Grice.
Brault 57
Why? Such methods were pursued so as to avoid information that didnt
necessarily need to be evaluated. AirBnb is a company that follows general
usability concepts well so I wanted to be sure this time was spent analyzing more
critical, relevant data.
Tasks:
1)
2)
3)
4)
Make an account
Figure out how it works/ browse through site and see what you can learn
Book a stay
Setting up Payment Methods
Aylara Navigating
Tasks
Screen shots
Brault 58
Task 1: Make an Account
Issue: (Minor Issue)
Aylara was first drawn to Become a
Host as that has more attention being
drawn to it with box
Recommendations:
Box sing up option in white as well or
make bold
To make even more clear, AirBnb might
choose to place this somewhere in the
center of the page; however they do
have a How it Works button under the
Welcome Home which may make most
sense to keep there.
Brault 59
Task 2: General Browsing; see what
you can learn
Issues (Cosmetic):
Host opportunity popped up first (S/S
1), so Aylara wasnt sure if she was going
in the right direction as nothing told her
what may lay just beyond that. But she
decided to scroll anyway to make sure
(S/S 2) Thus, there is no real assistance
and there seems to be a strong emphasis
on hosts not making her feel
(subconsciously) very welcome.
Once she clicked How it Works (S/S
3) the links on the next page (S/S
4)werent well highlighted (or rather
drawing attention to themselves)
She felt the page itself wasnt all that
helpful. She wanted more explicit info on
what each category actually was (S/S 4).
Aylara would have preferred current
clients face for these categories rather
than generic, animated figures. (S/S 4)
Recommendations:
Offer online tech support via live chat
offer at bottom of window with
highlighted or hovering link
It seems like it would make more sense
to have travel information located
primarily throughout site as that is an
introductory act; working with a
company is more of an in depth process
you become a representation of that
company more than just a travelling
client does Or they could categorize
these options a little better; maybe offer
general info on home page and
testimonials and then have a traveller
tab and a host tab at upper right hand
corner as first segues into the
corresponding information.
Put color on hyper-links
Brault 60
Use more direct, descriptive language
use suggestive figures with smiling
clients focused close ups
Brault 61
Task 3: Book a Stay
Issues:
(Cosmetic) Only potential issue was
overlapping prices on map didnt
separate themselves if she hovered over
them. Thus, she couldnt really tell what
she was looking at by price alone (on the
map). (S/S 1)
(Minor) Aylara had a hard time finding
any mention of a cancellation policy
She wanted to know just in case. (S/S 2 &
3)
Recommendations:
small software fix allow prices to be
differentiated from one another while
mouse hovers over particular ones.
create any easily found section for this
with highlighted header or hyperlink
have owner explain in his or her own
terms as long as it is clear
make PDF downloadable version with
signature from host
place information after availability
section (last category on left hand side)
Brault 62
Task 4: Setting Up Payment Methods
Issue (Major):
Aylara had a hard time finding this area
(S/S 1 & 2)
Once she did, vague language made
choices more difficult (S/S 3)
Additionally, form was not immediately
ready or out in front of her the
additional step of having to click add
payment method seems unnecessary/
redundant (S/S 3)
Recommendations:
Maybe have site direct someone to
setting up payment methods
automatically once any account is made
aka direct audience to it so that they are
already set up when they go to book
something and dont have to look for it
use more explicit language
and also just categorize users from the
get-go; on main page offer each user to
operate under a certain hat (host or
traveller) sort of like when you view
mycourses through a student view or
your own facebook via an outsider view.
From this AirBnb could then categorize
everything they are operating in under
terms of host needs; this way
information doesnt have to be split up,
lagged out, or juxtaposed awkwardly
amongst each other and different users.
Doing this would allow users to be both a
traveller and a host they just have to
choose the option of operating as one or
the other (but both are accessible in
their own account).
Brault 63
Tasks
Task 1: Make an Account
What works (Positive):
Making an account for Aylara was
made easy. It took her 3 minutes to
complete the task. On the site itself,
even with an initial scan of the page,
the sign up took 1:18 seconds. The
remainder of the time was spent
going into her email and confirming
her new airbnb account. The sign up
portion of this task (on the website)
only required 3 clicks. Aylara found
navigation simple as well. The initial
sign up button was where she
expected it to be. Immediately after
clicking it a floating window brought
sign up log in information to her.
Recommendations:
No recommendations
Screen shots
Brault 64
Task 2: General Browsing; see what
you can learn
What works (Positive):
Took Aylara 5 minutes to give the
site a brief once over (for this she
only veered 2 clicks away from the
main page in any direction).
Shorter pages show more is
available to scroll down to making
scroll down bar unnecessary. It also
engages the reader/ user further into
the site.
Issues (Minor):
When Aylara scrolled all the way
down, which was a ways, there was
no option to bring her back to top of
the page.
Links werent made immediately
identifiable. Images of places led to
further information about those
locations but are presented as 2D.
Recommendations:
Provide a back to the top option once
one has scrolled down passed a certain
point or a small icon that they can
identify will do this and click on that.
(The latter option may be more
aesthetically pleasing
Put a more distinguished look to
clickable items, i.e. have shadows make
images look 3D
Brault 65
Task 3: Book a Stay
(average clicks per locale 7)
What works (Positive):
Because Aylara was searching for
places in an entirely different
country/ culture she spent more time
looking through the options. On
average she reviewed 9 posts per
hour spending about 10 minutes with
each. Aylara found design elements
very user friendly for what content
was posted. One navigation feature
she particularly liked was the fact
that every individual location opened
up its own tab. This allowed Aylara to
keep open and compare various
locations at once, while still returning
to the mainframe to choose yet
another.
Aylara also appreciated ratings,
number of reviews posted, and
profile pictures of the hosts were
provided before clicking on a
location.
Recommendation:
Work with Google Earth and have
virtual tour of location
Brault 66
Task 4: Setting Up Payment Methods
Issues (Major):
Somewhat harder to find
Actions Not Clearly Expressed
Recommendations:
Have a primary payment method be
the first step someone organizes once
they have made an account; as soon
as they have returned from
confirming their email address. With
this though the site should also allow
additional payment methods and
future changes.
Much like when apps change and
show automatic tips or offer a
tutorials for those changes AirBnb
should also consider this. (especially
tutorial walk-throughs.)
More explicit direction and
language needed for payment
methods. Use different descriptive
titles.
Brault 67
Heuristic 3: Engaging: How well the interface draws the user into
the interaction and how pleasant and satisfying it is to use
makes user feel welcome*
Use Visuals to draw users in*
Use sound to enhance the experience*
Tasks
Task 1: Make an Account
What works (Positive):
Welcomes user directly with large font;
unique choice of the word home
connotes the website is a home for the
user comfy, trusted, place. Also a
unique inception for implying the
purpose of the site to share homes and
authentic experiences.
Offers Help link on home page
Addresses user directly
Visuals in background are actually
silent short films
Warm tones for button graphics
Issue (Cosmetic):
No Sound
Recommendations:
Add light background music to main
home page looping sequence
Screen shots
Brault 68
Issues (Minor):
Where do you want go bar gets lost
in Aylaras initial scroll down; she was
more drawn to the overpowering visuals
Travelling user (non-host) not initially
welcomed during first exploration; the
natural reaction to scroll down Aylara
was greeted with host information
Organization of content initially made
it harder for her to understand it was her
options that then followed after Host
opportunities
Recommendations:
Use different graphics and an emphatic
tactic to highlight search bar just a bit
more or place it in a white space to break
from
Re-arrange content/information:
possibly offer tabs for intended use of
site - host or traveller (but dont grid lock
them into this role)
or after main image scroll down to a
split screen that introduces these roles/
options
Brault 69
Task 3: Book a Stay
What works (Positive):
Aylara feels welcome in that the
site is giving her a choice from the
beginning on how she would like to
narrow her choices. Although she and
her friends are willing to make this a
worthwhile experience that are not at
a point in their lives where they want
to spend above the average per night.
Something that stood out to her the
most were the profile pictures of
hosts.
Issue (Major):
Generally speaking most hosts
offered pictures of good quality and
many of them. But, being that the
amount of pictures and their quality
are determined by the host, Aylara
felt some, potentially desirable sites,
were left untouched because she
could not get a good sense of the
place/ space.
Recommendation:
Have AirBnb certify and regulate
pictures stock
Have them prepare a bare minimum on
number and types of pictures hosts must
include
Offer virtual tour of space
Brault 70
Task 4: Setting Up Payment Methods
What works (Positive):
More personal information is reached
through (reasonably) clicking on ones
name, which is highlighted by a personal
profile picture.
General design layout and colors
implemented did feel welcoming to
Aylara.
Issues:
(Cosmetic) Aylara expected little to no
visuals in this process and this remained
true; no visuals or sounds enhancing any
portion of the experience
(Minor) Aylara was not immediately
drawn to her other options; text on the
3rd screen shot in this category may be
organized well but it does not have
emphasis portrayed in appropriate areas
Recommendations:
Have icons for actions near categorized
content/ sections
Offer better differentiation between all
primary options on page / implement
emphatic strategies more explicitly
Brault 71
Screen shots
Brault 72
Task 1: Make an
Account
What works
(Positive):
Aylara is drawn in my
silent short films
going on as soon as she
reaches the main page;
though she does not
know the individual(s)
of each new showing
they are engaged in
their atmosphere
appreciating it and
that engages her as a
user.
Aylara felt there were
no barriers and she did
not experience any
technical difficulties
when she needed help
on the current page she
was on there was
always a help button
available that actually
changed its content to
the corresponding page
she moved to.
On that same bar, and
thus on every page, was
the AirBnb icon that if
licked would bring you
right back to the home
page.
Issue (Cosmetic):
Other than silent
short films at top of the
page Aylara didnt feel
any other real presence
of personal experiences
until she later go to
host listing/
descriptions and
reviews
Recommendations:
Brault 73
On the main page
immediately offer a few
head shots with blurbs
containing testimonials
(customer experiences
do come through when
people leave reviews
but there is no real
place for people to
comment on their
experience using AirBnb
as the coordinating 3rd
party (the real host))
Brault 74
Task 2: General
Browsing; see what
you can learn
What Works
(Positive):
On the whole a lot
does do well to engage
Aylara; there are a
variety of places
immediately displayed
to her
Issues:
(Major) Through
exhausting every link
she could find, there
was no way to directly
contact the company;
no email or phone
number is provided
(Cosmetic) Aylara had
also become
accustomed to seeing
owners, Ceos,
Presidents, and the like
all be introduced on
previous/other
websites specifically
the home pages, almost
as a greeting. Now she
realizes it was a way
she felt the site offered
direct, explicit
credibility. Here she
infers an anonymous or
omniscient presence
and she is not sure how
she feel abouts that.
Brault 75
Task 3: Book a Stay
What works:
There are various
filters she discovered,
that she may use to find
the place best for her
and her friends (S/S 1)
Issue (Minor):
Host information and
how much of it existed
was dependent entirely
on how much the host
wanted to give; there
seems to be no format
meaning they can
provide as little (or as
much) info as they
deem necessary. Even
with descriptions,
Aylara found in some of
them, not enough
pertinent information
or even some in English
for international
Travellers (S/S 2 & 3)
Recommendations:
Being that this is a 3rd
party site whose entire
mission it is to connect
people around the
world and cultivate real,
worthwhile experiences
it would be in
AirBnbs interest to
ensure all language is
accessible they need
to make requirements
for what Hosts post and
how they do it.
(Brief Listing)
Brault 76
(Non-English Listing)
Brault 77
Task 4: Setting Up
Payment Methods
Issue (Minor):
Aylara was not
directly engaged with
possible methods of
payment nor did she
feel supported in
attempting to figure
this out generally. The
potential barriers were
not so much physical
inhibitors or technical
errors but rather lack of
guidance and/or explicit
information.
Recommendations:
When user signs up
have a video tutorial
from the company
founder take them
through the payment
process. This will
accomplish both
engaging the reading
for an effective purpose
as well as establish
some sort of
relationship between
the company and the
user.
Brault 78
Tasks
Brault 79
Task 1: Make an Account
What Works (Positive):
Once Aylara had the time, she
enjoyed browsing most of the site
and was happy to find how well her
own movements on the site were
recorded but even more so how
many interesting ways there were to
express herself.
Issues (Major):
Initially, Aylara anticipated that
having a tab titled Your Trips would
allows some sort of narrative record
to be shared, especially on the site.
But when she went to go view other
random travelers profiles there
werent any trip stories depicted.
Aylara felt this should have be an
option. (S/S 1)
Although Aylara does like that she,
as a traveller, can also have this sort
of rating system as seen in screen
shot 2 she does think, if not
monitored, some people could use
this as an opportunity to put down
others.
Recommendations:
Offer a virtual scrapbook feature
for people to (however creatively
they wish) construct a shareable,
virtual version that can not only be
shared with friends, but viewed by
other travelers on the site to display
what they felt were some highlights
of specific locations.
AirBnb should consider instituting a
policy (similar to the coordinator of
our tech works study case in ENL
265) that is monitoring these systems
in an appropriate manner. This is not
to say negative reviews should be
deleted, as they are just as helpful,
but rather to ensure no one is using
Brault 80
this as an opportunity for malicious
intent.
Brault 81
Recommendations:
For important areas of booking,
payments, posting any sort of
personal or sensitive information
there should be, if not a direct and
clearly seen explanation on the page,
at least a clearly marked symbol that
signals this information may have
disclaimers the user should be aware
of. Ideally I would think this symbol
would also offer a direct hyper link to
the page/ information.
Brault 82
Task 3: Book a Stay
What Works (Positive):
Aylara found she could contact any
potential Host even before booking
with them
Other member are allowed, and
very much encouraged, to post
reviews of their stays.
She felt that trying to book a stay
with someone was a pretty natural
step-by-step process.
The tone and language that the site
operates in for this section (as in
most others) is very accessible with a
laid back and casual tone. This
particular feature made Aylara feel
more welcomed and subconsciously
that maybe staying in someone
elses home wasnt as strange as
might as first sounds.
Recommendation:
Additional, helpful features hosts
should be able to include a welcome
introduction video of themselves and
of their spaces being offered.
AirBnb should also encourage hosts
to offer a list of what might be the
best attractions in their area.
Brault 83
Issue (Major):
No way to connect with real human
intelligence if a problem arises
Recommendations:
Offer more immediate and
personable opportunities for user to
directly engage with company at
least offer number; here live chat
would also be a good idea.
Brault 84
Conclusion:
Conducting this heuristic evaluation there was an interesting mix of issues
brought to light. Aylara, a representative form of users, highlighted variables I
expected would not be as major of an issue has they were. Ironically, she also
downplayed things I anticipated might be more of an issue. Through her I was
able to see that most users her age, that also have a similar work ethic, may
indeed have a sense of instant gratification but are willing to look for things. On
the website/ product end of the HE spectrum my predictions of content over any
catastrophic usability items proved true. AirBnb then, might be an aesthetically
appealing site actually meeting all expectation of Nielsons 10 (had I used that
heuristic). In reality however, this HE has revealed that some deeper thought
needs to be employed in and throughout their own digital persona.
Brault 85
APPENDIX ITEM: 2
Personas
Restaurant Assistant Manager, The Brooklyn Grange
Photo:
Fictional name:
Job title/
major
responsibilities:
Aylara Kovir
Restaurant Assistant Manager, The Brooklyn Grange
25 years old
Single
Demographics:
No children yet
Aunt to her older sisters son
Has an MBA and a minor in womens studies
Brault 86
Aylara feels she has more than earned her right to travel in the last few years of rigoro
work. For this reason, along with the fact the she seems to be the only one of her frien
who isnt making time for travelling, Aylara is ready to get out of the country. Hearing
things about AirBnb from family members and such friends, she wishes to become fam
with the site, make an account, book a stay, and become a continuous user. Naturally
ambitious, liberal, and a risk taker Aylara believes this is the site for her. She likes their
seemingly authentic approach to how experiences should be made
Environment:
Aylara will be finding and securing a place abroad for she and her friends. She is most
comfortable experiencing new sites in either her NY studio with Wes her pup, or, at he
favorite local coffee shop. Being of generation X she is comfortable with the idea of th
as well as the technology it is likely to be built on and influenced by.
This companys product is ideal for me but their visuals dont do it justice and
it takes too long for the payment platform to load. Im looking elsewhere.
Quote:
What she looks
for in a site:
Brault 87
Photo:
Fictional name:
William Blakemore
Clinical Nurse Leader, California Pacific Medical Center
Job title/
major
responsibilities:
56 years old
Married
Demographics:
Brault 88
Environment:
William is a seasoned traveller. Recently he has wanted to start using AirBnb to take
Weekend family trips as well as weekend getaways with his wife. His main goal is to
Continue growing a cultural and spiritual balance with such a hard work schedule. Additio
William has a secret affinity for cooking so he likes to pick up special recipes among his
Travels. William found AirBnb as a recommendation from a google search. Like Aylara,
He would also like to become familiar with the site, make an account, book a stay, and lik
Remain a continuous user.
William will likely be browsing such sites and possible vacations either in his office after
hours, or in his man cave at their beach house. For his first look through he likes to browse
and then after some research has been done present some options to his wife. Unlike most
of his friends, Williams job has kept him up to date with most technology. Although he is
no 24 year old who grew alongside every generation of the iphone, William is generally an
intermediate user. Going into new sites he does not get initially overwhelmed or confused.
That being said William does admit he cannot keep up with some of the apps on sites,
Language used, and methods of finding certain information.
This company offers an interesting product, what is it made of and where can I purchase some
Quote:
What he looks
for in a site:
clarity, simple and easy navigation, contact information, professionalism, written explanation
accessible language and information,
Brault 89
APPENDIX ITEM: 3
Brault 90
Aubrie Brault
Professor Gulbrandsen
11/16/15
Purpose:
The purpose of this usability test is evaluate the official AirBnb website. A
previously conducted Heuristic evaluation identified that, although AirBnb is
generally successful by practical standards (Nielsons 10), it is lacking in contentoriented ways. These more social, philanthropic issues seem to be especially
pertinent to such a company as they are more than producing a product they
are in the making of experiences; where this differs from the former is that the
process is much more personal. Monies, individual safety, constant
communication and in depth knowledge then, are all required for this process to
be appropriate and successful. With the location of these issues already,
generally identified, the purpose of this usability test then becomes to better
understand why those issues exist, how customers naturally respond to them,
and most importantly, to gather data that can be analyzed for offering solutions.
The surface of AirBnbs usability practices might present itself as clear and
steady, but beneath that surface, there seems to be a broken infrastructure
existing unintentionally and out of direct reach of the clients it is aiming to
attract.
Thus, this purpose can simply be broken down into, what I like to call, the 3 cs:
Credibility, comfortability, and community.
Problem statement: What are the issues that need to be addressed in this test?
What are the goals of this test?
- The issues needing to be addressed in this test are
o The ease of which a first time user can navigate the site
o
o The comfort and confidence levels an audience can build with the
company and the product its offering.
Brault 91
o The lack of an online community
o The lack of company persona, presence, and interaction
-
Thus, essentially our goal of this usability test is to evaluate the website on
its abilities to be:
o Efficient
o Effectively-engaging
o Accessible
User Profiles
o Users Typical Characteristics:
Young (20-30)
Ambitious
Risk-Taker
But willing to look for answers via navigational curiosity
Computer literate/ technologically advanced
Prefers simplicity and clarity
Enjoys visuals and multimodal experiences
Gives and expects ethical considerations
Likes to feel in control and aware of important variables
Can be known to rush through things or not pay attention to
detail (depends on severity of situation)
o Motives:
Gain authentic worldly experiences/ check out places
internationally
Also be able to compare those international options
To try new things/ make new memories
To network/ communicate internationally
Brault 92
To get to know third party (the site/ company)
Understand the company and their priorities (as knowing this
ensures their best interest is (hopefully) at heart)
Scope out potential, future living locations
Preview potential homes and locations for travel
-
Brault 93
Consent form:
This user test is for graduate research intended use only. Any information
generated during this user test is only for enl 631 graduate research to be
specific.
By signing this form I am allowing the tester of this user test, Aubrie brault, to
have control of the information given. I further allow any video or audio
recordings to be used as implied.
Print:______________________________________
Signature:____________________________________
Date:______________________
Pretest Questionnaire
I.
II.
Brault 94
V.
SCENARIOS:
Scenario 1
- You really want to get some more travelling in but if youre going to spend the
money you want to make the most out of it. It just so happened that a friend at
work overheard your conversation and mentioned shed heard good things
about a company called AirBnb. So, now you have come to the website for the
first time and want to get a good first impression/ feel for the site. How would
you go about doing this on the website?
Scenario 2
- Interest genuinely piqued, you want to make an account and some more
exploring. How would you go about this on the website?
o Follow up scenario 2A: You are now at your personal account platform
and would like to look around. What draws your attention first?
Scenario 3
- You have now learned that the company caters to both hosts and travelers. You,
being a traveler, now have your curiosity piqued; the idea of getting a true,
authentic experience elsewhere, through the near-eyes of a local is enticing. But
before you fall head over heels in love with just buying a plane ticket and getting
a taste of the world, you want to know more about the company. They sound
interesting and appear professional, but perhaps for being a new company, you
have, at most, heard their name only whispered. Thus, you dont how the
company got started, whom it was started by, or how the idea for this product
was even inspired. Before investing money, time, and, in this more-uniqueprocess, personal safety - you want to know - who is this company, for real? Can
I trust them? How would you go about finding information that satisfies this
need for you, the customer?
Scenario 4
- Well, things seem to be pretty solid and, despite any reservations you might
have at the moment, you would like to go ahead and pick two locations to see
how they compare. Simultaneously, this little comparison, you believe, will also
demonstrate to you a bit more clearly how the site operates. Your primary goal
here however is consider the information the hosts provide you. In this situation,
Brault 95
you have enough money to comfortably book a stay 3 nights in Amsterdam with
2 friends, for up to $200 a night. By these details how would you go about doing
this on the website? Is there enough satisfactory information for you to
ultimately choose one location over another?
o Follow up scenario 4A: You have now chosen one of those locations and
would like to get more direct information about the location youve
picked. What might be at least two ways you go about doing this on the
website?
o Follow up scenario 4B: Imagine you have read all of the reviews under
this chosen location and many of them are old or static. In this situation
you know you would want more updated information but dont want the
likely biased opinion of the host. Sso for the sake of getting more
opinions, where else on the page (or site) might you look for such
information?
Scenario 5
- Imagine you have already booked a stay via AirBnb in Amsterdam but, with only
3 weeks left until the trip, a family emergency asks that you cancel. Having
never cancelled before however, you are unsure of how this works. In fact, you
dont know where to begin. How would you going about seeking information on
this process?
o Follow up scenario 5A: Still trying to cancel your stay, you immediately
contact the host, letting them know you can no longer make it, but they
are not responding to you. Needing the money back to help with the
family emergency you become worried In this situation what would you
do now?
Brault 96
How would you rate your experience with being able to choose a desirable
location?
II.
How would you rate the general process of booking such a location?
III. How would you rate the information available to you, as a user, in these
scenarios?
IV. Is there satisfactory customer service?
V.
Brault 97
VI. Would you trust this site? Would you use it?
VII. Would you recommend it to a family member?
VIII. Do you feel you are leaving this testing with any valuable information?
IX. What did you enjoy or not enjoy about this experience?
Evaluation Methods
o Quantitative: (done via computer video recording):
How many clicks per task
How long the task takes them to complete
o Qualitative:
Audio record commentary
Video record body language/ facial expressions
Pre and post questionnaires
Test Environment and Equipment
o Environment
Brault 98
o Ideally UMD book a room in the library
o ??
o Equipment
o Laptops (2)
o Ipad
o Wifi
o Adobe Premier App
Moderator Checklist:
XX: Before participant arrives
- Make sure producr is loaded and read for first scenario
o Make sure
Have consent form ready
Instructions are printed out for user
Site is up (wifi functional)
Camera is set up recording ready
iPad camcorder set up and ready
have note pad for any of my own notes these may include clicks per
page
make sure questionnaires are also printed out and readily available
pen as well
have Pandora music playing in background
A) Welcome:
- introduce yourself, thank participant for participating
-
offer refreshment
Brault 99
-
Go over the consent form, allow time to read and sign; if this has been done
already, ask participant if he/she is comfortable being recorded
Explain that there are observers who very interested in learning from the
participant about his/ her experience
C) Instructions:
- Explain process of using scenarios, one at a time, while participant thinks out
loud
-
Show think aloud protocol video and ask if they have any questions
APPENDIX ITEM: 4
CONSENT FORMS
USER 1:
Brault 100
USER 2:
Brault 101
___________________________________________
USER 3:
APPENDIX ITEM: 5
Brault 102
Pre-Test Feedback:
Question
Kerr-Lee:
Mel:
Caty:
- Easy to navigate
- Straight forward
- Not an over-load of
information
- Runs quickly
- Professional,
engaging design
- The purpose of the
website should be
clearly
communicated (via
ethos/ pathos/
logos)
- Credibility
I expect it to be
easy to navigate,
clear, and not too
busy.
2) In thinking of a
travelling company,
what characteristics/
qualities would you
expect them to
embody?
- Information about
destination
- Pictures
- Reviews (Negative
and positive)
- Colorful
- Professionalism
- Experienced
- Savvy
- Knowledgeable
- Trustworthy
- Fun
- Exciting
I would expect
them to be wellknown, well
advertised. To have
bright fun ads with
vacation pictures & I
would expect them
to provide a
complete experience
i.e. Hotel, Rental Car
etc.,
It is a company that
rates and markets
private residences,
which are rented out
to travelers as
discounted rates. I
dont know too
much about it.
Brault 103
4) What do you
think of their idea/
product?
N/A
Things needed to
travel to the other
country so I can be
prepared for the
trip.
Any type of
violence, wars, or
crises going on in
the country to
ensure that I will be
safe during my visit.
Fun things to do
around where Ill be
staying so that I can
have fun while Im
there.
Sounds lovely in
theory in practice,
it kind of sounds like
a murder waiting to
happen
Background on the
residence owners
(jobs, reference,
criminal checks).
I would want to
know:
Kerr-Lee:
Mel:
Caty:
8 Very easy to
locate places. Pretty
pics.
Brault 104
2) How would you
rate the general
processing of
booking such a
location?
7 Relatively easy.
10 So many
reviews
4) Is there
satisfactory
customer service?
- Any policy on
background checks/
more information on
hosts
- BBB (Better
Business Bureau)
certified?
- Add data
comparison feature
for hosts locations
that are being listed
- Put a corporate
phone #
Yes, it appears
official.
7) Would you
recommend it to a
family member?
Maybe.
5) Considering the
scenarios, do you
think anything on
the website needs to
be added or altered?
If so, what and how?
9 Pretty Easy
Brault 105
9) What did you
enjoy, or not enjoy,
about this
experience?
I liked learning
about something
Ive never heard of
before. However it
was a little lengthy.
Very aesthetically
pleasing and, at
times, funny.
I enjoyed seeing a
new site I would not
have found
otherwise.
Brault 106
APPENDIX ITEM: 6
Moderator Script
Welcoming the participant: Thank you for coming. How are you? [continue
small talk] Can I get you any water or coffee?
State the purpose of the study:
The purpose of this usability test is
evaluate the official AirBnb website. A previous test identified, that although it
can be aesthetically appealing, it does lack in some essential areas. In other
words, AirBnb looks pretty good, navigates generally well, but we want to look a
layer deeper. So, although we will also be taking note of practical mechanics, we
will also be looking out how you interact with the site consciously. The areas that
we are primarily focusing on can be thought of as credibility, comfortability, and
community. We would like to know if there is enough on the site for you to
confidently travel internationally and nationally. Now, during this process if you
also think of ways this experience could be improved, please let me know. I
really want to know what works well for you and what does not. That is the most
foundational purpose behind this testing. I would like to sincerely remind you
that this test is not an evaluation of you. It is not you that we are testing but the
product.
Provide necessary forms: Provide and explain the video consent form Are you
comfortable with being recorded? ... In this particular project you will be video
recorded and audio recorded. I will also be tracking things like how many clicks
per page etc. Such surveillance is intended to help me keep track of this, but
more importantly, it will also help me to see your initial responses while in the
scenarios. It is important that you think out loud clearly as well so that we may
refer back to your comments. This footage and audio will not be officially
published but is used for a graduate research course on usability at UMass
Dartmouth.
Describe the environment: [I will point out where any cameras and audio devices
are] I, of course, will be the only one with your while you are going through the
given scenarios. I will sit beside you, a few feet away.
Explaining the testing process: The testing process will begin with a pre-test
questionnaire with only 5 questions. Once you have completed the pre-test
questionnaire I will refocus your attention to the computer. Again, I will be
Brault 107
sitting beside you and this is so that I may also take notes for record.
Additionally, I am there if you have a practical question about the scenario
process and how to go about it. However, I cannot answer any theoretical
questions about the website; the most I can do is try and work through a
problem with you... Once we are seated in the respective places I will then ask
you to go onto the computer and head to AirBnbs official site.
Once you have completed a scenario please look at the laptop camera and
confirm scenario [whatever number] is now complete. At this point, till facing
the camera, you may add any commentary you have about the scenario just
completed.
For the conclusion I will please ask you to end with a post test questionnaire.
Describing thinking out loud: The think out loud element to this process is very
important. Though it might seem strange at first it gives us the necessary context
to your choices and more importantly helps us to understand how you are
evaluating what you are seeing and doing. As an independent party, your
thoughts on such matter are priceless. An example of thinking out loud might
sound like, hmm okay, Im seeing a lot of text on this first page but its not really
telling me what I expected to find here or, That is not at all what I expected to
see when I clicked on that link or, why is there no link here? I would expect to
seen one here. I may ask questions or remind you to share your thoughts if you
fall silent.
Ask the participant to share any questions or concerns: At this point, do you
have any questions or concerns? Also, at any point in this process you may stop
if you feel the need to. Thank you again for coming. Lets begin.
Start the study: [I will hand the participant the first scenario. I will then take my
seat beside them while they read the scenario; once they have finished I will
ask:] For this scenario, you can you restate what you are supposed to do? [Once
they do, we will being.]
Brault 108
APPENDIX ITEM: 7
Video Logs upon request.