Lewis 1

Karyn N. Lewis
0105-440-01 Internet Marketing
Rochester Institute of Technology Winter 20082
Research Paper

Evaluating Web Site Quality: A Benchmarking Approach

Executive Summary
Quality web site design is a critical success factor for corporations with an e-commerce strategy. The
assessment of web site quality is considered a problem of measuring user satisfaction in order to analyze user
perceptions and preferences, requiring a set of guidelines—or criteria—from which corporations can benchmark their
design activities. An effective web site evaluation model is one that makes it possible to pinpoint the web
characteristics that contribute to improving virtual customers’ attitudes and purchasing intentions. Much of the
previous research done on web quality assessment proposes a multivariate approach because of the complexity and
the multi-dimensional nature of the problem. From specified quality features, a set of satisfaction criteria can be
assessed reflecting all aspects of user perceptions about web site quality. This method of data collection and
analysis assumes a multicriteria preference disaggregation approach following an ordinal regression model, which
provides quantitative measures of customer satisfaction considering the qualitative form of customers’ judgments.
The main objective of this method is the gathering of individual judgements into a collective value function, assuming
the client’s satisfaction depends on a set of criteria. From this data, a series of perceptual maps—comparative
performance diagrams—can be developed to benchmark a set of competing organizations and define possible
improvement actions according to customer satisfaction.

Introduction
In the virtual enterprise, quality web site design is a critical success factor for corporations with an e-
commerce strategy because most contact with customers hinges on their interaction through a web site (Joia &
Barbosa de Oliveira, 2008). Most B2C e-commerce web sites, however, were not developed to cater to this aspect.
Although companies may try to emulate human behavior with technology, the interaction is different due to many
aspects that technology cannot replace—courtesy, friendliness, helpfulness, care, commitment, flexibility, and
cleanliness. With the proliferation of web sites and the commercial Internet invested in them, the assessment of web
site quality has evolved as an important activity (Grigoroudis et al., 2008). Business organizations throughout the
world invest a great deal of time and money in order to develop and maintain quality sites to provide effective
communication and information channels to their customers.
The assessment of web site quality is considered a problem of measuring user satisfaction in order to
analyze user perceptions and preferences (Grigoroudis et al., 2008). Avoidance of poor web site design demands a
set of guidelines—or criteria—from which corporations can benchmark their design activities (Kim et al., 2003).
Having this basic set of criteria serves the corporation’s effort when evaluating, comparing, monitoring, managing,
and designing web sites.

The Need for Web Site Quality Assessment

Modern web sites present a wide variety of features, complexity of structure and offered services
(Grigoroudis et al., 2008). Evaluation is an aspect of site development and operation that often contributes to
maximizing invested resources in serving user needs and expectations. A quality-oriented approach to web site
assessment would consider the web site as the product and the user as the customer, focusing on the analysis and
 Lewis 2

assessment of web site features that affect overall user satisfaction (Zhang & von Dran, 2001). Thus, exploring web
site quality and user expectations is the crucial key for site improvement.
When a consumer accesses a corporation’s web site, the appearance, structure, and maintenance status all
influence the consumer’s perception of both the transaction experience and the corporate image (Kim et al., 2003).
These characteristics of customer and company interaction developed via a web site are linked to the subjective and
objective elements that influence purchase, which makes careful web site planning of paramount importance. An
effective web site evaluation model is one that makes it possible to pinpoint the web characteristics that contribute to
improving virtual customers’ attitudes and purchasing intentions. The goal is to distinguish clearly between the
external aspects of web sites—those related to the particular behaviors of the user—from their internal aspects, or
those related to their design (Joia & Barbosa de Oliveira, 2008). In this way, the relationships between the many
different factors affecting a web site’s rating can be found in order to understand their contribution to the closure of an
online purchase.

Existing Web Site Quality Assessment Research

Literature on web site quality assessment includes the perspectives of a broad range of experts in human
factors, cognitive psychology and web development, as well as research addressing issues associated with the
design and usability of web products (Grigoroudis et al., 2008). Traditional research on web site evaluation methods
offers insight in achieving usable web-based interfaces. Several authors have undertaken studies regarding B2C e-
commerce web site evaluation using a variety of conventional methodologies including usability testing (e.g.,
Zimmerman, et al., 1998 cited in Hassen & Li, 2005), expert review (e.g. Zhang & von Dran, 2000 cited in Hassen &
Li, 2005), case studies (e.g., Smith, Newman, & Parks, 1997 cited in Hassen & Li, 2005), and automated assessment
(e.g., Tausher & Greenberg, 1997 cited in Hassen & Li, 2005).
In several cases, web site quality has been related to the level of user expectations fulfillment and quality
standards (Grigoroudis et al., 2008). The SERVQUAL model (Parasuraman et al., 1985, 1988, 1991 cited in
Grigoroudis et al., 2008) has also been used for web site quality assessment, proposing a universal set of quality
dimensions (tangibles, reliability, responsiveness, assurance and empathy). Much of the previous research done on
web quality assessment proposes a multivariate approach because of the complexity and the multi-dimensional
nature of the problem. Several web site quality aspects can be found in the literature, which may be summarized in
the following principal quality dimensions listed by Grigoroudis et al. (2008):
a. Content: Related to the responsiveness of a web site to satisfy user inquiry and trust regarding the information
offered (Beck, 1997 cited in Grigoroudis et al., 2008); described in several dimensions such as utility of content
(Grose et al., 1998 cited in Grigoroudis et al., 2008), content integration (Winkler, 2001 cited in Grigoroudis et al.,
2008), completeness of information, subject specialization (Nielsen, 2002 cited in Grigoroudis et al., 2008), and
content credibility.
b. Personalization: Examined at the following levels: personalization of information (Blankenship, 2001 cited in
Grigoroudis et al., 2008), personalization of interface (Brusilovsky, 2001 cited in Grigoroudis et al., 2008), and
personalization of layout (Winkler, 2001 cited in Grigoroudis et al., 2008).
c. Navigation: Reflects the support provided to users when moving in and around a site, including the following
dimensions: convenience of navigation tools (Vora, 1998 cited in Grigoroudis et al., 2008), means of navigation,
ease of use of navigation tools, and links to other sites.
d. Structure and design: Examined at the following levels: loading speed (Virpi and Kaikkonen, 2003 cited in
Grigoroudis et al., 2008), technical integrity (Shredo, 2001 cited in Grigoroudis et al., 2008), real time
information, software requirements, and browser capability (Vora, 1998 cited in Grigoroudis et al., 2008).
e. Appearance and multimedia: Captures aspects related to a web site’s look and feel, giving emphasis on state
of the art graphics and multimedia artifacts. Examined at the following levels: graphics representation, existence
and usability of images, and voice and video.
 Lewis 3

In the current era of e-commerce, researchers must consider the addition of business-related evaluation
criteria to the more dominant ones (Kim et al., 2003). Researchers have recently suggested various personalization
schemes that ostensibly make the visitor interested in the site and guide him or her to important pages (Jenamani &
Mohapatra, 2006). Studies have been conducted to provide unique navigation structure for the individual user using
personalization techniques built on user behavior models. According to Zhang and von Dran (2001), however,
designers must first have a thorough understanding of the different quality dimensions that affect user expectations.
They must then be able to relate these quality characteristics to specific design features.

Application & Understanding of the Benchmarking Technique

Benchmarking is a measuring method widely to improve organizations (Elmuti, 1998). The essence of
benchmarking is the process of identifying the highest standards of excellence for products, services, or processes,
and then making the improvements necessary to reach those standards. It’s more than just a means of gathering
data on how well a company performs against others—it’s a method of identifying new ways to improve processes.
Benchmarking strategies establish methods of measuring each area in terms of units of output as well as cost,
ultimately helping organizations faithful to the process achieve cost savings. Successful implementation of
benchmarking has been credited with helping to improve quality, cut costs in manufacturing and development time,
increase productivity, and improve overall organizational effectiveness. In web site quality evaluation, benchmarking
can be used to measure the performance of a web site against its competitors’ in order to identify its strengths and
weaknesses.

Table 1
Web site quality assessment criteria
Criterion Definition
Relevance It is related to the user perception regarding the significance of web site’s content to the visitor’s inquiry. The web site is
relevant according to the degree that the information it contains is relevant to visitor’s interests
Usefulness Usefulness extends relevance to the nature of the specific visitor’s inquiry. Developers should continuously check
information contained in the web site to assess usefulness to a wide audience of visitors. Often, web site administrators
ask visitors to evaluate provided information, or offer star based qualification of web site material
Reliability Reliability is related to accuracy of information contained in the web site. Often designers include a note about last
update of information. Doing so they help the visitor in forming an opinion regarding web site’s reliability
Specialization Specialization captures the specificity of information contained in the web site. It contributes to web site relevance and
usefulness, yet places a heavier burden on reliability
Architecture Architecture concerns the way that the content is organized in a web site. This dimension is particularly focused on the
arrangement of objects, which are used to convey information to visitors
Navigability This dimension reflects both the easiness and the convenience of moving in and around the site
Efficiency Efficiency captures the technical performance characteristics of the web site (e.g. is it fast? does the visitor get advance
notice about the estimated time it may take to retrieve information?)
Layout Layout reflects the unique aspects of the web site involved in the presentation of objects. It is related to its architecture,
yet it is used to differentiate the web site based on its unique design characteristics
Animation This dimension concerns the moving objects involved in the presentation of information and the web site-user interaction
Source: Grigoroudis et al., 2008*

*
Grigoroudis et al., 2008, p. 1348 cited in Moustakis et. al, 2004
 Lewis 4

In order to analyze the data accrued from a satisfaction survey, statistical methods and multivariate analysis
techniques must be used (Joia & Barbosa de Oliveira, 2008). The former are useful for analyzing each variable
graphically, identifying outliers, and confirming necessary premises. The latter are useful for validating the proposed
model as a whole as well as the relationships among the constructs.
From the aforementioned quality features, a set of satisfaction criteria can be assessed reflecting all aspects
of user perceptions about web site quality, as seen in Table 1. These satisfaction criteria are based on research by
Moustakis et. al, 2004, who conducted a wide experimental survey in order to conclude a relevant set of quality
features pertaining to user perceptions and preferences (Grigoroudis et al., 2008). Customer satisfaction surveys
often include both overall and partial satisfaction judgments for a set of criteria, requiring satisfaction judgement to be
included in the qualitative data collected in a benchmarking analysis. Thus, satisfaction-benchmarking analysis is
usually a problem of exploring this ordinal data and evaluating collective measures of satisfaction performance.
This method of data collection and analysis assumes a multicriteria preference disaggregation approach
following an ordinal regression model which provides quantitative measures of customer satisfaction considering the
qualitative form of customers’ judgments (Grigoroudis et al., 2008). The main objective of this method is the gathering
of individual judgements into a collective value function, assuming the client’s satisfaction depends on a set of criteria
like the aforementioned characteristics used to assess web site quality. This ensures maximum consistency between
value functions and customers’ judgements. The results of the method would be based on evaluation criteria weights
and additive/marginal value functions.
From this data, a series of perceptual maps—comparative performance diagrams—can be developed to
benchmark a set of competing organizations and define possible improvement actions according to customer
satisfaction (Grigoroudis et al., 2008). Criteria weights and satisfaction indices could be combined to indicate the
strong and weak points of customer satisfaction and define required improvement efforts. These diagrams would
present the average satisfaction indices of a particular company in relation to its competition. They are divided into
four quadrants and can be used as a benchmarking tool in order to assess the performance of different
characteristics of a company’s web site against its competitors, as in Fig. 1 and Fig. 2 below.
Fig. 1. Action diagram. Fig. 2. Comparative performance diagram

Source: Grigoroudis et al., 2008

Web sites usually contain enormous amounts of information spanning hundreds of pages (Jenamani &
Mohapatra, 2006). Without proper guidance, a visitor often wanders aimlessly without visiting important pages, loses
interest, and leaves the site sooner than expected. Therefore, web sites need efficient design that aid users in
retrieving the right information at the right time in order to increase their competitiveness. The benchmarking process
makes it easy to identify the gap between where the organization is and where it would like to be, providing a
measurement of the improvement an organization would like to make (Elmuti, 1998).
 Lewis 5

The satisfaction benchmarking analysis is mainly focused on the performance evaluation of competitive
organizations against the satisfaction criteria, as well as the identification of the competitive advantages of each
company (Grigoroudis et al., 2008). The analysis is based on comparative performance diagrams produced by the
multicriteria preference disaggregation method, which can help each company to locate its position against
competitors, pinpoint weak points and determine which criteria will improve its overall site performance, as in Table 2.
Table 2
Example: Average satisfaction indices (%)
Criteria Company A Company B Company C
Relevance 81 83 73
Usefulness 79 73 70
Reliability 79 82 75
Specialization 77 76 68
Architecture 71 68 72
Navigability 72 72 62
Technical efficiency 46 51 47
Layout 67 74 62
Animation 31 23 21
Global 68 66 61
Source: Grigoroudis et al., 2008*

Conclusion
As e-commerce expands, the design of web sites becomes a critical success factor (Kim et al., 2003). Web
sites are often the main interface between businesses and consumers, making their design as important as a store’s
layout and aesthetics. In order to check the efficiency and effectiveness of a design, good evaluation criteria are
needed. It is critical to consider all aspects of the virtual arena, so as not to miss important explanations about a web
site’s performance (Joia & Barbosa de Oliveira, 2008). Moreover, the user’s characteristics must be established in
order to better understand his/her behavior in a digital environment.
The main objective of a user satisfaction analysis is the identification of customers’ attitudes and
preferences. This type of analysis would include relative importance and level of demand for different satisfaction
dimensions of customers (Grigoroudis et al., 2008). The main advantage of this type of method is that it fully
considers the qualitative form of customer judgements and preferences, as expressed through customer satisfaction
surveys. Furthermore, the method is able to assess an integrated benchmarking system, given the wide range of
results provided. Thus, discussion is not solely focused on the descriptive analysis of customer satisfaction data, but
is able to emphasize customer preferences and expectations.
Customer satisfaction benchmarking analysis is a useful tool for modern business organizations in order to
locate their position against competition (Grigoroudis et al., 2008). It provides an organization with the ability to
identify the most critical improvement actions and adopt the best practices of an industry. This type of coordination
and cooperation for conducting analysis demands a great deal of integration that is normally still difficult to do (Joia &
Barbosa de Oliveira, 2008).

*
Grigoroudis et al., 2008, p. 1353
 Lewis 6

Bibliography

Beck, S., 1997. Evaluation Criteria. The good, the bad and the ugly: Or, why it’s a good idea to evaluate web sources. [Online]
Available at: http://lib.nmsu.edu/instruction/evalcrit.html [Accessed 18 January 2009].

Blankenship, E., 2001. Portal design vs. web design. [Online] Available at:
http://www.sapdesignguild.org/editions/edition3/graphic.asp [Accessed 18 January 2009].

Brusilovsky, P., 2001. Adaptive Hypermedia. User Modeling and User Adaptive Interaction, 11 (1/2), pp. 87-110.

Elmuti, D., 1998. The perceived impact of the benchmarking process on organizational effectiveness. Production and Inventory
Management Journal, Third Quarter, 39 (3), pp. 6-12.

Grigoroudis, E., Litos, C., Moustakis, V.A., Politis, Y. & Tsironis, L., 2008. The assessment of user-perceived web quality:
Application of a satisfaction benchmarking approach. European Journal of Operational Research, 187 (2008), pp.
1346-1357.

Grose, E., Forsythe, C., & Ratner, J., 1998. Using web and traditional style guides to design web interfaces. In: Human Factors
and Web Development. New Jersey: Lawrence Erlbaum Associates, pp. 121-131.

Hassan, S. & Li, F., 2005. Evaluating the Usability and Content Usefulness of Web Sites: A Benchmarking Approach. Journal of
Electronic Commerce in Organizations, Apr.-Jun., 3 (2), pp. 46-68.

Jenamani, M. & Mohapatra, P., 2006. Design benchmarking, user behavior analysis and link-structures personalization in
commercial web sites. Internet Research, 16 (3), pp. 248-266.

Joia, L.A. & Barbosa de Oliveira, L.C., 2008. Development and Testing of an E-Commerce Web Site Evaluation Model. Journal
of Electronic Commerce in Organizations, Jul.-Sep., 6 (3), pp. 37-54.

Kim, S., Shaw, T. & Schneider, H., 2003. Web site design benchmarking within industry groups. Internet Research, 13 (1), pp.
17-26.

Misic, M.M. & Johnson, K., 1999. Benchmarking: A tool for Web Site evaluation and improvement. Internet Research: Electronic
Networking and Policy, 9 (5), pp. 383-392.

Moustakis, V., Litos, C., Dalivigas, A. & Tsironis, L., 2004. Website assessment criteria. In: Proceedings of International
Conference on Information Quality. Boston: MIT, pp. 59-73.

Neilsen, J., 2002. Designing web usability: The practice of simplicity. Indianapolis: New Riders Publishing.

Parasuraman, A., Zeithaml, V.A., & Berry, L.L., 1985. A conceptual model of service quality and its implications for future
research. Journal of Marketing, 49, pp. 41-50.

Parasuraman, A., Zeithaml, V.A., & Berry, L.L., 1988. SERVQUAL: A multiple item scale for measuring consumer perceptions of
service quality. Journal of Retailing, 64 (1), pp. 14-40.

Parasuraman, A., Zeithaml, V.A., & Berry, L.L., 1991. Refinement and reassessment of the SERVQUAL scale. Journal of
Retailing, 67 (4), pp. 420-450.

Smith, P.A., Newman, L.A. & Parks, L.M., 1997. Virtual hierarchy and virtual networks: Some lessons from hypermedia usability
research applied to the World Wide Web, International Journal of Human-Computer Studies, 47 (1), pp. 67-95.

Tauscher, L. & Greenberg, S., 1997. How people revisit web pages. International Journal of Human-Computer Studies, 47 (1),
pp. 97-137.

Virpi, R. & Kaikkonen,. A., 2003. Acceptable download times in the mobile internet. In: Proceedings of the 10th International
 Lewis 7

Conference on Human-Computer Interaction. New Jersey: Lawrence Erlbaum Associates, pp. 1467-1472.

Vora, P., 1998. Human factors methodology for designing web sites. In: Human Factors and Web Development. New Jersey:
Lawrence Erlbaum Associates, pp. 189-198.

Winkler, R., 2001. Portals-The all-in-one web supersites: Features, functions, definition, taxonomy. [Online] Available at:
http://www.sapdesignguild.org/editions/edition3/overview_edition3.asp [Accessed 18 January 2009].

Zhang, P. & von Dran, G.M., 2000. Satisfiers and dissatisfiers: A two-factor model for web site design and evaluation. Journal of
the American Society for Information Science, 57 (14), pp. 1253-1268.

Zhang, P. & von Dran, G.M., 2001. User expectations and rankings of quality factors in different web site domains. International
Journal of Electronic Commerce, 6 (2), 9-33.

Zimmerman, E.D., Muraski, M., Palmquist, M., Estes, M., McClintoch, C. & Bilsing, L., 1998. Examining WWW designs: Lessons
from pilot studies. [Online] Available at: http://www.microsoft.com/usability/webconf/zimmerman.htm [Accessed 13
January 2009].