Defining Website Comprehensibility From The Evaluators Perspective

Evaluation of Website Comprehensibility
Evaluation of Website Comprehensibility..........................................................1

Comprehensibility Project..............................................................................4
Defining Comprehensibility ..........................................................................5
Understanding the Evaluators Expertise..........................................................7
Defining the General Public Audience..............................................................8
STEM Topics.................................................................................................9
Web Site Sources........................................................................................10
Website Pre-qualification for General Public Audience...................................11
Site Selection Process...............................................................................13
Comprehensibility Rating Process ...............................................................15

Evaluation of Website Comprehensibility........................................................15
Site Selection..........................................................................................15
Review Process .......................................................................................15
Review Scaffolding...................................................................................16
Review Duration......................................................................................16
Rating Duration.......................................................................................16
Training Period.........................................................................................16
Comprehensibility Survey Questions.............................................................17

Site Classification.....................................................................................17
Reading Speed and Linking.......................................................................17
Prior Topic or Site Knowledge....................................................................17
Concept Recall.........................................................................................18
Overall Informational or Instructional Rating...............................................18
Detailed Rating Method.............................................................................18
Information Content Rating....................................................................18
Internet Construction Rating...................................................................19
Post Evaluation........................................................................................20
Processing Notes.....................................................................................20
Rating Matrix.............................................................................................21
Ray Garcia, June 2006 Confidential, University of Arizona 1

Comprehensibility Model Implications............................................................22
Appendix...................................................................................................26
Science Taxonomies....................................................................................26
Science Content Standards..........................................................................26
Science as Inquiry....................................................................................26
Physical Science......................................................................................26
Life Science.............................................................................................26
Earth and Space Science...........................................................................26
Science and Technology............................................................................26
Science in Personal and Social Perspectives.................................................26
History and Nature of Science....................................................................26
Science Center Exhibits...............................................................................27

Bridge of Knowledge................................................................................27
Vivarium.................................................................................................27
24-Hour Observatory................................................................................27
Earthworks..............................................................................................27
Earth and Planetary Materials....................................................................27
Reality Science Exchange..........................................................................27
Heath and Wellness..................................................................................27
Agriculture and Life Sciences.....................................................................27
Library of Congress Science Classification......................................................28

Cybernetics..........................................................................................28
Mathematics.........................................................................................28
Astronomy...........................................................................................28
Physics................................................................................................28
Chemistry............................................................................................28
Geology ..............................................................................................28
Natural history......................................................................................28
Botany ................................................................................................28
Zoology ..............................................................................................28
Human anatomy...................................................................................29

Physiology ...........................................................................................29
Microbiology ........................................................................................29

Comprehensibility Project
The University Of Arizona Science Center is conducting a research project with the
UofA Eller School of Management MIS Department to determine whether an
automated process can be used to source and evaluate web site pages for adequate
usage for learning purposes. The learning purposes are focused on self-directed free
choice learners who would use the web as a resource for finding information to
satisfy a learning objective. The automated process will attempt to simulate the
selection process for learning content that has a higher degree of potential
instructional value to the learner due to its ease of access to the main concept of the
information and ease of comprehensibility. The availability of enormous variety of
information on the web presents a challenge to the average person in selecting the
web pages that best suit their learning objectives, skills, and comprehension level.
The research involves the participation of four people who will apply their judgment
in the review and evaluation of web site pages. The evaluation criteria will be to
judge the web pages for comprehensibility of the hypertext information space
presented in the web page. We provide a description of the comprehensibility
criteria to the evaluators prior to conducting the review.
The review process consist of accessing the web site page, finding and reading the
central concept, linking to related pages as necessary to understand the central
concept, and evaluating the web site pages for adequacy for learning or instructional
purposes. The evaluation uses a rating system to rank the web pages relative to the
criteria and relative to other pages reviewed. Approximately eight hundred web site
pages will be reviewed with an average of 10 minutes allocated per web site page.
We are interested in the scanning for the web page site similar to what is typically
done when browsing the web site when seeking information.
The focus of the website is within the Science, Technology, Engineering, and Math,
(STEM) Domains. The selection of web site pages is based on a typical search
strategy using keywords. This requires the evaluators to have a general interest in
STEM but without any particular expertise in the area. The evaluators would have
the following general profile:
• Four people with similar professional profile.
• Internet and Media Literate with average daily internet Web Browser usage of
1 - 2 hours or more.
• Information literate so that they can discriminate between questionable

information sources and reliable sources.
• Adept at selecting internet content for learning activities either for themselves
or for others to use. They should have selected, evaluated, and

recommended website/pages for others such as the public, parents, friends,
children, peers, etc. that they thought were appropriate.
• Active self-directed free choice learners. They have learned something not
taught in school within the last year. They have read a non-fiction book,
visited a museum, watched a documentary, or participated in any free choice
learning activity where they can describe what they learned from the
experience or activity.
• Some interest in STEM without any particular expertise in any given area.
• They should have internal criteria for what makes a web site suited for
learning that they can express.
We have professional librarians who are not researchers and interface with the
public to select a broad range of appropriate web sites and pages for people to use to
learn about a topic.
The evaluation project was scheduled for July 2006 through August 2006.
The data generated from the project is used to understand what characteristics of a
web page may be more suited for learning insofar as ease of comprehensibility than
other web pages given the same level of interest and motivation in the learning
activity. It may require that we interview the evaluators after the review process to
understand why certain pages where rated higher than others. The specific ratings
provided by each evaluator will not be shared other that to the person providing the
data.
The results will permit the research to more accurately simulate the human
judgment in the selection process and rate pages for comprehensibility more
effectively. A computational model will be created to simulate the human judgment.
A research paper will be published describing the model and how it was created.
Summary statistics from the rating process and a description of how the data was
generated will be included in the research paper.
i ii
Defining Comprehensibility
The main interest in comprehensibility is the extraction of the instructional/learning
value of the information content in the web page and the related linked web pages.
The instructional value is a measure of the knowledge contained within the page,
including the degree of application, analysis, synthesis, inquiry or any other narrative
that provokes learning activity. If the information primarily motivates topic interest
in an entertaining manner this is of equal learning value.
Comprehensibility is the degree to which a web page provides direct access to the
substance of the information in the hypertext space without distractions. A
hypertext space is the web page being viewed plus all related linked pages which are
necessary for the reader to visit to understand the information within the related web

pages. A web page with high comprehensibility will focus on the central concept of
the text narrative and would only use visuals when they are more appropriate for
explaining a concept or audio when it is not spoken language and is relevant to the
concept. A highly comprehensible web page will limit the number of concepts
explained in the text to what can be easily scanned within two page scrolls with each
paragraph building on the concept. The use of the hypertext links are limited to
related concepts that are necessary to understanding the hypertext narrative where
clicking on the link the reader maintains a meaningful context and only optionally is
required to return to the previous page. The reader should easily be able to
determine where they are in the hypertext space relative to where they started.
Comprehensibility encompasses ease of finding and understanding of the concepts

presented assuming that the reading level of the text is equal or less than the
reading level of the evaluator. It does not require that the reader understand all the
text in absolute terms but is able to find meaning and make sense of the text relative
to their framework of understanding or mental map of the topic. If the reader
believes they understood some of the concepts presented this is adequate to
determine the degree of comprehensibility.
Comprehensibility would focus on the main topic with a narrative treatment that is
suited for quick scanning of the web page and uses summary presentations to isolate
the main ideas. The summary may be presented as, the use of short paragraphs
that are easy to scan, use of bullet points, tables, or other presentation that allows
for quick scanning of the information to find the central ideas.
Comprehensibility is evidenced by web page narrative that is suited for short term
memory that allows for ease of linking through and selectively discovering the
meaning relevant to the reader. It is web page narrative that combines both the text
on a single page and the associated pages navigated through links while still
maintaining the context of the related topics.
Comprehensibility of the hypertext space would allow the reader to know how they
navigated through the space and return to any previous page to traverse a different
path. Comprehensibility is enhanced by a site search and site map when a site trail
is not available. Findability of information adds to the comprehensibility.
Comprehensibility is enhanced by consistency and uniformity of the presentation of

the pages on the site. The use of graphical elements only adds to comprehensibility
when it shows information that is more easily presented by the graphical element.
The arbitrary use of colors, fonts, background, images distracts from the ease of
reading the text.
The usability and accessibility of the website contribute to comprehensibility. If a

site is not easy to use or cannot be adjusted for accessibility then its
comprehensibility is diminished.

Comprehensibility puts the readers attention on the most relevant information
quickly and build confidence in the reader that the information has utility for learning
to a degree that the reader will be satisfied that the time spent with the text as
having some value for learning.
Comprehensibility also includes the sense of credibility and trust and access to the
source, author and date of the information being presented. Without trust the
reader is unlikely to try and understand the concept or question it when found.
The opposite of comprehensibility is clutter. Clutter is the presentation of more

than a few related concepts on a single page or the partitioning of the page into
different sections. This is evidenced by Advertising, multiple topics on the same
page, and site navigation links, in areas that present a distraction. Clutter draws the
reader attention away from the central concept of the page or makes it difficult to
find the main idea. Clutter is when the noise on the page is more than 20% of the
page information content. Clutter is when it is difficult to find information and
understand it. Clutter is common on e-commerce and news sites. Clutter includes
chatter and ramblings typical of blogs. Clutter includes images that do not add
information to the text or is a concise visual of an idea.
Understanding the Evaluators Expertise

We require a brief professional biography from each evaluator to understand the
level of experience in selecting web content for the general public. The evaluators
should mention any activity that is relevant to the selection of websites such as
participation in social tagging, submitting sites to a virtual library or directory or
search engine, editorial or writing experience, web site development experience, or
any other Internet related experience that would influence their judgment in
selecting websites.
The evaluators will be asked to review this document prior to starting the activity.
They may provide feedback and suggestions to have the document more closely
reflect the criteria that they use in evaluating web site as Librarians.
We may ask the evaluator to optionally participate in a determination of their

Internet competency to establish a comparative level of experience. This may be
through a survey or by an interview process.
The evaluator may optionally participate in determining their preferred cognitive

style using the Cognitive Style Index or simply state whether they believe they are
more analytical or intuitive. The purpose of this information is only to understand
how to interpret the judgment of the ratings.
The ratings are done by the evaluators on a best efforts basis. We would prefer
that the evaluators engage in the rating process in sessions no longer than a few
hours per day to avoid fatigue or skewing of the judgments. If at any time confusion

over applying the rating criteria occurs we would like to be informed and given the
opportunity to discuss the issue.
The evaluators are allowed to discuss the project with their peers to expand the
professional judgment being applied but only within the first 25 website evaluations.
We want to avoid a shift in the judgment due to outside influences. The first 25
evaluations are considered the training period to practice applying the rating and to
adjust the process or rating if necessary.
Periodic communications during the evaluation process will provide feedback on the
status. Comments provided for particular websites would be discussed to
understand the rating. We are as interested in why a website is disqualified or rated
poorly as we are in why a website is rated appropriate. The rating statistic will not
be discussed with the evaluators during the project to avoid skewing the judgment.
After all the rating is completed we will analyze the data and provide the summarized
results to the evaluators.
We ask the Evaluators to classify the websites, state any prior knowledge, recall the
concepts presented on the website, provide an overall rating, and provide a specific
rating. This should model the selection process that a Librarian uses.
Defining the General Public Audience

The general public is assumed to have an average reading level of the 8th grade.
Reading competency would be sufficient to understand a newspaper or magazine.
The public is assumed to have access to the Internet either at home, work, library,
or public internet kiosk. The content is to be viewed in an Internet Browser and not
a cell phone or other internet enabled device.
Internet and Information Literacyiii is not assumed other than at a very rudimentary
level. The competency level in discriminating selection is assumed to be at a typical
Media Literacy level, which is relatively poor, often biased, and focused on
entertainment. For this audience the Internet may be falsely assumed to be
accurate and not equated to other media outlets. This is the audience that may be
familiar with Yahoo, MSN, AOL and possibly Google. They would use searching
strategies that are limited to single keywords in the search box and may expect that
the results in the first two pages would be satisfactory and valid. They would be
competent enough to use the Internet to find information and extend that activity to
learn about a topic.
The general public is not assumed to have an email account outside of work, buy
from e-commerce sites like Amazon or E-Bay, or use their personal computers for
other purposes. We do not want to bias the study to the top two million frequent
Internet users that are early adopters, know how to create a web page, or work in
STEM, and use computers as a tool. We believe that this population of users may

also lack appropriate search and information literary skills but they are likely to
overcome any deficiencies through the application of other skills.
Full competency in all the Browser features is not assumed. The typical Internet
user may not be aware of the configuration options for the Browser, parental
controls, history, search options, research, bookmarking, or optional features which
help the user gain control of their browser.
English reading proficiency is presumed for multi-lingual individuals. The websites

selected will be in English and may not have translations or may not be machine
translated without problems. Spanish Language Comprehensibility may be
supported in the future and tools to help English as a Second Language improve
comprehension. These are beyond the scope of this study but we believe that the
findings will be useful for European Languages other than English.
The general public is the audience that is targeted by spamming, phishing, viruses,
chain emails, and other activity that attacks the vulnerable Internet user. These are
the users that may not apply criteria that help them discriminate sources of
information.
Education level or age is not assumed to provide the user with an advantage. Older
professionals who do not need to use the Internet for work may have difficultly in
learning new Internet skills. Younger generations who have a lot of experience with
console games may not be any better at selecting credible and useful content on the
Internet.
Our belief is that the need for Librarians to assist the public in selecting appropriate
sources of information and learning material is even greater with the use of the
Internet across all segments of the population.
STEM Topics
The topics that are included in the evaluation are from Science Technology
Engineering and Mathematics. We have included four taxonomies for consideration,
National Academies Press (NAP), Flaudrau Science Center Exhibit subjects, Library of
Congress, and UNESCO. These taxonomies were chosen for their representative
sampling of STEM topics across a broad range of purposes related to the general
public. NAP publishes books for educators in Science, Engineering, and Medicine.
NAP Science Content Standards is expressed as conceptual phrases for science topics
to cover for each k – 12 grade. Flandrau Science Center taxonomy is related to the
types of exhibits that may be designed for the public. The taxonomies from the
Library of Congress and UNESCO represent a view of how STEM information may be
cataloged and used by librarians or researchers. The taxonomies may be used as
sources of keywords used in searching for content. This mimics the expressed
interest by an individual to a librarian who would use the request to determine the

area to start the search and the associated terms to use. (See appendix for sample
taxonomies.)
Web Site Sources

The sites are selected from Wikipedia, DMOZ.org, and the open web. Wikipedia
and DMOZ were chosen due their wide adoption by many search engines as the
encyclopedia and directory sources. While both of these sources are questionable
they reflect the state of the information available on the web and accessed by the
general public. The Wikipedia repository and DMOZ directory are freely available to
any website to reuse therefore much of the content and links appear on many
websites. The repository and directory are managed by volunteers. The volunteers
are not qualified through any certification process. The volunteers are individuals
who have an interest in the topic. They provide text and selections of the websites
within the topic category, accept or reject website submitted by the public, and
summarize a description of the website. The wikipedia entry or directory listing may
have several volunteers working within a topic but generally there are not more than
three volunteer editors and typically only one. The volunteers may post a short
biography of their expertise or interest in the topic but may choose to remain
anonymous. The encyclopedia repository and directory taxonomy of DMOZ have
grown substantially and has many cross links between portions of the websites
organization. The encyclopedia and directory reflects the many ways of organizing
the topics and may appear disorienting at first due it’s a lack of a consistent audience
perspective and level of competence of the volunteers who created it.
DMOZ represents an organic directory of public websites. It is not as qualified as a

public library directory and therefore would have many more website of questionable
value. The volunteers do not have consistent criteria for deciding which website is
accepted for inclusion in the directory. DMOZ has enough coverage that most
keyword searches will return a variety of choices of web sites. Similarly, Wikipedia is
organically edited by volunteers without specific editorial control. The quality of
entries varies and the authority of the information cannot be verified. Wikipedia and
DMOZ may be considered the lowest common denominator of web content created
and accessed by the general public. DMOZ has over 100,000 links related to Science
and Wikipedia has over a million pages in its encyclopedia. These are large corpus of
documents that should provide adequate variability in the site content and
construction.
We refer to Wikipedia and DMOZ as sources due to the general public selection of
the content and the links. A public search engine would return many links that
include Wikipedia pages and overlap with DMOZ links as well as content that may be
better or worst than these two sources. The public search engines lack of adequate
filtering of the results is therefore used as the third source of web site pages for
evaluation. A keyword search in a public search engine would return the most

popular websites based on link structure, not based on information conceptual
relevancy, which results in a lack of adequate filtering necessary to guide the user to
the most qualified source of information. With expert level searching strategy and
persistence it is possible to find quality web sites but this requires competency in
Internet searching and Information Literary neither of which is common within the
general public.
Our site selection goal is to find web site pages that reflect what the general public
would find plus sites that represent the full spectrum of what is available that is
better than the lower common denominator from Wikipedia and DMOZ. We want to
include Web site pages that would be recommended by a Librarian to the general
public interested in a particular topic. We expect that a Librarian may have expert
searching strategies and knowledge of quality sources as well as qualitative
judgment on information that is well beyond the competency of the general public.
We are interested in included web site page that reflect the level of quality that is
representative of a professional Librarian. The Librarians are asked to include web
sites page that may be of poor quality but easy confuse by the general public as a
valid source as well as web site pages that are of the highest information quality.
We prefer to exclude website that may already be included in a virtual library due
to the pre-determine quality of the website. We excluded any sources that are not
freely accessible from any where on the web. Digital libraries and web sites that
require paid subscription are excluded. We assume that the general public could go
to the local library or use the virtual library for their information and instructional
needs. We are interested in helping the general public when they are not accessing
quality directories.
A sample selection of web sites may be presented to the evaluator three times to
test for consistency in the rating judgment. The web sites will be presented within
the normal process and with time delays between the reviews. For example, if a web
site is evaluated on Monday it may appear again on Wednesday. The justification is
to allow enough time in between the reviews, such that the evaluator may not
remember that they already reviewed the site, or recall the ratings they provided
previously.
Website Pre-qualification for General Public Audience

Each selected web site will be pre-qualified for suitability for rating by reviewing the
sites prior to the rating activity and eliminating any sites that are not for the General
Public Audience seeking informational or instructional content. This will allow the
evaluators to focus on only rating web sites that may be of some informational or
instructional value.
The following types of website would be eliminated from the evaluation due to the
purpose of the site not being solely informational or instructional for the general

public; Audience Specific Content, Entity Information Sites, Connector Sites, Content
Format non-HTML, and Sites with Restrictions. Descriptions for each category
excluded are listed below.
Audience Specific Content:
• Scholarly, Expert, Professional site not suited for general public. These sites
require expert knowledge and vocabulary in the domain.
• Children site or too simplistic to be of general value. Includes any web site that
is highly graphically and specifically designed for children. These sites are likely
to be provided under supervision of a parent or teacher.
Entity Information:
• University, government, or other reputable organization site and was developed

specifically for information, instructional, and educational purposes. These sites
are identifiable with .EDU or .ORG domain.
• Commercial or Organizational site not suited for learning purposes, includes

company, e-commerce, auctions, portals, catalogs, brochures. These sites many
have interesting information related to the products and services they sell but the
primary purpose is to sell therefore the information would be biased.
• Personal or Promotional site, includes myspace or site with a bias. These are of
little informational or instructional value and are obvious to most Internet users.
Connector Sites:
• Directory or mostly of navigational value to orient oneself. Includes virtual or

digital libraries, social tagging sites, or rating sites. These are sites that do not
have content and only link to content on other sites. Connecting to other sources
of information is the purpose of these sites therefore evaluating them for
comprehensibility would not be beneficial.
• Databases or list of items sites that require prior knowledge of the topic to know
what to query.
Content Format non-HTML:
• Electronic distribution of printed documents and not designed for reading on the
web. (PDF, MS Word, or Flash Paper documents) (Other than scholarly papers).
The evaluation of this type of digital content should use method suited to printed
material.
Site Restrictions:
• Registration/login or restricted access or subscriber site. These sites would be

technically difficult to determine since the content is not accessible. The user

would need to use good sense criteria to evaluate the site owner’s reputation and
the content contained within the web site.
• Collaboration or News site used for communications such as groups, blogs,

forums, polls, open wiki, image galleries, email. These sources are available and
known to many Internet users and would not generally be confused with other
sources of information.
• Highly Interactive sites without a clear instructional value, includes flash

introduction pages. These sites are difficult to process and tend to focus on
entertainment media.
• Inappropriate web sites, includes all sites that would be blocked from further
review due to safe filtering. These included gaming, pornographic, and other
offensive content.
• Unknown due to site error, broken links, slow loading, installation of software or
other problem that hinders access to the information. Any sites that essentially
do not function are eliminated.
Site Selection Process

The web sites are selected by the research team and the librarians using the
Science Content Taxonomies provided. This is to allow for variety in the site
selections across, prior knowledge, interest, preferences, search strategies, and
sources. The four members of the research team will provide 200 sites. The four
Librarians will provide 600 sites each. This will total 800 sites for evaluation by the
Librarians. Only 25% of the sites being evaluated by any given Librarian would have
been submitted by that Librarian.
Each person may use their own search strategy for the highest quality web sites.
To include variety the researchers may follow a set sequence with the intention of
including both quality and poor site examples for the Evaluators. The search process
is as follows:
1. Go through each second level topic the taxonomy and pick a topic to use as
the keyword and associated concepts in the search.
2. Start with Wikipedia and find any pages that directly relate to the concepts.
3. Navigate through the internal and external links from Wikipedia. (It is
possible for the subject not to be represented in Wikipedia in which case just
go to the next step.)
4. As topically relevant sites are found submit them for rating. Include poor
and quality web sites as they are found.
5. Using DMOZ search for sites related to the topic.

6. Navigate through the DMOZ directory to find topic relevant sites.
7. As sites are reviewed navigate through any links on the page to find topic
relevant pages
8. Using Google, Gigablast.com, Ask.com or another search engine, with only

the keyword find pages.
9. Use advance searching to find pages and include terms like tutorial, learn
and exclude courses, lesson plans, or other terms as necessary
10. Go to any other known sources like Nasa, AMNH, National Geographic
11. Submit sites as they are found. Include variety in the selection with poor
and quality sites represented. Go to the next topic in the taxonomy.
We do not expect either the researchers or the Librarians to spend much time
searching for web sites for each term. The intention is to make a fast determination
of the website and not the rate them during the selection process. We want a
superficial determination of whether the website should be included for rating based
on the classification of the website previously described.
To store the web sites that are found we are using FURL from Furl.net. This is a
small utility that works with most internet browsers to help bookmark websites of
interest. The reason we selected Furl was due to its export feature which will permit
the researchers to load the data into our rating process. Each researcher should
install and test Furl before the site selection process begins.
All the web site pages will be selected prior to starting the rating process. This is to
allow for adequate representation of the topics and quality in the sites selected. The
site list will be reviewed periodically by the researchers in summary prior to the
evaluation to make recommendations to the Librarians in their choices. We want to
avoid a skewing of data based on any preferences or biases or prior knowledge in a
particular topic.
It is possible that not all topics or levels of quality are represented in the open web.
Web content is often developed to satisfy popular topics therefore science topics that
are of special interest may not be adequately represented.
It may be a common practice for Librarians to use multiple sources for information
to cross check facts as a technique to validate the information source. All the
sources used in this technique should be included for evaluation in the selection
process.

iv v vi
Comprehensibility Rating Process
The evaluator will login to the project website and select the URL presented. They
will review the website and evaluate it using the criteria described above. Once they
have read/scan/skim the website to a degree that they are confident in their ability
to make a judgment the evaluator will exit the site and complete a survey form. The
survey will ask some background questions, require that the website be categorized,
and rated for quality. The process is repeated with the next URL.
It is possible for the process to be paused at any point and restarted at a later
time. Once a site is selected the evaluator should complete the survey as a
continuous activity without distraction. The start and end time of the website review
is tracked by the application. The total number of pages visited within the site is
monitored.
Site Selection
The sites are pre-selected and presented in a particular order to each evaluator. No
description of the site is provided prior to it being presented. This removes the web
searching activity and the contextual evaluation of search results. Reviewing the site
in this manner is different from normal behavior since it is not typical for one to be
presented with a web page without some indication of what the web site is about.
The rational for this is to force a complete evaluation of the web site on its own
merits. The evaluator will need to scan the website to find the relevant information
and navigate through the website until they are confident that they understand what
is available and can rate the website content and construction adequately.
vii
Review Process
The review process should consider the rating questions but only after the web site
is read/scan/skimmed. It is permissible to scan or skim the contents and not read
the contents if the web site is designed for this type of reading. The evaluator
should use the same process they would use if they had a person ask them to find a
web site on the topic, they found the site through some search process, and they
need to make a quick determination as to the appropriateness of the web site to the
person’s goal. The assumption is that the topic is under represented and the website
is not available in a Virtual Public Library. The person may have a goal to either find
information about a topic or have a specific learning objective. The evaluator may
assume that the website being recommended is not the only source they would
provide.

Review Scaffolding
It is acceptable for the evaluator to use reference sites or dictionaries to
supplement their understanding of the topic. The assumption is that the evaluator is
learning about the Science topics and may need some assistance in specific
vocabulary used within the Science Domain. This activity should only be necessary
for a few words on the website. If the website has links to help understand specific
concepts this should be positive if the Evaluator needs to find other sources to help
understand the content then this should be accounted for in the rating. If sites are
found that are helpful in the scaffolding process the site should be included for
review.
Review Duration
The evaluation should remain within an average of 10 minutes for each site with
not more than 20 minutes for exceptional sites. If a site is commanding too much
time to comprehend this should be indicated in the rating. It is not necessary to
struggle with poor websites. For websites that are exceptionally interesting it also
not necessary to learn about the topic beyond understanding that the information is
of useful for learning. The websites that appear useful but are unusual or unfamiliar
are the websites that warrant time to understand how to rate them accurately.
Rating Duration
The rating should be done quickly under 5 minutes once the evaluator is familiar
with the rating questions. We are interested in the fast judgment applied in a real
setting where the evaluator is making a recommendation to an individual who is
waiting. This is similar to the behavior that one may have for their own searching
where quick judgments are made based on limited information in the search results
and a review of the first web page seen of the website. The difference is in the
expert judgment being applied using consistent criteria with a focus on adequacy and
appropriateness of the information for learning. The assumption is that web sites
and pages that are comprehensible are easy to read and understand and can be
evaluated quickly.
Training Period
The site selection process will have a training period the validate that content exist
for the range of topics in STEM. The training is also to help the Librarians refine an
effective search strategy and filtering of web sites. The research may review the
initial 25 web sites selected by each Librarian to help establish adequate topic and
quality variety in the selection.
The librarian evaluator will rate 25 to 50 web sites as a trial period to understand
the survey process and level set the criteria. They will pause to allow the

researchers to review the survey data and make any adjustments to the process.
Depending on the consistency of the rating the data may be discarded.
Comprehensibility Survey Questions
Site Classification
The web site is to be classified before rating. Websites that are not suited to the
general public audience as defined in this document are not to be rated. This step in
the process only validates that the web site belongs in the evaluation data.
Options: dropdown, default is General Public Audience Informational or Instructional Site,

Audience Specific Content, Entity Information Sites, Connector Sites, Content Format non-
HTML, and Sites with Restrictions
Reading Speed and Linking

We are interested in how the site is reviewed. Web sites may be skimmed to
understand the structure, scanned to find specific sections, read to find the main
concept, read for understanding, read for curiosity. Each reading technique is
subject to degrees of comprehensibility and therefore would impact the judgment
during the evaluation. We would like the Librarians to review the web sites on its
own merit and when the web site invokes deeper review to pursue their interest in
the topic within the short time period allotted for each web site. This forces
judgment about what is necessary to review for each web site to make an
evaluation.
The web is not a linear medium therefore clicking on links is common and
necessary to understand the information. The degree to which one links through an
web site page and retraces ones paths is an indicator of the information hypertext
space. The clicking through of links is considered part of the reading process for web
site pages.
Options: dropdown, skimmed to understand the structure, scanned to find specific sections,
read to find the main concept, read for understanding, read for interest
Options: dropdown, did not link through, active browsing of the website
Prior Topic or Site Knowledge

Do you have prior knowledge or interest in the topic of the website with well formed
notions or opinions? Prior knowledge does not include coursework in formal
education or general curiosity in the topic. Stable interest with an intrinsic
motivation to learn the topic where the person actively pursued a learning goal is
what we are interested in identifying.
Options: radio button, True or False; Default: False

Concept Recall
After reviewing the website read the description and website title and think about
whether this adequately capture the concepts you remember reading. The title and
description should be an adequate representation of what the website or page is
about.
Does the site description and directory category not match the site contents?
Options: radio button, True or False; Default: False
Overall Informational or Instructional Rating

General impression of the website based on what was learned and the presentation
facilitating the finding and reading of information and its suitability for the general
public. Is the information easy to understand, useful, and credible? This is a
judgment using intuition based on the content and appearance of the website.
Would the evaluator recommend this site as a source of informational or instructional
content to the public?
Options: radio button
1) 1 Poor, 2 Low, 3 Avg, 4 High, 5 Best
Detailed Rating Method

The rating method uses a five point scale with each point contributing to the value
of the overall rating. The scale is from one to five with one indicating that the
specific criteria do not add value and five indicating that it adds maximum value.
These are relative values based on judgment across the sites reviewed.
The ratings are normalized for the evaluator across all the web sites. The
normalized rating is adjusted using the evaluators weighted criteria then compared
to the other evaluators.
Options: radio button
1 Poor, 2 Low, 3 Avg, 4 High, 5 Best
Information Content Rating
Info r mat io n Va lue ( te xt co nte n t)
Is it easy to find and understand the main topic and concepts in the text? Is it
readable within the comprehension level of the general public? Are the web pages
informative?
Does the website adequately cover the topic? Does it have sufficient useful
information? Is the background and context for the topic provided?

Is the audience clear? Is it relevant to the audience? Is the information scope
adequate? Do the links help understanding the information? Are terms defined?
Info r mat io n C r e dib ili ty (cre d ibi li ty)
Does the content build confidence in its validity (expertise) and can be verified
(trustworthiness)? Does the content appear to have a editorial control?
Is the Authority of information known? Are the sources credible? Are the sources
listed and obtainable? Is the site owner available? Is the author expertise and
credentials listed? Can they be contacted? Is there a discussion of the topic or
opinions of the quality or a rating shown by peer review? Is there accountability for
the information?
Is the information objective, valid, and accurate? Is the information current? Is

the information free of biases or are the biases easy to determine?
Are there any broken links? Is this a small website that looks amateurish? Does it
link to other websites where the information does not look credible? Are their
typographical errors? Is it hosted by a 3rd party such as Geocities, Yahoo, or AOL?
Me dia Instr uc tio na l Va lue (v isual de sign clar i ty)
Does the use of graphics, icons, animation, or audio enhance the clarity of the
information? Is it necessary to communicate the concepts?
Is the site free of visual clutter? Is the site free of advertisements and unrelated
information? The site does not include popup ads or techniques that distract the
reader. Are the advertisements placed away from the information so that they are
easy to ignore?
Internet Construction Rating
Affe ct ive At te ntio n (o ver al l appe al)
Does the website make it easy to focus attention on the most relevant information?
Does the website look professionally designed?
Are the fonts, sizes, colors, background and sectioning of the site easy to read? Is
the content designed to be printed or have an option to have a printed version of the
same content?
Do the format, appearance, and aesthetic of the website enhance the information?
Is the web site organization uniform, consistent, and familiar? Is the page heading
and summary representative of the information on the web site?

O r ganiz at io n (s tr uctur e , naviga tio n)
Is the content presented for reading on the web? Does it make appropriate use of
short paragraphs, list, tables, headings, and links, to focus easily on the main ideas?
Is it easy to scan or skim?
Are the links related to the topic and make it easier to understand the information?
Are the links clearly organized? Are the navigational links separated from the links
within the narrative? Are the links to external sites relevant to the topic and clearly
indicated? Are links to media files indicated?
Usab il ity (fun ctio na li ty, inte r act ivi ty)
Do the Web site pages load quickly? Is the site easy to use? Does it have a search
feature and site map or menu that is organized? Is it easy to find the home page?
Does it provide an indicator of where the web page is relative to the site navigation?
Is the back button feature operable?
Does the site adequately support usability and accessibility?
If the website is highly interactive does the activity enhance the informational or
instructional value of the website? Is the interactivity necessary or the easiest
means of communicating the concept?
Post Evaluation
Comment on the evaluation process if necessary to explain an exceptional rating
and characteristic that was not included in the rating questions but was an important
factor in the evaluator criteria.
A calculation of the rating may be done and compared to the initial general
evaluation. If an inconsistency is found a re-evaluation may be done to correct for
the judgment. If some factor accounted for the discrepancy then a comment would
need to be provided to explain the difference.
Processing Notes
The survey time should be less than the time used to review the website. The
evaluation survey facilitates the rating process by displaying criteria questions the
evaluator should think about before providing the rating.
The evaluator is assumed to have an adequate comprehension level to review

general web pages on Science and Technology, and enough interest in the subject
matter to be motivated to find the central concept and attempt to understand it.

Rating Matrix
The rating matrix established the internal criteria of each factor considered in the
overall rating. It indicates when a factor is dependent on others or if it may be
considered independently. The relative value of each factor is proposed. This matrix
is used to weight the ratings against each other to emphasize one factor more than
another.
Each rating indicates the following:
D = Dependent
I= Independent
L = Familiarity may overcome hurdle, can learn to ignore factor
C = Constant Factor no matter how often it is reviewed
< > = Relative importance between the factors
IV IC MV AA SO UA
Information Value IV IC < > > < >
Information Credibility IC IC > > > >
Media Instructional Value MV IC > < <
Affective Attention AA D DL < <
Organization SO D D DC >
Usability UA D DL
Relative rating priority 5 6 2 1 4 3
The rating matrix is only applicable to the subject of this study. Websites that have
a purpose other than informational or instructional, such as web email or
promotional site would be a completely different prioritization.

Comprehensibility Model Implications

The comprehensibility model will need to be computed therefore the rating method
should provide sufficient conceptual equivalence to the model such that a heuristic
can be developed for each constituent part of the rating.
Website Pre-Qualification for General Audience is used to eliminate websites

from the evaluation that are not suited for self-directed learning. These website may
have value for informational purposes but are either likely to be easily filtered by the
learner visually or through an automated process. A secondary process may attempt
to automatically classify these types of websites in the future. The use of Wikipedia
and DMOZ as sources should provide an adequate topic filter and a balanced
selection of websites within a topic category. The classification data generation from
this project may be used to train an automated classifier based on the web site
structural patterns.
Prior Topic or Site Knowledge is requested to address a biased or preconceived

notion of the topic or website. The process may resubmit the same site multiple
times to test the consistency of judgment from the evaluators. If they recall the site
this may bias the subsequent evaluation. The model may determine the prior
knowledge of the learner through an indication that the website information is either
too easy or difficult for the learner. Using a readability index we may be able to
match the content to the learner within a reading range for the specific topic. The
readability index has the context of the topic and any jargon, nomenclature, or
vocabulary needed to understand the information.
Concept Recall test prior knowledge and understanding. Words that best describe
the concepts uniquely should be selected. General terms would indicate surface
understanding. Specific terms may indicate new knowledge if the topic was
unfamiliar to the evaluator. Since the sites are selected from Wikipedia and DMOZ
which use volunteers to manage the website we have no assurance of consistency or
accuracy of the site descriptions. If the site is not adequately described it may
correlate to a poorly selected site.
Overall Informational or Instructional Rating provides the first test of

judgment. It should correlate to the rating in the specific criteria. The degree of
matching signals that the evaluator has well formed internal criteria that they are
applying. A mismatch may signal a problem with the criteria of lack of consistency in
the evaluator.
The next process in the rating applies more specific criteria with probing questions
for each rating to help the evaluator apply consistent criteria and analyze their
judgments. Two rating criteria groups are used, Information Content Rating and
Internet Construction Rating. The Content Rating is focused solely on the
information apart from the presentation. The Construction Rating considered the

impact of web design on the information delivery. The ratings will apply a numeric
value to indicate a relative value of the website for the criteria versus other websites.
The rating could be weighted to emphasize one rating as more important than
another in the determination of comprehensibility.
Information Content Rating includes Information Value, Information Quality, and

Media Instructional Value. These are aggregated rating judgment with criteria
questions asked for each.
Information Value captures a rating for comprehensibility most directly, assuming

all other criteria are equal. This would be computed as an information
summarization or extraction function that determines the degree of meaningful
information is included in the text relative to the total text. It may include a
readability index and a classification against the bloom taxonomy. Word
concordances, vocabulary matches, and parts of speech grammatical analysis may
be included. The technical challenge is to determine the semantic text units of the
web page and the strength of association of the links from the web page.
Information Credibility determines the authority of the information. This would

be computed in the future through an eduction process for web page indicators once
the data is collected to use for training. Simple correlation may exist such as a
contact email or about page or access to a home page or author name and date.
Other text or graphical clues may signal a suspicious page. A whois lookup would
provide information on the website domain owner. The focus is to establish when
suspicious signals indicate loss of confidence in the source.
Media Instructional Value will be implemented as patterns within the objects

that may signal that the object is of instructional value. It is determined by analysis
of the web page characteristics. This may be the existence of meta tags within the
object or elements within the web page. A filter to eliminate advertisements and
interface graphics is needed to remove noise from the page. Advertisements will
discount the comprehensibility since they are fundamentally intended to distract the
readers’ attention from the main content on the web page. The anchor labels for
images may be indicators. Standard banner sizes and small graphics could be
filtered leaving only media of unusual size. Excessive Media within a clear purpose is
treated as noise on the web page. A determination of the space that the media file
occupies on the web page is an indicator of the prominence of the media file. This
value is the most difficult to compute since it is limited by the lack of information
provided for media files. We will focus on text processing as the most prevalent form
of communications on the web.
Internet Construction Rating is a grouping for Affective Attention, Organization, and

Usability.
Affective Attention is a rating on the effectiveness of the design to bring attention

to the highest value information content. The web page characteristics are likely to

impact this computation with certain patterns being more effective than others. This
is a subjective measure therefore difficult to compute. We may focus on page
characteristics that signal poor design choices that negatively impact the rating. This
is likely to be more feasible.
Organization is concerned with the hypertext linking structure and its usefulness
to enhancing the information of the webpage. The linking structure will be analyzed
to provide relative values to certain link patterns. These include links within the
page, links within the same site domain, and links to an external site. It may include
inbound link to the page from within the same website.
Usability and accessibility is based on the well established standards. This may be
computed as the function of lack of conformance to the standards. It is likely that
the computation will be more accurate that the human judgment for this particular
rating since to determine usability requires extensive testing which is unlikely to
occur under the normal course of the evaluation.
Post Evaluation is a free text comment provided by the evaluator to indicate

exceptional findings in the websites or omissions in the filter and rating criteria.
The calculated rating, based on what is provided by the evaluators, would be

compared to the overall rating they provided to determine how close they are able to
intuitively guess the rating prior to doing a more complete analysis. This is used to
cross check the evaluation process and allow the evaluators to improve their
consistency in applying the criteria or to expose problems with the process early.

The following chart summarizes the model implementation. The technique is either
based on a heuristic that is created as part of this study or based on pattern
matching using the data generated in the evaluation. The baseline refers to the how
the value is determined relative to a starting value. The value baseline either
assumes that it is either low or high and based on the characteristics either increases
or decreases the value.
Comprehensibility Model Implementation
Technique Baseline Priority
Information Value IV Heuristic Low Primary
Information Credibility IC Pattern Low Secondary
Media Instructional Value MV Heuristic Low Secondary
+Pattern
Affective Attention AA Pattern Low Secondary
Organization SO Heuristic Low Primary
Usability UA Heuristic High Primary

Appendix
Science Taxonomies
The following are four taxonomies of science content to use as a source for keywords and organization of
web sites and pages.
1. Science Content Standards - http://fermat.nap.edu/html/nses/6e.html#si

2. RAA document for Flandrau Science Center on exhibit design proposal
3. US Library of Congress - http://www.loc.gov/catdir/cpso/lcco/lcco.html
4. UNESCO - http://www2.ulcc.ac.uk/unesco/thesaurus.htm
Science Content Matter, energy, and organization in living

systems
Standards Behavior of organisms
Source: Earth and Space Science

http://fermat.nap.edu/html/nses/6e.html#si
Properties of earth materials
Science as Inquiry Objects in the sky
Changes in the earth and sky
Abilities necessary to do scientific inquiry Structure of the earth system
Understanding about scientific inquiry Earth’s history
Earth in the solar system
Physical Science Energy in the earth system
Geochemical cycles
Structure of Atoms Origin and evolution of the earth system
Light, heat, electricity, and magnetism Origin and evolution of the universe
Structure and properties of matter
Chemical reactions
Science and Technology
Motions and forces
Transfer of energy Abilities of the technological design
Conservation of energy and increase in Understandings about science and technology
disorder
Interactions of energy and matter Science in Personal and
Life Science Social Perspectives
The characteristics or organisms Personal and community health
Life cycles of organisms Populations, resources, and the environments
Organisms and environments Population growth
Structure and function of living systems Natural Resources
The cell Environmental quality
Reproduction and heredity Natural and human-induced hazards
Molecular basis of heredity Science and technology in local, national, and
Regulation and behavior global challenges
Biological Evolution
Populations and ecosystems History and Nature of
Interdependence of organisms
Diversity and adaptations of organisms
Science
Science as a human endeavor
Nature of scientific knowledge NanoTechnology
Historical perspectives Volcanology
Mathematics
Reality Science Exchange

Communications
Science Center Exhibits Ethics
Source; RAA document for Flandrau on exhibit Journalism
design proposal Technology
Current Events
Bridge of Knowledge
Heath and Wellness
Learning how to Learn
Information Literacy Physiology and Anatomy
Internet and Media Literacy Brain Science
Public Health
Geography Integrative and Alternative Medicine
Cartography Nutrition
Transportation History and Planning Medicine
Civil and Environmental Engineering Medical Communications
Urban Design Medical Ethics
Architecture Disease
Physical Sciences Aging
Mathematics Fitness
Hydrologic Engineering Human Genetics
Vivarium Agriculture and Life

Arts Sciences
Insects (Ants, Beetles, Butterflies, Moths)
Farming
Botany
Food Supply
Ecology
Animal Husbandry
Genetics
24-Hour Observatory Hydroponics / Aeroponics
Integrative Pest Management
Astronomy
Recycling
Spectroscopy
Biological and Environmental Engineering
Optical Sciences
Statistics/Economics
Physics
Horticulture
Astro-Archaeology
Landscape Architecture
Entomology
Earthworks
Geosciences
Ecology, Forestry
Hydrology
Oceanography
Climatology
Earth and Planetary

Materials
Mineral Sciences
Material Sciences
Speleology
Library of Congress Nuclear and particle physics, Atomic energy,
Radioactivity, Elementary particle physics,
Science Classification Radioactivity and radioactive substances
Geophysics, Cosmic physics
The source for this taxonomy is the Library of Geomagnetism
Congress. Meteorology, Climatology, Including the
http://www.loc.gov/catdir/cpso/lcco/lcco.html earth’s atmosphere, Meteorological optics,
Climatology and weather, Weather forecasting
The following is extracted from the Science
portion. The Technology, Agriculture, and Chemistry
Medicine taxonomies may be used as well.
Analytical chemistry
Cybernetics Inorganic chemistry
Organic chemistry, Biochemistry
Information Theory Physical and theoretical chemistry, Radiation
chemistry, Photochemistry
Mathematics Crystallography
Tables
Geology
Instruments and machines, Calculating
machines Mineralogy
Electronic computers, Computer science, Petrology
Computer software Dynamic and structural geology, Volcanoes and
Elementary mathematics, Arithmetic earthquakes, Structural geology
Algebra Stratigraphy
Probabilities, Mathematical statistics Paleontology
Analysis Paleozoology
Geometry, Trigonometry, Topology Paleobotany
Analytic mechanics
Natural history
Astronomy
Nature conservation, geographical distribution
Practical and spherical astronomy Microscopy
Geodesy Biology (General)
Theoretical astronomy and celestial mechanics Evolution
Astrogeology Genetics
Astrophysics Reproduction
Non-optical methods of astronomy Life
Descriptive astronomy, Solar system, Stars Ecology
Cosmogony, Cosmology Cytology
Economic biology
Physics
Botany
Weights and measures
Descriptive and experimental mechanics Spermatophyta, Phanerogams, Gymnosperms,
Atomic physics, Constitution and properties of Angiosperms
matter, Including molecular physics, relativity, Cryptogams
quantum theory, and solid state physics Plant anatomy
Acoustics, Sound Plant physiology
Heat, Thermodynamics Plant ecology
Optics, Light, Spectroscopy
Radiation physics (General) Zoology
Electricity, magnetism, Electromagnetic theory,
Radio waves (Theory), Electric discharge, Invertebrates, Insects
Plasma physics, Ionized gases Chordates, Vertebrates
Fishes
Reptiles and amphibians
Birds Physiology
Mammals
Animal behavior Neurophysiology and neuropsychology
Morphology Animal biochemistry
Anatomy Experimental pharmacology
Embryology
Microbiology
Human anatomy Bacteria
Brain Science Cyanobacteria
Regional anatomy Microbial ecology
Human and comparative histology Microorganisms in the animal body
Human embryology Immunology
Virology
References
i
“Towards a Hypertext Reading Comprehension Model”, A. Protopsaltis, V. Bouki, Department of
Computer Science University of Westminster, SIGDOC ‘05
ii
“Reading Strategies and Hypertext Comprehension”, L. Salmeron, University of Granada, Spain,
iii
American Library Association, Information and Internet Literacy
iv
“Evaluating the Content of Web Sites”, EETAP, Ohio State University Extension, Nov. 1999, www-
comdev.ag.ohio-state.edu/eetap/index.htm
v
“Teaching Media Literacy in the Age of the Internet”, Classroom Connect, Jan. 1999,
http://www.capecod.net/schrockguide/
vi
“What Makes Web Sites Credible?”, Persuasive Technology Lab, Stanford University, CHI April
2001
vii
“How Much Do We Understand When Skim Reading”, G. Duggan, S. Payne, University of
Manchester, UK, CHI 2006

Defining Website Comprehensibility From The Evaluators Perspective

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Defining Website Comprehensibility From The Evaluators Perspective

Uploaded by

Copyright:

Available Formats

Evaluation of Website Comprehensibility

Evaluation of Website Comprehensibility..........................................................1

Site Selection Process...............................................................................13

Comprehensibility Rating Process ...............................................................15

Review Process .......................................................................................15

Comprehensibility Survey Questions.............................................................17

Reading Speed and Linking.......................................................................17

Prior Topic or Site Knowledge....................................................................17

Overall Informational or Instructional Rating...............................................18

Detailed Rating Method.............................................................................18

Information Content Rating....................................................................18

Internet Construction Rating...................................................................19

Ray Garcia, June 2006 Confidential, University of Arizona 1

Earth and Space Science...........................................................................26

Science and Technology............................................................................26

Science in Personal and Social Perspectives.................................................26

History and Nature of Science....................................................................26

Science Center Exhibits...............................................................................27

Earth and Planetary Materials....................................................................27

Reality Science Exchange..........................................................................27

Heath and Wellness..................................................................................27

Agriculture and Life Sciences.....................................................................27

Library of Congress Science Classification......................................................28

Ray Garcia, June 2006 Confidential, University of Arizona 2

Ray Garcia, June 2006 Confidential, University of Arizona 3

• Four people with similar professional profile.

• Information literate so that they can discriminate between questionable

Ray Garcia, June 2006 Confidential, University of Arizona 4

Ray Garcia, June 2006 Confidential, University of Arizona 5

Comprehensibility encompasses ease of finding and understanding of the concepts

Comprehensibility is enhanced by consistency and uniformity of the presentation of

The usability and accessibility of the website contribute to comprehensibility. If a

Ray Garcia, June 2006 Confidential, University of Arizona 6

The opposite of comprehensibility is clutter. Clutter is the presentation of more

Understanding the Evaluators Expertise

We may ask the evaluator to optionally participate in a determination of their

The evaluator may optionally participate in determining their preferred cognitive

Ray Garcia, June 2006 Confidential, University of Arizona 7

Defining the General Public Audience

Ray Garcia, June 2006 Confidential, University of Arizona 8

English reading proficiency is presumed for multi-lingual individuals. The websites

Ray Garcia, June 2006 Confidential, University of Arizona 9

Web Site Sources

DMOZ represents an organic directory of public websites. It is not as qualified as a

Ray Garcia, June 2006 Confidential, University of Arizona 10

Website Pre-qualification for General Public Audience

Ray Garcia, June 2006 Confidential, University of Arizona 11

Audience Specific Content:

• University, government, or other reputable organization site and was developed

• Commercial or Organizational site not suited for learning purposes, includes

• Directory or mostly of navigational value to orient oneself. Includes virtual or

Content Format non-HTML:

• Registration/login or restricted access or subscriber site. These sites would be

Ray Garcia, June 2006 Confidential, University of Arizona 12

• Collaboration or News site used for communications such as groups, blogs,

• Highly Interactive sites without a clear instructional value, includes flash

Site Selection Process

5. Using DMOZ search for sites related to the topic.

Ray Garcia, June 2006 Confidential, University of Arizona 13

8. Using Google, Gigablast.com, Ask.com or another search engine, with only

Ray Garcia, June 2006 Confidential, University of Arizona 14

Ray Garcia, June 2006 Confidential, University of Arizona 15

Ray Garcia, June 2006 Confidential, University of Arizona 16