A study to analyse the effectiveness of multimedia

and text-only e-mail communications and discover

whether consumers attitudes towards e-mail reflect
their behaviour

By Owen Simon Ball

Submitted in partial fulfilment of the requirements for the Degree:

Bachelor of Arts in Marketing & Advertising

University of the Arts London

London College of Communication: School of Creative Enterprise

March 2007
1. Acknowledgements
This Dissertation would not have come to fruition without the help, wittingly or
unwittingly, of a great many people too numerous to mention. This section provides
a platform to recognise and acknowledge those fortunate enough to spring to mind.

My dissertation tutor Jon Kitto should be commended for the guidance, support,
and wealth of ideas he provided in order to see this work completed.

I would like to express my gratitude to the BA Marketing & Advertising staff,

specifically Colin Watson for his introductory coffee on my first visit to the institution
and his subsequent help with the questionnaire designs for this study.

A huge thank-you Kenji Lim from Jail Clothing, without whom this study would not
have been devised or executed.

The staff at Apple, TextMate, Mellel & Campaign Monitor for developing exceptional
software that made my life a little easier, and saved me an overwhelming amount of
time and effort.

Luke Branley, Aimee Gillett and Reynaldo Canlas Jr. for kindly taking the time to
assist me with my research and distract me in equal measure.

All the unnamed people who kindly responded to the Questionnaire, I am indebted
to you.

Martin Hall in the Reprographics department at EHDC for providing the facilities to
print and bind this study.

Finally, my family for providing help, advice, and motivation when it was required.

-1-
2. Abstract
Technological developments have given rise to a global network of computers and
servers allowing its users to communicate with each other without geographic or
time constraints. The most popular, and established, medium of communication in
this global network is e-mail and it is this of communication that will be scrutinised
in this study. In its early days e-mail was constrained to plain text messages.
Advances in software and internet protocols have enabled the medium to carry a far
richer subset of information including, but not limited to, colour information, images
and varied typefaces. This new breed of multimedia e-mail, facilitated by HTML, has
polarised opinion. Proponents laud the new design capabilities made possible
whilst at the same time detractors lament the surfeit of rendering issues,
incompatibility and broken links they argue plague the format.

Placed in this context, the objectives of the research were two-fold. The first was to
identify the format of e-mail that was most effective. The second was to discover
whether recipients attitudes toward e-mail format reflected the behaviour observed
in relation to the first objective. In order to conduct the study three methods of
research were utilised. Desk research provided data for comparison and contrast,
an experiment was conducted to obtain metrics for the effectiveness of each format
of e-mail. Finally, a questionnaire was sent to the same sample used in the
experiment to determine whether their attitudes toward e-mail correlated with the
observed behaviour.

The study found that e-mail open and click-through rates both provided significant
evidence to show that multimedia e-mail was in fact more effective than text-only.
These findings were backed up by statistical analysis that showed that the
probability of the results being significant was extremely high. In addition to the
findings from the experiment the questionnaire responses suggested that there is a
strong correlation between the the attitudes of recipients and their behaviour.
Respondents preferred multimedia over text-only by a ratio of 3:1, a ratio that when
compared to click-through rates for each format was almost identical.

This study ultimately produced two noteworthy conclusions. The first conclusion is
that HTML e-mail is more effective than text-only e-mail, thereby rendering the
research hypothesis invalid. The second conclusion, relating to the second
objective, is that consumers attitudes show a strong correlation to their observed
behaviour, both showing a significant preference for multimedia e-mail.

-2-
3. Table of Contents
1. Acknowledgements ..............................................................p. 1

2. Abstract ..............................................................p. 2

3. Table of Contents ..............................................................p. 3

4. List of Appendices ..............................................................p. 8

5. Abbreviations ..............................................................p. 9

6. Introduction ..............................................................p. 10

7. Literature Review ..............................................................p. 12

7.1 Introduction ..............................................................p. 12
7.2 E-mail Marketing ..............................................................p. 12
7.2.1 E-mail ..............................................................p. 12
7.2.2 E-mail Marketing ..............................................p. 13
7.3 E-mail by Format ..............................................................p. 15
7.3.1 Evidence for Text-only .....................................p. 15
7.3.2 Evidence for HTML-only .................................p. 16
7.3.3 Evidence for both formats ................................p. 17
7.4 Further Theory ..............................................................p. 18

8. Research Questions ..............................................................p. 21

8.1 Area for Further Research ........................................................p. 21
8.2 Research Aims ..............................................................p. 21
8.3 Research Hypothesis ..............................................................p. 22

-3-
9. Methodology ..............................................................p. 23
9.1 Overview ..............................................................p. 23
9.2 Research Considerations ...........................................................p. 23
9.2.1 Quantitative vs. Qualitative ..............................p. 23
9.2.2 Triangulation.....................................................p. 24
9.3 Research Methods ..............................................................p. 25
9.3.1 Experiment ........................................................p. 25
a. Process ..............................................................p. 25
i. Sample ..................................................p. 25
ii. Content ..................................................p. 26
iii. Measurement.........................................p. 26
b. Limitations ..............................................................p. 26
9.3.2 Survey ..............................................................p. 27
a. Process ..............................................................p. 27
i. Sample ..................................................p. 28
ii. Content ..................................................p. 28
iii. Measurement.........................................p. 28
b. Limitations ..............................................................p. 29

10. Results & Findings ..............................................................p. 30

10.1 JailMail Multimedia ..............................................................p. 30
10.1.1 Metrics for Multimedia Campaign ...................p. 30
10.2 JailMail Text-only ..............................................................p. 32
10.2.1 Metrics for Text-only Campaign ......................p. 32
10.3 Experiment Analysis ..............................................................p. 34
10.3.1 Comparisons .....................................................p. 34
10.3.2 Chi Squared ......................................................p. 35
10.4 JailMail Survey ..............................................................p. 39
10.4.1 Survey Statistics................................................p. 39
10.4.2 Gender Analysis................................................p. 40
10.4.3 Age Analysis .....................................................p. 42
10.4.4 E-mail software analysis ...................................p. 43
10.4.5 Browser based e-mail analysis..........................p. 44

-4-
 10.4.6 Frequency respondents check e-mail analysis ..p. 45
10.4.7 E-mail format analysis ......................................p. 46
10.4.8 Reasons for choosing multimedia e-mail analysis
..............................................................p. 49
10.4.9 Reasons for choosing text-only e-mail analysis
..............................................................p. 50
10.4.10 Respondent's preferred hyper-link format analysis
..............................................................p. 51
10.4.11 Preferred frequency for e-mail communications
analysis .............................................................p. 52
10.4.12 Behaviour towards unwanted commercial e-mail
communication analysis....................................p. 53
10.4.13 The reasons respondents gave for not choosing to
unsubscribe analysis .........................................p. 55
10.5 Survey Analysis ..............................................................p. 56
10.5.1 Comparisons .....................................................p. 56
a. Gender Vs. E-Mail Format ...............................p. 56
b. Age Vs. E-Mail Format ....................................p. 58

11. Conclusions ..............................................................p. 61

11.1 Key Conclusions ..............................................................p. 61
11.1.1 Experiment ........................................................p. 61
a. Open Rate .........................................................p. 61
b. Click-through Rate............................................p. 62
11.1.2 Questionnaire ....................................................p. 63
11.2 Qualifying Statement ..............................................................p. 63
11.3 Limitations ..............................................................p. 64
11.4 Further Research ..............................................................p. 65

12. Appendices ..............................................................p. 66

13. Bibliography ..............................................................p. 73

-5-
 List of Tables

Table 1: Comparing unique click-throughs by format .....................................p. 18

Table 2: Effectiveness of Image & Text dominant web banner ads .................p. 19
Table 3: A Taxonomy of Research Methodologies ..........................................p. 24
Table 4: Unique E-mail Opens Chi Squared Figures .......................................p. 35
Table 5: Unique E-Mail Clicks Chi Squared Figures .......................................p. 37
Table 6: Total E-mail Opens Chi Squared Figures ...........................................p. 37
Table 7: Survey - Gender Chi Squared .............................................................p. 40
Table 8: Survey - Multimedia vs Text Chi Squared .........................................p. 46
Table 9: Survey - Revised Multimedia vs Text Chi Squared ...........................p. 48
Table 10: Gender vs Preferred E-Mail Format .................................................p. 56
Table 11: Age vs Preferred E-Mail Format ......................................................p. 58
Table 12: Further Research Possibilities ..........................................................p. 65

List of Figures

Figure 1: Opens, Bounces & Unopened numbers for multimedia campaign ...p. 30
Figure 2: Total links clicked for multimedia campaign....................................p. 31
Figure 3: Opens, Bounces & Unopened numbers for text campaign ...............p. 32
Figure 4: Total links clicked for text campaign ................................................p. 33
Figure 5: Survey Statistics ..............................................................p. 39
Figure 6: Q1: Respondent Gender ..............................................................p. 40
Figure 7: Q2: Respondent Age ..............................................................p. 42
Figure 8: Q3: Respondent's e-mail software .....................................................p. 43
Figure 9: Q3a: Respondent's browser based e-mail choice ..............................p. 44
Figure 10: Q4: Frequency respondents check e-mail .......................................p. 45
Figure 11: Q5: Respondents preference of e-mail format ................................p. 46
Figure 12: Click-through compared to Preferred Format .................................p. 47
Figure 13: Q5a: Reasons for choosing multimedia e-mail ...............................p. 49
Figure 14: Q5b: Reasons for choosing text-only e-mail ...................................p. 50
Figure 15: Q6: Respondent's preferred hyper-link format ................................p. 51

-6-
Figure 16: Q7: Respondent's preferred frequency for e-mail communications
..............................................................p. 52
Figure 17: Q8: Behaviour towards unwanted commercial e-mail communication
..............................................................p. 53
Figure 18: Open & Unsubscribe Percentages comparison ...............................p. 54
Figure 19: Q8a: The reasons respondents gave for not choosing to unsubscribe
..............................................................p. 55
Figure 20: Gender vs Preferred E-Mail Format ................................................p. 57
Figure 21: Age vs Preferred E-Mail Format % by Age ....................................p. 59
Figure 22: Age vs Preferred E-Mail Format Frequency by Age ......................p. 59

-7-
4. List of Appendices
Appendix 1: JailMail Multimedia Format .......................................................p. 66
Appendix 2: JailMail Text-only Format ..........................................................p. 67
Appendix 3: JailMail Questionnaire ..............................................................p. 68
Appendix 4: Chi Squared Test Table ..............................................................p. 72

-8-
5. Abbreviations & Definitions
Click-through rate - The number of hyperlinks accessed from a campaign
E-mail - Electronic Mail
HTML - Hyper Text Mark-up Language
IPT - Interactive Prospect Targeting (company)
ISP - Internet Service Provider
Link(s) - Hyperlink (Method of navigating to content)
MIME - Multipurpose Internet Mail Extensions
Open-rate - The number of e-mails opened by recipients
Spam - Unsolicited Commercial E-mail

-9-
6. Introduction
The Global Reach research agency (2004) estimated that by the year 2005 there would be
over 1 billion users of the internet globally. Whilst it impossible to provide truly accurate
numbers this still represents a massive potential audience for marketing messages, and
because the internet transcends the limitations of time and space this audience can be reached
almost instantaneously. E-mail is one of the most established methods of communication
used online, according to IPT (2004), 99% of consumers use the internet for email which
must place the medium as a one of few communication methods with such a large reach.

If we accept then that e-mail has vast potential in terms of numbers, the next question to be
asked is what are the implications for marketers. According to the same IPT study, e-mail
marketing ranked second as the medium consumers felt was most effective. It is important
then, considering the numbers involved and consumers receptiveness to the format, that
marketers assess the best way with which to communicate their messages. There are currently
two clear options for e-mail messages. Traditional text formatted e-mail, or multimedia e-
mail facilitated by the use of HTML code. In some cases, it is very easy for a marketer to
make the choice to use one or the other, however, how does a marketer know whether to
spend the time and money required to develop an HTML based email campaign? This
research, in an attempt to provide empirical evidence on this issue, uses an experiment, and
questionnaire to discover what format of e-mail is most effective, and whether consumers
attitudes toward e-mail marketing reflect their behaviour in response to it. The aim of the
research then is to provide the marketer with quantifiable evidence that can be used to argue
in favour of either approach.

The dissertation is broken down into five broad section, beginning with a review of current
literature in the subject area. This is followed by the Research Questions and Research
Methodology chapters that define the area of research and how the research will be

- 10 -
conducted. The following section illustrates the findings from the experiment and
questionnaire. The dissertation is concluded by a chapter that draws together the key
conclusions from the findings.

- 11 -
7. Literature Review
7. 1. Introduction

The relatively recent development of email, in particular multimedia based email facilitated
by HTML, as a marketing tool ultimately restricts the pool of previous academic research in
the field. This study concerns itself with the effectiveness of two broad categories of email
communication, that of graphically rich multimedia email and that of text-only email. It is
perhaps to be expected that the majority of information available regarding email and email
marketing is based on the world wide web rather than in books and journals, and it is true to
say that there is a large amount of subjective and anecdotal evidence that can be found
supporting the hypothesis that either type of email is more effective than the other. This
literature review will provide an objective view of the material available using, where
appropriate web-based and print-based material. The literature review will be segmented into
four logical sections. The first section will focus on email marketing, citing work from
figures and companies well-regarded in the industry alongside more anecdotal evidence. The
second section will be more empirical in nature addressing previous studies and research
supporting the case for either format. The third section will discuss relevant theory relating
primarily to the composition of communications. The final section will draw conclusions
from the research presented.

7. 2. E-mail Marketing

7. 2. 1. E-mail

The aim of this section is to define email marketing and give the reader an understanding of
the importance of email and why it is an important element of a marketing mix. According to
the Oxford American Dictionary e-mail stands for "messages distributed by electronic means

- 12 -
from one computer user to one or more recipients via a network." The internet, in its current
form, was not a prerequisite for e-mail. Indeed, the first crude form of e-mail was sent by Ray
Tomlinson of ARPANET in 1971 (Griffiths, R.T., 2002), however, in order for the subject
area of this research, email marketing, to be utilised by marketers, the internet along with
more advanced software and networking technology was required. With this backbone in
place Hoffman (2000) argues that e-mail has become "the most important innovation since
the development of the printing press" able to "transform not just the way individuals go
about conducting their business with each other, but also the very essence of what it means to
be a human being in society." This is certainly an extreme viewpoint, but there is some
justification to Hoffman's views. The following figures make impressive reading and
certainly suggest that e-mail has become an intrinsic part of many peoples lives. According to
Messaging Online (2000) "e-mail is the fastest-growing communications technology in
history. From only 2m e-mail accounts in 1985, this grew to 891.1m e-mail accounts at the
beginning of 2001." This number is now even higher, according to The Radicati Group
(2006) there are a staggering "668 million email users worldwide, with over 1.2 billion active
email accounts." When this is coupled with "worldwide email traffic per day total[ing] about
135 billion messages" the result is a behemoth of a communication medium.

7. 2. 2. E-mail Marketing

Purely in terms of metrics the previous paragraph illustrates just how valuable e-mail could
be to an online marketing program, and it is true to say this is being followed up by
marketers. An IPT email marketing survey (2004) found that 92% percent of professional
respondents currently used email marketing with over half (51%) declaring that the budget
would increase the following year. It is perhaps not surprising that these figures are so
positive given the findings from the consumer part of the survey. According to IPT, 99% of
consumers use the internet for email creating a vast potential audience for marketing
messages. Even more encouraging for marketers is the medium consumers felt was the most
effective for marketing communication. Email ranked second at 32%, a score only 7% behind
television, but markedly higher than press, radio and other forms. Clearly these statistics have
to be approached with caution, the answers consumers give, compared to their actual

- 13 -
behaviour can often be distinctly different. Still, the figures do show consumers are still
receptive to email communication despite that fact that "the average email user gets between
54 and 93 e-mails per week, a number that increases in direct proportion to the length of time
the person has been using email" (Brondmo, H. P., 2001).

Utilised as a marketing tool email has both advantages and disadvantages. The key
advantages of email marketing are two-fold and intrinsically linked. The first is low costs and
the second is the ability to utilise a digital work-flow. (Jackson & DeCormier, 1999; Peppers
& Rodgers, 2000). Actual figures vary by campaign but Dave Chaffey (2006) estimates that
"currently, e-mail costs range from $5 to $7 per thousand compared to $500 to $700 per
thousand for direct mail." If e-mail costs just 1% per thousand to reach customers compared
to direct mail the casual observer may question whether, in the words of the age-old idiom,
"you get what you pay for." According to Hans Peter Brondmo (2001) "the best email
marketing programs, in fact, routinely enjoy response rates above 30 percent, while even
fairly simple promotional marketing campaigns routinely see a 4 to 6 percent response
(compared to 1 to 2 percent response rate from traditional direct mail campaigns)." The
figures are a positive re-enforcement for email, but Brondmo continues with a note of
caution. He says that "with all the information consumers are exposed to on a daily basis, it's
getting harder and harder to separate the good from the bad." This a viewpoint expanded by
Windham (2000) who suggests that email marketing has one major disadvantage, the scourge
that is spam, or unsolicited email. Not only does this taint the reputation of email as a form of
communication, but it directly affects response rates to campaigns. In addition many counter
spam measures introduced by ISPs and email providers have had a significant detrimental
effect on email open rates and created rendering issues (DoubleClick, 2005).

- 14 -
7. 3. E-mail by Format

7. 3. 1. Evidence for Text-only

The following paragraph will assess evidence that suggests text-only email messages are
more effective than HTML. It is to perhaps to the credit of what Jeanne Jennings (2004)
describes as "the anti-HTML group", a "small but vocal minority" that there is any debate on
the issue of what format of email is most effective. Evidence in favour of text-only is
underwhelming and, in addition, predominately based on figures from the first couple of
years of the twentieth century. A report on consumers' preferred e-mail advertisement formats
by eMarketer (2002) discovered that a staggering 62% of respondents preferred text email
messages , with only 35% favouring HTML. These results supported findings from another
eMarketer report carried out in 2001 assessing response rates to e-mail marketing campaigns.
It found that text email enjoyed an 18.5% click-through rate compared to 15.6% for the
HTML equivalent. Research undertaken in the years proceeding the data above tend to show
a slight preference for HTML as we shall see in the following section, however there is some
small consolation for text email. Jupiter Research (2005) attempted to identify what impact
the content of an email communication has upon response behaviour. Out of a sample size of
1166 consumers, written copy with 40% of responses was identified as the second most
important factor for respondents. This compares with 12% and 9% for one large image and
multiple smaller images respectively. This result does have to be put in some context.
Because there is no distinction between HTML and text email, it is entirely possible that
readers preferred HTML email, but still felt the body copy was the most important factor for
initiating response. Finally, A DoubleClick (2005) email trend report for Q1 2005 notes that
"Despite performing the least well, text showed the greatest year-on-year increase in click-
through rates up 19.6%, from 5.6% to 6.7%", and "the UK, in particular, saw its text click-
through rates increase by 136.7%." The report suggests this may be influenced by a trend
towards email access on mobile devices. This is a view shared by Dave Chaffey (2006) who
notes that there are many email readers that cannot easily display HTML, such as Blackberry
and PDA devices.

- 15 -
 7. 3. 2. Evidence for HTML-only

The previous section alluded to a trend where earlier research favoured text email and later
HTML. The metrics suggest that this is indeed the case, however, an eMarketer report (2001)
focused on consumer attitudes towards e-mail formats rather than email delivery statistics.
The results are of particular interest because they show a huge disparity with much of the
research at that time. The key finding was that 60% of respondents would rather receive
HTML over text email. This result supports more recent research suggesting that the case for
HTML email being the bastion of email marketing is stronger than that for text. This
viewpoint is supported by anecdotal alongside empirical data both of which will be detailed
in this section. To begin, a reader comments section on a text vs. HTML article by Jeanne
Jennings (2004 ) for Clickz resulted in "less than 3 percent of responses [being] anti-html."
This does not of course mean that 97 percent of respondents were pro HTML email, but it
does suggest that there is not a large swath of antipathy towards HTML. In addition
comments e-mailed to the author by marketers strongly suggest HTML is now the de-facto
standard for email. Paul Maloy (2004) says "the newsletter list that I maintain favors HTML
by about 12 to 1", a sentiment shared by Paula Skaper (2004) who adds that "more than 90
percent of readers choose the HTML version over text." And "until that metric changes
significantly, there's simply no solid business case for dumping HTML."

Data from DoubleClick (2005) echoes the views given in the preceding paragraph.
Discussing email format trends the report writes "HTML was, by far, the most popular email
format in Q1 2005 with 64.3% of e-mails being sent in this format. This was followed by
Multi-part ... at 20.9% and Text at 14.8%." The report continues "in Q1 2005, higher click-
through rates were achieved by formatted content with both HTML (11.5%) and Multi-Part
(11.0%) performing above the overall average of 10.0% for all types." However "Text
continued to perform least well at 6.7%". These results show that e-marketers clearly feel
HTML is the most effective method of communication, with the metrics backing up their
argument. Indeed the Q2 report from DoubleClick (2005) provides similar statistics. One of
the key findings in the report concluded that "although open rates declined from 36.0% in Q2
2004 to 27.5% in Q2 2005, click rates remained relatively stable, dipping slightly from 7.7%

- 16 -
to 7.2%". The report continues to say that "this relative stability in click rates, despite the
decline in opens, mirrors the HTML click-to-open ratio (which increased from 26.5% to
32.0% from Q2 2004 to Q2 2005)." A higher click-to-open ratio suggests that the content of
an email is effective in stimulating click-through once an email has been opened with HTML
the format that can facilitate this.

7. 3. 3. Evidence for both formats

In a survey of 50000 readers split evenly between HTML "lite", HTML and text based email,
it was found that almost no difference in effectiveness could be measured. The completion
rates were 7.2%, 7% and 7.5% respectively casting doubt on the supposition that either text
or HTML are more effective than the other (Marketing Sherpa, 2003). The article continues
to suggest that if a strong enough relationship between the business and consumer has been
forged, email format becomes a non-issue. In support of these findings, a survey of 600
people carried out by Lynda Partner (2003) for MarketingProfs discovered that just over half
preferred HTML email and just under half plain text.

Perhaps the most pertinent article is that of Tom O'Leary (2006) for GroupMail. O'Leary
proposes that due to technological advances in email composition software and readers the
best format to use is both. Using something called a multi-MIME message, it is possible to
send both HTML and text in the same communication. If a reader cannot read the HTML
correctly it can degrade gracefully down to the text version included, or alternatively the
reader can be programmed to show only the text-based version when available (Chaffey,
2006). To support the point O'Leary provides figures from a GroupMetrics report on unique
click-through from a list size of 10000.

- 17 -
Table 1: Comparing unique click-throughs by format
HTML Only 1436 Unique clicks

Text Only 1353 Unique clicks

HTML and Text 1518 Unique clicks

Source: GroupMetrics

Text is the worst performer, though not significantly below the performance of HTML. It is
clear however that the combination approach yields the best results. These results also
suggest that HTML would perhaps perform better without some of the technical and
rendering difficulties that can be associated with it, and which cause the reader to default to
text.

7. 4. Further Theory

A study conducted by Diameter for DoubleClick Media (2001) aimed to analyse the
effectiveness of each element of a web banner advertisement on ad recall. The study
concluded that in some situations visual images are second to brand logo in eliciting recall.
Whilst it did not discuss image versus text formatted advertisements, the findings suggest that
companies sending HTML formatted email should pay close attention to the branding of the
email above other image content. Two more studies, although assessing web banners as
opposed to email, can be applied to email formatting. The first of the studies carried out by
Electronic Telegraph/Ogilvy & Mather (1997) measured the effectiveness of each of the
creative elements incorporated into banner advertising. The findings of the study suggested
that image dominant adverts had a stronger brand recall than that of text dominant
advertising. However, closer scrutiny reveals only a slight difference. Image and text
dominated adverts had recall percentages of 60% and 56% respectively.

- 18 -
Utilising a similar comparison between image and text dominant web banner adverts, but
focusing on click-through-rate, BannerTips discovered that text dominant banners performed
better than their image dominant counterparts. In the context of email marketing these
findings suggest that the goal of each email communication should be reflected in the
formatting used, with the appropriate formatting utilised depending on whether the email was
designed to be a "call to action" or informative. The following table summarises these
findings.

Table 2: Effectiveness of Image & Text dominant web banner ads

Effectiveness

Attribute Brand-Reinforcement Click-through

Image Dominant Increase Decrease

Text Dominant Decrease Increase

Source: Park, M.H., 2002

It should be re-iterated that the findings in the studies preceding this paragraph provide data
based on the observation of banner adverts, not email. It is not then surprising that research
on the factors affecting click-through-rate in e-mail marketing (Chittenden, L. & Rettie, R.,
2002) produced contradictory results. Comparing click-through rate to creative information
they noted that "the number of images is less in the lower responding group, suggesting that
the more colourful and attractive e-mails generate greater response." If these results are
considered representative they provide a strong argument that not only are consumers
receptive to HTML email, but using HTML to provide colour and images actually improves
performance.

Eyetrack III (2004) undertook research studying the recall of information presented in text vs.
multimedia format. Although this study did not measure how effective each format was at
eliciting a physical response (click-through), it is important to remember not all email
marketing efforts are for that purpose. According to the IPT Email Marketing Survey (2004),

- 19 -
75% of marketers use email for customer retention, 15% for acquisition, 9% for cross selling
and 1% for branding. Retention may include information or news designed to keep the brand
in the customers mind rather than be a specific call to action. That said, the research shares
some similarities with email marketing and therefore it is possible to apply the results. The
aim of the research was to assess whether "the presentation of editorial content in multimedia
format [would] help readers understand and remember more [news] story information." Half
the participants viewed a news story in text only, and half in multimedia after which they all
completed the same recall quiz. Without delving too far into the data the study uncovered
three findings of relevance.

• Users that received information in text form seemed to have a better recall of
specific factual information, names, places etc.
• Information about an unfamiliar process or procedure was more correctly
recalled after viewing the multimedia.
• There was no significant difference between men and women regardless of
presentation format.

Source: Eyetrack III, 2004

In the context of e-mail marketing the first finding suggests that important information should
be presented to customers in text regardless of the format used. The second suggests that new
or unfamiliar products can be better represented through multimedia or a combination of
both. The final finding suggests that there will be no discernible difference in the
effectiveness of HTML or text email with regard to gender. This is a stance supported by a
Pew Internet report (2005) looking at how men and women use the internet. The report
concludes that although there are differences, men and women are more similar than different
in their online lives. Further research would be needed to identify whether this trend follow
through to email usage and effectiveness.

- 20 -
8. Research Questions
8. 1. Area for Further Research

The primary conclusion from the desk research was that although current research points to
HTML being the more effective format there is sufficient evidence to suggest that this is not
conclusive (see chapters 7.3.1 and 7.3.3). Indeed, if consider that the majority of evidence in
favour for HTML is based on metrics, and these metrics are not a "100% accurate measure"
(Patterson, 2007), then it would be ill-advised to infer that HTML is more effective. The
GroupMail report (2006) outlined in chapter 7.3.3, whilst concluding that sending e-mail in
both formats is preferable, also provide, numbers (Table 1) see that suggest there is virtually
no distinction in click-through rate between the formats. The lack of a unanimous agreement
on the topic provides scope for further research to be carried out. This study will attempt to
ascertain whether there is indeed a difference in effectiveness between HTML and text e-
mail. In addition, rather than focus entirely on metrics this study will explore consumer
attitudes toward the formatting of e-mail marketing communications and discover whether
these attitudes match the metric data.

8. 2. Research Aims

To determine if there is quantifiable difference in effectiveness obtained by utilising either

multimedia (HTML) or text-only email communications.

To identify if a sample groups attitudes toward multimedia (HTML) and text e-mail matched
their actions when presented with either type.

- 21 -
8. 3. Research Hypothesis

"There is no difference in effectiveness between HTML and text-only email"

- 22 -
9. Methodology
9. 1. Overview

This dissertation tests the hypothesis "There is no difference in effectiveness between

multimedia and text-only email." Two complimentary types of primary research were used,
an experiment and a questionnaire. The goal of the experiment was to discover what format
of e-mail, HTML or text, was the most effective. A questionnaire was then used to assess
whether the measured results reflected their thought process or whether the two were in
opposition. For this purpose it was important to send the experimental e-mail campaign first
to prevent any bias corrupting the results after a questionnaire had been carried out. It is
important to set down in words that although participants were unaware that their actions
were being measured they remained completely anonymous in the experiment. Those
participants who took part in the questionnaire also remained anonymous. Upon the
conclusion of the primary research, both sets of data could be cross-analysed and patterns,
trends and other conclusions could be drawn from the results. This chapter then provides the
reader with an overview of the research utilised, and a justification for the choices taken for
this study. This chapter will be broken down into sections, each covering one part of the
process.

9. 2. Research Considerations

9. 2. 1. Quantitative vs. Qualitative

A positivist or scientific approach to research asserts that reality is stable and may be
observed objectively (Levin, 1988), this approach is in contrast to an interpretivist outlook
which proposes that "only through the subjective interpretation of and intervention in reality
can that reality be fully understood" (Davison, 1998). In an attempt to nullify the "paradigm

- 23 -
wars" between positivists and interpretivists, Benbasat (1984) and Pervan (1994) observe that
the methodology chosen for research should be suited to the problem being investigated and
the objectives of the research. The specific nature of the research and the ability to prove or
disprove the hypothesis with purely quantitative data influenced the decision to follow a
positivist rather than interpretivist approach. The two methods of primary research used to
collect data for the study have been identified by Galliers (1991) to be positivist (see table 3),
and these along with justification for there use are detailed in the following sections.

Table 3: A Taxonomy of Research Methodologies

Positivist Interpretivist

Lab Experiment Argumentative

Field Experiment X Reviews

Surveys X Action Research

Case Studies Case Studies

Theorem Proof Descriptive

Forecasting Futures Research

Simulation Role/Game playing

9. 2. 2. Triangulation

Triangulation identifies the process of using multiple forms research in order to get two or
more viewpoints on topic being studied, utilising this technique prevents anomalous results
from one set of data biasing the overall findings. This study has used three sources of data for
analysis. Desk research conducted prior to the primary research unearthed an abundance of
metrics directly related to the subject topic. This research was then supplemented by
quantifiable primary research in the form of an experiment and a survey.

- 24 -
9. 3. Research Methods

9. 3. 1. Experiment

The objective of the experiment used for the study was to discover whether there is a
measurable difference in effectiveness between multimedia HTML email and text-only email.
In order for this to be achieved, an email communication formatted in both HTML and text
was proposed. The sample was split in half with each half receiving one format of email. The
effectiveness of each format of email would be assessed on two criteria. The first is open rate,
the number of unique opens from the sample. The second criteria is click-through, how many
people, after opening the email, ultimately went on to click one of the enclosed links.

a. Process

In order for the experiment to be fully realised a set of variables first had to be defined. This
section will deal with each of these variables in turn which, in turn, will describe the process
undertaken to complete the experiment.

i. Sample

To carry out a live experiment using e-mail is problematic because US and EU anti-spam
legislation requires that all recipients must have opted-in to receive the communication. This
issue was circumvented by conducting the research on behalf of a company called Jail
Clothing. Jail Clothing has a house list of opted-in subscribers totalling 166 at the time of the
experiment.

- 25 -
 ii. Content

The next issue was providing the content for both formats of email. It is difficult to obtain
measurements of effectiveness if there is no incentive for the viewer to open or respond to the
email. The partnership with Jail Clothing negated this problem. Jail send out regular email
newsletters, and it is one of these that was modified for the experiment. Recipients would
already recognise the sender ID and so the email was far less likely to be Blocked by an ISP
or filtered into junk or spam email folders by recipients. The two e-mails (see appendix 1 and
2) were crafted to have identical content as far as possible in order for the formatting of each
to be the only variable, and there-fore any difference in effectiveness between the two could
be measured. Both e-mails were hand-coded in TextMate and based on a template provided
by Campaign Monitor. Finally, because the text e-mail had to be sent in HTML format (see
Limitations below) a conscious decision was made to format hyperlinks in HTML so that
they were embedded in words. This way the links looked the same to the recipient regardless
of which format was delivered.

iii. Measurement

In order to measure the open and click through numbers for both e-mail campaigns Campaign
Monitor, an on-line e-mail service provider was utilised. The browser based software, already
in use by Jail for its e-mail communications, allows the user to measure, amongst other
occurrences, the open rate and click-through metrics that will used as a measurement of
effectiveness in this study. In addition Campaign Monitor generated and exported reports
directly saving the user the need to tackle this manually.

b. Limitations

There were three obvious limitation in the design of the experiment, one inherent to the
research methodology and the other two specific to this study. The inherent limitation of an

- 26 -
experiment is that it does not find out why the results have happened, in other words, it is
difficult to extrapolate meaning from quantified data. This limitation was bridged through the
use of triangulation (discussed above). The second limitation was a purely down to the
technology available. The text-only version of the e-mail had to be sent in HTML format. It
is impossible to measure open rates for text e-mail because in order for the measurement to
take place a tiny invisible image has to be requested from the host server, and this feature is
available only in HTML (Patterson, 2007). The final limitation is a limitation of format. It
was impossible for both versions of e-mail to be truly identical in everything except format
due to constraints with text-only formatting. Compromises had to be made, predominately
due to the lack of support for columns. The links down the left hand side of the multimedia e-
mail (appendix 1) had to be placed after the content of the e-mail for the text version
(appendix 2).

9. 3. 2. Survey

The objective of the survey questionnaire was to evaluate two things. The first, whether or
not the recipients views on e-mail format reflected the actions they had taken when receiving
the test campaign. And second, to identify if there were any other significant patterns or
trends that could be extrapolated from the answers given. These could have included
variables such as gender and e-mail provider.

a. Process

The following section will outline the processes undertaken to enable the questionnaire to be
carried out. In similar circumstances to the experiment there were a number of variables that
needed to addressed and a number of compromises that needed to be made.

- 27 -
 i. Sample

In order for the sample used for the questionnaire to provide meaningful results it was
imperative that the recipients were the same ones who had received the email campaign. With
the permission of Jail Clothing the questionnaire was sent out to the subscriber list for Jail’s
newsletter. Due to the “live” nature of the study and to stay compliant to US and EU anti-
spam regulation it was necessary to send the survey only to those people who had opted in to
receive information from Jail. The questionnaire was sent by e-mail invitation to 166
recipients.

ii. Content

The final content of the questionnaire (appendix 3) was written to satisfy the needs of the
study but, due to the link to Jail Clothing, the language used and nature of certain questions
were written to appeal to Jail’s current subscribers. The content of the questionnaire covered
basic demographic information, namely gender and age, alongside email usage and reasoning
behind email usage questions. The questionnaire was composed and piloted using online
software from Question Pro. The digital nature of the survey allowed changes to be
implemented instantly in response to feedback from test recipients improving the quality and
accuracy of the results obtained. The live questionnaire was subsequently delivered using the
software's built in e-mail list manager using a recipient list imported from Campaign
Monitor.

iii. Measurement

Conducting the survey online enabled the questionnaire to be delivered to recipients directly,
and responses recorded and collated in real-time. In addition, analogous to the Campaign
Monitor tools, the software provided real-time customised reports negating the need to
complete this procedure manually.

- 28 -
 b. Limitations

There were two limitations observed with the design and execution of the survey
questionnaire. The first was a relatively minor issue concerning the wording in certain
questions. Bartholomew (1963) suggested that uncommon words and jargon can be included
in questionnaires provided that the recipients are familiar with them. Piloting suggested that
the majority of terms were acceptable, however, "web-based e-mail" was deemed ambiguous
and this was changed in the live questionnaire to "browser-based e-mail" with examples
provided. In addition it was considered that "HTML"was too technical in nature and may not
be understood. "HTML" was subsequently substituted with "multimedia" to signify the use of
colour, formatting and images within an e-mail. This definition was provided to participants
along side the question.

The major limitation of the survey was the sample used and the consequences arising directly
from this decision. In order to discover a set of significant results it was necessary to use the
same sample group for the experiment and survey. Due to the remote nature of the sample the
only viable method of reaching them was to use e-mail. This in turn meant that the
questionnaire became vulnerable to the same problems that faced the e-mail campaign.
Indeed, the same metrics used to analyse the effectiveness of the e-mail experiment (open-
rate and click-through rate) would be significant factors in the response rate for the survey.
Walonick (1993) states that "Low response is the curse of statistical analysis" and it can
"dramatically lower confidence in the results." In order to aid response rates for the
questionnaire two techniques were used. The first was to place an article in the e-mail
campaign outlining Jails intention to conduct a survey. Participants who read the e-mail
would there-fore be expecting the questionnaire. The second technique was to provide the
recipient with an incentive to complete the survey. An agreement was made with Jail
Clothing for prizes to be provided for this purpose.

- 29 -
10. Results & Findings
10. 1. JailMail Multimedia

10. 1. 1. Metrics for Multimedia Campaign

Figure 1: Opens, Bounces & Unopened numbers for multimedia campaign

Sent to 83 recipients on 15 Feb 2007 at 5:00 PM

Delivered 79 (95.18%)
Total Opens 47
Unique Opened 36 (43.37%)
Clicks 15 (41.67%) recipients clicked 25 unique links
Unsubscribed 0 (0%)
Bounced 4 (4.82%)
Total Unopened 47 (56.63%)

- 30 -
Figure 2: Total links clicked for multimedia campaign

Figure 2 show the total, not unique, clicks on hyperlinks from the multimedia campaign. 8
different links were accessed a total of 28 times.

- 31 -
10. 2. JailMail Text-only

10. 2. 1. Metrics for Text-only Campaign

Figure 3: Opens, Bounces & Unopened numbers for text campaign

Sent to 83 recipients on 15 Feb 2007 at 5:00 PM

Delivered 75 (90.36%)
Total Opens 32
Unique Opened 27 (32.53%)
Clicks 5 (18.52%) recipients clicked 5 unique links
Unsubscribed 1 (1.2%)
Bounced 8 (9.64%)
Total Unopened 56 (67.47%)

- 32 -
Figure 4: Total links clicked for text campaign

Figure 4 show the total, not unique, clicks on hyperlinks from the text-only campaign. 3
different links were accessed a total of 5 times. Significantly less than the multimedia
campaign.

- 33 -
10. 3. Experiment Analysis

10. 3. 1. Comparisons

There is no significant difference in the delivery rate between the two campaigns. This
suggests that some technological issues such as ISP image blocking (DoubleClick, 2005)
have not directly affected the results. The same conclusion can be drawn for the "unique
opens" numbers which show that although there is a declining trend for open rates
(DoubleClick, 2005), this has not influenced the results. The two most significant
comparisons are the "unique opens" and the "click-throughs". Both criteria show the most
disparate results between the two campaigns. Using just the data from the table these two
results strongly suggest that the null-hypothesis should be abandoned in favour of declaring
HTML more effective. However, further statistical analysis is required (see 10.3.2)

- 34 -
 10. 3. 2. Chi Squared

The Chi Squared analysis is used "to test whether the number of individuals in different
categories fit a null hypothesis" (Deacon, J., n.d). The result, after being compared to a table
of values for Chi Squared, indicates the probability that the observations recorded in an
experiment are significant. If there is a statistically significant difference between the
observed numbers and the estimated numbers, this test provides the researcher with
supporting evidence to reject the null hypothesis. Conversely, a Chi squared value that is less
than the tabulated value for Chi squared gives the researcher no reason to reject the null
hypothesis. As a final note, according to Deacon (n.d) "When there are only two categories
(e.g. male/female) or, more correctly, when there is only one degree of freedom, the c2 test
should not, strictly, be used. There have been various attempts to correct this deficiency, but
the simplest is to apply Yates correction to our data." It is enough for this study to express
that this deficiency has been noted, and the Yates correction has been duly applied. Three Chi
Squared analysis will be presented in this section with accompanying explanation. Each of
the three tables represents one of the metrics extracted from the e-mail campaign that can be
used to assess effectiveness.

Table 4: Unique E-mail Opens Chi Squared Figures

- 35 -
When the Chi Squared Value of 1.02 is compared to the analysis table (appendix 4) it can be
seen that with 1 degree of freedom a value of 3.84 is required for a 95% probability that the
result is significant. The recorded value is distinctly lower than 3.84 and, with a probability at
less than 75% does not provide compelling evidence for the null hypothesis to be abandoned.

- 36 -
Table 5: Unique E-Mail Clicks Chi Squared Figures

In the case of Unique Clicks the Chi Squared value of 4.05 at 1 degree of freedom means that
we can say with a 95% probability that this result is a significant departure from the null
hypothesis. (Note: the values used represent the number of unique recipients who clicked on
a minimum of one link, not the total number of links clicked.)

Table 6: Total E-mail Opens Chi Squared Figures

The final table shows a Chi squared analysis for the total number of times the e-mail was
opened rather than just the unique open values. The Chi squared value of 2.48 along with 1

- 37 -
degree of freedom is lower than the value of 3.84 required for a 95% probability. However, it
does fall in the 90-95% probability range suggesting it is still a significant finding.

- 38 -
10. 4. JailMail Survey

Figure 5: Survey Statistics

10. 4. 1. Survey Statistics analysis

The statistics show that from a sample of 166 e-mail recipients 31 (18.67%) clicked through
to the questionnaire from the invitation. 21 (12.65%) respondents completed the
questionnaire. This a small response rate that jeopardises the reliability of the observations
recorded, however, if we consider the average click-through fell to 7.2% in 2005
(DoubleClick, 2005) then the click-through and response rate for this questionnaire is
statistically much greater than the average. Only 3 recipients dropped out of the survey before
completion suggesting that the majority of respondents felt comfortable with, and understood
the questions.

- 39 -
Figure 6: Q1: Respondent Gender

10. 4. 2. Gender Analysis

Table 7: Survey - Gender Chi Squared

- 40 -
The results show that there is a significantly higher proportion of male respondents to female,
and that this is significant. The value of 3.05 gives a 90-95% probability of significance.
There are a number of reasons this could be. The first possibility is for the JailMail list to be
gender biased. Gender statistics for the whole sample group were unavailable to test this
theory. The second possibility is that males were more likely to respond to the questionnaire
there-by biasing the results. The final explanation is that the sample size and resultant
response rate were too low for the gender sample to even out. Whatever the reason, this
finding will bias findings drawn from the questionnaire.

- 41 -
Figure 7: Q2: Respondent Age

10. 4. 3. Age Analysis

The key finding that two thirds of respondents are between the ages of 18 and 27 is
significant because it is likely to bias findings drawn from the questionnaire. Jail Clothing has
strong links with extreme sports, in particular snowboarding, and whilst a young age is not a
prerequisite for pursuing these sports a Mintel (2005) report into extreme sports suggests that
extreme sports "have much higher penetration levels amongst those under 25 ... due to the
physical nature of the sports in question."

- 42 -
Figure 8: Q3: Respondent's e-mail software

10. 4. 4. E-mail software analysis

It is clear from the results that accessing e-mail via a browser is the preferred method for the
sample group. It is possible that this factor would influence e-mail effectiveness. Most e-mail
clients provide superior rendering for HTML than their web-based counterparts and so this
would certainly not have a negative influence upon click through. However, many e-mail
clients have preview panes that automatically download tracking images thereby causing the
tracking software to register this as an open (Patterson, 2007). As 86% of the total
respondents (21) use browser-based e-mail, for the purpose of this research this factor can be
discounted as an influence on effectiveness.

- 43 -
Figure 9: Q3a: Respondent's browser based e-mail choice

10. 4. 5. Browser based e-mail analysis

Those users who selected browser-based e-mail in the previous question were asked to clarify
their choice. Over 60% chose Hotmail which is significantly high. As a comparison,
according to Emailcenter (2004) 40% of their test list email addresses were Hotmail or MSN.
It is certainly possible for the choice of e-mail provider to affect an e-mail campaign, in
particular a campaign utilising HTML code. A silverPOP (2005) report suggests that 40% of
e-mails contain missing or blocked images and this would negatively impact a multimedia
campaign. In the scope of this research there is not enough evidence to form a conclusion as
to whether the choice of e-mail provider directly affects campaign metrics.

- 44 -
Figure 10: Q4: Frequency respondents check e-mail

10. 4. 6. Frequency respondents check e-mail analysis

These results show that checking e-mail everyday has been selected by close to 100% of the
sample. The result is close enough to be classified as ubiquitous and accordingly it be
assumed that the presence of an e-mail will be detected within one day of its delivery. We
can also confidently predict that this factor will not have had an influence on the
effectiveness of the campaign.

- 45 -
Figure 11: Q5: Respondents preference of e-mail format

10. 4. 7. E-mail format analysis

Table 8: Survey - Multimedia vs Text Chi Squared

- 46 -
The Chi Squared table above uses the numbers of respondents from the questionnaire who
selected a preference for either multimedia (HTML) e-mail and text-only. The expected
numbers are, in this case, derived from the null hypothesis, assuming no difference between
the two formats of e-mail. The Chi squared value of 4.76 is very high signifying a 95-99%
probability that this is a significant result. It is interesting that this result is so distinctive from
the "total opens" numbers recorded in the experiment. If this result, on its own, is linked back
to the second research aim (8.2), we can see that there is a disparity. The sample claim to
prefer multimedia by a ratio of 3:1, however this is not born out in the total opens for each
format. The ratio is however almost identical to the "click through" numbers observed in the
experiment (Figure 12) which suggests a strong positive correlation between the two.

Figure 12: Click-through compared to Preferred Format

The previous section shows that there is a significant difference between recipients preferred
format of e-mail and the null-hypothesis that states there will be no difference in
effectiveness between the two. In order to fulfil the research aims (8.2) it is necessary to

- 47 -
repeat the Chi Squared analysis using estimated figures calculated from the numbers
observed in the experiment. In this way we can observe whether there is a significant
difference between the actions and attitudes of the sample group. Table 9 uses expected
numbers derived from the ratio of unique opens observed in the experiment.

Table 9: Survey - Revised Multimedia vs Text Chi Squared

We can see that there is still a difference between the observed numbers and the revised
expected numbers though not as evident. A Chi squared value of 2.42 suggests a 75-90%
probability that the result is significant.

- 48 -
Figure 13: Q5a: Reasons for choosing multimedia e-mail

10. 4. 8. Reasons for choosing multimedia e-mail analysis

The responses to this follow up question are distributed relatively evenly across the answers.
Taken in context with the recipients preferred choice of e-mail format these figures suggest
that Chittenden & Rettie (2002) were right to propose that "the more colourful and attractive
e-mails generate greater response." This factor could have been the driver behind the click-
through effectiveness of the multimedia campaign being dramatically higher than that for the
text-only message.

- 49 -
Figure 14: Q5b: Reasons for choosing text-only e-mail

10. 4. 9. Reasons for choosing text-only e-mail analysis

The number of responses recorded for a preference for text-only e-mail is too low to
extrapolate any significant findings from this follow up question. Of the answers received the
predominant reason for choosing text-only e-mail was to not be distracted by the colour and
images contained in a multimedia approach. This is perhaps due to the quantity of e-mails
received each week, 54 - 93 on average (Brondmo, 2000), limiting the time some users fell
they can spend reading each one. However, to reiterate the earlier point, there are not enough
responses to make confident assertions and the above analysis is conjecture only.

- 50 -
Figure 15: Q6: Respondent's preferred hyper-link format

10. 4. 10. Respondent's preferred hyper-link format analysis

The results show that respondents were split between preferring links to be shown in full, and
embedded in a word or phrase. These results are slightly contradictory to both the
observations taken in the experiment and the responses given thus far in the questionnaire.
One would expect the results for "embedded in a word or phrase" and "embedded in an
image" to be significantly greater than "link shown in full" which would then correlate with
the preference shown by respondents to multimedia e-mail and the higher click-through
numbers observed in the experiment. This is not the case suggesting one of two possible
causes. Firstly that respondents did not fully understand the question signifying an
inadequacy in the piloting of the questionnaire. Or, the second possibility is that there is a
difference between the sample groups actions and their attitude towards this question. Further
research would be needed to fully assess the influence the format of a hyperlink has on
effectiveness as it falls just outside of the scope of this study.

- 51 -
Figure 16: Q7: Respondent's preferred frequency for e-mail communications

10. 4. 11. Preferred frequency for e-mail communications

analysis

The results show a clear preference in the sample group for monthly e-mail communications.
Jail currently send information to customers on a monthly basis but only if there is anything
new to say. The campaign sent preceding the experiment was delivered in October 2006
which left approximately 4 months between messages. It is possible this may have influenced
the campaign effectiveness, however it is unlikely. Response rates tend to drop due to a high
frequency of e-mail messages which can annoy recipients (Chaffey, 2006). This is definitely
an area that, given the time and resources, would be beneficial to study more closely. E-mail
is so easy to filter and delete that any actions leading to the disenfranchisement of customers
could be costly.

- 52 -
Figure 17: Q8: Behaviour towards unwanted commercial e-mail communication

10. 4. 12. Behaviour towards unwanted commercial e-mail

communication analysis

This last question, and the follow-up (10.4.13) where applicable, primarily relate to the
formatting of future JailMail campaigns. There is one important observation that can be made
that has the potential to impact performance metrics. Over 50% of respondents did not select
unsubscribe as the option they would take if they no longer wished to receive messages. This
is significant because each would classify as a delivered message even though it were
unlikely to be read, and thus would lower the open rate accordingly. If we plot the figures for
the two campaigns (open % from delivered) next to the projected unsubscribe rate from the
table above there is a significant relationship between the two, in particular with the
multimedia campaign.

- 53 -
Figure 18: Open & Unsubscribe Percentages comparison

It is not in the remit of this study to explore this in depth although some indicators may be
found in the next section, these findings should be observed as stimulation for research
focusing solely on this factor.

- 54 -
Figure 19: Q8a: The reasons respondents gave for not choosing to unsubscribe

10. 4. 13. The reasons respondents gave for not choosing to

unsubscribe analysis

The final question in the survey was a follow up to the previous question for those people
who chose one of the option that was not "unsubscribe." The even spread of answers suggests
that there is not a unanimous reason why people choose not to unsubscribe given the choice.
The answers given by the respondents in this section, could, combined with the data from the
last section be used as a starting point for more focused research.

- 55 -
10. 5. Survey Analysis

10. 5. 1. Comparisons

The key question in the survey that relates to the hypothesis is the question that asks the
respondent "which format e-mail communications they would prefer to receive." The
questionnaire research method provided extra variables that can be used to analyse this
question in more depth. This section will compare two sets of data at a time in an attempt to
discover trends or draw conclusions that the previous analysis has been unable to provide.

a. Gender Vs. E-Mail Format

Table 10: Gender vs Preferred E-Mail Format

- 56 -
The table above compares gender to preferred e-mail format. It was noted in section 10.4.2
that there is a gender bias in the sample. The table shows that 72.73% of respondents were
male. The table also shows no significant difference between the genders with regards to e-
mail format preference. A higher percentage of females preferred multimedia, but the sample
size is too small to infer anything of value from this. The male sample is larger and the 3:1
ratio in favour of multimedia strongly suggests that males prefer the format, however, this
cannot be assumed for females. As noted in the Research Methods chapter (9.3.2) the
inherent flaw in the research is the sample size used for the questionnaire, and in order to get
an accurate finding for females further research is required with a significantly larger sample
size. With this limitation in mind, it is still valuable to visualise the data, clearly showing a
correlation between the two genders responses.

Figure 20: Gender vs Preferred E-Mail Format

- 57 -
 b. Age Vs. E-Mail Format

This section will assess what effect, if any, age has upon e-mail format preference.

Table 11: Age vs Preferred E-Mail Format

The most significant finding that can be drawn from the table is that 68.18% of the
respondents were aged between 18 and 27. Delving deeper into the data reveals that out of
this age group 80% prefer multimedia e-mail. These two results observed side by side suggest
that age is a factor in determining e-mail format preference, and whilst conjecture, is likely to
have been a significant factor in the results observed in the experiment.

- 58 -
Figure 21: Age vs Preferred E-Mail Format % by Age

Figure 22: Age vs Preferred E-Mail Format Frequency by Age

- 59 -
Figure 21 illustrates the percentages of each group that selected a preference for each format
of e-mail. These results should be viewed with caution, and in relation to Figure 22 which
shows the number of respondents who answered in each section. The three columns denoting
100% are misleading due to only having 1 or 2 respondents in each. The sample list used, and
the small list are the likely causes of this anomaly. Figure 22 clearly shows the correlation
between the age ranges and the format preferred. Jail Clothing's links to extreme sports is
likely to be the cause of the age bias in this research, however, there is room for additional
research in this area. It is possible that older age groups view the e-mail paradigm in a
different way to the younger audiences, and this would likely be an influencing factor in e-
mail effectiveness.

- 60 -
11. Conclusions

11. 1. Key Conclusions

11. 1. 1. Experiment

This section details the conclusions that can be drawn from the research experiment. They are
presented in such a way to build a convincing argument with respect to the null hypothesis
outlined in the Research Questions section (8.3) as the following;

"There is no difference in effectiveness between HTML and text-only email".

a. Open Rate

The "unique opens" observed for each format of e-mail appear to clearly favour the HTML
campaign, a result that, viewed alone, leads to the conclusion that the null hypothesis maybe
invalid. Further, statistical analysis using the Chi Squared method revealed a probability of
less than 70% that this finding was significant and so cast some doubt on the previous
finding. There is a high probability that the findings were obtained through chance instead of
indicating a trend, and therefore this study suggests that the data from "unique opens" is not
conclusive enough to disprove the hypothesis, but does suggest there may be some grounds
for this to occur.

The next conclusion that can be made is extrapolated from the "total e-mail opens" metric.
Unsurprisingly the figures show a strong correlation to the "unique opens" figures indicating
a significant trend in favour of HTML e-mail. The statistical analysis for these numbers

- 61 -
produced a 90-95% probability that the result is significant and so we can confidently assert
that the "total opens" figures give a strong indication that HTML is more effective. If this
assertion is assessed along side the conclusion outlined in the preceding paragraph then there
is now plausible evidence in place to conclude that HTML is indeed more effective rendering
the null hypothesis invalid.

There is an inherent problem with both conclusions drawn so far, they both rely on open
rates. Whilst being useful for measuring trends in a campaign, Mathew Patterson (2007)
from Campaign Monitor warns that "you should never take your open rate as a hard and fast
number, because you can never know the true figure." Accordingly, while the conclusions
drawn to date are valid, detached from subsequent findings they are not reliable enough to be
used to address the null hypothesis.

b. Click-through Rate

Click-through as a measure of effectiveness is significantly more accurate because a recipient

has to open, digest, and respond to an e-mail for a measurement to take place. The figures
used to analyse this factor were the number of unique recipients who clicked on a minimum
of one link. This method therefore produced results that showed for how many unique
recipients the e-mail campaigns had effected a response. The click-through numbers recorded
are compelling in their own right. 15 unique clicks to 5, or a 3:1 ratio in favour of HTML
formatted e-mail. This result is supported by the findings from a DoubleClick (2005) report
that found that the click-to-open ratio for HTML had increased from 26.5% to 32%. In
context "a higher click-to-open ratio suggests that the content of an email is effective in
stimulating click-through once an email has been opened with HTML the format that can
facilitate this." In support of this, Chittenden and Rettie (2002) found that "more colourful
and attractive e-mails generate greater response." Finally, when the Chi Squared value,
showing a >95% probability of significance, is factored in to the findings, this study
concludes that the results provide overwhelming evidence to render the null hypothesis
invalid. With the supporting evidence derived from the open-rate conclusions drawn above

- 62 -
the key conclusion from the experiment is that the null hypothesis has been confuted, HTML
is more effective than text-only e-mail.

11. 1. 2. Questionnaire

The second research question posed in this study related to the follow up questionnaire, and
asked whether the recipient’s attitude towards e-mail reflected the behaviour observed in the
experiment. The crucial question in the questionnaire asked the recipients to select whether
they prefer to receive multimedia or text-only e-mail (See 7.4.7). The results were definitive
and showed a strong preference for Multimedia e-mail. The Chi Squared analysis confirms
this and so this study concludes that the recipient’s attitude toward e-mail format is in accord
with their behaviour.

Finally, the results from the questionnaire showing gender and e-mail format preference were
cross-tabulated to determine if there were any correlation. This study concludes that gender
appears not to be an issue with respect e-mail format choice. Both genders favoured HTML
e-mail by a significant margin.

11. 2. Qualifying Statement

This study has concluded that HTML e-mail is more effective at eliciting a response,
regardless of recipient, than text-only e-mail. The null hypothesis proposed in the Research
Questions should be rejected and replaced with a hypothesis stating "HTML e-mail is more
effective than text-only e-mail".

- 63 -
11. 3. Limitations

Limitations anticipated prior to the research being conducted have been covered in the
Research Methodology Chapter and will not all be repeated here. However, a number of
limitations were identified during and post the research being conducted and they will be
included in this section. The first limitation was acknowledged earlier in the study, and it is
an important one. The sample size for questionnaire was very low and as a consequence this
factor is likely to have caused gender bias and age bias in the results.

The second limitation is a result of the constitution of recipients in the JailMail list that was
used for the sample. The market sector the company operates in attracts a certain
demographic of customer and this is likely to have created bias in the resulting findings.

- 64 -
11. 4. Further Research

There is a multitude of variables that could be used as the basis for further research, with
many highlighted in the questionnaire findings. The following table outlines a number of
areas where, given the time and resources, further research could have been used to obtain
more conclusive findings.

Table 12: Further Research Possibilities

- 65 -
12. Appendices
Appendix 1: JailMail Multimedia Format

- 66 -
Appendix 2: JailMail Text-only Format

- 67 -
Appendix 3: JailMail Questionnaire

http://www.questionpro.com/akira/frame.do?mode=editsurvey 03/08/2007 07:27 PM

Questions marked with a * are required

Hi,
You are invited to participate in our survey for JailMail about your email usage.
Approximately 150 of you will be asked to complete this survey to help Jail improve their
email marketing. It shouldn't take any longer than five minutes to complete the
questionnaire. And, as a big thankyou to those who complete the survey, you will be
entered into a draw to win some fabulous Jail Clothing goodies.

Taking part in this survey is completely voluntary, however, if you feel uncomfortable
answering any questions, you can withdraw from the survey at any point. Your opinions
will help us to make JailMail the best it can be, so we would really appreciate it if you can
finish the survey.

Don't worry, the responses you give and your information will all be kept strictly
confidential and will be coded to remain private. If you have questions at any time about
the survey or the procedures, you may contact Owen@JailMail by email at the email
address specified below.

Thank you very much for your time and support. Please start with the survey now by
clicking on the Continue button below.

What is your gender? *

Male
Female

How old are you? *

Under 18
18-22
23-27
28-32
33-37
38 and over

How do you receive email? (select all that apply) *

Browser based (Hotmail, Yahoo etc.)
Email Client (Outlook, Thunderbird, etc.)

http://www.questionpro.com/akira/frame.do?mode=editsurvey Page 1 of 4

- 68 -
http://www.questionpro.com/akira/frame.do?mode=editsurvey 03/08/2007 07:27 PM

Mobile Device (Blackberry etc.)

Other (please specify)

You selected that you use a browser-based email service, which do you use? (select all
that apply) *
Hotmail
Yahoo
Gmail
AOL

Other

How often do you you check your email? *

Everyday
2-3 times a week
Weekly
Bi-weekly
Monthly

Other

There are two main formats an email newsletter can take, multimedia or text. Multimedia
is defined as "using html to display pictures, images and colours in an email". Text
emails are just that, text only.

Given the choice of one of the following options, which would you prefer to receive? *
Multimedia email
Text-only email

You selected a preference for multimedia (html) email. Would you be able to provide a
reason for your choice? Select as many options as you need. *
I like how multimedia (html) emails can be presented
I like the use of colour in emails
I like to see images and pictures in my emails
I like changes in text fonts and styles

Other

You selected a preference for text-only email, Can you provide us with some more
information about your choice? Select as many option as you need. *
My email client or reader does not render multmedia (html) correctly
I just want the "meat" of the email without the distraction of colour or images

http://www.questionpro.com/akira/frame.do?mode=editsurvey Page 2 of 4

- 69 -
http://www.questionpro.com/akira/frame.do?mode=editsurvey 03/08/2007 07:27 PM

I like to read my email off-line and don't like the broken image links
I have slow internet access and multimedia (html) emails take longer to download

Other

When you are reading an email, in which situation are you most likely to click on a
hyper-link? *
When the link is shown in full
When the link is embedded in a word or phrase
When the link is embedded in an image or picture

Other (please specify)

How often do you prefer to receive email newsletters? *

Daily
Weekly
Bi-Weekly
Monthly
Quarterly

Other (please specify)

If you no longer wish to subscribe to an email newsletter which of the following are you
most likely to do? *
Unsubscribe
Bounce the email
Delete without reading
Filter in Junk/Spam

Other (please specify)

You didn't select unsubscribe as the option you would take if you no longer wanted to
receive email communications. What is the reason for this? *
It is difficult to find unsubscribe links in most emails
I don't trust the unsubscribe links given, they can lead to more spam
It takes less effort just to filter or block an email

Other

Is there anything you would like to add about email marketing that has not been
covered by the questionnaire? Please use the box below to inform us.

http://www.questionpro.com/akira/frame.do?mode=editsurvey Page 3 of 4

- 70 -
http://www.questionpro.com/akira/frame.do?mode=editsurvey 03/08/2007 07:27 PM

Please contact owen@jailclothing.com if you have any questions regarding this survey.

Privacy |
Security
Surveys | Email Marketing | Web Polls

http://www.questionpro.com/akira/frame.do?mode=editsurvey Page 4 of 4

- 71 -
Appendix 4: Chi Squared Test Table

Source: Adapted from (Deacon, J., n.d)

- 72 -
13. Bibliography
Benbasat, I.G. (1984) An Analysis of Research Methodologies, in: McFarlan, F.W. (ed.) The
Information Systems Research Challenge, Harvard Business School Press, Boston, MA,
47-85.

Pervan, G.P. (1994b) A Case for More Case Study Research in Group Support Systems, TC8
AUS IFIP Conference, Bond University, Gold Coast, Qld, May 8-11, 485-496.

Bartholomew, W., 1963, Questionnaires in Recreation; Their Preparation and Use. New
York: National Recreation Association.

Walonick, D., 1993, StatPac Gold IV: Survey & Marketing Research Edition. Minneapolis,
MN: StatPac Inc.

BannerTips, 2001, Comparison between image and text dominant banners and click rate,
http://www.bannertips.com/ BT025_answers.shtml

Bartholomew, W., 1963, Questionnaires in Recreation; Their Preparation and Use. New
York: National Recreation Association.

Benbasat, I.G. (1984) An Analysis of Research Methodologies, in: McFarlan, F.W. (ed.) The
Information Systems Research Challenge, Harvard Business School Press, Boston, MA,
47-85.

Brondmo, H.P. (2001) The Eng@ged Customer: The New Rules of Internet Direct
Marketing. Piatkus Books,

Chaffey, D., 2006, Total E-mail Marketing, Second Edition: Maximizing your results from
integrated e-marketing (Emarketing Essentials), Butterworth-Heinemann Newton, MA,
USA

Chittenden, L., and Rettie, R., 2003, An evaluation of email marketing and factors affecting
response, Journal of Targeting, Measurement and Analysis for Marketing, 11(3), 203 – 218.

- 73 -
Davison, R.M., 1998, An Action Research Perspective of Group Support Systems: How to
Improve Meetings in Hong Kong, http://www.is.cityu.edu.hk/staff/isrobert/phd/phd.htm

Deacon, J., n.d, The Really Easy Statistics Site, http://helios.bto.ed.ac.uk/bto/statistics/

table2.html#Chi%20squared%20test

DoubleClick, 2001, Online brand Effectiveness Study, http://www.doubleclick.com/us/

knowledge_central/documents/RESEARCH/dc_online_branding_0107.pdf

DoubleClick, 2005, DoubleClick EMEA Q1 2005 Email Trend Report, http:/

/emea.doubleclick.com/WEB_ADMIN/documents/
dc_Q1_05%20EMEA%20email%20trend%20report.pdf

DoubleClick, 2005, DoubleClick EMEA Q2 2005 Email Trend Report, http:/

/emea.doubleclick.com/WEB_ADMIN/documents/
dc_Q2_05%20EMEA%20email%20trend%20report.pdf

Electronic Telegraph/Ogilvy & Mather, 1997, How Internet Advertising Works, http:/
/www.intelliquest.com/resources/ whitepapers/interadvert.pdf

Emailcenter, 2004, Can your customers read your email newsletters?, http:/
/www.emailcenteruk.com/maxenews/ html_email_client_compatability.pdf

eMarketer, 2001, Response Rates of US E-Mail Marketing Campaigns, by Campaign Format,

2001 (as a % of respondents), http://www.emarketer.com

eMarketer, 2001, US Internet User Attitudes toward Commercial E-Mail Formats, 2001 (as a
% of respondents), http://www.emarketer.com

eMarketer, 2002, Consumers’ Preferred E-Mail Advertisement Formats Worldwide, 2002 (as
a % of respondents receiving commercial e-mail), http://www.emarketer.com

Eyetrack III, 2004, Recall of Information Presented in Text vs. Multimedia Format, http:/
/www.poynterextra.org/eyetrack2004/multimediarecall.htm

- 74 -
Galliers, R.D. (1991) Choosing Information Systems Research Approaches, in: Galliers, R.D.
(Ed) (1992) Information Systems Research: Issues, Methods and Practical Guidelines, Alfred
Waller: Henley-on-Thames, 144-162.

Global Reach, 2004, Evolution of Online Linguistic Populations, http://global-reach.biz/

globstats/evol.html

Griffiths, R.T., 2002, Chapter Three: History of Electronic Mail, http:/

/www.let.leidenuniv.nl/history/ivh/chap3.htm

Hoffman, D. I., 2000, The Revolution Will Not Be Televised: Introduction to the Special
Issue on Marketing Science and the Internet, Marketing Science, Vol. 19, No. 1

IPT, 2004, IPT Email Marketing Survey 2004, http://www.ipt-ltd.co.uk/downloads/press/

04_10_13_IPT_Research.pdf

Jackson, A. and DeCormier, R., 1999, ‘E-mail survey response rates: Targeting increases
response’, Journal of Marketing Intelligence and Planning, Vol. 17, No. 3, pp. 135–139.

Jennings. J., 2004, The E-Mail Text Vs. HTML Debate: Readers Weigh In, http:/
/www.clickz.com/showPage.html?page=3448841

Jupiter Research, 2005, E-mail Marketing Content Best Practices: Identifying the Impact of
Content on Response Behavior, http://www.jupiterresearch.com

Levin, W.C., 1988, Sociological Ideas: Concepts and Applications, Wadsworth: California.

Maloy, P., 2004, The E-Mail Text Vs. HTML Debate: Readers Weigh In, http:/
/www.clickz.com/showPage.html?page=3448841

Marketing Sherpa, 2003, Surprising Email Test Results: Text vs HTML Survey Versions,
http://www.marketingsherpa.com/sample.cfm?contentID=2440#

Messaging Online, 2000, Year End 2000 Mailbox Report, http://www.messagingonline.com

- 75 -
Mintel, 2005, Extreme Sports - UK, http://academic.mintel.com/sinatra/academic/
search_results/show&&type=RCItem&page=0&noaccess_page=0/display/id=114733/
display/id=185037&section?select_section=185040

O'Leary, T., 2006, HTML vs. Text: The Debate is Over, http://www.group-mail.com/asp/
common/articles.asp?id=191

Park, M.H., 2002, A Study of Effective Web Advertising Design to Maximize Click-Through
and Brand Awareness, http://www.idemployee.id.tue.nl/g.w.m.rauterberg/conferences/
CD_doNotOpen/ADC/final_paper/130.pdf

Partner, L., 2003, Why Readers Prefer Text or HTML, http://www.marketingprofs.com/3/

partner3.asp

Patterson, M., 2007, All about email open rates, http://www.campaignmonitor.com/blog/

archives/2007/01/all_about_email_open_rates.html

Peppers, D. and Rogers, M., 2000, ‘Email marketing maximized’, Peppers and Rodgers
Group, Stamford, CA.

Pervan, G.P. (1994b) A Case for More Case Study Research in Group Support Systems, TC8
AUS IFIP Conference, Bond University, Gold Coast, Qld, May 8-11, 485-496.

Pew Internet, 2005, How Women and Men Use the Internet, http://www.pewinternet.org/
pdfs/PIP_Women_and_Men_online.pdf

Radicati Group, 2006, The Radicati Group, Inc. Releases Q4 2005 Market Numbers Update,
http://www.radicati.com/uploaded_files/news/Q4-2005_PressRelease.pdf

silverPOP, 2005, SilverPOP's "2005 Broken Link Study", http://www.silverpop.com/

downloads/BrokenLink_Study.pdf

Skaper, P., 2004, The E-Mail Text Vs. HTML Debate: Readers Weigh In, http:/
/www.clickz.com/showPage.html?page=3448841

Walonick, D., 1993, StatPac Gold IV: Survey & Marketing Research Edition. Minneapolis,
MN: StatPac Inc.

- 76 -
Windham, L.,2000, ‘The soul of the new consumer’, Allworth Press, New York, NY.

- 77 -