Twitter Basic Training

Evaluation Plan
for

ACME Chem-Solutions, Inc.

by

Designed for Learning

July 2013

Twitter Basic Training, Evaluation Plan

Contents
Contents .....................................................................................................................................2
Introduction: ...........................................................................................................................3
Evaluation Process: ..................................................................................................................3
Formative Evaluation: ..........................................................................................................3
Summative Evaluation: ........................................................................................................3
Confirmative Evaluation: ......................................................................................................4
Alignment to the Five Levels of Evaluation:..............................................................................4
Formative Evaluation: ..........................................................................................................4
Summative Evaluation: ........................................................................................................4
Confirmative Evaluation: ......................................................................................................5
Alignment of Unit Goals to the Evaluation Process: .................................................................5
Evaluation Tools and Materials: ...............................................................................................5
Summary of Modifications: ......................................................................................................5
Reference List: .........................................................................................................................6
Twitter Basic Training Pre-Test ....................................................................................................8
Twitter Basic Post-Test ..............................................................................................................13
Twitter Basic Training Satisfaction Survey .................................................................................18
Twitter Basic Training Learning Transfer Evaluation ..................................................................20

2|Page

Twitter Basic Training, Evaluation Plan

Introduction:
This evaluation plan contains the processes for the formative, summative, and confirmative
evaluations that should be implemented for ACME, CHEM-Solutions, Inc. before, during, and
after the Twitter instruction. Kirkpatrick Level 1 through 4, and Phillips Level 5 evaluation
instruments and an evaluation checklist will be utilized to conduct these evaluations (Simonson,
2007). Both pilot-course and final implementation evaluations are included in this plan.
Evaluation instruments Level 1 through 3 are included as Appendices in this evaluation plan.
They will be implemented online; therefore, they are not included in printed format. These will
be developed in COURSEsites by Blackboard using the Evaluation Plan Appendices.

Evaluation Process:
Formative Evaluation:
Step 1: A Formative Evaluation Checklist (Appendix A) will be completed before, during, and
after implementation independently by each member of the design and development team
(Morrison, Ross, Kalman, & Kemp, 2011). This checklist helps to ensure quality and alignment of
the course content and activities with the learning goal and learning objectives.
Step 2: A pilot-course will be conducted prior to final implementation. A Level 2 pre-test
(Appendix B) will be administered prior to the start of the pilot-course to establish current
knowledge of Twitter. This will provide a baseline for assessment of the learning outcome
(Simonson, 2007).
Step 3: A Level 2 post-test (Appendix C) will be given again at the end of the pilot course to
assess whether learning has occurred (Simonson, 2007). If the scores received are below 70%
on average, an analysis of the cause will be conducted. Suggested analyses are frequency
distributions, frequency graphs or histograms, and descriptive statistics, i.e. percentages,
means, and medians (Morrison, Ross, Kalman, & Kemp, 2011). Relevant course corrections will
be made.
Step 4: Learners will be asked to take notes and provide feedback about any difficulties or
concerns during the pilot. Their notes will be given to the project manager. After combined
feedback analysis, the D4L project manager will address and note any concerns for inclusion in
the list of potential revisions. Course corrections or revisions may need.

Summative Evaluation:
Step 1: Prior to the final course implementation, a Level 2 pre-test (Appendix B) will be
administered to the learners to establish their current knowledge of Twitter. As in the pilot
course, this will provide a baseline for assessment of the learning outcome (Simonson, 2007).
Step 2: A Level 2 post-test (Appendix C) will be given again at the end of the course to assess
whether learning has occurred (Simonson, 2007). If the scores received are below 70% on

3|Page

Twitter Basic Training, Evaluation Plan

average, an analysis of the cause will be conducted. Suggested analyses are frequency
distributions, frequency graphs or histograms, and descriptive statistics, i.e. percentages,
means, and medians (Morrison, Ross, Kalman, & Kemp, 2011). Course corrections or revisions
may need to be made depending on the cause of the low scores.
Step 3: At the conclusion of the training, learners will complete a Level 1 Satisfaction Survey
(Appendix D) to measure their attitude and satisfaction overall with the instructional
experience (Simonson, 2007). After all surveys have been collected, an analysis of the results
and learner comments will be conducted. Appropriate course revisions will be made before
further implementing the course.

Confirmative Evaluation:
Step 1: Level 3 Learning Transfer Evaluations (Appendix E) will be implemented at 6 months
following Twitter instruction. This evaluation will assess whether transfer of learning has
occurred in the workplace and the service has been successful as intended by the company for
communication and business development. Further instruction, appropriate information
bulletins, or establishment of company policies may be required at this point to further
describe and promote the use of Twitter at ACME, Chem-Solutions, Inc.
Step 2: A Level 4 (Results) evaluation is recommended at 6 months to 1 year after the Twitter
instruction to assess whether the training and use of Twitter is producing the expected results
for in-company communication. Evaluation instruments have not been developed for this
evaluation. D4L will be available for further consulting and development at the request of
ACME, Chem-Solutions, Inc.
Step 3: A Level 5 (Return on Investment) evaluation is recommended at 1 year to determine if
the instruction and use of Twitter is producing the sales and revenue that was expected.
Evaluation instruments have not been developed for this evaluation. D4L will be available for
further consulting and development at the request of ACME, Chem-Solutions, Inc.

Alignment to the Five Levels of Evaluation:

Formative Evaluation:
The formative evaluation steps are aligned to Kirkpatrick Level 2, pre- and post-tests to
measure the extent to which learners have advanced in skills, knowledge, and attitude
(Simonson, 2007). They will be questioned about specific aspects of the instruction.

Summative Evaluation:
The summative evaluation steps are aligned to Kirkpatrick Level 1 and 2. Level 1 measures how
the learners feel about the educational activity (Simonson, 2007). They will be asked what they
did and did not like about the instruction. Level 2 evaluations will measure the extent to which
learners have advanced in skills, knowledge, and attitude (Simonson, 2007). They will be
questioned about specific aspects of the instruction in the final implementation just as they
were in the pilot-course.

4|Page

Twitter Basic Training, Evaluation Plan

Confirmative Evaluation:
Confirmative evaluation steps are aligned with Kirkpatrick Levels 3, 4 and 5. The level 3
evaluations will assess how well learning has been transferred to job performance (Simonson,
2007). Level 4 evaluations will assess whether the instruction has resulted in an increase in
communication across the company (Simonson, 2007). Level 5 evaluations will assess whether
there has been an adequate return on investment (Simonson, 2007).

Alignment of Unit Goals to the Evaluation Process:

Goal 1: Create a personal micro-blogging account with Twitter.

Goal 2: Send messages or postings using Twitter.
Goal 3: Follow and unfollow other users on Twitter.

All three goals will be evaluated at Level 1 in the summative evaluation. All goals will be
evaluated at Level 2 in the formative and summative evaluations. Additionally, they will be
evaluated by Level 3 evaluations in the confirmative evaluation. Levels 4 and 5 evaluations were
recommended at 6 months and 1 year respectively for this instruction, but they were not
developed.

Evaluation Tools and Materials:

This evaluation plan contains four evaluation tools: Formative Evaluation Checklist, Pre-Test,
Post-Test, and Learning Transfer Evaluation. The formative evaluation checklist was developed
from experience as a checklist to ensure quality and alignment of the course content and
activities with the learning goal and learning objectives. Pre- and post-tests were developed
directly from the learning objectives to ensure each was properly assessed. The learning
transfer evaluation was developed from the topic goals and overall course goal which is
educate employees about the use of Twitter on the topics of account signup, how to tweet, and
how to follow other Twitter users.

Summary of Modifications:

Included a pilot-course as part of the formative assessment with a small user group to
ensure any glitches in the online training were corrected before implementing the
instruction fully

Included the Level 1 survey at the end of the summative assessment after
implementation to assess how the learners felt about the training. Feedback will be
used to assess learner attitudes, indicate corrections, and increase user satisfaction.

Removed any references to instructors facilitating the instruction since the instruction is
online only

5|Page

Twitter Basic Training, Evaluation Plan

Removed any references to direct observation of instructions since the instruction is

online only

Reference List:
D4L. (2013). Instructional Plan for Twitter 101 Course. Retrieved from
http://d4l.wikispaces.com/Implementation+Document+%28Final%29
D4L. (2013). Implementation Plan for ACME Chem-Solutions, Inc. Retrieved from
https://class.waldenu.edu/@@/3209B6D6B60DA305BA7128E685B9864F/courses/1/US
W1.41500.201360/attempt/_11243219_1/s/Wk3ProjWrightC.pdf
Morrison, G. R., Ross, S. M., Kalman, H. K., & Kemp, J. E. (2011). Designing effective instruction
(6th ed.). Hoboken, NJ: John Wiley & Sons, Inc.
Simonson, M. (2007). Evaluation and distance education: Five steps. Quarterly Review of
Distance Education. 8(3), 191194.

6|Page

Appendix A
Twitter Basic Training Formative Evaluation Checklist
Formative evaluations include steps and checks required to ensure the course is designed and
developed according to the learning need and with the highest quality. Complete the specific
checks below before, during, and after implementation. Write any comments or
recommendations on the back. Include the evaluation context, your name, date, and signature
at the bottom of this checklist. Give the completed checklist to your project manager.
Compiled job tasks fully define the job
Identified learning gaps are supported by at least one objective
Training fully supports the actual job tasks
Course Goal fully describes the training
TLOs fully support and fulfill the Course Goal
TLO verbs are at the proper learning level in accordance with Blooms Taxonomy
TLOs are properly sequenced
ELOs adequately support their TLO
ELO verb is at the proper learning level in accordance with Blooms Taxonomy and the
TLO it supports
Outline under each ELO fully supports the ELO
Content under each outline level supports the outline level
Outline flows and is congruent from one level to the next
Outline content is at the proper learning level for the ELO it supports
Learning activities are fully supported by content, and they can stand alone without
direct instructor input
Learning activities convey the learning intended
Evaluation instruments are worded properly to measure course effectiveness
Evaluation Context (before, during, or after implementation):
Printed Name:

Date:

Signature:

7|Page

Appendix B

Twitter Basic Training Pre-Test

(Answers and Learning Objective Assignments)
Purpose: This is a pre-test learning objective measurement instrument. It provides the learning
objectives referenced by the question and the answer to the question. An actual pre-test
measurement instrument will be provided in the instructional package online.
Questions:
1. (TLO 1, ELO 1) The @ sign in Twitter is used to.
a. identify a special friend.
b. signal the end of a Tweet.
c. call out usernames in a Tweet.
d. block a user who sends offensive messages.
2. (TLO 1, ELO 1) The # sign in Twitter is used to.
a. call out usernames in a Tweet.
b. find related topics within Twitter.
c. search the internet for a related term.
d. block a user who sends offensive messages.
3. (TLO 1, ELO 1) Hashtags can be used to.
a. find trending topics.
b. identify yourself in a Tweet.
c. relate your Tweet to a friends Tweet.
d. search the internet for a related term.
4. (TLO 1, ELO 1) Usernames in a Twitter message are identified by a. symbol.
a. ?
b. @
c. #
d. %
5. (TLO 1, ELO 1) Twitter recommends no more than . hashtags per Tweet.
a. 8
b. 6
c. 4

8|Page

Appendix B
d. 2
6. (TLO 1, ELO 1) What does the term direct message or DM refer to?
a. Sending a public message to a user on Twitter
b. Sending a private message to a user on Twitter
c. Sending a public message to a user group on Twitter
d. Sending a private message to a user group on Twitter
7. (TLO 1, ELO 1) A mention is a.
a. direct messaging technique.
b. direct marketing technique.
c. means of crediting an author of an article.
d. means of including someone in your message.
8. (TLO 1, ELO 1) What does the term follow mean?
a. You will receive all of their messages.
b. You will receive all of their direct messages.
c. You will receive only of their retweeted messages.
d. You will receive only of their hashtagged messages.
9. (TLO 1, ELO 1) Usernames are also referred to as a.
a. name.
b. handle.
c. user account.
d. direct connection.
10. (TLO 1, ELO 1) What does the term retweet mean?
a. Tweeting a message tweeted by another user
b. Tweeting a message to a select group of users
c. Tweeting a private message to another user
d. Tweeting a private message to a group of users
11. (TLO 1, ELO 2) What personal information is required to sign up for Twitter?
a. Full name and Facebook account
b. Full name and email address
c. Email address and birthday
d. Email address and job title

9|Page

Appendix B
12. (TLO 1, ELO 2) What must be avoided when signing up for a Twitter account?
a. Using your real name
b. Using more than 15 characters
c. Using any offensive terms in the username
d. Using words that do not contain the word twitter
13. (TLO 2, ELO 1) When composing a tweet, it is important to remember.
a. to always hashtag any person in your message.
b. to always mention any person in your message.
c. to use a maximum of 160 characters.
d. to use a maximum of 140 characters.
14. (TLO 2, ELO 2) What is a tweet?
a. 120-character message sent on Twitter
b. 130-character message sent on Twitter
c. 140-character message sent on Twitter
d. 150-character message sent on Twitter
15. (TLO 2, ELO 2) Which of the following is NOT a valid means of sending a tweet?
a. internet
b. text message
c.

voice message

d. mobile device app

16. (TLO 2, ELO 3) Sending a message that someone already sent is called.
a. retweeting.
b. mentioning.
c. hashtagging.
d. direct messaging.
17. (TLO 2, ELO 4) Including another users handle in your tweet so that it can be clicked is
called.
a. mentioning.
b. retweeting.
c. hashtagging.
d. direct messaging.

10 | P a g e

Appendix B
18. (TLO 2, ELO 5) Sending a message to only one Twitter handle is called.
a. retweeting.
b. mentioning.
c. hashtagging.
d. direct messaging.
19. (TLO 2, ELO 6) Including a subject in your tweet that can be accessed by your followers is
called.
a. retweeting.
b. mentioning.
c. hashtagging.
d. direct messaging.
20. (TLO 3, ELO 1) What can be searched for in Twitter?
a. Topics, hashtags, tweets, or people
b. People, mentions, hashtags, or topics
c. Mentions, hashtags, topics, or direct messages
d. Hashtags, topics, tweets, or direct messages
21. (TLO 3, ELO 1) Twitter searches are primarily conducted on the . tab.
a. Home
b. @ Connect
c. # Discover
d. Me
22. (TLO 3, ELO 2) What must you do to follow a Twitter handle?
a. Click the users handle
b. Click the # Discover tab
c. Click the @ Connect tab
d. Click on the follow button
23. (TLO 3, ELO 2) When following a Twitter handle.
a. you will receive all of their followers messages.
b. you will receive only their hashtagged messages.
c. you will receive all of their retweeted messages.
d. you will receive only their direct mention messages.

11 | P a g e

Appendix B
24. (TLO 3, ELO 3) What must you do to unfollow a Twitter handle?
a. Email the user to stop
b. Select the @ connect tab
c. Go to the users account page
d. Click the follow button again
25. (TLO 3, ELO 3) Which ways of unfollowing below are NOT valid?
a. Clicking on an persons account and clicking unfollow
b. Clicking on the # Discover tab and clicking unfollow
c. Clicking on Following and clicking on unfollow
d. Sending a text message to Twitter

12 | P a g e

Appendix C

Twitter Basic Post-Test

(Answers and Learning Objective Assignments)
Purpose: This is a post-test learning objective measurement instrument. It provides the
learning objectives referenced by the question and the answer to the question. An actual posttest measurement instrument will be provided in the instructional package online.
Questions:
1. (TLO 1, ELO 1) The @ sign in Twitter is used to.
a. identify a special friend.
b. signal the end of a Tweet.
c. call out usernames in a Tweet.
d. block a user who sends offensive messages.
2. (TLO 1, ELO 1) The # sign in Twitter is used to.
a. call out usernames in a Tweet.
b. find related topics within Twitter.
c. search the internet for a related term.
d. block a user who sends offensive messages.
3. (TLO 1, ELO 1) Hashtags can be used to.
a. find trending topics.
b. identify yourself in a Tweet.
c. relate your Tweet to a friends Tweet.
d. search the internet for a related term.
4. (TLO 1, ELO 1) Usernames in a Twitter message are identified by a. symbol.
a. ?
b. @
c. #
d. %
5. (TLO 1, ELO 1) Twitter recommends no more than . hashtags per Tweet.
a. 8
b. 6
c. 4

13 | P a g e

Appendix C
d. 2
6. (TLO 1, ELO 1) What does the term direct message or DM refer to?
a. Sending a public message to a user on Twitter
b. Sending a private message to a user on Twitter
c. Sending a public message to a user group on Twitter
d. Sending a private message to a user group on Twitter
7. (TLO 1, ELO 1) A mention is a.
a. direct messaging technique.
b. direct marketing technique.
c. means of crediting an author of an article.
d. means of including someone in your message.
8. (TLO 1, ELO 1) What does the term follow mean?
a. You will receive all of their messages.
b. You will receive all of their direct messages.
c. You will receive only of their retweeted messages.
d. You will receive only of their hashtagged messages.
9. (TLO 1, ELO 1) Usernames are also referred to as a.
a. name.
b. handle.
c. user account.
d. direct connection.
10. (TLO 1, ELO 1) What does the term retweet mean?
a. Tweeting a message tweeted by another user
b. Tweeting a message to a select group of users
c. Tweeting a private message to another user
d. Tweeting a private message to a group of users
11. (TLO 1, ELO 2) What personal information is required to sign up for Twitter?
a. Full name and Facebook account
b. Full name and email address
c. Email address and birthday
d. Email address and job title

14 | P a g e

Appendix C
12. (TLO 1, ELO 2) What must be avoided when signing up for a Twitter account?
a. Using your real name
b. Using more than 15 characters
c. Using any offensive terms in the username
d. Using words that do not contain the word twitter
13. (TLO 2, ELO 1) When composing a tweet, it is important to remember.
a. to always hashtag any person in your message.
b. to always mention any person in your message.
c. to use a maximum of 160 characters.
d. to use a maximum of 140 characters.
14. (TLO 2, ELO 2) What is a tweet?
a. 120-character message sent on Twitter
b. 130-character message sent on Twitter
c. 140-character message sent on Twitter
d. 150-character message sent on Twitter
15. (TLO 2, ELO 2) Which of the following is NOT a valid means of sending a tweet?
a. internet
b. text message
c.

voice message

d. mobile device app

16. (TLO 2, ELO 3) Sending a message that someone already sent is called.
a. retweeting.
b. mentioning.
c. hashtagging.
d. direct messaging.
17. (TLO 2, ELO 4) Including another users handle in your tweet so that it can be clicked is
called.
a. mentioning.
b. retweeting.
c. hashtagging.
d. direct messaging.

15 | P a g e

Appendix C
18. (TLO 2, ELO 5) Sending a message to only one Twitter handle is called.
a. retweeting.
b. mentioning.
c. hashtagging.
d. direct messaging.
19. (TLO 2, ELO 6) Including a subject in your tweet that can be accessed by your followers is
called.
a. retweeting.
b. mentioning.
c. hashtagging.
d. direct messaging.
20. (TLO 3, ELO 1) What can be searched for in Twitter?
a. Topics, hashtags, tweets, or people
b. People, mentions, hashtags, or topics
c. Mentions, hashtags, topics, or direct messages
d. Hashtags, topics, tweets, or direct messages
21. (TLO 3, ELO 1) Twitter searches are primarily conducted on the . tab.
a. Home
b. @ Connect
c. # Discover
d. Me
22. (TLO 3, ELO 2) What must you do to follow a Twitter handle?
a. Click the users handle
b. Click the # Discover tab
c. Click the @ Connect tab
d. Click on the follow button
23. (TLO 3, ELO 2) When following a Twitter handle.
a. you will receive all of their followers messages.
b. you will receive only their hashtagged messages.
c. you will receive all of their retweeted messages.
d. you will receive only their direct mention messages.

16 | P a g e

Appendix C
24. (TLO 3, ELO 3) What must you do to unfollow a Twitter handle?
a. Email the user to stop
b. Select the @ connect tab
c. Go to the users account page
d. Click the follow button again
25. (TLO 3, ELO 3) Which ways of unfollowing below are NOT valid?
a. Clicking on an persons account and clicking unfollow
b. Clicking on the # Discover tab and clicking unfollow
c. Clicking on Following and clicking on unfollow
d. Sending a text message to Twitter

17 | P a g e

Appendix D

Twitter Basic Training Satisfaction Survey

Purpose: This is a sample of the assessment/survey that will be used to measure learner
satisfaction. The answers are indicated in Likert scale from 1 to 5 1 = disagree, 2 = slightly
disagree, 3 = undecided, 4 = agree, 5 = highly agree. An actual assessment survey will be
provided in the instructional package online.
Name:

Date:

Job Title:
How long have you been using Twitter?
Instructions: Place an X in the box that indicates your satisfaction with the idea in the question
or statement: 1 = disagree, 2 = slightly disagree, 3 = undecided, 4 = agree, 5 = highly agree. If
you rate any a 1 or 2, please include a comment as to why in the comment section at the
bottom.
Rating
#

Question or Statement

1.

I like using Twitter.

2.

The instruction was beneficial in helping me learn about Twitter.

3.

I enjoyed participating in the online instruction.

4.

The instruction was paced correctly for me.

5.

The instructional content was at the right level for me.

6.

The course overviews provided good introductions to the sections.

7.

The demonstrations or screencasts assisted with my learning.

8.

The presentations were well designed and easy to follow.

9.

The student activities such as creating an account and tweeting helped me

to learn the material.

10.

The online quizzes helped me remember some of the key terms and
concepts.

11.

The optional learning activities, such as Tour of Twitter, were beneficial in

giving me a better understanding of Twitter.

12.

The course reviews adequately recapped the section and helped me

remember key learning points.

13.

The tests were well written and measured the learning objectives.

18 | P a g e

Appendix D
14.

I believe the training will help me become an avid Twitter user.

15.

I plan on using Twitter for personal as well as business purposes.

Comments: (Please include comments below for any 1 or 2 ratings.)

19 | P a g e

Appendix E

Twitter Basic Training Learning Transfer

Evaluation
Purpose: This is a sample of the assessment/survey that will be used to measure learner
transfer of knowledge to the job at 6 months after the Twitter instruction. The answers are
indicated in Likert scale from 1 to 5 1 = disagree, 2 = slightly disagree, 3 = undecided, 4 = agree,
5 = highly agree. An actual assessment survey will be provided in the instructional package
online.
Name:

Date:

Job Title:
How long have you been using Twitter?
Instructions: Place an X in the box that indicates your satisfaction with the idea in the question
or statement: 1 = disagree, 2 = slightly disagree, 3 = undecided, 4 = agree, 5 = highly agree. If
you rate any a 1 or 2, please include a comment as to why in the comment section at the
bottom.
Rating
#

Question or Statement

1.

I like using Twitter.

2.

I am involved with business development in my company.

3.

I tweet at least 2 times a daily.

4.

I check my Twitter account 2 or more times daily.

5.

I am following more than 10 people.

6.

I use Twitter on a regular basis to communicate with friends.

7.

I use the direct message function of Twitter.

8.

I use Twitter to send company product updates.

9.

I use Twitter for advertising company products.

10.

I am following the company and leadership on Twitter.

11.

I use Twitter to communicate with others in the company about important

updates related to my job function.

12.

Twitter provides me with relevant information.

20 | P a g e

Appendix E
13.

Twitter often points me to the right places to find answers.

14.

Twitter has enhanced my social life by linking me with other employees I

would not normally know.

15.

Twitter has enhanced the team aspect of our company by linking me with
other employees I would not normally know.

16.

Twitter has provided more of a family feeling in our company.

17.

I receive company product updates frequently.

18.

I have witnessed increased sales due to the use of Twitter.

19.

I have received purchase requests due to the use of Twitter.

20.

I am an advocate of using Twitter for communication.

21.

I am an advocate of using Twitter for business development.

Comments: (Please include comments below for any 1 or 2 ratings.)

21 | P a g e