Module Evaluation 1

Running head: INSTRUCTIONAL MODULE EVALUATION

Evaluation of Reading Nutrition Labels Module

Tara Peters
Carrie Sebora
Mary Pat Vargas
California State University Monterey Bay

IST622 Assessment and Evaluation

Dr. Su
July 28, 2015

Module Evaluation 2

Table of Contents
I. Introduction ---------------------------------------------------------------------------------------- 3
II. Methodology -------------------------------------------------------------------------------------- 3
a. Prototype ------------------------------------------------------------------------------------- 3
b. Learners -------------------------------------------------------------------------------------- 5
c. Tryout Process ------------------------------------------------------------------------------- 5
1) Pre- & Post-Test ---------------------------------------------------------------------- 5
2) Observations --------------------------------------------------------------------------- 5
3) Class Discussion ---------------------------------------------------------------------- 6
d. Tryout Conditions --------------------------------------------------------------------------- 6
III. Results -------------------------------------------------------------------------------------------- 6
a. Entry conditions ----------------------------------------------------------------------------- 6
b. Instruction ------------------------------------------------------------------------------------ 7
c. Outcomes ------------------------------------------------------------------------------------- 7
d. Recommendations -------------------------------------------------------------------------- 9
IV. Summary and Conclusion -------------------------------------------------------------------- 11
V. Summary and Conclusion --------------------------------------------------------------------- 12
VI. Appendices ------------------------------------------------------------------------------------- 14
a. California State Health Standards ------------------------------------------------------- 14
b. Observation Checklist -------------------------------------------------------------------- 15
c. Discussion & Response Summary ------------------------------------------------------ 16
d. Pre- and Post-test result charts ---------------------------------------------------------- 18
e. T-Test Paired Two Sample for Means ------------------------------------------------- 20
f. Pre- and Post-test Questions ------------------------------------------------------------ 21
g. Pre- and Post-test Questions Statistics ------------------------------------------------- 23
h. Pre- and Post-test Results ---------------------------------------------------------------- 24

Module Evaluation 3

I. Introduction
Although childhood obesity rates are declining, a serious problem remains; 17% of
children and adolescents are obese (Centers for Disease Control and Prevention, 2015). The
Reading Nutrition Labels Module seeks to empower children with the knowledge of product
ingredients so that they can make informed decisions regarding what they eat.
The California State Health Standards (appendix a) represent an additional impetus for
creating this module. In their preface to the standards, State Superintendent of Education, Jack
OConnell and State Board of Education President, Theodore Mitchell stressed the importance of
student health: Healthy children make better students, and better students become healthy,
successful adults who are productive members of their communities. (2009).

II. Methodology
a. Prototype
The prototype, Reading Nutrition Labels, used for evaluation is a preliminary
capstone project of one of the authors of this paper and the goal of the module is to
deliver education regarding the statistics and data printed on food labels. It is likely that
after-school enrichment programs will implement the use of the Reading Nutritional
Labels module with groups of children completing the module at the same time.

This

prototype was tested in a regular classroom environment at the end of the school year
when teachers typically curate interesting, yet informative lessons that may or may not
cover the assigned curriculum. This module incorporates colorful clipart, provides a

Module Evaluation 4
voiced dialog, written instructions, informative lessons and checkpoints with immediate
feedback for the learner. The module begins with a brief introduction regarding nutrition
labels. The learning outcomes are stated before the learner explores the module.

The

color-coded arrows provided allow for additional explanation and to assist the learner in
discovery. Young looking character clipart provide additional engagement for the
learner.
All learners received earbuds, although the use of them was not required, every
student noticed the sound button and they used it resulting in most learners completing
the classroom prototype module in fifteen minutes or less. While the module provides
written instructions, several learners communicated that using the earbuds assisted with
comprehending the information conveyed.
The targeted learners of the module comprised two similar sixth grade classes at
a suburban middle school. The last two weeks of school is an optimal time to have
students use the Reading Nutrition Labels prototype module because expectations of
receiving letter grades are absent, and it eliminates the stress associated with
performance. The instructor, one of the authors of this paper, explained that the module
was a prototype and that she needed input regarding information contained in the
module, ease of use and opinions regarding the appeal. The content did not relate to any
area of study (these were math students); however the students were enthusiastic to be
the first users of the module and to share their experiences. Upon conclusion of the
module (pre and post-testing), the two classes participated in discussions concerning the
importance of understanding food labels. The students reached a general agreement and
concluded that recognizing how information on a food label can assist consumers is

Module Evaluation 5
essential.
b. Learners
The recruitment of a total of fifty-two users (sixth-grade students) for the
purpose of the prototype module test, represented varied class demographics that
included English Language Learners, resource students and identified gifted learners.
Both classes had students with varied socio-economic status. None of the test subjects
announced total familiarity with food labels prior to taking the pretest.
c. Tryout Process
The process of evaluation consists of three parts; 1) pre- and post-tests to measure
learning, 2) classroom observations, 3) verbal discussions to measure reaction.
1) Pre- and post-tests: After evaluating the information presented in the module, pre
and post-test were prepared using Socrative. Socrative is a free application that
teachers can use for assessments and students may use tablets, laptops or
smartphones. With Socrative, the teacher can create assessments, receive instant
feedback and detailed reports. The application also makes it easy to track student
progress as they are taking the test. Students are familiar with this testing method,
they easily logged into the classroom account using Google Chromebooks and
they completed their pre and post-tests. The typical student did not spend more than
ten minutes taking the test. Only one student took thirty minutes. The pretest results
were not shared with the students increasing the reliability of post-test results and,
therefore, the validity of learning.
2) Observation: One mode of observation was conducted in the classroom as the
students were using the module. The educator utilized a usability checklist

Module Evaluation 6
(appendix b) as a reference while the learners experienced the module. The main
purpose of conducting an observation in this manner was to determine the usability
and appropriateness of the module in a classroom environment. No user required
instructor assistance. The instructor remained as an observer only and did not
provide encouragement, advice nor was any student questioned regarding his or her
progress.
3) Class Discussion: Upon completion of the module, the teacher questioned the class
regarding the module - specifics included ease of use, appeal, and knowledge
gained (appendix c).
d. Tryout Conditions
The module was used in a classroom environment to emulate the potential use
with groups of children. Student district Gmail accounts were used to communicate the
Internet address for the module. Login was accomplished with ease, as the students are
familiar with receiving these types of links via their school email accounts. All of the
students used the provided earbuds or headphones. This activity was the first of the class
period, and there was no time limit expressed. Students typically spent no more than
fifteen minutes on the module. The discussion regarding elements of the module took
place immediately after the posttest.

III. Results
a. Entry Conditions
The instructor did not expect or observe any entry problems.

The conditions

were good; students finished California State mandatory testing, and they were
energized to assist the instructor and to participate in an activity without an associated

Module Evaluation 7
grade. Users had never received instruction via an online teaching module, and they
expressed their willingness use the module. These sixth-grade students can be
considered digital natives and could be considered experts at maneuvering their way
around their Chromebooks.
b. Instruction
Students were given several instructions: a reminder regarding where to find the
link to the module, a review of classroom behavior, and encouragement to acquire as
much knowledge as they could regarding reading nutrition labels. Students understood
they would complete a post-test and that the expectation was to receive a better score
than he or she received on the pretest. All students completed the module without
interventions from the instructor.
c. Outcomes
Does the Reading Nutrition Labels Tutorial increase knowledge of nutrition
labels?
IN SYMBOLS

Ho

1 - 2 <= 0

H1

1 - 2 > 0

IN WORDS
The Reading Nutrition Labels Tutorial has no effect on the
students knowledge of nutritional product labels
The Reading Nutrition Labels Tutorial increases student
knowledge regarding nutritional product labels

Assumptions:
The null hypothesis: 1 - 2 <= 0
Our subjects were part of a convenience sample chosen from a Middle School; we
used two sixth grade math classes.
Sampling distributions of the difference between means (pretest/post-test) are
normal in shape; however the data is slightly skewed to the left (appendix d)

Module Evaluation 8
The groups are matched and dependent.
We used a two-tailed test and an alpha level of .05.

This research expects an

improvement in learning: Alternate Hypothesis 1 - 2 > 0

The level of reading comprehension is a concern.

This module, Reading

Nutrition Labels, should be used by students age 10 and up.

Employing a

readability formula, The Flesch-Kincaid Grade Level, a common indicator

utilized by public schools, establishes a comprehension level of the material.
An audio version of the module was available, and all students used this feature;
however, an appropriate level of word comprehension is still necessary.
Use of the audio feature of the Reading Nutrition Labels module may have
contributed to the learning of certain ELL students who were not successful at
sharing their knowledge on the pre- or post-tests.
An examination of pre- and post-test results from both classes combined (see appendix d)
indicates that learners achieved learning gains as a result of completing the Reading
Nutritional Label learning module. The students pre- and post-test results were graded
based on a nominal score out of 10 possible correct answers. A paired two-sample t-test
was performed to evaluate whether the Nutrition Label module contributed to knowledge
gained by the students.

The absolute value of t-stat was greater than the t critical value

(|t|=5.61911 > 2.00758) and the conclusion is that the instructional module resulted in
learning gains (appendix e).
The ten pre/post-test questions (appendix f) are presented, and an appraisal of
each question provides further evidence that critical learning occurred (appendix g).
example the following question: What information does the Food and Drug

For

Module Evaluation 9
Administration require to be listed first on the Nutritional Label? The combined
percentage of all students who selected the correct answer, ingredients, on the pretest was
31%.

The percent of all students who selected correctly on the post-test rose

significantly to 79%.
d. Recommendations
Post-test results range from a score of 4 to 10, a range of 6 points. The cluster of scores
towards the right of the chart is convincing concerning student learning. The established
proficiency level of 70% may not be acceptable as proficient in a true educational setting.
Also, typically an instructor may find it appropriate to weight the information of some
questions as more important than others; this process would affect the described scores.
As the module exists, sustainable success at a level higher than 70% may be doubtful. It
is also possible that the pre-test does not reflect the learning of the facts in the module in
its entirety or that the pre-test may not address all items of importance. The module
works well, and the students were interested to use it. In recognition that all work can be
improved if time and resources are available, the following recommendations could be
considered:
Begin with movie and provide a link to real-life
A story line that establishes the importance of knowing how to read nutritional labels
and how the information can be used to eat healthy will immediately capture the
interest of the user. Many students at the middle school age have never been grocery
shopping. Include a clip that demonstrates how to grocery shop and then add a clip
that demonstrates how to choose snacks. Middle school children eat snacks all the
time, and they can easily relate.

Module Evaluation 10
The About/Intro/Overview page could be more engaging.
The first page should provide text; it is not obvious that the audio button must be
pressed.
Provide real life vignettes of people and the effects of their eating habits.
A discussion of the importance of healthy eating through real life examples is
important. Consider using a variety of people: ages, occupations and ethnicities.
These people could share how they eat and how they feel. Not all examples should be
good. Examples of poor eating or overeating should be shared.
Link the scenarios to the first story in the module
The continued use of cartoon characters should be considered; students will enjoy
seeing how a real character evolves into a cartoon.
Continue the cartoon scenarios but add dialog between two characters.
The dialog between two characters draws the reader into the decision, and it becomes
more fun to participate in the module. Introducing students to the characters will
encourage them to feel a connection from the beginning.
Add a game where students match the facts to pictures
The use of a game will be engaging and if by using a time clock the students can
compare who matched the correct answers to the pictures in the shortest amount of
time.

Students love online games, and a game that is designed as a competition will

provide additional motivation to learn.

This module represents essential information for young learners. It should be
developed into a series.
This module is engaging and provides excellent links to important knowledge. The

Module Evaluation 11
health information presented in this manner is engaging, and the established
characters could appear in successive, related, nutrition modules that cover more of
the California State Health Standards.

IV. Summary and Conclusion

Observations, pre- and post-tests, and class discussions provide confirmation that learning
occurred.

The most powerful information gleaned from this study is that student users shared

that they had never learned in this manner (using a module) and that they enjoyed doing so.
This knowledge and the statistical evidence of learning should provide the impetus for educators
to develop or curate various modules related to the standards then use them to augment the
curriculum.
It is the author's observation that when grades are removed and acquiring knowledge is
expected, this eliminates some of the stress of learning. Use of this module in an educational
setting where the norm is to receive grades may provide results that are higher.

There is a

contrast between learning for the sake of learning and learning for rewards (grades); however, in
both cases, it is challenging to judge effort.
An experienced educator constructed the pre- and post-test; however, that educator did
not create the learning module.

This disconnect may have contributed to the construction of

questions that did not provide the perfect representation of the information in the module.

On

the other hand, approaching the analysis in this manner replicates what a different, potential,
virtual educator may do with his or her class.

Despite any shortcomings of the pre/post-test, it

demonstrated that the module increases student knowledge in regards to reading nutrition labels.
The method constructed for analyzing student usability was remarkably different than
testing a small group, or one learner at a time. There are limitations with administering the

Module Evaluation 12
prototype to an entire classroom of learners; it is impossible to be in more than one place at a
time, and the educator could not possibly see everything. It is entirely likely that students
problem solved without announcing their concerns or requesting assistance from the educator.

V. References

Module Evaluation 13
Childhood obesity facts | child | data | obesity | DNPAO | CDC Retrieved 7/3/2015, 2015,
from http://www.cdc.gov/obesity/data/childhood.html
Health education content standards - content standards (CA dept of education) healthstandmar08.pdf Retrieved 7/3/2015, 2015, from
http://www.cde.ca.gov/be/st/ss/documents/healthstandmar08.pdf

Module Evaluation 14
VI. Appendices
a. California State Health Standards

Module Evaluation 15
b.

Observation Checklist
Checklist
Does the learner begin the module without
assistance?
Does the learner follow the instructions
correctly and complete the module?
Does the learner need any help resolving
problems?
Does the learner use the voice feature?
Does the learner need help navigating the
module?

Yes

No

Comments

The checklist was useful for the educator; however, student use of the module did not
present any problems the educator had to address. No student needed help to navigate the
module and all students used the voice feature (the educator passed out earbuds).

Module Evaluation 16
c. Discussion & Response Summary

REFLECTIVE DISCUSSION QUESTIONS

1. Did you have any problems navigating through the module?
There were no problems. I was surprised that the speaking started on the first page and
there were no words to read.
2. Did you enjoy using the module?
It was very helpful. I feel like I learned a lot.
It was a fun way to learn about calories and serving sizes.
3. Was there information presented that you did not know previously?
I know where I can look for information to help with diabetes.
Eat everything in moderation means dont eat too much.
Serving sizes are tricky and we should pay attention to those.
4. How would you improve the module?
There was a lot of information at once.
I liked the clipart characters, although they seemed a bit young.
5. What did you like about the module?
The information is important if you want to go on a diet.
My mom already teaches me about food labels so it was nice to see things I already knew.
This is good for the Boy Scouts to use. We need to earn a cooking/merit badge. I already
have the badge so this information was familiar to me.
My mom watches the Food Network so I knew most of the stuff already.

Module Evaluation 17
d. Pre- & Post-test result charts

Pretest Score Distributions

20
18
16
14
12
10
8
6
4
2
0
0

10

If we establish 70% as marginally proficient, then pretest scores reveal that 37% of the students
tested had relevant knowledge and could be considered knowledgeable about nutritional labels.
Because the test was a multiple-choice test, a margin of error should be respected because
guessing is possible.

Interest in the test could also affect the scores; some students may not

have felt motivated to do their best. The instructor was able to view the test results as the
students responded and the students were aware of this.

Module Evaluation 18

Post-test Score Distributions

20
18
16
14
12
10
8
6
4
2
0
0

10

If we establish 70% as marginally proficient, then post-test scores reveal that 69% of the students
tested acquired relevant knowledge and could be considered knowledgeable about nutritional
labels. Because the test was a multiple-choice test, a margin of error should always be respected
because guessing is possible. The post-test was given one week after the pretest and although it
is considered unlikely the questions were remembered (there was no motivation to do so) it is
always a consideration that the students participated in the module in hopes of answering
questions on the test.

Module Evaluation 19
e. Pre- and Post-test result chart

Module Evaluation 20

f. Pre- and Post-test Questions

Module Evaluation 21

Module Evaluation 22
g. Pre- and Post-test Question Statistics

Module Evaluation 23
h. Pre- and Post-test Results
Pre + Post test scores (0-100)
Class 4

Class 6

Student
Names

Pre-Test

Post-Test

Student
Names

Pre-Test

Post-Test

Ma

50

70

Sh

60

90

Ja

30

40

Tr

64

90

Hw

30

50

Ka

80

70

Ri

60

80

Al

70

70

Br

60

80

Ju

60

70

Ad

60

80

Ke

50

80

Se

60

60

Kn

60

50

So

90

90

Lu

80

80

Le

40

70

As

50

70

El

50

70

No

40

50

Ja

60

50

Ca

60

80

Al

70

90

Gr

60

100

Ai

80

90

Sa

70

70

Rh

70

50

Ch

80

50

Be

30

60

Tri

60

70

Sh

70

70

Ki

36

60

Ju

70

90

Wi

40

60

Ca

70

50

Ma

70

70

Na

40

50

Br

60

70

Kr

80

80

Em

70

70

Module Evaluation 24
Yu

50

80

Dy

36

60

Sk

50

91

El

60

70

Mar

50

80

Am

50

70

Kyl

50

70

Ty

50

70

Ti

70

60

Qu

70

80

Class
Scoring

57.6%

70.4%

Ja

40

70

Ky

40

60

Class
Scoring

58.0%

70.4%