A & HL 4088: Second Language Assessment

A Reading and Writing Diagnostic Test:

Assessing L2 Learners in an Academic Setting

Shawn Kamp
Seth Silberner
Columbia University, Teachers College

Table of Contents
Introduction..3
REVIEW OF THE LITERATURE
The nature of academic writing..4
Theoretical definition of the construct of academic writing ability18
The nature of readin
g..................22
Theoretical definition of the construct of reading ability30
The relationhip between reading and writing ability..33
TEST CONSTRUCTION
Design Statement.....35
Operationalization......42
APPENDIX
Appendix A: The Test45
Bibliography....56

Introduction
Gregorio Luperon High School is a newcomer school located in Washington Heights. Our
students are 100% native Spanish speaking. The students are tracked into 8 levels
depending on their mastery of the English language. English language literacy, in both its
receptive and productive forms is particularly difficult for our students. Recently, the
level 6 class composed primarily of seniors, several juniors, and a single sophomore took
the English Language Arts Regents exam. Students are required to pass this test to
receive their Regents Diploma, and as most are in the process of applying to colleges and
universities their ability to succeed in this and future academic reading and writing
contexts are paramount. Yet, almost all of them failed the ELA Regents. The best way to
help students acquire requisite literacy skills to pass these exams and succeed on
University entrance exams is to ascertain their particular strengths and weaknesses, and
then design instruction to specifically target those areas of weakness. Unfortunately, we
did not have enough data concerning student strengths and weaknesses to design future
targeted instruction.
The purpose of this assessment is to provide the specific diagnostic information about
our students strengths and weaknesses in reading and writing in an academic context,
which we could then use to create lessons with targeted instruction. Given the
We have determined four research questions that will assist us in constructing a test th
at suits our specific context. The first two are of particular relevance, since we must know
what it is we are going to test if we are to create a valid assessment tool (find something f
rom B&P).These are as follows:

1) What is the nature of reading ability as measured by this test?

2) What is the nature of academic writing ability as measured by this test?
3) What is the correlation between reading and writing abilities as measured by this
test?
4)

What is the relationship between scores and the classroom teacher perception of
ability?

The first question is a reflection of our encountering Bernhardt (1991) during the early sta
ges of our research into the construct of writing. Bernhardt created a classification schem
e for types of second-language learners, which also outlined their particular language nee
ds. In terms of writing skills, for the group who we were expecting to test, minority group
members in a bilingual program, educational success calls for academic writing skills. It
was clear to us that our concern is not each and every factor associated with writing, but t
hose that will be called upon and assessed in our student future academic life. As Horowi
tz (1986, Pg. 445) tells us, without such information, creating realistic writing tasks in the
English-for-academic-purposes (EAP) classroom remains largely a matter of guesswork.
For this reason, we decided to focus our first question specifically on the construct of aca
demic writing.
In

assessing reading and writing abilities we also wanted to consider factors, such

as the interrelation between the two abilities. We believe that reading and writing, the pro
ductive and receptive aspects of literacy, are inextricably linked. In examining the two, w
e hope to better understand any causative effects in their relationship, then employ that
knowledge in constructing a test that is conscious of their reciprocal nature.
4

Our final research question concerns how one peripheral variable and its effect on
performance. We wanted to investiage the relationship between teacher perceptions of
students abilities and their actual scores on the test. In order to facilitate our answering
this question, raters will score essay tests before matching tests to test takers.

The nature of reading

The literature that we reviewed on reading tended to fall into two types: those that
explored reading from a product oriented perspective, and those that examined reading
from a process oriented perspective. When examining reading from a product
perspective, researchers are collecting data by asking students questions about texts and
evaluating the answers that those students produce. Examining with a processes
perspectiveis oriented towards examining the cognitive processes that occur during
reading. We found that the more current models tended to examine reading from the latter
perspective.
Mackworth (1970) was an early example of a process oriented perspective. His
research into the sub-systems that are employed during the reading process, however,
went beyond the cognitive.Mackworth created a model that attempted to describe the
interaction between the short- and long-term memory and the sensory input from the
visual, auditory, and motor sub-systems. He hypothesized that there are feedback loops.
For instance, visual input that occurs during a fixation pause, when we spend more time
on particular words or phrases within the text, involves: selection, attention, expectation,
and prediction. Unfortunately Mackworth is unclear on what actually occurs during each
of these
5

Lennon (1970), in a more product oriented model, examined usedwhat is

measurable in reading as the basis for his model. He recognized four overarching
components to the reading process. The first, verbal factor connotes word knowledge.
Lennon notes that extensive vocabulary mastery is a prerequisite to high competence in
reading. The second, comprehension of explicitly stated material,concerns the ability to
locate specific details and understand the literal meaning of the text. The third,
comprehension of implicitmeanings, is the outcome of reasoning in reading. (Pg. 30).
This would include drawing inferences; making predictions; deriving the meanings of
words from context; perceiving the structure of what is being read - the main idea;
noticing the hierarchy of ideas in the text; and interpreting text to find a solution or derive
a generalization, principle, or theory. The fourth component is appreciatio, sensing the
authors purpose by noticing the mood, tone and literary devices.
Smith (1985) created a process oriented model that echoed Mackworths focus on
the interaction between sensory input and memory. In Smiths this is referred to as visual
and non-visual information. The visual information is simply the text, and the decoding
of it. Smiths sums it up as what disappears when the lights go out (pg. 14). The nonvisual information is all the information that is in your head, already(pg. 14),
essentially your background knowledge. These two types of information act reciprocall;
In other words, the more prior knowledge the reader has, the easier it will be for them to
comprehend the text. Reciprocally, the less prior knowledge a reader has about the
content of the text, the more they must rely on visual information, which may include
headings, graphics, pictures, or any other piece of information that the writer has
provided to enable comprehension
6

Anderson (1991)was part of the movement that expressly sought to shift the focus
of reading research from product to process of reading. Andersons investigation sought
to outline the processes by expressly identifying the strategies that are being used to
comprehend the visual information a reader encounters. In examining data collected from
readers using think-aloud protocols, 47 processing strategies were identified. These were
then placed into 5 overarching categories: supervising strategies, those metacognitive
strategies a reader uses to keep track of their understanding of the text; support strategies,
which are used to repair misunderstandings; paraphrase strategies, which are primarily
used to understand the lexical items; strategies for establishing coherence in text, which a
reader uses to follow the organization and development of ideas in the text; and testtaking strategies. Interestingly, aside from the category of supervising strategies, all other
categories constantly rely on the aforementioned interaction between visual and nonvisual information.
Anderson also listed five aspects of the content of text that a reader seeks to
comprehend:
1)

Main ideas

2)

Textually explicit or direct statements, if both the question and the answer

are derivable from the text and if the relationship between the question and the
answer was explicitly cued by the language of the text. (pg. 48) Essentially,
textually explicit questions are those a reader can answer by finding details within
the text.
3)

Drawing inferences

4)

Textually implicit, if both the question and answer are derivable from the

text but there is no logical or grammatical cues tying the question to the answer
and the given answer is plausible in light of the question, (p.48) Textually
implicit questions require the reader to make connections between the question
and indirect information within the text, and evaluate this information in making
an inference in answering the question.
5)

Scriptally implicit, whenever a plausible non-textual response is given to

a question derivable from the text. (p.48)

One major drawback of Andersons model is that h did clearly define not
delineate all aspects of his model. We can interpret main idea and drawing inferences in a
wide variety of ways, none of which may match Andersons intended meaning. In terms
of his category, scriptally implicit, his own definition is similarly opaque.
Grabe (1991) was expressly interested in creating a process model that outlined
the nature of fluent reading, which he described as: rapid - the flow of information is fast
enough for the reader to make the connections and inference necessary for reading
comprehension; purposeful -good readers have reason for reading; interactive - the
readers prior knowledge, along with many other cognitive skills, are being used to
understand and make connections to the text; comprehending - the reader expects to
understand the text; and flexible - the reader is able to employ the multiple strategies
needed to read efficiently, such as skimming, considering titles, headings and text
structure, and anticipating information.
Grabe (1991) goes on to discuss 6 component skills, all of which play a part in
each of the aforementioned descriptions of fluent reading:
8

1) Automatic recognition skills. These have widely been recognized by cognitive and
educational psychologists as important, but are only now being recognized within L2
reading research.Automaticity is achieved when a reader is unconscious of the controlling
process in decoding the text, and uses little processing capacity in doing so. It is
particularly important in terms of word identification skills.
2) Vocabulary and syntactic knowledge. These are essentially the size of a readers
vocabulary and familiarity with the structural aspects of the text. Vocabulary is largely
measured by looking at a readers recognition of words in the target language. There is
some debate as to the number of words a reader must be able to recognize in order to be a
fluent reader of the language. Grabe suggest that fluent L1 readers have a vocabulary
range of 10,000 100,000 words, while he argues an L2 readersmay be able to get by
with a range of only 2,000 7,000 words. Regardless of the outcome of this debate,
there is strong evidence that word recognition plays an important role in reading fluency.
Furthermore, without the aforementioned syntactic knowledge, the great variety of text
structures that exist may serve only to confuse readers.
3) Formal discourse structure knowledge. This is a readers formal schemata, or
understanding of how a text is organized. This enables a reader to better comprehend a
text. For example, a readers realizing that a particular passage was written in the style of
a comparative essay will contribute to their understanding of the way the author presents
and develops the information, and thus to their understanding of the information itself.
4) Content/world background knowledge. This is a readers content schemata, or their
ability to use prior knowledge related to the text. It plays an important role in
comprehension.
9

5) Synthesis and evaluation skills/strategies. The particular skills and strategies that a
reader uses to integrate their prior knowledge with the new information they have gained
through accessing the text. This is done by taking prior knowledge and comparing and
evaluating it within the framework of the new information that they have just read, then
synthesizing the information into a new understanding or comprehension. Fluent readers
are also continuously in the process of predicting the development of the text, during
which they use evaluation skills to reconcile their predictions with the actual
developments.
6) Metacognitive knowledge and skills monitoring. These are composed of what
strategies a reader uses to monitor their understanding while reading, as well as the
strategies they employ to better understand the text. An example would be a reader
repairing a misunderstanding by considering the context, or adjusting their reading rate to
meet the complexity of the text.

Table #1
Summative table of Grabes construct of
reading (by Kamp and Silberner)

10

Reading Construct
Component Skills in Reading

Fluent Reading
Rapid

Automatic recognition skill

Purposeful

Vocabulary and structural knowledge

Interactive

Formal discourse structure knowledge

Comprehending

Content / world background knowledge

Flexible

Synthesis and evaluation

skills / strategies
Metacognative knowledge
and skills monitoring

Figure 2: Summative table of Grabes construct of reading (by Kamp & Silberner, 2007

Marshall and Campbells (2006) also weighed in on the relationship between

fluency and comprehension. Similarl to Grabe (1991), they expressed the importance of
fluency in achieving comprehension. Their model of fluency, however, was more product
oriented. For them, fluency is composed of speed, accuracy, expression. Speed and
accuracy are both measured in words per minute. Speed is simply the total words read per
minute, while accuracy is the words read correctly per minute. They also viewed
expression, the ability to read in such a way as to sound like one is speaking
conversationally when orally reading, as an important indicator of fluency. They judge
expression through the tone, junction (or phrasing), and pitch that the reader use. The
inclusion of this latter aspect of fluency is potentially problematic, as students may be
highly competent oral readers while displaying little actual comprehension. Marshall and
11

Campbell felt that by using expression as their means of evaluation, they were addressing
this problem. However, there is as yet no evidence that supports their conclusion.
Medina and Pilonieta (2006) focused on the comprehension of narrative texts.
They presented a process model that was very much aligned with Grabes (1991), and
once again echoed the notion of integrating visual and non-visual information. In their
view, comprehension starts with an understanding of how words are involved in creating
meaning. They conceptualize the reading process as involving contextualization,
analyzing, synthesizing, and evaluating words, phrases, sentences, and longer passages.
The text read is integrated into the readers prior knowledge and experiences of the
world, utilizing short- and long-term memory.
According to Medina and Pilonieta, comprehension in reading is the connecting
of new and prior knowledge. The process is both integrative and expansive. The reading
of a text brings new information to the reader, who then accesses their prior knowledge,
or file folder, (pg. 227) of related information. As new knowledge is added and
integrated with prior knowledge of related concepts, prior knowledge reciprocally
contextualizes the new knowledge. For example, when one learns about winter, the
information is added and integrated with prior knowledge of seasons, while seasons
simultaneously contextualizes the information about winter.
For
Hill and Parry (1994) reading is an act of communication. The relationship of the
writer to the reader is one of social reciprocity in negotiating the meaning of the text.
They distinguish three skills that are necessary for this to occur: knowledge of the writing
system, essentially the decoding process; possessing appropriate background knowledge;
12

and being able to engage in a reciprocal exchange with the text being read. Thus their
exploration of the reading component of literacy further integrates the models that we
have already discussed. Knowledge of the writing system is where Smiths (1985)
integration of visual and non-visual information first occurs. Possessing appropriate
background knowledge aligns them with Grabe (1991) and Medina and Pilonieta (2006)
by emphasizing the connection of prior knowledge to the new information being read.
Within their framework, prior knowledge is a crucial factor in engaging in a reciprocal
exchange with the text being read, and thus in reading comprehension. Hill and Parry
give the example that a native language reader attempting to understand a text about
elementary particle theory in a physics textbook. Despite being able to decode and
understand the individual words, unless they were familiar with the concepts discussed in
the text they could not understand the new information, could not communicate with the
text, and thus would lack comprehension.

Theoretical definition of the construct of reading ability

After reviewing the literature related to the construct of reading, we have defined the
construct as being composed of these three aspects:
1) Details. That which is explicitly located in the language of the text.
2) Inferences. The connections between the details and prior knowledge in order to
locate the implied meanings in the language of the text.
3) Gist. The authors main ideas, purpose, or message.

13

Reading

Details

Inferences

Gist

Figure 3: Theoretical definition of the construct of reading ability

Despite the fact that most researchers have made the switch to examining what
processes are occurring within the reader, we felt it would be more fruitful to define our
construct in terms of the purposes readers are putting those processes to. Certain
processes were almost universally discussed in the various theoretical definitions of the
construct. One such process was the connecting of the decoded text to the readers prior
knowledge (Mackworth, 1970; Smith, 1985; Grabe, 1991; Medina and Pilonieta, 2006;
Hill and Parry, 1994).
In attempting to determine how readers used employedthis process, Lennons (1970)
early model was highly instructive. He mentions verbal factor, a readers general
vocabulary knowledge; comprehension of explicitly stated material, being able to
comprehend the literal details; comprehension of implicit meanings, all those outcomes of
reasoning about the text; and appreciation, recognizing the authors intentions through
14

noticing the language and literary devices that have been used. All of these necessarily
rely on prior knowledge, and thus we felt that our categories must be in some way related
to these factors.
Anderson (1991) had listed five aspects of textual comprehension that, while
imperfect, helped us to further clarify our understanding of this connecting of visual and
non-visual information. Anderson mentioned textually explicit or direct statements, by
which he meant comprehending the details explicitly stated in the text. Certain other
categories he mentioned textually implicit and drawing inferences, seemed to us to both
be conceptually similar. Referring to Lennons (1970) category, comprehension of
implicit meanings, confirmed this in our minds. We felt that drawing inferences, though
not clearly defined, was what all three of these categories (textually implicit, drawing
inferences, and comprehension of implicit meanings) were basically concerned with.
Anderson also included the category main idea. He did not define this, but we felt
that it was conceptually related to both general comprehensions, and Lennons
appreciation. Often a reader gets the main idea, tone, or purpose, without understanding
every word, detail, or inference. We felt that these deserved their own category, and
decided that they could be better understood if they were to be combined into one. We
felt the term gist most accurately described our intentions.
In terms of other authors inclusions of sub-skills, such as vocabulary knowledge
(Lennon, 1970; Grabe, 1991; Marshall and Campbell, 2006; Medina and Pilonieta, 2006)
and textual knowledge (Grabe, 1991; Medina and Pilonieta, 2006), we felt that the
process of recognizing and analyzing these were merely tools a reader used to carry out
their purpose, rather than purposes unto themselvs.
15

The nature of academic writing

There are two ends of the continuum along which the construct of academic writing
could be defined. At one end of the continuum, a model could be constructed that was
entirely a theoretical representation of the skills involved in academic writing. In contrast,
researchers could create a syllabus-based construct definition by analyzing the actual
academic tasks needed in academic writing in order to ascertain which skills are being
called upon, and afterwards use their data to build a model. Herein we will cover
construct definitions that exist along several points of the continuum.
Bereiter and Scardamalias (1987) model lies towards the theoretical end of the sp
ectrum. They were interested in advancing understanding of the mind. They believed that
one important avenue of research was studying the relationship between what the mind d
oes naturally and with ease, and those cognitive functions that take a sustained effort to d
o well. In their estimation, writing was an excellent vehicle through which to study this re
lationship;

On the one hand, writing is a skill traditionally viewed as difficult to acquire, and on it is
developed to immensely higher levels in some people than others. Thus it is a suitable do
main for the study of expertise. On the other hand, it is based on linguistic capabilities tha
t are shared by all normal members of the species (p. 4).

16

Bereiter and Scardamalias (1987) research led them to propose two different models of t
he writing process. The first, knowledge telling, involves writing as a natural task, where
an individual makes use of cognitive structures that are already in place in the mind. An i
ndividual will examine the writing assignment, looking for topical and genre identifiers t
hat relate to their prior knowledge. Information retrieval is cued, and relevant knowledge
of the topic and discourse type called for are brought to the surface, without the writer ha
ving to plan or monitor for coherence. Once retrieved, knowledge is subjected to a test of
appropriateness, the extent of which may greatly vary according to the context of the assi
gnment and the level of expertise of the writer. Then the writer will write what they have
deemed appropriate. In this model, as a writer gains greater maturity, syntactic fluency, a
nd discourse knowledge, they may become a highly effective writer.
Bereiter and Scardamalias (1987) second model, which is more difficult for write
rs to employ, is knowledge transformation; When an individual makes writing a task that
keeps growing in complexity to match the expanding competence of the writer (p.5). As
an individual gains more knowledge and skills, they replace old problems with new ones.
Essentially, it is a process where thoughts come into existence through the writing proces
s itself, beginning formlessly, and through much rethinking and restating becoming fully
developed. It is important to note that in this model the knowledge telling process become
s embedded in two types of problem solving spaces. There is a content space, where issue
s of beliefs and knowledge are worked out; and a rhetorical space, where issues surround
ing achievement of the discourse goals are dealt with. Output from each also serves as inp
ut to the other, which may necessitate further problem solving, with the text being contin
uously modified as the writers beliefs change. Eventually the text will reach a point wher
17

e it finally represents the writers latest thinking on the subject. According to Bereiter and
Scardamalia, it is this latter model which offers the opportunity for greater literary quality.
Bereiter and Scardamalia (1987) go on to discuss the difficulties associated wi
th selecting languageto suit an audience that is not present. We all learn language in
conversational contexts, thus learning to write depends on revamping the language-pr
oduction system so that it can function autonomously(Pg. 57). Simultaneously,stude
nts are learning the mechanics of writing; how to access working memory - essential
to the recall of content and rhetorical knowledge; and executive control of internal fe
edback - without which planning and revising is impossible.
The process that Bereiter and Scardamalia describe, looking for topical and genre
identifiers, cueing information retrieval from memory, testing the informations
appropriateness with audience in mind, are what Douglas (2000) has termed strategic
competence. For Douglas, whose model also lies toward the theoretical end of the
continuum, this includes four separate aspects; assessment, where the task is evaluated to
determine what type of response is called for; goal setting, determining how to respond;
planning, deciding which elements of language and content knowledge are called for; and
control of execution, accessing the necessary language knowledge. For Douglas, strategic
competence is the guide, the metacognitive skills that writers call on in managing their
anguage knowledge: their grammatical knowledge of lexicon and syntax; textual
18

knowledge of organization; functional knowledge of how to use language to develop

ideas; and sociolinguistic knowledge of what language is appropriate in the context.
Vhpssi (1982) constructed a syllabus-based model that presented a general
picture of the nature of written discourse, comprised of two intersecting dimensions of
cognitive processing and dominant intention, and the criteria for success within that
picture.
The dimension of cognitive processing includes the types of mental activities that
are being carried out during writing: reproduce, organize/reorganize, and
invent/generate. These three are related to the primary content of the text, as well as the
authors general verbal ability. The latter category, invent/generate, suggests an alignment
towards Bereiter and Scardamalias model of knowledge transforming.
Dominant intention is the purpose for writing, or mode of discourse, and according to
Vhpssi is related to the primary audience. In his examination of dominant intentions,
he chose to concentrate on the communicative act of writing, and therefore left out
writing with an archival purpose. What he included was, to learn (metalingual), to
convey emotions (emotive), to inform (referential), to convince/persuade, and to entertain
(poetic). Vhpssi concludes that traditional literary genres may fall into one or more of
the four latter purposes. In our view, however, not including to learn as a possible literary
purpose seems an oversight in light of Bereiter and Scardamalias model of knowledge tr
ansforming.
Vhpssi (1982) also noted that within the domain of academic writing there are
several other related factors that affect the final product. The major ones are the content

19

of the writing, and the writing situation (e.g. clues, stimuli). Awareness of being
evaluated and perceptions of criteria used in evaluation also have effects.
Vhpssi (1982), constantly aware of the contrastive rhetoric that exists in
different cultures, went on to hypothesize a set of four criteria for successful school
writing, each with sub-criteria, that he claims work across many cultural boundaries.
His first, articulateness, includes a broad range of factors: adherence to standard
Dialect and conventions; clarity and comprehensibility, which Vhpssi defines as
mastery of the cognitive content, the process dimension of his model of written
discourse; and coherence, or the order of ideas or topics and the flow of sentences
The second, Fluency, is how Vhpssi refers to the actual rate or amount of writing
done. The third criteria is , Flexibility, which includes the ability to write different types
of compositions, for different purposes, for different kinds of audiences, and to adopt
different points of view, can be evaluated only by looking at multiple assignments.
The final criteria, appropriateness, which Vhpssi claims also can only be judged
across a number of assignments, is essentially the ability to select the appropriate role in a
given assignment (purpose, audience, type, point-of-view), and to then adhere to
conventions associated with that role, genre or discipline.
Vhpssis (1982) scheme has several overlapping layers. Articulateness seems
to encompass all the aspects that Douglas (2000) refers to as language knowledge; the
organization or structural format, the logical development of the reasoning within the
text, pragmatic and sociolinguistic aspects of writing, and grammar; Vhpssis
flexibility and appropriateness are more overarching abilities. The sub-criterion of
articulateness changes with shifts in dominant intention, under which umbrella we would
20

include role, genre, and discipline. Appropriateness, then, is primarily the ability to assess
the task and recognize the fitting dominant intention, while flexibility is being able to
carry off articulateness within those varieties of dominant intentions.
It would seem that Vhpssis appropriateness and flexibility are together aligned with
Douglas strategic competence. The latter is bringing relevant knowledge into use at the
right time, and in the right relationship to the resources demanded by the task (Douglas,
2000, p. 79), while the former is the ability to use this

Both Vhpssis inclusion of fluency, whether it has any actual bearing on the level of
expertise of the writer, is an open question. A wide variety of physical conditions could
alter speed of writing. Furthermore, we question whether the length of an article has any
bearing on its success in fulfilling its intended purpose. In many cases succinctness would
be considered a virtue.

Bereiter and Scardamalias models knowledge telling and knowledge

transforming, dynamic explanations of the cognitive processes as they occur linearly in th
e writer, could both be easily integrated with Vhpssis (1982) criteria for successful ac
ademic writing. When writers who have expertise in the use of knowledge telling access
memory and subject it to a test of appropriateness,they are simultaneously attempting to
meet Vhpssis criteria of appropriateness and articulateness. As knowledge
transforming encompasses the cognitive processes that occur during knowledge telling,
there is similarly no friction between this model and Vhpssis. Unfortunately, while
evaluating a writers work with Vhpssis criteria of flexibility in mind would be
21

helpful in determining expertise, it would not be helpful in determining whether

knowledge telling or knowledge transforming has occurred
Ballard and Clanchy (1991) lay further towards a syllabus-based definition of the
construct of academic writing. Rather than examining academic task themselves, howeve
r, they looked at the comments that university lecturers made on 500 first-year student es
says, in order to determine the criteria for success in academic writing. The four criteria t
hat Ballard and Clanchy found represent a fairly succinct model upon which to judge suc
cessful academic writing.
The first criterion is that students focus on the topi. This means that they must stay
relevant, developing their argument in the rigid, linear fashion of English speaking acade
mia. The second criterion is evidence of critical reading. In Ballard and Clanchy own w
ords, That the writer has considered a wide range of alternative views and sources of ev
idence, and that he has considered these critically in developing his own position on the is
sues under discussion (pg. 31). The third criterion is that students present a reasoned arg
ument. In other words, the writer must demonstrate that the issues have been understood,
then present an opinion about these issues, and finally give reasons for their viewpoint. T
he fourth criterion is competent presentation. That is, following of the conventions of spel
ling, grammar, idiom, register, format, and styles of referencing.
It is interesting that Ballard and Clanchy (1991) and Vhpssis (1982) models b
oth contain four distinct criterion, and both compress all of Douglas (2000) language
knowledge into a single criterion. We find this compression problematic, as a writer may
display one competence without having control of the other. To give an example, an L2
writer who develops the ideas within their text with great aplomb may be displaying a
22

high degree of functional and sociolinguistic competence, while their grammatical

competence may be at a much lower level.
Ballard and Clanchys (1991) notions that evidence of critical reading and reasoned a
rgument are both important components of success, align their definition of academic
writing with Bereiter and Scardamalias knowledge transforming. However, as Vhpss
is general picture of written discourse will show, these two criteria are not a part of every
type of writing. Nor do Bereiter and Scardamalia insist that all academic writing calls upo
n the use of the knowledge transformation model. These two criteria, which may be
selected as appropriate after reviewing the purpose and audience for the assignment, are
not true criterion in and of themselves, and therefore present a problem for Ballard and Cl
anchys model. In the end, we must consider their model subordinate to those of Douglas
and Vhpssi.
Reid (1989), in a more theoretically minded model, weighed in on a particular aspect
of the academic writing construct. Reid points out that to write successfully in academia
students must be able to employ pre-writing strategies. During this part of the writing
process students should not only be generating ideas, but also considering the purpose
and audience in formulating their plan. Reids understanding of the pre-writing process is
remarkably similar to Bachman and Palmers definition of strategic competenc. Where
they diverge is Reids assertion that, In academic writing, the purpose of a writing
assignment is usually designed, assigned, and evaluated by the audience (the professor),
so the two are closely connected. (pg. 220). Goal setting, then, is primarily out of the
hands of the writer. Thus, pre-writing involves understanding the rhetorical expectations
of content and form required by the instructor, and then generating ideas and planning
23

based on that understanding. In other words, students must, as they plan their writing,
consider the types of criterion for successful academic writing that have been outlined by
both Vhpssi (1982) and Ballard and Clanchy (1991).
The components that an author is trying to include in their product, beyond
mechanics, as outlined by Reid, include: format, such as the summary, the five paragraph
expository theme, the argumentative essay, and the response to written materials;
organization, what goes into the introduction, the body and the conclusion; and syntactic
form, those aspec primarily associated with academic register.
Reids three latter categories are what we would associate with Vhpssis
(1982) articulateness, Ballard and Clanchys (1991) competent presentation, and
Douglas (2000) language knowledge. Unfortunately, Reid does not assign mechanics,
what Douglas would call grammatical knowledge, as a criterion unto itself. For those
concerned with L2 writing, where grammatical knowledge is an ever present concern,
this is an oversight that seriously undermines Reids model.
Johns (1991) created a syllabus-based model following an investigation that was p
rompted by her diverse freshman ESL students lack of success on essay tests. She set up
a series of interviews with two of their political science instructors in order to determine s
teps she could take to improve her students performance. Her report on these discussions
concerning the nature of academic literacy may have bearing on the construct of academi
c writing in general. The instructors identified six specific areas of weakness.
Many students lack of background knowledge. Without depth of culturally relevan
t knowledge students can not contextualize the information in their readings. Lacking the
background to understand the informations historical significance, they can not feel comf
24

ortable thinking about it with certainty. Their writing thus became constrained and derivat
ive.

They also have problems interpreting and producing the macropurposes of texts. I

n other words, they find it difficult to locate the main ideas of the texts about which they
were writing, and instead focused on specific details. Another problem is that of lack of p
lanning in approaching reading or writing. Students tend to read or write without precon
ceived ideas about what their goals should be. When an essay prompt was given students
did not deconstruct it and plan responses based on the linguistic clues.Instead, they tende
d to list details that they had read from the readings, rather than presenting an organized a
nd focused argument. An additional problem is lack of conceptual imagination. Students
have trouble making connections between concepts and examples. A further problem,
they have a lack of vocabulary. This by and large consisted of the sub-technical academic
vocabulary. In other words, all the language that allows students to write formally, in an
academic sounding register. Finally, they have
conflicting values. Western academics often expect suspension of personal beliefs when
examining controversial issues. When their basic values are threatened, rather than
responding with the desired detachment, their writing becomes emotional and personal.
Johns extrapolates from these six areas of reported weaknesses, three criterion for
successful academic writing. The first is audience awareness,which is extrapolated from l
ack of planning, lack of vocabulary, and value conflicts. The second criteria is exploitatio
n of common academic genres. Johns makes the point that this primarily consists of argu
mentation and problem-solution. The third criteria is relationship with subject matter. Thi
s addresses student difficulties with lack of background knowledge, conceptual imaginati
on, and also elements of lack of planning and lack of vocabulary.
25

We can easily draw connections between Johns areas of weaknesses and

Douglas (2000) notions of strategic competence. Audience awareness was an
extrapolation from interpreting and producing macropurposes, essentially assessment,
and both texts use the term planning in more or less the same way. Inclusion of backgrou
nd knowledge and conceptual imagination, what she eventually included under the
criterion of relationship with subject matter, indicates that Johns also believed that
writing can not be separated from mastery of the content. In this sense she aligns herself
with Bereiter and Scardamalia, for whom the content space was an essential part of the
cognitive process, and Vhpssi (1982), whose inclusions for whom articulateness
could not occur without mastery of the process dimension of his model.
Cummings (2001) is another researcher whose definition of the construct comes
closer to the syllabus-based end of the continuum. He conducted 48 interviews with
highly experienced instructors in both ESL and EFL classrooms. Of the ESL classrooms,
19 were in English speaking academic situations and 12 were in continuing educations
programs for adult immigrants. The programs that these professors taught in had both
specific purpose and general purpose syllabi. Cummings spoke with instructors, about the
curriculum and assessment of writing classes. Cummings analysis of the interviews
turned up several interesting finding, amongst which was a set of 5 indicators of
achievement in writing. The first language and style, encompasses development and cohe
rence of the ideas within the text, use of morphemes, tense, and linking words, control ov
er sentence length and complexity, and referencing ideas. The second, self-confidence an
d expressive ability, is the range of writing, and ability to express oneself forcefully and c
oherently, as well as having a unique voice. The third, composing processes, is metacogni
26

tve awareness, goal setting, and understanding of the various parts of the writing process,
especially revision. The fourth , rhetoric, involves being conscious of the audience, and b
eing able to state and support ideas. The fifth, acculturation, is acculturation into the acad
emic community being evident in the writing. The final indicator of achievement is langu
age and style, a very broad category, in a similar vein to Vhpssis articulateness or
Ballard and Clanchys competent presentation, that is contained within the boundaries of
language knowledge (Bachman and Palmer, 1996; Douglas, 2000). It likewise
encompasses Reids (1989) format, organization, and syntactic format, and touches upon
control of the topical knowledge. So too does rhetoric, as it includes the ability to support
the ideas presented in the text. We would suggest that rhetoric encompasses functional
knowledge and elements of strategic competence. It is in composing processes, however,
that we find the main elements of Douglas (2000) strategic competenc. In composing pr
ocesses, though, the element of knowledge of the revising process is added. This is
interesting in light of Reids renaming of strategic competence as pre-writing.
Cummings self-confidence and expressive ability is also of interest, as it includes notions
of forcefulness and voice. The question of whether these are necessarily called upon in
academic writing is one we will return to in our discussion of how we arrived at our own
theoretical definition of academic writing.
Connor and Mbaye (2002), further towards the theoretical end of the continuum, p
ropose that the construct of writing be examined through text analysis. The model that the
y proposed expanded on Canale and Swains (1980) model of communicative competenc
e. Thus, they proposed four corresponding types of competencies in their writing compete
nce model: grammatical competence includes grammar, vocabulary, spelling and punctua
27

tion; discourse competence includes discourse organization, cohesion, and coherence. In

other words, the ability to combine sentences to form a
text that is meaningful on the supra-sentential level. This would be demonstrated through
topical structure analysis, or measuring the semantic unity of the text by looking at the rel
ationship between sentences and the overall discourse topic; sociolinguistic competence i
ncludes genre appropriacy, register, and tone. It is influenced by a writers awareness of b
oth audience and purpose; and strategic competence includes audience/reader awareness,
appeals, pertinence of claims, and warrants. In Canale and Swains (1980) model this cate
gory was primarily concerned with repairing communication breakdowns. In terms of dis
course and sociolinguistic competence, there is a certain amount of overlap. The latter inc
ludes use of genre appropriate language, while within the former the writer must organize
in a way that is also appropriate to the genre.
The categories grammatical competence and sociolinguistic competence are
basically the same as the categories grammatical knowledge and sociolinguistic
knowledge proposed by Bachman and Palmer. Within discourse competence, however, C
onnor and Mbaye have condensed all of textual and functional knowledge.
Strategic competence here is defined quite differently than in Douglas (2000) or B
achman and Palmer (1996). In Connor and Mbaye it concerns the use of transitions and m
etatextual markers.The inclusion of metatextual markers in the category of strategic comp
etence is of great interest. Metatextual marker are those ways an author helps the reader f
ollow the organization and development of ideas within the text. For instance, an author
might use connectives to indicate sequence within a text, or illocution markers, such as
to give an example, in order to show the reader what speech act they are performing.
28

Metatxtual markers also allow the author to expresses feelings and attitudes, and otherwis
e interact with the audience. The separation of this class of rhetorical tools into their own
category is unique amongst the models we have reviewed. Bachman and Palmer (1996),
for instance, would have considered these as to be an aspect of functional knowledge
Silva et al. (2003) analyzed five texts written by L2 writers about their own writin
g processes and experiences. The model they presented is partly made up of the elements
of written text, and of the writing process itself. The text features that were included wer
e an understanding of rhetorical features, such as coherence, creativity, directness, explici
tness, linearity, logic, organization, originality, and verbosity; rhetorical elements, such as
introduction, thesis statement, paragraphs, arguments, and conclusion; and mechanical as
pects, such as grammar and lexicon. The writing process is the time spent choosing which
of the above element would be employed, and then revising as needed, similar to
Cummings (2001) composing process. While the textual categories are fairly broad, it is
clear from Silva et al.s discussion of the writing process that their conception is aligned c
losely with Bereiter and Scardamalias knowledge transforming model.

After reviewing the literature related to the construct of academic writing, we hav
e defined the construct as being composed of these three aspects:
1) Organization. The structure, such that the text follows the appropriate genre fo
rmat, and proceeds in a coherent fashion.
2) Development. The ideas of the author being elaborated and supported with rel
evant evidence.

29

3) Mechanics. Academic standards of spelling, punctuation, capitalization, gram

mar, usage, and vocabulary.
Mechanics
Writing

Organization
Development

Figure 1: Theoretical Model of Academic Writing Ability

Johns (1991) criterion of audience awareness and exploitation of common acade
mic genres had seemed quite useful to us in light of assertion that audience (the
professor) is primarily responsible for setting the purpose in academic writing (Horowitz,
1986; Reid, 1989). Certainly, it would be this awareness which drives decisions in terms
of which academic genre to exploit. In other word, during the course of pre-writing
(Reid, 1989) the writer would carry out assessment (Douglas, 2000). During this, they
would determine what was called for in terms of both language knowledge (Bachman and
Palmer, 1996; Douglas, 2000) and rhetoric (Cummings, 2001), which is another way of
saying exploitation of common academic genre; finally they would subject what they
have determined is called for to a test of appropriateness (Bereiter and Scardamalia,
1987; Vhpssi, 1982).
While all of the above is an acknowledgment of the importance of strategic
competence, in light of our realization that the whole of its elementsare not called on in
all writing assignments, we feel that it is best viewed as its own separate
construct.Writing itself, we feel, is best defined as only being inclusive of language
knowledge.
30

Our next step was to determine what was universally called upon, or rather what
the overarching categories were for these elements. We had discovered certain elements
that were consistently referred to. For example, while Vhpssi (1982), Cummings (200
1), and Silva et al. (2003) discussed grammatical knowledge only as a subset of other crit
erion, all of the authors we looked at included it in some fashion. Connor and Mbaye (20
02) had included aspects such as spelling and punctuation in Grammatical competence.
We also wanted to include these elements. We chose to call it mechanics, however, as we
felt that spelling and punctuations were not actually grammar.
Another element that was universal was organization of the text, whatReid (1989)
calls, format, and Bachman and Palmer (1996) refer to as textual knowledge. While they
were the only authors to assign these their own distinct category, when other authors
referred to this element, the term organization is used. We felt this to be a more explicit
term and chose to adopt it.
The final universal element was what Cummings (2001) called rhetoric. He was
the only author who assigned this element a clearly distinct category to what is generally
thought of as developing and supporting ones ideas. Yet all the authors had included this
notion in some form or another. As the word rhetoric has been thrown aroundand used in
so many different ways, we thought it best to find a name that would cause less
confusion. Development was the term we felt to be most clear. We felt that development,
within the academic writing context would contain certain other universally discussed
elements as sub-elements. Academic register, style, and academic language(which
includes Connor and Mbayes (2002) metatextual markers) are all ways that a writer
develops their ideas during academic writing.
31

We also wanted to consider aspects that were included in various construct definit
ions that were entirely unique. Bereiter and Scardamalias (1987) two models of writing
is one such example. While these would perhaps be useful in terms of instruction, we felt
that the models were ultimately too vague regarding the cognitive processes that were
occurring in the writer to be useful additions to our model.
Vhpssis fluency was another unique inclusion, though we felt it was
questionable for reasons already outlined. Vhpssis inclusion of flexibility, on the
other hand, is a useful concept. Unfortunately it lacks clear parameters. How many genres
of writing would an individual need to be able to employ successfully in order to be
deemed flexible? Until that question is answered, we could not include it in our definition
of the construct. Furthermore, even if some parameters had been defined, we would not
have been able to operationalize it within our particular context
Ballard and Clanchy (1991) presented a model whose basic criterion we found
useful. Their inclusion of the categories evidence of critical reading and reasoned argum
ent, however, are fallacious in light of Horowitzs (1986) review of academic discourse
types. These may be called for in certain types of academic discourse, but they are not a
given. In the end it seemed to us that these categories were a subordinate of our own
category, development.
Reid had been somewhat vague about what he meant when he presented his
category organization, saying only that its what the writer decides to include in the
various parts of the text. It was clear that his format was composed of the same elements
as our organization. Since a writers decisions about what to put into the various portions

32

of the text is primarily a function of the how they are developing their ideas, we felt what
he termed organization is more closely alligned with what we have termed development.
Cummings (2001) category, Self-confidence and expressive ability, which
included notions of forcefulness of the writing and voice, is another inclusion that we had
to question in light of Horowitz. This ability may be one that the most expert writers
possess, but it is not necessarily one that is necessary for success in academic writing.
Thus we chose not to include it in our theoretical model.
Johns (1991), Cummings (2001), and Ballard and Clanchy (1991) had all included
relatively similar notions under different names. Cummings called it acculturation, Johns
called it values, Ballard and Clanchy referred to focus on the topic. What they all meant,
essentially, is that writing is organized and ideas developed in the rigid, linear fashion of
Western academia. We are in agreement with the above, but feel the notion is best
encompassed in the two separate categories we have defined, organization and
development.

The relationhip between reading and writing ability

Research in to nature of reading and writing, the productive and receptive aspects of liter
acy, have shown that the two abilities are reciprocal in many ways. Shanahan and Lomax
(1986) analyzed and compared three theoretical models of this relationship using an
extensive corpus of reading and writing data. It was in the first model that was reviewed,
the interactive model, that they found the best description of this reciprocity
In the interactive model reading is composed of word analysis (decoding),
vocabulary knowledge, and text comprehension; writing is made up of spelling,
33

vocabulary, syntax, and structure. The model postulates that reading development, which
proceed forward from letters to sentences to extended texts, influences writing
development. For instance, word analysis, the ability to decode words, will influence
spelling, though knowledge of spelling will not have much influence on the decoding
process. In other words, students ability to interpret words in texts will influence their
early writing. As learning advances, lower level writing begins to exert an influence on
higher level reading. Knowledge of spelling, at this point, can influence vocabulary
knowledge in reading, and thus is used in the process of reading comprehension.
While our own definition of the constructs differ from this presented in the
interactive model, we can similarly hypothesize similar relationships. We know that
writing is an attempt to communicate with an invisible audience. Hill and Parry (1994)
remind us that reading is also an act of communication with the writer. As a readers
skills in taking on the audience role in this communicative relationship grow, the more
familiar they become with the tools the writer uses to enhance this communication. For
instance, the more genres a reader becomes familiar with through extensive reading, the
more capable they will be of employing that discourse patterns of the genre. At the same
time, the more experience a writer has of using the tools of writing to enhance the act of
communication, the easier it will become for them to recognize the use of these tools by
another author, and hence the easier it will be for them to interpret the authors messag.

Details

34

Reading

Inferences
Gist
Mechanics

Writing

Organization
Developmen

Figure 3: Theoretical Model of the Relationship Between Reading and Writing

Design Statement
1 Test Purposes
A Inferences
About test takers ability in high school English Language Arts (ELA) domains where
writing in necessary.
B Decisions
1 Stakes: very low; test results will be used to inform future teacher instructional
decision making.
2 Individuals affected: teacher and test takers in the classroom.
3 Specific decisions to be made
a Diagnosis: teacher will use the test to identify reading and writing problems of
35

specific students and the class in general.

2 Description of the TLU Domain/and task types.
A Identification of tasks
1 TLU domain: High school ELA classroom and high stakes assessments
2 Identification and selection of TLU tasks for consideration as test tasks: The TLU
task was modeled on task types that likely to be encountered in high school ELA classes,
Regents exams, and introductory college level English composition courses. In order to
determine what task types students are likely to encounter in high school ELA and on
Regents exams, we consulted Seths cooperating teacher and examined the last several
Regents exams. To determine what types of writing task students are likely to encounter
at the college level, we relied on the task type surveys discussed in the literature review
(Horowitz, Johns).
B Description of TLU task types.
See tables

TLU Task 1

TLU Task 2

ELA Literary Response Essay

SETTING
Physical characteristics

Location: classroom, home, library.

Noise level: varied on campus,
including quiet to loud and crowded,
varied off campus, but likely
adequate to poor. Lighting:
Fluorescent lighting (no windows in
the classroom) to varied off campus,
but likely adequate to poor.
Temperature: Hot and stuffy in the
classroom to varied off campus.
Seating conditions: two person
tables, computer consoles, varied off
campus. Paper, pencils, word
processing programs. Students are
allowed dictionaries and thesauruses,
but rarely use them. Likely not to
have computers at home. When they

36

Regents style reading

comprehension passage
Same as Task 1

do type assignments at school they

use spell checker.
Participants

Teacher, classmates, friends who

might help, all who are likely to be
familiar with the test taker and have a
positive attitude towards them.

Students

Time of task
INPUT
Format
Channel

40 minutes to 1 week

10 15 minutes

Typically visual, but could include

audio, as there are class discussion
and oral instructions and meeting
with the teacher.

Visual

Form

Language

Same as Task 1

Language

Target

Same as Task 1

Type

Prompt

Item

Length

Extended Discourse

One page (300 500 words)

Speededness

Unspeeded

Same as Task 1

Vehicle

Live

Same as Task 1

Vocabulary: wide range of general

and literary criticism vocabulary.
Morphology and syntax: wide range
of organized structures. Graphology:
generally typed.

Same as Task 1

Variable: Wide range of cohesive

devices and organizational pattern.

Cohesion: cohesive. Rhetorical:

focused extended discussion and
analysis.

Language
characteristics
Organizational
characteristics
Grammatical

Textual
Pragmatic
characteristics
Functional
Sociolinguistic

Topical characteristics

Variable: ideational, manipulative,

and imaginative.
Dialect/variety: variable, standard
and regional. Register: formal and
informal. Naturalness: natural.
Cultural references and figurative
language: variable
Variable: literary and historical topics

37

Ideation, manipulative, and heuristic.

Dialect/variety: standard. Register:
formal. Naturalness: natural.
Cultural: references and figurative
language: variable.
Same as Task 1

EXPECTED
RESPONSE
Format
Channel

Visual

Same as Task 1

Form

Language

Language

Target

Non-language (circling the correct

letter)
Same as Task 1

Length

1- 2 pages

Short (12 MC items)

Type

Prompt

Selected response

Speededness

Unspeeded

Same as Task 1

Vocabulary: wide range of general

and literary criticism vocabulary.
Morphology and syntax: standard
English. Graphology: generally
handwritten, but occasionally typed.

Vocabulary: general. Morphosyntax:

standard English.

Textual

Variable: Wide range of cohesive

devices. Organizational patterns
usually include an introduction with
a thesis statement, a body that
develops an argument through use of
direct evidence and quotations from
the literature, and a conclusion
summarizing and reiterating.

Cohesion: cohesive. Rhetorical:

extended discussion and analysis.

Pragmatic
characteristics
Functional

Ideational, heuristic.
Dialect /variety: Standard. Register:
Formal. Naturalness: natural.
Cultural references: variable.

Ideation and heuristic.

Sociolinguistic

Variable: literary and historical

topics.

Dialect/variety: standard. Register:

formal. Naturalness: natural.
Cultural: references and figurative
language: variable.

Topical characteristics

Variable: literary and historical topics

Same as Task 1

Language
Characteristics
Organizational
characteristics
Grammatical

38

RELATIONSHIP
BETWEEN INPUT
AND RESPONSE
Reactivity

Non-reciprocal

Same as Task 1

Scope

Broad

Broad to narrow

Directness

Indirect

Direct

3 Description of characteristics of the test takers

A Personal characteristics
1
Age: 16 to 18, mostly 17.
2
Sex: male and female
3
Nationalities: 32 Dominican, 1 Honduran, 1 Ecuadorian
4
Immigrant Status: Immigrants
5
Native Language: Spanish
6
Level and type of general education: high school students, most with
elementary and middle school in their country of origin, two to three with part of middle
school in New York City.
7
Type and amount of preparation or prior experience with a given test: all test
takers are familiar with the ELA Regents exams. Most failed the last time the test was
administered.
B Topical knowledge of the test takers
1
In general, fairly limited range of topical knowledge, due to limited educational
experiences, cultural variety, and travel experience.
2
Highly specific topical knowledge, such as home culture, popular culture,
personal experiences studying English, and of Immigrating to New York City.
C Levels and profiles pf language knowledge of test takers
1
General level of language ability: High intermediate with a few low
intermediate.
2
Specific writing ability: widely varied. Very poor, barely comprehensible, to
intermediate. Some students are very capable of determining what the requirements of
tasks are organizationally, and developing their ideas, others have tremendous difficulties
with this.
D Possible affective response to taking the test
We hope that students will feel positive about taking the test, as it is meant to be used for
their own gain. Some students may feel it is purposeless, since they are not being graded,
and not take it in a motivated way. Less proficient students may feel frustrated, and also
not be motivated.
4 Definition of the construct(s)
39

A Language ability
The reading construct is theoretically based. It includes , while the writing construct falls
somewhere between a theoretical and syllabus-based definition.The goal of the ELA
classroom is to prepare students to pass exams and be able to function academically at the
high school level. In our review of the literature we found reading was much more
difficult to test. We felt we had to, therefore, rely on a theoretical model. Writing ability is
more directly testable. We thus prepared a theoretical model, and compared it the
syllabus-based constructs, which researchers had prepared through taking surveys and
collecting data on assignments at the first year college level (Horowitz, Johns). We then
attempted to analyze which factors from the theoretical model were most salient in
evaluations of our own our syllabus, and which we estimated would more relevant in our
students future syllabus-based models. Finally, we synthesized them into a theoreticalsyllabus-based construct definition.
B Strategic Competence
Strategic competence, as defined by Douglas (2000, pg.35), is not part of the definition of
the construct of reading, as we are interested only in making inferences about the
language components. Strategic competence is part of the construct of writing. We will
make inferences about assessment, evaluating the communicative situation or test task
and engaging in an appropriate discourse domain, by examining whether the idea they
chose to develop and their organization of their essay is appropriate to the situation;
planning, deciding what elements of language knowledge and background knowledge
are required to reach the established goal, by examining what elements they employed to
develop and organize their arguments; and control of execution, retrieving and
organizing the appropriate elements of language knowledge to carry out the plan, by
evaluating the final product. We have not included goal setting, deciding how (and
whether) to respond to the communicative situation, as our examination of the TLU
domain revealed that this aspect of strategic competence is primarily assigned by the
instructor.
C Topical Knowledge
Not included in the construct, since we are interested only in making inferences about
control of language knowledge.
5 Plan for evaluating qualities of usefulness
A Reliability
1 Setting minimum acceptable levels
a Relevant considerations: as this is an extremely low stakes test, minimum level of
acceptable reliability is low. The task types are very similar to the TLU tasks, and there
are only three tasks and two raters who will confer, so reliability will most likely be
relatively high.
40

2 Procedures for collecting evidence: As participants are few, and the test will have only
one administration, there will be no opportunity to collect evidence.
B Construct validity
1 Setting minimum acceptable levels
Relevant considerations
a Purpose: Low stakes, so limited evidence needs to be collected.
b Construct definition: In each construct, evidence needs to be collected related to
three components of language knowledge.
c Domain of generalization: evidence supporting generalizability in of score
interpretations to the intended domain of generalization needs to be collected.
2 Logical evaluation: The construct definition is clear, relevant to the purposes of the
test, and are reflected clearly in the test tasks and scoring procedures. The scores will be
useful in making the desired inferences about the test takers language abilities.
3 Procedures for collecting empirical evidence: Regents exams and classroom tasks
will be thoroughly examined and the classroom teacher will be consulted.
Task bias: The only potential source of bias is in the writing section, where students must
write about the reading passages. Evaluation of the essay may then be biased against poor
readers.
C Authenticity
1 Setting minimum acceptable levels
a Relevant considerations: domain of generalization is narrow, so minimum levels for
authenticity must be high.
b How quantified: Perceptions of the test developers, classroom teacher, and test takers.
2 Logical Evaluation: The description of the TLU domain is complete, and corresponds
highly to the test task.
3 Procedures for collecting empirical evidence: The classroom teacher will be
consulted, and along with the test developers will make visual comparison to determine
correspondence of task tasks and tasks in the TLU domain.
D Interactiveness
1 Setting minimum acceptable levels: The test does not rely on topical knowledge,
however, students are familiar with the general topic, and metacognitive skills and
language knowledge will be called upon in completing these test tasks. Interactiveness is
therefore, medium.
Language Level: High
Topical Knowledge: Moderate
Affective Response: Low
2 Logical Evaluation: Personal characteristics are included in the design statement, and
characteristics of the test task are suitable. Processing involves a relatively wide range of
language knowledge and language functions. The tasks are very interdependent.
Opportunities for strategy involvement are limited due to the speededness of the test, but
41

this is necessary to maintain authenticity. The input contains emotionally charged

material, but this is also necessary to maintain authenticity.
3 Procedures for collecting empirical evidence: Discussions with the students after
taking the test.
E Impact
1 Setting minimum acceptable levels
a Relevant considerations: This is only a student teacher prepared test, for the purposes
of practicing second language assessment. Therefore impact should be kept to a
minimum.
2 Logical Evaluation: The experience of taking the test is not likely to impact students,
the classroom teacher, or society
. 3 Procedures for collecting empirical evidence: Discussions with the students and
teacher after the test is administered.
F Practicality
1 Logical Evaluation during design and operationalization stages: Paper and pencils,
internet access, and a word processing program are all that is necessary throughout. All
materials are available.
6 Inventory of available resources and plan for their allocation and management:
A Design stage
1 Human resources: 2 test developers/raters.
2 Material resources: Paper, pencils, a computer with internet access and word
Processor.
B Operationalization stage: same
C Administration: Same
7 Allocation of resources & costs: All space, equipment, and materials, will be provided
by Teachers College and Gregorio Luperon High School.
8 Test development time line
February 2007: Design and develop test
March 2007: Administer and score test
Operationalization
In this project the tasks in the TLU domain are not widely varied, since its purpose is to
prepare students to complete ELA Regents exams, and have the skills necessary for first
year college level English composition courses. While this means we had students with a
relatively similar level of topical knowledge, the TLU domain is more concerned with
general language knowledge and strategic competence. Also aware that L2 learners have
difficulty with unfamiliar topics, we decided to use passages that were related to the topic
they had been studying, but which would not rely on any of that topical knowledge
explicitly. For the reasons of authenticity and construct validity we developed a writing
task with relatively specific direction.
42

Blueprint
Test structure
1 Number of parts/tasks: The test consists of two parts: A reading part with one task
consisting of two passages, each followed by 6 multiple choice items, and a writing part
with one task.
2 Salience of parts: parts are clearly distinct.
3 Sequence of parts: Reading first, writing second.
4 Relative importance of parts or tasks: The parts are of equal importance. However,
the first part could stand alone, while the second part relies on understanding the first
part.
5 Number of tasks per part: Two in the reading, one in the writing.
Test task specifications
1 Purpose: see design statement
2 Definition of the construct: see design statement.
3 Setting: see the classroom discussed in the design statement.
4 Time allotment: 1 hour and 30 minutes
5 Instructions:
a Language: target language (English). Test takers read instructions as they are read
aloud by the proctor.
b Channel: visual (writing)
c Instructions (see Appendix 1)

6 Characteristics of input and expected response:

SETTING
Physical Characteristics

See TLU description in design statement

Participants

See TLU description in design statement

Time of task
INPUT
Format
Channel

Friday morning, 11:20 -12:50

Form

Language

Visual

43

Language

Target

Length

2 typed pages, double spaced, 12 pt. font

Type

1 Narrative and one historical passage, and a

complex prompt that that specifies a number of
criteria the writing sample should meet.

Speededness

unspeeded

Vehicle

Live

Language characteristics
Organizational Characteristics
Grammatical

Textual
Pragmatic Characteristics
Functional

Vocabulary: General and specifically related to the

topic of the reading passage. Morphology and
syntax: Standard English and regional. Graphology:
typewritten
Cohesion: Cohesive within each passage and step of
the prompt. Organization: list of procedures.
Heuristic, ideational, and manipulative

Sociolinguistic

Dialect/variety: Standard and regional. Register:

Formal and informal. Naturalness: Natural. Cultural
references: None. Figurative language: moderate.

Topical characteristics

Readings about reactions to Nazi policies in Europe,

and a prompt for synthesis of the given information.

EXPECTED RESPONSE
Format
Channel

Visual

Form

Language

Language

Target

Length

12b multiple choice items (Reading) , and a 1 2

pages handwritten essay (Writing).

Type

Selected response (Reading) and extended

production (Writing)

Speededness

Speeded

Language Characteristics (Writing)

Organizational characteristics
Grammatical

Vocabulary: General and specifically related to the

topic of the reading passage. Morphology and
syntax: Standard English. Graphology: handwritten.

Textual

Cohesion: Cohesive throughout. Rhetorical: Written

44

in essay form.
Pragmatic Characteristics
Functional

Ideational and manipulative

Sociolinguistic

Dialect/variety: Standard. Register: Formal.

Naturalness: Natural. Cultural References: None.
Figurative language: None

Topical Characteristics

Prompt for comparing and evaluating reactions to

Nazi policies in Europe.

RELATIONSHIP BETWEEN INPUT AND

RESPONSE
Reactivity

Non-reciprocal

Scope of relationship

Broad

Directness of relationship

Direct

7 Scoring Method
A Criteria for correctness: For reading, a criterion-referenced multiple choice answer
key language ability scales. For writing, a criterion-referenced rubric based on theory and
syllabus based definitions of the ability to use language in a writing task. Writing samples
will be scored on a five point scale for three separate criterions: organization,
development, and mechanics.
B Procedures for scoring responses: Each essay will be read by two raters, and scored
will be averaged. No rating session will last longer than one hour.
C Explicitness of criteria and procedures: the test takers are informed in general terms of
about the scoring procedures.

Directions to be read at the beginning of the test

This is a test of your reading and writing abilities. As we have been studying the history
and literature relating to Nazi Germany and the Holocaust, the test is related to this topic.
Though you may use the knowledge you have learned in this class, all of the information
you will need to complete this test will be provided in the two passages that you will be
asked to read.
45

The first part of the test is comprised of two reading passages. Each is approximately one
page in length. After reading each passage, you will answer six multiple questions. You
will be given 45 minutes for this part of the test.

The second part of the test is comprised of one writing prompt related to the two reading
passages. You will compose a 1 2 page essay in response. You may use your
background knowledge.

Good luck.

46

Twilight
by Elie Wiesel
The Nazis issued a proclamation directing all residents to gather in the town
square, in front of town hall, the next Saturday at 10 in the morning.
That day, the Jews went to the designated place in silence. As did the Poles. No
one knew what to expect. At precisely ten oclock, the Nazi commander exited the town
hall, flanked by officers in full dress uniform. The picture of vitality, he jumped up on a
stage that had been built for the day. He addressed the audience in a gravelly voice:
The Fhrers triumphant army is here to keep order. It will attend to the security
of civilians as long as they dont do anything to jeopardize its own security. At the
smallest hostility, the German army will deal harshly with the offenders.
Unfortunately, a serious incident has occurred. A soldier has been assaulted in
the street by a criminal. Fortunately he was only wounded and able to see his attacker.
He fell silent, took a deep breath, and continued:
The attacker was a Jew.
From the Poles came a sigh of relief. As for the Jews, they tried to make
themselves invisible.
Polish citizens, boomed the commander, I have summoned you here, together
with the Jews, so you may recognize the harm they are doing to you. Because of them,
you may have to endure hardships you dont deserve. I want you to remember that.
Among the Poles there were a few well known anti-Semites. It took only a
moment for the first one to shout:
Death to the Jews, death to the Jews! We let them stay in our country and this is
how they repay us. Let them die!
Allowing the agitation to spread, the German commander turned to his officers to
speak. Then he continued:
Today we will set an example. We shall punish the Jewish criminal who dared to
attack a soldier wearing the Fhrers uniform.
Two soldiers emerged from the town hall, pushing in front of them a Jew whose
face was covered in blood.
This crime warrants death, and thats what hell get, said the commander
looking pleased.
Hands tied behind his back, the condemned Jew ascended the gallows and faced
the crowd. If he was afraid he did not show it.
Are you sure you have nothing to say, asked the commander.
Nothing.

47

Twilight
by Ellie Wiesel
Directions:
Answer the following questions based on the Twilight passage.
1. What does the author want us to know about the Nazis?
a)
b)
c)
d)

They believed in justice.

They offered security to people.
They used fear to control people.
They were great military leaders.

2. What did the commander tell the Poles to arouse anti-Semitic feelings?
a) The Jews hated the Poles.
b) The Jews were killing the Poles.
c) The Jews were invading Poland.
d) The Jews caused the Poles to suffer.
3. What was the Jewish prisoner's punishment?
a) death
b) deportation
c) paying a fine
d) going to prison
4. What did the Jewish prisoner have on his face?
a) tears
b) blood
c) some dirt
d) a blindfold
5. What was the Jewish prisoner accused of?
a) killing a Pole
b) robbing a Pole
c) lying to a soldier
d) attacking a soldier
48

6. The author's main purpose in this passage was to show how the Nazis _________.
a) punished criminals
b) occupied countries
c) looked up to the Fhrer
d) created hatred of the Jews

49

Resistance in Denmark
Most individuals in occupied Europe did not actively collaborate in the Nazi genocide. Nor did
they do anything to help Jews and other victims of Nazi policies. Throughout the Holocaust,
millions of people silently stood by while they saw Jews, Gypsies, and other "enemies of the
Reich" being rounded up and deported. Many of these bystanders told themselves that what they
saw happening was none of their business. Others were too frightened to help. In many places,
providing shelter to Jews was a crime punishable by death.
In spite of the risks, a small number of individuals refused to stand by and watch. These people
had the courage to help by providing hiding places, underground escape routes, false documents,
food, clothing, money, and sometimes even weapons.
Denmark was the only occupied country that actively resisted the Nazi regime's attempts to
deport its Jewish citizens. On September 28, 1943, a German diplomat secretly informed the
Danish resistance that the Nazis were planning to deport the Danish Jews. The Danes responded
quickly, organizing a nationwide effort to smuggle the Jews by sea to neutral Sweden. Warned of
the German plans, Jews began to leave Copenhagen, where most of the 8,000 Jews in Denmark
lived, and other cities, by train, car, and on foot. With the help of the Danish people, they found
hiding places in homes, hospitals, and churches. Within a two-week period fishermen helped ferry
7,220 Danish Jews and 680 non-Jewish family members to safety across the narrow body of
water separating Denmark from Sweden.
The Danish rescue effort was unique because it was nationwide. It was not completely successful,
however. Almost 500 Danish Jews were deported to a ghetto in Czechoslovakia. Yet even of these
Jews, all but 51 survived the Holocaust, largely because Danish officials pressured the Germans
with their concerns for the well-being of those who had been deported. The Danes proved that
widespread support for Jews and resistance to Nazi policies could save lives.

50

Resistance in Denmark
Directions:
Answer the following questions based on the Resistance in Denmark passage.
1. How many Jews were helped by the fishermen?
a) 680
b) 7,220
c) 7,900
d) 8,000
2. Why did the Danes ferry the Jews to Sweden?
a) The Nazis ordered it.
b) The Danish army was strong.
c) The Swedes were politically neutral.
d) The Jews had relatives living there.
3. The authors main purpose in this passage was to show that_____________.
a)
b)
c)
d)

the Danes were frightened of the Nazis

not all Germans wanted to persecute the Jews
resistance to the Nazis could have saved the Jews
during the Holocaust most people didnt help the Jews

4. Where were almost 500 Danish Jews deported to?

a) a city in Sweden
b) a concentration camp
c) a town in Czechoslovakia
d) a ghetto in Czechoslovakia
5. The Danes were successful in protecting their Jewish citizens because they ______________.
a) had a lot of boats
b) were unified in their efforts
c) fought hard against the Nazis
d) were allies with the United States

51

6. Which of the following statements would the Danes agree with?

a) All people are equal.
b) Laws must always be obeyed.
c) Its too dangerous to fight injustice.
d) Nazi policies are none of my business.

52

Writing Prompt
Twilight & Resistance in Denmark

1. Denmark was one of the few countries to actively resist the Nazi policies. What have
you learned from the passages that might explain this lack of action?
o Support your position with direct evidence and quotations from the passages.
2. Use the space below for planning your essay.
3. Use the sheets of paper attached to write your essay on.
(If you need additional paper, please ask)
Your essay should be:
1 2 pages.
Clear
Organized
Legible

(oral directions for clarification)

The Criteria your writing will be evaluated on are:

Development

Organization

Mechanics:

The extent to which ideas are

elaborated using specific and
relevant evidence

The extent to which the

response exhibits essay
structure and coherent focus.

The extent to which the

response exhibitsstandard
spelling, punctuation,
capitalization, grammar,
usage, and vocabulary

After 45 minutes, teacher says:

You should be finishing your MC and be starting your writing prompt now. You
have 45 minutes left to finisher your test.

53

______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
54

______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
55

Bibliography
Anderson, N., Bachman, L., Perkins, K., & Cohen, A. (1991). An exploratory study into
the construct validity of a reading comprehension test: triangulation of datasource.
Language Testing, 8(1), 41-65.
Bachman, L., & Palmer, A., (1996). Language testing in practice: Designing and
developing useful language tests. New York: Oxford University Press.
Ballard, B., & Clanchy, J. (1991). Assessment by Misconception: Cultural Influences and
Intellectual Traditions. In Hamp-Lyons, L. (Ed.), Assessing second language
writing in academic contexts. New Jersey: Ablex Publishing Corporation.
Bereiter, C., & Scardamalia, M. (1987). The Psychology of Written Composition. New
Jersey: Lawrence Erlbaum Associates, Publisher.
Connor, U., & Mbaye, A. (2002). Discourse approaches to writing assessment. Annual
Review of Applied Linguistics. 22, 263-278.
Cummings, A. (2001). ESL/EFL instructors practices for writing assessment: specific
purposes or general purposes? Language Testing. 18(2), 207-224.
Douglas, D. (2002). Assessing Languages for specific purposes. New York: Cambridge
University Press.
Grabe, W. (1991). Current development in second language reading research. TESOL
Quarterly, 25(3), 375-406.
Hill, C., & Parry, K. (1994). Models of literacy: The nature of reading tests. In C. Hill &
K. Parry (Eds.), From testing to assessment. New York, Longman Group Limited.
Horowitz, D. M. (1986). What professors actually require: Academic talks for the ESL
classroom. TESOL Quarterly, 20(3), 445-462.
Johns, A. (1991). Faculty Assessment of ESL Students Literacy Skills: Implications for
Writing Assessment. In Hamp-Lyons, L. (Ed.), Assessing second language
writing in academic contexts (pp 167-179). New Jersey: Ablex Publishing
Corporation.
Lennon, R. (1970). What can be measured?. In Farr, R. (Ed.) Measurement and
evaluation of reading. New York: Harcourt, Brace and World, Inc.
Farr, R. (1970). Measurement and evaluation of reading. New York: Harcourt, Brace and
World, Inc.
Marshall, J., & Campbell, Y. (2006). Practice makes permanent: Working toward fluency.
In J. Schumm (Ed.), Reading assessment and instruction for all learners. New
York: The Guilford Press.
Medina, A., & Pilonieta, P. (2006). Once upon a time: Comprehending narrative text. In J.
Schumm (Ed.), Reading assessment and instruction for all learners. New York:
The Guilford Press.
Reid, J. M. (1989). English as a Second Language Composition in Higher Education: The
Expectations of the Academic Audience. In Johnson, D. M., & Roen, D. H. (Eds.),
Richness in writing: Empowering ESL students. New York: Longman Press.
Silva, T., Reichelt, M., Chikuma, Y., Duval-Couetil, N., Ruo-Ping, J.M., Vlez-Rendn,
G., & Wood, S. (2003). Second language writing up close and personal: Some
56

success stories. In Kroll, B. (Ed.) Exploring the dynamics of second language

writing. New York: Cambridge University Press.
Spack, R. (1988). Initiating ESL students into the academic discourse community: How
far should we go? TESOL Quarterly, 22(1), 29-51.
Smith, F. (1985). Reading without nonsense. New York: Teachers College Press.
Vhpassi, A. (2002). On the specification of the domain of school writing. Evaluation in
Education, 5, 265-289.
Weigle, S. C. (2002). Assessing writing. New York: Cambridge University Press.
2003 in Second language writing up close and personal by Kroll
Rubric for Mid-term Projects
Introduction and overview

Lit review part 1

Lit review part 2

3 3.5 4

Lit review part 3

2 2.5 3

TLU domain

Test development and administration

1
2
(Design Statement, Operationalization, Administration Procedures)

3 3.5 4

Formatting (headings, tables, etc.)

Language (typos, grammatical errors, register, APA issues) (weighted double)

The Test (weighted double)

6 7

10

4
6
8
Total: 38.5 /55

10

Seth and Shawn,

This needs a lot of work. First off, you didnt exactly follow the directions and format of
the paper. You skipped the TLU domain section, and included a lot of unnecessary info in
the design statement and operationalization, and left other info out that you should have
included. The lit review needs a lot of revision as wellsee our comments. The first part
of the lit review was especially difficult to follow. There were also too many typos and
APA citation lapses (and lack of proofreading. Remember, this is only 20% of your final
grade, so I expect that the final project will be much stronger.
Elvis

57