DEVELOPING AND USING RUBRICS IN LANGUAGE ASSESSMENT James Dean Brown University of Hawaii at Mnoa

A. What is a rubric? According to Brown (2012): The term rubric has existed in English for more than 600 years and, during some of that time, it has meant a set of printed rules or instructions (Encarta Encyclopedia, 2004). However, more recently, rubric has come to be used widely in education. In the classroom, rubric may mean a set of categories, criteria for assessment, and the gradients for presenting and evaluating learning. When grading a students essay, for example, a teacher may apply a rubric for its quality of organization, giving a 3 for Advanced Proficient, 2 for Proficient, and a 1 for Partially Proficient (Cooper & Gargan, 2009, p. 54). A rubric in language teaching is typically a grid set up in one of two ways: (a) with scores along one axis of the grid and language behavior descriptors inside the grid for what each score means in terms of language performance or (b) with language categories along one axis and scores along the other axis and language behavior descriptors inside the grid for what each score within each category means in terms of language performance. B. What sorts of assessment benefit most from rubrics? C. How can we create an analytic rubric? D. How can we create a holistic rubric? E. Are there useful online tools that can be used to develop rubrics? F. Are there other examples of rubrics in the literature for say group assessment, other types of writing assessment, reading fluency, etc.? G. What are some useful referencs for further reading? H. If there is time: What are formal steps should be considered in rubric development? I. If there is time: How might task-dependent and task-independent rubrics differ?

J. If there is time: Are rubrics just a North American thing?

Table 2: Advantages, Disadvantages, & Item Writing Guidelines for Personal-Response Assessments _____________________________________________________________________________________________ Overall Advantages - Personal assessment; Directly related to and integrated into curriculum; Appropriate for assessing learning processes Disadvantages - Difficult to create and structure; Scoring is subjective Overall Guidelines - Make sure students have a voice in everything that takes place Conferences Advantages - Can help students understand learning processes and learning strategies; Help students develop better self-images; Teachers can elicit specific skills or tasks that need review; Can be used to inform, observe, mold, and gather information about Ss Disadvantages - Requires a lot of time; Difficult to use for grading purposes; Typically not scored or rated at all Guidelines 1. insure that students feel in control of conference 2. focus discussion on the students worries and views about the learning process 3. consider focusing students on self-image issues 4. consider eliciting performances on specific tasks or skills that need to be reviewed 5. hold conferences regularly and frequently consider scoring and grading conferences by applying rubrics Advantages - Enhances student learning; teachers role; assessment process; curriculum Disadvantages - Design decision problems; Logistical problems; Interpretation problems; Reliability problems; Validity problems Guidelines 1. decide with students in advance who will do what 2. introduce and explain portfolios to the students 3. have students select and collect meaningful work 4. periodically have students reflect on their portfolios 5. periodically have students, teachers, outsiders, etc. review the portfolios 6. consider scoring portfolios by using rubrics

Portfolios

Self-assessments

Advantages - Takes less time; Involves students in the assessment process; Encourages student autonomy; Can increase learner motivation Disadvantages - Accuracy varies depending on skill levels and material involved; Higher level students may tend to underestimate their abilities; Prone to subjective errors; May function differently depending on the consequences Guidelines 1. decide in advance what aspect of their language performance they will be assessing 2. where suitable, develop a written rating scale for the learners to use in scoring (and decide on whether to use a holistic or analytic scoring) 3. plan in advance the logistics of how the students will score themselves 4. use directions that give the students descriptions of concrete linguistic situations that they can score in behavioral terms if at all possible 5. make sure the students understand the self-scoring they will do 6. have students score all of the items of one type in one session 7. consider having students score themselves on two different occasions or have another student (or the teacher) do the same scoring _____________________________________________________________________________________________

Table 5: Holistic Version of the Scale for Rating Composition Tasks

SCORES

DESCRIPTORS
Appropriate title, effective introductory paragraph, topic is stated, leads to body; transitional expressions used; arrangement of material shows plan (could be outlined by reader); supporting evidence given for generalizations; conclusion logical & complete. Essay addresses the assigned topic; the ideas are concrete and thoroughly developed; no extraneous material; essay reflects thought. Native-like fluency in English grammar; correct use of relative clauses, prepositions, modals, articles, verb forms, and tense sequencing; no fragments or run-on sentences. Correct use of English writing conventions; left & right margins, all needed capitals, paragraphs indented, punctuation & spelling; very neat. Precise vocabulary usage; use of parallel structures; concise; register good. Adequate title, introduction, & conclusion; body of essay is acceptable but some evidence may be lacking, some ideas arent fully developed; sequence is logical but transitional expressions may be absent or misused. Essay addresses the issues but misses some points; ideas could be more fully developed; some extraneous material is present. Advanced proficiency in English grammar; some grammar problems dont influence communication, although the reader is aware of them; no fragments or run-on sentences. Some problems with writing conventions or punctuation; occasional spelling errors; left margin correct; paper is neat and legible. Attempts variety; good vocabulary; not wordy; register OK; style fairly concise. Mediocre or scant introduction, or conclusion; problems with the order of ideas in body; the generalizations may not be fully supported by the evidence given; problems of organization interfere. Development of ideas not complete or essay is somewhat off the topic; paragraphs arent divided exactly right. Ideas getting through to the reader, but grammar problems are apparent and have a negative effect on communication; run-on sentences or fragments present. Uses general writing conventions but has errors; spelling problems distract reader; punctuation errors interfere with ideas. Some vocabulary misused; lacks awareness of register; may be too wordy. Shaky or minimally recognizable introduction; organization can barely be seen; severe problems with ordering of ideas; lack of supporting evidence; conclusion weak or illogical; inadequate effort at organization. Ideas incomplete; essay does not reflect careful thinking or was hurriedly written; inadequate effort in area of content. Numerous serious grammar problems interfere with communication of the writers ideas; grammar review of some areas clearly needed; difficult to read sentences. Serious problems with format of paper; parts of essay not legible; errors in sentence-final punctuation; unacceptable to educated readers. Poor expression of ideas; problems in vocabulary; lacks variety of structure. Absence of introduction or conclusion; no apparent organization of body; severe lack of supporting evidence; writer has not made any effort to organize the composition (could not be outlined by reader). Essay is completely inadequate and does not reflect college level work; no apparent effort to consider the topic carefully. Severe grammar problems interfere greatly with the message; reader cant understand what the writer is trying to say; unintelligible sentence structure. Complete disregard for English writing conventions; paper illegible; obvious capitals missing, no margins, severe spelling problems. Inappropriate use of vocabulary; no concept of register or sentence variety.

5 4 3 2 1

Table 6: Analytic Oral Presentation Rubric: Class presentations

Teacher Name: JD Brown Student Name: ________________________________________

CATEGORY Enthusiasm

4
Facial expressions and body language generate a strong interest and enthusiasm about the topic in others. Student is completely prepared and has obviously rehearsed. Speaks clearly and distinctly all (10095%) the time, and mispronounces no words. Uses vocabulary appropriate for the audience. Extends audience vocabulary by defining words that might be new to most of the audience.

3
Facial expressions and body language sometimes generate a strong interest and enthusiasm about the topic in others. Student seems pretty prepared but might have needed a couple more rehearsals. Speaks clearly and distinctly all (10095%) the time, but mispronounces one word. Uses vocabulary appropriate for the audience. Includes 1-2 words that might be new to most of the audience, but does not define them. Stands up straight and establishes eye contact with everyone in the room during the presentation.

2
Facial expressions and body language are used to try to generate enthusiasm, but seem somewhat faked.

1
Very little use of facial expressions or body language. Did not generate much interest in topic being presented.

Preparedness

The student is Student does not somewhat prepared, seem at all prepared but it is clear that to present. rehearsal was lacking. Speaks clearly and distinctly most ( 9485%) of the time. Mispronounces no more than one word. Uses vocabulary appropriate for the audience. Does not include any vocabulary that might be new to the audience. Sometimes stands up straight and establishes eye contact. Often mumbles or cannot be understood OR mispronounces more than one word. Uses several (5 or more) words or phrases that are not understood by the audience.

Speaks Clearly

Vocabulary

Posture and Eye Stands up straight, looks relaxed and Contact

confident. Establishes eye contact with everyone in the room during the presentation.

Slouches and/or does not look at people during the presentation.

Date Created: Jan 08, 2011 11:32 pm (UTC)

Created at http://rubistar.4teachers.org/

Some Useful References

Arter, J., & McTighe, J. (2001). Scoring rubrics in the classroom: Using performance criteria for assessing and improving student performance. Thousand Oaks, CA: Corwin/Sage. Bachman, L. F., & Savignon, S. J. (1986). The evaluation of communicative language proficiency: A critique of the ACTFL oral interview. Modern Language Journal, 70, 380-390. Brindley, G. (1991). Defining language ability: The criteria for criteria. In S. Anivan (Ed.), Current developments in language testing (pp. 139-164). Singapore: Regional Language Centre. Brown, J. D. (2005). Testing in language programs: A comprehensive guide to English language assessment (New edition). New York: McGraw-Hill. Brown, J. D. (Ed.) (2012). Developing, using, and analyzing rubrics in language assessment with case studies in Asian and Pacific languages. Honolulu, HI: National Foreign Languages Resource Center. Brown, J. D. (Ed.). (2013). New ways of classroom assessment (2nd ed.). Alexandria, VA: Teachers of English to Speakers of Other Languages. Brown, J. D., & Bailey, K. M. (1984). A categorical instrument for scoring second language writing skills. Language Learning, 34(4), 21-42. Brown, J. D., Hudson, T., Norris, J. M., & Bonk, W. (2002). Investigating second language performance assessments. Honolulu, HI: NFLRC. Buttner, A. (2007). Activities, games, assessment strategies, and rubrics: For the foreign language classroom. Larchmont, NY: Eye on Education. Campbell Hill, B., & Ekey, C. (2010). The next-step guide to enriching classroom environments: Rubrics and resources for self-evaluation and goal setting for literacy coaches, principals, and teacher study groups, K-6. Portsmouth, NH: Heinemann. Cooper, B. S., & Gargan, A. (2009). Rubrics in education: Old term, new meanings. Phi Beta Kappan, 19(1), 54-55 Fulcher, G. (1987). Tests of oral performance: The need for data-based criteria. ELT Journal, 41, 287-291. Glickman-Bond, J., & Rose, K. (2006). Creating and using rubrics in today's classrooms: A practical guide. Norwood, MA: Christopher-Gordon. Hutson-Nechkash, P. (2003). Help me write: Frames and rubrics for classroom writing success. Greenville, SC: Thinking Publications. Jacobs, H. L., Zinkgraf, S. A., Wormuth, D. R., Hartfiel, V. F., & Hughey, J. B. (1981). Testing ESL composition: A practical approach. Rowley, MA: Newbury House. Mertler, C. A. (2001). Designing scoring rubrics for your classroom. Practical Assessment, Research & Evaluation,7(25). Moskal, B. M. (2000). Scoring rubrics: what, when and how? Practical Assessment, Research & Evaluation, 7(3). Norris, J. M., Brown, J. D., Hudson, T. D., & Bonk, W. (2002). Examinee abilities and task difficulty in task-based second language performance assessment. Language Testing, 19(4), 396-418. Norris, J. M., Brown, J. D., Hudson, T., & Yoshioka, J. (1998). Designing second language performance assessments. Honolulu, HI: National Foreign Languages Resource Center, University of Hawaii at Mnoa (also available from ERIC: ED 451 701). North, B., & Schneider, G. (1998). Scaling descriptors for language proficiency scales. Language Testing, 15(2), 217-263. Popham, W. J. (1997). Whats wrong and whats right with rubrics. Educational Leadership, 55(2), 72-75. Tierney, R., & Simon, M. (2004). Whats still wrong with rubrics: Focusing on the consistency of performance criteria across scale levels. Practical Assessment, Research & Evaluation, 9(2). Upshur, J. A., & Turner, C. E. (1995). Constructing rating scales for second language tests. ELT Journal, 49(1), 312.

Appendix A Suggested Stages and Steps in the Rubric Development Process

Stages and Steps Stage 1 Planning Step 1.1 Define your goal. Step 1.2 Go to the source material. Step 1.3 Brainstorm. Step 1.4 Analytic or holistic? Step 1.5 Decide the categories. Step 1.6 Decide the range of scores your want to use. Stage 2 Design the Rubric. Step 2.1 Put scores on one axis. Step 2.2 Put the categories on the other axis. Step 2.3 Fill in the rubric descriptors for each score level. Stage 3 Planning the Assessment Procedures and Using the Rubric Step 3.1 Decide on the stimulus formats. Step 3.2 Decide on the response formats. Step 3.3 Write clear instructions. Step 3.4 Make sure the instructions and stimulus materials are ready. Step 3.5 Arrange for the mechanics of assessment. Step 3.6 Actually do the assessment. Step 3.7 Train raters to use the rubric. Stage 4 Evaluate the Reliability/Fairness of Your Rubric Stage 5 Evaluate the Quality of Your Rubric Step 5.1 Evaluate the validity of your rubric. Step 5.2 Evaluate the usability of your rubric. Stage 6 - Plan Feedback and Revise for Pedagogically Useful Ratings Step 6a Plan for student and teacher feedback. Step 6b Set up a cycle of revision and improvement.

Appendix C Common European Framework of Reference for Languages http://www.coe.int/T/E/Cultural_Cooperation/education/Languages/Language_Policy/Common_Framework_of_Reference/

Mahalo!