Tips for Writing Questionnaire Items

Office of Educational Assessment Last Updated: October 2006

ThishandoutisprovidedbytheUWOfficeofEducationalAssessment(OEA)asaresourceforWebQ workshopattendeeswhowantsomebasictipsforwritingquestionnaireitems.Itdoesnotaddress broaderaspectsofsurveyandresearchdesign,butadditionalresourcescanbefoundonourwebsite: http://www.washington.edu/oea.OEAalsoprovidesavarietyofconsultingandresearchsupport services;pleasefeelfreetocontactusifyouareinterestedinassistancewitharesearch,assessment,or evaluationproject(baytam@u.washington.edu,2065435190).

TIP 1: BE RELEVANT
Beforestartingyoursurvey,youshouldspecifyoneorseveraloverarchingquestionsyouwishto address(i.e.,researchquestionsorevaluationquestions).Yourquestionnaireitemsshouldbefocused aroundthesequestions.

Different Domains of Questions: Behavior, Beliefs, and Evaluation

Doesprovidingoptional,onlinepracticequizzeshelpstudentslearnthematerialinmycourse? Behavior Of the ten optional online practice quizzes, how many did you complete? Completing the online practice quizzes helped me learn the material in this course. How satisfied were you with the online quizzes?
0 12 3 5 67 8 9 10

Beliefs

Strongly Disagree Extremely Dissatisfied

Disagree

Neutral

Agree

Strongly Agree Extremely Satisfied

Evaluation

Somewhat Dissatisfied

Somewhat Satisfied

Thethreeitemsabovehavedifferentfoci.Thefirstcouldhelpansweryourresearchquestionifyouwere abletocorrelateresponsestotheitemwithactualgrades.

Copyright2006UniversityofWashingtonOfficeofEducationalAssessment

Different Types of Questions: Open- and Closed-Ended

HowdotransferstudentsbenefitfromgoingtoTransferThursdaysessionsintheGatewaycenter? Closedended: WhichofthefollowingdoyouseeasbenefitsofattendingTransferThursday sessions?(checkallthatapply) o o o o Openended: Learningaboutmajorrequirements MeetingwithaUWadviser Meetingothertransferstudents Other(specify):__________________

WhatdoyouseeasthebenefitsofattendingTransferThursdaysessions?

Datafromanopenendedquestioncanhelpyoucreatecheckallthatapplylistsorotherclosedformat items.Theprimarydrawbackofopenendedquestionsisthetimeittakestoanalyzethedata;if resourcesarelimited,useopenendedquestionssparingly.

TIP 2: BE SPECIFIC Avoid abstract terms and jargon

BAD GOOD

The in-class activities in this class facilitate active learning. The in-class activities (e.g., illusory correlation demonstration, eye-witness testimony activity) help keep me engaged in class.

Strongly Disagree Strongly Disagree

Disagree

Neutral

Agree

Strongly Agree Strongly Agree

Disagree

Neutral

Agree

Intherevision,thetermsinclassactivitiesandactivelearninghavebeenspecified.

Provide clarifying details

BAD GOOD

How many times did you visit CLUE? How many times did you visit the Center for Learning and Undergraduate Enrichment (CLUE) -evening tutoring in Mary Gates Hall?

Writeoutacronymsorabbreviationsthe firsttime.Providefurtherdescriptorsif appropriate.

Frequency estimates
BAD GOOD

How many times do you visit CLUE? During a typical quarter, how many times do you visit CLUE?

Never Never

Not often 1-3 times

Fairly Often About every other week

All the time About once per week 2 or more times per week

Thelessdefinitetheresponseoptions,themorenoiseorerrorinthedata(i.e.,participantswillvaryin theirinterpretationoftheoptions).

UWOEA:TipsforWritingQuestionnaireItems

TIP 3: AVOID CONFUSION Avoid double-barreled questions

BAD GOOD

The TAs in this course were helpful and responsive.

Strongly Disagree

Disagree

Neutral

Agree

Strongly Agree

Rate the performance of the TAs on the following dimensions. Helping you understand the material Responding to inquires about your grade
Poor 1 Poor 1 Excellent 5 Excellent 5

2 2

3 3

4 4

Avoid double-negative wording

BAD GOOD

The instructor for this course did not respond to students' comments during class. The instructor for this course was response to students' comments during class.

Strongly Disagree Strongly Disagree

Disagree Disagree

Neutral Neutral

Agree Agree

Strongly Agree Strongly Agree

Inthefirstcase,studentswhothoughttheinstructorwasresponsive(apositivetrait)wouldhaveto choosestronglydisagree(anegativeanswer).Watchoutforsuchmentalpuzzles.

TIP 4: USE APPROPRIATE SCALES Provide a spectrum of options

BAD GOOD

Did you enjoy the interactive online activity about Newton's 3rd Law? To what extent did you enjoy the interactive online activity about Newton's 3rd Law?

No Not at all

Yes A little Somewhat Very much

Typical scales
Agreement Satisfaction Evaluation
Strongly Disagree Extremely Dissatisfied Poor Very unfair Not at all important Disagree Dissatisfied Fair Unfair Not too important Not too familiar Neutral Somewhat Dissatisfied Good Fair Somewhat important Somewhat familiar Agree Somewhat Satisfied Very good Very fair Very important Very familiar Strongly Agree Satisfied Excellent Extremely Satisfied

Knowledge

Not at all familiar

UWOEA:TipsforWritingQuestionnaireItems

Unipolar vs. Bipolar scales

Unipolar Bipolar
None Extremely negative Some Very negative Very Somewhat negative Extremely Somewhat positive Very positive Extremely positive

Often,thepresenceofoneattributedoesnotnecessitatetheabsenceofanother.Forexample,onecan enjoybothartsandsciences,sotheyshouldnotbepresentedwithascalethatforcesthemtochooseone ortheother.Bewareofimposingastructureormodelthatmaynotbevalid.

Providing a (neutral) midpoint

With midpoint Without midpoint
Extremely negative Extremely negative Very negative Very negative Neither negative nor positive Somewhat negative Somewhat positive Very positive Very positive Extremely positive Extremely positive

Opinionsvaryregardingtheappropriatenessorusefulnessofamidpoint,particularlyforagreement scales.Researchsuggestsaforcedresponsewillyieldroughlythesameproportionofagree/disagreeasa scalewithamidpoint.Forexample: Resultswithamidpoint: Resultswithoutamidpoint: 30%con,50%neutral,20%pro 60%con,40%pro

Number of response anchors

Shouldonelabelalloftheresponseoptions?Ingeneral,themorespecificyouareabouttheresponse options,thelessyouwillhavetoworryaboutidiosyncraticinterpretationsofthoseoptionsbythe respondents.Insomeinstances(e.g.,frequencyestimates)itisimperativethatyoulabelalloptions.In othercases,youcouldincreaseerrorbytryingtoforcealabel.Whateversystemyouchoose,makesure thatyourlabelingissymmetrical(e.g.,donotlabelonly1and2and5ona5pointscale).

N/A: Use sparingly!

NO

I would have completed the online practice quizzes, even if they did not count towards my grade. The optional online practice quizzes improved my grade on the midterm.

Strongly Disagree Strongly Disagree

Disagree

Agree

Strongly Agree Strongly Agree

N/A

OK

Disagree

Agree

N/A

Anotheralternativewouldbetodirectrespondentstoskipasetofitemsiftheyhadnotcompletedanyof theoptionalonlinepracticequestions,makingtheN/Aoptionunnecessary. Notapplicableanddontknowshouldbeavoidedforopinionandattitudeitems.

"Check all that apply" lists

Makelistsasexhaustiveaspossible.Usepreviouslycollectedopenendeddataifavailable.Unlessall optionsareknown,itisalmostalwaysappropriatetoprovideaspaceforawriteinotherresponse.

UWOEA:TipsforWritingQuestionnaireItems

TIP 5: AVOID BIAS Avoid leading questions

BAD IFFY GOOD BEST

The UW is considering having wireless capacity for the entire campus within 3 years. Given the potential impact on students' behavior in the classroom, what do you think of this proposal? How do you think all-campus wireless capacity will affect students' behavior in the classroom? How, if at all, do you think all-campus wireless capacity will affect students' behavior in the classroom? Do you think all-campus wireless capacity will affect students' behavior in the classroom? Definitely not Possibly Definitely yes If yes or possibly, in what ways do you think it will affect students' behavior?

Inthefirstexample,thereisanimplicitassumptionthatexpandedcapacitywillaffectstudentsbehavior inclass.Thesecondwordingisbetter,butitstilldoesnotmakeclearthatrespondentsarefreetodeny anyeffect;thethirdwordingprovidesthatoption.Inthelastexample,theadditionofaclosedformat introductoryquestionallowsforeasytabulationofoverallopinionsfromtheopenendedquestion.

Beware hidden assumptions and contingencies

BAD GOOD

During the past quarter, how often have you skipped evening social events to study? In the course of a typical seven-day week, how many hours do you spend doing course-related work outside of class? In the course of a typical seven-day week, how many hours do you spend socializing with friends for non-academic purposes?

Theoverallquestionorobjectiveistofindouthowstudentsindifferentmajorsbalancetheirsociallives andstudytime.Thefirstexampleassumesthattherespondentwouldnormallytakepartinsocial events.Furthermore,thereisanimplicationthatallsocialeventstakeplaceintheevening,thatonly eveningsocialeventscount,orthatrespondentstypicallyonlystudyduringtheevening.Finally,the respondentwouldbeobligedtothinkbackandaverageoveranentirequarter.

REFERENCES/RESOURCES
Bradburn,N.M.,Sudburn,S.&Wansink,B.(2004).Askingquestions:Thedefinitiveguideto questionnairedesign.SanFrancisco,CA:JosseyBass. Czaja,R.&Blair,J.(2005).Designingsurveys:Aguidetodecisionsandprocedures.ThousandOaks, CA:SagePublications. Fowler,F.J.(2002).Surveyresearchmethods,3rdEd.ThousandOaks,CA:SagePublications. Fowler,F.J.(1995).Improvingsurveyquestions:Designandevaluation.ThousandOaks,CA:Sage Publications. Peterson,R.(2000).Constructingeffectivequestionnaires.ThousandOaks,CA:SagePublications

UWOEA:TipsforWritingQuestionnaireItems