Pageiii

ResearchMethodsinAppliedSettings:
AnIntegratedApproachtoDesignandAnalysis
JeffreyA.Gliner
and
GeorgeA.Morgan
ColoradoStateUniversity

Pageiv

Copyright2000byLawrenceErlbaumAssociates,Inc.
Allrightsreserved.Nopartofthisbookmaybereproducedinanyform,byphotostat,microfilm,retrievalsystem,oranyothermeans,withoutpriorwritten
permissionofthepublisher.
LawrenceErlbaumAssociates,Inc.,
Publishers10IndustrialAvenue
Mahwah,NJ07430
CoverdesignbyKathrynHoughtalingLacey
LibraryofCongressCataloginginPublicationData
Gliner,JeffreyA.
Researchmethodsinappliedsettings:anintegratedapproachto
designandanalysis/byJeffreyA.GlinerandGeorgeA.Morgan.
p.cm.
Includesbibliographicalreferencesandindex.
ISBN080582992X(cloth:alk.paper)
1.SocialsciencesResearch.I.Morgan,GeorgeA.
(GeorgeArthur),1936.II.Title.
H62.G5232000
300'.7'2dc219938652
CIP
BookspublishedbyLawrenceErlbaumAssociatesareprintedonacidfreepaper,andtheirbindingsarechosenforstrengthanddurability.
PrintedintheUnitedStatesofAmerica
10987654321

Pagev

CONTENTS
vii

Preface
I.IntroductoryChapters

1
Definitions,Purposes,andDimensionsofResearch

2
ATaleofTwoParadigms:QuantitativeandQualitative

15

3
EthicalProblemsandPrinciples

31

4
ResearchProblems,Variables,andHypotheses

43

II.ResearchApproachesandDesigns

5
OverviewofResearchApproachesandQuestions

61

6
InternalValidity

81

7
SpecificResearchDesignsforRandomizedExperimentalandQuasi
ExperimentalApproaches

93

8
SingleSubjectDesigns
III.UnderstandingtheSelectionandUseofStatistics

109

9
MeasurementandDescriptiveStatistics

127

10
SamplingandExternalValidity

143

Pagevi

11
IntroductiontoInferentialStatisticsandProblemswithNullHypothesis
SignificanceTesting

165

12
GeneralDesignClassifications

181

13
SelectionofStatisticalMethods

197

IV.IntegratingDesignsandAnalyses:InterpretingResults
14
SingleFactorbetweenGroupsDesigns:AnalysisandInterpretation

213

15
SingleFactorwithinSubjectsDesigns:AnalysisandInterpretation

237

16
BasicAssociationalDesigns:AnalysisandInterpretation

251

17
AddingIndependentVariablesComplexbetweenGroupsDesigns:Analysisand
Interpretation

263

18
MixedFactorialDesigns:AnalysisandInterpretation

277

19
InterpretationofComplexAssociationalAnalysesandMANOVA

291

V.Measurement,Instruments,andProcedures

20
MeasurementReliabilityandValidity

309

21
TypesofDataCollectionTechniques

329

22
StepsintheResearchProcess:PracticalandEthicalIssues

345

VI.ResearchValidity,Replication,andReview

23
ResearchValidityandReplication

367

24
EvaluatingEmpiricalResearchStudies:ASynthesis

385

AppendixA
ConfusingTermsandGlossary

401

AppendixB
AnatomyofaResearchArticle

427

AppendixC
WritingResearchProblemsandQuestions

435

References

441

AuthorIndex

449

SubjectIndex

453

Pagevii

PREFACE
Inthisbook,wehaveprovidedanintegratedapproachtoquantitativeresearchmethodsandtotheselectionandinterpretationofdataanalysesforgraduatestudents
intheappliedbehavioralsciences.Thetargetdisciplinesincludeeducation,alliedhealth,andotherappliedbehavioralscienceareasaswellaspsychology,whichisour
trainingandbackgrounddiscipline.
Thebookoffersseveralunusualandvaluablefeatures.Thecontentisbasedontheconceptualizationsofrespectedauthorsofresearchmethodsbooksandarticles
(e.g.Cook&Campbell,Kerlinger,etc.),butwehavetriedtomakethisbookstudentfriendlyaswellassophisticated,partlybybeingconsistentandclearin
terminologyandpartlybyorganizingthematerialsothatthevariouschaptersareconsistentandfittogetherlogically.Manyauthorstreatthedifferentpartsoftheir
researchmethodsbooksasessentiallyunrelated.Forexample,samplingisdiscussedasifitonlyappliedtosurveyresearchinternalvalidityisdiscussedonlyin
relationtoexperimentalandquasiexperimentalresearchandoftenlittleattemptismadetoshowhowthereliabilityandvalidityofameasurement(testorinstrument)
arerelatedtoanddifferentfrominternalandexternalvalidity.Furthermore,chaptersonstatisticsusuallyseemunrelatedtothoseondesign,sostudentscantake
statisticsandhavelittleideawhenorwhytousethem.Inthisbook,wediscussindetailbothresearchdesignandinterpretationofstatisticalanalyses,andweshow
howtheresearchapproachanddesigndeterminetheappropriatestatisticalanalysis.However,thisisnotastatisticsbooksotherearefewformulasandcomputations.
Ourapproachtodesignandanalysisissomewhatnontraditionalbecausewehavefoundthatstudentshavedifficultywithsomeaspectsofstatisticsbutnotothers.
Mostcan"crunch"thenumberseasilyandaccuratelywithacalculatororwithacomputer.However,manyhavetroubleknowingwhatstatisticstouseandhowto
interprettheresults.Theydonotseemtohavea"bigpicture"orseehowtheresearchquestionsanddesigninfluencedataanalysis.Partoftheproblemisinconsistent
terminology.WeareremindedofBruceThompson's

Pageviii

frequentlyrepeated,intentionallyfacetiousremarkathismanynationalworkshops:"Weusethesedifferenttermstoconfusethegraduatestudents."Forthesereasons,
wehavetriedtopresentasemanticallyconsistentandcoherentbigpictureofwhatwecallresearchapproaches(experimental,comparative,andsoforth)andhow
theyleadtothreebasickindsofresearchquestions(difference,associational,anddescriptive)which,inturn,leadtothreekindsorgroupsofstatisticswiththesame
names.Werealizethattheseandotherattemptstodevelopanduseaconsistentframeworkarebothnontraditionalandsomewhatofanoversimplification.However,
wethinktheframeworkandconsistencypayoffintermsofstudentunderstandingandabilitytoactuallyusestatisticstoanswertheirresearchquestions.
Astrongfeatureofthebookistheemphasisonhelpingstudentsbecomegoodconsumersofresearchbyhelpingthemtoanalyzeandevaluateresearcharticles.The
lastchapter(24)integratesmanypointsfromearlierchaptersintoaframeworkfortheanalysisandevaluationofresearcharticlesbyusing16questionsandthesix
ratingscales,presentedinearlierchapters,toevaluateresearchvalidity,whichisthevalidityofawholestudy.Becauseevaluationofresearchstudiesisamajorgoalof
ourcourses,wehavefoundithelpfulforstudentstoreadchapter24partwaythroughthecourse,perhapsafterchapter5,aswellasattheend.Theapplication
problemsattheendofeachchapterarealsodesignedtohelpstudentsbecomegoodconsumersofresearch.
Asecondfeatureisthatthetextisheavilyannotatedwithexamplesfromourcombined60yearsofteachingandresearchexperience.Italsohasalargenumberof
diagramsandtablesthatsummarizevarioustopicsandshowinadifferentwayareasthatareoftenconfusingorprovedifficulttolearnwellenoughtoapply.Visual
learnersmayespeciallyappreciatethefiguresandtables.
Athirdfeatureisthat,althoughthebookisbasedprimarilyonthequantitativeresearchparadigm,chapter2dealsindepthwithhowthequalitativeor
naturalist/constructivistparadigmdiffersfromthequantitativeorpositivistparadigm.Inanumberofplaces,includingchapter2,thebookpointsoutthevalueof
qualitativeresearchandhowitshouldleadquantitativeresearcherstobemoreflexibleandtakeintoaccountthecriticismsofqualitativeresearchers.Wealsopoint
out,ofcourse,whatwethinkaretheweaknessesofthequalitativeparadigm.
Afourthfeatureofthebookisthedivisionofallquantitativeresearchquestions(andwethinkqualitativeresearchaswell)intofivecategoriesthatwecallresearch
approaches:
1.RandomizedExperimental(hasrandomassignmentofparticipantstogroupsandanactiveormanipulatedindependentvariable).
2.QuasiExperimental(hasanactiveormanipulatedindependentvariablebutwithoutrandomassignmentofparticipantstogroups).
3.Comparative(hasafewlevelsorcategoriesofanattributeornonmanipulatedvariablethatarecompared).
4.Associational(sometimescalledcorrelational,hastwoormorevariablesthatarerelatedorassociatedforthesamegroupofsubjects).

Pageix

5.Descriptive(researchthatanswersdescriptivequestionsusingonlydescriptivestatistics).
Complexstudiesmayusemorethanoneoftheseapproachesforexample,"survey"studiesoftenhavedescriptiveaswellascomparativeandassociationalresearch
questions.Wehavebeenabletofitallofthehundredsofstudiesthatweandourstudentshaveevaluatedintooneormoreofthesecategoriesbasedontheresearch
questionsthattheinvestigatoraskedandanalyzed.
Thiscategorizationofresearchor,moreaccurately,researchquestions,intooneoffiveapproacheshasbeenhelpfulforseveralreasons.Oneconcernsdiscussionsof
causeandeffect.Wefeelthatcausalquestionscanbeappropriatelyansweredonlywithwellcontrolledrandomizedexperimentsandtoalesserextentwiththequasi
experimentalapproach.Neitherthecomparativenortheassociationalapproachiswellsuitedtodealwithcauseandeffect,butwerealizethatsomecomplexstatistics,
suchascrosslagpanelcorrelationsandstructuralequationmodeling,mayprovideevidenceforcausalityfromnonexperimentalstudies.
Anotherreasonthatourclassificationofresearchapproachesishelpfulisthatonecanfollowtheresearchprocessfrompurposetoquestionorhypothesistodata
analysis.Forexample,ingeneral,theexperimental,quasiexperimental,andcomparativeapproachesusewhatwecall"differenceinferentialstatistics,"suchasthet
testoranalysisofvariance,whereastheassociationalapproachuses"associationalinferentialstatistics,"suchascorrelationandmultipleregression.Weknowthatall
parametricinferentialstatisticsarerelational(specialcasesofcanonicalcorrelation),butwethinkthatitishelpfuleducationallytomakethisdistinction,whichis
consistentwithourframeworkforresearchapproachesaswellaswiththeinformationfoundinmostbasicstatisticsbooks.
Afifthfeatureofthebookiswhatwecallthedesignclassificationthatisbasedonthethreemajortypesofdesign:betweengroups,withinsubjects,andmixed
designs.Thesegeneralclassificationsapplytothecomparativeapproachaswellastotheexperimentalandquasiexperimentalapproaches,whichhasbeenmore
traditional.Weshowthat,althoughthesethreetypesofapproachusethesamegeneraltypeofstatistics(e.g.,ANOVA),thespecificstatisticsforbetweengroups
designaredifferentfromthoseforwithinsubjectsandfromthoseinmixeddesigns.Wealsopointoutthattheassociationalapproachusesadifferentsetofstatistics
butthedataresemblethoseinawithinsubjectsdesign.
Althoughourbackgroundsareinpsychology,wehaveoverthelasttwentyormoreyearsworkedinandtaughtresearchcoursesinapplieddepartmentsthatinclude
occupationaltherapy,education,humandevelopmentandfamilystudies,andconsumerscienceandmerchandising,aswellaspsychology.Inaddition,wehavehadin
ourclassesstudentsfromhomeeconomics,business,musictherapy,socialwork,andcommunicationdisorders,tomentionafewofthemorecommonareas.Thus,
wefeelthatwehaveagoodgraspofthetypesofresearchproblemsfacedbymaster'sanddoctoralstudentsinthesediverseareasandhavedesignedabookthatwe
feelisuserfriendlyaswellassophisti

Pagex

cated.Wethinkthatwehaveagoodtextonresearchdesignanddataanalysisthatwillhelpgraduatestudentslearnaboutthisimportantareathatisoften
unnecessarilyscarytostudentsinappliedfields.
Wewanttoacknowledgetheassistanceofmanypersonswhosehelpandadvicehasbeencriticalinthedevelopmentofthisbook.Specialthanksgotoourvery
competentandhelpfulwordprocessor,LindaWhite.PhyllisBeardandRuthMcNealdidmuchofthewordprocessingforearlyversionsofthistext.Attimesour
manyfiguresandtableshavepushedthelimitsofthesoftware.MaeMackie,ChaoHsienYeh,andMeiHueiTsaydevelopedmanyofthefiguresinthetext.Joan
Anderson,DavidStephen,andLorettaTengdidthefirstdraftoftheglossary.Severalteachingassistantsandworkstudystudentshelpedwiththetasksrelatedtothe
book.Forexample,ShellyHaddock,NancyLeech,MauraMacPheeandSusanTungatewrotemanyoftheapplicationproblems.Weespeciallywanttothankthe
studentsinourclasseswhohaveusedseveralearlierversionsastheirtextandhaveprovidedextensivefeedback,muchofwhichwehavetriedtoincorporate.Finally,
wewanttothankourwives,GailandHildy,fortheirsupportandhelp.GailGliner,OrlandoGriego,HelenaChmuraKraemer,DavidMacPhee,PatSample,Celia
Walker,andseveralreviewersprovidedhelpfulfeedbackonvariouschaptersofthebook.
J.A.G.&G.A.M.
FORTCOLLINS,COLORADO

Page1

PARTI
INTRODUCTORYCHAPTERS

Page3

Chapter1
Definitions,Purposes,andDimensionsofResearch
WhatIsResearch?
Definition
PurposesofResearch
IncreasingKnowledgewithintheDiscipline
TheoryDevelopment
PracticalApplication
DevelopingResearchTools
UnderstandingResearchinOne'sDiscipline
ResearchDimensionsandDichotomies
TheoreticalVersusApplied
LaboratoryVersusField
SelfReportVersusResearcherObservation
QuantitativeVersusQualitativeResearch
PositivistVersusConstructivistParadigms
QuantitativeVersusQualitativeDataandDataCollection
QuantitativeVersusQualitativeDataAnalysis
RelationshipsBetweentheFourDichotomies
Summary
StudyAids
Concepts
Distinctions
ApplicationProblems

Page4

WhatIsResearch?
Definition
Whatisresearch?Manydefinitionshavebeengiven,includingasystematicmethodofgainingnewinformation,orawaytoanswerquestions.Smith(1981)suggests
thatresearchbebroadenedtodisciplinedinquiry,which
mustbeconductedandreportedsothatitslogicalargumentcanbecarefullyexamineditdoesnotdependonsurfaceplausibilityortheeloquence,status,orauthorityofits
authorerrorisavoidedevidentialtestandverificationarevaluedthedispassionatesearchfortruthisvaluedoverideology.Everypieceofresearchorevaluation,whether
naturalistic,experimental,survey,orhistoricalmustmeetthesestandardstobeconsidereddisciplined.(p.585)

Smith's(1981)definitionofdisciplinedinquiryisworthconsideringindetail.AninquiryisdefinedbyWebster(1972)asasystematicinvestigationofamatterofpublic
interest.Thepublicinterestpartdebatable,butthesystematicinvestigationiscertainlyapartofanydefinitionofresearch.Regardlessoftheparticularresearch
paradigmtowhichtheinvestigatoradheres,theremustbeunderlyingguidelinesforhowtheresearchistobecarriedout.
Intheexperimentalapproachwithinthesocalledpositivist,orquantitative,framework,theinvestigatorestablishesadetailedplanpriortothestudy.Thisplan
includesdetailssuchaswhotheparticipantsintheinvestigationwouldbe,howtheywouldbeselected,howthetreatmentgroup(s)wouldbeassigned,whatthe
treatmentwouldbe,howthetreatmentwouldbemeasured,andotherdetailsofthestudy.
Ontheotherhand,iftheinvestigatorusesaqualitativeframework,oftencalledconstructivist,ornaturalistic,therearestillgeneralguidelinestofollow.Although
manyoftheseguidelinescomeintoplayaftertheinvestigationbegins,theyareimportantforguidingtheresearch.
Smith(1981)statesthattheresearch''mustbeconductedandreportedsothatitslogicalargumentcanbecarefullyexamined"(p.585).Thetermconductedinthe
definitionimpliesthattheresearchmustbecarriedout.(Designingresearchservesnousefulpurposeiftheresearchisnotactuallydone).Also,theresearchmustbe
reported,thatis,writtenforajournalordeliveredasatalkataprofessionalmeeting.Thisdisseminationfunctionisimportantiftheresearchistobeexaminedby
othersindetail.Smith(1981)isreallysayingthatunlesstheresearchisconductedandreported,otherscannotexamineittodeterminewhether,givensimilar
circumstances,theywouldcometothesameconclusionastheinvestigators.
ThelastpartofSmith's(1981)definitionreferstothefactthattheresearchmuststandonitsownmerit.Itshouldnotmatterwhocompletestheresearch,how
eloquentitis,oreventhenatureoftheproblem.Iftheresearchhasbeendonesystematicallybyfollowingguidelineswithinaparticularresearchparadigm,andithas
beendisseminatedwithinaparticulardiscipline,thenthatresearchcanbetested,orverified,byothers.Althoughtherehavebeennumer

Page5

ousattemptstodefineresearch,wefeelthattheSmith(1981)definitionencompassesmost.
Whydowedoresearch?Tofindsomethingout?Whatisitthatwewanttofindout?Questionsrecentlyaddressedwithinthefieldofeducationareasfollows:Does
classsizeaffectstudentoutcomes?Iscooperativelearningmoresuccessfulthanindividualizedlearning?Islearningmathematicsapersonalconstructionorasocial
construction?Shouldstudentswithspecialneedsbemainstreamedintotheschoolsystem?Severalquestionsalsoneedtobeaddressedinthealliedhealthfields:Does
aparticulartreatmentwork?Arecertaincharacteristicsoftherapistsbetterthanothers?Whatmakesagoodtherapist?Issupportedemploymentmoresuccessfulfor
communityintegrationthanshelteredwork?
PurposesofResearch
Therearemanyreasonstoconductresearch,nottheleastofwhichistocompleteyourgraduateprogram.Onamore(orless)seriousnote,therationaleforlearning
aboutresearchservestwogeneralpurposes:(a)increasingknowledgewithinthediscipline,and(b)increasingknowledgewithinoneselfasaprofessionalconsumerof
researchtounderstandnewdevelopmentswithinthediscipline.Increasingknowledgewithinthedisciplineiscertainlyaworthyandnoblepursuit.Forexample,most
researchtextsacknowledgesomethingsimilartothefollowing"Theselectionofprogramsandtechniquesoftherapeuticinterventionforaparticularhandicapped
populationorhandicappingconditionmustbebasedonclearempiricalevidenceofeffectiveness"(Ottenbacher,1986,p.5).
IncreasingKnowledgewithintheDiscipline
Increasingknowledgewithinthedisciplinecantakemanydirectionsthreearediscussedhere.Thefirstofthesedirectionssupportsthetheoreticalbasisofthe
discipline.
TheoryDevelopment
Atheory,accordingtoKerlinger(1986),"isasetofinterrelatedconstructs(concepts),definitions,andpropositionsthatpresentasystematicviewofphenomenaby
specifyingrelationsamongvariables,withthepurposeofexplainingandpredictingthephenomena"(p.9).Forexample,purposefulactivityisaconstructwithinthe
theoryofoccupationwithinthefieldofoccupationaltherapy.Numerousstudieshavebeenpublishedattemptingtodemonstratethatiftheactivityis"purposeful,"the
individualperformingtheactivitywillbemoreinvestedintheactivity(e.g.,Bakshi,Bhambhani,&Madill,1991Kircher,1984Nelson&Peterson,1989).These
studiesmightbeconceptualizedwithinthefollowingresearchdesign.Twogroupsareformedthroughrandomassignment.Onegroupreceivesaconditionofexercise,
forexample,jumpinginplace.Theothergroupalsojumpsinplace,butdoessowithajumpropeandthegoalofdoingitwell.Attheendofagiventimeperiod,the
twogroupsaremeasuredforperformance,satisfaction,or

Page6

motivationtodetermineiftheexercisewithpurposeconditionwasdifferentthantheexercisewithoutpurposecondition.
PracticalApplication
Asecondapproachtoincreasingknowledgewithinthedisciplineinvolvesprovidingevidencefortheefficacyofatherapeutictechnique,acurriculum,oran
administrativechange.AnexampleofexaminingtherapeutictechniquescanbeseeninastudybyJongbloed,Stacey,andBrighton(1989).Theycomparedthe
effectivenessoftwoapproachesfortreatingcerebrovascularaccidentpatients:afunctionalapproachandasensorimotorintegrativeapproach.Afterrandomly
assigningpatientstooneoftwotreatmentgroups,bothgroupswereassessedforfunctioninselfcareandmobility,mealpreparation,andsensorimotorintegration
priortothetreatmentinterventionsandat4and8weeksaftertheintervention.Thestudyisexemplaryasamethodtotesttheeffectivenessofdifferenttherapeutic
interventions.
DevelopingResearchTools
Athirdapproachtoincreasingknowledgewithinthedisciplineinvolvescreatingmethodstoassessbehaviors.Forexample,Morgan,Harmon,andMaslinCole
(1990)developedanewstandardizedtestingprocedureandsetoftaskstoassessmasterymotivationinyoungchildren.Theprocedurewasdesignedtobeusefulfor
normallydevelopingchildrenandalsoforchildrenwhoareatriskfordevelopmentalproblems.Tocomparechildrenwithdifferentabilitylevels,anindividualized
approachtomeasuringmasterymotivationwasdeveloped.Thisapproachvariesthedifficultyofthetaskinaccordancewiththechild'sabilitylevel,sothateachchild
isgiventasksthataremoderatelydifficultforhimorher.Eachchild'smotivationisassessedwithonetask,fromeachofseveralgradedsetsofsimilartasks,thatis
challenging(nottoodifficultortooeasy)forthatindividualchild.Goodinterobserverreliabilityhasbeenobtainedforthetwokeymeasuresinthisprocedure:
persistenceattasksthatarechallengingandpleasureaftercompletingapartofthetask.Thevalidityofthesemasterytaskmeasureshasbeenconfirmedinseveral
ways.First,groupsofchildren(suchasthosewhoarehandicappedoratriskfordevelopmentalproblems)whohadbeenpredictedtoscoreloweronmastery
motivationmeasuresthantheappropriatecomparisongroupsdidscorelower.Second,thereweresignificantrelationshipsbetweenpersistenceattasksandseveral
othermeasuresthatwereexpectedtoreflectaspectsoftheconceptofmasterymotivationforexample,ratingsofpersistenceduringmentaltests,engrossmentduring
freeplay,andmotherandteacherratingsofmotivation.
A.G.Fisher(1995)providesanotherexampleofinstrumentdevelopmentwithherfunctionalassessmenttoolcalledtheAssessmentofMotorandProcessSkills
(AMPS).TheAMPSallowsthetherapisttoassesstheclientwhiletheclientisperformingataskofhisorherchoice,usuallyintheclient'shome.Thusthesettingfor
theAMPSisnatural,i.e.,ithasecologicalvalidity.However,whatisspecialabouttheAMPSisthatbyusingasophisticatedpsychometricmethodcalledRasch
analysis(A.G.Fisher,1993),theAMPScanbe(a)usedwithdifferenttherapists,(b)usedwithdifferenttasks,and(c)

Page7

analyzedwithparametricstatisticsbecausethedataapproximateanintervalscale.FisherhassetaboutthetaskofvalidatingtheAMPSthroughnumerousstudiesthat
examinecrossculturaldifferences(A.G.Fisher,Liu,Velozo,&Pan,1992),diagnosticdifferences(e.g.,Puderbaugh&A.G.Fisher,1992),andagedifferences
(Dickerson&Fisher,1993).
UnderstandingResearchinOne'sDiscipline
Formoststudents,theabilitytounderstandresearchinone'sdisciplinemaybemoreimportantthanmakingaresearchcontributiontotheprofession.Disseminationof
newknowledgeoccursfortheprofessionalthroughnumerousprofessionaljournals,workshops,andcontinuingeducationcourses,aswellasthroughpopular
literature,suchasdailynewspapers.Today'sprofessionalcannotsimplyrelyonthestatementsofaworkshopinstructorornewspapertodeterminewhatshouldbe
includedornotincludedforfutureinterventionintheclassroom,clinic,orcommunity.Evenjournalarticlesneedtobescrutinizedforweakresearchdesign,
inappropriatedataanalyses,orincorrectinterpretationoftheseanalyses.Thecurrentprofessionalmusthavetheresearchandreasoningskillstobeabletomake
sounddecisionsandtosupportthem.Inaddition,researchskillscanhelpprofessionalsineducationortherapeuticsciencestobebetterserviceprovidersbecause
theywillknowhowtoexaminetheirownschool,classroom,orclients,andtheywillnoteifimprovementinvariousareashasoccurred(e.g.,seeOttenbacher,1986).
Wepresentaframeworkforevaluatingarticlesinseveralchaptersandsummarizeitinthelastchapterofthebook.
ResearchDimensionsandDichotomies
Althoughwehavediscussedbrieflytheimportanceoflearningaboutresearch,littlehasbeensaidaboutthedifferentformsthatresearchmaytake.Several
dichotomies,ordimensions,areusedtodescriberesearch.
TheoreticalVersusApplied
Researchersinmostsocialsciencedisciplinesconductresearchwithaspecific,practicalapplicationinmind,suchastreatment,learningenhancement,orevaluation.
Theoreticalresearchisalsodone,asmentionedearlier,butmostoftheresearchprojectsweexamineinthisbookareattheappliedendofthisdimension.
LaboratoryVersusField
Aseconddimensionforexaminingresearchisthesetting.Thetermfieldtakesonmanydifferentmeanings.Fieldcouldbeaclinic,hospital,school,workplace,or
homesetting.Laboratoryimpliesastructuredsettingthatisnotwherethesubjectsorparticipantsusuallylive,work,orreceivetherapy.Inthesocialandhealth
sciences,alaboratorymostoftenreferstoaroomwithavideocameraandmicrophones(i.e.,asomewhatunnaturalsetting).Researchintheso

Page8

cialsciencesisusuallyslantedtowardthefieldendofthedimensiontobemoreecologicallyvalid,butlaboratorysettingsprovidebettercontroloverextraneous
variables.
SelfReportVersusResearcherObservation
Insomestudiestheparticipantsreporttotheresearcher(inwritingororally)abouttheirattitudes,intentions,orbehavior.Inotherstudiestheresearcherdirectly
observesandrecordsthebehavioroftheparticipant.Sometimesinstrumentssuchasheartratemonitorsareusedbyresearchersto"observe"theparticipants'
physiologicalfunctioning.
Mostresearchintheappliedsocialsciencesandineducationreliesontheselfreportsoftheparticipants.However,suchreportsarealwaysinfluencedbythefactthat
theparticipantsknowtheyareinastudytheymaywanttopleasetheresearcher,theymaywanttohidethings,ortheymayhaveforgottenornotknowsomethings.
Manyinvestigatorspreferresearcherobservedbehavioraldata,eventhoughthesedataalsohavepotentiallimitations.Ontheotherhand,sensitive,welltrained
interviewersmaybeabletoestablishenoughrapportwithparticipantstoalleviatesomeofthebiasesinherentinselfreports.
QuantitativeVersusQualitativeResearch
Thisisthemostconfusingandcontroversialdichotomy.Webelievethatthistopicismoreappropriatelythoughtofasthreerelateddimensions.Thefirstdimension
dealswithphilosophicalorparadigmdifferencesintheapproachtoresearch.Theseconddimension,whichiswhatmanystudentsmeanwhenreferringtothis
dichotomy,dealswiththetypeofdataanddatacollectionmethod.Thethirddimensionreferstothetypeofdataanalysis.Wethinkthatindistinguishingbetween
qualitativeandquantitativeresearch,thefirstdimension(i.e.,thephilosophicaldichotomycalledpositivistvs.constructivist)isthemostimportant.
PositivistVersusConstructivistParadigms1
Thesimilaritiesanddifferencesbetweentheseparadigmsarediscussedindetailinchapter2.Fornow,notethatastudycouldbetheoreticallypositivistic,butthedata
couldbesubjectiveorqualitative.Infact,thiscombinationiscommon.Ontheotherhand,aresearchermayembracetheconstructivistparadigm,butsomeofthe
supportingdatamaybequantitativeorobjective.Thus,thetypeofdataandeventhedataanalysisisnotnecessarilythesameastheresearchparadigm.However,
qualitativedata,methods,andanalysesoftenaccompanytheconstructivistparadigm,andquantitativedata,methods,andanalysesareusuallyassociatedwiththe
positivistparadigm.Theapproachofthisbooklies
1

Althoughwebelievethatthetermpositivistisnotanaccuratelabelformostquantitativesocialscientists,itiscommonlyusedbyqualitativeorconstructivistwriters.Italso
helpsusseparatethephilosophicalorparadigmdistinctionfromthedatacollectionandanalysisissues.Likewise,thetermconstructivistmaynotbethebestormostcommon
identifierforwhatisoftencalledthenaturalistorqualitativeparadigm,but,again,ithelpsusmakeimportantdistinctions.

Page9

withintheframeworkofthesocalledpositivistparadigm,buttheconstructivistparadigmremindsusthathumanparticipantsarecomplexanddifferentfromother
animalsandinanimateobjects.Thus,weshouldnotbeoverlydependentonthephilosophyandmethodsofnaturalscienceresearch.Althoughyoumayfindthis
paradigmdistinctionconfusingnow,itsmeaningshouldbeclearafteryoureadchapter2.
QuantitativeVersusQualitativeDataandDataCollection
Quantitativedataaresaidtobeobjective,whichindicatesthatthebehaviorsareeasilyclassifiedorquantified,eitherbytheparticipantsthemselvesorbythe
researcher.Examplesofquantitativedataarescoresonanachievementtest,timetorecovery,anddemographicvariablessuchasageandgender.Thedataare
usuallygatheredwithaninstrument(e.g.,atest,physiologicaldevice,orquestionnaire)thatcanbescoredreliablywithlittletrainingrequired.
Qualitativedataaremoredifficulttodescribe.Theyaresaidtobesubjective,whichindicatesthattheycouldbehardtoclassifyorscore.Examplesofqualitativedate
areperceptionsofpain,feelingsaboutwork,andattitudestowardschool.Usuallythesedataaregatheredfrominterviews,observations,ordocumentssuchas
biographies.Quantitativeorpositivistresearchersalsogatherthesetypesofdata,buttheyusuallytranslateperceptions,feelings,andattitudesintonumbersbyusing,
forexample,ratingscales.Qualitativeorconstructivistresearchers,ontheotherhand,usuallydonottrytoquantifysuchperceptions.Liketheauthorsofother
quantitativeorpositivistresearchbooks,webelievethattheapproachinthisbookisusefulfordealingwithbothqualitativeorsubjectivedataandquantitativeor
objectivedata.
Itisimportanttopointoutthatresearcherswithinboththequantitativeorpositivistparadigmandthequalitativeorconstructivistparadigmuseinterviewand
observationalmethodsandbothareinterestedinobjectiveaswellassubjectivedata.However,constructivistresearcherspreferopenendedinterviews,observations,
anddocumentssuchasdiaries.Positivistresearcherspreferstructuredinterviews(orquestionnaires),observations,anddocumentssuchasschoolorclinicrecords.
Wedescribedatacollectionmethodsinchapter21.
QuantitativeVersusQualitativeDataAnalysis
Theinterpretationandunderstandingofquantitativedataanalysisisamajorthemeofthisbook.Itisdealtwithindetailinchapters14to19,whichdiscussmanyofthe
mostcommoninferentialstatisticsandshowhowtheyarerelatedtotheapproachesanddesignsdiscussedinchapters5to8.Qualitativedataanalysisinvolvesvarious
methodsforcoding,categorizing,andassigningmeaningtothedata,whichareusuallywordsorimages.Thisbookdoesnotdealwithqualitativedataanalysis
techniquessuchascontentanalysis,butsomeresearcherswithinthepositivistparadigmdoso.Constructivistresearchersrarelyuseinferentialstatistics,butsometimes
theyusedescriptivestatistics(chap.9).
RelationshipsbetweentheFourDichotomies
Certainaspectsofthefourdimensionsordichotomiestend,inpractice,togotogether.Forexample,appliedresearchtendstobedoneinfieldornaturalset

Page10

tings,oftenbyusingselfreportsoftheparticipants.Qualitativeresearchisalmostalwaysconductedinthefield.Ontheotherhand,theoreticallyorientedresearch
tendstobedoneinthelabbyusingresearcherobservations.
However,thereisnotnecessarilyanassociationbetweenanyofthesefourdimensions.Forexample,appliedresearchcanbedoneineitherthelaborinthefield,by
usingeitherobservationorselfreport,anditcanbeeitherpositivist(quantitative)orconstructivist(qualitative).Tohelpyouvisualizethatthesedimensionscanbeand
oftenareindependent,wepresentFig.1.1,whichisadaptedfromHendricks,Marvel,andBarrington(1990).Examplesofresearchstudiesthatfallintoeachofthe
eightcellsinthecubewouldnotbehardtofind.
Summary
Researchisdefined,accordingtoSmith(1981),as"disciplinedinquiry,"whichmustbeconductedandreportedsothatitsargumentscanbecarefullyexamined.
Inquiryisasystematicinvestigationofamatterofpublicinterest.Inthesocalledlogicalpositivistorquantitativeframework,whichisoneofthetwomainresearch
paradigms,aspecificplanisdevelopedpriortothestudy.Intheconstructivistorqualitativeapproach,lessstructuresurroundstheuseofspecificguidelinesinthe
researchdesign.However,generalguidelinesarefollowedinqualitativeresearch.Allresearchmustbeconductedandreportedsothatitcanbetestedandverifiedby
others.
Therearetwomainpurposesforresearch:(a)toincreaseknowledgewithinone'sdiscipline,and(b)toincreaseknowledgewithinoneselfasaprofessionalconsumer
ofresearch.Threedirectionsaretakentoincreaseknowledgewithinadiscipline:expandingthetheoreticalbasisofthedisciplinetestingtheeffec

Fig.1.1
Aresearchcubeshowingtheconceptualindependenceoftheseresearchdimensions
ordichotomies.From"TheDimensionsofPsychologicalResearch,"byB.Hendricks,
M.K.Marvel,andB.L.Barrington,1994,TeachingofPsychology,17,79.Copyright
1990byLawrenceErlbaumAssociates.Adaptedwithpermission.(Note,thevertical
dimensionisdifferentfromHendricksetal.)

Page11

tivenessofpracticalapplications,anddevelopingresearchtools.Researchskillsarerequiredtoexamineresearchinone'sdiscipline.
Fourimportantresearchdimensionsare(a)theoreticalversusapplied,(b)laboratoryversusfield,(c)selfreportversusresearcherobservation,and(d)quantitative
versusqualitativeresearch.Thelatterdichotomyhasthreeaspects:thephilosophicalortheoreticalframework,thetypeofdataandhowdataarecollected,andthe
typeofdataanalysis.Althoughqualitativedataareoftencollectedwithintheconstructivistparadigm,thatisnotalwaysthecase.Qualitativedataanddataanalysiscan
beusedwithinthepositivistparadigm.Thefourdimensionsordichotomiesarerelatedinpractice,butconceptuallytheyareindependent.Itispossibletohaveastudy
thatfitsanycombinationofthefourdichotomies.
StudyAids
Thefollowingconceptsanddistinctionsarediscussedinthischapterandmanyoftheconceptsaredefinedintheglossary(AppendixA).Itwillhelpyoutolearnthe
materialifyouunderstandthemeaningofeachconceptandcancompareandcontrastthepaireddistinctions.
Concepts
Disciplinedinquiry
Research
Theory
Distinctions
Laboratoryversusfieldresearch
Positivistversusconstructivistparadigmortheoreticalframework
Producingknowledgeversusunderstandingresearchasaconsumer
Quantitativeversusqualitativedataanalysis
Quantitativeversusqualitativedataanddatacollection
Selfreportversusresearcherobservation
Theoreticalversusappliedresearch
ApplicationProblems
1.Thechapterintroducedseveraldifferentformsthatresearchcantake:theoreticalversusapplied,laboratoryversusfield,selfreportversusresearcherobservation,
andquantitativeversusqualitative.Usingthesedimensionsordichotomies,identifytheformofresearchforeachofthefollowingexamples.Rememberthatsome
researchprojectsusebothendsofadimension.Forinstance,manyprojectsincorporatebothquantitativeandqualitativedatacollectionmethodsothersuseboth
selfreportandobservationalmeasures.

Page12
2

a.Aresearcherwasinterestedtoknowifthereweredifferencesinthephysiologicalarousalofmenandwomenduringarguments.She recruited30couples,
andaskedthemtocometotheHappyFamilyCounselingCenter.Coupleswerecomfortablyseatedinanattractivelydecoratedroom.Theresearcherplaced
heartrateandbloodpressuremonitorsoneachpersoninthecouple.Theyweretheninstructedtoidentifyanddiscussaproblemareaintheirrelationshipfor
20minutes.Theresearcherrecordedtheheartrateandbloodpressureforeachindividualtodetermineifthereweredifferencesbetweenmenandwomenand
toimprovetherapy.
b.Atherapistwasinterestedinlearningthecharacteristicsofmarriagesthatwerebasedonequality.Shealsowantedtolearnwhatthebenefitsandcostsof
equalitywereforwomen,men,andtheirrelationship.Sheinterviewedcouplesintheirhomefor2hours,askingthemopenendedquestionsabouttheir
previousrelationships,thehistoryoftheirmarriage,theevolutionoftheiregalitarianfeelingsandbehaviors,andhowtheyhandledcommunication,conflict,sex
andintimacy,money,children,job,andlifestyleissues.Toanalyzethedata,shecodedtheconversationaccordingtocommonthemesthatemergedfromthe
interviews.
c.Amodelwasdevelopedtoexplainafamily'sresponsetoastressfulevent.Morespecifically,themodelwasdevelopedtoexplainafamily'sadaptationover
timegivenseveralvariables,suchasthenatureanddegreeofastressor,andthefamily'sresourcesandperceptions.Aresearcherisinterestedindeterminingif
thismodelappliestoaparticularcatastrophiceventthelossofone'shometoanenvironmentalcatastrophe.Theresearcherrecruitsfamiliesfromatownthat
recentlyexperiencedanearthquake.Membersofthesefamiliesaremailedaquestionnaireaboutparticularvariables(e.g.,familyresources,perceptionsofthe
stressorevent)thesemeasureshavebeenusedinpriorresearchofthemodel.
2.Aresearcherwasinterestedtolearnhowworkenvironmentinfluencesemployees'experienceofworkfamilyconflict,orthedegreetowhichtheirwork
responsibilitiesimpingedontheirhomeresponsibilitiesandviceversa.Theresearchergainedpermissionfromseveralcompanypresidentstocollectdatafrom
companyemployees.
a.Describehowtheresearchermightproceedifthiswereafieldresearchdesign.Whataboutalaboratorydesign?
b.Howmighttheresearcherusequalitativedatacollectionmethodstogaininformation?Howmightsheusequantitativedatacollectionmethods?
c.Describehowshemightuseselfreportmeasurestogathercertaininformation?Whataboutobservationalmethods?
d.Wouldthisresearchbeappliedortheoretical?
3.Aresearcherisinterestedtolearnthequalitiesofandstrategiesusedbydualearnercoupleswhoaresuccessfulinbalancingworkandfamilyre
2

TheauthorsusethepronounheandsherandomlythroughouttheApplicationProblemssections.

Page13

sponsibilities.Sheaskseachmemberofthecoupletocompleteseveralmeasuresofvariablesthatshebelieveswillbeparticularlyrelevant,suchascreativity,
optimism,andselfesteem.Shealsoplanstointervieweachcoupletolearnabouttheirstrategiesforbalancingworkandfamily.Theseinterviewswillbeginwiththe
question:"Whatisitaboutyouoryourlifethatyoubelievemostleadstoyoursuccessinbalancingworkandfamily?"
a.Whichofthepreviouslymentionedmethodsforcollectingdataisquantitative?Whichisqualitative?
b.Iftheresearcherusesqualitativemethodsofdataanalysisfortheinterview,howmighttheyconductthisanalysis?
4.Arecentlyhiredpresidentofauniversityiscommittedtoincreasingthenumberofminoritystudentswhograduatewiththeirbachelors'degrees.Thepresidentcalls
toarrangeameetingwithyou.Inthismeeting,thepresidentexplainsthatshewantsyou"todosomeresearchonthistopic."Sheexplainsthatsheisawareofother
universitieswhohavesetandachievedthisgoalinprioryears.Sheisalsoawarethatthisuniversityhasdevelopedseveralprogramsinprioryearsintheefforttoreach
thisgoal.Shewantsyoutoprovideherwithinformationthatwillhelpherdesignspecificinitiativesthataremostlikelytoproducetheresultsshewants.Whatisthe
Presidentaskingyoutodo?(Or,isthePresidentaskingyoutobeaconsumerorproducerofknowledge?)Whatkindofskillsmustshebelievethatyouhave?

Page15

Chapter2
ATaleofTwoParadigms:
QuantitativeandQualitative
Confusion
Paradigms
Axiom1:TheNatureofReality
PositivistVersion
ConstructivistVersion
ExplanationandComments
Axiom2:TheRelationshipofKnowertoKnown
PositivistVersion
ConstructivistVersion
ExplanationandComments
Axiom3:ThePossibilityofGeneralization
PositivistVersion
ConstructivistVersion
ExplanationandComments
Axiom4:ThePossibilityofCausalLinkages
PositivistVersion
ConstructivistVersion
ExplanationandComments
Axiom5:TheRolesofValuesinInquiry
PositivistVersion
ConstructivistVersion
ExplanationandComments
ThePositivistApproachtoResearch
TheConstructivistApproachtoResearch

Page16

TypeofDataDoesNotImplyaParticularParadigm
ARapprochementBetweenQualitativeandQuantitative
Approaches?
Summary
StudyAids
Concepts
Distinctions
ApplicationProblems
Confusion
Aquantitativeoraqualitativeparadigm,whichshouldIchoose?Thereisconfusioninthesocialandhealthsciencesaboutthedifferencebetweenquantitativeand
qualitativeapproachestoresearch.Muchoftheconfusionabouttheseparadigms,asindicatedinchapter1,comesfromequatingthemwiththewaydataare
collected.Investigatorsoftenassociatesubjectivemethodsofdatacollection,suchasopenendedinterviews,withthequalitativeapproachandobjectivemethodsof
datacollectionwiththequantitativeapproach.Aswewillsee,however,thetypeofdatacollectioncannotbeusedtodistinguishadequatelybetweenthetwodifferent
paradigms.
Paradigms
WhatIsaParadigm?
ThetermwascoinedbyThomasKuhn(1970)buthasbeeninconsistentlydefined(Komesaroff,1986).OneinterpretationofaparadigmbyKuhnwasthatofa
scientificcommunity,or,morespecifically,thebeliefsthatmembersofthatscientificcommunityshare.LincolnandGuba

Page17

(1985)refertoaparadigmasasystemofideas:''Weshallcallsuchasystematicsetofbeliefs,togetherwiththeiraccompanyingmethods,aparadigm"(p.15).Inour
viewaparadigmisawayofthinkingaboutandconductingresearch.Itisnotstrictlyamethodology,butmoreofaphilosophythatguideshowtheresearchistobe
conducted.Moreimportant,aparadigmdeterminesthetypesofquestionsthatarelegitimate,howtheywillbeanswered,andinwhatcontexttheywillbeinterpreted.
Therearecurrentlytwomajorparadigmswithinthesocialandhealthsciences.Oneparadigm,oftenreferredtoasthequantitativeparadigm,hasbeenthedominant
paradigmandisusuallyassociatedwiththesocalledscientificmethod.Qualitativetheorists(e.g.,Lincoln&Guba,1985)usethetermlogicalpositivisttoidentifythe
dominantparadigmhowever,thistermiseasilymisunderstoodandnotreallyrepresentativeofthequantitative,dominantparadigm(Phillips,1992).Philosophically,
thelogicalpositivistsdidnotagreewithmostquantitativeresearchersthatonecanidentifycausalrelationships,becausetheyarguedthattheserelationshipscouldnot
bedirectlyverifiedbysensoryexperience.Ontheotherhand,thepositivistsdidhaverespectfortraditionalscientificmethods(Phillips,1992).Itappearsthatitisto
thelatterthatLincolnandGuba(1985)referwhentheyassociatepositivismwiththedominantparadigm.Thus,eventhoughthetermpositivistisnottotallyaccurate,
we,too,willuseitwhenreferringtothedominantorquantitativeparadigm,inparttodistinguishitfromquantitativedatacollectionandanalysis,asdiscussedin
chapter1.ThepositivistcaseisdescribedbyPhillips(1992)asfollows:
Acasecanbemadethatforthepurposesofsocialscience,meaningsandintentionscanbeinvestigatedusingtraditionalscientificmethods.Thatis,itcanbearguedthatthere
isnoepistemologicaldifferenceinkindbetweengainingknowledgeabouttheotherobjectsofscienceandgainingknowledgeaboutmeaningsandintentions.Manybranchesof
sciencecanprovidecaseswheretheobjectsofinterestarenotdirectlyobservableormeasurable,butwheretheirpresenceisinferredfromwhatisobservable.Thisprocessis
hypothetical,anditisnotguaranteedtobesuccessfulbutitisselfcorrectivebyabootstrappingprocessinvolvingtestingandeliminationoferrors.(p.18)

Takenastepfurther,theimplicationisthatsciencethatis,thescientificmethodwillallowustodiscovertruthsabouttheworld,bothphysicalandsocial.
Theotherparadigm,referredtobyLincolnandGuba(1985)asthenaturalisticorconstructivistapproach,isusuallyassociatedwithaqualitativeapproachto
research.Phillips(1992)pointsout,"Humansarenotmerephysicalobjectsandtounderstandorexplainwhyapersonhasactedinaparticularmanner,themeaning
(ormeanings)oftheactionhavetobeuncoveredandtodothistherolesoflanguageandofsocialsymbolismsandvalueshavetobetakenintoaccount"(p.5).
LincolnandGuba(1985)statedfiveaxiomsthatfromtheirpointofviewasqualitativeresearchers,separatethetwoparadigms.Sincethattime,theyhavemodified
theirposition(E.G.Guba,personalcommunication,1995),butitis

Page18

instructivetoexaminetheseoriginalfiveaxiomsindetailtogainaclearerpictureofthedifferencesbetweenthetwoparadigms.Notethatwehavechangedtheword
naturalist,whichwasoriginallyusedbyLincolnandGuba(1985),toconstructivistbecausetheformercouldbeconfusedwith"naturalism"or"naturalistic
interpretationofsocialscience,"whichsaysthatthesocialsciencesshouldhavethesamegoalsandmethodsasthenaturalsciences(Ary,Jacobs,&Razaviek,1996).
Axiom1:
TheNatureofReality
PositivistVersion
Thereisasingletangiblereality"outthere"thatcanbefragmentedintoindependentvariablesandprocesses,anyofwhichcanbestudiedindependentlyoftheothers
inquirycanconvergeontothatrealityuntilfinally,itcanbepredictedandcontrolled.
ConstructivistVersion
Therearemultipleconstructedrealitiesthatcanbestudiedonlyholisticallyinquiryintothesemultiplerealitieswillinevitablydiverge(eachinquiryraisesmorequestions
thanitanswers)sothatpredictionandcontrolareunlikelyoutcomes,althoughsomelevelofunderstandingcanbeachieved(Lincoln&Guba,1985,p.37).
ExplanationandComments
Theissueofrealitythatseparatesthepositivistparadigmfromtheconstructivistparadigmisthemostimportantoftheaxioms.Thenatureofrealityisanissuethat
seeminglycannotberesolvedortreatedonarelativescale.Eithertherearemanyrealitiesorthereisonereality,butbothpositionscannotexistatthesametime.We
thinkthatpartoftheissuecomesfromthelackofanagreedondefinitionofreality.Phillips(1992)pointsout:
First,thereisasimpleconfusionherebetween,ontheonehand,thefactthatdifferentpeopleanddifferentsocietieshavedifferentviewsaboutwhatisreal(afactthatseems
undeniable),andontheotherhandtheissueofwhetherornotwecanknowwhichoftheseviewsisthecorrectone(orindeed,whetherthereisacorrectoneatall).Fromthefact
thatwemightnotbeabletoreachanagreement(anepistemologicalmatter),itdoesnotfollowthatthereismorethanone"reality"(ontologicalmatter).(p.59)

Phillips(1992)followswithanexampleaboutthisdistinction,"Andcertainly,itdoesnotfollowfromthefactthatatribeofheadhunterssociallydeterminesitsown
reality,thatwetherebyhavetoacceptthatrealityastrue.Whatistrueifwehavedoneourresearchproperlyisthatthemembersofthattribeactuallydobelieve
intheirownrealities"(p.60).
Theissueleadstothefollowingquestion.Inthesocialsciences,isthepurposeofresearchtopredictandcontrol(understandthecauses)?Theanswertothisquestion
raisesakeydistinctionbetweenthetwodifferentparadigmsorbeliefsystemsonhowresearchoughttobecarriedout.Webelievethatifthepurposeofresearchisto
beabletoidentifycausesandpredictbehavior,thenamethodologyforcarryingoutthisresearchfromtraditionalscientificmethodsshouldbeundertaken.Onthe
otherhand,ifthepurposeofresearchistode

Page19

scribeparticipants'beliefsfully,thenamethodologybasedonconstructivismseemstobewelladaptedtothisproblem.
Considerthefollowingexample.AgrantfromtheKelloggFoundationwasgiventotheFortCollinscommunitythroughtheFamilyMedicineCenter(PoudreValley
PrenatalProgram)tohelpprovidemedicalcareforlowincomepregnantwomen.Fromtheperspectiveoftheprogramevaluator,oneofthegeneraloutcome
variablesthatneededtobeassessedwaswhetherthemethodofhealthcaredeliveryaffectsthehealthstatusofthemothersandinfants.Todeterminetheeffectofthe
intervention,theprogramevaluatorundertookapositivistapproachbyusingatimeseriesdesignandmeasuringoutcomessuchasbirthweight,Apgarscores,andso
forth.Ontheotherhand,aprogramevaluatoralsomightbeinterestedintheeffectoftheintervention(program),asseenthroughtheeyesoftheparticipants.Whatisit
likebeingaparticipantinthePoudreValleyPrenatalProgram?Toanswerthisquestion,aconstructivistapproachappearstobetheappropriatemethodofresearch.
Anethnographywasconducted,wheretheinvestigatorinterviewedparticipantsthroughindividualinterviewsandfocusgroups.Accordingtotheethnographer(Gerst,
1994),thereweremanydifferentviewpoints(realities),dependingonthecontext.
Quantitativeresearchershavealsorecognizedthatparticipantshavedifferentperspectives,orpointsofview,andreportthoseasvariability.Inaddition,quantitative
studiesoftenexaminefactorsthatarerelatedtoandperhapscausedifferentperceptions.Justbecauseapersonstatesabelief,doesnotnecessarilymeanthatitis
"real."Sometimespeoplelieorhidetheirbeliefsorareconfused.Itseemstous,astoPhillips(1992),thatitisimportanttoacknowledgethatpeoplehavedifferent
perceptions,butitisalsobesttoinvestigatewhyperceptionsseemtobedifferentandtobecautiousaboutassumingthatwhatonesaysiswhatonereallybelieves.
Axiom2:
TheRelationshipofKnowertoKnown
PositivistVersion
Theinquirerandtheobjectofinquiryareindependenttheknowerandtheknownconstituteadiscretedualism.
ConstructivistVersion
Theinquirerandthe"object"ofinquiryinteracttoinfluenceoneanotherknowerandknownareinseparable(Lincoln&Guba,1985,p.37).
ExplanationandComments
Axiom2isabouthowmuchtheinvestigatorandparticipantsinfluenceeachother.Thepositivistapproachtoresearchhasprideditselfonthenotionthatthe
investigatorisobjectiveduringtheexperiment.However,mostinvestigatorsrealizethattheycanbeaffectedbytheimpressionoftheparticipants.Acutebabyis
reactedtodifferentlythanafussyone.Ontheotherhand,participantswhoknowtheyarebeinginvestigatedmaychangeoractdifferently.Suchpsychological
phenomenaastheHawthorneeffect(participantsarelikelytochangeortobemore[orless]motivatedsimplybecausetheyknowoftheirparticipationinastudye.g.,
Roethlisberger&Dickson,1939)ordemandcharacteristics(participants'awarenessofthepurposeofthestudy)havebeenknownforyears.

Page20

Inmedicalresearch,placebosareusedsothatthesubjectswillnotknowwhethertheyareinthetreatmentorinthecontrolgroup.Anddoubleblindstudiesareused
sothatthephysicianwillnotknowwhoisgettingtheplacebo.Theissueforthepositivististodeterminehowmuchoftheoutcomemightresultfromtheseeffects.For
example,Drew(1980)suggestedthat"iftheHawthorneeffectcannotreliablyberemovedfromtheexperimentalgroup,themosteffectivecontrolmethodwouldseem
tobeto'Hawthorne'bothgroupsequally"(p.161).However,thesameissueisaproblemfortheconstructivistfamilyofapproaches.Howmuchdifferencedoesit
makeiftheobserverisaparticipantascomparedtoasilentobserver?WewouldarguethatAxiom2separatesthetwoapproachesmoreonarelativethanonan
absolutebasis.Thepositivistisusuallymoreconfidentthatbiascanbeovercome,butbothparadigmsneedtobesensitivetothisissue.
Axiom3:
ThePossibilityofGeneralization
PositivistVersion
Theaimofinquiryistodevelopanomotheticbodyofknowledgeintheformofgeneralizationsthataretruthstatementsfreefrombothtimeandcontext(theywillhold
anywhereandatanytime).
ConstructivistVersion
Theaimofinquiryistodevelopanideographicbodyofknowledgeintheformof"workinghypotheses"thatdescribetheindividualcase(everythingiscontextually
bound)(Lincoln&Guba,1985,p.38).
ExplanationandComments
TheissueforAxiom3concernsthegeneralizationoftheresultsfromtheindividualstudytopopulations,settings,treatmentvariables,andmeasurementvariables
(Campbell&Stanley,1966).Thetheoryunderlyingthisgeneralization(externalvalidityofastudy)comesfrompropersamplingtechniques.LincolnandGuba(1985)
andothersarecorrectintheirclaimsthatfewstudiesthattakeanempiricistorpositivistapproachhaveusedpropersamplingtechniques.Unfortunately,empirical
clinicalstudiesareevenmoresusceptibletotheseclaims.Usually,participantsarenotrandomlysampledfromatargetpopulation.Instead,participantsareusually
obtainedfromaconveniencesample(i.e.,getthemwhereyoufindthem).Theconveniencesampleisoftenfoundattheinvestigator'sclinic.
Thereispresentlylessconcernoverthegeneralizationissueamongpositivistsprimarilybecauseoftherelativelynewmethodologycalledmetaanalysis(Glass,
McGaw,&Smith,1981).Metaanalysisallowsresearcherstocombinetheresultsfromdifferentstudiesperformedinthesameareaofinterest(byusinganeffectsize
index).Effectsizeinitssimplestformforanystudyisdeterminedbysubtractingtheaverageresultsofthecontrolgroupfromtheinterventiongroupanddividingbythe
pooledstandarddeviationofbothgroups.Theresultyieldsaneffectsizeinstandarddeviationunits.Effectsizesthencanbecombinedwithother,similarstudies,
resultinginametaanalysis.Whenoneconsidersthattherearesomemetaanalyseswhereover200studieshavebeencombined(Lipsey,1992),thecasefor
generalizationgainsconsiderablesteam.

Page21

Theconstructivistoftenmakesnoclaimsforgeneralizingtheresultsbeyondwhatwasfoundinthestudy.Ifonebelievesthattheresultsofastudyarealwayscontext
specificandtheinvestigator'sbiasesplayalargeroleinthedatagatheringandinterpretation,thenitfollowsthattheresultsfromthestudycannotbegeneralizedto
othersituations.Thislaststatementiscontradictoryhowever,asoneofthemethodsusedbyconstructiviststoestablishcredibilityistohaveapeerexaminethedatato
determineifsimilarinterpretationsarefound.Manyofthequalitativestudieswehavereviewedhaveattemptedtomakeinferenceswellbeyondthespecificsofthe
findings.Again,wewouldarguethat,similartoAxiom2,Axiom3separatesthetwoapproachesmoreonarelativethanonanabsolutebasis.Furthermore,
researchersfrombothapproachesshouldbecautiousaboutgeneralizingtheirresults.
Axiom4:
ThePossibilityofCausalLinkages
PositivistVersion
Everyactioncanbeexplainedastheresultofarealcausethatprecedestheeffecttemporally.
ConstructivistVersion
Allentitiesareinastateofmutualsimultaneousshapingsothatitisimpossibletodistinguishcausesfromeffects(Lincoln&Guba,1985,p.38).
ExplanationandComments
TheissueforAxiom4becomesphilosophical.Foryears,positivistsbelievedthatundertheproperexperimentalconditionsonecouldconcludethattheindependent
variablecausedthechange(effect)inthedependentvariable.Theseproperconditionsalwaysincludedtherandomassignmentofparticipantstogroups.However,
fewpositivistswerewillingtomakeastrongstatementaboutthecausalnatureofthestudyevenif(a)theyusedtheproperexperimentalmethods,and(b)the
dependentvariablechangedinthepredicteddirectionafterthemanipulationoftheindependentvariable.Instead,theyfrequentlyobserved,"Thedataarehighly
suggestivethatvariableXcausedthechangeinvariableY."Furthermore,statisticalanalysesalonedonotallowforacausalstatement.Instead,outcomesareviewed
asprobabilitystatementsforexample,"TheprobabilitythatvariableXcouldhavecausedthechangeinoutcomeYislessthan5chancesin100,assumingthatthenull
hypothesisistrue."
Constructivistsruleouttheconceptofcausalityonmanygrounds,butperhapsthemostsalientisthatthepositivistsseecausalityinalinearfashion,whereasothers
suggestthatmosteventshavemultiplecauses.LincolnandGuba(1985)takeamuchstrongerstanceontheissueofcausality,suggestingthatitisimpossibleto
separatecauseandeffect.Instead,theyintroducetheconceptofmutualsimultaneousshaping.Theirideaisthat
everythinginfluenceseverythingelse,inthehereandnow.Manyelementsareimplicatedinanygivenaction,andeachelementinteractswithalloftheothersinwaysthatchange
themallwhilesimultaneouslyresultinginsomethingthatwe,asoutsideobservers,labelasoutcomesoreffects.Buttheinteractionhasnodirectionality,noneedtoproducethat

Page22
particularoutcomeitsimplyhappenedasaproductoftheinteractionmutualshaping.(pp.151152)

ThereisnowaytoresolvedifferencesbetweenthetwoparadigmsforAxiom4.Weareunsurewhetheronecouldplacethedifferencesonanabsoluteoronarelative
basis.However,ourviewsoncausalityaresimilartothepositiontakenbyDavis(1985):
"causation"isanotoriousphilosophicaltarpit.Oneoftheimportantprinciplesofthephilosophyofscienceisthattheworkingscientistdependsonformulationsthatdon'tquite
holdwaterunderthescrutinyofaprofessionalphilosopher.Thus,theideasIwillspelloutaretheunspokenassumptionsoftheprofessionalsocialscienceresearcher,notthe
honedformulationsoftheprofessionallogician.(p.8)

LincolnandGuba(1985)arecertainlycorrectinpointingoutthatmuchbehaviorisbothacauseandaneffect,thatthereismutualsimultaneousshapingofbehavior,
andthatcausesandeffectsaredifficult,ifnotimpossible,todistinguish.Also,itistruethatstudentsandscholars,nottomentionjournalistsandthepublic,aretoo
looseinusingwordslikecause,impact,anddeterminant.Infact,oneofthekeypointsthatwewanttomakeinthisbookisthatsuchwordsshouldbeusedwith
cautionandthenprobablyonlyaftercompletingastudywithatight,randomizedexperimentaldesign.
Axiom5:
TheRoleofValuesinInquiry
PositivistVersion
Inquiryisvaluefreeandcanbeguaranteedtobesobyvirtueoftheobjectivemethodologyused.
ConstructivistVersion
Inquiryisvalueboundinatleastfiveways,ascapturedinthecorollariesthatfollow:
Corollary1.Inquiriesareinfluencedbyinquirervaluesasexpressedinthechoiceofaproblem,evaluand,orpolicyoption,andintheframing,bounding,and
focusingofthatproblem,evaluand,orpolicyoption.
Corollary2.Inquiryisinfluencedbythechoiceoftheparadigmthatguidestheinvestigationintotheproblem.
Corollary3.Inquiryisinfluencedbythechoiceofthesubstantivetheoryusedtoguidethecollectionandanalysisofdataandintheinterpretationoffindings.
Corollary4.Inquiryisinfluencedbythevaluesthatinhereinthecontext.
Corollary5.WithrespecttoCorollaries1through4,inquiryiseithervalueresonant(reinforcingorcongruent)orvaluedissonant(conflicting).Problem,evaluand,
orpolicyoptionparadigmtheoryandcontextmustexhibitcongruence(valueresonance)iftheinquiryistoproducemeaningfulresults(Lincoln&Guba,1985,p.
38).

Page23

ExplanationandComments
Axiom5is,inpart,aboutintroducingbiasinresearch.Itisimportantforallresearcherstorecognizethatresearchisnotvaluefree.Considerthepositivistwho
assumesthathisorherresearchisobjective.Onemightask,"Whoselectedtheindependentvariable(s)orthedependentvariable(s)?Whoselectedthesampling
frame?Whoselectedtheparticulartreatment?"Ofcourse,theinvestigatorselectedallaspectsofthestudy.Therefore,theconstructivistsarecorrectinassumingthat
researchisnotvaluefree.Inaddition,apartofconstructivistresearchistostatethebiasoftheinvestigator.
Ontheotherhand,justbecausetheinvestigatormayhaveabiasinthestudydoesnotnecessarilymeanthatnothingcanbedoneaboutbiasorthattheoutcomewill
beinthedirectionoftheinvestigator'schoice.Thereareprobablyfarmorestudiesthatfailtofindsignificantdifferencesbetweentreatmentandcontrolgroupsthan
thosethatfindsignificantdifferences.Furthermore,replicationattemptsbyotherresearchershelptoinsurethatsomethingotherthantheinvestigator'sbiasisinfluencing
theresults.
Axiom5appearstoseparatethetwoparadigmsonarelativebasis.Thedegreeofsubjectivityinmostconstructiviststudies(theinvestigatoristheinstrumentto
measureoutcomes)seemstoususuallyfargreaterthanthatseeninthepositivistapproach.However,thisdifferenceisnotallornothing.Researchersinboth
paradigmsshouldacknowledgethattherewillbesubjectivityandbias.However,webelievethatresearchersshoulddowhattheycantominimizethepotentialeffects
ofbiases.
ThePositivistApproachtoResearch
Thepositivistapproachisbasedonthesocalledscientificmethod.Drew(1980)presentssixstepstotheresearchprocessthatarerelevantforeducation,health,and
humanservicesdisciplines.
1.Thefirststepinvolvestheresearchquestion.Kerlinger(1986)referstoitastheproblemobstacleideastage.Thisinitialstageinvolveschoosingaquestionthathas
thepotentialtoworkintoaresearchableproject.Wheredoestheproblemcomefrom?Formany,especiallythoseinapplieddisciplines,theproblemoftencomes
fromaclinicalsituation.Willaparticulartypeoftherapyleadtoimprovement?Willadaptivetechnologyincreasecommunicationskills?Willaparticularassessment
yieldtheinformationIneed?Anotherplacefromwhichproblemsmayariseisthepreviousliterature.Apublishedstudymayhelptoformulatequestionsthatleadtoa
newstudy.
2.ThesecondstagesuggestedbyDrew(1980)involvesdevelopinghypotheses.Briefly,theresearchproblemisdirectedintospecificquestionsthataretestable.
3.Thethirdstepisdevelopingaresearchdesignthatallowstheinvestigatortotestthehypotheses.Themajorfocusoftheresearchdesignistoallowtheinvestigator
tocontroloreliminatevariablesthatarenotofdirectin

Page24

teresttothestudy.Thisallowstheinvestigatortotestoranswertheresearchquestiondirectly.
4.Thefourthstageinvolvesdatacollectionandanalysis.Dataarecollectedinanunbiasedandobjectivefashion.Inthetraditionalmethod,thescientistdoesnot
examinethedataindetailuntilthestudyhasbeencompleted.Thedata,whichareusuallyquantitativenumbers,arethenanalyzedbyusingstatistics.
5.Thefifthstepinvolvesmakinginferencesorinterpretationsfromthedata.However,theseinterpretationsmustbebasedontheoriginalhypotheses.
6.Astatementisthenmadeastowhetherthehypothesesareacceptedorrejected.
Drew's(1980)exampleoftheresearchprocesstakestheformofafeedbackloop.Afterthelaststep,anewresearchquestionisasked.Ifthehypothesisis
confirmed,thenanewquestioncanbeaskedtogainadditionalinformation.Ifthehypothesisisnotconfirmed,thenewquestionmaybeamodificationoftheoriginal
question.
Thesesixsteps,whicharetypicalofthoseinmostsocialscienceresearchmethodsbooks,arediscussedinmoredetailinchapter22andthroughoutthisbook.It
shouldbenoted,however,thatquantitativeresearchoftenvariesfromthisidealizedformat,whichhasprovideda"strawman"forproponentsofthequalitativeor
constructivistparadigmtocriticize.Inpractice,thescientificapproachisnotasdeductive(literature/theory deducehypotheses testhypotheses)orasrigidas
implied.Forexample,interestingfindingsoftenemergeinthedataanalysisstagethatwerenotbasedontheoriginalhypotheses.
TheConstructivistApproachtoResearch
Theprocessoftheconstructivistapproachtoresearchalsoissequential,butitismoreflexiblethanthatofthepositivist.AccordingtoLincolnandGuba(1985),
"Designinthenaturalistic(constructivist)sense,asweshallsee,meansplanningforcertainbroadcontingencieswithout,however,indicatingexactlywhatwillbedone
inrelationtoeach"(p.226).Theconstructivistapproachalsostartswithaproblem.Itwouldbeinconceivabletostudyagroupororganizationwithouthavingany
thoughtastowhythestudyisbeingconducted.However,althoughtheproblemmaybewellthoughtoutatthetimeofdecidingonagrouptostudy,theconstructivist
researcherisopentothepossibilitythatamajorchangeinemphasisabouttheproblemcouldoccuratanytimeduringthestudy(emergentdesign).Inotherwords,
althoughtheconstructivistresearcherhasageneralideaofwhattobelookingforinthestudy,nospecifichypothesesareformedpriortotheonsetofthestudy.
AccordingtoLincolnandGuba(1985,pp.226249),ageneralsequenceorplanofconstructivistresearchwouldbeasfollows:
1.Determiningafocusfortheinquiryanddeterminingtheconstraintsoftheproblem.
2.Determiningthefitoftheparadigmtothefocus.

Page25

3.Determiningthefitoftheinquiryparadigmtothetheory.
4.Determiningwhereandfromwhomthedatawillbecollected.
5.Determiningsuccessivephasesoftheinquiry.
6.Determininginstrumentation.
7.Planningforcollectingandrecordingdata.
8.Planningdataanalysis(andinterpretationouraddition).
9.Planningthelogistics.
10.Planningfortrustworthiness.
Thefirstthreestepsaddresstheproblem.However,morethanjustknowingthataproblemneedstoberesearched,Steps2and3indicatethatadecisionmustbe
madeaboutwhethertheconstructivistparadigmisthebestmethodtoapproachtheproblem.
Step4addressestheparticulargroupthatwillbestudied,butmorespecifically,whointhegroupwillprovidetheinitialinformation.Steps4,5,and6aregeneralin
thattheconstructivistresearcherisawarethatoncethestudyunfolds,differentpersonswillneedtobestudiedwhowerenotspecifiedpriortotheinitiationofthe
study.Inaddition,theconstructivistresearcherhasnowayofknowingwhatwillhappenafterthestudyisinitiatedasfarasplanningfurtherstages.Whereasthe
positivistresearcherplansastartandendtothestudy,theconstructivistresearchercannotknowinadvancehowlongthestudywilltakeorinwhatdirectionthestudy
willgo.However,LincolnandGuba(1985)suggestthreegeneralphasesofanyconstructivistinquiry:focusingonwhattheinvestigatorneedstodetermine,
determiningit,andcheckingthefindings.Step6,instrumentation,usuallyimpliesthehumaninstrumentasobserverorparticipant,butothertypesofinstrumentation
suchasrecordsordocumentsmaybeneededtohelpverifythefindingsofthestudy.Again,theseinstrumentsmaynotbeestablishedpriortotheinitiationofthe
study.
Steps7(collectingandrecordingdata)and8(dataanalysis)appearonthesurfacetobesimilartothosefollowedbythepositivist.However,therearedifferences
betweenthetwoapproaches.Thetypeofdatathattheconstructivistdealswithistypicallyqualitative(words,photographs,etc.),orsubjective.Thisisnottosaythat
theconstructivistcannotordoesnotusequantitativedata,butdoessolessfrequentlythanthepositivistwould.Inaddition,thetypeofdataanalysismethodsforthe
constructivistisnotbasedoninferentialstatistics.Instead,thesemethodsaresimilartodescriptivestatisticsindeterminingagreementamongthemes,opinions,andso
forth.Moreimportant,itisnotjustthetypeofdatathatiscollectedandanalyzedthatdiffersfromthatcollectedbythepositivist,butwhenthecollectionandanalyses
takeplace.Inthescientificmethod,datacollectionandanalysesfollowaparticularorder:thedatausuallyarenotanalyzeduntilallofthedatahavebeencollected.
Thentheanalysisphasetakesplace,whichisfollowedbytheinterpretation.Fortheconstructivist,datacollection,analysis,andinterpretationtakeplacecontinually
andsimultaneouslythroughoutthestudy.Inasense,afeedbacksystemiscreatedthatoperatescontinuously.Fromthissystem,newhypothesesmayemerge,andnew
typesofdataarecollected.
Step9,planningthelogistics,involvesthenutsandboltsofthestudy,atleastinitially,intermsofwhowillcarryouttheinquiry,whereitwillbecarriedout,

Page26

andwhatscheduleswillneedtobeobserved.Someofthelogisticsincludemakinginitialcontactandgainingentree,negotiatingconsent,buildingandmaintainingtrust,
andidentifyingandusinginformants(Lincoln&Guba,1985).
Thelaststep,planningfortrustworthiness,isanalogoustothepositivist'smethodstoevaluateresearch.LincolnandGuba(1985)havemadeconsiderableeffortsto
developamethodofevaluationforconstructivistinquirythatwouldcarrytheweightthatinternalandexternalvaliditycarryforstudiesthatusethepositivistparadigm.
Gliner(1994)suggeststhatsomeofthesemethodsshouldbegivenhigherprioritythanothers.
TypeofDataDoesNotImplyaParticularParadigm
Ifoneusesthenatureofrealityasadistinctionbetweenthepositivistparadigmandtheconstructivistparadigm,thenitcanbearguedthatnotallqualitativedatawillbe
constructivist,norwillallquantitativedatafitthepositivistparadigm(Gliner,1994).Patton's(1990)examplesofdifferentprogramevaluationapproacheshelpto
demonstratethispoint.Twoofhisdesignswerelabeledaspuredesigns.Onepuredesign,labeledasthepurehypotheticaldeductiveapproach,fitsthetraditional
positivistframework.Thisapproachusesexperimentaldesign,quantitativedata,andtraditionalstatisticalmethods.Theotherpuredesignwaslabeledasthepure
qualitativestrategyandfitstheconstructivistparadigm.Thisapproachusesconstructivistinquiry,qualitativedata,andcontentanalysis.
Patton(1990)alsointroducedmixedformapproachesthatincludedcharacteristicsfromeachpureformapproach.Ofparticularimportancefordistinguishing
constructivistresearchfromotherresearchthatusesqualitativedataweretwomixedformapproaches.Onemixedformapproachconsistedofexperimentaldesign,
qualitativedatacollection,andstatisticalanalysis.Theothermixedformapproachconsistedofexperimentaldesign,qualitativedatacollection,andcontentanalysis.
Bothoftheseapproachesshouldbeconsideredaspartofthepositivistparadigmbecausethedesignwasexperimental.Experimentaldesignsareusedtoanswer
questionsaboutcauseandeffect.Therefore,eventhoughthesetwoapproachesusedqualitativedata,theyshouldbeevaluatedbyusingtraditionalcriteriaestablished
forpositivistparadigms.
ARapprochementbetweenQualitativeandQuantitativeApproaches?
Aswehaveseen,philosophically,thetwoparadigmsaredifferent,yetthetwoapproachesmaybefoundtogetherinoneresearcharticle.Sometimesthetwoare
includedinthesamearticlebutareseparate.Othertimes,thetwoparadigmsareblendedsothatoneparadigmsetsthestagefororleadstotheotherparadigm
(Caracelli&Greene,1993).Thefieldofprogramevaluationhasnumerousexamplesofbothsituations.
Aclassicexampleofusingbothqualitativeandquantitativeapproachesinonestudy,butkeepingthemseparate,istheworkbySmith,Gabriel,Schott,andPadia
(1976).TheseinvestigatorsevaluatedanOutwardBoundprogram.Inthequantitativepartoftheevaluation,auniquetimeseriesdesignwasper

Page27

formedonover300participantsintheprogram.Thequalitativepartoftheevaluationusedacasestudyapproach,whereoneparticipantkeptadiaryofhis
experiencesthroughouttheOutwardBoundprogram.
Anexamplewhereoneapproachleadstotheotherisalsocommoninthefieldofprogramevaluation.ConsideraprogramevaluationbyGlinerandSample(1996),
whichevaluatedaprogramdesignedtoincreasecommunityopportunitiesforpersonswithdevelopmentaldisabilities.GlinerandSample(1996)examinedquantitative
informationcollectedfromaqualityoflifequestionnairetoselectparticipantsforfurthercasestudy.
Thenaturalisticparadigmandthepositivistparadigmrepresenttwodifferentphilosophicalapproachestoresearch.TheaxiomsoutlinedbyLincolnandGuba(1985)
emphasizethedifferencesbetweenthetwoapproaches.Ontheotherhand,writerslikeHowe(1985)donotseethetwoasmutuallyexclusive.Afewexamplesofthe
possibleparadigmcombinationswerelistedearlier.OthercombinationshavebeendescribedbyPatton(1990),CaracelliandGreene(1993),andCreswell(1994).
Lancy(1993)notesthat''asthepolemicaldebateheatsup,therealsoisincreasingevidenceofrapprochement"(p.11).
Summary
Aparadigmisthebeliefsthatmembersofascientificcommunityshare.Therearetwomajorsocialscienceparadigmsatpresent.Thedominantparadigm,sometimes
calledpositivism,isusuallyassociatedwithquantitativeresearchandmethodssimilartothoseinthenaturalsciences.Theconstructivist,ornaturalist,paradigmis
usuallyassociatedwithqualitativeresearch.Suchresearchersarguethatonemusttakeintoaccounttherolesthatlanguageandvaluesplay.Therefore,unlikea
positivist,aconstructivistdoesnotputtotalfaithinscientificmethodologybutleavesroomforhumannature.
LincolnandGuba(1985),whoareconstructivistinvestigators,arguethattherearefiveaxiomsthatseparatethetwoparadigms.Theaxiomsdealwiththefollowing:
1.Thenatureofreality.Positivistsbelieveinasinglereality.Constructivistsbelieveinmultipleconstructedrealities.
2.Therelationshipofknowertoknown.Positivistsmightsaytheinvestigatoristotallyobjective.Constructivistssaytheinvestigatorcannotbetotallyobjectivein
fact,participantandresearcherinteract.
3.Thepossibilityofgeneralization.Positivistsmightsaytruthstatementsarefreefrombothtimeandcontext.Constructivistssaythatthebestthatcanbe
accomplishedisaworkinghypothesiseverythingiscontextuallybound.
4.Thepossibilityofcausallinkages.Positivistsbelievethatcauseandeffectcanbedeterminedatleastasaprobability.Constructivistsbelievethatweareina
constantstateofmutualshapinganditisimpossibletodistinguishcauseandeffect.
5.Theroleofvaluesininquiry.Positivistsmightsayinquiryisvaluefreeandobjective.Constructiviststhinkthatinquiryisvalueboundbyinquiry,choice,theory,
values,andconflict.

Page28

Althoughthesedichotomieshighlightdifferencesintheparadigms,webelievethattosomeextentthepositionsattributedtothesocalledpositivistsdonotreflectthe
realpositionofmostquantitativeresearchers.Thesedichotomiesdoreflect,however,relativedifferences.Thepositivistapproachtoresearchfollowsthesocalled
scientificmethod.Stepstakentocreatethismethodaretostatearesearchquestion,developahypothesis,developaresearchdesign,collectdata,interpretthedata,
andacceptorrejectahypothesis.
Theconstructivistapproachtoresearchissequentialyetflexible.Thestepsaretodetermineanareaofinquiry,abelieftofittheinquiry,afittoatheory,acollecting
site,aplanforfuturesteps,andtheinstrumentformeasurementandtocollectandrecorddata,interpretdata,planlogistics,andplanforvalidity.
Apurepositivistapproachisdeductive.Withinthisapproachyouwillfindexperimentaldesign,quantitativedata,andstatisticalmethods.Apureconstructivist
approachisinductive.Withinthisapproachyouwillfindnaturalinquiry,qualitativedata,andcontentanalysis.Sometimesthesetwoparadigmsareusedtogetherin
research.
StudyAids
Concepts
Deductivereasoning
Epistemology
Generalize
Inductivereasoning
Paradigm
Reality
Distinctions
Identifiablecausesversusmutualsimultaneousshaping
Nomotheticorgeneralizableknowledgeversusideographicknowledge
Positivistversusconstructivistparadigm
Positiveapproach(steps)versusconstructivistapproachtoresearch
Qualitativeversusquantitativeresearch
Researcherandparticipantsareindependentversusresearchandparticipantalwaysinfluenceeachother
Singletangiblerealityversusmultipleconstructedrealities
Valuefreeversusvalueladeninquiry
ApplicationProblems
1.Oneofthemajordifferencesbetweenthequalitativeparadigm(constructivistinquiry)andthequantitativeparadigm(logicalpositivism)is

Page29

themethodofsampling.Howdothetwoparadigmsdifferinsamplingandwhatimplicationsmightthishaveforgeneralization?
2.Whatistheorderofthestepsinthepositivistapproachtoresearch?Putthefollowingstepsinorderfrom1to6.
Interpretdata
Developahypothesis
Acceptorrejecthypothesis
Stateresearchquestion
Collectdata
Developaresearchdesign
3.AccordingtoLincolnandGuba(1985),whatistheusualorderofaconstructivistapproachforresearch?Putthefollowingstepsinorderfrom1to10.
Determineafittoatheory
Determinetheinstrumentation
Determineafocusofinquiry
Planfortrustworthiness
Determinewhereandfromwhomtocollectdata
Determinethefitofparadigmtothefocus
Planfordataanalyses
Determineaplanforphasesoftheinquiry
Plantocollectandrecorddata
Planlogistics
4.Aresearcherwantstofindoutifthereisarelationshipbetweengenderandtestanxiety.
a.Usingthequalitative(constructivist)approach,describehowaresearcherwouldcreatearesearchprojecttoanswerthisquestion.
b.Usingthequantitative(positivist)approach,describehowaresearcherwouldcreateaprojecttoanswerthisquestion.

Page31

Chapter3
EthicalProblemsandPrinciples
HistoricalOverview
PrinciplesandPoliciesRelatedtoHumanResearchParticipants
TheBelmontReport:PrinciplesandNorms
RespectforPersons
Beneficence
Justice
VoluntaryInformedConsent
Information
Comprehension
Voluntariness
Privacy
AssessmentofRisksandBenefits
Summary
StudyAids
Concepts
Distinctions
ApplicationProblems
Thischapterisdividedintotwomajorsections.Thefirstprovidesabriefoverviewoftherecenthistoryofethicalproblemsinresearchwithhumansubjects.The
secondisadiscussionofcurrentprinciplesandpolicies.

Page32

HistoricalOverview
Althoughtherehavebeenethicalproblemsregardingthetreatmentofhumansubjectsthroughouthistory,itiscommontobeginabout60yearsagowiththeNazi
researchatrocitiesof1933to1945.Noticethat,incontrasttotheterminologyweuseintherestofthisbook,wehaveusedthephrase"humansubjects"ratherthan
"participants."Thelatterisarelativelyrecentchangethatemphasizesthecollaborativeandvoluntaryrelationshipofinvestigatorandparticipant.Thiswasdefinitelynot
therelationshipinNaziGermanynor,sadly,intoomanyexamplesofAmericanresearch.
ByNaziresearchatrocities,werefertoexperimentsconductedbyrespectedGermandoctorsandprofessorsonconcentrationcampinmatesthatledtotheir
mutilationordeath.Althoughitistemptingtothinkthattheseatrocitiescouldbeblamedonprisonguards,soldiers,orroguescientists,theevidenceindicates
otherwise(e.g.,Pross,1992).Notonlyweremanyofthesedoctorsrespected,butGermanyhadmoreadvancedmoralandlegalregulationsconcerningbonafide
consentandspecialprotectionsforvulnerablesubjectsthananyothercountryatthattime(Young,1999).Asaresultofthetrialofthesedoctors,theNuremberg
codewasprescribedbyaninternationalcourtin1947(seeShuster,1997).Thefirstprincipleofthecodestatesthatvoluntaryconsentofhumansubjectsis
absolutelyessential.Principles2through8dealwithexperimentaldesignandtherisksandbenefitsoftheresearch.Principle9statesthesubject'srighttorefuseto
participateorcontinue,andPrinciple10dealswiththeinvestigator'sobligationtostoptheexperimentswhencontinuingthemwouldlikelyleadtoharm.
LestwethinkthatethicalproblemswithhumanresearchhavebeenconfinedtoNaziGermany,examplesofAmericanresearchcanalsobecited.In1963,mentally
impairedchildrenfromtheWillowbrookStateSchoolinNewYorkweregivenlivehepatitisAvirus.Theirparentswerenotadequatelyinformedandwereeven
coercedintovolunteeringtheirchildrenforthestudy.TheTuskegeesyphilisstudy,whichbeganin1932,continueduntilitbecamepublicknowledgein1972(Heller,
1972).ThestudyinvolvedseveralhundredpoorAfricanAmericanmeninAlabamawhowerestudiedbutnottreatedovera40yearperiod,eventhoughantibiotics
wereavailableandcommonlyusedtotreatsyphilisformorethan25yearsofthestudy.ThelongtermeffectsofthisstudyincludemistrustandsuspicionintheAfrican
Americancommunityofmedicalresearchandofdoctorsingeneral(Jones,1982).
Seriousethicalproblemsarenot,however,confinedtothebiomedicalsciences.Milgram(1974)conductedaseriesofwellknownexperimentsonobediencethat
sparkedethicaldebatebothinsideandoutsideofthebehavioralsciences.Hisintenttoperformtheseexperimentswasbasedonhisdismayattheeffectsofblind
obediencetoNazicommandsinWorldWarII.Hedecidedthatitwasimportanttostudythepsychologicalmechanismthatlinkedblindobediencetodestructive
behavior.Hewantedtoknowhowfarordinaryadultswillgoincarryingouttheordersofaseeminglylegitimateauthority.Hedeceivedsubjectsintobelievingthat
theywouldbegivingpainfulelectricshockstoathirdperson,the"learner,"whenthatpersonmadeamistakeonaparticular

Page33

task.Theresultswerestartling.Agreatmanyofthe"teachers,"whoweretheactualsubjectsinthestudy,obeyedwithouthesitationtheexperimenter'surgingto
continuetoincreasethepresumedleveloftheshocks,nomatterhowmuchthelearnerpleadedandscreamed.Milgramwasespeciallysurprisedthatnoneofthe
subjectsrefusedtoapplytheshocksordroppedoutofthestudy.ThelearnerinthesestudieswasaconfederateofMilgram'sandnoactualshocksweretransmitted
bytheteacher.Nevertheless,concernsaboutthestudiesandtheuseofdeceptionhavecontinuedtothisday.Milgramdefendedhisworkasshowingthatremarkable
obediencewasseentimeandtimeagainatseveraluniversitieswheretheexperimentwasrepeated.Heemphasizedthewillingnessofadultstogotoalmostanylengths
whencommandedbyanauthority.Hedidfullydebriefthesubjectsandprovidedanopportunityforafriendlyreconciliationwiththepresumablyshockedlearner,who
wasshownnottohavereceivedanyactualelectricshocks.Furthermore,hesentfollowupquestionnairestotheformersubjectsandfoundthatlessthan1%regretted
havingparticipatedinthestudy.Inspiteofthis,it'sdoubtfulthatinstitutionalreviewboards(IRBs)wouldallowthiskindofstudytodaybecausesubjectswere
trickedintoparticipatinginastudythattheyprobablywouldfindunacceptableiftheyhadunderstooditcorrectly.
Ifyouthinkthatethicalproblemsaboutresearchhavebeenconfinedtoexperimentalstudies,Humphreys'(1970)researchonthe"tearoom"tradeindicatesissuesthat
arepotentiallyraisedbyparticipantobservationsandqualitativemethodology.Forthisstudyofmalehomosexualbehavior,Humphreysreceivedaprestigiousaward.
Heusedconsiderabledeceptionandviolatedthesubjects'privacybysurreptitiouslynotingthelicenseplatesofmenheknewhadhadfellatioinpublicrestrooms.He
thenobtainedtheiraddressesfromtheDivisionofMotorVehiclestointerviewthemwhilepretendingtobeahealthserviceworker.Hesuspectedthatthemenwould
notgrantaninterviewiftheyhadknownhisrealpurposebecausemostofthemenweremarriedandlivedwithwiveswhowouldnothaveapprovedofthisbehavior.
In1974theDepartmentofHealth,Education,andWelfarepublishedregulationsontheprotectionofhumansubjects.ItmandatedthattherebeIRBsateachresearch
institutionthatacceptedfederalfundingtodeterminewhethersubjectsareplacedatrisk,andifso,whetherthebenefitsandimportanceoftheknowledgetobegained
sooutweightherisksthatthesubjectsshouldbeallowedtoaccepttheserisks.Theguidelinesalsomandatedthateffectiveinformedconsentbeobtainedfrom
participantsinresearch.
PrinciplesandPoliciesRelatedtoHumanResearchParticipants
TheBelmontReport:
PrinciplesandNorms
TheNationalCommissionfortheProtectionofHumanSubjectsofBiomedicalandBehavioralResearchidentifiedethicalprinciplesandguidelinesfortheprotectionof
humansubjects.Thisreport(NationalCommission,1978)isoftencalledtheBelmontReport.Thereportlistedthreeethicalprinciplestoguidehumanresearch.

Page34

RespectforPersons
Thisprincipleincorporatestwoethicalconvictions.First,participantsshouldbetreatedasautonomousagents,whichmeansthattheindividualiscapableof
deliberatingandmakingindividualdecisionsandchoices.Second,personswithdiminishedautonomyareentitledtoprotection.Suchpersonsincludechildren,the
mentallyretarded,thedementedelderly,personswithvariousemotionalormentaldisorders,andprisoners.
Beneficence
Researchersshouldnotharmparticipantsandgoodoutcomesshouldbemaximizedfortheparticipantsaswellasforscienceandhumanity.Thisprinciple,likethe
others,iscomplexandrequiresbalancingthepotentialbenefitsandtherisks,aswewillseefollowing.
Justice
Researchshouldnotbeexploitativeandthereshouldbeafairdistributionofrisksandbenefits.Forexample,thosewhobearmostoftherisksduringresearchshould
benefitthemostfromit.Fairdistributionisclearlynotalwaysthecaseandleadstoethicaldilemmasthatneedtobeconsideredcarefully.
Sieber(1992)listssixnormsofscientificbehaviorthatfollowfromtheBelmontReport:
1.Theresearchdesignshouldbevalidbecauseonlyvalidresearchyieldsmeaningfulresults.
2.Theresearchermustbecompetentandcapableofcarryingouttheprocedures.
3.Therisksandbenefitsshouldbeidentifiedfromallperspectives.Ethicalresearchwillmaximizethebenefitsandminimizetherisks.
4.Theselectedsubjectsmustbeappropriateforthepurposesofthestudyandrepresentativeofgroupsthatwillbenefitfromtheresearch.
5.Theresearcherisresponsibleforwhathappenstothesubjectsandmustinformthemaboutcompensationiftheyareharmed.
6.Theremustbevoluntaryconsent.
Thenextsectionexpandsthetopicofinformedconsent.Thatdiscussionandthoseonprivacyandtheassessmentofrisksandbenefitsthatfollowarebasedon
Sieber's(1992)helpfulbook,PlanningEthicallyResponsibleResearch:AGuideforStudentsandInternalReviewBoards.
VoluntaryInformedConsent
Consent,liketheotherissuestobediscussed,isconsiderablymorecomplexthanitmightappearonthesurface.Voluntaryinformedconsentistheprocedureby
whichparticipantschoosewhetherornottheywishtoparticipateinastudy.AccordingtotheBelmontReport(NationalCommission,1978)therearethreeaspects
toinformedconsent:information,comprehension,andvoluntariness.Eachisdescribedbrieflyandthenvariousissuesarediscussedinmoredetail.

Page35

Information
Theinformationprovidedtoparticipantsshouldincludetheresearchprocedure,purpose,risks,andanticipatedbenefits.Itshouldprovidetheinformationthata
reasonablevolunteerwouldwanttoknowbeforegivingconsent.Howtheinformationisgivenisalsoimportant.Itmustbeinlanguageorinaformthattheparticipants
canunderstand,andeffortsshouldbemadetocheckthatitisunderstood,especiallyincaseswhererisksareinvolved.Also,itshouldbenotedthatconsentisan
ongoingprocessandthatitmaybewithdrawnatanytimeduringthestudy.
Comprehension
Theparticipantsshouldhavethelegalcapacityandtheabilitytounderstandtheinformationandrisksinvolvedsothattheycanmakeaninformeddecision.Some
participants(e.g.,children)arenotlegallyqualifiedtomakedecisionsofconsentforthemselves,soothersmustmakethedecisionforthem.Theirrepresentativeis
usuallytheparentorguardian,butchildren7yearsorolderalsomustassenttotheprocedure,indicatingthattheyagreeandwanttoparticipate.
Comprehensionalsomaybeimpairedincasesofmentallyretardedoremotionallydisabledpersons.Again,totheextentpossible,thispersonshouldbeallowedto
assentornottotheprocedures,butathirdparty(e.g.,thelegalguardianorsomeonewhoislikelytounderstandthesubject'ssituation)shouldbechosentoactinthat
person'sbestinterest.Specialcaremustbetakenwhentheremightbeaconflictofinterestbetweentheparentorguardianandtheinterestsofthechildorother
participantswhodonothavethelegalcapacitytograntconsent.Bothpersonsmustvoluntarilyagreetotheparticipation.
Voluntariness
Thethirdaspectofinformedconsentmeansthattheparticipanthasdecidedtoparticipateinthestudyfreelywithoutthreatorundueinducement.Thereshouldnotbe
anyelementofforce,deceit,duress,orotherformofconstraintorcoercion.AswehaveseenintheMilgramstudies,personsinauthoritycanelicitunjustifiable
obedienceevenwithwelleducatedadults.Also,voluntarinessisreducedwhentheresearchoffersfinancialorotherinducementsthatthepotentialparticipantswould
findhardtorefuse,forexample,largepaymentstopoorpeopleorpromisesofreducedsentencestoprisoners.
Sieber(1992)listsanumberofaspectsoftheconsentprocessthatshouldbeconsidered.Rapportshouldbeachievednotonlybecauseparticipantsaremorelikelyto
cooperate,butbecauseitcanstrengthentheecologicalvalidityofthestudy.Thereshouldbeacongruencebetweenwhattheresearchersaysandbodylanguage.Itis
importantthattheresearchernotrushthroughthis"unnecessary"aspectofthestudyorevengivethatimpression.Developingthetrustoftheparticipantsand
understandingtheirpersonalandculturalsituationsisalsoimportant.Thisawarenessisespeciallytrueforcommunitybasedresearchandresearchdoneinculturesthat
aredifferentfromtheresearcher'sculture.Theresearchalsoshouldberelevanttotheconcernsoftheresearchpopulationandexplainedinthoseterms.
Theissueofwhoshouldprovidetheconsentiseasywhenthepotentialparticipantisanadultwhoisassumedtobeafreeagentandhasthecapacitytoconsentor
not.Theissueislessclearwithchildrenorotherpersonswhomay

Page36

nothavethecapacitytogiveconsent.Theissuehereisthat,insituationswheretheparticipantscannotprovidetheirownconsent,weshouldnotautomaticallyassume
thatparentalorguardianconsentissufficient,althoughinmostcasesitshouldbe.Researchersshouldbeawarethatinsomesituationstheremaybeaconflictof
interest.Forexample,parentsofadjudicatedyouthorpoorparentswhoneedthemoneyofferedbyaresearchstudymightnothavetheinterestofthechildforemost.
Howisconsentobtained?MostIRBsrequireaformalsignedconsentform,exceptincertainsituationsspecifiedinthefederalregulations.First,asignedconsentform
maybeomittedwhensubjectscaneasilyrefusebydiscontinuingaphonecallwithaninterviewerorbynotreturningthesurveythatwasreceivedinthemail.Itis
important,however,thattheinterviewerorquestionnairecoverletterdescribesthepurposeoftheresearchandanyriskinvolved,andstatesthatparticipationis
voluntary.Insuchcasesit'sgenerallydeemedunnecessaryforthesubjecttoactuallysignandreturnaconsentform.Returningthesurveyoransweringthequestionsis
thesubject'swayofimplyingconsent.Second,signedconsentformscanbeomittedwhenprovidingthemwouldjeopardizethewellbeingoftheparticipants.For
example,iftheresearchisoncriminalbehavior,aconsentformcouldbesubpoenaedand,therefore,couldputthepersoninjeopardy.Inresearchonpopulations
vulnerabletostigmatizingeffects(e.g.,personswithAIDS),asignedconsentformmaywellpresentmoredangerthanprotectiontotheparticipants.
Thefactthatasignedconsentformisnotrequireddoesnotmeanthatconsentisnotnecessary.Forconsentinthecaseofinterviewsandsurveys,theparticipantsstill
mustbeinformedandmusthavethecapacitytoconsent.Inthecaseofcriminalorstigmatizedbehavior,theparticipantsshouldbeprovidedwithacopyofthe
consentstatementbuttheydonothavetosigntheagreementtoparticipate.
Anotheraspectoftheconsentprocessoccursinstudieswherethereisdeceptionorwheretheremaybequestionsabouttheproceduresafterthefact.Debriefing,as
itiscalled,isrequiredwhendeceptionisinvolvedandisagoodideaevenifthereisnodeception.Toooftenresearchersskiparesponsibilitytodebriefsubjectsby
promisingtomaketheresultsavailablelater.Thispromiseisusuallynotagoodideabecauseitisoftenbrokenandishardtofollowupbythetimetheresultsare
known.Morerealisticandimportantisprovidingtheopportunityrightafterthestudytodiscussitinthecontextoftheliteratureandwhatledtotheresearch.This
approachisoftenmoreusefultotheparticipantsthanthefindingsoftheparticularstudy.Participantswhoreceivedsuchinformativedebriefingaremorelikelytofeel
goodaboutresearchandmorelikelytoparticipateinfuturestudies.
Privacy
Stemmingfromtheethicalprincipleofrespectforparticipants,acomponentofvoluntaryinformedconsentisthatparticipants'privacywillberespected.Tosome
extentmuchofbehavioralresearchinvolvesaskingparticipantstorevealaspectsoftheirbehaviororattitudes.Privacyreferstoaperson'sconcern

Page37

aboutcontrollingaccesstoinformationaboutthemselves(Sieber,1992).Voluntaryinformedconsentinvolvestheparticipantagreeingtorevealcertainaspectsthat
mayhavebeenprivatepreviously.Ifparticipantsfeelthatprivacyisbeinginvaded,answersthattheyprovidemaybedistortedand,therefore,givemisleadingorfalse
information.Theessenceofprivacyisthattheparticipantisfreetochoosetheextenttowhichhisorherattitudes,beliefs,andbehaviorsaretobesharedwithor
withheldfromothers.Becausebehavioralscientistsoftencollectandanalyzeprivatedata,thereisalwaysthepotentialforaconflictbetweentherightofprivacyand
thegoaloftheresearch.
Severalfactorsinfluencetheextenttowhichparticipantsviewtheirprivacyasbeinginvaded.Oneisthatifthedataareanonymous,thatis,unidentifiablebythe
researcher,theparticipantmaybemorewillingtoshareandtobereassuredthattheirprivacyisprotected.Itisimportantheretomakeadistinctionbetween
confidentialityandanonymity.Anonymitymeansthattheperson'snameandotheridentifierssuchassocialsecuritynumberoraddressareneverattachedtothedata
orevenknownbytheresearcher.Inmanystudiesthedataarenotanonymous,becausetheresearchermustmatchmorethanonepieceofdataortheparticipantis
seenfacetofaceandcannotbeanonymous.Inallcasesitisimportantthatthedataremainconfidential.Thatis,thereisanagreementthatprivateinformationwill
remainprivatetotheresearcher,andtheparticipantwillnotbeidentifiableinthereportsorinconversationswithotherpersonsoutsideoftheresearchteam.
Federallawhaslittletosayaboutprivacyandsocialresearch.Forexample,itisnotdiscussedasaseparatetopicindepthintheBelmontReportmentionedearlier.
However,sensitiveresearcherswillbeverycarefulnottoinvadetheprivacyofparticipantsintheirresearchIRBsaretypicallyalerttothisissue.Thisconsideration
impliesthatfullyinformedvoluntaryconsentwillbeobtainedaheadoftimeandthattheresearcherwillassureconfidentialityofthedata.Theparticipantscanthen
decidewhetherornottoparticipateundertheseconditions.Presumablytheparticipantwhoviewstheresearchasaninvasionofprivacycansimplydeclineto
participate.However,itmaybethat,feelingsubtlepressuretoparticipate,thesubjectwilldistortanswers.Thus,bothtobesensitivetotheparticipant'sconcerns
aboutinvasionofprivacyandtoobtainthebestdata,itisimportanttoconsiderwhethermembersoftheplannedparticipantgroupwillviewtheresearchasan
invasionofprivacy.Tolearnabouttheprivacyinterestsofyourresearchpopulation,youshouldasksomeonewhoworkswithorisamemberofthepopulation.For
example,askteachers,parents,therapists,orexperiencedresearcherswhethertheythinkotherpeopleinthisgroupmightthinkyourquestionsareaninvasionof
privacy.
Researchersalsoneedtoprotectthefactthatanindividualparticipatedintheresearchatall.Confidentialityisespeciallyimportantiftheresearchisaboutastigmatized
group(e.g.,homosexuals,depressedmothers).Thus,listsofparticipantsneedtobekeptconfidentialandshouldbedestroyedaftertheproject.
AssessmentofRisksandBenefits
Probablythemostimportantconcernaboutresearchethicsisthattheindividualsnotbeharmedbyservingasparticipantsinthestudy.TheBelmontReportstates,

Page38

''Therequirementthatresearchbejustifiedonthebasisofafavorablerisk/benefitassessmentbearsacloserelationtotheprincipleofbeneficence,justasthemoral
requirementthatinformedconsentbeobtainedisderivedprimarilyfromtheprincipleofrespectforpersons"(NationalCommission,1978,p.6).
Riskrefersbothtotheprobabilityofharmoccurringandtothemagnitudeandtypeofharm.Therearemanykindsofpossibleharmsandbenefitsthatneedtobe
takenintoaccount.Forexample,therearepsychological,physical,legal,social,andeconomicharms,whichhavecorrespondingpotentialbenefits,although
psychologicalandphysicalpainorinjuriesarethemostoftendiscussedtypesofrisks.Others,suchaspotentiallossofjoborembarrassment,shouldbeconsidered.
Infact,invasionofone'sprivacycouldbeviewedasatypeofharmresultingfromresearchthatisnotsensitivelyorcarefullyplanned.
Althoughitisraretoattempttoquantifytherisksandbenefitsofaparticularresearchstudy,thereshouldbeasystemicassessmentofthesefactors.TheBelmont
Report(NationalCommission,1978)statesthattheassessmentofwhethertheresearchisjustifiableshouldreflectatleastfiveconsiderations:(a)brutalorinhumane
treatmentisneverjustified(b)risksshouldbereducedtothosethatarenecessaryandconsiderationshouldbegiventoalternativeproceduresthatreducerisks(c)
whenresearchinvolvessignificantrisksofseriousharm,reviewcommitteesshouldbeverycarefulthatthebenefitsjustifythoserisks(forexample,inmedicalresearch,
anunproventreatmentmaypromisesignificantbenefitseventhoughtherearerisksofserioussideeffects)(d)whenvulnerablepopulationsareinvolved,the
appropriatenessofusingthemshouldbedemonstrated,includingjudgmentaboutthenatureanddegreeofriskandthelevelofanticipatedbenefitsand(e)relevant
risksandbenefitsmustbefairlyexplainedintheinformedconsentprocedureandform.
Sieber(1992)identifiesfourmisconceptionsaboutrisksandbenefits:
1.Aratioisnotactuallycomputedmostrisksandbenefitsofresearchcannotbequantified.
2.Somerisksandbenefitscannotbeidentifiedaccuratelybeforetheresearchisperformed.
3.Itisimpossibletoconsiderallrisksandbenefits...theresearchersneedtofocusonthemostimportantofthese,butcannotpossiblyanticipateeverything.
4.Risksandbenefitscannotbeidentifiedforeachsubjectindividually.Onesubject'sriskmaybeanother'sbenefit.(pp.7576)
Sieber(1992)alsopointsoutthatfederalregulationsgoverningtheIRBsareconcernedonlywithharmtohumansubjects,notwithresearchersocialsensitivity.
However,ignoringthesensitivitiesofgatekeepers,communitymembers,andsocietyatlargeinviteshavingtheresearchprevented,interrupted,maligned,or
invalidated.Furthermore,IRBsmaywellconsiderhowtheresearchmayaffectthecommunity.
Inadditiontominimizingtherisks,itisimportantforresearcherstomaximizethebenefits.Thisstepmayberelativelyeasyincommunitybasedandmedicalresearch
wheresomeclearbenefittotheindividualparticipantsisen

Page39

visioned.However,beforethestudy,suchbenefitscanonlybeanticipated,ortherewouldbenoneedforthestudy.
Itislesseasytoachievebenefitsfortheparticipantsinsocalledsurveyresearchandcertainkindsoflaboratoryexperiments.Nevertheless,researchersmustthink
abouttheissueofmaximizingbenefitsinarealisticmannertoavoidfalsepromisesorgrandioseclaimsaboutbenefitstoscienceandsociety.Sieber(1992)statesthat
itismorerealistictoprovidebenefitstothesubjectsortheircommunity,researchinstitutions,theresearcher,orthefunderthanitistoscienceandsociety.Benefitsto
participantscouldincludeaninformativedebriefing,workbooksormaterials,achancetoshareconcernsorinterestswiththeresearcher,and,insomecases,the
effectsoftheexperimentaltreatment.
Benefitstothecommunitycouldincludeimprovedrelationshipswithauniversity,moreunderstandingabouttheproblemsunderstudy,specialtraining,theprestigeof
beingassociatedwiththeprogramanduniversity,andmaterialssuchasbooks.Thebenefitstotheresearcher,institutions,andfunderaremoreobvious,butinclude
subtlethingslikefutureaccesstoparticipantsandcommunitysitesandimproved"towngown"relationships.Thesebenefitsareimportantbut,intheviewofIRBs,
takeabackseattobenefitstotheindividualandtoscience.
Summary
Thischapterprovidesanoverviewoftherecenthistoryofethicalproblemsinresearchwithhumansubjects.Inthelast60years,therehavebeenanumberof
violationsrangingfromtheNazidoctors'researchatrocitiestostudiesthatseeminretrospecttohavegonetoofarintermsofdeceptionorinvadingprivacy.The
chapteralsodiscussesprinciplesandnormsforresearchwithhumanparticipants.Theprinciplesofrespectforparticipants,beneficence,andjusticeformthebasisof
appropriatehumanresearch.Voluntaryinformedconsentbyparticipantsisrequired,anditscomplexitiesarediscussed.Theethicalprinciplesofprivacyaredesigned
toassurethattheparticipantsarefreetochoosehowmuchtorevealandthatwhatisrevealedwillbekeptconfidential.Finally,theassessmentofrisksandbenefitsis
discussedandthepointmadethatrisksshouldbeminimized.Variouskindsofrisks,psychologicalaswellasphysical,arediscussed,asarevarioustypesofpossible
benefits.
StudyAids
Concepts
Assent
Beneficence
Debriefing
Institutionalreviewboard(IRB)
Justice
Privacy

Page40

Respectforpersons
Voluntaryinformedconsent
Distinctions
Anonymousversusconfidential
Risksversusbenefits
ApplicationProblems
1.Readthefollowingscenario,thenanswerthequestionsbelow.
Dr.Jones,oftheCollegeofEducationatMajorUniversity,isinterestedintheemotionalwellbeingofchildrenraisedinatraditionalreligiousschoolsetting.She
hypothesizesthatthesechildrenwillbeemotionallystrongerthanthegeneralnationalnorms.BecauseDr.JonesservesontheBoardofDirectorsfortheschool,the
Principal,SisterMary,readilyagrees.Dr.Jonesmaymeetwiththe5thgradersandcaninterviewall20studentsabouttheirfamilyattitudestowardalcohol,tobacco,
anddruguse,andtheirresistancetoviolence.Shealsoknowsofastandardizedinstrumentofemotionalhealth.SisterMarydeterminesthattheassessmentofthe
children'semotionalhealthwillbeusefulinformationfortheschooltohaveinthestudents'records,sostudentswillbetoldtheinterviewsarepartoftheclass.Because
itispartoftheclassassignment,thereisnoneedtoespeciallyinformtheparents.Besides,thenoticessenttoparentsnevercomebackwhentheyaresentoutin
lunchboxesanyway!The20studentsareaboutevenlysplitbetweenboysandgirls,threeCubanstudentsattend,andtherestarewhite.Alongwiththestandardized
psychologicalinstrument,sheshouldbeabletosnapshotthechildrenreasonablywell,anddifferentiatewellbeingbygender,ethnicity,andfamilyattitudes.
a.Whoaretheplayers(bothapparentandnotapparent)andwhatmightbetheirissues?
b.WhichBelmontReportprinciples(respectforperson,beneficence,justice)pertain,andhow?
c.WhatquestionsmightanInstitutionalReviewBoardhaveaboutthisproject?
d.Shouldtheprojectbeapprovedascurrentlyproposed?
e.HowcouldtheprojectberedesignedtoaddresssomeoftheIRB'sconcerns?
Inthefollowingscenariosdescribewhatethicalissueswereraised.
2.Aresearcherisinterestedinchocolateconsumptionandreactiontime.Sherandomlyassigns16studentstoeitheranexperimentalorcontrolgroup.Thestudents
aretoldthataspartoftheirfinalgradeinthecourse,theymustbeasubjectinthestudy.Aftergivingeightofthestudentsfivecandybarseachtoeat(whileshesits
andwatchestoensuretheyeatallofthem)shegivesall16studentsatestforreactiontime.Whenthestudentshavecompletedthetest,sheallowsthemtoleave.

Page41

3.Atalargeuniversityaresearcherwantstofindoutifgraduatestudentshavebetterdecisionmakingskillsthanundergraduates.Theresearchertells30graduates
and30undergraduatesthathewillgivethem$50eachiftheycompleteadifficultdecisiontask.Aftertheresultsweretabulated,theresearcherpoststhestudents'
socialsecuritynumbersanddecisionmakingscoreonhisdoorsothestudentscanknowhowtheydidonthetask.

Page43

Chapter4
ResearchProblems,Variables,andHypotheses
ResearchProblems
TwoSampleProblems
SourcesofResearchProblems
CharacteristicsofaGoodResearchProblem
BroadVersusNarrow
WidespreadVersusLimitedInterest
WellResearchedVersusUnknownTerritory
Variables
DefinitionofaVariable
OperationalDefinitionsofVariables
IndependentVariables
ActiveIndependentVariables
AttributeIndependentVariables
TypeofIndependentVariableandInferencesAboutCauses
LevelsoftheIndependentVariable
MoreThanOneIndependentVariable
DependentVariables
ExtraneousVariables
OrderedVersusUnorderedorNominalLevelsofVariables
OtherConsiderationsAboutVariables
GroupsorSetsofVariables
HypothesesandResearchQuestions
TheAnxietyandHeartRateProblemRevisited
TheMasteryMotivationProblemRevisited

Page44

Theory
Summary
StudyAids
Concepts
Distinctions
ApplicationProblems
ResearchProblems
Theresearchprocessbeginswithaproblem.Whatisaresearchproblem?Kerlinger(1986)describesaproblemas"aninterrogativesentenceorstatementthat
asks:Whatrelationexistsbetweentwoormorevariables?"(p.16).Wewouldliketopointoutthatalmostallresearchstudieshavemorethantwovariables.
Kerlingersuggeststhatpriortotheproblemstatement"thescientistwillusuallyexperienceanobstacletounderstanding,avagueunrestaboutobservedand
unobservedphenomena,acuriosityastowhysomethingisasitis"(p.11).
TwoSampleProblems
ConsiderfirstthefollowingexamplethathappenedtoJ.A.G.Someyearsago,whenIwasworkingintheareaofenvironmentalphysiology,Iwastogiveatalkatthe
FederationofAmericanSocietiesforExperimentalBiology.Thereweremanywellknownscientistsatmytalk,andIwasnervoustosaytheleast,especiallybecause
IfeltthatothersintheaudienceknewmoreaboutmytopicregionaldistributionofbloodflowduringalcoholintoxicationthanIdid.Duringthetalkimmediately
precedingmine,acolleaguesittingnexttomeaskedhowIfelt.IansweredthatIfeltfine,butItookmypulseandfound

Page45

myhearttobebeatingatarateofabout110beatsperminute,considerablyabovemynormalrestingheartrateof60beatsperminutebutsimilartomyrateafter
moderateexercise.Iwonderedifthiscouldbeahealthyresponse.ThusIbegantoformulatemyproblem.Iwouldphrasemyproblematthisstageas,Couldahigh
heartrateintheabsenceofexercisebenormal?
FromhereIreadtheliteratureonanxietyandheartrate,andfoundnumerousstudiesthatexaminedheartrateunderconditionsofspeakingbeforeaudiences.The
heartratecouldgetexceedinglyhigh,muchhigherthanminehadbeen.Alsoofinterestwastheuseofpharmacologicalagentssuchasbetablockerstoreduceheart
rateandanxietyundertheseconditions.However,noneofthepreviousstudiesexaminedthemetabolicrequirements(e.g.,oxygenuptakeandcardiacoutput)under
theseanxietysituations.Ontheotherhand,severalstudiesthathadexaminedmetabolicrequirementsonheartrateaboverestingwereperformedduringthecondition
ofexercise.Thesestudiesconsideredelevatedheartratefollowingexercisetobenormalbecausetheheartmustdeliveranincreasedamountofoxygentothetissues
underhighermetabolicdemands.Theproblemnowbecameageneralquestion:Ifwemeasuredthemetabolicdemandsofasituationunderanxiety,woulditbe
similartoasituationunderexercise?Nowanobstaclewascreatedthathadtoberemoved.Howcouldwecreatetwosituations,oneunderanxietyandoneunder
exercise,thatyieldedsimilarheartrates?Toremovethisbarrier,wedecidedtouseawithinsubjectsdesign,whereeachparticipanttookpartinallconditionsofthe
study.First,wecoulddeterminetheheartrateandmetabolicrequirementsunderananxietyprovokingsituation(e.g.,priortogivingatalk).Next,wecouldhavethe
participantexerciseonatreadmillataworkloadhighenoughtogiveusaheartrateidenticaltothatexperiencedunderanxietyandwecouldalsomeasuremetabolic
requirements.
Nowwecouldstateourproblemas,Howareheartrateandmetabolicrequirementsrelatedunderconditionsofanxiety?Ournextstepwouldbetochangethe
problemstatementintoapredictionstatement,orhypothesis,thatcouldbedirectlytested.Again,borrowingfromKerlinger(1986),"hypothesisstatementscontain
twoormorevariablesthataremeasurableorpotentiallymeasurableandthatspecifyhowthevariablesarerelated"(p.17).Wewillreturntothisexamplelaterinthe
chapter.
ThesecondexampleisaresearchproblemfacedbyG.A.M.andhiscolleagues.Wehadobservedthatinfantswhowerebornprematurelyandalsothosewhohad
beenabusedorneglectedseemedtohavelowermotivationtomasternewskillsandseemedtogetlesspleasurefromtrying.Thisclinicalobservationraisedseveral
issues.First,couldthemotivationofpreverbalinfantsbemeasured?Achievementmotivationinadultsandolderchildrenhadbeenassessedfromstoriestheytoldin
responsetoambiguouspictures.Someothermethodwouldneedtobedevelopedforinfants.Second,wasitreallythecasethatprematureandabusedorneglected
infantswerelessmotivatedtomastertasks?Thesecondpartoftheresearchproblemmightbephrased,Istherearelationshipbetweenprematurityandabuseor
neglectandmasterymotivation?Wewillalsoreturntothisexampletoexaminetheselectionanddevelopmentofagoodresearchproblem.

Page46

SourcesofResearchProblems
Theexamplesjustdiscussedillustratetwocommonsourcesofresearchproblems:personalexperienceandclinicalobservation.Bothassumeaknowledgeofthe
literatureinthefieldandtheabilitytorelateittotheexperiencesorobservations.Oftenexperiencesatworkorschoolcanbethesourceofaresearchproblem,ifyou
knowwhatquestionsareunansweredatpresentandhowtotranslatethe"vagueunrestaboutobservedorunobservedphenomena"(Kerlinger,1986,p.11)intoa
testableresearchproblem.
Animportantdistinctionthatissometimesconfusingtostudentsisthatthewordproblemmightconveythefalseimpressionthataresearchproblemisthesameasa
personalorsocietalproblem.Personalorsocietalproblems,however,mayleadtoresearchproblemsandquestionsorhypothesesthatcanbeansweredbycollecting
andanalyzingdata.ForJ.A.G.toworrythathewouldbenervousduringhispresentationisnotaresearchproblem.Likewise,forG.A.M.tobeconcernedaboutthe
apparentlowmasterymotivationofabusedorneglectedchildrenisasocietalbutnotaresearchproblem.Thediscussionatthebeginningandendofthischapter
indicateshowthesepersonalandsocietalproblemsmightevolveintoresearchproblems.
Oneofthefirststepsintheresearchprocessistoreadtheresearchliteratureonandaroundthetopicofinterestsothatyouwillbeabletoidentifygapsin
knowledge.AppendixBdiscussestheresearcharticleandprovideshintsabouttheliteraturereview.Locke,Silverman,andSpirduso(1998)andFink(1998)provide
usefulinformationaboutreadingandunderstandingresearchandconductingliteraturereviews.
Theoryisanothermajorsourceofresearchproblems.Intextbookdescriptionsofthescientificmethod,thedeductionoftestablehypothesesfromtheoriesis
presentedasthemethodofchoiceforthebehavioralsciences(e.g.,Drew,1980).Wedonotwanttodiscounttheimportanceorvalueofthissourceofresearch
problems,butwewouldliketomaketwopoints.Qualitativeresearchisoftensaidtoworktheotherwayaround:observationsleadtotheoryinductively.Also,in
appliedfields,relevanttheoriesmaybehardtofind.Nevertheless,wethinkitisimportantforanygoodresearchproblemtobegroundedintheempiricaland
theoreticalliteratureandtohaveaconceptualaswellasapracticalbase.
CharacteristicsofaGoodResearchProblem
Inadditiontobeinggroundedintheempiricalandtheoreticalliteratureandexaminingtherelationshipsbetweentwoormorevariables,agoodresearchproblemhas
severalothercharacteristics.Asindicatedearlier,itshouldholdthepromiseoffillingagapintheliterature,orprovidingatestofatheory,orboth.
Agoodresearchproblemshouldalsostatethevariablestoberelatedclearlyandunambiguously.Oftenresearchproblemsstartouttoobroadlyorvaguelystated.
AppendixCprovidesseveralexamplesandtemplatesforwritinggoodresearchproblems.Problemsshouldalsoimplyseveralresearchquestions.

Page47

Again,AppendixCprovidesexamplesofresearchquestionsinformatsthatareconsistentwiththeframeworkofthisbook(seeespeciallychapter5,whichalso
providesexamplesofresearchquestions).
Agoodresearchproblemshouldbetestablebyempiricalmethodsitshouldnotbejustastatementofyourmoral,ethical,orpoliticalposition.Youshouldbeableto
collectdatathatwillanswertheresearchquestions.
Ofcourse,themethodsusedmustbeethicalandconsistentwiththeguidelinesoutlinedinchapter3.Theproblemalsoneedstobefeasible,givenyourresourcesand
abilities.Finally,itisdesirable,especiallyforgraduatestudents,tochooseaproblemthatisofvitalinteresttoyousothatyoucansustainthemotivationtofinish,often
adifficultthingtodo.
Anotherwayofdecidingonagood,appropriateresearchproblemforathesisordissertationistoexaminewhereitwouldlieonseveraldimensions.
BroadVersusNarrow
Westatedthatresearchproblemsoftenstartouttoobroadly.Forexample,Whatfactorscauselowmasterymotivation?istoobroadandprobablynotafeasible
problem.Rememberthatyourtimeandresourcesarelimited,sopracticalityrequiresthatyoulimitthescopeofyourproblem.Itisalsoimportanttorealizethat
scienceprogressesinsmallsteps.Evenlarge,wellfundedresearchprojectsoftenraisemorequestionsthantheyanswer,andusuallyaddressonlyalimitedpieceofa
broadresearchproblem.Ontheotherhand,youdonotwanttheproblemtobetoonarrow.Usuallyitshoulddealwithmorethanasingle,limitedresearchquestion
andtwovariables(seeAppendixC).
WidespreadVersusLimitedInterest
Itmightseemthatyouwouldwantasmanypeople(scholarsandthepublic)aspossibletobeinterestedinyourresearch.Certainlyyoushouldbeinterestedanditisa
goodstrategytopickaproblemthatisofinteresttoyouradvisor.Youwillgetmoreandbetterfeedbackfromyouradvisorandcommitteeiftheyhaveinterestinand
knowledgeaboutthearea.Itisalsodesirabletochooseatopicthatisofwidespreadinterest,butsometopicsbecomealmostfaddishandhavesomanystudiesabout
themthatitishardtomakeacontribution.Ifyouchooseatopicthatiscurrentlyverypopular,trytofindasmuchrecentunpublishedliteratureaspossiblebyattending
conferences,searchingtheWebandERIC(EducationResourcesInformationCenters)documents,andwritingtoresearcherswhohaverecentlypublishedinthearea
toseeiftheyhavesomethingnew.Itisadifficulttasktoidentifygapsintheliteratureofacurrentlypopulartopicbecausealotofworkmaybestillinprogress.This
pointoverlapswiththenext.
WellResearchedVersusUnknownTerritory
Itisexcitingtothinkthatyoumightbethefirstonetoexploreanarea.However,ifthatisthecase,onemightwonderwhyitisunexplored.Isthetopicoflimited
interest?Aretherepractical,ethical,orfinancialreasons?Isthetopictoospecializedornarrow?Ofcourse,thereareinterestingandimportanttopicsthatare
relativelyunexploredandarenotfacedwiththeseobjections,buttheyarenoteasytoidentify.

Page48

Quantitativeresearcherstendtoemphasizefindinggapsintheliterature,sotheytendtostudyrelativelywellresearchedareas.Qualitativeresearchers,ontheother
hand,deemphasizefindingliteratureaheadoftimeandtendtoexplorelesswellresearchedtopics,seeingwheretheirobservationsleadthem.
Variables
Avariablehasonedefiningquality.Itmustbeabletovaryorhavedifferentvalues.Forexample,genderisavariablebecauseithastwovalues,femaleormale.Ifwe
arestudyingdifferencesbetweenmenandwomen,genderisavariable.Ontheotherhand,ifwearestudyingonlywomen,genderisnotavariableitisaconstant.
Ageisavariablethathasalargenumberofvalues.Typeoftreatmentorintervention(ortypeofcurriculum)isavariableifthereismorethanonetreatmentorthereis
atreatmentandacontrolgroup.Numberofdaystorecoverfromanailment,acommonmeasureoftheeffectofatreatment,isalsoavariable.Similarly,amountof
mathematicsknowledgeisavariablebecauseitcanvaryfromnonetoalot.
DefinitionofaVariable
Thus,wecandefinethetermvariableasacharacteristicoftheparticipantsorasituationinagivenstudythathasdifferentvaluesinthatstudy.Inquantitative
research,variablesaredefinedoperationallyandarecommonlydividedintoindependentvariables(activeorattribute),dependentvariables,andextraneousvariables.
Eachofthesetopicsaredealtwithinthefollowingsections.
OperationalDefinitionsofVariables
Anoperationaldefinitiondescribesordefinesavariableintermsoftheoperationsusedtoproduceitortechniquesusedtomeasureit.Whenquantitative
researchersdescribethevariablesintheirstudy,theyspecifywhattheymeanbytellinghowtheymeasuredthevariable.Demographicvariableslikeage,gender,or
ethnicgroupareusuallymeasuredsimplybyaskingtheparticipanttochoosetheappropriatecategoryfromamongthoselisted.Eachtypeoftreatment(orcurriculum)
isusuallydescribedordefinedmuchmoreextensivelysothatthereaderknowswhattheresearchermeantby,forexample,acognitivelyenrichingcurriculumor
shelteredwork.Likewise,abstractconceptssuchasmathematicsknowledge,selfconcept,ormathematicsanxietyneedtobedefinedoperationallybydescribingin
detailhowtheywillbemeasuredorimplementedinaparticularstudy.Todothis,theinvestigatormayprovidesamplequestionsorappendtheinstrument,orprovide
areferencewheremoreinformationcanbefound.
IndependentVariables
ActiveIndependentVariables
Thisfirsttypeofindependentvariableisoftencalledamanipulatedvariable.Afrequentgoalofresearchistoinvestigatetheeffectofaparticularinterventionona
particularoutcome.Astudymight

Page49

examinetheeffectofanewkindofsplinttoincreasefunctionalmovementinpeoplewhohavehadaspinalcordinjury.Theindependentvariableisthetypeofsplint.
Asecondstudymightexaminetheeffectofanewteachingmethod,suchascooperativelearning,onstudentperformance.Theindependentvariableinthiscaseisthe
typeofteachingmethod.Inthesetwoexamples,thevariableofinterestissomethingthatisgiventotheparticipants.Therefore,anactiveindependentvariableisa
variable,suchasaworkshop,newcurriculum,orotherintervention,onelevelofwhichisgivenorpresentedtoonegroupofparticipantsbutnottoanother,usually
withinaspecifiedperiodoftimeduringthestudy.
Intraditionalexperimentalresearch,independentvariablesarethosethattheinvestigatorcanmanipulatetheypresumablycauseachangeinaresultingbehavior,
attitude,orphysiologicalmeasureofinterest.Anindependentvariableisconsideredtobemanipulatedwhentheinvestigatorhasgivenonevaluetoonegroup
(experimentalcondition),andanothervaluetoanothergroup(controlcondition).
However,therearemanycircumstances,especiallyinappliedresearch,whenwehaveanactiveindependentvariable,butthisvariableisnotdirectlymanipulatedby
theinvestigator.Considerthesituationwheretheinvestigatorisinterestedinanewtypeoftreatment.Toconductthestudy,itturnsoutthatrehabilitationcenterAwill
beusingthattreatment.RehabilitationcenterBwillbeusingthetraditionaltreatment.Theinvestigatorwillcomparethetwocenterstodetermineifonetreatmentworks
betterthantheother.Noticethattheindependentvariableisactivebuthasnotbeenmanipulatedbytheinvestigator.
Thus,activeindependentvariablesaregiventotheparticipantsinthestudybutarenotnecessarilymanipulatedbytheexperimenter.Theymaybegivenbyaclinic,
school,orsomeoneotherthantheinvestigator.Fromtheparticipants'pointofview,thesituationwasmanipulated.
AttributeIndependentVariables
Unlikesomeauthorsofresearchmethodsbooks,wedonotrestrictthetermindependentvariabletothosevariablesthataremanipulatedoractive.Wedefinean
independentvariablemorebroadlytoincludeanypredictors,antecedents,orpresumedcausesorinfluencesunderinvestigationinthestudy.Attributesofthe
participants,aswellasactiveindependentvariables,wouldfitwithinthisdefinition.Forthesocialsciences,education,anddisciplinesdealingwithspecialneeds
populations,nonactiveindependentvariablesareespeciallyimportant.Typeofdisabilityorlevelofdisabilityisoftenthemajorfocusofastudy.Disabilitycertainly
qualifiesasavariablebecauseitcantakeondifferentvalueseventhoughtheyarenot''given"inthestudy.Forexample,cerebralpalsyisdifferentfromDown
syndrome,whichisdifferentfromspinabifidayetallaredisabilities.Also,therearedifferentlevelsofthesamedisability,whichmaytakeonvaluesdefinedby
functionalleveloranatomicallevel.Spinalcordinjuryisagoodexampleofthelatter.However,wecannotgivelevelsofspinalcordinjury.Specifically,wecannot
assignormakeonegroupofpeopleintothecategoryofquadriplegiaandanothergroupofpeopleintothecategoryofparaplegia.Peoplealreadyhavedefining
characteristics,orattributes,thatplacethemintooneofthesetwocategories.

Page50

Thedifferentdisabilitiesarealreadypresentwhenweperformourstudy.Thus,wearealsointerestedinstudyingaclassofvariablesthatcannotbegivenduringthe
study,evenbyotherpersons,schools,orclinics.
Avariablethatcannotbegiven,yetisamajorfocusofthestudyiscalledanattributeindependentvariable(Kerlinger,1986).Inotherwords,thevaluesofthe
independentvariableareanattributeoftheperson.Forexample,gender,age,ethnicgroup,ordisabilityareattributesofaperson.Anothercommonlabelforan
attributeindependentvariableisameasuredvariable.However,wedislikethislabelbecauseonecouldeasilyconfuseanattributeindependentvariablewiththe
dependentvariable,whichisalsomeasured.
Othermethodologists,especiallyexperimentalpsychologists,havereferredtoattributeindependentvariablesasnuisancevariables(Kirk,1982)because,fromastrict
experimentaliststandpoint,theresearcherwouldmakeanattempttogetridof,orcontrol,thesevariables.Sometimesdemographicvariablessuchasgenderorethnic
grouparecalledmoderatorormediatingvariablesbecausetheyservethesefunctions.However,ineducationandthehealthsciences,weoftenhaveasubstantive
interestinattributevariables.
TypeofIndependentVariableandInferencesaboutCauses
Whenweanalyzedatafromastudy,thestatisticalanalysisdoesnotdifferentiatebetweenwhethertheindependentvariableisanactiveindependentvariableoran
attributeindependentvariable.However,eventhoughwegivethelabelindependentvariabletobothactiveandattributevariables,thereisacrucialdifferencein
interpretation.Asignificantchangeordifferencefollowingmanipulationoftheactiveindependentvariablemayreasonablyleadtheinvestigatortoinferthatthe
independentvariablecausedthechangeinthedependentvariable.However,asignificantchangeordifferencebetweenoramongvaluesofanattributeindependent
variableshouldnotleadonetotheinterpretationthattheattributeindependentvariablecausedthedependentvariabletochange.Thisimportantdistinctionisamplified
inthenextchapterwhenwecomparetheexperimental(active)andindividualdifferences(attribute)approachestoresearch.
LevelsoftheIndependentVariable
Earlier,indefiningavariable,wesaidthatitmusttakeondifferentvalues.Whendescribingthedifferentvaluesofanindependentvariable,thewordlevelis
commonlyuseditdoesnotnecessarilyimplythatthevaluesareordered.1Supposethataninvestigatorisconductingastudytoinvestigatetheeffectofatreatment.
Onegroupofpatientsisassignedtothetreatmentgroup.Asecondgroupofpatientsdoesnotreceivethetreatment.Thestudycouldbeconceptualizedashavingone
independentvariable,treatmenttype,withtwolevels,treatmentandnotreatment.Theindependentvariableinthisexamplewouldbeclassifiedasanactive
independentvariable.Supposeinsteadthattheinvestigatorwasinterestedprimarilyincomparing
1

Thetermscategories,values,groups,andsamplesaresometimesusedinterchangeablywiththetermlevels,especiallyinstatisticsbooks.Likewise,thetermfactorisoftenused
insteadofthetermindependentvariable.

Page51

twodifferenttreatmentsbutdecidedtoincludeathirdnotreatmentgroupasacontrolgroupinthestudy.Nowthestudycouldbeconceptualizedashavingone
independentvariable,treatmenttype,withthreelevels,thetwotreatmentconditionsandthecontrolcondition.Again,theindependentvariableinthisexampleisan
activeindependentvariable.Thisvariablecouldbediagramedasfollows:

Now,let'sthinkaboutgender,whichisanattributeindependentvariablewithtwolevels,maleandfemale.Itcouldbediagramedasfollows:

Thedesignofastudywithoneindependentvariableisoftencalledasinglefactordesignandtheappropriatestatisticsarecalledsinglefactororbasicstatistics(see
chap.1415).
MoreThanOneIndependentVariable
Wecouldexpandourstudyofthreelevelsoftreatmentbyaddinggenderasasecondindependentvariable.Whenmorethanoneindependentvariableisincludedina
study,theinvestigatorisofteninterestednotonlyintheeffectofeachindependentvariablebyitself,butintheinteractionbetweenthetwoindependentvariables.We
discussvariables,levels,andinteractionsinmoredetailinlaterchaptersbecausemostpublishedstudiesandtheseshavemorethanoneindependentvariable.Studies
withtwoindependentvariablesarecalledtwofactor,orfactorial,designs.
DependentVariables
Thedependentvariableistheoutcomeorcriterion.Itisassumedtomeasureorassesstheeffectoftheindependentvariable.Dependentvariablesarescoresfroma
test,ratingsonquestionnaires,readingsfrominstruments(electrocardiogram,galvanicskinresponse,etc.),ormeasuresofphysicalperformance.Whenwediscuss
measurement,weareusuallyreferringtothedependentvariable.Specifically,scaleofmeasurement,standardization,reliability,andvalidityarealltermsthatare
importantforchoosinganddescribingthedependentvariable(seechap.9and20).Justasitiscommonforathesisorpublishedstudytohavemorethanone
independentvariable,itiscommontohaveseveraldependentvariablesormeasures,forexample,performanceandsatisfaction.Dependentvariablesalsohaveat
leasttwolevels,butitiscommonforthemtohavemanyorderedlevels(i.e.,fromlowtohigh).

Page52

ExtraneousVariables
Extraneousvariablesarenotofinterestinaparticularstudy,buttheycouldinfluencethedependentvariable.Environmentalfactors(e.g.,temperatureordistractions),
timeofday,otherattributesoftheparticipants,andcharacteristicsoftheinvestigator,teacher,ortherapistaresomepossibleextraneousvariablesthatneedtobe
controlledbymethodssuchasholdingthemconstant,randomization,statistics,ormatching.Wediscussthesevariablesinmoredetailinseverallaterchapters.
OrderedVersusUnorderedorNominalLevelsofVariables
Anotherimportantfeatureofanyvariableisthatthelevelsmaybeunorderedcategoriesortheymaybeorderedand,perhapsevencontinuousthatis,variablesmay
haveaninfinitenumberoforderedcategoriesonacontinuum.Remember,earlieritwasstatedthatavariablemustvarythatis,itmustbeabletotakeondifferent
values.Mostofourexamplesofindependentvariables(e.g.,gender,treatmenttype)hadlevelsorcategoriesthatwerenotordered.Thecategoriesinsuchvariables
areessentiallylabelsornamesandthevariablesaresaidtobenominal(from"name")variables.Forexample,theindependentvariablegenderhastwo,nonordered
levels,femaleandmale.Nominalvariables,accordingtoKerlinger(1986),"havesimplerequirements:allthemembersofasubsetareconsideredthesameandallare
assignedthesamenameandthesamenumeral"(p.36).Forexample,allfemalesareconsideredthesameandassignedtothecategoryoffemale.Allmalesarethe
sameandassignedtothecategoryofmale,butthecategoriesarenotorderedsuchthatfemalesaremoreorlessthanmales.
Ontheotherhand,orderedvariableshaveasetofvaluesthatvaryfromlowtohighwithinacertainrange(e.g.,a17ratingofsatisfaction),suchthatalargervalueof
thevariableindicatesmoreofitthanasmallervalueofthevariable,andthereisanassumptionthatthereareorcouldbeaninfinitesetofvalueswithintherange
(Kerlinger,1986).Weightandtimetorecoveryarecontinuousvariables,butmanyorderedvariableshaveonlyafewlevelsorcategories,suchashigh,medium,and
low.Weexpandonthisintroductiontomeasurementinchapter9.
OtherConsiderationsaboutVariables
Forthemostpart,thestudieswediscusshaveindependentvariablesthathaveafewlevelsanddependentvariablesthathavemanyorderedlevels.However,inthe
associationalapproachdiscussedinchapter5,bothindependentanddependentvariablesusuallyhavemanyorderedlevels.Also,therearesomestudiesinwhichthe
independentvariableshavemanylevelsandthedependentvariablehastwoorafewlevels,andthereareevenstudieswherebothvariableshaveonlyafewlevels.
Wediscussthesedifferentcombinationsofindependentanddependentvariablesandhowtheyareanalyzedlaterinthebook.

Page53

Somevariables(e.g.,knowledgeofmathematicsorselfconcept)couldbeeithertheindependentvariableorthedependentvariable(orevenanextraneousvariable),
dependingonthestudy.Thesevariablesareusuallyachangeablecharacteristicoftheparticipant(likeanattitudeorpersonalitycharacteristic)ifoneoftheseisused
astheindependentvariable,itisanattributeindependentvariable.
Individualparticipantsdonothavetovaryonacharacteristicorvariableitisthegroupthatmusthavemorethanonevalue(e.g.,somemenandsomewomen).In
somestudies,participantsmaychangeovertimeorasaresultofanintervention.Inthesestudies,therearerepeatedmeasuresofthesamevariable(e.g.,apretestand
aposttestonmathknowledge).
GroupsorSetsofVariables
Inanalyzingcomplexresearcharticles(seechap.24),studentsoftenhavedifficultydistinguishingbetweenvariablesandthelevelsofvariables.Thereasonthat
thissortingisdifficultisthatincomplexstudiesresearchershavemanyvariablesthattheyoftengroupintowhatmightbecalledsetsofsimilarvariables.Forexample,
age,gender,education,andmaritalstatusvariablescouldbegroupedtogetherandreferredtocollectivelyinanarticleasdemographics.Similarly,verbal,quantitative,
andanalyticalscoresonGraduateRecordExams(GRE)couldbecalledGREscores.Theconfusionarisesifonemistakenlyassumesthatthesetsorgroupsof
variables(demographicsandGREscores)arethevariables,andtheactualvariables(age,gender,GREverbal,etc.)arethelevels.
Howcanoneavoidthisconfusion?Thoughtfulreadingisthekey,butsometipsmayhelp.Rememberthatavariablehastohaveatleasttwolevels,butalevelor
categoryisasinglevalue.2Thus,ifsomethingcanvaryfromlowtohigh(e.g.,ageorGREverbal)orhastwoormorenominalvalues(e.g.,gender),ithastobea
variable,notalevel.
HypothesesandResearchQuestions
Nowthatwehavediscussedvariablesindetail,let'sreturntohypotheses.Broadlyspeaking,ahypothesisisapredictivestatementabouttherelationshipbetweentwo
ormorevariables.Inlaterchapters,weexpandonthisdefinition.Inchapter11,wediscussnull(nodifference)hypothesesthataretestedwhenweuseinferential
statistics.However,fornowwecanusethegeneraldefinition.Inchapter5wedescribetwobasickindsofresearchquestions(differenceandassociational),which
arelikehypothesesinquestionformat.However,itisimportanttopointoutthatanyresearchhypothesisorquestioncanbephrasedasarelationshipbetween
variables.Alsonotethatmoststudieshaveseveralhypothesesorresearchquestions.Weexaminethisscenariointhenextchapterandthroughoutthebook.
2

Insomecasesalevelmaybearangeofvalues(e.g.,ages2130years),butinthesecasesthevaluesinagivenrangearetreatedasiftheywereallthesame(e.g.,youngadultor
givenasinglegroupcodesuchas3).

Page54

TheAnxietyandHeartRateProblemRevisited
Earlier,westatedoneresearchproblemas,Howareheartrateandmetabolicrequirementsrelatedunderconditionsofanxiety?Perhapsaclearerwaytostate
theproblemis,Dometabolicrequirementsdifferbetweentheconditionsofanxietyandexercisewhenheartrateisheldconstant?Byexaminingourproblem
statementclosely,wecandeterminethatanumberofvariablesmaybeinvolved.Heartrate,metabolicrequirements,exercise,andanxietyallmightbevariables.But
remember,avariablemustbeabletovary.Let'sreexamineourstudy.Underonecondition,calledanxiety,heartrateandmetabolicrequirementsaremeasured.
Underasecondcondition,whichisexercise,heartrateandmetabolicrequirementsarealsomeasured.Therefore,onevariablemightbelabeledtypeofcondition
withtwolevels,anxietyandexercise.Typeofconditionisanactiveormanipulatedindependentvariable,andanxietyandexercisearelevelsorcategoriesofthat
variable,notvariablesthemselves,becauseinthisstudytheydonotvary.Asecondsetofvariablesismetabolicrequirements,whichisoperationallydefinedby
cardiacoutputandoxygenuptake.Thesearethetwodependentvariables.Dowehaveanothervariable?Whataboutheartrate?Rememberthat,inourstudy,weare
attemptingtocreatetwodifferentsituations,anxietyandexercise,withidenticalheartrates.However,thetwoconditionsyieldidenticalheartratestherefore,heart
ratecannotbeavariablebecauseitwillnotvaryinthetwosituationsofthisstudy.Thatis,heartrateiscontrolledratherthanallowedtovaryinthisstudy.
Broadlyspeaking,ourresearchproblemaskswhethertherewillbearelationshipbetweenthesituationandmetabolicrequirements.Morespecifically,we
hypothesizedthattheexerciseconditionwouldproducesignificantlyhighercardiacoutputandoxygenuptakethantheanxietyconditionwhenheartratewasequal.
Werethehypothesestested?Yes,Gliner,Bedi,andHorvath(1979)testedthehypothesisforthemetabolicrequirementofcardiacoutput.Gliner,Bunnell,and
Horvath(1982),replicatedthestudyforcardiacoutputandalsomeasuredoxygenuptake.Inbothstudies,thehypothesiswassupported.
TheMasteryMotivationProblemRevisited
Earlierwestatedthesecondresearchproblemas,Istherearelationshipbetweenprematurityandabuseorneglectandmasterymotivation?Actually,this
generalproblemledtotwoseparatestudies.Onestudycomparedprematureinfantstofullterminfantsmatchedongestationalage,andtheothercomparedthree
groupsofinfants:abused,neglected,andlowrisk(controlgroup).
Inthefirststudy,theindependentvariablewasgestationalstatus,withtwolevels,pretermandfullterm.Thisisanattributeindependentvariable.Inthesecondstudy,
theindependentvariablemightbecalledriskstatus,ortype,withthreelevels:abused,neglected,andlowrisk.Again,thisisanattributevariablebecauseitwasnot
andcouldnotethicallybemanipulatedorgiventothechildren.Ifaninterventionweregiventosomeinfantsbutnotothers,thatvariableinterventiontypewouldbe
anactiveindependentvariable.

Page55

Whataretheoutcomeordependentvariablesinthisstudy?Twoaspectsofmasterymotivationweremeasured,taskpersistenceandmasterypleasure,eachofwhich
isavariable.Theyarenottwolevelsofonevariablebecauseeachvariesfromlowtohighandinfantshadscoresonboth.
Whataboutage?Itwasnotavariablebecauseallinfantswere12monthsgestationalage.Likewise,infantsarenotavariable.Alloftheparticipantswereinfants,so,
asweseeinchapter10,theyarethepopulationstudied.
Theory
Wheredoestheorycomeintoplayforourproblems,variables,andhypotheses?Firstweshoulddefinetheory.Kerlinger(1986)suggeststhatatheoryexplains
naturalphenomena,whichisagoalofscience,andhedefinesitas,"Atheoryisasetofinterrelatedconstructs(concepts),definitions,andpropositionsthatpresenta
systematicviewofphenomenabyspecifyingrelationsamongvariables,withthepurposeofexplainingandpredictingthephenomena"(p.9).Krathwohl(1993)
suggeststhatatheoreticalorientationshouldbepresentedatthebeginningofanarticleasthebasisforunderstandingtherestofthearticle.Hedoesnotbelievethata
theorymustbeapartofthearticleinstead,explanation,rationale,orpointofviewcouldbesubstitutedtosatisfy,tosomeextent,thesamepurposeastheory.
However,noneoftheseconceptsareasstrongasatheory.Anumberofbooksdealextensivelywiththeroleoftheoryinresearch,butwehavechosentoemphasize
researchdesignandhowitinfluencesdataanalysisandtheinterpretationofresults.
Theissueincontemporarysocialandhealthscienceresearchisnotsomuchwhethertheoryisimportantandhowitshouldfitintoanarticle,buthowimportantshould
theorybeindesigningresearch?Weagreethattheoryisimportantandthatthevalueofastudydepends,inpart,onwhethertheresultssupportatheoryornot.
However,theorydrivenresearchisnotnecessarilywelldesignedandanalyzed,whichisthefocusofthisbook.
Summary
Aresearchproblemisastatementthataskswhetherthereisarelationshipbetweentwo,orlikelymore,variables.Itresultswhenaresearcherhasexperiencedan
obstacletounderstanding.
Avariableisacharacteristicoftheparticipantsorsituationofagivenstudythathasdifferentvaluesorlevels.Thereareanumberofpointstorememberabout
variables:
1.Therearethreemaintypesofvariables:
a)Independentvariables,whicharethepresumedcauses,influences,orantecedentsinthestudy.Wedifferentiatedtwotypesofindependentvariable:
1)Active,whichisavariablethatisgiventotheparticipants,usuallyforsomespecifiedtimeperiod.Itmaybe,butisnotalways,manipulatedand
controlledbytheinvestigator.

Page56

2)Attribute,whichisanobservedormeasuredcharacteristicoftheparticipantsorenvironmentthateitherwasnotorcannotbemanipulatedbythe
investigator.
b)Dependentvariables,whicharethepresumedoutcomesorcriteria.
c)Extraneousvariablesarenotofinterestinthisstudybuttheycouldinfluencethedependentvariable.
2.Ifeveryoneinastudyisthesameonagivencharacteristic,thatcharacteristicisconstantnotavariableinthestudy.Forexample,inastudyof9yearoldboys'
learningofmathematicsinoneoftwocurricula,theparticipants'ageandgender(9yearoldboys)arenotvariables.
3.Thedifferencebetweenthevariableandthelevels,orcategories,ofavariableisimportant.Thevariableitselfhasorisgivenanamethatencompassesallofthe
levelsorcategories(e.g.,treatmenttype,gender,orethnicity).Thelevelsarethenamesofthespecificcategoriesorgroupsorvalues(e.g.,experimentalversus
control,maleversusfemale,AsianversusAfricanversusEuropean).Inthiscontext,leveldoesnotnecessarilyimplyorder,andonelevelisnotnecessarilyhigheror
lowerthananother.
4.Variablescanhaveeithernominal(unordered)levelsorcategoriesorhaveorderedlevels.
5.Somevariables(e.g.,knowledgeofmathematicsorselfconcept)couldbeeithertheindependentvariableorthedependentvariable(orevenanextraneous
variable),dependingonthestudy.Thesevariablesareusuallyachangeablecharacteristicoftheparticipant(likeanattitudeorpersonalitycharacteristic)ifoneof
thesevariablesisusedastheindependentvariable,itisanattributeindependentvariable.
6.Individualparticipantsdonothavetovaryonacharacteristicorvariableitisthegroupthatmusthavemorethanonevalue(e.g.,somemenandsomewomen).In
somestudiestherearerepeatedmeasuresofthesamevariable(e.g.,apretestandaposttestonmathknowledge)andindividualsmaychangeovertimeina
longitudinalstudy.
Theresearchproblemisusuallystatedmorebroadlythantheresearchhypothesesorquestions.Moststudieshaveseveralhypothesesorquestionsthatindicate
predictedorpossiblerelationshipsbetweenvariables.Inchapter5wedescribesixspecifictypesofresearchquestionsandfivetypesofresearchapproachesthat
formthebasisforanunderstandingofresearchdesignanddataanalysis.
StudyAids
Concepts
Hypothesis
Literaturereview

Page57

Operationaldefinition
Researchliterature
Researchproblem
Researchquestion
Testable
Theory
Variable
Distinctions
Activeversusattributeindependentvariable
Independentversusdependentversusextraneousvariable
Levelsofonevariableversusasetorgroupofvariables
Orderedversusunorderedornominalvariables
Researchproblemversuspersonalorsocietalproblem
Thevariable(itself)versuslevelsorcategoriesofthevariable
ApplicationProblems
Providethefollowinginformationforeachhypothesis.
1.Nametheindependent/antecedent/predictorvariable.
2.Istheindependentvariableactiveoranattribute?
3.Howmanylevelsofthisvariablearethere?
4.Arethelevelsorderedornominal?
5.Namethedependent/outcomevariable?
6.Isthepopulationofinterestnamed?Whatisit?(seep.55)
7.Giveanoperationaldefinitionofeachvariable.Ifactive,howmighttheindependentvariablebemanipulated?Ifanattribute,howwilltheattributebemeasured?
Howwillthedependentvariablebemeasured?
a.Familyconflictisassociatedwithabsenteeismratesinclericalworkers.
b.Aworkshoponvisualimageryimprovesmemoryincollegestudents.
c.Thenumberoffacultymembersatacommitteemeetingisrelatedtothelengthofthemeeting.
d.Theamountofchildabuseisrelatedtotheageofparentswhentheymarried.
e.Avoters'politicalpartyisrelatedtotheirattitudetowardguncontrol.
f.Whetherapregnantwoman'sdietwashigh,medium,orlowinfolicacidaffectsthebirthweightofherchild.
g.Studentsgivenanexerciseprogramhavereducedlevelsofstress.
h.Thegenderoftheinstructorisrelatedtostudents'evaluationoftheinstructor.
i.Participationinananxietyreductionworkshopisrelatedtotestperformance.

Page59

PARTII
RESEARCHAPPROACHESANDDESIGNS

Page61

Chapter5
OverviewofResearchApproachesandQuestions
Overview
ResearchApproacheswithanActiveIndependentVariable
TheRandomizedExperimentalResearchApproach
RandomAssignment
RandomAssignmentVersusRandomSelectionorSampling
AnExample
InferringCausation
SummaryoftheRandomizedExperimentalApproach
TheQuasiExperimentalResearchApproach
ResearchApproacheswithAttributeIndependentVariables
TheComparativeResearchApproach
AddingLevelstotheIndependentVariable
ControllingforNoTreatmentEffects
DeterminingtheCompleteRelationship
SummaryoftheApproachesThatCompareGroups
TheAssociationalResearchApproach
SummaryDiagramsfortheFirstFourApproaches
TheDescriptiveResearchApproach
CombinationsofResearchApproaches
ResearchQuestionsorHypothesesandStatistics
ThreeTypesofBasicHypothesesorResearchQuestions
DifferenceVersusAssociationalInferentialStatistics
SixTypesofResearchQuestions:BasicandComplex
Summary

Page62

StudyAids
Concepts
Distinctions
ApplicationProblems
Overview
Thegeneralpurposeofallresearchstudies,exceptthosethatwecall(purely)descriptive,istolookforrelationshipsbetweenvariables(seeFig.5.1).Wedivide
approachestoresearchintothreegeneraltypes:experimental,individualdifferences(ornonexperimental),anddescriptive.Thefirsttypeofapproachhasan
activeindependentvariable,thesecondhasanattributeindependentvariable,andthedescriptiveapproachdoesnothaveanindependentvariable.Weusethelabel
descriptiveapproachmorenarrowlythansomewriterstoindicatestudiesthatdonotuseinferentialstatisticstotesthypotheses.
Next,wedivideboththeexperimentalandtheindividualdifferencesapproachesintotwospecificapproaches:randomized(ortrue)experimentalversusquasi
experimentalandcomparativeversusassociational.Thesefourspecificapproachesallseektofindrelationshipsamongvariablestheydifferintermsofpurposesand
inwhatkindsofhypothesesorresearchquestionstheyanswer.Figure5.1indicatesthespecificpurposeforeachofthefivespecificapproaches.Noticethatthe
randomizedexperimentalapproachisthebestsuitedtodeterminecauses.Noneoftheotherapproachescanprovidesolidevidenceaboutwhethertheindependent
variablecausedachangeinthedependentvariable,butthequasiexperimentalapproachprovidessomeevidenceforcausalityiftheapproachisstrong.Fora
discussionofthestrengthofquasiexperimentsseechapter7.
Inthesectionthatfollowswediscusstheutilityofthesefiveapproachesinproducingconclusionsaboutcauseandeffect.Wealsoexaminethesimilari

Page63

Fig.5.1
Schematicdiagramoffivebasicapproachestoresearchandtheirpurposes.

tiesanddifferencesbetweeneachofthefiveapproachesanddemonstratethatmoststudiesareactuallymorecomplexbecausetheycombinetwoormoreofthe
approaches.Table5.1showsthefivespecificresearchapproachesandthecriteriathatdistinguishesthem.
ResearchApproacheswithanActiveIndependentVariable
TheRandomizedExperimentalResearchApproach
Foraresearchapproachtobecalledrandomized(ortrue)experimental,twocriteriamustbemet.Thefirstcriterionisthattheresearchermustrandomlyassign
participantstogroupsorconditions.(Wehaveusedthewordconditioninadditiontogroupbecauseundercertaincircumstancesagroupcanundergoboththe
controlandtheinterventionconditions.)AsyoucanseefromTable5.1,thiscriterioniswhatdifferentiatesrandomizedexperimentsfromquasiexperiments,butitis
themostdifficulttoachieve.Muchappliedresearchinvolvesgroupsthatarealreadyintactorinexistencesuchasclassroomsorrehabilitationsettings,anditisnot
possibletochangethoseassignments.Thus,suchresearchisnotconsideredtoberandomizedexperimental.
Thesecondcriterionthatmustbesatisfiedfortheresearchapproachtobeconsideredrandomizedexperimentalisthattheindependentvariablemustbeactiveas
definedinthelastchapter.Inadditiontheresearcherusuallyisableto

Page64

controltheindependentvariable.Inotherwords,theresearchercandecideexactlywhatthetreatmentwillbeandwhenandtowhomitwillbegiven.Forexample,
sheorhewillbeabletorandomlyassignoneleveloftheindependentvariabletotheinterventionconditionandtheotherleveloftheindependentvariabletothe
controlcondition.TheseaspectsofrandomizedexperimentsarealsoshowninTable5.1.
RandomAssignment
Whyisrandomassignmentsoimportant?Theconceptofrandomnessimpliesthatthereisnobias.Whentheinvestigatorrandomlyassignsparticipantstogroups,it
meansthateachparticipanthasanequalchancetobeineithertheinterventiongrouporthecontrolgroup.Randomdoesnotmeanhaphazardoranyoldway,asit
sometimesdoesinpopularlanguage.Onecoulduseacomputergeneratedrandomnumbertableoramethodliketheonedescribedinthenextparagraphto
randomlyassignparticipantstogroups.
Forexample,supposethereare20participantsinastudy.Thestudycallsfor10personstobeineachoftwogroups.Theinvestigatorcantake20piecesofpaper,
andonhalfofthepiecesprintazeroandontheotherhalfprintaone.
TABLE5.1
AComparisonoftheFiveBasicQuantitativeResearchApproaches
Criteria

Randomized
Experimental

Quasi
Experimental

Comparative

Associational

Descriptive

Randomassignmentof
participantstogroups
byinvestigator

Yes

No

No

No(onlyone
group)

Nogroups

Independentvariableis
active

Yes

Yes

No(attribute)

No(attribute)

Noindependent
variable

Independentvariableis
controlledbythe
investigatora

Usually

Sometimes

No

No

No

Independentvariable
hasonlyafewlevelsor
categories

Usually

Usually

Usually

Sometimes

Noindependent
variable

Relationshipsbetween
variablesorcomparison
ofgroups

Yes
(comparison)

Yes
(comparison)

Yes
(comparison)

Yes
(relationship)

No

Althoughthisisadesiredqualityofrandomizedexperimentalandquasiexperimentaldesigns,itisnotsufficientfor
distinguishingbetweentherandomizedexperimentalandquasiexperimentalapproaches.

Page65

Then,theinvestigatorplacesall20piecesofpaperinabowl,andshakesthebowltomixupthepiecesofpaper.Now,wheneachparticipantcomesinforthestudy,
theinvestigatorreachesintothebowlandpullsoutapieceofpaper.Ifthepaperhasazeroonit,theparticipantisassignedtothecontrolgroup.Ifthepaperhasa
oneonit,theparticipantisassignedtotheinterventiongroup.(Itisimportantinthissituationthattheinvestigatordoesnotputthepieceofpaperbackintothebowl
aftereachparticipant'sassignmentismade.)Thisprocedurecontinuesuntilall20participantshavebeenassignedtoeithertheinterventionorthecontrolgroup.
Therefore,priortotheintervention,theassumptionismadethattheparticipantsinthetwogroupsareequivalentinallotherrespects,includingdemographic
characteristicsandthedependentvariable,ifthatistobemeasuredbeforetheintervention.Inthepracticalsituation,twosmallgroupsof10personseach,asinthe
aboveexample,maynotbeequalbecausetheconceptofrandomnessonlymakesthingsequalinthelongrun,withrelativelylargenumbersofparticipantsineach
group(e.g.,30).However,afterrandomassignment,evenifthetwogroupsarenotexactlyequal,thedifferencesbetweenthemareconsideredtobeunbiasedand
canbeadjustedstatistically.
RandomAssignmentVersusRandomSelectionorSampling
Itisveryimportanttounderstandthesetwoconceptsandtoknowthedifferencebetweenthem.Theconceptrandom,orunbiased,is,ofcourse,commontoboth
conceptsandtoseveralotherphrases,suchasrandomorderandrandomassignmentoftreatmentstogroups,whichwediscussinthisandlaterchapters.Random
samplingorrandomselectionofparticipantsfromthepopulation,ifdoneinastudy,comesbeforerandomassignmenttogroupsintheprocedure.Asweseein
chapter10,randomselectionhastodowithwhotheparticipantsinthestudywillbe,andhowtheyareselected.Intheidealsituation,describedindepthinchapter
10,thesampleisselectedtoberepresentativeofallthepossibleparticipantswhofittheselectioncriteria.Forexample,intheDiPasqualeLehnerz(1994)study
describedinthefollowingsection,thetheoreticalpopulationofinterestmighthavebeenallthepersonsintheworldwhohadspinalcordinjuries.Forpracticalreasons,
theresearcherchosetostudypersonswithsuchinjuriesfromonerehabilitationcenterratherthanrandomlyselectingasampleofsuchpersonsfromaroundtheworld
ortheUnitedStates.Asweshowinchapter10,thistypeofconveniencesampleiscommonandoftennecessary.Nonrandomsamplingaffectsthegenerality
(externalvalidity)ofthestudy,butitisnotrelevantasfarastheapproachisconcerned.Thus,randomselectionorsamplingisnotshowninTable5.1.A
randomizedexperimentmayormaynotuserandomselectionorsampling.Althoughastudywithaweaksamplingprocedurecanstillbearandomizedexperiment,its
overallqualitywillbereduced,asdiscussedinchapter24.Itisalsotruethatinferentialstatisticsassumethatthesamplestudiedisarandomsampleofthepopulation
ofinterest.Ifitisnot,thestatisticalresultsmaybemisleading.Nevertheless,arandomizedexperimentdoesnotnecessarilyinvolverandomsampling.
Randomassignmentoftheparticipantstotheinterventionversuscontrolgroupsisrelevantforinternalvalidityandforinferringcausation.Randomassignmenthasto
dowithhowparticipantsgotintotheirparticulargroups.Was

Page66

theassignmentdonerandomly,asillustratedbythe20piecesofpaperexample?Orwasthereadegreeofbias?
AnExample
ConsiderastudybyDiPasqualeLehnerz(1994),whowasinterestedinusingorthoses(handsplints)toincreasehandfunctionwithpersonswhohadcervical6level
spinalcordinjuries.Usingparticipantsattherehabilitationcenterwheresheworked(note,therewasnotrandomsampling),DiPasqualeLehnerzrandomlyassigned
participantstooneoftwogroups.Onegroup(intervention)receivedthehandsplints.Asecondgroup(control)didnotreceivethehandsplints.DiPasqualeLehnerz
measuredhandfunctionat4weeks,8weeks,and12weeksintothestudybyusingtheJebsonhandfunctiontestandotherstrengthmeasures.
IstheDiPasqualeLehnerzstudyarandomizedexperiment?Thestudysatisfiedthesecondcriterionbecausetheindependentvariablewasactive(manipulated).She
alsodecidedwhatthetreatmentwouldbeandwhichgroupshouldgetthetreatment,soshehadcontrolovertheindependentvariable.Thefirstcriterionwasalsomet
becauseshewasabletorandomlyassignparticipantstogroups.Thus,thestudywasarandomizedexperiment.
InferringCausation
Porter's(1997)threecriteriaforcausearenecessaryforpostulatingthatanindependentvariablecausedachangeinadependentvariable.Thethreecriteriaarethat
theindependentvariablemustprecedethedependentvariable,theindependentvariablemustberelatedtothedependentvariable,andtheremustbenothirdvariable
thatcouldexplainwhytheindependentvariableisrelatedtothedependentvariable.
DidtheindependentvariableprecedethedependentvariableintheDiPasqualeLehnerzstudy?Theproceduresofthestudywere,first,therandomassignmentof
participantstogroups,andtheassigningoftheindependentvariable(treatment)tooneofthegroups.Next,thetreatmentwasgivenand,finally,theresearcher
measuredthechangeinthegroups.Thus,theindependentvariableprecededthedependentvariableinthestudy.Wereextraneousvariablesruledout?DiPasquale
Lehnerzrandomlyassignedparticipantstogroups,andthenrandomlyassignedtreatmentstogroups.Thegroupswerepresumedtobeequalpriortotheintroduction
oftheinterventionandweretreatedidenticallyduringthestudyexceptforwhetherornottheyhadasplint.Therefore,iftherewasachangeinthedependentvariable
inthegroupthatreceivedtheintervention,andnochangeinthegroupthatdidnotreceivetheintervention,itwouldbedifficulttopostulateathirdvariablebeing
responsibleforanychangeinthedependentvariable.Wastherearelationbetweentheindependentvariableandthedependentvariable?Unfortunately,therewereno
statisticallysignificantdifferencesbetweengroups.Therefore,eventhoughDiPasqualeLehnerzsatisfiedtheconditionsofarandomizedexperiment,inthisparticular
studytheindependentvariablefailedtocauseachangeinthedependentvariable.
SummaryoftheRandomizedExperimentalApproach
TheDiPasqualeLehnerzstudyusedoneofseveralspecificexperimentaldesignsthepretest

Page67
1

posttestcontrolgroupdesignwhichwillbedescribedinchapter7. Whatthisspecificexperimentaldesignhasincommonwith,andwhatdistinguishesitfrom,the
otherfourresearchapproachesthataredescribedinthischapterisshowninTable5.1.Notethatrandomassignmentoftheparticipantsbytheinvestigatortolevels,
orgroups,orconditionsiswhatdistinguishestherandomizedexperimentalapproachfromthequasiexperimentalapproach.Intherandomizedexperimentalapproach
therearealsoseveralcharacteristicsincommonwithquasiexperiments:anactiveindependentvariable(e.g.,treatmenttype)thathasonlyafewcategoriesorlevels
(e.g.,treatmentandcontrol).Thetablealsoshowsthat,inexperiments,usuallytheinvestigatorhassomecontrolovertheindependentvariable,andthegroupsare
compared(e.g.,thetreatmentiscomparedtothecontrol).
TheQuasiExperimentalResearchApproach
Wedividethequasiexperimentalapproachintothreecategories,whichwedescribeinmoredetailinchapters7and8.Thesecategoriesarepretestposttest
designs,timeseriesdesigns,andsinglesubjectdesigns.
Thepretestposttestquasiexperimentalresearchapproachissimilartotherandomizedexperimentalapproach,butfailstosatisfytheconditionofrandom
assignmentofparticipantstogroups.Inthesedesigns,forexample,participantsarealreadyinintactgroups,suchastwodifferentclassrooms,priortothestudy.Both
groupsaremeasured(pretest)priortotheintroductionoftheindependentvariable.Onegroupreceivestheindependentvariableandtheothergroupdoesnotgetthe
independentvariable.Attheendofthestudy,bothgroupsaremeasuredagain(posttest).
NoteinTable5.1thatquasiexperimentaldesignshaveanactiveindependentvariablewithafewcategoriesandalsoinvolveacomparisonbetween,forexample,an
interventionandacontrolcondition.However,thereisawordofcautionabouttheactiveindependentvariable.Intherandomizedexperimentalapproach,the
researcherusuallyhascontrolovertheindependentvariableinthatonelevelcanberandomlyassignedtotheexperimentalcondition,andonelevelcanberandomly
assignedtothecontrolcondition.Thestrengthofthequasiexperimentaldesignisbased,inpart,onhowmuchcontroltheinvestigatoractuallyhasinmanipulatingthe
independentvariableanddecidingwhichgroupwillreceivewhichtreatment.Inchapter7weillustratehowdifferentlevelsofcontroloftheindependentvariableaffect
thestrengthofthequasiexperimentaldesign.Thestrengthofthedesigninfluenceshowconfidentwecanbeaboutwhethertheindependentvariablewasthecauseof
anychangeinthedependentvariable.
Timeseriesdesignsincludesinglegrouptimeseriesdesignsandmultiplegrouptimeseriesdesigns.Timeseriesdesignsaredifferentfrompretest
1

Itshouldbenotedthatifthereisapretestandposttest,asinthisstudy,thedesignisnolongerabasicsinglefactordesign.Wediscussothertypesofdesigninlaterchapters.
Fornow,ignorethepretestandthinkoftherandomizedandquasiexperimentaldesignsascomparingtwogroupsonaposttestthatmeasuresthedependentvariable.

Page68

posttestdesignsseenintherandomizedandquasiexperimentalapproach.InPretestPosttestdesignsthereareonlytwomeasurementperiods.Intimeseriesdesigns
therearenumerousmeasurementperiodspriortoandaftertheintroductionoftheintervention.Whereassinglegrouptimeseriesdesignsincludeonlyonegroup,the
interventiongroup,multiplegrouptimeseriesdesignsaddacontrolgroupalongwiththemultiplemeasurements.
Singlesubjectdesignsalsoincludemultiplemeasurementperiodssimilartotimeseriesdesigns.However,singlesubjectdesignsincludefewparticipants,rarelymore
thanfour.Eachparticipantbecomesaseparatestudy.Inonetypeofsinglesubjectdesign,theinterventionisgivenandwithdrawnrepeatedlyforeachofthe
participants.Thesinglesubjectdesignisespeciallyappealingtothoseworkinginclinicalorappliedsettingsbecauseofthedifficultyofobtaininglargesamplesof
participants.Becauseoftheuniquenatureofsinglesubjectdesigns,wediscusstheminaseparatechapter.
ResearchApproacheswithAttributeIndependentVariables
Table5.1alsoshowsthattheassociationalandcomparativeapproachesaresimilarinseveralwaysforexample,theyareusedtostudyattributeindependent
variables,theinvestigatordoesnotuserandomassignment,andtheinvestigatordoesnothavecontrolovertheindependentvariables.Forthesereasons,associational
andcomparativeapproachesareoftenreferredtoasnonexperimentalapproaches.Becauseoftheirfocusonstudyingpatternsofindividualdifferencesinattributesof
participants,wehavegroupedthemtogetherunderageneralapproachwiththatname.Infact,mostsurveytyperesearchincludesbothcomparativeandassociational
researchquestions,soitiscommonforonestudytousebothapproaches.Neitherapproachprovidesgoodevidencethattheindependentvariableisthecauseof
differencesinthedependentvariable.
TheComparativeResearchApproach
Thecomparativeapproachdiffersfromtherandomizedexperimentalandquasiexperimentalapproachesbecausetheinvestigatorcannotrandomlyassignparticipants
togroupsandbecausethereisnotanactiveindependentvariable.Table5.1showsthat,likerandomizedexperimentsandquasiexperiments,comparativedesigns
usuallyhaveafewcategoriesoftheindependentvariableandmakecomparisonsbetweengroups.Studiesthatusethecomparativeapproachexaminethepresumed
effectofanattributeindependentvariable.Someauthorscallthisapproachcausalcomparative,butwethinkthattermisunwisebecausethisapproachisnotwell
suitedtoestablishingcauses.
AnexampleofthecomparativeapproachcomesfromastudybyMurphyandGliner(1988),whocomparedtwogroupsofchildrenonaseriesofmotorperformance
tests.Theinvestigatorswereattemptingtodeterminewhetherthe

Page69

differencesbetweenthetwogroupsresultedfromperceptualormotorprocessingproblems.Onegroupofchildrenwhohadmotorproblemswascomparedtoa
secondgroupofchildrenwhodidnothavemotorproblems.Noticethattheindependentvariableinthisstudywasanattributeindependentvariablewithtwolevels:
withmotorproblemsandwithoutmotorproblems.Thus,itisnotpossiblefortheinvestigatortorandomlyassignparticipantstogroups,ortoprovidetheindependent
variabletheindependentvariablewasnotactive.However,theindependentvariablehadonlyafewcategoriesandastatisticalcomparisonbetweenthetwogroups
wasdone.
NotethatcomparativestudiesdonotmeetPorter's(1997)criteriaforattributingcausality,becauseitisimpossibletocontrolperfectlyforothervariablesthatare
extraneoustothisstudy.Forexample,childrenwithmotorproblemsmightdifferfromthecontrolchildreninmanyways(e.g.,education,ethnicgroup,economic
status)inadditiontomotorproblems.Agoodcomparativestudywouldtrytocontrolforsomeofthesefactorsbymatchingorbyusinganothertechnique,butwe
couldneverbecertainthatthegroupswereequivalentinallrespects,aswecouldbeifrandomassignmenttogroupswerepossible.Thus,weshouldnotstateinour
conclusionsthatthisdisabilitycausedanydifferencesinmotorperformancethatwerefound.Wecansaythatthereweresignificantdifferences,iftheywereindicated
byourstatistics.
Youmightask,''Whydoacomparativestudyifwecannotmakeconclusionsaboutwhatcausedwhat?"Inparttheansweristhatifyouareinterestedinattribute
independentvariables,youhavenootherchoicethananindividualdifference(comparativeorassociational)approach.Attributes,ingeneral,cannotbegivenor
manipulatedinastudy.Someattributes,suchasselfconfidenceoranxiety,dovaryfromtimetotime,orsituationtosituation,sotheycouldbeactive,ormanipulated,
variables.However,inrecentyearsitisusuallyconsideredunethicaltodoso.Thus,withsomeexceptions,wemustusethecomparativeapproachifwewanttostudy
anattributeofparticipants.
AddingLevelstotheIndependentVariable
Intherandomizedexperimental,quasiexperimental,andcomparativeapproaches,itisoftendesirabletohavemorethantwogroups(i.e.,morethantwolevelsofthe
independentvariable).Anexamplefromthecomparativeapproachwouldbeastudythatcomparedthreegroupsfirst,third,andfifthgradestudentsonanaspect
ofcognitivedevelopment.Again,thisstudydoesnotmeettherequirementsforarandomizedexperimentalorquasiexperimentalstudy,becausetheindependent
variableisanattributeofthestudentsandwasnotmanipulated.Notethattheindependentvariable(gradeinschool)inthiscasehasthreeorderedlevels.2
Itisalsopossibletocomparearelativelylargenumberofgroupsorlevels(e.g.,5or10)ifonehasenoughparticipantssothatthegroupsizesareadequate(e.g.,20
ormoreineach).However,thisisatypical,andiftherearefive
2

Thistypeofstudyiscalledcrosssectional(versuslongitudinal)becausedifferentchildrenareassessedateachage.

Page70

ormoreorderedlevelsoftheindependentvariable,wewouldrecommendthatyouusetheassociationalapproachandstatisticsthatwediscusslaterinthischapter.
Therearetworeasonsforaddinglevelstoasingleindependentvariable:(a)tocontrolfor"notreatment"effectsinanexperimentalstudy,and(b)tohelpdeterminethe
completerelationshipbetweentheindependentanddependentvariables.
ControllingforNoTreatmentEffects
Theadditionofathird(ormore)leveltoasingleindependentvariableinexperimentalstudies,wheretwodifferenttreatmentsorinterventionsarecompared,may
improvetheresearchconsiderably.Forexample,astudybyJongbloed,Stacey,andBrighton(1989)comparedtwotypesoftreatmentinstrokerehabilitation.The
independentvariable,treatment,hadtwolevels,sensoriintegrativetreatmentandfunctionaltreatment,andthustwoexperimentalgroups.Althoughtheauthors
randomlyassignedparticipantstogroupstoproducegoodinternalvalidity,theresultswerehardtointerpretbecauseofthefailuretoincludeacontrolgroupthatdid
notreceivetreatment.Inotherwords,thestudycouldonlycomparewhichofthetreatmentsworkedbetter,buttherewasnowaytoevaluatewhethereithertreatment
wasbetterthannotreatmentatall.Hadtheauthorsaddedathirdlevelorgroup,whichdidnotreceiveanytreatment,thestudywouldhavebeenimproved.Similarly,
ifyoustartedwithanewtreatmentandacontrolornotreatmentgroup,youwouldobtainadditionalinformationifyouaddedagroupwithanalternativeortraditional
treatment.
DeterminingtheCompleteRelationship
Thesecondreasonforaddingaleveltoasingleindependentvariableistodeterminemorepreciselyarelationshipbetweentheindependentanddependentvariables.
Anexampleinvolvestherelationshipbetweentaskdifficultyandmasterymotivation(Redding,Morgan,&Harmon,1988).Masterymotivation,thedependent
measure,wasdefinedaspersistenceatthetask.Taskdifficulty,theindependentvariable,wasdeterminedwithanothergroupofchildren.Ifthestudyhadusedonly
veryeasyandverydifficulttasks,theinvestigatorswouldhavefoundnodifferenceinpersistence,asshowninthefollowinggraph:

Itwouldappearthattaskdifficultydidnotaffectpersistence,orthattherewasnorelationshipbetweentaskdifficultyandpersistence.Nowconsidertherelationship
whenanotherlevel(mediumdifficulty)isaddedtotheindependentvariable.Therelationshipisdescribedinthefollowinggraph:

Page71

NoticethataddingthethirdlevelindicatesthatthereisaninvertedUrelationshipbetweendifficultyandpersistence.Persistenceincreasesasdifficultyincreases,upto
apoint,butthenislowerasdifficultygetshigh.Hadthemediumdifficultyconditionnotbeenincluded,theactualrelationshipbetweentaskdifficultyandpersistence
wouldhavebeenmissed.
Althoughitisoftenconsidereddesirabletoaddathirdormoreleveltoasingleindependentvariable,itdoesnotchangethegeneraldesignclassification.Astudythat
hastwotreatmentsisasinglefactor(oneindependentvariable)designwithtwolevels.Ifweaddathird(ormore)level,itwouldstillbeasinglefactordesignthere
arejustmorelevelsorgroupstocompare.
SummaryoftheApproachesThatCompareGroups
Ineachofthethreepreviousapproaches(randomizedexperimental,quasiexperimental,andcomparative)anattemptwasmadetocomparetwoormorelevelsor
groupsoftheindependentvariableintermsoftheirscoresonthedependentvariable.Regardlessofwhethertheindependentvariablewasactiveorattribute,ithada
fewcategories,usuallylessthanfive.Inotherwords,thelevelsoftheindependentvariablefitintoafewnonoverlappingcategoriesorlevels.Forexample,inthe
DiPasqualeLehnerz(1994)experimentalstudy,theparticipantseitherreceivedahandsplintordidnotreceiveahandsplint(twolevels).Likewise,inthe
comparativestudybyMurphyandGliner(1988),theparticipantseitherhadamotorproblemordidnot.Aswehaveseen,studiesthatcomparegroupscanhave
morethantwocategories(e.g.,twotreatmentsandacontrol),andthecategoriescanbeordered(e.g.,high,medium,andlowonanattribute)ornot(e.g.,three
nominalcategorieslikeProtestants,Catholics,andJews).
TheAssociationalResearchApproach
Nowwewouldliketoconsideranapproachtoresearchwheretheindependentvariableisusuallycontinuousorhasseveralorderedcategories,typicallyfiveormore.
Supposethattheinvestigatorisinterestedintherelationshipbetweengiftednessandselfconceptinchildren.Assumethatthedependentvariableisselfconceptas
measuredbythePerceivedCompetenceScaleforChildren(Harter,1985).Theindependentvariableisgiftedness.Ifgiftednesshadbeendividedintohigh,average,
andlowgroups(afeworderedcategories),theresearchapproachwouldbethecomparativeapproachdiscussedpreviously.Ontheotherhand,inthetypical
associationalapproach,theindependentvariablegiftedness

Page72

iscontinuousorhasatleastfiveorderedlevels.Inotherwords,allparticipantswouldbeinasinglegroupmeasuredontwocontinuousvariables:giftednessandself
concept.Acorrelationcoefficientcouldbeperformedtodeterminethestrengthoftherelationshipbetweenthetwovariables(seechap.16).Nevertheless,evena
verystrongrelationshipbetweenthesevariablesdoesnotjustifytheconclusionthathighgiftednesscauseshighselfconcept.Someauthorslabelthisapproach
correlationalbecausethetypicalstatisticisacorrelationcoefficient.However,thatisnottheonlystatisticusedandwethinkitisbettertohaveamoregenerally
applicablelabel,namely,associational.
Wediscussthecomplex(morethanoneindependentvariable)associationalapproachandstatisticsusedwithitindetailinchapter19.Wewanttomentionherethat
multipleregressionisacommonassociationalstatisticthatisusedwhenthequestioniswhetheracombinationofseveralindependentvariablespredictsthedependent
variablesbetterthananyonepredictoralone.Forexample,multipleregressionisusedbyschoolsandcompaniestodeterminethebestcombinationofentranceor
applicationfactors,suchastestscores,grades,andrecommendations,topredictsuccessincollegeoronthejob.
Itisarbitrarywhetherastudyisconsideredtobecomparativeorassociational.Forexample,acontinuousvariablesuchasagecanalwaysbedividedintofewerlevels
suchasyoungandold.However,wemakethisdistinctionfortworeasons.First,wethinkitisusuallyunwisetodivideavariablewithmanyorderedlevelsintoafew
becauseinformationislost.Forexample,ifthecutpointfor"oldage"was65yearsold,personswhoare66and96yearsoldwouldbelumpedtogether,aswould
personswhoare21and64yearsold,andpersonswhoare64and65yearsoldwouldbeindifferentagegroups.Second,differenttypesofstatisticsareusually,but
notalways,usedwiththetwoapproaches.Wethinkthisdistinctionandthesimilaronemadeinthesectiononresearchquestionswillhelpyoudecideonan
appropriatestatistic,whichwehavefoundisoneofthehardestpartsoftheresearchprocess.
SummaryDiagramsfortheFirstFourApproaches
Figure5.2isaschematicdiagramillustratingthefourapproachesdiscussedsofar:randomizedexperimental,quasiexperimental,comparative,andassociational.
ThesediagramspresenttheinformationfromTable5.1inadifferentway.TheyalsoserveasapreviewofTable7.2,whichillustratesseveraldifferentspecific
experimentaldesigns.
TheDescriptiveResearchApproach
Thisapproachisdifferentfromtheotherfourinthatonlyonevariableisconsideredatatimesothatnocomparisonsorrelationshipsaremade.Table5.1showsthat
thislackofcomparisonsorrelationshipsiswhatdistinguishesthisapproachfromtheotherfour.Ofcourse,thedescriptiveapproachdoesnotmeetanyoftheother
criteria,suchasrandomassignmentofparticipantstogroups.

Page73

Fig.5.2
Schematicdiagramsoffourresearchapproaches.

Mostresearchstudiesincludesomedescriptivequestions(atleasttodescribethesample),butmostdonotstopthere.Infact,itisrarethesedaysforpublished
quantitativeresearchtobepurelydescriptivewealmostalwaysexamineseveralvariablesandtheirrelationships.However,politicalpollsandconsumersurveysare
sometimesonlyinterestedindescribinghowvotersasawholereacttoanissueorwhatproductsconsumersingeneralwillbuy.Ex

Page74

ploratorystudiesofanewtopicmayjustdescribewhatpeoplesayorfeelaboutthattopic.Furthermore,qualitativeorconstructivistresearchmaybeprimarily
descriptive.
Mostresearchbooksuseaconsiderablybroaderdefinitionfordescriptiveresearch.Forexample,someexperimentaldesignbooksusethetermdescriptiveresearch
toincludeallresearchthatisnotrandomizedexperimentalorquasiexperimental.Othersdonothaveacleardefinitionandusetheworddescriptivealmostasa
synonymforexploratoryorsometimescorrelationalresearch.Wethinkitisclearerandlessconfusingtorestrictthetermdescriptiveresearchtoquestionsandstudies
thatuseonlydescriptivestatistics,suchasaverages,percentages,histograms,andfrequencydistributions,thatarenottestedforstatisticalsignificancewithinferential
statistics.
CombinationsofResearchApproaches
Itisimportanttonotethatmoststudiesaremorecomplexthanimpliedbytheearlierexamples.Infact,almostallstudieshavemorethanonehypothesisorresearch
questionandmayusemorethanoneofthepreviouslydiscussedapproaches.Itiscommontofindastudywithoneactiveindependentvariable(e.g.,typeof
treatment)andoneormoreattributeindependentvariables(e.g.,gender).Thistypeofstudycombinestherandomizedexperimentalapproach(iftheparticipantswere
randomlyassignedtogroups)andthecomparativeapproach.Whensuchastudyconsidersbothindependentvariablestogether,itnolongerhasabasicorsingle
factordesign.Wediscussstudieswithtwo(ormore)factorsinlaterchapters.Mostsurveystudiesincludeboththeassociationalandcomparativeapproaches.Most
studiesalsohavesomedescriptivequestions,soitiscommonforpublishedstudiestousethreeapproachesorevenmore.
ResearchQuestionsorHypothesesandStatistics
ThreeTypesofBasicHypothesesorResearchQuestions
Inchapter4,weintroducedtheconceptofhypothesis,whichwasdefinedasapredictivestatementabouttherelationshipbetweenvariables.Researchquestionsare
similartohypothesesbutinquestionformat.Nowwewanttoexpandonthatgeneraldefinitionbysplittingresearchquestionsintothreetypes:basicdifference
questions,basicassociationalquestions,andbasicdescriptivequestions.Fordifferenceandassociationalquestions,basicmeansthatthereisoneindependent
andonedependentvariable.Fordescriptivequestions,basicmeansthatthereisonevariable.
Rememberthatbothdifferenceandassociationalquestionshaveasageneralpurposetheexplorationofrelationshipsbetweenvariables(seeFig.5.3).Thissimilarity
agreeswiththestatementbystatisticiansthatallparametricinferentialstatisticsarerelational,anditisconsistentwiththenotionthatthedistinctionbetweenthe
comparativeandassociationalapproachesissomewhat

Page75

Fig.5.3.
Schemaicdiagramshowinghowthegeneraltypeofstatisticandhypothesis
orquestionusedinastudycorrespondtothepurposesandtheapproach.

arbitrary.3However,webelievethatthedistinctioniseducationallyuseful.Notethatdifferenceandassociationalquestionsdifferinspecificpurposeandthekindsof
statisticstheyusetoanswerthequestion.
Forthedifferenttypeofquestions,wecomparegroupsorlevelsoftheindependentvariableintermsoftheirscoresonthedependentvariable.Thistypeofquestion
typicallyisusedwiththerandomizedexperimental,quasiexperimental,andcomparativeapproaches.Foranassociationalquestion,weassociateorrelatethe
independentanddependentvariables.Descriptivequestionsarenotansweredwithinferentialstatisticstheymerelydescribeorsummarizedata.
3

Weusethetermassociationalforthistypeofresearchquestion,approach,andstatisticratherthanthetermsrelationalorcorrelationaltodistinguishthesetermsfromthegeneral
purposeofdifferenceandassociationalquestionsorhypotheses.Also,wewantedtodistinguishbetweencorrelation,asaspecificstatisticaltechnique,andthebroadertypesof
approach,question,andgroupsofstatistics.

Page76

TheformatandanexampleofeachofthesetypesofquestionorhypothesesisgiveninTable5.2.Differencehypothesesareusedwhentheresearchapproachis
randomizedexperimental,quasiexperimental,orcomparative.Asyoumightguess,associationalhypothesesareusedwiththeassociationalapproach.Wethinkitis
advisabletousetheresearchquestionformatwhenonedoesnothaveacleardirectionalpredictionandforthedescriptiveapproach.Moredetailsandexamplesfor
statingresearchproblemsandquestionsareprovidedinAppendixC.
DifferenceVersusAssociationalInferentialStatistics
Wethinkitiseducationallyuseful,althoughnotcommonlyexplicitinstatisticsbooks,todivideinferentialstatisticsintotwotypescorrespondingtodifferenceand
associationalhypothesesorquestions.Differenceinferentialstatisticsareusually(butnotalways)usedfortherandomizedexperimental,quasiexperimental,and
comparativeapproaches,whichtestfordifferencesbetweengroups(e.g.,usinganalysisofvariance,alsocalledANOVA).Associationalinferentialstatisticstestfor
associationsorrelationshipsbetweenvariablesanduseatypeofcorrelationormultipleregressionanalysis.4Weusethiscontrastbetweendifferenceandassociational
inferentialstatisticsinchapter13todescribehowtoselectanappropriatestatistic.
SixTypesofResearchQuestions:
BasicandComplex
Rememberthatmanystudiesaremorecomplexthanimpliedbytheearlierexamples.Infact,moststudieshavemorethanonehypothesisorresearchquestionand
mayusemorethanoneoftheresearchapproaches.Itiscommontofindastudywithoneactiveindependentvariable(e.g.,treatment)andoneormoreattribute
independentvariables(e.g.,gender).Thistypeofstudycombinestherandomizedexperimentalapproach(iftheparticipantswererandomlyassignedtogroups)and
thecomparativeapproach,andhastwodifferencehypotheses.Weseeinchapter17thatthereareactuallythreehypotheses.Thissetofthreequestionscanbe
consideredacomplexdifferencequestionbecausethestudyhastwoindependentvariables.Theycouldbothbeactiveorbothattributeitwouldstillbeastudywitha
complexdifferencequestion.Likewise,complexassociationalquestionsareusedinstudieswithmorethanoneindependentvariableconsideredtogether.
Table5.3expandsouroverviewofresearchquestionstoincludebothbasicandcomplexquestionsofeachofthethreetypes:descriptive,difference,and
associational.Weusethetermsbasicandcomplexbecausethemorecommonnames,univariateandmultivariate,arenotusedconsistentlyintheliterature.
4

Wealsorealizethatallparametricinferentialstatisticsarerelational,sothisdichotomyofusingonetypeofdataanalysisproceduretotestfordifferences(whenthereareafew
levelsoftheindependentvariable)andanothertypeofdataanalysisproceduretotestforassociations(whenthereisacontinuousindependentvariable)issomewhatartificial.
Bothcontinuousandcategoricalindependentvariablescanbeusedinagenerallinearmodel(regression)approachtodataanalysis.However,thepracticalimplicationsarethat
mostresearchersadheretotheabovedichotomyindataanalysis.

Page77
TABLE5.2
ExamplesofThreeKindsofResearchQuestions/Hypotheses
1.Basic
Difference
(Group
Comparison)
Hypotheses
or
Questions

Useforrandomizedexperimental,quasiexperimental,andcomparativeapproaches

Forthistypeofhypothesis,thelevelsorcategoriesoftheindependentvariable(e.g.,
anxiety)areusedtosplittheparticipantsintogroups(e.g.,highandlow),whicharethen
comparedtoseeiftheydifferinrespecttotheaveragescoresonthedependentvariable
(e.g.,gradepointaverage[GPA]).

Anexample.Personswithlowandhighanxietywilldifferonaveragescores.Inother
words,theaverageGPAofthehighanxietypersonswillbesignificantlydifferentfromthe
averageGPAforlowanxietypersons.

2.Basic
Associational
(Relational)
Hypotheses
or
Questions

Usefortheassociationalapproach

Forthistypeofhypothesisorquestion,thescoresontheindependentvariable(e.g.,
anxiety)areassociatedwithorrelatedtothedependentvariable(e.g.,GPA).

Anexample.Therewillbeanegativeassociation(correlation)betweenanxietyscoresand
GPA.Inotherwords,thosepersonswhoarehighonanxietywilltendtohavelowGPAs,
thosewithlowanxietywilltendtohavehighGPAs,andthoseinthemiddleontheone
variablewilltendtobeinthemiddleontheothervariable.

3.Basic
Descriptive
Questions

Usefordescriptiveapproach
Forthistypeofquestion,scoresonasinglevariablearedescribedintermsoftheir
centraltendency,variability,orpercentagesineachcategory/level.
Anexample.TheaverageGPAwas2.73.

Thetablealsoincludesreferencestotablesinotherchaptersandexamplesofthetypesofstatisticsthatweincludeundereachofthesixtypesofquestions.Appendix
Cprovidesexamplesofresearchquestionsforeachofthesixtypes.
Notethatcomplexdescriptivestatistics(e.g.,acrosstabulationtable)couldbetestedforsignificancewithinferentialstatisticsiftheyweresotestedtheywouldno
longerbeconsidereddescriptive.Wethinkthatmostqualitativeorconstructivistresearchersaskcomplexdescriptivequestionsbecausetheyconsidermorethanone
variableorconceptatatime,buttheyseldomuseinferentialorhypothesistestingstatistics.Furthermore,complexdescriptivestatisticsareusedtocheckreliabilityand
toreducethenumberofvariables(e.g.,withfactoranalysis).

Page78
TABLE5.3
SummaryofTypesofResearchQuestions
TypeofResearchQuestion(NumberofVariables)

Statistics(Example)

1)Basicdescriptivequestions:onevariable.

Table9.2,chap.9(mean,standard
deviation,frequencydistribution)

2)Complexdescriptivequestions:twoormorevariables,butnouseof Chap.9,19,20(boxplots,cross
inferentialstatistics.
tabulationtables,factoranalysis,
measuresofreliability)
3)Basicorsinglefactordifferencequestions:oneindependentand
Table13.1,chap.14,15(ttest,one
onedependentvariable.Independentvariableusuallyhasafewlevels wayANOVA).
(orderedornot)
4)Complexormultifactordifferencequestion:threeormorevariables. Table13.3,chap.17,18factorial
Usuallytwoorafewindependentvariablesandoneormore
ANOVA)
dependentvariablesconsideredoneatatime.
5)Basicassociationalquestions:oneindependentvariableandone
dependentvariable.Usuallyatleastfiveorderedlevelsforboth
variables.Oftentheyarecontinuous.

Table13.2,chap.16(correlation
testedforsignificance)

6)Complexormultivariateassociationalquestions:twoormore
independentvariablesandonedependentvariable.Usuallyfiveor
moreorderedlevelsforallvariablesbutsomeorallcanbe
dichotomousvariables.

Table13.4,chap.19(multiple
regression)

Note.Manystudieshavemorethanonedependentvariable.Itiscommontotreateachoneseparately(i.e.,
todoseveralttests,ANOVAs,correlations,ormultipleregressions).However,complexstatistics(e.g.,
MANOVAandcanonicalcorrelation)areusedtotreatseveraldependentvariablestogetherinone
analysis.

Summary
Figure5.3andTable5.1providemostofthekeypointsmadeinthischapter.NotethatthetoprowofFig.5.3liststwogeneralpurposesofquantitativeresearch:
discoveryofrelationshipsanddescription.Rememberthatchapter4beganwithadefinitionofaresearchproblemasaquestionabouttherelationshipbetweentwoor
morevariables.Thisisthebroadsenseinwhichalltheapproaches,exceptthedescriptive,seektoestablishrelationshipsbetweenvariables.
Formorespecificpurposes,youcanseefromFig.5.3thattherandomizedexperimentalapproachistheonlyonewhosepurposeistodetermineoridentifycauses
however,quasiexperimentshelpusexaminepossiblecauses.Thecomparative,thequasiexperimental,andtherandomizedexperimentalapproachesenableusto
comparegroups.Thus,allthreeoftheapproachesontheleftsideofFig.5.3usedifferencehypotheses(asdiscussedearlier)andinferentialstatisticsthattestfor
differencesbetweengroups(e.g.,ttestsandanalysisofvariance).Notethatthereisnodistinctionbetweenthestatisticsusedinexperimentstodeterminecausesand
thoseusedincomparativestudiesthatonlytellusthatthereisadifferencebetweengroups.

Page79

NotealsoinFig.5.3thatthespecificpurposeoftheassociationalapproachincludesfindingassociations,relatingvariables,andalsomakingpredictionsfromthe
independentorpredictorvariablestoscoresonthedependentorcriterionvariable(s).Althoughsomewhatofanoversimplification,theassociationalapproachusesa
differenttypeofhypothesis(associational)anddifferentinferentialstatistics(correlationandmultipleregression)thandothecomparative,quasiexperimental,and
randomizedexperimentalapproaches.
Table5.4providessomeoftheinformationinTable5.1inadifferentway.Itbringstogetherthediscussionoforderedversusnominalvariablesfromchapter4andthe
discussionofthefiveapproachesinchapter5.
TABLE5.4
MostcommonTypesofVariablefortheIndependentandDependentVariableswithinEachoftheFive
ResearchApproaches
ResearchApproach

IndependentVariable

DependentVariable

Randomizedexperimental

Nominal

Manyordered

Quasiexperimental

orafew

levels

Comparative

orderedlevels

(approximately

Associational

Manyorderedlevels

continuous)

Descriptive

NA

StudyAids
Concepts
Basicorsinglefactorapproaches,designs,andquestions
Causeorinferringcausation
Complexdesignsandquestions
Researchhypothesis
Researchquestion
Variablecontrolledormanipulatedbytheexperimenter
Distinctions
Activeversusattributeindependentvariable
Differenceversusassociationalversusdescriptiveresearchquestionsandstatistics
Experimentalversusindividualdifferenceresearchapproaches
Pretestposttestversustimeseriesversussinglesubjectquasiexperimentaldesigns
Randomassignmentofparticipantstogroupsversusrandomselectionofparticipantstobeincludedinastudy
Randomizedexperimentalversusquasiexperimentalversuscomparativeversusassociationalversusdescriptiveapproachtoresearch
Relationshipsamongvariablesversusdescriptionofavariable

Page80

ApplicationProblems
1.Listedbelowaresomedifferencesbetweenthefiveapproachestoresearch.Matchthedescriptionthatbestfitsthetypeofapproach.Explain.
1.Experimental

A.Comparesgroups

2.Quasiexperimental

B.Asksquestionsthatdescribethedata

3.Comparative

C.Examinescausality

4.Associational

D.Associatesthemanylevelsofonevariablewiththemany
levelsofanother

5.Descriptive

E.Randomizedassignment,triestodeterminecausality

Choosewhichresearchapproachbestdescribesthefollowingscenarios.Describewhy.
2.Aresearcherwantstoknowifdrinkingcaffeinehelpsstudentsgetbettergradesonamathexam.Herandomlyassignsstudentstotwogroupsonethathegives
caffeinetodrinkandonehedoesnot.Hegiveseachparticipantamathematicsexamination.
3.Astudyisdonetoinvestigatetypeofclassroomseatsandtestperformance.ThesubjectsarefromtwoEnglishclassesatalocalhighschool.Oneclassisassigned
tomeetinaroomwithpillowsonthefloorforseats.Theotherclassistomeetinatraditionalclassroom.
4.Agradeschoolteacherisinterestedinwhethermoremalesorfemalesusetheirlefthandastheirdominanthand.Sheasksherclassof28studentstowritedown
whethertheyarerightorlefthanded.
5.Astudyisdonetoanalyzewhetherahighlevelofstress(measuredona0100scale)isrelatedtoahighlevelofloneliness(measuredona0100scale).
6.Youareinterestedincomparingtheeffectsoftwodifferenttypesoftherapy,musictherapyandoccupationaltherapy,onpainperceptioninpeoplewithchronic
arthritis.Youhavetwodifferentrehabilitationsettingsatyourdisposal.Describehowarandomizedexperimentaldesignwoulddifferfromaquasiexperimentaldesign.

Page81

Chapter6
InternalValidity
ResearchValidity
InferringCause
InternalValidity
EquivalenceofGroupsonParticipantCharacteristics
EquivalenceofGroupsinComparativeStudies
EquivalenceinAssociationalStudies
CookandCampbell'sThreatsRelatedtoParticipantCharacteristics
StatisticalRegression
ExperimentalMortality
Selection
InteractionsWithParticipantAssignment
ControlofExtraneousExperienceorEnvironmentVariables
CookandCampbell'sThreatsRelatedtoExperiencesorEnvironment
Maturation
HistoryorEnvironmentalEvents
RepeatedTesting
Instrumentation
Summary
StudyAids
Concepts
Distinctions
ApplicationProblems

Page82

ResearchValidity
Oneofthemainobjectivesofthisbookistohelpyoulearntoevaluatethemeritofaresearchstudy.Validityisthetermmostoftenusedtojudgethequalityormerit
ofaparticularstudy.Weusethetermresearchvaliditywhendiscussingthemeritofawholestudytodistinguishthetermfromvalidityofthemeasurementofa
variable.Itisnecessarytolearnaboutresearchvalidityfortworeasons.Thefirstisthatithelpsdevelopcriteriabywhichtodesignandjudgeone'sownresearch.
Second,andperhapsmoreimportant,itmakesyouabetterconsumerofresearch.Personsembarkingoncareersinappliedareasneedtobeabletojudgethemerit
ofresearch.Forexample,ifoneistopickandchooseamongalternativetreatmentorinterventionmodalities,oneneedstobeabletoevaluatetheresearchliterature.
FromtheworkofCookandCampbell(1979),wehavedividedresearchvalidityintofourcomponents:
1.measurementreliabilityandstatistics
2.internalvalidity
3.measurementvalidityandgeneralizabilityoftheconstructs,and
4.externalvalidity.
Ourpresentdiscussionfocusesoninternalvaliditybecauseofitsdirectrelationtothedesignofresearch.Thediscussionoftheotherthreecomponentsofresearch
validityarefoundinchapter23(measurementreliabilityandstatisticsmeasurementvalidityandgeneralizabilityoftheconstructs)andchapter10(externalvalidity).In
chapter24,wepresentasummaryoftheframeworkforanalyzingandevaluatingresearchthatisbasedonearlierchaptersandthesefourdimensionsofresearch
validity.
InferringCause
Amajorgoalofscientificresearchistobeabletoidentifyacausalrelationshipbetweentwovariables.Forthoseinapplieddisciplines,theneedtodemonstratethata
giveninterventionortreatmentcausesachangeinbehaviororperformanceisextremelyimportant.However,thereisconsiderabledisagreementamongscholarsasto
whatisnecessarytoprovethatacausalrelationshipexists.Thoseprofessingtheconstructivistparadigmdonotbelievethatacausalrelationshipcanbedetermined
(Lincoln&Guba,1985).However,mostscientistswouldatleastsubscribetosomeprobabilisticstatementaboutthecausalrelationshipbetweentwovariables.
Howiscauseinferred?Inchapter5weintroducedthreecriteriathatmustoccurtoinferacausalrelationship(Porter,1997).Werepeatthesethreecriteria:(a)the
independentvariablemustprecedethedependentvariableintime,(b)arelationshipmustbeestablishedbetweentheindependentvariableandthedependentvariable
(inthesocialsciencesthisrelationshipisusuallydeterminedstatistically),and(c)theremustbenoplausiblethird(extraneous)variablethatalsocouldaccountfora
relationbetweentheindependentanddependentvariables.

Page83

Fourofthefivespecificresearchapproaches(randomizedexperimental,quasiexperimental,comparative,andassociational)discussedinthelastchapterattemptin
differentwaystosatisfyPorter'sthreeprerequisites.Allfourcan,butdonotalways,meetthefirsttwocriteria.Therandomizedexperimentaland,toalesserextent,
thequasiexperimentalapproachesareusuallysuccessfulinmeetingPorter'sthirdcondition.Thecomparativeandassociationalapproachesarenotwellsuitedto
establishingcauses,butsomethingscanbedonetocontrolforextraneousvariables.Inthischapterandthenextweshowthatthedegreetowhichadesignmeets
Porter'sthreeconditionsforinferringcauseishighlyrelatedtoitsstrengthandinternalvalidity.
Althoughthecomparativeandassociationalapproachesarelimitedinwhatcanbesaidaboutcausation,theycanleadtostrongconclusionsaboutthedifferences
betweengroupsandaboutassociationsbetweenvariables.Furthermore,theyaretheonlyavailableapproaches,ifthefocusofyourresearchisonattribute
independentvariables.Thedescriptiveapproach,aswedefineit,doesnotattempttoidentifycausalrelationshipsor,infact,anyrelationships.Itfocusesondescribing
variables.
Now,wewilldiscussinternalvalidity,whichis,inpart,dependentonwhichoftheapproachesisused,andonthestrengthsandweaknessesofthedesignswithineach
ofthefourapproaches.Rememberthatconclusionsaboutcausescanbemadewithmorecertaintyfromrandomizedexperimentalthanfromthequasiexperimentalor
comparativeapproaches,eventhoughthesethreeapproachesusesimilargroupcomparisonordifferencestatisticsfordataanalysis.
InternalValidity
CookandCampbell(1979)definedinternalvalidityas''theapproximatevaliditywithwhichwecaninferthatarelationshipiscausal"(p.37).Internalvaliditydepends
onthestrengthorsoundnessofthedesignandinfluenceswhetheronecanconcludethattheindependentvariableorinterventioncausedthedependentvariableto
change.
CookandCampbell(1979)proposedalistof"threats"tointernalvalidity,severalofwhichareespeciallylikelytobepresentinthepoorandweakquasiexperimental
designsdescribedinchapter7.Randomizedexperimentaldesignshavefewerthreatstointernalvalidityand,thus,aremorelikelytoallowvalidconclusionsabout
causation.
WehavegroupedCookandCampbell'sthreatstointernalvalidityintotwomaintypes:equivalenceofgroupsonparticipantcharacteristics(e.g.,equivalenceof
theinterventionandcontrolgroups)andcontrolof(extraneous)experienceorenvironmentalvariables.Randomassignmentofparticipantstothegroups,whichis
acharacteristicofrandomizedexperimentsbutnotofquasiexperiments,isthebestwaytoassureequivalenceofthegroups.Controlofextraneousexperiencesand
theenvironmentdependsonthespecificstudy,butisgenerallybetterforrandomizedexperimentsandforstudiesdoneincontrolledenvironmentslikelaboratories.
Althoughinternalvalidityisoftendiscussedonlywithrespecttorandomizedandquasiexperiments,webelievethe

Page84

conceptalsoappliestothecomparativeandassociationalapproaches.Thus,ourdiscussionisintendedtoapplytoalltypesofresearch.
EquivalenceofGroupsonParticipantCharacteristics
CampbellandStanley(1966)describedanumberofspecificthreatstointernalvalidity,severalofwhich(e.g.,selection,statisticalregression,experimentalmortality,
andvariousinteractions)areparticipantfactorsthatcouldleadtoalackofequivalenceoftheparticipantsinthetwo(ormore)groupsandthusinfluencethe
dependentvariable.WehavefoundthelabelsofCampbellandStanley'sthreatsconfusingandtheircategoriesmorecomplexthannecessaryforabasicunderstanding
ofinternalvalidity.Anotherproblemwiththeemphasisonthreatstointernalvalidityisthatathreatoftenonlyinstructsyouaboutwhatmightresultifthegroupsarenot
equivalent.Inotherwords,theytellyouwhatiswrong.Theydonotgetatwhatwefeelisoneofthemostimportantpointsaboutinternalvalidity:howdowecorrect
theproblem?
Inresearchthatcomparesdifferencesamonggroups,akeyquestioniswhetherthegroupsthatarecomparedareequivalentinallrespectspriortotheintroduction
oftheindependentvariableorvariables.Inrandomizedexperimentalresearch,bydefinition,equivalenceisachievedthroughrandomassignmentofparticipantsto
groups,ifthereareatleast30ineachgroup.However,inquasiexperimental,comparative,orassociationalresearch,randomassignmentofparticipantstogroupshas
notorcannotbedone.Othermethods,suchasrandomassignmentoftreatmentstosimilarintactgroups,analysisofcovariance,matching,orcheckingforpretest
equalityofgroupsafterthefact,areattemptstomakethegroupsequaloratleasttoseehowdifferenttheyare.
Therandomassignmentoftreatmentstosimilarintactgroups,anaspectofastrongquasiexperimentaldesign,isoftenusefulforreducinginitialparticipantdifferences
whenrandomassignmentofparticipantstogroupscannottakeplace.Theeffectofthismethodisenhancedifthereisnoreasontobelievethatthegroupsareunequal,
aswouldbetrueifparticipantsassignedthemselvestothegroups,forexample,iftheychoseaparticularclassorteacherinaschool.ConsiderastudybyTuckman
(1992),whowasinterestedindetermininghowplanningchangedstudentmotivation.Studentmotivationwasdefinedastheamountofeffortputforthbycollege
studentsonavoluntary,courserelatedtask.AllofTuckman'sparticipantswereenrolledinoneoffoursectionsofacourseineducationalpsychology.Tuckman
randomlyassignedtwoofthesectionstoreceivetheintervention,whiletheothertwosectionsdidnotreceivetheintervention.However,thestudentswerenot
randomlyassignedtotheindividualsections.Iftherewerenoreasonstobelievethatthestudentsinthefourclassesweredifferentandiftheseturnedouttobe
significantdifferencesbetweentheinterventionandnoninterventionconditions,thenaconvincingargumentcouldbemadethatitwastheinterventionandnotthe
characteristicsoftheparticipantsthatmadethedifferenceinthestudy.Onewaytocheckifthefourclassesweredifferentbeforetheinterventionwouldbeto
comparerelevantpretestscores.

Page85

Acommonmethodofattemptingtoachieveparticipantorgroupequivalence,whenrandomassignmentofparticipantstoconditionscannotbeundertaken,isanalysis
ofcovariance(ANCOVA),whichisdiscussedinchapter18.Itisoftenusedinadesignthatcomparestwogroupsandtheparticipantsineachgroupreceiveapretest
andaposttest.Becauseparticipantsarenotrandomlyassignedtogroups,thedesigniscalledanonequivalentcontrolgroupdesign(chap.7).However,inquasi
experiments,ratherthanusethepretestasthecovariate(whichiscommonwhenparticipantshavebeenrandomlyassignedtogroups),itisbesttouseaseparate
measure,whichisrelatedtothedependentvariable,asthecovariate.Forexample,Griffin(1992)usedacademicaptitudeasthecovariateinastudythatmeasured
changeinconceptacquisition.Iverson,Iverson,andLukin(1993)usedgradepointaverageasacovariatetocontrolforselectionbiasacrossgenderinastudy
investigatinginstructionalstrategy.AproblemwithANCOVAisthatcertainstatisticalassumptionsmustbemet(seechapter18).
EquivalenceofGroupsinComparativeStudies
ANCOVAcanalsobeusedinstudieswithanattributeindependentvariable.Inthiscaseaseparatemeasure,suchasIQoreducation,isusedasthecovariateto
controlforgroupdifferencesinsuchimportantvariables.
Matchingofparticipantsoncharacteristicsotherthantheindependentvariableisanothermethodofapproachingparticipantorgroupequivalence.Thistechniqueis
especiallypopularinthecomparativeapproach,whereadiagnosticgroupiscomparedtoanormalgroup.Forexample,BeattyandGange(1977)compared26
personswithmultiplesclerosistoanequalnumberofpersonswithoutthediagnosistoassessmotorandintellectualfunctioning.Beforethestudy,theymatchedthe
participantsonage,gender,andeducation.Ifparticipantsarenotdifferentwiththeexceptionofthediagnosis,thentheauthorscouldconcludethatdifferences
betweenthetwogroupsmightbeattributedtothedisease.Oftenincomparativestudies,investigatorscheckafterthestudytoseehowwellmatchedthegroupswere
fordemographicmeasurescollectedduringthestudy.Ifthegroupsaresimilar,adegreeofinternalvalidityisshown.
Evenifoneormoreofthemethodslistedearlierisundertakentoachievegroupequivalenceinplaceofrandomassignmentofparticipantstogroups,actual
equivalencecanneverbeassumed.Thatiswhythespecificquasiexperimentaldesignsdescribedinchapter7arelabelednonequivalentgroupsdesigns.
Additionalproblemsinvolvedinmakinggroupsequivalentincludecharacteristicsthatseemequalinitially,butleadtodifferentdevelopmentaltrajectoriesovertime
and,thus,groupdifferencesthatarenotaresultoftheindependentvariable.Differencesbetweengroupsinmaturationrates,geneticpredisposition,orfactorsleading
toparticipantattritionareexamples.Regressiontothemeanofextremegroupsisanotherexampleofwheredifferencesincharacteristicsofparticipantsthatmaynot
beinitiallyobviousappearovertimebecausetheinitialscorewasnotthetruescore.Althoughthisproblemisunlikelytohappeniftherehasbeenrandomassignment
ofparticipantstogroups,theissuearisesofteninquasiexperimentaldesigns,wherechangeovertimeisone

Page86

oftheindependentvariables.TheseproblemsarediscussedingreaterdetailinthenextsectionaboutCookandCampbell's(1979)threatstointernalvalidity.
EquivalenceinAssociationalStudies
Iftheresearchapproachisassociational,thatis,thereisonlyonegroup.(Notethatourassociationalapproachhasoftenbeenreferredtoasthecorrelational
approach,butwepreferassociationalsothatthetermisnotconfusedwithacorrelationcoefficient).Intheassociationalcase,equivalenceofparticipantcharacteristics
comesdowntothequestionofwhetherthosewhoscorehighontheindependentvariableofinterestaresimilartothosewhoscorelowforotherattributesthatmay
becorrelatedwiththedependentvariable.Forexample,iftheindependentvariablewaseducationandthedependentvariablewasincome,weshouldbecautious
aboutinterpretingahighcorrelationasindicatingthatmoreeducationcausesahigherincome.Arethehighlyeducatedparticipantsequaltothepoorlyeducatedfor
otherpossiblecausalfactorssuchasIQ,parent'seducation,andfamilysocialstatus?Ifitislikelythatthehighscorersarenotequivalenttothelowscorersfor
variablessuchasage,gender,orethnicity,statisticallycontrollingforthevariablesonwhichthehighandlowparticipantsareunequalisonemethodofachievingsome
degreeofthisaspectofinternalvaliditywithintheassociationalresearchapproach.
CookandCampbell'sThreatsRelatedtoParticipantCharacteristics
AlthoughwehavefoundsomeofCookandCampbell's(1979)namesforthevariousthreatstobeconfusinganddifficulttoremember,weusethemherebecause
theyarecommonlyseenintheliteratureandresearchmethodsbooks.
StatisticalRegression
Sometimesthepurposeofastudyistobenefitaparticulargroupwhomaybeaboveorbelowtheaverage.AnexamplemaybechildreninaHeadStartprogram.
Whatoftenhappenswhenperformingthistypeofresearchisthatthescoresfromthisgroupontheposttestimproverelativetotheirscoresonthepretestregardlessof
theintervention.Becausethesescoreswerelowtostartwith,theymayhavemovedor"regressed"towardthemeanoraverageofallscores.Conceptually,whatis
happeningintheinternalvaliditythreatofregressiontothemeanisthatthereismeasurementerrorinthedependentvariable(reliabilityislessthanperfect).Because
thedependentvariable(pretest)isusedinthescreening,onlythosechildrenwhoscorelowonthepretestmaybeselectedtobeinthestudy.However,sincethereis
measurementerror,aproportionofstudentsselectedintothe"catchup"groupshouldnotactuallybeinthatgroup.Hence,whentestedasecondtime(posttest),their
"truescore"ismoreapttobereflected,whichwillnaturallyrepresentanincreasefromthepretest.Therefore,theinvestigatordoesnotknowiftheposttestscoreisa
reflectionoftheinterventionorofthestatisticalproblemofregressiontothemean.
ExperimentalMortality
Althoughthenamesoundsbad,thetypeofmortalityreferredtointhissituationinvolvesparticipantsleavingthestudy,calledattrition.Problemsarecreatedifthe
percentagelostislarge,orifthereisdif

Page87

ferentiallossbetweenoramonggroups,orboth.Foranyofthesemortalityproblems,theresultcouldleadtoabiasedposttestscoreiftherationaleforleavingthe
studyisrelatedtotheindependentvariableinthestudy.Thisisespeciallythecaseifeithertheinterventionorthecontrolconditionpromptsparticipantstodropout.
Forexample,iftheinterventionisfoundbyparticipantstobeonerousornoteffective,theymayquitthestudy.Likewise,ifparticipantsareinthecontrolconditionand
feelcheated,theymaywithdraw.
Selection
Wethinkthisthreatshouldbecalledparticipantassignmentbecausetheproblemarisesinhowparticipantsareassignedtoaparticulargroup,treatment,or
intervention,notinhowtheywereselectedfromthepopulation.Problemsarecreatedwhenparticipantsarenotrandomlyassignedtothetreatmentgroupsevenifa
pretestsuggeststhatthegroupsareequal.Althoughthegroupsshouldbecallednonequivalent,theextentofthisproblemdependsonhowtheparticipantsenteredinto
thegroupsandwhetherthatprocesswasbiased.
InteractionswithParticipantAssignment
Howparticipantswereassignedtogroupspresentsproblems.Inaddition,iftherearebiasesinassignment,theymayinteractdifferentiallybetweenoramongthe
groups,withotherfactorssuchasmaturation,environmentalevents,orinstrumentation.Theseassignmentinteractionsmakeitdifficulttoconcludethatposttest
differencesamonggroupsareduetotheeffectoftheintervention.
ControlofExtraneousExperienceorEnvironmentVariables
Wehavegroupedseveralotherthreatstointernalvalidityunderacategorythatdealswiththeeffectsofextraneous(variablesotherthantheindependentvariables)
experiencesorenvironmentalconditionsduringthestudy.Thus,wehavecalledthisinternalvaliditydimensioncontrolofextraneousexperienceandenvironment
variables.CookandCampbell(1979)addressedthisproblemwhendiscussingthreatstointernalvaliditythatrandomassignmentdoesnoteliminate(p.56).Manyof
thesethreatsoccurbecauseparticipantsgaininformationaboutthepurposeofthestudywhilethestudyistakingplace.Thefirstaspectofthiscategoryhastodowith
whetherextraneousvariablesoreventsaffectonegroupmorethantheother.Forexample,ifstudentslearnthattheyareinacontrolgroup,theymaygiveupandnot
tryashard,exaggeratingdifferencesbetweentheinterventionandcontrolgroups.Or,theoppositemayoccurandstudentsinthecontrolgroupovercompensate,
eliminatingdifferencesbetweenthetwogroups.Intheassociationalapproach,theissueiswhethertheexperiencesoftheparticipantswhoarehighontheindependent
variablearedifferentfromthosewhoarelowontheindependentvariable.
Asecondissueiswhethersomethingotherthantheindependentvariableisaffectingthedependentvariableforallgroups.Thisproblemis,ofcourse,morelikelyto
beseriouswhenthetreatmentcontinuesoveranextendedtimeperiodandinlongitudinalresearch.Historicaleventsandmaturationcouldaffectbothgroupsequally,
obscuringanygroupdifferences.Theresultscouldbemisinter

Page88

pretedif,forexample,oneconcludedthatsignificantpretesttoposttestgains(orlosses)intwointerventiongroupsresultedfromthetreatmentsratherthanfrom
uncontrolledcommon(e.g.,Hawthornetype)experienceormaturation.Thisisnotuncommoninstudiesthatinvestigateaclinicalintervention.Asaresultofthe
problemsofwithholdingtreatmenttopatients,oftentwodifferenttreatmentsarecomparedwithoutacontrolgroupthatdidnotreceiveanytreatment.Ifbothgroups
improve,butdonotdifferfromeachother,thenaconclusionthatbothtreatmentsworkedequallywellmaynotbecorrect.Instead,maturationoracommon
experiencecouldhaveaffectedbothgroupsequally.Ofcourse,inlongitudinalstudies,maturationisthevariableofinterest.Thisissueiscomplex.Inrandomized
experimentaldesignsinalaboratory,theseexperientialandenvironmentalvariablesareusuallywellcontrolled,butinquasiexperimentaldesigns,andespeciallyinthe
comparativeandassociationalapproaches,suchexperiencesmaybeinadequatelycontrolled.
Agoodstudyshouldhavemoderatetohighinternalvalidityonbothdimensionsofinternalvalidity(equivalenceofgroupsonparticipantcharacteristicsandcontrolof
extraneousexperienceorenvironmentvariables)or,ifnot,theauthorshouldbe,attheveryleast,cautiousnottosaythattheindependentvariablesinfluenced,
impacted,orcausedthedependentvariablestochange.Fortheevaluationofinternalvalidity,refertotheratingscalesinFig.6.1.Eachscaleshouldbeusedtorate
thestudyasawhole.
CookandCampbell'sThreatsRelatedtoExperiencesorEnvironment
Maturation
Theinternalvaliditythreatcalledmaturationtakesplacewhenparticipantsinthestudychangeasafunctionoftime,notnecessarilyfromphysicalmaturation.Time,in
thecaseofrandomizedexperimentalandquasiexperimentalresearch,isfromthepretestmeasurementperiodtotheposttestmeasurementperiod.Someofthese
changesarephysicalwhichcouldresultfromgrowingolder.Butotherchangesarelessobvious.Forexample,patientsmaygetbetterovertimewithoutanytreatment.
Thematurationthreatmakesitdifficulttodeterminewhetheritwastheinterventionorsomethingelsethatmadethedifferenceinthedependentvariable.
HistoryorEnvironmentalEvents
Thisthreattointernalvaliditytakesplacewhenanevent,whichmayberelatedtotheindependentvariablebutisnotofspecificinteresttotheinvestigator,occursat
sometimebetweenthepretestandtheposttest.Considerasituationinwhichyouareinterestedintheeffectofaparticulartypeofteachingmethod.Duringtheperiod
thatyourinterventionistakingplace,thestudentsinyourclassareexposedtoareadingfromanotherclassonthemeritsofyourteachingmethod.Theenvironmental
eventsthreatdoesnotallowforaconclusionthatwouldsaythatitwasyourteachingmethod,andonlyyourteachingmethodthatmadethedifferenceinthestudy.
RepeatedTesting
Therepeatedtestingthreattointernalvaliditymostoftenoccurswhentheinvestigatorusesapretestandaposttestinthestudy,andthetwoareidenticalorsimilar.
Theproblemresultsfromapossiblecarryover

Page89

Fig.6.1.
Evaluatingtheinternalvalidityofthefindingsofastudy.

fromthepretestthatmightalerttheparticipantsaboutthestudyandhowtheymightbehave.Or,ifthestudyinvolveslearning,thepretestmayhaveinformationthatis
includedintheposttest,whichwouldmakeitdifficulttoeliminatefromthetrueposttestscores.Severalstudieshavenotfoundrepeatedtestingtobeamajorproblem.
Instrumentation
Whenusingthesamepretestandposttestinaresearchdesign,itispossiblethatthescoringofthetestmaychange,especiallyiftheintervalbetweenthepretestand
theposttestisrelativelylong.Anexampleofthistypeofchangecouldbeacalibrationdriftinaninstrumentthatmeasuresreactiontime.Evenslightchangeswill
preventtheinvestigatorfromconcludingwhetherthechangeresultedfromtheinterventionortothechangeincalibration.CookandCampbellcalledthistypeofthreat
tointernalvalidityinstrumentation.Acommonprobleminvolvingtheinstrumentationthreatiswhenthepretestandposttestmeasurementtoolinvolvesraters.People
oftenchangetheircriteriaovertime.Evenworse,oneormoreoftheratersmay

Page90

leavethestudyandhavetobereplacedwithdifferentraters.Repeatedlyestablishinghighinterraterreliabilityisonemethodofcircumventingthisproblem.
Summary
Inthischapter,weintroducedtheimportanceofresearchvaliditytotheresearchprocess.Internalvalidity,oneofthefourcomponentsofresearchvalidity,was
definedanddividedintodimensions,equivalenceofgroupsonparticipantcharacteristics,andcontrolofextraneousorenvironmentalvariables.Threatstointernal
validityweredescribedundereachdimensionofinternalvalidity.Controllingthesethreatstointernalvaliditythroughspecificresearchdesignsisthetopicofournext
chapter.
StudyAids
Concepts
Controlofextraneousexperienceorenvironmentvariables
Equivalenceofgroupsonparticipantcharacteristics
Experimentalmortality
Historyorenvironmentalevents
Instrumentation
Interactionswithparticipantassignment
Internalvalidity
Maturation
Repeatedtesting
Researchvalidity
Statisticalregression
Selection
Threatstointernalvalidity
Distinctions
Equivalenceofgroupsonparticipantcharacteristicsversuscontrolofextraneousexperienceorenvironmentvariables
Internalvalidityversusexternalvalidity
Maturationversushistoryorenvironmentalevents
Repeatedtestingversusinstrumentation
Researchvalidityversusmeasurementvalidity
ApplicationProblems
1.Matcheachresearchexamplewiththemainthreattointernalvaliditythatitcontains.
a.maturation

Page91

b.historyenvironmentalevents
c.repeatedtesting
d.instrumentation
e.selectionassignment
f.experimentalmortality
g.statisticalregression
____Acontrolgrouptakesapretestaboutsocialstudiesknowledge.Someofthemareintriguedanddecidetoreaduponthetopicbeforetheposttest.
____Theresearchassistantsbecomeboredanddon'tdotheirobservationsascarefullyneartheendofthestudy.
____Anexperimentisconductedtoassessanewhistoryteachingmethod.Schooldistrictsthatvolunteerserveastheexperimentalgroupandthosethatdon't
volunteerserveasthecontrolgroup.
____Aresearcherisinterestedinthelongtermeffectsofanelectiononthepoliticalattitudesofvoters.Priortotheelection,theviewsof100votersare
assessed.Afterwardstheresearcherisabletoreassesstheattitudesof74voters.
____Thisisdefinedasanyeventotherthanthemanipulationoftheindependentvariablethatoccursbetweenthepretestandtheposttest.
____Atthestartoftheschoolyear,themathachievementofagroupofchildrenisassessed.Theyarethenallexposedtoanewmathprogram,andre
examinedattheendoftheschoolyear.
____Apsychiatristselectsagroupofpatientswiththemostextremesymptoms.After1monthoftherapy,thepatientshavemarkedlyimproved.
Forthefollowingfourstudies:
a.Whatresearchapproachwasused?
b.StatetheresearchquestionintheformatillustratedinTable5.2.
c.Whatgeneraltypeofstatisticwouldprobablybeused?
d.Evaluatetheequivalenceofthegroupsonparticipantcharacteristicsandevaluatethecontrolofextraneousexperiences.
2.Researcherswereinterestedineffectsofdifferenttypesoftelevisionprogrammingontheaggressivebehaviorofpreschoolagedchildren.Childrenfromapre
schoolwererandomlyassignedtospend30minutesviewingoneoftwodifferenttypesoftelevisionprogramming.Onegroupwatchedviolentanimatedcartoonssuch
asPowerRangers,andinanadjacentroom,theothergroupwatchedprogrammingthatmodeledprosocialbehaviorsuchasBarney.Duringthehourafterthe
viewing,aggressiveactsinitiatedbyindividualchildrenfrombothviewgroupswerecountedbyobservers.Theycomparedthetwogroupsonnumberofaggressive
behaviors.
3.Inthisstudy,theresearcherswereinterestedincomparingthewayinwhichthreetypesofreinforcementaffectedtheconditioningofchildrentousetheword"they"
whenmakingupsentences.Subjectswerebroughttothelabandthenrandomlyassignedtothreegroups:(a)Childreninthematerialreinforcementconditionreceived
anM&Mcandyimmediatelyafterusingtheword"they"atthebeginningofasentence(b)Childrenassignedtothe

Page92

praiseconditionwerereinforcedbytheexperimentersaying"good"and(c)Childreninthesymbolicreinforcementconditionweresimplygivenaplusmark.
4.Aprofessorwantstoknowwhetherstudentanxiety(onananxietyinventorywithscoresfrom110)influencestestperformancescoresonthemidtermexam.
5.Theorganizersofarequiredweeklonggraduatecoursewereinterestedinwhichoftwoteachingtrainingapproacheswasmosteffective.Twosectionsofthe
course(ontraumaassessmentandintervention)weretaught.Oneteacherutilizedatraditionalstructureddidacticapproach.Theotherteacherusednewapproach
curriculum,utilizingahighproportionofexperientialcomponents.Onesectionmetintheafternoonandtheothersectioninthemorning.Studentscouldsignupfor
eithersession.However,theinstructorswereunknowntothestudentsandthestudentshadnopriorawarenessofthedifferingapproachescurriculums.Thesections
wereofequalsizeandthestudentsweredemographicallysimilar.Studentswerepretestedtoassesstheirpriorknowledge,andattheendofthecoursestudents
weretestedonthecontentofthecourse.

Page93

Chapter7
SpecificResearchDesignsforRandomizedExperimentalandQuasiExperimentalApproaches
ResearchDesign
DesignTerminology
PoorQuasiExperimentalDesigns
OneGroupPosttestOnlyDesign
OneGroupPretestPosttestDesign
PosttestOnlyDesignWithNonequivalentGroups
BetterQuasiExperimentalDesigns
PretestPosttestNonequivalentComparisonGroupDesigns
WeakQuasiExperimentalDesigns
ModerateStrengthQuasiExperimentalDesigns
StrongQuasiExperimentalDesigns
TimeSeriesDesigns
SingleGroupTimeSeriesDesigns
MultipleGroupTimeSeriesDesigns
Conclusions
RandomizedExperimentalDesigns
PosttestOnlyControl(orComparison)GroupDesign
PretestPosttestControlGroupDesign
SolomonFourGroupDesign
RandomizedExperimentalDesignWithMatching
Summary
StudyAids

Page94

Concepts
Distinctions
ApplicationProblems
ResearchDesign
Inchapter4weintroducedtwodifferenttypesofindependentvariables,activeandattribute.Wealsodescribedhowanindependentvariablehasdifferentvalues,
whichwecalledlevels.Inaddition,wedescribedthedependentvariableastheoutcomemeasureorcriterionofthestudy.Howparticipantsbecomeassignedtothe
levelsoftheindependentvariable,inpart,determinesthetypeofquantitativeresearchapproach,whichwasthetopicofchapter5.Inchapter6weintroducedthe
conceptofinternalvalidity,andhowitshouldbeusedtojudgethemeritsofastudy.Itisworthgoingbacktochapter5andreviewingFig.5.1toexaminethe
relationshipbetweentypeofindependentvariableandtypeofquantitativeresearchapproach.Therandomizedexperimentalandquasiexperimentalapproacheshave
anactiveindependentvariable,whereasthecomparativeandassociationalapproacheshaveanattributeindependentvariable.
Inthischapterweintroducetheconceptofresearchdesign.Wediscussspecificresearchdesignsthatfitintotherandomizedexperimentalandquasiexperimental
researchapproaches.Aspecificresearchdesignhelpsusvisualizetheindependentvariablesofthestudy,thelevelswithintheseindependentvariables,andwhen
measurement(dependentvariable)willtakeplace.Whentheoperationsfromthesespecificresearchdesignsareexamined,theyhelptheresearcherdeterminethe
internalvalidityofthestudy.Weintroducespecificresearchdesignshere,sothatwecanexaminethesedesignswithrespecttointernalvalidity,andpointoutsomeof
thethreatstointernalvaliditythatwediscussedinchapter6.

Page95

Bothrandomizedexperimentalandquasiexperimentalapproacheshaveanactiveindependentvariable,withatleastonelevelbeingsometypeofinterventionor
manipulationgiventoparticipantsintheinterventiongroup.Usuallythereisalsoacomparisonorcontrolcondition(ortreatment),whichisgivenasanotherlevelofthe
independentvariable.Therecanbemorethantwolevelsorgroups.Unfortunately,insomepoorquasiexperimentaldesigns,thereisonlyonelevel,sono
comparisonscanbemade.Beforediscussingspecificdesigns,wewanttointroducesometerminologytohelpconceptualizeeachdesign.
DesignTerminology
R=randomassignmenttothegroup
NR=nonrandomassignmenttothegroup
O=observationofthedependentvariable
X=intervention(oneleveloftheindependentvariable)
~X=nointervention(ortheusualintervention)
E:=experimentalorinterventiongroup1
C:=controlorcomparisongroup2
M=matching
PoorQuasiExperimentalDesign
OneGroupPosttestOnlyDesign
Anexampleofthisdesign,oftenreferredtoastheoneshotdesign,wouldbeanevaluationofanewcurriculuminaschoolsystem.Theinvestigatorintroducesthe
curriculum(X),andthendecidesthatitmightbeusefultodetermineifitisworking.Attheendofthesemester,theinvestigatorusessomeformofmeasurement(O)to
determinethestudents'responsetothenewcurriculum.Thedesignisshownasfollows:
NRE:XO
Thisdiagramandthosethatfollowindicateatimesequence.First,thereisnonrandomassignmentofparticipantstotheinterventiongroup,thenthetreatment,and,
finally,aposttesttakesplace.Theproblemwiththedesignisitdoesnotsatisfyeventheminimumconditionforaresearchproblem,whichisinvestigationofa
relationshiporcomparison.Notethattheinterventionisnotavariablebecausethereisonlyonelevel.Doestheonegroupposttestonlydesignhaveanyvalue?If
nothingelse,itprovidespilotdata(acommontermtoindicateexploratorydata)forafuturestudy.Theinvestigatorcouldcomparethe
1

Tosimplifytheexamplesinthischapter,wehavemainlydescribeddesignswithonlyoneinterventiongroupandonecontrolgroup.However,itiscommontohavemorethan
twogroups.Wehavediscussedsomesuchexamplesandthereasonsfortheminchapter5.
2

Inquasiexperiments,itisbettertousethetermcomparisongroupratherthancontrolgroupbecause,especiallywithpoorandweakquasiexperiments,thereislittlethatis
''controlled."Forsimilarreasons,quasiexperimentsarelabelednonequivalentgroupdesigns.

Page96

resultstodatafromanearliergrouporfromthesamegroupatanearliertime.However,ifthisweredone,thedesignwouldnolongerbeaonegroupposttestonly
design.
OneGroupPretestPosttestDesign
TheonegroupPretestPosttestdesigncanbeshownasfollows:
NR

E:

O1

O2

Pretest

Intervention

Posttest

Theoperationsforthisdesignarethatanobservationintheformofapretestisgivenfirst,thentheinterventionisgiven,andfinally,asecondobservationintheformof
aposttestisrecorded.AnexampleofthistypeofdesignisdemonstratedinastudybyMoisan(1990).Moisanwasinterestedintheeffectsofanadaptiveski
programontheselfperceptionsandperformanceofchildrenwithphysicaldisabilities.Unfortunately,earlyinthestudy,thechildrenfromthecontrolgroupwithdrew,
leavingherwithonlytheinterventiongrouprelativelyintact(thishighdropoutratemayhaveoccurredbecausetheinterventiongroupgottoparticipateinaski
program,whereasthecontrolgroupdidnotreceivethisinterventionoranythingcomparable).Therefore,theonlycomparisonleftforMoisanwasfromthepretestto
theposttest.
Theproblemwiththeonegrouppretestposttestdesignisthatthecomparisonisnotwithasecondgroup(acontrolgroup).Instead,thecomparisonintheonegroup
pretestposttestdesignisbetweenthepretestandtheposttestwithinthesamegroup.Becausethereisnocomparisongroup,thedesignissusceptibletomostofthe
threatstointernalvalidity.
Environmentaleventsareapossiblethreattointernalvalidityinthisdesign,becausethelackofacontrolgrouppreventstheinvestigatorfromknowing,forexample,
whetherotheractivitiesinschoolatthesametimeastheinterventionmightbeproducingthefacilitation.Maturationisapossiblethreattointernalvaliditybecausethe
studentsaregettingolderandmaybebettercoordinatedandstrongeratthesametimeastheintervention.Carryovereffectsareapossibleprobleminthisdesign
becausetakingthepretestcouldinfluencetheposttest.
IntheMoisan(1990)study,equivalenceofthegroupswasdefinitelyaproblembecausestudentsvolunteered(wereselfassignedratherthanrandomlyassigned)to
beintheinterventiongroup.Eveniftheinterventiongrouphadinitiallybeensimilartothecontrolgroup,thehighdropoutrate(calledexperimentalmortalitybyCook
&Campbell,1979)wouldlikelyhavemeantthatthegroupswhotooktheposttesthaddifferentcharacteristics.
Ifonecannotfindanacceptablecomparisongroupfortheonegrouppretestposttestdesign,increasingthenumberofdependentvariablesonecanaddcontrol
towardinterpretingtheresults.Forexample,amongthemeasuresthatMoisan(1990)usedinherstudywastheSelfPerceptionProfileforChildren(Harter,1985).
Thismeasurementtoolhasseveraldomains.Someofthesedomainsarescholasticcompetence,socialacceptance,athleticcompetence,and

Page97

behavioralconduct.Notallofthesemeasureswouldbeexpectedtoimproveasaresultoftheintervention.Therefore,Moisancouldhavepredictedwhichmeasures
shouldchangefromtheintervention.IfonlythosemeasureschangedthatMoisanpredictedwouldchange,thenmoreconfidencecouldbeplacedintheinterventionas
beingresponsibleforchangesinthosemeasures.CookandCampbell(1979)refertothismodificationasthenonequivalentdependentvariablesdesign.
PosttestOnlyDesignwithNonequivalentGroups
Thisdesigncanbeshownasfollows:
NR

E:

NR

C:

~X

Thereisnorandomassignmenttogroupsandnopretest,soitisimpossibletodeterminetheequalityofthegroupspriortothetreatment.Hence,thedesignisweak.
Cosby(1989)providesanexampleofthistypeofdesign.Aninvestigatorisinterestedinarelaxationtrainingprogramtoreducecigarettesmoking.Theparticipantsin
theinterventiongrouparegivenarelaxationtrainingprogram.Theparticipantsinthecomparisongroupdonotreceiverelaxationtraining.Attheendofthestudy,both
groupsaremeasuredonasmokingfrequencymeasure.AsCosby(1989)pointsout:
Itislikelyinthiscasethatparticipantsinthefirstgroupchosetoparticipateintheprogram,andthesubjectsinthesecondgrouparesimplysmokerswhodidnotsignupforthe
training.Theproblemofselectiondifferencesarisesbecausesmokerswhochoosetoparticipatemaybedifferentfromthosewhodonot.(pp.7071)

Thus,thisdesignislikelytobeweakonequivalenceofthegroupsonparticipantcharacteristics.
BetterQuasiExperimentalDesigns
PretestPosttestNonequivalentComparisonGroupDesigns
Thepretestposttestdesignswithinthequasiexperimentalapproachareusuallyreferredtoasnonequivalentcomparisongroupdesigns.Thedesignappearsas
follows:
NR

E:

O1

O2

NR

C:

O1

~X

O2

Noticethatthereisnorandomassignmentoftheparticipantstothetwo(ormore)groupsinthisdesign.Thesequentialoperationsofthenonequivalentcomparison
groupdesignareasfollows.First,measurementsaretakenontwo

Page98

differentgroupspriortoanintervention.Then,onegroupreceivestheintervention,andtheothergroupdoesnotreceivetheintervention.Attheendoftheintervention
period,bothgroupsaremeasuredagaintodetermineiftherearedifferencesbetweenthetwogroups.Thedesignisconsideredtobenonequivalentbecauseevenif
thetwogroupshavethesamemeanscoreafterthepretest,theremaybecharacteristicsthathavenotbeenmeasuredthatmayinteractwiththetreatmenttocause
differencesbetweenthetwogroupsthatarenotcausedstrictlybytheintervention.Wehaveclassifiedthenonequivalentcomparisongroupdesignintothreedifferent
strengthsofquasiexperimentaldesignthatlookalikewhendiagramed,butvaryinhowparticipantsareassignedtogroupsorconditionsandhowmuchcontrolthe
investigatorhasovertheindependentvariable.
WeakQuasiExperimentalDesigns
Thisdesignoccurswhentryingtoevaluatesituationswhereattendanceisvoluntary.Ithassomeoftheproblemsmentionedearlier.Theresearcherdoesnotrandomly
assignparticipantstogroups.Infact,participantsassignthemselvestothegroups.Asecondproblemisthattheresearcherdoesnothavecontroloverthe
independentvariablebecausetheparticipantspresumablychooseaparticulargrouptoreceiveaparticularinterventionortreatment.Adesignofthistypeisespecially
commonwhentryingtoevaluateeducationalortherapeuticworkshops.Peoplewhowanttoattendtheworkshopvolunteertobeinthatconditionorgroup.A
comparisongroupiscomposedofpeoplewhodonotwishtobeintheworkshop,orasampleofpeoplewhomaynothavecaredonewayoranother.Regardless,
sinceatleastonegrouphasvolunteered,theresearchercannotrandomlyassignonegrouptoonecondition,andtheothergrouptotheothercondition.Therefore,any
eventualdifferencebetweenthegroupthatreceivedtheinterventionandthegroupthatdidnotreceivetheinterventionmustbetemperedbythispotentialbias.
ModerateStrengthQuasiExperimentalDesigns
Thisdesignalsoinvolveslesscontrolbytheinvestigatorovertheindependentvariable,and,asinallquasiexperiments,theparticipantsarenotrandomlyassignedto
groups.Themoderatestrengthquasiexperimentaldesignfitsbetweentheweakquasiexperimentaldesignandthestrongquasiexperimentaldesignfromequivalence
ofgroupspriortotheintervention.Inthisdesign,participantsdonotselectthemselvesintointactgroups,butareinthesegroupsasaresultofotherfactors,whichare
notrelatedtotheintervention.Examplesofthesefactorsincludestudentsschedulingclassesaroundavailability,orpeoplechoosinghospitalsforconvenience.The
criticaldifferencebetweenthisversionofthedesignandstrongquasiexperimentsisthattheinvestigatorisnotabletorandomlyassignthetreatmenttocertain
groups.Instead,theinvestigatortakesadvantageofasituationwhereitisknownthatonesetting(e.g.,schoolorhospital)willreceivetheinterventionandanother
settingwillnotreceivetheintervention.Anexampledemonstratesthisdesign.
GilfoyleandGliner(1985)wereinterestedinattitudechangestowardhandicappedchildrenbynonhandicappedchildren.Aneducationalpuppetshow(AlloftheKids
ontheBlock)wasusedastheactiveindependentvariablein

Page99

thisstudy.Threegroups(schools)wereused.Twointerventiongroupsviewedthepuppetshowbuttheothergroup(control)didnot.Fortheinterventiongroups,a
pretestwasgivenpriortotheintervention,andaposttestwasgivenfollowingtheintervention.Thecomparisongroupalsoreceivedapretestandposttestduringthe
sametimeintervalsastheinterventiongroups.Thedependentvariablefromthepretestandposttestwasascaleusedtomeasureattitudesofpersonalfeelingsor
perceptionsofdisability.
DidtheGilfoyleandGliner(1985)studymeetthecriteriaforarandomizedexperiment?Therewasanactiveindependentvariable(thepuppetshow),butdidthe
investigatorrandomlyassignparticipantstogroups?No,thegroups(schools)wereintactpriortotheinvestigation.Didtheinvestigatorhavecontroloverthe
independentvariable?No,theinvestigatorcouldnotrandomlyassignthetreatmenttotwogroupsandnotreatmenttoathirdgroup.Instead,thegroupswereselected
becausetheresearcherknewthattwoschoolsweregoingtoviewthepuppetshowandanotherschoolwasnotgoingtoviewthepuppetshow.Therefore,thisdesign
isnotasstrongasthenextdesignbecausetheinvestigatorcouldnotrandomlyassigntheintervention.Therelativestrengthofthisdesignrestsontwoquestions.First,
whydidtwoschoolsgettheinterventionandtheotherschoolnotgettheintervention?Second,arestudentsintheschoolsthatreceivedtheinterventiondifferentfrom
studentsintheschoolthatdidnotreceivetheintervention?Ifthereisnoreasontosuspectbiasrelativetothedependentvariableforeitherquestion,thenthedesignis
almostasstrongasthetypediscussednext.Ontheotherhand,ifthereisareasontobelievethatthereisbiasinthegroupsthatreceivedthetreatmentanddifferences
betweenthestudentsintheinterventionandcontrolschools,suchasfamiliaritywithhandicappedchildrenorethniccomposition,thenthedesignisconsiderably
weaker.
StrongQuasiExperimentalDesigns
Thisdesignmightbeusedinanenvironmentsuchasanelementaryschoolthathastwofifthgradeclassrooms,andwithanactiveindependentvariablesuchasanew
curriculum.Inthisenvironment,thechildrenalreadyhavebeenassignedtooneofthetwoclassroomsorgroupsforaparticularsemesterthustheinvestigatorcannot
randomlyassignparticipantstogroups.However,inthestrongquasiexperimentalapproach,theinvestigatorhascontrolovertheindependentvariablebyrandomly
assigningthenewcurriculum(treatment)conditiontoone(intact)classroomandtheoldcurriculumconditiontotheother.Thestrengthofthisquasiexperimental
designisthatitissimilartoarandomexperimentaldesignexceptthatparticipantshavenotbeenrandomlyassignedtogroupsorconditions.Insomeintactsituations,
suchasclassroomswithinaschool,theassignmentofstudentstodifferentclassroomsmaybealmostrandom(i.e.,therewasnointentionalbiasintroducedinthe
assignment)inthosecases,thestrongquasiexperimentaldesignisalmostequivalenttoarandomizedexperimentaldesignoninternalvalidity.
Inchapter6,wementionedastudybyTuckman(1992).Werecallthestudyasanexampleofthestrongquasiexperimentaldesign.Tuckmanwasinterestedin
determininghowplanningchangedstudentmotivation.Studentmotivation(thedependentvariable)wasdefinedastheamountofeffortputforthbycollegestudentson
avoluntary,courserelatedtask.Forourpurposes,theim

Page100

portantpointwastheassignmentofparticipantstogroups.Tuckmanhadfourdifferentgroups.AllofTuckman'sparticipantswereenrolledinoneoffoursectionsofa
courseineducationalpsychology.Tuckmanrandomlyassignedoneleveloftheindependentvariable,"usingtheplanningform,"totwosections.Theothertwo
sectionsdidnotusetheplanningform.IstheTuckman(1991)studyarandomizedexperiment?
Therewasanactiveindependentvariable.Tuckmanhadcontrolovertheindependentvariableherandomlyassignedtheplanningcondition(treatment)totwogroups
andtheothertwogroupsdidnotgettheplanningcondition.However,Tuckmanwasnotabletorandomlyassignparticipantstogroups,sothiswasnotarandomized
experiment.Notsatisfyingthisconditionmayormaynotbeamajorproblem,dependingonhowstudentsassignedthemselvesintoeachclasssection.Iftheirchoices
weresimilartochance,thentheTuckmanstudycouldbeconsideredalmostasstrongasarandomizedexperiment.Ontheotherhand,iftherewassomesystematic
reasonforstudentschoosingonesectionoveranothersection(suchaspreferenceforteacherortimeofday),thentherewasabiasinthemethodology,andall
conclusionsmusttakethisbiasintoconsideration.
Table7.1summarizesthetwoissuesthatdeterminethestrengthofapretestposttestquasiexperimentaldesign:controlovertheindependentvariable(indicatedby
randomassignmentoftreatmentstointactgroups)andequivalenceofparticipantcharacteristics.Rememberthatthereisnorandomassignmenttothegroupsinany
quasiexperimentaldesign,sothegroupsarenevertotallyequivalent.
TimeSeriesDesigns
Asecondgeneralcategoryofquasiexperimentaldesignsiscalledtimeseriesdesigns.Similartothequasiexperimentaldesignsmentionedearlier,thereisnorandom
assignmentofparticipantstogroups.Thetwomostcommontypesoftimeseriesdesignsaresinglegrouptimeseriesdesignsandmultiplegrouptimeseries
designs(seeCook&Campbell,1979Huck,Cormier,&Bounds,1974).Also,subcategorieswithineachtypeoftimeseriesdesignaretemporaryversus
continuoustreatment.
TABLE7.1
IssuesThatDeterminetheStrengthofQuasiexperimentalDesigns

StrengthofDesign

RandomAssignmentofTreatments
toIntactGroups

ParticipantCharacteristics
LikelytobeSimilar

Strong

Yes

Yes,assumingnobiasinhow
participantswereassignedto
groups

Moderate

No

Maybe

Weak

No

No,becauseparticipantsself
assignedtogroups

Page101

SingleGroupTimeSeriesDesigns
Thelogicbehindthesedesigns,oranytimeseriesdesign,involvesconvincingothersthatabaseline(i.e.,severalpretests)isstablepriortoaninterventionsothatone
canconcludethatthechangeinthedependentvariableresultsfromtheinterventionandnototherenvironmentalevents.Forexample,considertheonegrouppretest
posttestdesignthatwediscussedundertheheadingofpoorquasiexperimentaldesigns.
Theonegrouppretestposttestdesigncanbeviewedasfollows:
NR

E:

O1

O2

Pretest

Intervention

Posttest

Theproblemwiththisdesignisthatifthereisachangefromthepretesttotheposttestscore,itisnotknownwhetherthechangeresultedfromtheinterventionorfrom
anothereventthatcouldbehappeningatthesametimethathadnotbeencontrolled.Now,supposeweaddanearlierobservationperiod(pretest1,taken6months
priortothestudy)tothisdesignasfollows:
NR

E:

O1

O2

O3

Pretest1

Pretest2

Intervention

Posttest

SupposealsothattherewasnochangeobservedbetweenPretest1andPretest2priortotheintervention.Butaftertheintervention,achangewasobservedinthe
posttest.Thisdesignwouldbemoreconvincingifevenmoreobservationstookplacepriortotheintroductionoftheindependentvariable,andstillnochangehad
occurred.Itiscommonintimeseriesdesignstohavemultiplemeasuresbeforeandaftertheintervention,buttheremustbemultiplepreteststoestablishabaseline.
Thesinglegrouptimeseriesdesignwithtemporarytreatmentisasfollows:
NR

E:

O1O2O3O4XO5O6O7O8

Anexampleofthissinglegrouptimeseriesdesigncouldinvolveacompanythatwasinterestedintheeffectsofaworkshoponbeingateamplayer.Observations
wouldtakeplacepriortotheworkshoponsomerelevantmeasure,suchascooperativeinteractions.Theworkshopisgivenafterfourbaselinemeasureson
cooperativeinteractions.Theworkshopisatemporaryintervention,andobservationsarerecordedimmediatelyaftertheinterventionandthreelatertimes.Onewould
expectthatiftheworkshopwassuccessful,therewouldbeanimmediateincreaseaftertheinterventionrelativetotheprecedingbaselineperiods,andtheeffectsmight
ormightnotbelonglasting.
Thesinglegrouptimeseriesdesignwithcontinuoustreatmentisavariantofthedesignwithtemporarytreatment.Thisdesignisasfollows:
NR

E:

O1O2O3O4~XO5XO6XO7XO8

Page102

Anexampleofthistypeofdesignmightbeaschoolimplementinganewcurriculum.Observationsoftheoldcurriculummighttakeplacewithstandardizedreading
scoresfromprevioussemesters.Thesesamemeasurementscouldbeexaminedduringthenewcurriculumintervention.Thenewcurriculumisnotatemporary
interventionlikeaworkshop,buttakesplacecontinuouslyuntilreplaced.Thisdesignisespeciallypopularwhentherearestudentrecordswithmanyrepeated
measuresthatcanbeusedforobservationsandwhenitisnotpossibleorpracticaltoimplementacontrolgroup.
MultipleGroupTimeSeriesDesigns
Thesetimeseriesdesignsaresimilartothoseforthesinglegroup,butarestrongerbyaddingacomparisongroupthatreceivesthesamenumberofmeasurement
periods,butdoesnotreceivetheintervention.
Themultiplegrouptimeseriesdesignwithtemporarytreatmentisseenasfollows:
NR

E:

O1O2O3O4XO5O6O7O8

NR

C:

O1O2O3O4~XO5O6O7O8

Wecanprovideanexampleofthistypeofdesignbyextendingourworkshopexamplefromthesinglegrouptimeseriesdesign.Supposethatthecompanythatis
tryingtopromotecooperationthroughtheteamplayerworkshopestablishesacomparisongroupbyexaminingcooperativeinteractionsamongthoseworkersthatdid
notattendtheworkshop.Or,amorecommonoccurrencewouldbetoexamineworkersatasimilarcompany(perhapsanotherbranch)thatdidnotreceivethe
workshop.
Themultiplegrouptimeseriesdesignwithcontinuoustreatmentisthefinaltimeseriesdesignthatwediscuss.Thisdesigncanbeseenasfollows:
NR

E:

O1O2O3O4XO5XO6XO7XO8

NR

C:

O1O2O3O4~XO5~XO6~XO7~XO8

Ifwereturntoourschoolcurriculumexample,thesinglegrouptimeseriesdesignwithcontinuoustreatmentscouldbeextendedtothemultiplegrouptimeseries
designwithmultipletreatmentsbyaddingacomparisongroup,perhapsfromanotherdistrict.Thiscomparisongroupwouldjustreceivethetraditionalcurriculum.
Conclusions
Timeseriesdesigns(especiallysinglegrouptimeseriesdesigns)havebecomeimportantdesignsineducationalsettings,whereitisoftennotpracticaltointroducea
controlgroup.Thekeyadvantageofsuchatimeseriesdesign,incontrasttotheonegrouppretestposttestpoorquasiexperimentaldesign,istherepeated
observationsorrecords,whichprovideadegreeofassurancethatchangesarenotcausedbyotherenvironmentaleventsormaturation.Anothertypeofquasi
experimentaldesign,singlesubjectdesigns,isdiscussedindetailinthenextchapter.

Page103

RandomizedExperimentalDesigns
PosttestOnlyControl(orComparison)GroupDesign
Theposttestonlycontrolgroupdesigncanbeshownasfollows:
R

E:

C:

~X

Thesequentialoperationsoftheposttestonlycontrolgroupdesignaretoassignparticipantsrandomlytoeitheraninterventionorcontrolgroup(itshouldbe
rememberedthatmorethantwogroupsmaybeusedwithanyofthesedesigns).Then,theinterventiongroupreceivestheintendedinterventionandthecontrolgroup
receiveseitheradifferentinterventionornointervention.Iftwodifferentinterventionsareused,thisdesigncouldbecalledaposttestonlycomparisongroupdesign.
Attheendoftheinterventionperiod,bothgroupsaremeasuredbyusingaformofinstrumentationrelatedtothestudy(dependentvariable).
Anexampleoftheposttestonlycontrolgroupdesignisastudyinwhichtheinvestigatorisinterestedinaparticularintervention.Laconte,Shaw,andDunn(1993)
investigatedtheeffectsofparticipationinadevelopmentallyappropriateaffectiveeducationprogram.Studentswhowereatriskfordroppingoutwererandomly
assignedtoaninterventiongrouporacontrolgroup.Theinterventiongroupreceiveda15weekgrouptreatmentthatinvolvedarationalemotivecurriculum.Atthe
endofthe15weeksession,bothgroupsweretestedonaselfconceptscale.
Thekeypointfortheposttestonlycontrolgroupdesignistherandomassignmentofparticipantstogroups.Onecanassumethatifparticipantsareassignedrandomly
toeitheroneortheothergroup,thetwogroupsareequivalentpriortotheintervention.Therefore,iftherearedifferencesonthedependentmeasurefollowingthe
intervention,itcanbeassumedthatthedifferencesresultfromtheinterventionandnotfromdifferencesinparticipantcharacteristics.Rememberthatequivalenceof
participantcharacteristicsisakeydimensionofinternalvalidity.Doesrandomassignmentofparticipantstogroupsalwaysmakethegroupsequivalent?Withlarge
numbers(say30ormoreinagroup)ofparticipantsorahomogeneoussampleofparticipants,theinvestigatorcanbeconfidentthatrandomassignmentwillyield
equivalentgroups.However,withsmallernumbersinthesample,orveryheterogeneousparticipants,lessconfidencecanbeplacedinrandomassignmentproviding
equivalentgroups.Inthelattercases,adifferentexperimentaldesign,thepretestposttestcontrolgroupdesign,issuggested.
PretestPosttestControlGroupDesign
Thepretestposttestcontrolgroupdesigncanbeshownasfollows:

Page104

E:

O1

O2

C:

O1

~X

O2

Thesequentialoperationsofthepretestposttestcontrol(orcomparison)groupdesignareasfollows.First,participantsarerandomlyassignedtogroups.Then,each
groupismeasuredonthedependentvariable(pretest).Theinterventiongroupthenreceivestheintervention,andthecontrolgroupreceivesthetraditionaltreatment.
Becausethecontrolgroupparticipantsmaydropoutofthestudyornottryhardtodowellontheposttest,itisrareandnotdesirableforthecontrolgrouptoreceive
notreatmentatall.Aftertheinterventionperiod,bothgroupsaremeasuredagainonthedependentvariable(posttest).
Thepretestposttestcontrolgroupdesignisthemostcommonrandomizedexperimentaldesign.Anytimeatreatmentiscomparedtoacontrolgroupacrosstwotime
periods,usuallypretestandposttest,thisisthedesignthatisused.Itisrandomizedexperimentalbecausetheparticipantsarerandomlyassignedtogroupspriortothe
initialmeasurement(pretest)period.Thereasonforusingthisdesignascomparedtotheposttestonlycontrolgroupdesignistocheckforequivalentgroupsbefore
theintervention.Ontheotherhand,thedisadvantageisthatifapretestisused,itcouldbiastheparticipantsaboutwhattoexpectofthestudy,anditcouldinfluence
theminsomewaythatis,therecouldbecarryovereffects.Theinvestigatormustweightheadvantagesofgivingapretest,thatis,gaininginformationaboutthe
equivalencyofgroups,withthedisadvantageofpossiblybiasingtheposttest.Oftenthedecisionismadebythesamplesize.Ifeachgroupiscomposedofatleast30
participantsafterrandomassignment,theresearchermaychoosetousetheposttestonlycontrolgroupdesign,becausewiththatnumberofparticipantsitisexpected
thattheconceptofrandomnessshouldworkandbothgroupswouldbeexpectedtobeequivalent.Ontheotherhand,ifeachgrouphasonly10participants,andthe
groupsareheterogeneous,thenthepretestposttestcontrolgroupdesignisprobablybestbecauseitispossiblethatrandomassignmentdidnotmakethegroups
equivalentandfurtherstatisticaladjustment(analysisofcovariance)maybenecessary.
SolomonFourGroupDesign
Onemethodofdealingwiththepossibleeffectofthepretestintherandomizedexperimentalapproachistoincludeinterventionandcontrolgroupsthatreceivethe
pretest,andinterventionandcontrolgroupsthatdonotreceivethepretest.Thisrandomizedexperimentaldesign,calledtheSolomonfourgroupdesign,appearsas
follows:

E1:

E2:

C1:

C2:

O1

O1

O2

O2

~X

O2

~X

O2

Page105

ThesequentialoperationsoftheSolomonfourgroupdesignareasfollows.First,participantsarerandomlyassignedtooneofthefourdifferentgroups.Then,twoof
thegroups(E1andC1)aremeasuredonthedependentvariable(pretest).Theothertwogroups(E2andC2)donotreceiveapretest.Then,twogroups(E1andE2)
receivetheintervention.Onegroupthatreceivestheinterventionwaspretested(E1)andonegroupthatreceivestheinterventionwasnotpretested(E2).Inaddition,
twogroupsdonotreceivetheintervention:onethatwaspretested(C1)andonethatwasnotpretested(C2).Therefore,theSolomonfourgroupdesignallowsthe
investigatortotesttheeffectsofapretestinadditiontotestingtheeffectsoftheintervention.However,todeterminetheeffectsofthepretestontheposttest,the
investigatormustdoublethenumberofparticipantshardlyworththecostandeffortinmostsituations.
RandomizedExperimentalDesignwithMatching
Thelastspecificexperimentaldesign,whichiscommonlyused,isonewhereparticipantsarematchedonsomecharacteristicpriortotheintroductionofanyofthe
conditionsofthestudy.Thecharacteristicthatisusedforthematchmustberelatedtothedependentvariableotherwise,matchingisawasteoftimeandresultsina
lossofpower.Thesequentialoperationsoftheexperimentaldesignwithmatchingareasfollows.First,theinvestigatormeasuresalloftheparticipantsonsome
characteristic(variable)thatappearstoberelatedtothedependentvariable,suchasintelligence.Next,iftheindependentvariablehastwolevels,theinvestigator
dividesalloftheparticipantsintopairsofparticipantsfromtheirscoresontheintelligencetest.(Iftherewerethreelevelsorgroups,theparticipantswouldbedivided
intriads.)Theideaistohavepairsthatareascloseaspossibleonthevariableofintelligence.Forexample,iftherewere6participantswithIQscoresof122,110,
99,102,113,and120,thenthethreepairsmightbe122with120,113with110,and102with99.Afterallpairsareformed,theinvestigatorrandomlyassignsone
memberofthepairtotheinterventiongroupandtheothermemberofthepairtothecontrolgroup.Thekeytotherandomizedexperimentaldesignwithmatchingisto
makeitasifthe2participantsareidentical(atleastasfarasthecharacteristicsofinterest).Therefore,itisasthough1participantisreceivingbothconditionsofthe
study,eventhoughthereareactually2differentparticipantsineachpair.
Summary
Table7.2providesasummarydiagramofthevariousspecificdesignsdiscussedinthischapter.Rememberthatrandomassignmentofparticipantstogroupsiswhat
differentiatesrandomizedexperimentsfromquasiexperiments.Wehavediscussedthestrengthsandweaknessesofeachdesign,pointingoutthreatstointernalvalidity
whereappropriate.Rememberthattherandomizedexperimentaldesignshavethestrongestinternalvalidityandprovidethebestinformationaboutwhetherthe
independentvariablecausedchangesinthedependentvariable.

Page106
TABLE7.2
SummaryofSpecificDesignsforExperimentsandQuasiExperiments
TypeofDesign

Assign.

Grp.

Poorquasiexperimentaldesigns

Pre.

Onegroupposttestonlydesign

NR

E:

NR

E:

Posttestonlydesignwith

NR

E:

Nonequivalentgroups

I.V.

Post.

OnegroupPretestPosttest
design

NR

C:

Quasiexperimentaldesigns

PretestPosttestnonequivalent

NR

E:

NR

C:

~X

Withtemporarytreatment

NR

E:

OOO

OOO

Withcontinuoustreatment

NR

E:

OOO

XOXO

XOXO

NR

E:

OOO

OOO

NR

C:

OOO

~X

OOO

NR

E:

OOO

XOXO

XOXO

NR

C:

OOO

OO

OO

Randomizedexperimental
designs

Posttestonlycontrolgroup
designs

E:

C:

~X

comparisongroupdesigns

~X

Singlegrouptimeseriesdesign

Multiplegrouptimeseries
designs
Withtemporarytreatment

Withcontinuoustreatment

PretestPosttestcontrolgroup
designs

E:

C:

~X

~X

Solomonfourgroupdesign

E1:

E2:

C1:

C2:

~X

E:

C:

~X

Randomizedexperimentaldesign MR
withmatching
MR

Note.Assign.=assignmentofsubjectstogroups(NR=nonrandom,R=random,MR=matchedthen
randomlyassigned)Grp.=grouporcondition(E=experimental,C=controlorcomparison)Pre=pretest
(O=anobservationormeasurementablankmeanstherewasnopretestforthatgroup)I.V.=active
independentvariable(X=intervention,~X=controlorcomparisontreatment)Post=posttest(O=a
posttestobservationormeasure).

Page107

StudyAids
Concepts
Multiplegrouptimeseriesdesigns
Onegroupposttestonlydesign
Onegrouppretestposttestdesign
Pretestposttestcontrolgroupdesign
Pretestposttestnonequivalentcomparisongroupdesigns
Posttestonlycontrolgroupdesign
Posttestonlydesignwithnonequivalentgroups
Randomizedexperimentaldesignwithmatching
Singlegrouptimeseriesdesigns
Solomonfourgroupdesign
Distinctions
Controlgroupversuscomparisongroup
Poorquasiexperimentalversusbetterquasiexperimentalversusrandomizedexperimentaldesigns
Randomassignmentofparticipantstogroupsversusrandomassignmentoftreatmentstogroups
Randomassignmentversusnonrandomassignmentofparticipantstogroups
Weakquasiexperimentsversusmoderatestrengthquasiexperimentsversusstrongquasiexperiments
ApplicationProblems
Forthefollowingthreescenarios:
a.Identifytheindependentvariable(s).Foreach,statewhetheritisactiveorattribute.
b.Identifythedependentvariable(s).
c.Identifythespecificdesignname(e.g.posttestonlycontrolgroupdesign).Iftheapproachisquasiexperimental,evaluateitsstrength.
1.Youarearesearcherinscienceeducationwhoisinterestedintheroleofdiagramsininstruction.Youwishtoinvestigatewhetherusingdiagramsinplaceoftextwill
facilitatecomprehensionoftheprinciplesandconceptstaught.Todoso,youhavedevelopeda12thgradephysicsunitthatincorporatestheliberaluseofdiagrams.
Youplantocomparestudent'sknowledgeofphysicsbeforeandaftertheinstructionalunit.Youwillteachoneofyourclassesusingthediagramunitandtheother
usingthetextonlyunit.
2.Thepurposeofthisstudywastodeterminewhethertypeofclasscouldalterattitudestowardpersonswithdisabilities.Twoclassesatalargeuniversitywere
studied.Oneclass,SurveyofHumanDisease,placedemphasisonspecificdiseasesandhandicappingconditions.Theemphasiswasonhowthesecondi

Page108

tionsdifferedfromeachother.Theotherclass,HandicappedIndividualinSociety,placedemphasisonabilitiesanddidnotaddresshowhandicappingconditions
weredifferentforthisstudy.Twentydifferentvolunteersfromeachclassservedassubjectsforthisstudy.Attheendofthefirstsemester,allsubjectsweretestedon
theAttitudeTowardDisabledPersonsScale(ATDP).
3.Aresearcherwantstostudytheeffectsofsocialworkersupportonhomelesspeoples'jobattainment.Therearetwosimilarmissionsites.Asocialworkerspendsa
monthatoneofthesites,butnottheother.Thepeopleatthesitesdidnotdifferinaverageage,gender,andeducation.Attheendofayear,shecollectsthefollowing
dataonthetwogroupsfromlabordepartmentrecordsfortheprevioustwoyears:monthlytotalsofthenumberofdaysofemployment.
4.Explaintherationaleforarandomizedexperimentaldesignwithmatching.
5.Healtheducatorsadministeringalargewellnessprogramareinterestedinwhetherstructuredclassesorsupportgroupsseemtohavethegreaterinfluenceonhealthy
attitudestowardfood.Individualsvoluntarilysignupforeithertheclassesorthesupportgroups.Theirplanistorandomlyselect30participantsfromtheclassesand
30fromsupportgroupsand(withtheirpermission)administeraneatingattitudesinstrumentasapretestandasaposttesttoassesschangeinattitudesovertime.One
ofthehealtheducatorsexpressedtheconcernthattakingtheeatingattitudestestpriortothecoursewouldhaveanaffectonposttestscoresbecauseparticipantswill
alreadybefamiliarwithitemsontheinstrumentandmayattempttoprovidethesociallydesirableresponse.Practiceeffectswouldbeanissue.Whatcouldtheydoto
addressthis?
6.Describehowaresearchercouldexploretheimpactofanewcurriculumonattendance:
a.Utilizingasinglegrouptimeseriesdesign.
b.Diagramthedesignandgivethespecificdesignname.
c.Whyisatimeseriesdesignstrongerthanasimilardesignthatisnotatimeseriesdesign?
7.Subjectsarematchedinpairsonkeyattributevariablesoftestscoresandage,andthenthechildrenineachmatchedpairarerandomlyassignedtooneoftwo
groups,onereceivingtheinterventionandonereceivingnointervention.
a.Whatspecifictypeofexperimentaldesignisthis?Explain.

Page109

Chapter8
SingleSubjectDesigns
ReversalDesigns
MultipleBaselineDesigns
MultipleBaselineAcrossSubjectsDesign
MultipleBaselineAcrossBehaviorsDesign
MultipleBaselineAcrossSettingsDesign
Flexibility,RandomAssignment,andMultipleBaselineDesigns
MeasurementPeriodsandInstruments
PaperPencilTests
BehavioralObservation
EvaluationoftheResultsofSingleSubjectDesigns
VisualAnalysisofSingleSubjectDesigns
StatisticalAnalysisofSingleSubjectDesigns
ConsiderationsofInternalandExternalValidityofSingleSubjectDesigns
Summary
StudyAids
Concepts
Distinctions
ApplicationProblems

Page110

Inthischapter,wedescribeasubcategoryofquasiexperimentaltimeseriesdesignsthatcanbeusedwithveryfewparticipants.Thesesinglesubjectdesignshave
manyofthecharacteristicsthatgoverntraditionalgrouptimeseriesdesigns,suchasthenumerousrepeatedmeasuresoneachparticipantandtheinitiationand
withdrawaloftreatment.However,usingveryfewparticipantsincreasestheflexibilityofthedesignandleadstocompletelydifferentmethodsofdataanalysis.
Therefore,wehavechosentodevoteaseparatechaptertothesetypesofdesigns.Thetopicofsinglesubjectdesignsiscomplex,andcontainstoomuchmaterialto
becoveredcompletelyinasinglechapter.Forthoseinterestedinacompletetreatmentofthetopic,wesuggestthetextbyKazdin(1982).Foraclinicalperspective
onsinglesubjectdesign,Ottenbacher(1986)isanexcellentwork.Fortheanalysisofsinglesubjectdesigns,wesuggestKratochwillandLevin(1992).
Singlesubjectdesignsbecameprominentinthefieldofpsychologyinthe1960s,resultingintwojournals,TheJournaloftheExperimentalAnalysisofBehavior
andtheJournalofAppliedBehaviorAnalysis.Theconceptualizationofsinglesubjectdesignsisexplainedasfollows.Inatraditionalgroupdesign,10participants
mightbeassignedtoreceivethetreatment,and10participantswouldnotreceivethetreatment.Attheendofaparticulartimeperiod,thetwogroupsarecompared
todetermineifthetreatmentwassuccessful.Ifthegroupthatreceivesthetreatmentperformssignificantlybetterthanthegroupthatdoesnotreceivethetreatment,
thenajudgmentismadethatthetreatmentwassuccessful.Notethatonlyonetreatmentwasgiven1timeto10participantsandnotreatmentwasgiventotheother
10participants.Participantsweremeasuredpriortotheinterventionandaftertheintervention.Nowconsiderasituationwhere1participant(orsometimesasmanyas
3or4participants)receivesthesametreatment10timesand,inaddition,thetreatmentiswithheldover10differenttimestothesameparticipant.Eachparticipant
wouldbemeasured20times.Ifeachtimethetreatmentwasgiven,anincreaseinthedesiredbehavioroccurred,andeachtimethetreatmentwaswithdrawn,the
desiredbehaviorfailedtooccur,onecouldconcludethatthetreatmentwassuccessfulinincreasingthedesiredbehavior.
Wecandescribesinglesubjectdesignsastimeseriesdesignswhereanintervention(activeindependentvariable)isgiventoveryfewparticipants,fourorfewer.In
mostsituations,theindependentvariableisinitiatedandwithheldnumeroustimesthroughoutthestudy.Insomesituations(e.g.,multiplebaselinesinglesubject
designs),theremovaloftheindependentvariableisnotnecessarytobeincludedasasinglesubjectdesign.Singlesubjectdesignsarequasiexperimentaldesigns
becausetheymustincludeanactiveindependentvariable.Inaddition,thereisnorandomassignmentofparticipantstotreatments.Thisshouldnotbeconfusedwith
casestudytypesofdesigns.Casestudydesignsfallunderqualitativeresearchmethods,wheredescriptionsofparticipantsinnaturalsettingsaretherule.Casestudies
areoftenusedtodescribeanunusualcase,ortoprovidemoredescriptiveevidencetosupportaquantitativestudysuchasaprogramevaluation.
Inthischapterweintroducethetwomajortypesofsinglesubjectdesigns,ABABorreversaldesigns,andmultiplebaselinedesigns,andweprovideex

Page111

amplesforboth.Next,wediscussthemethodsofobservationandlengthsoftimeofmeasurementperiodsinsinglesubjectdesigns.Thenwediscusstheanalysesof
thesetypesofdesigns.Finally,wediscusstheinternalandexternalvalidityofsinglesubjectdesigns.
ReversalDesigns
Reversaldesigns,oftenreferredtoasABABdesigns,aretheoriginalsinglesubjectdesignsandarestillthemostcommontypeofsinglesubjectdesign.Inthese
designs,thefirstAstandsforthebaselineperiod,wheretheparticipantisusuallyobservedforanumberoftimeperiods.Thekeyhereisthattheparticipantis
observeduntilthebaselineisrelativelyflat.Thisisalargedeparturefromtraditionalgroupdesigns,wheretheamountoftimeallottedtotheexperimentalandcontrol
treatmentsisdecidedpriortothestudy.Insinglesubjectdesigns,theinvestigatorplotsthedataforeachmeasurementperiodongraphpapertodetermineifthe
behaviorduringbaseline(ortreatment)isincreasing,decreasing,orlevelingoff.ThefirstBperiodreferstothefirstinterventionperiod.Afterthebaselinehasleveled
off,theinvestigatorinitiatesthetreatmentoractiveindependentvariable.Again,theinvestigatorplotsthedatafromeachsessiontodeterminetheeffectofthe
treatment.Ifoneweretostophere(henceanABdesign),itwouldbeeasytoseethethreatstointernalvalidity,especiallycontrolofextraneousvariables,thatcould
arise.Becausetherehasbeenonlyonebaselineandonetreatmentphase,itisdifficulttoknowwhetheritisthetreatmentoranothervariablethatismakingthe
difference.Therefore,oncethetreatmentdataappeartoleveloff(arelativelyflatline),theinvestigatorwithdrawsthetreatment,andinitiatesasecondAphase.The
investigatorobservesthisphaseforseveralperiods(threeattheminimum)untilthebehaviorlevelsoff.ThentheinvestigatorinitiatesthesecondBortreatmentphase.
Thiscompletestheminimumreversaldesign,withtwoAorbaselinephasesandtwoBortreatmentphases.ItshouldbenotedthathavingtwoAandtwoBphases
doesnoteliminateallthreatstointernalvalidity,andthatthemoreAandBphasesinserted,themoreconvincingisthedesignofthestudy,similartoanytimeseries
design.
WhatshouldhappeninatypicalABABsinglesubjectstudy?Figure8.1demonstratesasinglesubjectABABdesign.Onewouldexpectthatduringtheinitialbaseline
period(A1)theremaybesomefluctuationofresponses,butafterthefirstfewperiods,theparticipant'sresponses(dependentvariable)shouldhabituateorleveloff.
Duringtheinitialtreatmentperiod(B1),behaviorshouldincrease(ordecreaseifthetreatmentisdesignedtoreduceanundesirablebehaviorsuchasaggression).One
wouldexpectthisbehaviortoincreaseuptoapoint,andthentoleveloff.Next,duringthewithdrawalofthetreatment(A2period),theexpectationisthatperformance
willdecrease(althoughperhapsnotaslowastheA1period)andthenbegintostabilize.Whenthestabilizationhasoccurred,thereintroductionofthetreatment(B2
period)takesplace,andperformanceshouldincreaseabovethatofalloftheprecedingphases.
AlthoughtheprecedingparagraphoutlinestheidealresultsofanABABsinglesubjectdesign,thingsrarelyhappenexactlyasplanned.Perhapsthemost

Page112

commonprobleminvolvesstabilizationofperformanceforeachphaseofthestudy.First,itoftentakestimetoreachastablebaseline.Howlongshouldthe
investigatorwaituntilthebaselinestabilizes?Usually,thebaselineshouldstabilizewithinfiveorsixperiods.Iftheperformanceisstillirregular(i.e.,highonedayand
lowthenext),thentheinvestigatorshouldlookforexternalinfluencestoexplainwhytheparticipantisperformingsoirregularly.Sometimes,itjusttakesalittlewhileto
habituatetothesetting.Otherproblemsthatmayinterferewithstabilizationmayinvolvereactivitytothemeasureofperformance.Ifthemeasureofperformance
requiresthatanobserverbepresent,thismaycausereactivitybytheparticipant.
StabilizationoftheinitialbaselineisacommonprobleminABABdesigns,butitisnottheonlyproblem.Asecondproblemisrelatedtotreatmentwithdrawal.Many
singlesubjectdesignsinstituteatreatmentthathasapermanenteffect.Ifthisisthecase,theninthesecondbaseline(A2)phase,onewouldexpectthattheremightbe
littleornodropinperformance.Ifthereisnodropinperformance,thenonewouldexpectanincreaseinperformanceinthesecondintervention(B2)phasetoalevel
substantiallyhigherthanthatofthefirstintervention(B1)phase.
TheABABsinglesubjectreversaldesigndoesnotnecessarilymeanthatthereshouldonlybetwobaselinephasesandtwointerventionphases.MostABABdesigns
useatleastthreeAandthreeBphases,andmanyusemore.Actually,usingonlytwoAandtwoBphasesistheminimumthatcouldpassforasinglesubjectstudy.
MoreAandBphasesmakethestudymoreconvincing,andhenceincreaseinternalvalidity.Inaddition,theinvestigatorisnotlimited

Fig.8.1.
Agraphicrepresentationofthehypotheticaldataofan
ABABsinglesubjectexperimentaldesign.

Page113

tojustthephasesofAandB.ConsiderasituationwhereaftertheinitialAphase,theinvestigatorinitiatesatreatmentintheBphase.However,thetreatmentfailsto
increaseperformanceabovethatobservedduringthebaselineperiod.Ifthiswereatraditionalbetweengroupstypeofdesign,theinvestigatorwouldbestuckwitha
studythatfailedtorejectthenullhypothesis.Instead,inasinglesubjectdesign,theinvestigatorcouldmodifythetreatmentandintroduceit(C)aftertheBphase.Thus
thedesignmightbesomethinglikeABCAC.Thepointtorememberisthatastrengthofsinglesubjectdesignsisthattheyareflexible,andjustasyouarenot
predeterminedbythenumberofsessionscontributingtoanyparticularphase,youarealsonotpredeterminedtousejustanAandBphase.
AnexampleofareversaldesignwithonlytwobaselineperiodsandtwointerventionperiodscanbeseeninastudybyDunlap,FosterJohnson,Clarke,Kern,and
Childs(1995).Theyattemptedtoproducefunctionaloutcomeson3participantswhohadseveredisabilities.Theyexaminedtwodependentvariablesatthesame
time,ontaskbehaviorandproblembehavior.Theirbaselineperiodsconsistedofstandardoutcomes,whereastheinterventionperiodsconsistedoffunctional
outcomes.Figure8.2(adaptedfromDunlapetal.,1995)showsthedatafor1ofthe3participants.Noticethatduringthebaselineperiod,theontaskbehavioris
dropping,andthenlevelingoff,whereastheproblembehaviorislevelforthemostpart.Duringthefirstinterventionphase,theontaskbehaviorincreasesimmediately,
andthenlevelsoff,whiletheproblembehaviorremainsuniformlylow.Inthesecondbaselineperiod,theontaskbehaviorhasdecreasedtothatseenintheinitial
baselineperiod,andtheproblembehaviorhasincreasedabovethatseenintheinitialbaselineperiod.Duringthesecondintervention,whichonlyhadonesession,the
ontaskbehaviorincreased,andtheproblembehaviordecreasedbelowtheaverageofthepreviousperiod.WereturntoFig.8.2whenwediscusstheevaluationof
singlesubjectdesigns.
MultipleBaselineDesigns
Multiplebaselinesinglesubjectdesignswereintroducedmorerecentlythanreversaldesigns.Themajorreasonsfortheintroductionofmultiplebaselinedesignsisthat
(a)inclinicalsituationstheremovaloftreatmentwasoftenconsideredunethical,especiallyifthetreatmentappearedsuccessfuland(b)manyofthesestudieswere
beingperformedwhenthepatient,inoneformoranother,wasresponsibleforthepaymentofthetreatment.Inmultiplebaselinestudies,intheinitialstagesofthe
study,asmanyasthreebaselinesmayberecordedsimultaneously.Thesebaselinesmayrepresentresponsesofthreedifferentparticipants,responsesofthree
differentbehaviorsofthesameparticipant,ortheresponsesofthesameparticipantinthreedifferentsettings.Thekeytomultiplebaselinesinglesubjectstudiesis
thattheinvestigatorintervenesatarandomlyselectedtime,andobservestheeffectononlyoneofthebaselineswhiletheothertwobaselinesshouldbeunchanged.
Thistypeofdesigneliminatestheinternalvaliditythreatofhistorybecauseonewouldexpectthatifsomeexternaleventwasalteringbehavior,itwouldaffectall
participants,settings,orbehaviors,notjustone.

Page114

Fig.8.2.
Resultsshowingtheeffectsofthestandardversusfunctionaloutcomeassignments
onontaskandproblembehavior.From''ModifyingActivitiestoProduceFunctional
Outcomes:EffectsoftheProblemBehaviorsofStudentswithDisabilities,"byG.
Dunlap,L.FosterJohnson,S.Clark,L.Kern,andK.E.Childs,1995,TheJournalof
theAssociationofPersonswithSevereHandicaps(JASH),Vol.20,No.4,p.253.
Copyright1995byTheAssociationforSevereHandicaps(TASH).Adaptedwithpermission.

MultipleBaselineacrossSubjectsDesigns
Themostcommonmultiplebaselinedesignismultiplebaselineacrosssubjects.Itspopularityispartiallyasaresultoftheeaseofcompletingthistypeofstudy,
especiallyinaclinicalsetting.Theprocedureforcarryingoutthistypeofdesignisasfollows.Initially,theinvestigatorselects3(orperhaps4)differentparticipantsfor
thestudy.All3participantsareobservedconcurrentlyinabaselinephase,andtheirresponsesforeachbaselineperiodareplottedonagraph(seeFig.8.3).Next,
theinvestigatorgivestheinterventionto1oftheparticipants,whilecontinuingtoobtainabaselineontheother2participantsatthesametime.Afteragivennumberof
periods,theinterventionisstartedwiththesecondparticipant,continuedwiththefirstparticipant,andabaselineiscontinuedforthethirdparticipant.Again,aftera
numberofbaselineperiods,theinterventionisstartedwiththethirdparticipant,andcontinuedwiththefirst2participants.
AnexampleofamultiplebaselineacrosssubjectsdesigncanbeseeninastudybyBambaraandAgar(1992),whoexaminedthefrequencyofselfdirectedleisure
activitiesin3adultswithmoderatedevelopmentaldisabilities.Theinterventionwasselfscheduling.Themainpointtonoticeisthestaggeredbaselineacrossthe3
participants,whichissimilartothatshowninFig.8.3.Thefirstparticipant,P1,hadabaselineofseveralweeksandthenreceivedtheintervention.Meanwhile,P2and
P3continuedtobeinthebaselinecondition.Afterseveralmoreweeks,P2startedtheintervention,whileP3continuedthebaselinecondition.Finally,P3receivedthe
intervention.

Page115

Fig.8.3.
Resultsofamultiplebaselinearcosssubjectsdesign.

MultipleBaselineacrossBehaviorsDesign
Thissecondtypeofmultiplebaselinedesignislesspopularthanthemultiplebaselineacrosssubjectsdesign,especiallyinclinicalsettings.Theprocedureforthistype
ofdesignisthatthreedifferentbehaviorsofthesameparticipantaretargetedforchangebytheinvestigator.Insteadofrecordingbaselinedatafor3different
participants,asinmultiplebaselineacrosssubjectsdesigns,baselinesforthreedifferentbehaviorsarerecordedconcurrently.Then,aninterventionisstartedwithone
ofthebehaviors,whilebaselinescontinuetoberecordedontheothertwobehaviors.
Next,thesecondbehavioristargetedwithanintervention,whilethefirstbehaviorcontinueswiththeintervention.Last,thethirdbehavioristargetedwithan
intervention,whiletheothertwobehaviorscontinuetoreceivetheintervention.Forthistypeofdesigntobesuccessful,onemustassumethatthetreatmentsaffect
eachtargetedbehaviorindependently.Inotherwords,whenonebehaviorisbeingtreated,itisimportantthattheotherbehaviorsarenotaffected.Ontheotherhand,
iftreatingonebehavioraffectsthesecondbehavior,thenthedesignwillnotbesuccessfulbecauseallbehaviorswillchangeatthesametime.Forexample,inthefield
ofoccupationaltherapy,wheremosttreatmentsareassumedtobeholistic,itisdifficulttofindbehaviorsthatwouldbe

Page116

increasedoreliminatedbytreatmentsthatareindependentofeachother.Ontheotherhand,theprominentuseofoperantconditioningtechniquesinthefieldofspecial
educationmakesthistypeofdesignidealbecausespecificbehaviorscanbetargetedwithoutaffectingotherbehaviors.
MultipleBaselineacrossSettingsDesign
Thistypeofmultiplebaselinedesignissimilartomultiplebaselineacrossbehaviorssinglesubjectdesign.Usuallyasingleparticipantisusedinthestudy.However,in
multiplebaselineacrosssettingsstudies,asinglebehaviorisusuallytargeted,butinatleastthreedifferentsettings.Theprocedureforthistypeofdesignisthatbaseline
responsesarecollectedon1participantinthreedifferentsettings.Forexample,onesettingmightbethetherapist'sofficeorclinic.Asecondsettingmightbeatthe
participant'shome.Thethirdsettingmightbeattheparticipant'sschool.Afteranumberofbaselineperiods,interventioncouldbeginattheclinic.Duringthistime,
baselineresponsesatschoolandathomewouldstillberecorded.Next,afterafewperiods,interventionmightstartathome.Meanwhile,nointerventionwouldbe
attemptedatschool.Last,theinterventionatschoolwouldbetried.Themultiplebaselineacrosssettingssinglesubjectdesignsuffersfromsomeofthesameproblems
asthemultiplebaselineacrossbehaviorsdesign.Forthedesigntobesuccessful,onewouldexpectchangesinresponseineachsettingtobeindependentofeach
other.However,realistically,atherapistwouldhopethattreatmentwouldgeneralizeacrosssettings,justasitmightbeexpectedtogeneralizeacrossbehaviors.Ifthis
istheexpectation,thenmultiplebaselineacrossbehaviorsdesignandacrosssettingsdesignareprobablynotgooddesignstouse.
Flexibility,RandomAssignment,andMultipleBaselineDesigns
Oneofthestrengthsofreversaldesignsisflexibility.Ifonetreatmentisnotworking,whynotmodifythetreatment?Moreimportant,acarefullyperformedreversal
singlesubjectdesignpayscloseattentiontostablebaselines.Theinvestigatorhastheflexibilitytowaitwithinaparticularphaseofthedesignuntiltheresponseis
stable.Ontheotherhand,thestrongerthemultiplebaselinedesign,thelessflexibility.Thekeytoastrongmultiplebaselinedesignisrandomlydecidingaheadoftime
whichofthe3participants,behaviors,orbaselineswillgettheinterventionfirstandwhenthetreatmentwillbeginforeachparticipant,behavior,orbaseline.Random
assignmentoftreatmenttoaparticularparticipant,behavior,orsettingmeansthattheinterventionmustgoahead,evenifthebaselineisnotstable,whilebaselinesof
otherparticipants,behaviors,orsettingsmightbequitestable.Therandomassignmentlendscredibilitytothedesign,butitreducesflexibilityconsiderably.Afurther
problemwithrandomassignmentmightbethataparticularorderofinterventionswithbehaviorsorsettingsmightbeconsideredadvisable.Forexample,ifoneis
planningamultiplebehavioracrosssettingsdesignatclinic,home,andschool,itisdoubtfulthathomeorschoolwouldbeplannedasatargetsettingbeforetheclinic.
However,randomassignmentprecludesthisfromhappening.

Page117

MeasurementPeriodsandInstruments
Thenumberofmeasurementperiodsmaychangebetweenonephaseandanotherinareversaldesign.Oneshouldwaituntileachphaseisstablebeforeinstigatingor
withdrawingtreatment.Thisaddstotheflexibilityofthedesign.Ontheotherhand,eachmeasurementperiod(session)mustbethesamelengthoftime.Youcannot
recordresponsesonaparticipantfor1/2hronedayand1hronthenextday.Thisinvalidatesthedesign,becausethenumberofresponsesperperiodorsession
wouldhavenomeaningforcomparison.
Asecondmeasurementissuetoconsiderwhenperformingsinglesubjectdesignsisthatthetypeofinstrumentselectedcouldseriouslycompromisethestudy.Each
sessionmustyieldascoreoranumberofresponses.Iftherearealimitednumberofresponsespersession,thenyourinstrumentmaynotbesensitiveenoughforthe
study.Therearetwopopulartypesofmeasures(dependentvariables)usedinsinglesubjectdesigns:paperpenciltestsandbehavioralobservation.
PaperPencilTests
Thesetypesofinstruments,whichoftenarestandardized,helpinsuretheinternalvalidityofthestudy(instrumentreliabilityandstatisticalvalidity).However,ifyou
decidetouseastandardizedinstrumentsuchasapaperandpenciltest,thenyoumustdetermineboththelengthoftheinstrument,andhowoftenyoucoulduseit
withouttheparticipantgettingbored,orbecomingunreliableinresponding.Typically,paperandpenciltestsareusedonlyonceaweekandusuallyinconjunction
withsomeothermeasuresuchasobservation.
BehavioralObservation
Observationoftheparticipant'sbehaviorisprobablythemostcommonformofmeasureinsinglesubjectdesigns.Certainrulesshouldbefollowedwhenusing
observation.
1.Itisbesttohavetheobserverbesomeonedifferentfromtheteacher,parent,ortherapist.
2.Itisbesttohavetheobserverbeasdiscreteaspossible(i.e.,apassiveobserverwhoisanotherstudent,studentsintheclassroom,oranobserverwatchingthrough
aonewaymirror).
3.Thecriticalresponsestobejudgedshouldbewelldefinedpriortothestudy.
4.Morethanonejudgeshouldbeusedtorecordtheresponses.
5.Interraterreliabilityshouldbecarriedoutamongjudgespriortothestudy.
EvaluationoftheResultsofSingleSubjectDesigns
Theearlystudiesusingsinglesubjectdesigns,especiallythosedonewithratsorpigeons,typicallyhadverystablebaselinesandinterventionperiods.Inad

Page118

dition,thenumberofbaselineandinterventionperiodsfarexceededthosethatareusedinstudieswithhumans,especiallyclinicalstudies.Forbothofthesereasons,
theearlysinglesubjectstudiesdidnotusestatisticalanalysistoconvincetheappropriateaudiencethatinterventionsweresuccessful.Instead,theinvestigatorsbelieved
thatthegraphicdisplayswereconvincing.However,singlesubjectstudieswithhumans,especiallyreversaldesigns,usuallyhavefewerbaselineandintervention
periodsthananimalstudies.Inaddition,thebaselineperiodsareoftenshorterandlessstable.Perhapsanevengreaterproblemforvisualanalysisofsinglesubject
designsisserialdependency(Parsonson&Baer,1992).Sincesinglesubjectdesignsarerepeatedmeasuresorwithinsubjectsdesigns,aproblemarisesthateach
datapointisusuallynotindependentfromthepreviousorfollowingdatapoint.Inotherwords,ifoneknowsthevalueofaparticulardatapoint,thevalueofthenext
datapointcouldbepredicted.Itappearsthatserialdependencymaycauseinconsistencyinagreementontheeffectoftheinterventioninsinglesubjectdesigns.
Consideringalloftheseproblems,therehasbeenanincreasingemphasisgiventousingsomeformofstatisticalanalysisinadditiontovisualanalysis.
VisualAnalysisofSingleSubjectDesigns
Whenexploringasinglesubjectgraphvisually,thekeyistolookforpatternsinthedata,especiallyasthephaseschangefrombaselinetointervention,andbackto
baseline.AreexaminationofFig.8.2showsdefinitechangesinontaskbehavioraseachphaseofthestudychanges.Ontheotherhand,thechangesinproblem
behaviorarenotasconvincing,especiallywhenchangingfromthefirstbaselinetofirstinterventionphase.Kazdin(1982)discussedtheuseofcertaincriteriaforvisual
inspectionofsinglesubjectdesigns.OnecriterionintroducedbyKazdinisreferredtoaslevel.Levelisthechangefromthelastmeasurementinaphasetothefirst
measurementinthenextphase.Forexample,inFig.8.2,anincreaseinlevelcanbeseenfortheontaskbehaviorbetweenthefirstbaselineandfirstintervention
periods,andthesecondbaselineandsecondinterventionperiods.Adecreaseinlevelcanbeseenbetweenthefirstinterventionperiodandthesecondbaselineperiod
fortheontaskbehavior.Sometimes,justexaminingchangeinlevelcanbemisleading.Forexample,inFig.8.2,noticethattheleveldecreasesforproblembehavior
betweenthesecondbaselinephaseandthesecondinterventionphase.However,carefulexaminationoftheproblembehaviorduringthesecondbaselinephaseshows
avariablepatternofincreaseandthendecrease.Fromthispattern,onemightexpectadecreaseonthenextmeasurementperiod,regardlessofanintervention.
Becausethecriterionofleveldoesnotalwaysreflectthepatternofaparticularphase,onecouldusemeanlevel,asasecond,morestablecriterion(Kazdin,1982).
Meanlevelreferstotheaverageofthepointsinonephasecomparedtotheaverageofthepointsinthenextphase.
AthirdcriterionforvisualanalysissuggestedbyKazdin(1982)wasthatoftrend.Atrendindicatesadefinitedirectionofthepointswithinaphase.Thetrendcould
bepositive(goingup),negative(goingdown),orflat(goinginneitherdirection).InFig.8.2,thetrendinontaskbehaviorduringthefirstbase

Page119

linephaseisnegative,butshiftstopositiveduringthefirstinterventionphase.Ontheotherhand,thetrendisfairlyflatduringthefirstinterventionphaseforproblem
behavior.
Ottenbacher(1986)usedtwomorecriteriaforvisualanalysisofsinglesubjectdesigns.OnecriterionforvisualanalysisrecommendedbyOttenbacheristhatof
variability.Ifonelookswithinanyparticularphase,alinecouldbefitthroughthepointsinthephase.Theamountofdistancethepointfallsfromthatlineisagood
measureofthevariabilitywithinthephase.Forexample,inFig.8.2,thesecondbaselinephaseshowsalargeamountofvariabilityfortheproblembehavior,yetvery
littlevariabilityfortheontaskbehavior.
ThesecondcriterionforvisualanalysissuggestedbyOttenbacher(1986)wasslope.Theslopereferstotheangleofincreaseordecreaseofthemeasurementpoints.
Slopehasbecomeimportantasanindicatorforvisualanalysisofsinglesubjectdesignsbecauseoften,asoneshiftsfrombaselinetointervention,thetrendmayremain
thesame,buttheangleincreasesduringtheintervention.Thiscanhappenifthebaselineperiodfailstostabilize.
Tohelpevaluatesinglesubjectdesigns,someresearcherssuggest,inadditiontotheprecedingcriteria,thattrendlinesbedrawnforeachphaseofthestudy.These
trendlinescouldbe"splitmiddle"lines(White,1974)orleastsquaresregressionlines.Thesplitmiddleprocedureusesmediansandproducesatrendlinethrougha
baselinesuchthathalfofthedatapointsareabovethelineandhalfarebelowtheline.Conceptually,ifthislineisextendedthroughtheinterventionperiod,thenone
couldjudgetheeffectoftheinterventionrelativetothebaseline.Ifapproximatelyhalfofthebaselinepointsareabovethelineandhalfarebelowthelineduringthe
interventionperiod,thenonecouldconcludethatnoeffecthastakenplace.Ontheotherhand,ifalmostallofthedatapointsareabovethelineduringtheintervention,
thenonewouldconcludethataneffecthastakenplace.Forthesplitmiddleorleastsquaresapproachestohavemeaning,aconsiderablenumberofdatapointsmust
beused.ParsonsonandBaer(1992)commented,however,thatwhenjudgesusedtrendlines,theytendedtoignoresomeoftheotherimportantcriteriathatwere
listedearlier.
StatisticalAnalysisofSingleSubjectDesigns
Althoughvisualanalysishasbeenoneofthestrengthsofsinglesubjectdesigns,sometimesthedatafromthesedesignsarenotconvincing.Therefore,investigators
whousethesedesignshaveresortedtostatisticalteststodetermineifinterventionshavemadeadifference.Kazdin(1982)discussedtheuseoftraditionalstatistical
tests,suchasattesttocomparethedifferencebetweenabaselineandinterventionperiod,orasinglefactoranalysisofvariancetocompareallphasesofanABAB
design.However,healsocautionedthatthesetestsshouldnotbeusedifserialdependencyexisted.Kazdin(1982)suggestedtheuseoftimeseriesanalysiswhen
serialdependencyproblemsexisted.However,therequirementofatleast20datapointsperphasebringsthesetypesofanalysesforhumansinglesubjectdesignsinto
questionbecauseweseldomcanaffordthatmanydatapoints(seeMcCleary&Welsh,1992foramoredetaileddescriptionoftimeseriesanalysesinsinglesubject
designs).

Page120

NonparametrictestssuchastheMannWhitneyUtest,theFisherexacttest,andthesigntesthavealsobeensuggested(Edgington,1992).
Thereisanincreasinginterestinusingrandomizationtestsinsinglesubjectdesigns.Levin,Marascuilo,andHubert(1978)demonstratethesimplicityofthistypeof
procedure.Theyprovide,asanexample,theanalysisofanABABdesign.Aswestatedearlier,inthetraditionalreversaldesignwithonlytwobaselinephasesandtwo
interventionphases,theresultsthatwouldbeexpectedareasfollows.Thelowestlevelofresponsewouldbeexpectedinthefirstbaselinephase.Thenextlowest
levelofresponsewouldbeexpectedinthesecondbaselinephase.Thefirstinterventionphaseshouldhaveahigherlevelofresponsethaneitherofthetwobaseline
phases.Thehighestlevelofresponseshouldbethesecondinterventionphase.ThiscouldbedepictedasA1<A2<B1<B2Ifoneweretohypothesizethisoutcome,
andtheresultsdid,infact,occurinthisorder,thentheprobabilityofthisoccurrenceis1in24orabout.05.Theideaisthattherearefourfactorialpossibleoutcomes
oftwoAphasesandtwoBphases,i.e.,4 3 2 1possibilities,whichequal24.Inadditiontothepredictedoutcome,thereare23otherssuchasA1>A2>A3>
A4.Althoughthisisanoversimplification,randomizationtestshavebecomepopularinsinglesubjectdesigns(seeEdgington,1992foramoreindepthanalysis).
ConsiderationsofInternalandExternalValidityofSingleSubjectDesigns
Levinetal.(1978)discussedtheinternalvalidityandexternalvalidityproblemsinsinglesubjectdesigns.Theinternalvalidityproblemsrelatetoproblemsinrandom
assignment.Withonly1participant,therecannotberandomassignmentofparticipantstotreatments(althoughthemultiplebaselineacrossparticipantsdoesallowfor
randomassignmenttodifferentlengthsofthebaseline).Moreimportant,theorderofthetreatmentphasesalsocannotberandomlyassigned.Athirdproblemisthe
possiblecarryovereffectsfromonephasetoanother.Ontheotherhand,theABABdesignreducesthethreatsofconfoundingvariables(outsideinfluencesotherthan
theindependentvariable).Theproblemsinexternalvalidityforsinglesubjectdesignsareevenmoreobvious.Therandomselectionof1participant,orevenasmall
numberofparticipants,isunusualbecausetheparticipantsareusuallyselectedbecauseofaparticularbehavioralorphysicalproblem.However,asstatedbyLevinet
al.(1978),"sincetheABABdesignhasfoundfavorwithappliedbehavioralresearchers,fornowwewillhavetoacceptitasagivenanddothebestwithitwecan,
despiteitslimitations"(pp.171172).
Althoughflexibilityintryingoutdifferenttreatmentshassomeadvantages,italsohasdrawbacks.Whateventuallyworksforoneclientorparticipantmaynotworkfor
another.Ofcourse,someoftheunsuccessfultreatmentsmayworkforanotherperson.Thus,ifwetriedmanytreatments,weshouldbecarefulinconcludingthatthe
successfultreatmentwillworkforallclientsorparticipants.

Page121

Summary
Wedescribedsinglesubjectdesignsasasubcategoryofquasiexperimentaltimeseriesdesignsthatcanbeusedwithveryfewparticipants.Usingveryfew
participantsincreasestheflexibilityofthedesignandleadstocompletelydifferentmethodsofdataanalysis.Thesesinglesubjectdesignsusenumerousrepeated
measuresoneachparticipantandtheinitiationandwithdrawaloftreatment.
Weintroducedthetwomajortypesofsinglesubjectdesigns,ABABorreversaldesigns,andmultiplebaselinedesigns,andprovidedexamplesforboth.TheABAB
designismoreflexiblethanmultiplebaselinedesigns,butoftentakeslongertocarryoutandisdependentonstablebaselines.Multiplebaselinedesignsareeasierto
carryout,buttoinsurestronginternalvalidity,randomassignmenttoaparticularinterventiontimeperiodmustbeinstituted.Thisreducesflexibilityinmultiplebaseline
designs.
Themethodsofmeasurementforsinglesubjectdesignsareusuallyobservationandpaperandpenciltests.Wediscussedtheirstrengthsandweaknesses.Twotypes
ofevaluationofsinglesubjectdesignsarevisualanalysisandstatisticalanalysis.Onemustbecautiouswheninterpretingtheresultsfromasinglesubjectdesignstudy,
especiallywithrespecttoexternalvalidity.
StudyAids
Concepts
Behavioralobservations
Level
Multiplebaselineacrossbehaviorsdesign
Multiplebaselineacrosssettingsdesign
Multiplebaselineacrosssubjectsdesign
Multiplebaselinedesigns
Paperpenciltests
Reversaldesigns
Slope
Trend
Variability
Distinctions
Levelversusmeanlevel
Levelversustrendversusslope
Multiplebaselineacrosssubjectsdesignversusmultiplebaselineacrossbehaviorsdesignversusmultiplebaselineacrosssettingsdesign
Reversaldesignsversusmultiplebaselinedesigns
Singlesubjectdesignsversustraditionalgroupdesigns

Page122

ApplicationProblems
1.AnABABreversaldesignisconsideredtobemoreflexiblethanmultiplebaselinedesigns.Why?
2.Whataretheadvantagesofselectingamultiplebaselineacrosssubjectsdesignascomparedtoareversaldesign?
3.Aninvestigatorperformsamultiplebaselineacrosssubjectsdesign.Hehasthreeparticipantsinthestudy,labeledA,B,andC.Allthreeparticipantswillreceivethe
sametreatment.Afterobservingtheparticipantsfor5days,theresearcherdecidestostartthetreatmentwithparticipantBsinceherbaselinewasthemoststable.
After3moredays,treatmentisinstigatedwithparticipantCsincehisbaselineismorestablethanparticipantA.After6moredays,treatmentisstartedwithparticipant
A.Whataretheadvantagesanddisadvantagesofthismethodofdecidingtheorderoftreatment?
4.Youareconductingastudytodeterminetheeffectsofaspecifictreatmentusingasinglesubjectdesign.Youdecidetouseareversaldesign(ABAB).Afterfive
sessions,astablebaselinewasestablished(phaseA).YouintroduceyourtreatmentduringphaseB,andaftereightsessionstherehasbeennoincreaseonyour
measure.YoudecidetomodifyyourtreatmentandintroducethenewtreatmentasphaseC.Afterfivesessionsyounoticeanincreaseeasilyvisualizedonyourgraph.
Whatshouldbeyournextsteps(phases)toestablishinternalvalidity?
5.ThefollowingsinglesubjectstudyisanABAdesign.Cathywashavingadifficulttimesucceedingathermailsortingjob.Herjobcoachdecidedtotryanewcuing
systemwithCathy,whichinvolvedverbalredirectswhenCathy'sattentionwouldgetoffofherwork.Forthebaselineperiod,thejobcoachcountedhowmanyletters
Cathycouldsortin15minutes.Aftereightmeasurementperiods,Cathyhadastablebaseline.Herjobcoachtheninstitutedtheverbalredirects,andmeasuredCathy
foreightmoremeasurementperiods.Afterthis,thejobcoachstoppedtheredirectsandmeasuredCathyforthelasteightmeasurementperiods.Themeasuresfor
eachsectionarelistedbelow:
A

50

65

60

60

70

65

35

70

65

45

75

70

50

80

70

45

80

65

50

85

60

50

90

65

a.Graphthemeasurements.
b.Describehowlevel,trend,andsloperelatestothegraph.
c.Formaconclusionaboutthestudy.
6.ThefollowingsinglesubjectstudyisanABABdesign.Bradisayoungmanwithmentalretardation.Anoccupationaltherapist(OT)hasbeenas

Page123

signedtoassistBradinlearninghisjob.Heisatriskforlosinghisjobbecauseheisforgettingtodocertaintasksonaregularbasis.TheOTdecidestotryaself
monitoringchecklistwithBradtoseeifthatwillhelphimkeeptrackofhisduties.For2workweekstheOTmonitorsBrad'swork,countinghowmanyassignedtasks
Bradcompleteswithoutcuing.Attheendof2weeks,shedecidestostartBradwithachecklist,andcounthistaskcompletion.After8workdays,Bradseemstobe
fairlystableinhisworkroutine.TheOTthenremovesthechecklistandmonitorsBradfor10moreworkdays,andcountshistaskcompletion.Afterthisperiod,the
OTreinstatesthechecklist.Themeasuresforeachsectionarelistedbelow:
A

11

17

14

19

12

18

13

20

13

16

14

22

11

18

15

21

12

17

13

23

18

13

22

10

18

14

21

11

17

13

22

12

23

13

23

12
9

a.Graphthemeasurements.
b.Describehowlevel,trend,andsloperelatetothegraph.
c.Formaconclusionaboutthestudy.

Page125

PARTIII
UNDERSTANDINGTHESELECTIONANDUSEOFSTATISTICS

Page127

Chapter9
MeasurementandDescriptiveStatistics
Measurement
Stevens'ScalesofMeasurement
NominalScalesorVariables
OrdinalScalesorVariables(UnequalIntervalScales)
IntervalandRatioScalesorVariables
DistinguishingbetweenOrdinalandIntervalScales
Kraemer'sLevelsofMeasurement
DichotomousVariables
NominalVariables
OrdinalVariables
NormallyDistributedVariables
ConfusionaboutTerms
TheNormalCurve
PropertiesoftheNormalCurve
AreasundertheNormalCurve
TheStandardNormalCurve
DescriptiveStatistics
TheImportanceofLevelsofMeasurementforDescriptiveStatistics
MeasuresofCentralTendency
CalculationoftheMean
MeasuresofVariability
CalculationoftheStandardDeviation
ConclusionsaboutMeasurementandtheUseofStatistics
Summary

Page128

StudyAids
Concepts
Distinctions
ApplicationProblems
Measurement
AccordingtoS.S.Stevens(1951),''Initsbroadestsensemeasurementistheassignmentofnumeralstoobjectsoreventsaccordingtorules"(p.1).Aswehaveseen
inchapter4,theprocessofresearchbeginswithaproblemthatismadeupofaquestionabouttherelationshipbetweentwoormorevariables.Measurementis
introducedwhenthesevariablesareoperationallydefinedbycertainrulesthatdeterminehowtheparticipants'responsesaretranslatedintonumerals.Thesedefinitions
canincludeunorderedcategoriesinwhichthenumeralsdonotindicateagreaterorlesserdegreeofthecharacteristicorthevariable.Stevenswentontodescribefour
scalesofmeasurementthathelabeled:nominal,ordinal,interval,andratio.Thesetypesofmeasurementsvaryfromtheunordered(nominal)tothehighestlevel(ratio).
However,becausenoneofthestatisticsthatarecommonlyusedinsocialsciencesoreducationrequiretheuseofratioscales,wewillnotdiscussthemextensively.
Furthermore,thiscategorizationisnotthemostusefulonefordeterminingwhichstatisticsareappropriate,sowepresentanalternativecategorizationproposedby
HelenaChumuraKraemer(personalcommunication,March16,1999).
BeforepresentingKraemer'slevelsofmeasurement,itisimportanttoatleastsummarizewhatS.S.Stevens(1951)meantbynormal,ordinal,interval,andratio
scalesbecausethesetermsareseenintheliterature.However,firstwewouldliketoanticipatesomecommonconfusions.Recall,thebriefdiscussion

Page129

inchapter4onorderedversusunorderedvariablesandcategoricalversuscontinuousvariables.Thesetermsandthoseforthescale(nominal,ordinal,interval)are
sometimesusedinconsistently.Figure9.1shouldbehelpfulinclarifyingthesesomewhatoverlappingandconfusingterms.
Stevens'ScalesofMeasurement
NominalScalesorVariables
Thesearethemostbasicorprimitiveformsofscalesinwhichthenumeralsassignedtoeachcategorystandforthenameofthecategory,buthavenoimpliedorderor
value.Forexample,inastudymalesmaybeassignedthenumeral1andfemalesmaybecodedas2.Thisdoesnotimplythatfemalesarehigherthanmalesorthat
twomalesequalafemaleoranyoftheothertypicalmathematicalusesofthenumerals.Thesamereasoningappliestomanyothertruecategoriessuchasethnic
groups,typeofdisability,orsectionnumberinaclassschedule.Ineachofthesecasesthecategoriesaredistinctandnonoverlapping,butnotorderedthuseach
categoryinthevariableethnicgroupisdifferentfromtheother,butthereisnonecessaryordertothecategories.Thus,thecategoriescouldbenumbered1forAsian
American,2forLatinAmerican,3forAfricanAmerican,and4forEuropeanAmericanorthereverse,oranycombinationofassigninganumbertoeachcategory.
Whatthisimpliesisthatyoumustnottreatthenumbersthatareusedforidentifyingthecategoriesinanominalscaleasiftheywerenumbersthatcouldbeusedina
formula,addedtogether,subtractedfromoneanother,orusedtocomputeanaverage.Averageethnicgroupmakesnosense.However,ifoneasksacomputertodo
averageethnicgroup,itwilldosoandgiveyoumeaninglessinformation.Theimportantthingaboutnominalscalesistohave

Fig.9.1.
Definitionsandcorrespondenceofseveralmeasurementterms.

Page130

clearlydefined,nonoverlappingormutuallyexclusivecategoriesthatcanbecodedreliablybyobserversorbyselfreport.
Itshouldbepointedoutthatqualitativeorconstructivistresearchersrelyheavily,ifnotexclusively,onnominalscalesandontheprocessofdevelopingappropriate
codesorcategoriesforbehaviors,words,andsoforth.Qualitativecodingmayseemdifferentbecauseoftenitismuchmoredetailedandsophisticated,andbecauseit
isunusualtoassignnumeralstothevariouscategories.Althoughusingqualitativeornominalscalesdoesdramaticallyreducethetypesofstatisticsthatcanbeused
withyourdata,itdoesnotaltogethereliminatetheuseofstatisticstosummarizeyourdataandmakeinferences.Therefore,evenwhenthedataarenominalor
qualitativecategories,one'sresearchmaybenefitfromtheuseofappropriatestatistics.Wereturnshortlytoadiscussionaboutthetypesofstatistics,bothdescriptive
andinferential,thatareappropriatefornominaldata.
OrdinalScalesorVariables(UnequalIntervalScales)
Inordinalscalestherearenotonlymutuallyexclusivecategoriesasinnominalscales,butthecategoriesareorderedfromlowtohighinmuchthesamewaythatone
wouldranktheorderinwhichhorsesfinishedarace(i.e.,first,second,third,...last).Thus,inanordinalscaleoneknowswhichparticipantishighestormost
preferredonadimension,buttheintervalsbetweenthevariousranksarenotequal.Forexample,thesecondplacehorsemayfinishfarbehindthewinnerbutonlya
fractionofasecondinfrontofthethirdplacefinisher.Thus,inthiscasethereareunequalintervalsbetweenfirst,second,andthirdplacewithaverysmallinterval
betweensecondandthirdandamuchlargeronebetweenfirstandsecond.
IntervalandRatioScalesorVariables
Intervalscalesnotonlyhavemutuallyexclusivecategoriesthatareorderedfromlowtohigh,butthecategoriesareequallyspacedthatis,theyhaveequalintervals
betweenthem.Mostphysicalmeasurements(length,weight,money,etc.)haveequalintervalsbetweenthecategoriesand,infact,arecalledratioscalesbecausethey
have,inaddition,atruezero,whichmeansintheaboveexamples,nolength,noweight,ornomoney.Fewpsychologicalscaleshavethispropertyofatruezeroand
thuseveniftheyareverywellconstructedequalintervalscales,itisnotpossibletosaythatonehasnointelligenceornoextroversionornoattitudeofacertaintype.
Thedifferencesbetweenintervalandratioscalesarenotimportantforus,becausewecandoallofthetypesofstatisticsthatwehaveavailablewithintervaldata.As
longasthescalehasequalintervals,itisnotnecessarytohaveatruezero.
DistinguishingbetweenOrdinalandIntervalScales
Itisusuallyfairlyeasytotellwhethercategoriesareorderedornot,sostudentsandresearcherscandistinguishbetweennominalandordinaldata.Thatisgood
becausethisdistinctionmakesalotofdifferenceforwhichstatisticsareappropriate,asweshallsee.However,itisconsiderablyhardertodistinguishbetweenordinal
andintervaldata.Whilealmostallphysicalmeasurementsprovideeitherratio

Page131

orintervaldata,thesituationislessclearforpsychologicalmeasurements.Whenwecometothemeasurementofpsychologicalcharacteristicssuchasattitudes,often
wecannotbecertainaboutwhethertheintervalsbetweentheorderedcategoriesareequal,asrequiredforanintervallevelscale.Supposewehavea5pointscaleon
whichwearetorateourattitudeaboutacertainstatementfromstronglyagreeas5tostronglydisagreeas1.Theissueiswhethertheintervalsbetweenaratingof1
and2,or2and3,or3and4,or4and5areallequalornot.Onecouldargue,andweoftendo,thatbecausethenumbersareequallyspacedonthepageand
becausetheyareequallyspacedintermsoftheirnumericalvalues,theparticipantswillviewthemasequalintervals,andthustheywillhavepsychologicallyequal
intervals.However,especiallyiftheinbetweenpointsareidentified(e.g.,stronglyagree,agree,neutral,disagree,andstronglydisagree),itcouldbearguedthatthe
differencebetweenstronglyagreeandagreeisnotthesameasbetweenagreeandneutralthiscontentionwouldbehardtodisprove.
Somequestionnaireorsurveyitemshaveresponsesthatareclearlynotequalintervals.Forexample,let'stakethecasewheretheparticipantsareaskedtoidentify
theirageasoneoffivecategories:lessthan21,21to30,31to40,41to50,and51andabove.Itshouldbeclearthatthefirstandlastcategoriesaremuchlargerfor
numberofyearscoveredthanthethreemiddlecategories.Thus,theageintervalswouldnotbeequal.Anotherexampleofanorderedscalethatisclearlynotinterval,
wouldbeonethataskedhowfrequentlyparticipantsdosomething.Theanswersgosomethinglikethis:everyday,onceaweek,onceamonth,onceayear,andonce
every5years.Youcanseethatthecategoriesbecomewiderandwider,andthereforearenotequalintervals.Thereisclearlymuchmoredifferencebetween1year
and5yearsthanthereisbetween1dayandIweek.
Kraemer'sLevelsofMeasurement
HelenaChumuraKraemer,professorofbiostatisticsattheStanfordUniversityMedicalSchool,proposedthefollowingfourlevelsofmeasurement,whicharesimilar
toStevens',butmoreusefulfordecidingonanappropriatestatistic.
DichotomousVariables
Adichotomousvariable,onewithtwolevelsorcategories(e.g.,YesorNo,PassorFail)issometimesassumedtobenominal.Wecontendthat,althoughsomesuch
dichotomousvariablesareclearlynominal(e.g.,gender)andothersareclearlyordered(e.g.,mathgradeshighandlow),alldichotomousvariablesformaspecial
case(seeTable9.1).Statisticssuchasthemeanorvariancewouldbemeaninglessforathreeormorecategorynominalvariable(e.g.,ethnicgroupormaritalstatus,
asdescribedearlier).However,suchstatisticsdohavemeaningwhenthereareonlytwocategories.Forexample,iftheaveragegenderwas1.55(withmales=1and
females=2),then55%oftheparticipantswerefemales.Furthermore,asweshowinchapter19formultipleregression,dichotomousvariables,called"dummy"
variables,canbeusedasindependentvariablesalongwithothervariablesthatareintervalscale.Thus,itisnotnecessarytodecidewhethera

Page132

dichotomousvariableisnominal,anditcanbetreatedinsomecasesassimilartoanintervalscalevariable.
NominalVariables
Thesevariableshavethreeormoreunorderedcategories.ThislevelisthesameasStevens'nominalscaleofmeasurement,sowewillnotrepeatthatinformation.
OrdinalVariables
Thesevariableshavethreeormoreorderedcategories.Thevaluesofthevariabledonothaveequalspacesbetweenthem,ortheresponsesarenotnormally
distributed,orboth.Again,thislevelissimilartoStevens'ordinalscaleofmeasurement,sowewillnotrepeatthatinformation.
NormallyDistributedVariables
Thesevariablesnotonlyhavemutuallyexclusivecategoriesthatareorderedfromlowtohigh,butalsotheresponsesorscoresareatleastapproximatelynormally
distributedinthepopulationsampled.Normallydistributedisanassumptionofmanyparametricinferentialstatistics(seechap.13),andisalsoimportantforthe
appropriateuseofanumberofthemostcommondescriptivestatisticsdiscussedinthischapter(e.g.,meanandstandarddeviation).
Table9.1showsthisinformationintabularform.
ConfusionaboutTerms
Unfortunately,theliteratureisfullofconfusingtermstodescribethemeasurementaspectsofvariables.Categoricalanddiscretearesometimesusedinterchangeably
withnominal,butwethinkthatnominalisbetterbecauseitispossibletohaveordered,discretecategories.Continuous,dimensional,andquantitativeareterms
thatyouwillseeintheliteratureforvariablesthatvaryfromlowtohigh,andareassumedtobenormallydistributed.
MostoftheprecedinginformationissummarizedinthetoppartofTable9.2.Thisshouldprovideagoodreviewoftheconceptoflevelsofmeasurementofavariable.
ThebottomsectionofTable9.2providesadditionalinformationandexamplesabouttheappropriateuseofvariouskindsofdescriptivestatisticsfordichotomous,
nominal,ordinal,ornormaldata.Weshouldpointoutherethatitisalwaysimportanttoknowthelevelofmeasurementofthedependentvariableinastudy.Also,
whentheindependentvariableisanattribute,weshouldmakeajudgmentaboutthelevelmeasurement.Usuallywithanac
TABLE9.1
DescriptionsofKraemer'sLevelsofMeasurement
Level

Description

Dichotomous

=2categorieseitherunorderedorordered

Nominal

=3ormoreunorderedornominalcategories

Ordinal

=3ormoreorderedcategories,butclearlyunequalintervalsbetweencategoriesor
ranks

Normal

=3ormoreorderedcategories.Scoresonthistypeofvariableareatleast
approximatelyanormal(bellshaped)distribution

Page133

tiveindependentvariable,theassumptionisthatthecategoriesoftheindependentvariablearenominalcategories,butincertaincases(drugdosages,forexample),
evenanactiveindependentvariablecouldbemeasuredonanintervallevelscalesuchasnodrug,10mg,20mg,and30mg.
Anexamplethatillustratesthreelevelsofmeasurementandmaybehelpfulisoneofanafternoonatthehorseraces.Thenumberswornbythejockeysrepresenta
nominalscalebecausethenumbersidentifythenamesofthejockeys,butthenumbersdonottellusanythingelseaboutthejockeys,exceptperhapstheirpoleposition
foracertainrace.Thebettingisbasedonanordinalscale:whethertheselectedhorsecomesinfirst,second,orthird(i.e.,win,place,orshow).Itdoesnotmatterif
thehorsewinsbyanoseorby10lengths,awinisawin.Thus,theseranksformanordinalscalethatdoesnotnecessarilyhave
TABLE9.2
SelectionofAppropriateDescriptiveStatisticsforOneDependentVariable

LevelorScaleofMeasurementofVariable

Nominal

Ordinal

Normal

Characteristicsofthe
variable

Qualitativedata

Quantitativedata

Quantitativedata

Notordered

Ordereddata

Ordereddata

Truecategories:
onlynames,labels

Rankorderonly

Normallydistributed

Examples

Gender,school,curriculum
type,haircolor

1st,2nd,3rdplace, Height,goodtest
rankedpreferences
scores,goodrating
scales

Frequencydistribution

Redhead

37%

Best

33%

9%

Blond

44%

Better

37%

18%

Brunette

22%

Good

25%

27%

27%

1
Yes

22%

Frequencypolygon

Noa

OK

Bargraphorchart

Yes

Yes

OK

No

Yes

Yes

Boxandwhiskersplot
CentralTendency

Mean

No

No

Yes

Median

No

Yes

OK

Mode

Yes

OK

OK

Standarddeviation

No

No

Yes

Interquartilerange

No

Yes

OK

Howmany
categories

Yes

OK

OK

Variability

Nomeansnotappropriateatthislevelofmeasurement.
OKmeansnotthebestchoiceatthislevelofmeasurement.
c
Yesmeansagoodchoicewiththislevelofmeasurement.
a

Page134

equalintervals.However,themoneyyoureceive,ifyoupickedawinner,isnormallydistributed.
TheNormalCurve
Figure9.2isanexampleofanormalcurve.Manyofthedependentvariablesusedinthebehavioralsciencesthatarescaledattheintervalorratiolevelfitanormal
curve.Examplesofsuchvariablesthatfitanormalcurveareheight,weight,andmanypsychologicalvariables.Noticethatforeachoftheseexamples,mostpeople
wouldfalltowardthemiddleofthecurve,withfewerpeopleattheextremes.IftheaverageheightofmenintheUnitedStateswere5ft101n.,thenthismeasurewould
beinthemiddleofthecurve.Theheightsofmenwhoaretallerthan5ftl0in.wouldbetotherightofthemiddleonthecurve,andthoseofmenwhoareshorterthan
5ft10in.wouldbetotheleftofthemiddleonthecurve.
Thenormalcurvecanbethoughtofasderivedfromafrequencydistribution.Itistheoreticallyformedfromcountingan"infinite"numberofoccurrencesofavariable.
Usuallywhenthenormalcurveisdepicted,onlytheXaxis(abscissa)isshown.Todeterminehowafrequencydistributionisobtained,youcouldtakeafaircoin,and
flipit10times,andrecordthenumberofheadsonthisfirstsetortrial.Thenflipitanother10timesandrecordthenumberofheads.Ifyouhadnothingbettertodo,
youcoulddo100trials.(Wehopeyouhavesomethingbettertodowithyourtime.)Afterperformingthistask,youcouldplotthenumberoftimes,outofeachtrialof
10thatthecointurnedupheads.Whatwouldyouexpect?Ofcourse,thelargestnumberoftrialsprobablywouldshow5headsoutof10.Therewouldbeveryfew,
ifanytrials,where

Fig.9.2.
FrequencydistributionandprobabilitydistributionfortheNormalcurve.

Page135

0,1,9,or10headsoccur.Itcouldhappen,buttheprobabilityislow,whichbringsustoaprobabilitydistribution.Ifweperformedthisexperiment100times,or
1,000times,or1,000,000times,thefrequencydistribution"fillsin"andlooksmoreandmorelikeanormalcurve.
Thenormalcurveisalsoaprobabilitydistribution.Visualizethattheareaunderthenormalcurveisequalto1.0.Therefore,portionsofthiscurvecouldbeexpressed
asfractionsof1.0.Forexample,ifweassumethat5ft10in.istheaverageheightofmenintheUnitedStates,thentheprobabilityofamanbeing5ft10in.ortaller
is.5.Theprobabilityofamanbeingover6ft5in.orlessthan5ft5in.isconsiderablysmaller.Itisimportanttobeabletoconceptualizethenormalcurveasa
probabilitydistributionbecausestatisticalconventionsetsacceptableprobabilitylevelsforrejectingthenullhypothesisat.05or.01.Asweshallsee,wheneventsor
outcomeshappeninfrequently,thatis,only5timesin100or1timein100(wayoutintheleftorrighttailofthecurve),wewonderiftheybelongtothatdistributionor
perhapstoadifferentdistribution.Wereturntothispointnumeroustimeslaterinthebook.
PropertiesoftheNormalCurve
Thenormalcurvehasfivepropertiesthatarealwayspresent.
1.Thenormalcurveisunimodal.Ithasone"hump,"andthishumpisinthemiddleofthedistribution.Themostfrequentvalueisinthemiddle.
2.Themean,median,andmodeareequal.
3.Thecurveissymmetric.Ifyoufoldedthenormalcurveinhalf,therightsidewouldfitperfectlywiththeleftsidethatis,itisnotskewed.
4.Therangeisinfinite.ThismeansthattheextremesapproachbutnevertouchtheXaxis.
5.Thecurveisneithertoopeakednortooflatanditstailsareneithertooshortnortoolongithasnokurtosis.ItsproportionsarelikethoseinFig.9.2.
AreasundertheNormalCurve
Allnormalcurves,regardlessofwhethertheyarenarroworspreadout,canbedividedintoareasorunitsintermsofthestandarddeviation.Approximately34%of
theareaunderthenormalcurveisbetweenthemeanandonestandarddeviationaboveorbelowthemean.Ifweincludeboththeareatotherightandtotheleftof
themean,68%oftheareaunderthenormalcurveiswithinonestandarddeviationfromthemean.Another13.5%oftheareaunderthenormalcurveisaccountedfor
byaddingasecondstandarddeviationtothefirststandarddeviation.Inotherwords,twostandarddeviationstotherightofthemeanaccountsforanareaof
approximately47.5%,andtwostandarddeviationstotheleftandrightofthemeanmakeupanareaofapproximately95%ofthenormalcurve.Ifwewereto
subtract95%from100%,theremaining5%relatestothateverpresentprobabilityorpvalueof.05neededforstatisticalsignificance.Valuesnotfallingwithintwo
standarddeviationsofthemeanareseenasrelativelyrareevents.

Page136

TheStandardNormalCurve
Allnormalcurvescanbeconvertedintostandardnormalcurvesbysettingthemeanequaltozeroandthestandarddeviationequaltoone.Becauseallnormalcurves
havethesameproportionofthecurvewithinonestandarddeviation,twostandarddeviations,ormoreofthemean,thisconversionallowscomparisonsamongnormal
curveswithdifferentmeansandstandarddeviations.Figure9.2,ofthenormaldistribution,hasthestandardnormaldistributionunitsunderneath.Theseunitsare
referredtoaszscores.Ifyouexaminethenormalcurvetableinanystatisticsbook,youcanfindtheareasunderthecurveforonestandarddeviation(z=1),two
standarddeviations(z=2),andsoforth.Anynormaldistributioncanbeconvertedintoastandardnormaldistribution.
DescriptiveStatistics
TheImportanceofLevelsofMeasurementforDescriptiveStatistics
ThebottomofTable9.2alsoillustrateswhetherandhowanumberoffamiliardescriptivestatisticswouldbeusedifthedata(i.e.,dependentvariable)werenominal,
ordinal,ornormallydistributed.Frequencydistributionswouldlooksimilarinallthreecasestheonlydifferenceisthatwithnominaldatatheorderinwhichthe
categoriesarelistedisarbitrary.Thus,wehavelistedredhead,blond,andthenbrunette.Iftherearethreeredheads,fourblonds,andtwobrunettes,then33%,44%,
and22%areineachcategory.However,redheadscouldbeputafterbrunettesorbetweenblondsandbrunettesbecausethecategoriesarenotordered.Inordinal
datayoucanseethattheorderwouldbeinvariant.Fornormallydistributeddata,thefrequencydistributionwouldlooksimilartothenormalcurveinFig.9.2.
Frequencydistributionsindicatehowmanyparticipantsareineachcategory,whetherthosebeorderedorunorderedcategories.Ifonewantstomakeadiagramofa
frequencydistributionthereareseveralchoices:threeofthemarefrequencypolygons,abarchart,andboxandwhiskerplots.AsshowninTable9.2andinFig.9.3,
afrequencypolygonorhistogram,whichconnectsthepointsbetweenthecategories,isbestusedwithnormaldata.Frequencypolygonsshouldnotbeusedwith
nominaldata,becausethereisnonecessaryorderingofthepoints.Thusitisbettertomakeabargraphorchartofthefrequencydistributionofvariableslikegender,
ethnicgroup,schoolcurriculum,orothernominalvariablesbecausethepointsthathappentobeadjacentinyourfrequencydistributionarenotbynecessityadjacent
(seeFig.9.4).
Forordinalandnormaldata,theboxandwhiskerplotisuseful(seeFig.9.5)itshouldnotbeusedwithnominaldata,becausewithnominaldatathereisnonecessary
orderingoftheresponsecategories.Theboxandwhiskerplotisagraphicalrepresentationofthedistributionofscoresandishelpful,asindicatedinthefollowing
section,indistinguishingbetweenordinalandnormallydistributeddata(seeTukey,1977).

Page137

Fig.9.3.
Samplefrequencypolygonsfornormalleveldata.

Fig.9.4.
Samplefrequencydistributionbarchartforthenominalscaleofhaircolor.

Fig.9.5.
Aboxandwhiskerplotforordinalandnormaldata.

Page138

MeasuresofCentralTendency
Threemainmeasuresofthecenterofadistributionareavailable:mean,median,andmode.AsyoucanseefromTable9.2,anyofthemcanbeusedwithnormaldata,
whereaswithordinaldata,themeanisusuallynotappropriate.Themeanorarithmeticaveragetakesintoaccountalloftheavailableinformationincomputingthe
centraltendencyofafrequencydistributionthus,itisusuallythestatisticofchoice,assumingthatonehasnormaldata.Themedianormiddlescoreisanappropriate
measureofcentraltendencyforordinalleveldata.Themedianmaybeabettermeasureofcentraltendencythanthemeanundercertaincircumstances,namely,when
thefrequencydistributionisskewedmarkedlytooneside.Forexample,themedianincomeof100midlevelworkersandamillionaireissubstantiallylowerand
reflectsthecentraltendencyofthegroupbetterthantheaverageincome,whichwouldbeinflatedinthisexample,andforthecountryasawholebyafewpeoplewho
makeverylargeamountsofmoney.Fornormallydistributeddata,themedianisinthecenteroftheboxandwhiskerplot.NoticethatinFig.9.5themedianisnotin
thecenterofthebox.Finally,themode,ormostcommoncategory,canbeusedwithanykindofdatabutgenerallyprovidesyouwiththeleastpreciseinformation
aboutcentraltendency.Onewouldusethemodeasthemeasureofcentraltendencyifthereisonlyonemode,itisclearlyidentified,andyouwantaquick
noncalculatedmeasure.
Wedemonstratethecalculationofthemeanhere,becauseofitscommonuseinbothdescriptiveandinferentialstatistics.Theformulaforthepopulationmeancalled
isthesameastheformulaforthesamplemeancalled( ).Tocalculatethemeanoraverageofasetofnumbers,weaddorsum( )thenumbers(X),andthendivide
bythenumberofentries(N).WeusecapitalXtorefertoindividualscores(oftenreferredtoasrawscores).
Theformulaforthesamplemean(*)isasfollows:

*Where:X=individualorrawscoresinthesample,and
N=theumberofscoresinthesample.
CalculationoftheMean
Supposethat10peopletookatestthatwewillcallFormB.Wewanttoknowthemeanofthe10scores.ThescoresareshowninTable9.3.
WecalculatethemeanbyusingEquation1:

Page139
TABLE9.3
ScoresforTenParticipantsonFormB
Participant

FormB

75

62

93

66

77

63

54

43

82

10

85

MeasuresofVariability
Variabilitytellsusaboutthespreadordispersionofthescores.Intheextreme,ifallofthescoresinadistributionarethesame,thereisnovariability.Iftheyareall
differentandwidelyspacedapart,thevariabilitywillbehigh.YoucanseefromTable9.2thatthestandarddeviation,themostcommonmeasureofvariability,isonly
appropriatewhenonehasnormallydistributeddata.
Forordinaldata,theinterquartilerange,seenintheboxplot(Fig.9.5)bythedistancebetweenthetopandbottomofthebox,isthebestmeasureofvariability.
Notethatthewhiskersindicatetheexpectedrangeandscoresoutsidethatrangeareshownasoutliers.Thepresenceofoutliersisawarningofnonnormalityor
errorswiththedata.Withnominaldataonewouldneedtoaskhowmanydifferentcategoriesthereareandwhatarethepercentagesineach.
CalculationoftheStandardDeviation
Becauseitiscommoninbothdescriptiveandinferentialstatistics,weshowhowtocalculatethestandarddeviation.
Theformulaforthesamplestandarddeviation(s)isasfollows:

Inthisformulawearegoingtosumthesquaresofnumbers,butnoticethatthenumbersthatwesquareandaddarenotrawscores(largeXs),butdeviationscores
(themeansubtractedfromeachrawscore),becausethestandarddeviationisameasureofhowscoresvaryaboutthemean.Wewilldoanexamplecalculationofthe
standarddeviationbyusingthesamesampleofscoresthatweusedforthemeanintheprevioussection.
WestartbycreatingTable9.4fromthescoresonFormB.TheprocedureisasfollowsasseeninTable9.4.Wesubtractthemean,70,fromeachoftherawscores.

Page140
TABLE9.4
CalculationoftheStandardDeviation
x

x2

70

25

70

64

93

70

23

529

66

70

16

77

70

49

63

70

49

54

70

16

256

43

70

27

729

82

70

12

144

10

85

70

15

Participant

X(FormBScore)

75

62

Total

225
2,166

Thesedeviationscorescanbeseenunderthecolumnx.Ifweaddedthesescores,thetotalwouldbezero.(Thesumofthedeviationsaroundthemeanalwaysequals
zero.Thispropertyofthemeanbecomesimportantwhenweconsiderdegreesoffreedomofthettestinchapter14.)Nextwesquareeachofthedeviationscoresin
columnx,whichgivesusthescoresincolumnx 2.Thenweaddthescoresincolumnx 2,whichyieldsthenumber2,166.Wedividethisnumberbythenumberof
participantsminusoneandtakethesquareroottoarriveatthestandarddeviation.
Wecompletethecalculationofthestandarddeviationasfollows:

Thecalculationofthestandarddeviationisconceptuallyeasytofollow.Oneproblemisthatthesamplemeanisnotusuallyanevennumber,makingsubtraction
tediousandpossiblyleadingtocalculationerrors.Therawscoreformulalooksworse,butiseasiertoaccomplishbecausewearedealingwiththeoriginalnumbersor
scores.
Therawscoreformulaforthesamplestandarddeviationisasfollows:

Page141

ConclusionsaboutMeasurementandtheUseofStatistics
First,researchersshouldtrytomeasureeachconstructonthehighestlevelconsistentwithreliabilityandvalidity.Theaccuracyofestimationandthepoweroftests
dependsstronglyonhowsensitivethemeasurementistoindividualdifferencesamongparticipantsandonaccesstothemorepowerfulparametricmethods.Thus,
giventhechoice,onewouldhopetoobtainsounderresultsbyusingnormalleveldatathanbyusingordinalleveldata,andordinalmeasurementratherthannominal
measurement.However,sacrificingreliabilityorvaliditytohaveahigherlevelofmeasurementmeansthatyouwillhavegreateraccuracyorpowertogetthewrong
answers!
Second,researchersalsoshouldusethemostpowerfulmethodsconsistentwiththenatureoftheirdata.Thus,aswediscussinchapters13and14,independent
samplesttests,whicharebasedonthemeansandstandarddeviation,arevalidfornormaldatawithequalvariancesinthetwogroupsbeingcompared.Typically,
theyarethemostpowerfultestsavailabletocomparetwogroups.However,ifthedataareorderedbutgrosslynonnormal,ornormalbutwithgrosslyunequal
variances,thetwosamplettestmaynotgivetherightanswers.ThentheMannWhitney(MW)nonparametrictestwouldbepreferred.Thesacrificeinpoweris
relativelyminor.However,theMWtestrequiresordinaldata.So,ifthedatawerenominalordichotomous,onewoulduseinsteadthechisquarecontingencytest.
Thentherewouldbeamajorsacrificeinpower.Thus,aresearcherwhochoosestodichotomizenormaldataandusethechisquare(oftendone)isbeingvery
wasteful.Aresearcherwhousesthettestfornominaldata(seldomdone)wouldproducemisleadingresults.
Summary
Thechapterprovidesanoverviewoflevelsofmeasurementandhowtheyinfluencetheappropriateuseofstatistics.Wedividevariablesintofourlevels:dichotomous,
nominal,ordinal,andnormallydistributed.Dichotomousvariableshaveonlytwolevelsnominalhavethreeormoreunorderedlevels.Ordinalvariableshavethreeor
moreorderedlevels,but,likeranks,theintervalsbetweenlevelsarenotequal.Finally,normalvariableshaveatleastapproximatelyanormal(bellshaped)frequency
distribution.Propertiesofthenormalcurvearediscussedbecausemanystatisticsassumethatresponsesarenormallydistributed,andmanybehavioralvariablesare
distributedatleastapproximatelynormally.Threewaysofplottingfrequencydistributions(frequencypolygons,boxandwhiskerplots,andbarcharts)aredescribed
andillustrated.Similarly,thethreemeasuresofcentraltendency(mean,median,andmode)aredescribed,asarethreemeasuresofvariability(standarddeviation,
interquartilerange,andnumberofcategories).

Page142

StudyAids
Concepts
Boxandwhiskerplot
Interquartilerange
Normaldistributionorcurve
Outlier
Skewness
Standarddeviation
Standardnormalcurve
Distinctions
Dichotomousversusnominalversusordinalversusnormallevelsofmeasurement
Meanversusmedianversusmode
Nominalversusordinalversusintervalscalesofmeasurement
Orderedversusunorderedcategoriesorlevelsofavariable
ApplicationProblems
1.NameanddescribeStevens'fourlevelsofmeasurementprovideexamples.
2.HowareKraemer'slevelsofmeasurementsimilaranddifferentfromStevens'?
3.Whyisitimportanttoknowordeterminethelevelofmeasurementforyourdata?
4.Whichmeasuresofcentraltendencyareappropriatetousewithdataateachlevelorscaleofmeasurement?
5.Fortheexamplesbelow,statethelevelorscaleofmeasurement:
a.Urban,suburban,rural
b.Young,middleaged,old
c.15,16,17,18,19,20,21...years
d.Stronglyagree,agree,neutral,disagree,stronglydisagree
6.Bothdichotomousvariablesandnominalvariablesarecategorical.Whyisitinformativetocalculatethemeanwithadichotomousvariable,butnotwithanominal
variable?
7.Howdoesthenormalcurvedifferfromthestandardnormalcurve?
8.Thefollowingscoreswererecordedfromstudentsinastatisticsclass.Determinethemean,median,mode,andstandarddeviation.89,93,81,93,73,93,85,89,
75,85,90,70
9.Astudentinalargeundergraduateclass(approximately500)scoresonestandarddeviationabovethemeanonherfirstmidterm.Herscoreishigherthanwhat
percentageoftheclass?
10.Anotherstudentinthissameclassscorestwostandarddeviationsbelowthemean.Whatpercentageofstudentshavehigherscores?

Page143

Chapter10
SamplingandExternalValidity
WhatIsSampling?
AdvantagesofSampling
KeyConcepts
Participants,Cases,orElements
TheoreticalorTargetPopulation
AccessiblePopulation
SelectedSample
ActualSample
SamplingDesign
Strata
Clusters
RepresentativeSample
StepsinSelectingaSampleandGeneralizingResults
TypesofSampling
ProbabilitySampling
SimpleRandomSampling
SystematicRandomSampling
StratifiedRandomSampling
StratifiedSamplingwithDifferentialProbabilitiesofSelection
Cluster(Random)Sampling
ConcludingCommentsonProbabilitySampling
NonprobabilitySampling
QuotaSampling
PurposiveSampling
ConvenienceSampling
SnowballSampling

Page144

WhyareNonprobabilitySamplesUsedSoFrequently?
AspectsofaStudythatLeadtoanUnrepresentativeSample
HowManyParticipants?
ExternalValidity
EvaluatingExternalValidity
PopulationExternalValidity
EcologicalExternalValidity
SamplingandtheInternalandExternalValidityofaStudy
Summary
StudyAids
Concepts
Distinctions
ApplicationProblems
WhatIsSampling?
Samplingistheprocessofselectingpartofalargergroupofparticipantswiththeintentofgeneralizingfromthesmallergroup,calledthesample,tothepopulation,the
largergroup.Ifwearetomakevalidinferencesaboutthepopulation,wemustselectthesamplesothatitisrepresentativeofthetotalpopulation.
Politicalpollstersandmarketresearchershavedevelopedandrefinedtheprocessofsamplingsothattheyareusuallyabletoestimateaccuratelythevotingor
purchasingintentionsofthepopulationoftheUnitedStatesfromsamplesassmallasafewhundredparticipants.Weareallfamiliarwithpublicopinionandvoting
surveysthatareusuallydonebytelephoneinterviewers,whomayuserandomdigitdialingtechniquestoselectthepersonswhomtheychoosetocall.Ifthequestions
areclearandtheparticipantsanswerthemtruthfullyandaccurately,arandomsampleofapproximately1,000participantsis

Page145

enoughtopredict,withinplusorminus3%,whatthewholepopulationoftheUnitedStateswouldsayorfeelaboutacertainissue.
YoumayalsobefamiliarwiththeNielsentelevisionratingsthatarebasedoninformationgatheredabouttheTVviewingofafewthousandpresumablyrepresentative
households.TheseratingsarethenextrapolatedtoindicatethepresumedpercentageofthetotalTVviewingpopulationoftheUnitedStatesthatwouldhavewatched
acertainshow,andthisdeterminesadvertisingrates.AsimilarNielsensystem,calledScantrack,hasbeendevelopedtoassessthespecificshoppingbehaviorsofa
smallsampleofAmericanconsumers.ConsumersscanthebarcodesontheitemsthattheypurchasesothatNielsencanreporttomanufacturersinformationnotonly
aboutthenumberandtypesofitemspurchased,but,perhapsmoreimportantly,profilesofthepeoplewhoarebuying.
Withafewnotableexceptions,thesemodernsurveytechniqueshaveproventobeusefulandaccurateinpredictingorreportinginformationabouttheattitudesand
behaviorsoftheAmericanpublic.Historically,however,therehavebeenexamplesofmajormiscalculationsthatcanbetracedinparttoinadequatesampling
techniques.Oneoftheoftencitedexamplesisthatofthegrosslyerroneousprediction,byaLiteraryDigestpoll,ofasampleofseveralmillionrespondents,that
FranklinRooseveltwouldlosethe1936presidentialelection,when,infact,hewonbyalandslide.Oneoftheproblemswiththispollwasthatthesamplewasselected
fromautomobileregistrations,telephonedirectories,andotherrelatedsources.Thisledtooversamplingofaffluentandhighereducatedindividualswhowerenot
representativeofthevotingpublic,especiallyduringthemiddleoftheGreatDepression.Inaddition,onlyabout20%oftheselectedsampleactuallyreturnedtheir
questionnaires.Thus,oneneedstoobtainarepresentativesampletodescribeapopulationformarketingorelectionpurposes.
AdvantagesofSampling
Selectinglessthanthetotalpopulationisanadvantageforresearchersforseveralreasons.First,itislessexpensivetointerview,observe,orsendsurveystoasmaller
groupofpeoplethantoaverylargeone.Second,itclearlytakeslesstimetostudyasampleofparticipantsthanitwouldtostudyawholepopulation.Third,better
qualitycontrolcanbeobtainedifonehasareasonableamountoftimetodevotetotheassessmentofeachparticipantratherthantryingtospreadoneselftoothinover
alargergroup.
KeyConcepts
Itisnecessarytodefineafewkeyconceptsrelatedtosamplingtounderstandsamplingasitoccursintheresearchreportedinthesocialscienceandeducation
literature.Figure10.1shouldhelpyouvisualizethesekeyconceptsandtherelationshipsamongthem.
Participants,Cases,orElements
Thesearethepeopleorobjectsoreventsthatareofinterestinaparticularstudy.Inthesocialsciencestheparticipantsareusuallyindividualpeople(childrenor
adults),buttheyalsocouldbe

Page146

Fig.10.1.
Schematicdiagramofthesamplingprocess.

groupsofpeoplesuchasmarriedcouples,siblings,families,teams,schools,andsoon.Theycouldbeanimalssuchaswhiterats,oreventssuchastelevision
programsorcaraccidents.
TheoreticalorTargetPopulation
Thisincludesalloftheparticipantsoftheoreticalinteresttotheresearcherandtowhichheorshewouldliketogeneralize.Examplesoftheoreticalpopulationsareall
thirdgradechildrenintheUnitedStates,allHispanicmalesintheSouthwest,andallwomenover80yearsintheworld.Itisrarelypossibletostudyorevensample
thesetargetortheoreticalpopulationsforbotheconomicandpracticalreasons.Weusuallydonothaveaccesstosuchbroadgroups,especiallyifweareattempting
toobserveormeasuretheminafacetofacesituation.Unfortunately,thetheoreticalpopulationisusuallynotspecifiedinpublishedresearcharticles.Onehastoinfer
itfromthecontextandanygeneralizationsmadeintheresultsanddiscussion.
AccessiblePopulation
Thisissometimescalledthesurveypopulationandfrequentlycalledthesamplingframe.Asthenameimplies,theaccessiblepopulationisthegroupofparticipantsto
whichtheresearcherhasaccess,perhapsthroughatelephoneormembershipdirectory.Theaccessiblepopulation

Page147

mightalsobeanorganizationorgroupsuchasaclasstowhichtheresearcherhasentry.Examplesofaccessiblepopulationsmightbethethirdgradersinaparticular
schoolorschooldistrict,HispanicmenwhobelongtocertainfraternalorganizationsinselectedcitiesintheSouthwest,orcoupleswhoareonamailinglistdeveloped
byaninternationalmarketingfirm.
SelectedSample
Thisisthesmallergroupofparticipantswhoareselectedfromthelargeraccessiblepopulationbytheresearcherandaskedtoparticipateinthestudy.Insomecases,
theaccessiblepopulationmaybesosmall,ordefinedsonarrowly,orboth,thatitisnotnecessarytosampletheparticipants.Rather,everyparticipantinthe
accessiblepopulationisaskedtoparticipateinthestudy.Unlessthisaccessiblepopulationisrepresentativeofthetheoreticalpopulation,thisselectedsampleiscalled
aconveniencesample,eventhoughnoactualsamplingwasdoneatthisstep.
TheselectioncanbeperformedinseveraldifferentwaysasdescribedfollowingunderTypesofSampling.Thepointisthattheselectedsampleiscomposedofthe
participantsthattheresearcherhasselected,butnotallofthissamplenecessarilyparticipatesinthestudy.
Therearemanyreasonsthatparticipantsintheselectedsampledonotendupintheactualdataforthestudy.Somedeclineparticipation,perhaps,bynotreturningthe
researcher'squestionnaireorbytellingthetelephoneinterviewerthattheydonotwishtoparticipate.Othershavemovedorarenotabletobereachedonthe
telephone.Stillothersmaydropoutlaterintheprocess,eitherbyreturningapartiallycompletedquestionnaireorbyprovidinganswersthatarejudgedtobesuspect
or,perhaps,bydroppingoutinthemiddleofthestudy.
ActualSample
Thesearetheparticipantswhocompletethestudyandwhosedataareactuallyusedinthedataanalysisandinthereportofthestudy'sresults.Theratioofthesizeof
theactualsampletotheselectedsampleisknownastheresponserate.Alowresponseratemaylowerthequalityofthesample,orevenmakethestudyinvalid,if
thepersonswhorespondedaredifferentinimportantwaysfromthosewhodidnotrespond.Thus,inevaluatingresearchitisimportanttoknowtheresponserateand
toknowwhethertherespondersweresimilartononresponders.
SamplingDesign
Thisistheprocessbywhichtheselectedsampleischosen.SeveralofthemostoftenusedsamplingdesignsaredescribedlaterunderTypesofSampling,Probability
Sampling,andNonprobabilitySampling.
Strata
Thesearevariables(e.g.,race,geographicalregion,age,orgender)thatcouldbeusedtodividethepopulationintosegments.Theresearcherisusually
knowledgeableaboutthesedimensionsandassumesthattheyareimportantinobtainingarepresentativesample.Thestrataareused,asdescribedfollowing,in
obtainingastratifiedrandomsample.Theyarealsousedinspecifyingthequotasinaquotasample.
Clusters
Sometimescalledsamplingunits,clustersarecollectionsorgroupsofpotentialparticipantsthatdonotoverlap.Theparticipantsinagiven

Page148

clusterareusuallygeographicallygroupedtogether.Clustersincludetowns,schools,andhospitalstheyareimportantforclusterprobabilitysampling,whichis
describedlaterinthechapter.
RepresentativeSample
Byrepresentative,wemeanasamplethatrepresentsthepopulation,thatis,isasmallreplicaofthepopulation.Ithas,onallofthekeyvariables,thesameproportions
asinthewholepopulation.Eachparticipantrepresentsaknownfractionofthetheoreticalpopulationsothatthecharacterizationofthepopulationcanberecreated
fromthesample.Arepresentativesampleismostlikelyobtainedbyusingthetechniquesdescribedbelowasprobabilitysampling.
StepsinSelectingaSampleandGeneralizingResults
Therearemanywaystoselectasamplefromapopulation.Thegoalistohaveanactualsamplethatisrepresentativeofthetargetortheoreticalpopulation.
Obtainingarepresentativesampleisnoteasy,becausethingscangowrongatseveralstagesoftheresearchprocess.Figure10.1showstheconceptsthatwehave
describedandalsothethreesteps(shownwitharrows)fromthetheoreticalpopulationtotheactualsample.
Thefirststepisfromthetheoreticalpopulationtotheaccessiblepopulation.Itmaybethattheaccessiblepopulationorsamplingframeisnotrepresentativeofthe
theoreticalpopulation.Thisisacommonproblembecauseresearchersoftendonothaveaccesstothegeographical,socioeconomic,orotherrangeofparticipantsto
whichtheywouldliketomakeinferencesorgeneralizations.Often,especiallyifweneedtomeasureparticipantsinafacetofacecontact,wearelimitedtoaspecific
locationandtogroupsthatareavailabletoussuchaspersonsinacertainschool,hospital,ororganization.
Thesecondstepinthesamplingprocessiscalledthesamplingdesignorselectionofparticipants.Thisstep,betweentheaccessiblepopulationandtheselected
sample,isthestepthatisusuallydescribedinthemethodssectionofarticlesandisthestepoverwhichtheresearcherhasthemostcontrol.Weexpandonthisstepin
thenextsection,TypesofSampling.
Thethirdsteptakesplacebetweentheselectedsampleandtheactualsample.Theproblemhereisthatparticipantsmaynotrespondtotheinvitationtoparticipate,or
maydropoutofthestudy,orboth,sothattheactualsamplemaybeconsiderablysmallerthantheselectedsamplethatis,thereisalowresponserate.Theactual
samplemaybeunrepresentativeoftheselectedsample.Thisisoftenaproblemwithmailedsurveys,especiallyifthesurveyissenttobusypeoplesuchassmall
businessowners.Inthesecases,lessthan25%ofthequestionnairerecipientsmayreturnthem.Thus,eveniftheselectedsamplewasrepresentativeofthetheoretical
population,theactualsamplemaybeunrepresentative.
TypesofSampling
Therearetwomajortypesofsamplingdesignsorproceduresthatareusedinobtainingtheselectedsample:probabilityandnonprobabilitysamplingtech

Page149

niques.Probabilitysamplinginvolvestheselectionofparticipantsinawaythatisnonbiased.Inaprobabilitysample,everyparticipantorelementofthepopulation
hasaknown,nonzeroprobabilityofbeingchosentobeamemberofthesample(Stuart,1984).Innonprobabilitysampling,thereisnowayofestimatingthe
probabilitythateachparticipanthasofbeingincludedinthesample.Therefore,biasisusuallyintroduced.Nonprobabilitysamplesareusedwhenprobabilitysamples,
whichrelyonrandomorsystematicselectionofparticipants,arenotfeasible.Theadvantagesofnonprobabilitysamplesareeconomyandconvenience.Infact,most
publishedstudiesinthesocialsciencesandeducationusenonprobabilitysampling,ortheentireaccessiblepopulation,ifitissmall.Thesesamplesmaybeusefulin
examiningtherelationshipbetweenvariablesorthedifferencesbetweengroups,buttheyareclearlynotthebestwaytodescribeormakegeneralizationsaboutthe
wholepopulation.
ProbabilitySampling
Whenprobabilitysamplingisused,inferentialstatisticsenableresearcherstoestimatetheextenttowhichresultsfromthesamplearelikelytodifferfromwhatwe
wouldhavefoundbystudyingtheentirepopulation.Fourtypesofprobabilitysamplingaredescribedbriefly.
1
SimpleRandomSampling
Thebestknownandmostbasicoftheprobabilitysamplingtechniquesisthesimplerandomsample,whichcanbedefinedasasampleinwhichallparticipantsor
elementshaveanequalandindependentchanceofbeingincludedinthesample.Ifweput100piecesofpaper(numberedfrom1to100)inahat,shakethehat,and
drawout10withoutreplacingthem,thisprocesswouldapproximateasimplerandomsample.Insuchasample,eachparticipanthasanequalandindependent
chanceofbeingselectedorpickedas1ofthe10personstobeaskedtoparticipateinthestudy.Thistypeofprobabilitysamplewillproducearepresentativesample
ifthenumberofparticipantsselectedisrelativelylarge.However,ifthenumberselectedissmall,likethe10numbersdrawnfromthehat,thesamplemightnotbea
smallreplicaofthetotalpopulation.
Inanactualresearchstudywewoulddraworselectourrandomsamplebyusingarandomnumbertableorcomputergeneratedrandomnumbersratherthanby
selectingnumbersoutofahat.Assumingthatthereare900participantsintheaccessiblepopulation,allofthepossibleparticipantsintheaccessiblepopulationor
samplingframewouldbelistedandnumberedfrom1to900.Then,ifwedecidedtoselectasampleof90,wewouldstartbyunsystematicallypickingastartingpoint
intherandomnumbertableandproceedinginasystematicandplannedmannerdowntherowsoracrossthecolumnstoselectthefirst90nonrepeatednumberslisted
intherandomnumbertable.Table10.1isanexampleofasmallpartofarandomnumbertable.(Completetablescanbefoundinmoststatisticsbooks.)Toselect
numbersfrom001to900,onewouldneedthreedigits,soonecoulduse,forexample,thethreerighthandcolumnsinthesetofrandomnumbers.Let's

Page150

saywestartedbynonsystematicallypickingthenumber11508,whichisabouthalfwaydownthelefthandsetof5digitnumbers.Thethreerighthanddigitsare508,
sothefirstparticipanttobeselectedwouldbenumber508,thesecondparticipantwouldbenumber449,andthethird515.However,wewouldskipnumber986
becausethatnumberisoutsidetherangeof1to900inthesamplingframe.Wewouldcontinuedownthelistbyskippingnumberslargerthan900andanythathad
alreadybeenpickeduntilwehadselected90oftheoriginal900potentialparticipantsforthissimplerandomsample.
Althoughthesimplerandomsampleistheprototypeofaprobabilitysamplingmethod,itisusedrelativelyinfrequently,inpartbecauseitmaybetimeconsumingto
numbertheentirelist,ifitislong.Also,manytimesthereisnolistofthepopulationofinterest.Amorefrequentequivalentofthesimplerandomsampleissystematic
samplingwitharandomstart,whichwillbediscussednext.Touseeithersimplerandomorsystematicsampling,thepopulationhastobefiniteandtherehastobea
listordirectoryofpersonsinthepopulation.
2
SystematicRandomSampling
Toobtainthistypeofsample,westartbyusingtherandomnumbertabletoselectanumberbetween1and10because,asinthepreviousexample,wehavedecided
onaonetenthsampleofthepopulation.Ifwerandomlyselected,forexample,thefourthpersononthelistasthefirstparticipant,thenwewouldsystematicallyselect
everytenthparticipant,startingfromthefourth.Thus,thesamplewouldincludethepersononthelistwhowas4th,14th,24th,34th,andsoon,andwouldinclude90
participants.Someresearchbookswarnagainstsystematicsamplesifalistisordered,especiallyinareoccurringpattern,whichwouldhaveadifferentialeffectonthe
resultingsample,dependingonwhereonestarted.Forexample,supposewehadalistof90youthsoccerteams,eachofwhichhad10players,andtheirgoaliewas
alwaysthefourthpersonlistedforeachteam.Ifweusedtheselectionprocessfromourexample,startingrandomlywithnumberfour,wewouldselectonlygoaliesfor
this90personsample(ornogoaliesiftherandomstarthadbegunatadifferentnumber).Thus,weshouldexaminethelistwiththeinterval(e.g.,10)tobeusedin
mind.However,Fowler(1993)statesthatthisisrarelyaproblemandinal
TABLE10.1
SmallSectionofaRandomNumberTable
55515

81899

04153

79401

46375

81953

etc.

etc.

15792

35101

37824

etc.

11508

37449

46515

30986

63798

Page151

mostallcasesasystematicsamplewitharandomstartwillproducetheequivalentofasimplerandomsample.
3
StratifiedRandomSampling
Ifimportantcharacteristicsoftheaccessiblepopulationorsamplingframe,suchasgenderorrace,areknownaheadoftime(i.e.,arenotedonthesamplingframe),
thenwecanreducethesamplingvariationandincreasethelikelihoodthatthesamplewillberepresentativeofthepopulationbystratifyingthesampleonthebasisof
thesekeyvariables.Inourpreviousexample,supposethatwewantedtobesurethatarepresentativenumberofgoalieswaschosenaspartofthesample.Wewould
useastratifiedrandomsamplingtechnique.Thelistorsamplingframewouldberearrangedsothatallthegoalieswerelistedtogether,andthenonetenthofthem
wouldbeselectedrandomlywitheitherasimplerandomsamplingtechniqueorasystematicsamplingtechniquewitharandomstart.Thesametechniquescouldbe
usedforselectingasamplefromeachoftheotherpositions.Stratifyingensuresthatthesamplecontainsexactlytheproportionofgoalies(onetenth)asintheoverall
population.
Whenparticipantsaregeographicallyspreadacrossthecountryorastate,itiscommontostratifyfromgeographysothatappropriateproportionsoftheselected
samplecomefromthedifferentregionsofthecountryorstate.Itisalsocommontostratifyontherural,suburban,andurbancharacteristicsofthesampleiftheseare
identifiableinthesamplingframe.
Wewouldnowliketodescribetwomorecomplextypesofsampling.Oneisavariantofthestratifiedsamplingprocedurejustdescribed,andthesecondisa
multistagesamplingproceduredesignedtomakesamplinggeographicallydiverseparticipantsmorepractical.
3a
StratifiedSamplingwithDifferentialProbabilitiesofSelection
Sometimesstratifiedsamplingwillleadtooneormoresizablegroupsofparticipantsandoneormoreverysmallgroupsofparticipants.Forexample,ifwewantedto
comparevariousethnicgroups,thenumberofHispanics,AfricanAmericans,andespeciallyAsianAmericansandNativeAmericanswouldbesmallinamoderate
sizedsamplethatwasrepresentativeofthetotalpopulationofthecountry.Ifwewantedtocomparedifferentethnicgroups,itisdesirabletohavethegroupsequalor
atleastofasubstantialsize(maybe30ormore).Therefore,onemightwanttooversampletheminoritygroupmemberstohaveenoughineachgrouptomake
reasonablecomparisonswiththeCaucasianorWhitesample.
Inourexampleofthesoccerteams,thegoalieswouldbesimilartominorityethnicgroupmembersinthat,ifwedidaonetenthsampleofthe90goalies,wewould
endupwithasampleofonly9goalies,whichistoosmallforreasonablecomparisonswiththegroupofnongoalies.Wemight,forexample,wanttosamplehalfofthe
90goaliestogetalargeenoughsampletocompare.
Weshouldtakecautionbecause,ifwedrawconclusionslateraboutatotalpopulationfromthesample,adjustmentforthefactthatthegrouphadbeenoversampled
wouldneedtobeundertaken.Thatis,ifwewereinterestedintheheightofsoccerplayers,wecouldnotjusttaketheaverageoftheheightsoftheoversampled
goaliesandofthenongoalies.Wewouldhaveto

Page152

weightthegoalieslesssothattheoverallaverageheightwouldnotbedistortedbythefactthattherewerefivetimesasmanygoaliesinoursampleaswouldbe
representativeofthepopulation.
4
Cluster(Random)Sampling
Clustersamplingisatwostagesamplingprocedurethatisespeciallyusefulwhenthepopulationisspreadoutgeographically,orthereisnosingleoveralllistof
individualsintheaccessiblepopulation,orboth.Thebasicstrategyistofirstselectspecificclustersorgroupsofparticipantsbyusingaprobabilitysamplingmethod
suchassimplerandomsampling.Thesecondstageistorandomlyselectall,oraproportionof,participantsfromtheclusters.
Take,forexample,thesituationwhereweareinterestedinsamplingonetenthofthestudentsfromalargenumberofschools.Thetaskofgoingtoeachof,say,100
schoolsandselecting1outofevery10studentswouldbedifficultintermsoftimeandexpense.Alessexpensivealternativewouldbetoselectrandomly1outoffive
schools(i.e.,20),andthenrandomlyselecthalfofthestudentsinthoseschoolsastheonetenthsample.Thereare,ofcourse,variouscombinationsoftheproportions
thatonemightbeselectedonthefirststepandonthesecondstep.Forexample,acommonstrategyistorandomlyselectjustenoughgeographicallycompactclusters
(e.g.,schoolsorcommunities)sothatonewillhavetheneedednumberofparticipantsifoneselectedallofthestudentsintheselectedclustersanddidnotselectany
studentsfromtheotherschools(i.e.,select10schoolsandthenallstudentsfromthose10schools).Someprecisioninsamplingissacrificedbecausethesampleof
schools,evenifrandomlyselected,couldbeunrepresentativeofthelargerpopulationofschools,butoftencostconsiderationsoutweighthisminorlossinprecision.
ConcludingCommentsonProbabilitySampling
Figure10.2providesdiagramsofthefiveprobabilitysamplingmethodsthatwejustdescribed.Noticethatalloftheminvolverandomizationatsomepointinthe
processofselectingparticipants.However,theydifferinwhethertheaccessiblepopulationisstratifiedinsomeway(indicatedbyhorizontallines)andinwhat
proportionastratumorclusterisselected.Thenumbersintheboxesrepresentasinglepotentialparticipantandhisorhersubjectnumber.Thesediagramsdonot
includeallofthepossiblecombinationsofthefourmainsamplingstrategies,butgiveagoodideaofthemethodsusedbyresearcherswhoattempttoobtaina
representativesampleandahighpopulationexternalvalidity.Withaprobabilitysample,descriptivestatisticsfromthesamplealsodescribethepopulation.However,
withstratifiedsamplingswithdifferentproportions,onewouldneedtoweighttheresultsappropriatelytodescribethepopulation.
NonprobabilitySampling
Nonprobabilitysamplesarethoseinwhichtheprobabilityofbeingselectedisunknown,oftenbecausethereisnosamplingframeorlistofthemembersofthe
accessiblepopulation.Timeandcostconstraintsalsoleadresearchersandpollsterstousenonprobabilitysamples.Althoughnonprobabilitysamples

Page153

Fig.10.2.
Schematicdiagramsoffiveprobabilitysamplingmethods
forasampleof10fromanaccessiblepopulationof50.

mayappeartobesimilartoprobabilitysamplesinthedemographicsoftheselectedparticipants,theresultscanbedistorted,andtheassumptionsofprobabilitytheory
andsamplingerrordonotapply.Thefirsttypeofnonprobabilitysampling,quotasampling,isoftenusedbypublicopinionpollsters,politicalpollsters,andmarket
researchersbecausetheresultingsampleslookrepresentativeofthepopulationandthecostofobtainingthedataisconsiderablylessthanwouldberequiredtoobtain
aprobabilitysample.

Page154

QuotaSampling
Inquotasampling,theinvestigatorsetscertainparametersfortheinterviewersheorshehashiredtofollow,butsomedegreeoflatitudeordiscretionisallowedinthe
selectionoftheactualparticipants.Forexample,theinterviewermaybedirectedtocertaincityblocksortelephoneexchangesthatmayhavebeenchosenrandomly
orinsomeotherwaytoberepresentative,andthenisaskedtofindandinterviewacertainnumberofparticipantsfromeachblockortelephoneexchange.Itis
possiblethattheremayevenbefurtherrestrictionssuchasobtainingacertainproportionofmenandwomen,youngerandolderparticipants,andsoforth,butthe
actualparticipantsareselectedbytheinterviewerbecausetheyarehomeandwillingtoparticipatewhenasked.Thistechniquesavesmoneyinpartbecause
participantswhoarenotavailablearenotcalledback.Inhousetohousesurveying,theinterviewermaybetemptedtoskipcertainhouseholds,thatis,thosethatare
locatedonupperfloors,thatareindilapidatedcondition,orhaveabarkingdog.
PurposiveSampling
Thisprocedureisusedtoselectasamplefromtheparticipants,orgroupsofparticipantssuchasvotingdistricts,thatarejudgedtobeappropriateorespecially
informativeforthepurposeoftheresearch.Theparticipantsarehandpickedfromtheaccessiblepopulationsothattheypresumablywillberepresentativeortypicalof
thepopulation.Thistechniqueissometimesusedbypoliticalpollsterswhohaveinformationaboutpreviouselectionsthatindicateswhichvotingdistrictsaretypicalin
forecastinghowlargerentities,suchasthestate,willvote.Theinterviewerthenpollspeopleinthatdistricttomakeextrapolationsandgeneralizationsaboutthelarger
unit.
Thistypeofsamplingisespeciallyconfusingtostudentsandnewresearchersbecauseitdoesnotseemtobeclearlydefined.Researchersareoftentemptedtosaythat
theirsampleispurposivebecausetheyselectedthetypesofparticipantsinwhichtheywereinterested.Qualitativeresearchersalsomayclaimthattheirsamplewas
purposivelyselectedasmayfocusgrouporganizers.Sometimesthismaybethecase,butothertimesitappearstobeawayofavoidingacknowledgingthattheir
sampleisreallyaconvenienceoraccidentalsample(tobediscussedfollowing).Wethinkthatapurposivesampleismuchlikeaquotasampleinthatonemustmake
anattempttoselectasamplethatisrepresentativeofthepopulationofinterest.
Takethesoccersampleasanexampleifwewantedtoknowhowplayersevaluatedthefairnessoftheleagueorthequalityoftherefereeing,wemightpurposively
selectplayerswhowereonteamsthatwereaboutaverageintermsofwins,losses,andgoals.Orwemightselectsomeplayerswhodidwellintermsofcompetition,
somewhodidaboutaverage,andsomewhodidpoorly.Asanotherexample,perhapsonewouldaskteacherstoselectchildrentheyfeltwererepresentativeofthe
class.However,unlessmoredetailedinstructionsaregiven,itmightwellturnoutthattheteacherswouldselectchildrenthattheyfeltwouldmakethemortheschool
lookgoodratherthanrepresentativechildren.
Wewouldnotconsideritpurposivesamplingifteacherswereaskedtoidentifychildrenwith,forexample,dyslexia,orifasocialserviceagencywasaskedtoidentify
familycaregiversofAlzheimerpatientswhowereintheir

Page155

files.ThistechniqueissimilartowhatFowler(1993)callsscreeninginthecaseoftelephoneinterviewerswhoexcludehouseholdsthatdonothaveparticipantsofthe
desiredtypes(e.g.,womenconsumersover60yearsold).Thesamplingwouldonlybepurposiveif,giventhelistofdyslexicchildrenorfamilycaregiversof
Alzheimerpatients,weattemptedtopicktypicalorrepresentativecasesfromthoselists.
Asanaside,iftheselectionofparticipantstobecontactedisdonebyusingprobabilitytechniquessuchasrandomdigitdialing,andwescreenoutparticipantswhodo
notfittheselectioncriteria,wewouldstillendupwithaprobabilitysample.However,ifweselectcasesfromtheaccessiblepopulationpurposefully,notrandomlyor
systematically,theresultingsamplewillbeanonprobabilitysample.Likequotasampling,purposivesamplingisanattempttomakethesamplerepresentativeofthe
population,butitwillprobablynotachievethegoalofcompleterepresentativeness.
Apurposivesampleisdifferentfromaconveniencesampleinthatatleastanattempthasbeenmadetoselectparticipantssothattheyarerepresentativeofthat
accessiblepopulation,notjustthosewhoareconvenientandavailable.
ConvenienceSampling
Unfortunately,thisisprobablythemostcommonsamplingmethodusedinstudentprojects,includingthesesanddissertations.Itisalsooftenusedbyresearchersin
experimentallaboratorystudiesandbythosewhodoqualitativeresearch,aswellasbyanyresearcherwithlimitedresourcesavailableforsampling.Asthename
implies,theparticipantsareselectedonthebasisofconvenienceratherthanchoseninaseriousattempttoselectparticipantswhoarerepresentativeofthetheoretical
population.Examplesofconvenienceoraccidentalsamplingaretheuseofstudentsinones'class,theuseofpassersbyatacertainlocation(e.g.,thestudentcenteror
amall),membersofcertainclubs,churchgroups,studentsinaschool,oremployeesofacompanythathappentobewillingtocooperate.Afterthedataarecollected,
researchersoftenexaminethedemographiccharacteristicsoftheirconveniencesampleandconcludethattheparticipantsaresimilartothoseinthelargerpopulation.
Thisdoesnotmeanthatthesampleis,infact,representative,butitdoesindicateanattemptbytheresearcher,atleastafterthefact,tocheckonrepresentativeness.
Conveniencesamplesresultwhenevertheaccessiblepopulationisnotrepresentativeofthetheoreticalpopulation(Step1inFig.10.1),evenifallofthemembersof
theclass,club,orclinicwereassessed.Thesampleisalsooneofconvenienceiftheparticipantsareselected(Step2)inahaphazardwayfromthepopulation(e.g.,
volunteersareused).
SnowballSampling
Snowballsamplingisamodificationofconvenienceoraccidentalsamplingthatisusedwhentheparticipantsofinterestarefromapopulationthatisrareoratleast
whosemembersareunknowntoyou.Thesemightbepersonswithunusualattributes,beliefs,orbehaviorpatternswhodonotbelongtoknowngroupswith
identifiablelistsofmembers,forexample,drugaddicts.Afewparticipantswhomeetthecharacteristicsarefoundandthenaskedforreferencesornamesofother
peopletheymayknowwhofitintothesamecategory.Thentheseotherpeopleareaskedforadditionalreferences,andsoforth,thusthenamesnowballsampling.
Thisisclearlyaconve

Page156

nienceoraccidentalsample.Samplingsimilartothisiscommoninthequalitativeparadigm.
WhyareNonprobabilitySamplesUsedSoFrequently?
Inadditiontothecostandtimeefficiencyadvantagesthatwerementionedearlier,thereareotherreasonsforusingnonprobabilitysamples.First,itmaynotbe
possibletodoaprobabilitysampleoftheparticipants.Thisistrueforstudentresearchersandothersonlimitedbudgetswhocannotaffordthecostsofpostageorof
purchasingacomprehensivemailinglist,orthecostoftraveltointerviewgeographicallydiverseparticipants.
Someresearchers,perhapsespeciallythosewhousecontrolledlaboratoryandexperimentaldesigns,arenotprimarilyinterestedinmakinginferencesaboutthe
populationfromthedescriptivedata,asisthecaseinsurveyresearch.Theseresearchersaremoreinterestedinwhethertheexperimentaltreatmenthasaneffecton
thedependentvariable,andtheyassumethatifthetreatmentispowerful,theeffectwillshowupinmanykindsofparticipants.Infact,theuseofnonhumananimalsin
medicalandbehavioralresearchassumesthatwecangeneralizesomeresults,evenfromotherspecies,tohumans.
Inothertypesofresearch,theinvestigatorisprimarilyinterestedintherelationshipbetweenvariablesandmayassumethattherelationshipwillholdupinawide
varietyofhumanparticipants.Thus,somesay,perhapsinappropriately,thatitisnotnecessarytohavearepresentativesampleofthepopulationtomakegeneralizable
statementsabouttherelationshipbetweentwoormorevariables.Implicitly,manyresearchersimplythatexternalpopulationvalidity,whichisdirectlyrelatedtothe
representativenessofthesample,islessimportantthaninternalvalidity.
AspectsofaStudyThatLeadtoanUnrepresentativeSample
Thefollowingsummarydescribessomeofthethingsthatleadtoasamplethatisunrepresentativeofthetargetpopulation.
1.First,anaccessiblepopulationthatisnotrepresentativeofthetheoreticalpopulation,butispickedforitsconvenience,maybeselected.Withthiskindof
accessiblepopulation,forexampleschoolsinacertaincityinanunrepresentativepartofthecountry,thesamplewouldnotberepresentativeofthetheoretical
populationevenifitwaschosenrandomlyfromthisaccessiblepopulation.
2.Theobviouswaytoobtainanunrepresentativesampleistouseanonprobabilitysamplingdesignormethod.
3.Ifthereisapoorresponserate,therepresentativenessofthesampleislikelytobecompromised.Theresponserateisthenumberofpeopleinterviewedor
respondingdividedbythetotalnumberofpeoplesampled.Thisdenominatorincludesallthepeoplewhowereselectedbutdidnotrespondforavarietyofreasons:
refusals,languageproblems,illness,orlackofavailability.Itdoesnotincludethosewhowerescreenedoutbecausethey

Page157

didnotfittheselectioncharacteristics,didnothaveaworkingtelephone,orwhosequestionnairewasreturnedbecauseitwasnotdeliverable.Theeffectof
nonresponsesontheresultsofthesurveydependsonboththepercentageofpeoplewhoarenotrespondingandtheextenttowhichthosewhodidnotrespondare
biasedinsomeway,thatis,differentfromtherestofthesamplewhodidrespond.
4.Highexperimentalmortalityforcertaingroupsandnototherscanproduceanonrepresentativesample.Forexample,ifanexperimentturnedouttobeunpleasantor
irritatingtomalesbutnottofemales,theremightbeamuchlargerpercentageofmaleswhodropoutduringthestudy,thusleadingtoabiasedsample,evenif
everythinguptothatpointhadbeenbasedonprobabilitysampling.
5.Certainassumptions,suchas''minorityyoutharemorelikelytoengageindeviantbehavior,"canleadtosamplingbiases.Inthiscase,minorityyouthsmaybe
oversampled(seeMacPhee,Kreutzer,&Fritz,1994).
HowManyParticipants?
Oneofthemostoftenaskedquestionsis,"HowmanyparticipantsdoIneedforthisstudy?"Theanswercanbecomplex,butwewilltrytogivesomegeneral
guidelineshere.Onepartoftheanswerdependsonthepeopleyouaskandthedisciplineinwhichtheywork.Nationalopinionsurveysalmostalwayshaveathousand
ormoreparticipants,whilesociologicalandepidemiologicalstudiesusuallyhaveatleastseveralhundredparticipants.Ontheotherhand,psychologicalexperiments
andclinicaltrialsinmedicinewith10to20participantspergrouparecommon,andinsomeclinicalandeducationalareas,singlesubjectdesignsareoftenused.To
anextentthesedramaticdifferencesinsamplesizesdependondifferencesintypesofdesigns,measures,andstatisticalanalyses,buttheyarealsobasedingoodpart
oncustom(Kraemer&Thiemann,1987).
Someauthorssuggestthatthesamplebeaslargeasisfeasiblefortheinvestigatorsandtheirbudget.Otherthingsbeingequal,itistruethatalargersamplewillmore
likelydetectasignificantdifferenceorrelationshipandleadtotherejectionofthenullhypothesis.However,twopointsshouldbemade.
First,representativenessisamoreimportantconsiderationthansamplesize.Ifthesampleisnotrepresentativeofthepopulation,itcanbehugeandstillgive
misleadingresults.Forexample,rememberthatthereweretwoandahalfmillionrespondentstothe1936LiteraryDigestpollthatpredictedthedefeatofPresident
Roosevelt.
Second,verylargesampleswilldetectdifferencesorrelationshipsthatmayhavelittlepracticalorsocietalimportance.Ifwearetryingtodescribeapopulationwitha
statisticsuchasthemeanorpercentage,wewanttobeasaccurateaspossible,andalargesample,ifappropriatelydrawn,willreducethesamplingerror.However,
inmostsocialscienceandeducationalresearch,wearenotinterestedindescribingthepopulation.Rather,wewanttoidentifythekeyfactorsthatmayinfluencethe
dependentvariableorhelpustopredictit.Wehaverelativelylessinterestinfindingfactorsthataccountforverysmallper

Page158

centagesofthevariance.Thus,insomewaysalargesamplecanbedetrimentaltoidentifyingimportantresults.Forexample,with1,000participants,alarge
proportionofcorrelationsprobablywillbestatisticallysignificantlydifferentfromzero,butmanyofthemmayaccountforonly1%or2%ofthevarianceand,thus,
notbeofmuchpracticalimportance.
Thus,thesizeofthesampleshouldbelargeenoughsoonedoesnotfailtodetectsignificantfindingsbecausethesamplewastoosmall,butalargesamplewillnot
necessarilyhelponedistinguishbetweenthemerelystatisticallysignificantandthesocietallyimportantfindings.Thisimportantpointraisestheissueofstatisticalpower
thatwediscussindepthinchapter23.
Fornowwewillonlymentionaruleofthumbthatsomeauthorssuggest:oneshouldincludeaminimumof30participantspermajorgroupinasample.Thusfor
associational(onegroup)designs,onemighthaveasfewas30participants,butforcomparative,quasiexperimental,andexperimentaldesignsoneshouldhave
approximately30ineachgroupthatisbeingcompared.Thelatterpartoftherecommendationseemsexcessivetous.Withastrongtreatmentorlowwithingroup
variability,groupsassmallas10canbecompared.
Inchapter23weaddressthetopicofpowerandhowtocalculateit,whichisthetechnicallycorrectwaytoplanaheadoftimehowmanyparticipantsareneededto
detectaresultofacertaineffectsize.TheKraemerandThiemann(1987)bookprovidesarelativelyeasywaytofindtheneededsamplesize.
ExternalValidity
Inthischapter,weagaindiscussaspectsofresearchvalidity,thevalidityorqualityofawholestudy.Inchapter6,wediscussedinternalvalidity,thevalidityrelated
tothedesignofthestudy.Researchvalidityalsodependsonsampling.Nowwediscussexternalvalidity,anaspectofresearchvaliditythatdependsinpartonthe
qualityofthesample.
ExternalvaliditywasdefinedbyCampbellandStanley(1966)asfollows:"Externalvalidityasksthequestionofgeneralizability:Towhatpopulations,settings,
treatmentvariables,andmeasurementvariablescanthiseffectbegeneralized?"(p.5).Someresearchershaveatendencytojudgeexternalvalidityascontingenton
internalvalidity(discussedinchap.6).Forexample,theymightsuggestthatbecausethestudyhadpoorinternalvalidity,thenexternalvalidityalsomustbepoor.
However,wethinkthatexternalvalidity,likeinternalvalidity,shouldbejudgedseparately,beforethefact,andnotbejudgedonthebasisofinternalvalidity.
EvaluatingExternalValidity
Questionsdealingwiththeexternalvalidityofastudyarebasedontheprinciplethatagoodstudyshouldberatedhighonexternalvalidity,or,ifnot,theauthorshould
atleastbecautiousaboutgeneralizingthefindingstoothermeasures,populations,andsettings.Figure10.3providesscalestorateeachofourtwomainaspectsof
externalvalidity.

Page159

Fig.10.3.
Evaluatingtheexternalvalidityofthefindingsofastudy.

PopulationExternalValidity
Thisfirstaspectofexternalvalidityisaselectionproblemthatinvolveshowparticipantswereselectedtobeinthestudy.Wereparticipantsrandomlyselectedfroma
particularpopulation,orweretheyaconveniencesample?Asdiscussedearlierinthischapter,mostquantitativestudiesinthesocialscienceshavenotusedrandom
selectionofparticipants,andthus,arenothighonpopulationexternalvalidity.However,theissueofpopulationexternalvalidityisevenmorecomplexthanan
evaluationofthesamplingdesign,thatis,howthesamplewasselectedfromtheaccessiblepopulation.Theevaluativeratingofpopulationvalidityshouldbebasedon
allthreeissuesinFigure10.3.
Therealquestioniswhethertheactualfinalsampleofparticipantsisrepresentativeofthetheoreticalortargetpopulation.Toevaluatethis,itishelpfulto
identify(a)theapparenttheoreticalpopulation,(b)theaccessiblepopulation,(c)thesamplingdesign,(d)theselectedsample,and(e)theactualsampleof
participantswhocompletedthestudy.Itispossiblethattheresearchercouldusearandomorotherprobabilitysamplingdesign,buthasanactualsamplethatisnot
representativeofthetheoreticalpopulationeitherbecauseofalow

Page160

responserateorbecauseoftheaccessiblepopulationisnotrepresentativeofthetheoreticalpopulation.Thelatterproblemisalmostuniversal,inpartasaresultof
fundingandtravellimitations.Exceptinnationalsurveyresearch,wealmostalwaysstartwithanaccessiblepopulationfromthelocalschooldistrict,community,clinic,
animalcolony,andsoon.
EcologicalExternalValidity
Theotheraspectofexternalvalidityiscalledecologicalvalidity.Ithastodowithwhethertheconditions,settings,times,testers,procedures,orcombinationsofthese
factorsarerepresentativeofreallife,and,thus,whethertheresultscanbegeneralizedtoreallifeoutcomes.Obviously,fieldresearchismorelikelytobehighon
ecologicalexternalvaliditythanarelaboratoryprocedures,especiallyiftheyarehighlyartificial.Wewouldratemostoftheselfreportmeasures,especially
questionnaires,tobeartificialbecausetheyarenotdirectmeasuresoftheparticipants'actualbehaviorinatypicalenvironment.
Anexampleofaproblemwithecologicalvalidityisthetraditionalstrangerapproach(e.g.,seeMorgan&Ricciuti,1969Spitz,1965).Althoughmethodsvaried,it
wastypicalinthe1960stotest612montholdinfantsinasomewhatunnaturalsetting(labplayroom)withamalestrangerwhoapproachedandpickedupthebaby
inashortseriesofpredeterminedsteps.Inthenameofexperimentalcontrol,noattemptwasmadetohavetheresearcherorstranger'sbehaviorbecontingentonthe
baby'sbehavior.Thisprocedure,andeventheexistenceoffearofstrangers,wascriticized.byRheingoldandEckerman(1973)whoshowedthataslower,more
"natural"approachbyafemalestrangerproducedalmostnocryingorattemptstogetaway.Ofcourse,thedeterminantsofinfantfeararecomplex,butitwasclear
thatearlystudieswerenothighonecologicalvalidity.Theyhadtradedecologicalvalidityforbettercontroloftheenvironmentalandindependentvariableaspectsof
internalvalidity.
Asanotherexampleofaprobleminecologicalvalidity,ifaneducatorisinterestedintheeffectofaparticularteachingstyleonstudentparticipation,theclassroom
shouldbesimilartothatofanormalclassroom.Similarly,iftheinvestigatoraskedstudentstocomeatnightforthestudy,butthesestudentsnormallyattendedclass
duringtheday,thenthereisaprobleminecologicalexternalvalidity.Theinvestigatormustaskifarepresentativemethodwasusedforselectingthesettingandtime,
orifaconveniencemethodwasused.Forhighecologicalvalidity,aninterventionshouldbeconductedbyaculturallyappropriateintervenor(teacher,therapist,or
tester)foranappropriatelengthoftime.
Finally,thereisthequestionofwhetherthestudyisspecificorboundtoacertaintimeperiod,orwhethertheresultswillbeapplicableoveranumberofyears.
Attitudesaboutcertaintopics(e.g.,schoolvouchers)maychangeoverashortperiodoftimesothatresultsmaynotbegeneralizableevenafewyearsafterthestudy.
AnevaluativeratingoftheecologicalvalidityofastudyshouldbebasedonallfiveissuesraisedinthissectionandinFigure10.3.

Page161

SamplingandtheInternalandExternalValidityofaStudy
Wehavediscussedtheinternalandexternalvalidityofastudyandnotedthatexternalvalidityisinfluencedbytherepresentativenessofthesample.Muchofthis
chapterhasbeenabouthowtoobtainarepresentativesampleandwhatproblemsmayariseintheprocessofsampling.Itisimportanttonoteagain,asindicatedin
Fig.10.4,thattheinternalvalidityofastudyisnotdirectlyaffectedbythesamplingdesignorthetypeofsampling.Thus,astudy,asisthecasewithmanyrandomized
experiments,mayhaveasmallconveniencesample,andstillhavehighinternalvaliditybecauserandomassignmentofparticipantstogroupseliminatesmanythreatsto
internalvalidity.Figure10.4isaschematicdiagramthatextendsandsimplifiesFig.10.1toshowhowthetwo

Fig.10.4.
Randomsamplingversusrandomassignmenttogroupsandtheir
relationshipstoexternalandinternalvalidity.

Page162

usesofthewordrandomhavedifferentmeaningsanddifferenteffectsoninternalandexternalvalidity.Randomselection,orsamplingofwhoisaskedtoparticipate
inthestudy,isimportantforhighexternalvalidity.Ontheotherhand,randomassignment,orplacementofparticipantsintogroups,isimportantforhighinternal
validity.Thisdistinction,whichisoftenconfusedormisunderstood,isimportantforevaluatingthequalityofaresearchstudyanditsinternalandexternalvalidity.
Summary
Samplingistheprocessofselectingpartofalargergroup(theaccessiblepopulation)withtheintentofgeneralizingfromthesmallergroup(thesample)tothe
population.Weidentifytwokindsofpopulations,theoreticalortargetversusaccessible,andwediscussdifficultiesinobtaininganaccessiblepopulationthatis
representativeofthetheoreticalpopulationofinterest.Sampling(thesamplingdesign)isthemethodusedtoselectpotentialparticipants(theselectedsample)fromthe
accessiblepopulation.Severalgoodsamplingdesigns(probabilitysampling)arediscussedthesedesignsincludesimplerandom,systematicwitharandomstart,
stratified,andclustersampling.Severalcommonbutlessdesirablesamplingmethods(quota,purposive,andconvenience)arealsodescribed.Finally,wediscuss
externalvalidityandhowtoevaluateitstwomajorcomponents,populationexternalvalidityandecologicalexternalvalidity.Randomselectionofparticipantsisuseful
toproducehighpopulationexternalvalidity,whereasrandomassignmentofparticipantstogroupsisimportantforhighinternalvalidity.
StudyAids
Concepts
Cluster
Externalvalidity
Representativesample
Sampling
Samplingdesign
Strata
Distinctions
Populationversussample.
Populationversusecologicalexternalvalidity
Probabilityversusnonprobabilitysampling
Quotaversuspurposiveversusconveniencesampling
Randomsamplingversusrandomassignmentofparticipantstogroups
Selectedsampleversusactualsample

Page163

Simplerandomversussystematicrandomversusstratifiedrandomversuscluster(random)sampling
Stratifiedsamplingwithequalversusdifferentialproportions
Theoreticalortargetpopulationversusaccessiblepopulation
ApplicationProblems
1.Aresearcherdistributedquestionnaires(surveys)toallemployeesofamunicipalagencytoobtainfeedbackregardingtheirjobsatthisparticularagency.Of720
questionnairesdistributed,605completed,usablesurveyswerereturned.Inthisprojectwhatwas:(a)thetargetpopulation?(b)theaccessiblepopulation?(c)the
selectedsample?(d)theresponserate?Wasanysamplingdone?Evaluatetheexternalpopulationvalidityoverall.
2.TheFortChoicemunicipalagencywasinterestedinemployeefeedback.Adecisionwasmadetosurveyarepresentativesampleofemployees.Theunits
comprisingtheagencyrangedfromverysmall,14to18employees,tofairlylargeunitsofmorethan100employees.Theresearcherswantedtobecertainthatall
unitswererepresentedinproportiontotheirsizeinthesurvey.Whatkindofsamplingapproachmighttheyuse?
3.Thecountyofficeonagingisinterestedintheperceivedneedsofolderadultsintheirservicearea.Atelephonesurveyisplanned.Asystematicrandomsampleof
25%oftheolderadultswithbirthdatespriorto1938isgeneratedfromvoterregistrationlists.Describehowthiswouldbedoneandthendiscussthestrengthsand
weaknessesoftheexternalvalidityofthisapproach.
4.Aresearcherisinterestedinstudyingmen'sandwomen'sreactionstoaviolentcrimeshowonnationalTV.
a.Describeanappropriateprobabilitysamplingtechniqueshemightuse.
b.Whataresomeproblemsthatmightaffectexternalvalidity?
5.Aresearcherhasalimitedresearchbudget,sohedecidestolookonlyatthehighschoolswithinamidwesterncommunityof50,000people.Therearethreehigh
schools.Hemakesalistofallthestudentsforeachgradelevel(8th12th).Herandomlysamples10studentsfromeachgradelevelateachschool(150students
total).Nameandcritiquethesamplingusedinthisstudy.
6.Aresearcherdecidestodoalaboratoryexperimentalstudyofsleepdeprivationonmathperformance.Herandomlyassignsstudentsfromhisconveniencesample
totwogroups.Onegroupiskeptawakeallnightandgivenamathtestinthemorning.Theothergroupisallowedtosleepaslongastheywantbeforetheytaketheir
mathtestinthemorning.Critiquethisstudyonthebasisofecologicalvalidity.
7.Dr.Gisevaluatingalargenationalgrant.Thepurposeofthegrantistoevaluatehowwellrevisedscienceeducationcoursesaretaughtatthecommunitycollege
anduniversitylevel.Thereare10communitycollegesand10universitiesinvolvedwiththegrant,eachwithonescienceeducationcoursewith40studentsinaclass.
Describehowyouwouldcarryouta10%samplingprocedurefor:
a.Simplerandomsample
b.Stratifiedrandomsample

Page164

c.Cluster(random)sample
Labelthetypeofsamplingtechniqueinthefollowingexample,andstatewhetheritisaprobabilityornonprobabilitysamplingtechnique.
8.Auniversityprofessorwasinterestedinwhetherhernewandimprovedcurriculumwasturningoutthebestpossibleteachers.Whenthestudentscompletedtheir
studentteacherexperience,theprofessorlookedoverthestudents'cumulativeGPAandtheirstudentteacherevaluations.Ofthe75studentswhocompletedthe
program,theprofessortookthe10studentswhohaddonewell,10fromthemiddle,andthe10studentswhohaddonepoorly,andinterviewedthem.

Page165

Chapter11
IntroductiontoInferentialStatisticsandProblemswithNullHypothesisSignificanceTesting
HypothesisTesting
ADifferenceHypothesisExample
TheNullHypothesis(H0)andAlternativeHypotheses(H1)
ThreeWaystoStatetheAlternativeHypothesis
TheInferentialProcess
TypeIandTypeIIErrors
StatisticalDecisionMaking
InterpretingInferentialStatistics
ProblemswithNullHypothesisSignificanceTesting
AddingConfidenceIntervalsHelps
IncludingEffectSizeIsNecessary
InterpretingEffectSizes
Summary
StudyAids
Concepts
Distinctions
ApplicationProblems
Thegoalofthischapteristointroduceinferentialstatisticsandthentodiscusssomeproblemswiththetraditionaluseofinferentialstatisticstotestnullhypotheses.Key
conceptstobecoveredinthischapterarehypothesistesting,exploratorydataanalysis,typeIandtypeIIerrors,confidenceintervals,andeffectsizes.

Page166

HypothesisTesting
Asdiscussedinthelastchapter,rarelyareweabletoworkwithanentirepopulationofindividuals.Instead,weusuallytryourtreatmentorinterventiononasampleof
individualsfromthepopulation.Ifourtreatmentissuccessful,wecaninferthattheresultsapplytothepopulationofinterest.Inferentialstatisticsinvolvesmaking
inferencesfromsamplestatistics,suchasthesamplemean( )andthesamplestandarddeviation(s)topopulationparameterssuchasthepopulationmean( )
andthepopulationstandarddeviation( ).Whenwerefertosamplestatistics,weuseRomanletters(ouralphabet)whenwerefertopopulationparameters,weuse
Greekletters.
ADifferenceHypothesisExample
Supposeweareinterestedintherelationshipbetweenexerciseandcardiovascularhealthinwomen.Ahypothesisisformedthatwomenwhoexerciseregularlywill
havebettercardiovascularhealththanthosewhodonotexerciseregularly.(Asdescribedinchap.5,thisisadifferencehypothesis.)Inferentialstatisticsprovidesus
withawaytotestthishypothesis,thatis,tomakeadecisionabouttherelationshipbetweenexerciseandcardiovascularhealth.(Wearenever100%certainthatthis
decisioniscorrect.)Totestourhypothesis,weneedtoreformulateitintotwostatementsorhypotheses,thenullhypothesisandthealternativehypothesis.
However,beforeweactuallyspecifythenullandalternativehypothesesforourstudy,weneedtooperationalizeourvariables.Theindependentvariable,exercise,
willbedefinedaseitheruseofastationarybicycle45minperday,5daysperweek,for6weeksataworkloadof50%ofmaximumcapacity,ornoexercise.The
dependentvariable,heartrate,isanindicatorofcardiovascularhealthandwillbemeasuredinbeatsperminute.Ifourhypothesisiscorrect,wewouldexpectthat
personswhoexercisewillhavealowerheartratethanthosewhodonotexercise,becauselowheartrateisconsideredtobeanindexofcardiovascularfitness.
TheNullHypothesis(H0)andAlternativeHypotheses(H1)
Thesehypothesescanbeshownasfollows:

Wherem I=interventionpopulationmean,and
m C=controlpopulationmean.
Thenullhypothesisstatesthatthemeanheartrateofthepopulationofthosewhoreceivetheinterventionisequaltothemeanheartrateofthepopulationofthose
whodonotreceivetheintervention.Ifthenullhypothesisistrue,theinterventionofexercisehasnotbeensuccessfulinchangingheartrate.Thealternative
hypothesisstatesthatthemeanheartrateofthepopulationofthosewhoreceivetheinterventionwillbelessthanthemeanheartrateofthepopulation

Page167

ofthosewhodonotreceivetheintervention.Ifthenullhypothesisisfalse,orrejected,theinterventionofexercisehasbeensuccessfulinalteringrestingheartrate.In
mostcases,thegoaloftheresearchistorejectthenullhypothesisinfavorofthealternativehypothesis.
Noticethatwedidnotsetupournullandalternativehypothesesbyusingsamplemeans,whichwouldappearasfollows:

where( I=interventiongroupsamplemean,and
C=controlgroupsamplemean.)
Theuseofsamplemeansisnotcorrect.Whynot?(Rememberthatwearediscussinginferentialstatistics.)Thismethodofwritingthehypothesesstatesthatweare
comparingonesampletoanothersample.Butourgoalistogeneralizetoapopulation.Therefore,westateournullandalternativehypothesesintermsofthe
populationmeans.
ThreeWaystoStatetheAlternativeHypothesis
Specifyingouralternativehypothesisintheexerciseexampleastheinterventionpopulationmeanbeinglowerthanthecontrolgrouppopulationmeanisjustone
methodofexpressingthealternativehypothesis.Actuallytherearethreechoices.Onechoiceistospecifythealternativehypothesisasnondirectional.Thisis
expressedasfollows:

Thisequationindicatesthatyoupredicttheinterventionwillbesignificantlydifferentfromthecontrol,butyouarenotsureofthedirectionofthisdifference.A
nondirectionalalternativehypothesisisoftenusedwhencomparingtwodifferenttreatmentmethods.Theothertwochoicesforalternativehypothesesaredirectional
positive:

anddirectionalnegative:

Directionalalternativehypothesesareusedmostoftenwhencomparingatreatmenttoacontrolcondition.Althoughitmayappearthatthesechoicesarearbitrary,two
thingsareimportanttokeepinmind.First,thetypeofalternativehypothesisselectedshouldbeinformedbytheliteraturereview.Whenthereispreviousresearchto
supportyourintervention,thenadirectionalhypothesisshouldbeused.Sometimesthereisnotstrongsupportforyourintervention.Thiscouldbeduetoconflicting
reportsfrompreviousstudiesorfromlittleresearchdonewithyourintervention.Inthesecasesanondirectionalalternativehypothesisshouldbeused.Asecond
importantpointtorememberaboutselectingthealternativehypothesisisthattherearestatisticalconsequencesattachedtothetype

Page168

ofhypothesisyouselect.Theseconsequencesoftenleadtotheconclusionof''nofreelunch."Wediscussthissecondissuelaterinthechapter.
Nowthatwehavetheproperterminologyforournullandalternativehypotheses,weneedtoconsiderourpopulationofinterestfortheexercisestudy.Ifweare
interestedingeneralizingtoallwomen,thatwouldbeourtheoreticalpopulation.Perhapsasubsetofallwomen,suchasallwomengraduatestudentsintheUnited
Statesisourtheoreticalpopulation.However,wemayhaveaccessonlytowomengraduatestudentswhoarecurrentlyattendingacertainuniversity.Therefore,
womengraduatestudentsatthatuniversityareouraccessiblepopulation.
TheInferentialProcess
Figure11.1,adaptedfromShavelson(1981,p.335)forourstudy,providesinsightintotheinferentialprocess.Atthefarleftofthefigureisaboxthatrepresentsthe
population.Theheartratesofwomengraduatestudentsattheuniversityisourdependentvariable.Forourpurposes,wewillassumethatheartrateisdistributed
normally.Whatdowemeanbydistributednormally?(Weexaminedthenormalcurveinsomedepthinchapter9.)Wemeanthatmostofthewomengraduate
studentshaverestingheartratesneartheaverage,about70beatspermin(bpm).Aswemovetowardtheextremesor"tails"ofthedistribution,say85bpmor55
bpm,therearefewwomenwhohavetheseheartrates.Whatisimportantforinferentialstatisticsisthatvaluesthatoccurinfrequently(i.e.,attheextremesofthe
distribution),areoftenregardedasbelongingto,perhaps,adifferentpopulationdistributionwithadifferentpopulationmean.
Notethatwerarelyknowtheactualcharacteristicsofthepopulationdistribution.Undermanyconditionswecanassumeittobenormal.However,there

Fig.11.1.
Schematicdiagramoftheprocessofmakinganinferenceaboutthedifferencebetweentwo
groups.FromStatisticalReasoningfortheBehavioralSciences(p.335),byR.J.Shavelson,
Copyright1981,Boston:AllynandBacon.Adaptedbypermission.

Page169

maybetimeswhenwecannotassumethisdistributiontobenormalthenwewillusenonparametric,ordistributionfreestatisticaltests.
Fromouraccessiblepopulation(womengraduatestudents)wesampleorselect,perhapsrandomly,72women.(Notethatthisisstep(a)inFig.11.1.)Thisstepis
bestdonebyselectingnamesfromatotallistofwomenstudentsinsuchawaythatallstudentshaveanequalchanceofbeingselectedforourstudy.Frequentlythe
sampleisoneofconvenience,notrandomlyselected.
Inthenextstep(b),weassignparticipantstogroups.AssumingapprovalfromtheHumanResearchCommitteeandthatall72participantshavegivenvoluntary
consenttobeineithergroupattheresearcher'sdiscretion,werandomlyassign36womentobeintheexercise(intervention)groupand36womentobeinthe
nonexercise(control)group.1Again,randomassignmentimpliesthateachstudenthadanequalchancetobeineithergroup.Iftheparticipantscannotberandomly
assignedtoeithergroup,thestudyisaquasiexperimentratherthanarandomizedexperiment.Rememberthedifferencebetweenrandomselection,whichis
importantforexternalvalidity,andrandomassignment,whichisimportantforinternalvalidity.
MovingtotherightinFig.11.1,weseethatthenextstep(c)istoconductthestudy.Now,theinterventiongroup(oneleveloftheindependentvariable)willexercise
onastationarybicyclefor45minaday,5daysaweek,for6weeks(training).Thecontrolgroup(theotherleveloftheindependentvariable)willrefrainfrom
exerciseforthenext6weeks(notraining).Youmayask,Whatifsomemembersofthecontrolgroupwerehabitualexerciserspriortothestudy?Itishardtosay
whateffectthiswouldhaveonourstudy,butitillustratesatypeofproblem(extraneousvariable)thatyoumightencounter.
After6weeks,weasktheparticipantstocomeintoourlaboratory,andwemeasuretheirheartrate,step(d).Wefindthemeanheartrateoftheinterventiongroupto
be65bpmandthemeanheartrateofthecontrolgrouptobe73bpm.Weknowthetwomeansaredifferent.Themeanoftheinterventiongroupislower,supporting
ourhypothesisthatexerciseincreasescardiovascularhealth.Fromtheseresults,canweformaconclusiontorejectthenullhypothesis(thatthereisnodifference
betweentheexerciseandnoexerciseconditions)insupportofthealternativehypothesis(thattheexerciseconditionwillreduceheartrate)?Beforewemakethis
decision,asecondstudyishelpful.
Supposethatwerepeatedthesamestudywithnewparticipants,butusedthesamenumberofparticipantsandthesamemethodofselectionandrandomassignment.
However,inthisnewstudy,neithergroupexercisesfor6weeks.Now,attheendofthe6weekperiod,wemeasurethemeanheartrateofbothgroups.Willthe
meansbeidentical?Probablynot,becausethereareindividualdifferencesamongthemembersofeachsample.Wearenotmeasuringthewholepopulation,onlya
sampleofthepopulation,sowewouldexpectthemeanstobedifferent,asaresultofrandomfluctuation.Thatis,evenwithoutintroducingatreatment,andevenifthe
twosampleswereequivalent,we
1

Itisimportanttopointoutthatthetwodifferentsamplesof36participantsineachgroupcouldbeconsideredassamplesfromtwodifferentbutidenticallydistributed
populationsratherthanfromasinglepopulation.

Page170

wouldexpectthetwomeanstobedifferentbychancealone.Therefore,weneedtouseinferentialstatisticstohelpmaketheproperdecision.
Nowbacktoouroriginalstudy.Afterperformingtheproperstatisticaltest,wecanformoneoftwoconclusions.Wecanconcludethattheinterventiongroupmean
reallyislessthanthecontrolgroupmean.Inotherwords,wecanconcludethattheinterventiongroupmeannowrepresentsthemeanofapopulationoftrained
women,andthecontrolgroupmeanrepresentsthemeanofapopulationofuntrainedwomen.Thisconclusiondefinesastatisticallysignificantdifferenceandis
demonstratedintheupperportionoftherighthandsectionofFig.11.1.
Asecondconclusioncouldbethatthereisnodifferencebetweenthetwomeans(lowerportionoftherighthandsectionofthefigure).Inotherwords,thedifference
betweenmeanswassimplyfromrandomfluctuation.Thislatterconclusionwouldimplythatthetwogroupsstillcomefromthesameunderlyingpopulation,andthat
thisamountofexercisedoesnotmakeadifferenceincardiovascularhealthasdefinedforourstudy.
Whichconclusiondowemake?Howmuchofadifferencebetweenthetwomeansisneededbeforeconcludingthatthereisasignificantdifference?Inferential
statisticsprovidesuswithanoutcome(astatistic)thatinformsusofthedecisiontomake.Whichconclusiondowemake?Howmuchofadifferenceisneeded
betweenthetwo?Evenafterperforminginferentialstatisticalproceduresonourdata,wearestillmakingadecisionwithadegreeofuncertainty.
Inourexampleofwomenandexercise,westatedthatthereweretwopossibledecisionsthatwecouldmakefromoursampledata.Eitherwewouldrejectthenull
hypothesisandconcludethatthetwogroupscomefromtwodifferentpopulations(topofFig.11.1,step(e)),orwewouldnotrejectthenullhypothesisandconclude
thatthesamplescomefromthesamepopulation(bottomofFig.11.1,step(e)).Thedecisiontorejectornotrejectthenullhypothesisisdeterminedforusby
subjectingoursampledatatoaparticularstatisticaltest.Anoutcomethatishighlyunlikelyunderthenullhypothesis(i.e.,onethatresultsinalowprobabilityvalue)
leadsustorejectthenullhypothesis.Anoutcomethatisnotunlikelywillresultinafailuretorejectthenullhypothesis.
TypeIandTypeIIErrors
Althoughinferentialstatisticsinformsusofthedecisiontomake(i.e.,rejectordonotrejectthenullhypothesis),thereisstillapossibilitythatthedecisionwemake
maybeincorrect.Thisisbecauseourdecisionisbasedontheprobabilityofagivenoutcome.Thestatisticisassociatedwithaparticularprobability.Forexample,at
valuecalculatedonoursampledatamayresultinaprobabilityof.04.Thisstatesthattheprobabilityofouroutcome(adifferencethislargebetweenthetwosample
groups)wouldoccuronly4timesin100,ifthenullhypothesisistrue.Whilearesultofthismagnitudewouldleadustoadecisiontorejectthenullhypothesisinfavor
ofanalternativehypothesis,thereisapossibilitythatweareinerror.Inotherwords,thenullhypothesismaybetrue.Whatistheprobabilityofmakinganerrorabout
thenullhypothesisinthissituation?Theansweris4outof100or.04.Therefore,whileinferentialstatisticsinforms

Page171

ustoeitherrejectornotrejectthenullhypothesisfromoursampledata,eachdecisioncaneitherbecorrectoritcanbeinerror.
Anytimeweconductastudyfromthesampledata,fouroutcomesarepossible.Twooftheoutcomesarecorrectdecisionsandtwooftheoutcomesareerrors.
CorrectDecisions:
1.Donotrejectthenullhypothesiswhenitistrue(i.e.,thereisnodifference).
2.Rejectthenullhypothesiswhenitisfalse(i.e.,thereisadifference).
Errors:
1.Rejectthenullhypothesiswhen,infact,itistrue(typeIerror).
2.Donotrejectthenullhypothesiswhenitisfalse(typeIIerror).
Weareneversureifthedecisionwehavemadeiscorrect(whatisactuallytrueinthepopulation),becausewearebasingourdecisiononsampledata.
Figure11.2(adaptedfromLoftus&Loftus,1982,p.225)helpsustoconceptualizethefourpossibleoutcomesthatwehavejustdiscussed.Thecurveontheright
representsthepopulationdistributionifthenullhypothesisistrue,andthecurveontheleftrepresentsthepopulationdistributionunderthealternativehypothesisifthe
nullhypothesisisfalse.ThelinedrawnperpendiculartotheXaxisinthisillustrationrepresentsthe.05decisionpoint,orsignificancelevel.Weestablishthislevelprior
tothestudy.Itiscustomarytodecidethatanydifferencebetweenourtwosamplemeansthatislargeenoughtoyieldastatisticaloutcomethatcouldoccurlessthan5
timesin100( =.05)ifthenullhypothesisistrueshouldresultinarejectionofthenullhypothesis.
First,wewilldescribethecurveontheright,orthepopulationdistributionifthenullhypothesisistrue.Mostofthiscurve(95%)istotherightofthe.05decisionpoint
orstatisticalsignificancelevelline.Onlyasmallportionofthiscurveistotheleftofthestatisticalsignificanceline.Wecalltheportiontotheleftofthestatistical
significancelinealpha( ).ThisalsoreferstotheprobabilityofmakingatypeIerror.Therefore,inthisexample,becausetheareaofthecurvetotheleftofthe
significancelevellineis5%,theprobabilityofatypeIerror(a)is.05.Theremainingportionofthecurve(thepartofthecurvetotherightofthesignificanceline)is
theprobabilityofmakingacorrectdecision.Thisis1 Itis1 ,because is.05.Becausewearedealingwiththepopulationdistributionassociatedwiththenull
hypothesis,thecorrectdecisionwouldbetonotrejectthenullhypothesis,assumingitistrue.Inourpresentexample,theprobabilityofmakingacorrectdecisionto
notrejectthenullhypothesisis.95(1 =1.05=.95).
ThecurveontheleftinFig.11.2isreferredtoasthepopulationdistributionrelatedtothealternativehypothesis.Theareaofthiscurvetotherightofthestatistical
significancelineiscalledbeta( ).Itisrelativelysmall,butnotusuallyassmallasalpha.Betadepictsanareaofthecurve(thealternativehypothesis

Page172

Fig.11.2.
TypeIandTypeIIerrorsrelatedtothenullandalternativepopulationdistributions.
AdaptedfromEssenceofStatistics(p.225),byG.R.LoftusandE.F.Loftus,1982,
Monterey,CA:Brooks/Cole.Adaptedbypermission.

curve)associatedwithanotherothertypeoferror.Thearea providestheprobabilityofmakingtheerrorofnotrejectingthenullhypothesiswhenitisfalse,or
shouldberejected.ThistypeoferroriscalledatypeIIerror.Theareaofthealternativehypothesiscurvethatfallstotheleftofthesignificancelineistheprobability
ofmakingacorrectdecision.Thiscorrectdecision,becauseitdealswiththealternativehypothesiscurve,isrejectingthenullhypothesiswhenitisfalse,thatis,whenit
shouldberejected.Theprobabilityofmakingthiscorrectdecisionis1 .Ourgoalinresearchusuallyistorejectthenullhypothesisinfavorofanalternative
hypothesis,sotheareaorprobability1 isveryimportant.Wewouldliketoincreasethisareaasmuchaspossible.Becauseofitsimportance,1 iscalled
power.Wediscusspowerinmoredetailinchapter23.Table11.1summarizesourdiscussionoftypeIandtypeIIerrors.
TABLE11.1
TypeIandTypeIIErrors

Trueinthepopulation

DecisionBasedonDataFromSample

NullisTrue

NullisFalse

Rejectnull

TypeIerror( )

Correctdecision(1 )Power

Donotrejectnull

Correctdecision(1 )

TypeIIerror( )

Page173

StatisticalDecisionMaking
Wereturnnowtodirectionalandnondirectionalalternativehypotheses.Rememberthatearlierinthechapterwestatedthattherearestatisticalconsequences
associatedwiththetypeofalternativehypothesisselected.Ifwehypothesizeadirectionalnegativealternativehypothesis,itwouldbeconceptualizedsimilartoFig.
11.2.Thedistributionofscoresunderthealternativehypothesisistotheleftofthedistributionofscoresunderthenullhypothesis.If,asinFig.11.2,weestablishour
significancelevel, ,at.05,thenastatisticaloutcomethatistotheleftofthis.05valuewouldresultinarejectionofthenullhypothesis.Asimilarconceptualization
wouldresultforanalternativehypothesisthatisdirectionalpositive.Here,astatisticaloutcomethatistotherightofthe.05valuewouldresultinarejectionofthenull
hypothesis.
Now,supposeinsteadthatouralternativehypothesisisnondirectional,asseeninFig.11.3.Wewouldhavetwodistributionsofscoresunderthealternative
hypothesis.Onedistributionwouldbetotherightofthedistributionofscoresunderthenullhypothesis,andtheotherdistributionwouldbetotheleftofthedistribution
ofscoresunderthenullhypothesis.Ifwekeepoursignificancelevelat.05,thenitwouldmeanthattorejectthenullhypothesisineitherdirection,thestatistical
outcomewouldhavetoexceedthe.025levelratherthanthe.05level.Therefore,itismoredifficulttorejectthenullhypothesisusinganondirectionalhypothesis.
However,youarelesslikelytomakeacriticalmistake.Inotherwords,ifyouselectadirectionalalternativehypothesis,andtheresultwasasignificantdifferencein
theoppositedirection,youcanonlyconcludeafailuretorejectthenullhypothesis,ornosignificantdifference.Itshouldbenotedthatbecauseweareusingbothends
ortailsofthedistribution

Fig.11.3.
Alternativenondirectionalhypotheses.

Page174

underthenullhypothesiswhenstatinganondirectionalhypothesis,atestofthishypothesisisreferredtoasatwotailedtest.
Alastwordonthephilosophyofhypothesistestingisthatwhenthenullhypothesisisnotrejected,itisneveractuallyaccepted.Thecorrectconclusiontoformis
thatthenullhypothesiswasnotrejected.Whileonemayquestionthedifferencebetweenthetermsacceptandnotreject,theproblemwiththeformeristhatthere
couldbemanyreasonswhyourstudydidnotresultinarejectionofthenullhypothesis.Perhapsanother,morepowerfulstudy,mightresultinarejectionofthenull
hypothesis.LoftusandLoftus(1982,p.242)provideaninterestingandhumorousdiscussionofthelogicalimplicationsoffindingsomething,asopposedtothelogical
implicationsofnotfindingsomething.
InterpretingInferentialStatistics
Therationalefornullhypothesissignificancetestingwasprovidedearlierinthischapter,whenweusedasanexampleabasicdifferencequestioncomparingtwo
groups.Herewepresentanoverviewofhowtointerpretanyoftheinferentialstatisticsdescribedinthisbook.Selectionandinterpretationofeachofthespecific
statisticsinthetablesispresentedinthefollowingchapters.Foreachstatistic(i.e.,t,c,X2,r,etc.),thecalculationsproduceanumberorcalculatedvaluefromthe
specificdatainyourstudy.Tointerpretthatcalculatedvalue,itiscomparedtocriticalvaluesfoundinastatisticaltableorinthecomputer'smemory,takinginto
accountthedegreesoffreedom,whichisusuallybasedonthenumberofparticipants.SeetheleftsideofFig.11.4forapproximatecriticalvalueswhenthestudyhas
about50participantsorisa22chisquare.
ThemiddlecolumnofFig.11.4showshowtointerpretanyinferentialtestonceyouknowtheprobabilitylevel(p)fromthecomputerorwhetherthecalculatedvalue
isgreaterthanthecriticalvalue.Ingeneral,ifthecalculatedvalueofthestatistics(t,F,etc.)isrelativelylarge,theprobability,orp,issmall,(e.g.,.05,.01,.001).Ifthe
probabilityislessthanthepresetalphalevel(usually.05),wecansaythattheresultsarestatisticallysignificantorthattheyaresignificantatthe.05levelorthatp
<.05.Wecanalsorejectthenullhypothesisofnodifferenceornorelationship.Wedonotusuallysayso,butwecouldthinkaboutthelevelofconfidence(1p)in
theresultsasshownontherightsideofFig.11.4.
NotethatcomputerprintoutssuchasthosefromSPSS(StatisticalPackagefortheSocialSciences)makeinterpretationofthevariousstatisticseasybyprintingthe
actualsignificanceorprobabilitylevel(p)soyoudonothavetolookupacriticalvalueinatable.Thistranslatesallofthecommoninferentialstatisticsintoacommon
metric,thesignificancelevel(Sig.inSPSS).ThislevelisalsotheprobabilityofatypeIerrorortheprobabilityofrejectingthenullhypothesiswhenitisactuallytrue.
Thus,regardlessofwhatspecificstatisticyouuse,ifpissmall(usuallylessthan.05),thefindingisstatisticallysignificantandyourejectthenullhypothesis.
Whenyouinterpretinferentialstatistics,itisnotenoughtodecidewhethertorejectthenullhypothesis.Ifyoufindthatthestatisticisstatisticallysignificant,youneedto
answertwomorequestions.

Page175

Fig.11.4.
Interpretinginferentialstatistics.

First,whatisthedirectionoftheeffect?Differenceinferentialstatisticscomparegroupssoitisnecessarytostatewhichgroupperformedbetter.Wediscusshowto
dothisinchapters14,15,17and18.Forassociationalinferentialstatistics(e.g.,correlation),thesignisveryimportant,soyoumustindicate

Page176

whethertheassociationorrelationshipispositiveornegative.Wediscusshowtointerpretcorrelationsinchapter16.
Second,whatisthesizeoftheeffect?Inthenextsectionwediscussproblemswithnullhypothesissignificancetestingandrecommendthatyouincludeconfidence
intervalsoreffectsizemeasuresorbothinthedescriptionofyourresults.Thegeneralmethodofcomputingeffectsizesconvertsthedifferencebetweenthegroupsinto
acommonmetric,standarddeviations,asdiscussedinchapter9.
ProblemswithNullHypothesisSignificanceTesting
Therehasbeenanincreaseinresistancetonullhypothesissignificancetesting(NHST)inthesocialsciencesduringrecentyears.Althoughresearchershave
periodicallyobjectedtotheuseofNHSTthroughoutthepastthreedecades(e.g.,Bakan,1966Rozeboom,1960),theintensityofobjectionhasincreasedrecently,
especiallywithinthedisciplinesofpsychology(Cohen,1990,1994Schmitt,1996a)andeducation(Robinson&Levin,1997Thompson,1996).Theseobjections
focusaroundnumerouscriticisms.Weaddresstwomajorproblemsherebecauseoftheirrelevancefordataanalysis.Bothproblemsinvolvemisinterpretationof
statisticalsignificance.Inaddition,wedescribetwotypesofsolutionstotheseproblems,calculationofconfidenceintervalsandeffectsizes.
AddingConfidenceIntervalsHelps
ThefirstproblemwithNHSTinvolvestheinterpretationofastatisticallysignificantdifferenceintheformofanallornonedecision.Thereisatendencytoaccept
somethingassignificantornotsignificant,ratherthanacknowledgethatstatisticalsignificanceimpliesaprobabilityofuncertainty.Inaddition,whenwedichotomize
statisticalsignificanceinthisway,webecomeremovedfromtheactualdataofourstudy.OnealternativeapproachtoNHSTistocreateconfidenceintervals.When
weconductastudytodeterminewhichoftwointerventionsortreatmentsworksbetter,ourresultisusuallyexpressedasadifferencebetweenthemeansofthetwo
interventiongroups,orbetweenaninterventionandacontrolgroup.Itisimportanttorememberthatthisdifferencebetweenmeansisadifferencebetweentwo
samplemeans.Thisdifferencebetweenthetwosamplemeansiscalledapointestimate.Wewouldliketobelievethatthismeandifferenceorpointestimateisagood
estimationoftheactualmeandifferencebetweenthetwopopulationmeans(populationparameter).However,becausethisisonlyadifferencebetweentwosample
means,wearenotsurehowclosethisestimationistotheactualpopulationdifferencebetweenmeans.Todeterminehowcloseourdifferencebetweentwosample
meansisfromourpopulationdifferencebetweenmeans,weneedtocreateaconfidenceinterval.Thisconfidenceintervalisarangeofthedependentvariablethat
shouldcontainthetruepopulationdifferencebetweenmeans.Wearenevertotallysurethatourconfidenceintervalincludesthepopulationparameter,sowemust
estimatetheconfidenceintervalwith,forexample,a95%con

Page177

fidenceinterval.Thisintervalmeansthat95%ofthedifferencesbetweensamplemeansthatwecomputewouldcontainthetruepopulationparameter.
ConfidenceintervalsprovidemorepracticalinformationthandoesNHST.Forexample,supposeoneknewthatanincreaseinreadingscoresof15pointsobtainedon
aparticularinstrumentwouldleadtoafunctionalincreaseinreadingperformance.Twodifferentmethodsofinstructionarecompared.Onemethodresultsina
statisticallysignificantgaincomparedtotheothermethod.AccordingtoNHST,wewouldrejectthenullhypothesisofnodifferencebetweenmethodsandconclude
thatournewmethodisbetter.Ifweapplyconfidenceintervalstothissamestudy,notonlycanwedeterminea95%intervalthatcontainsourpopulationparameter,
butwecanseeifthelowerboundofthatintervalisgreaterthan15pointsonourreadingtest.Ifthelowerboundisgreaterthan15points,weconcludethatusingthis
methodofinstructionwouldincreasefunctionalreadinglevels.
CortinaandDunlap(1997)havearguedthattocreateaconfidenceintervalyouneedthesameinformationthatwouldbeusedforNHST(e.g.,inattest,the
differencebetweensamplemeans,theestimatedstandarderrorofthedifferencebetweenmeans,andthecriticalvalueforthe.05alphalevelfortheparticulardegrees
offreedom).Therefore,wewilldemonstratehowtocreateconfidenceintervalsforthettestforindependentsamplesdescribedinchapter14.
IncludingEffectSizeIsNecessary
AsecondmajorproblemwithNHSTalsoconcernstheinterpretationofstatisticalsignificance.Thissecondmisinterpretationofstatisticalsignificanceoccurswhenone
assumesthatastatisticallysignificantoutcomegivesinformationaboutthesizeoftheoutcome.Astatisticallysignificantoutcomeonlydescribesarelationshipthatis
unlikelytooccur,assumingthenullhypothesisistrue,butitdoesnottelltheextentofthatrelationship.Therefore,itisimportanttostate,inadditiontoinformationon
statisticalsignificance,thesizeoftheeffect.Aneffectsizeisdefinedasthestrengthoftherelationshipbetweentheindependentvariableandthedependentvariable.
Forexample,ifoneiscomparinganinterventiongrouptoacontrolgroup,theeffectsize(d)couldbecomputedbysubtractingthemeanofthecontrolgroupfromthe
meanoftheinterventiongroupanddividingbythepooledstandarddeviationofbothgroupsasfollows:

Anothermethodofexpressingeffectsizesisasacorrelationcoefficient,r,(Rosenthal,1994).Chapter16providesadiscussionofcorrelationandhowtointerpretit.
Usingthismethod,effectsizesarealwayslessthan1.0,varyingbetween1.0and+1.0.
Unfortunately,therearemanydifferenteffectsizemeasuresandlittleagreementaboutwhichtouse.Althoughdisthemostcommonlydiscussedeffectsizemeasure,it
isnotusuallyavailablefromthepopularcomputerpackages

Page178

suchasSPSS.Thecorrelationcoefficient,r,orsimilarmeasuresofthestrengthofassociation,aremorecommonlyavailableinstatisticaloutputs.
AlthoughthePublicationManualoftheAmericanPsychologicalAssociation(APA,1994)recommendsthatresearchersreporteffectsizes,fewresearchersdid
sobefore1999whenTheAPATaskForceonStatisticalInferencestatedthateffectsizesshouldalwaysbereportedforyourprimaryresults(Wilkinson&TheTask
Force,1999).Becauseinthepastfewpublishedarticleshavepresentedeffectsizes,wedonotdiscussthemmuchinchapters1419.Inthefuture,itislikelythat
mostarticleswilldiscussthesizeoftheeffectaswellaswhetherornottheresultwasstatisticallysignificantandthedirectionoftheeffect.
InterpretingEffectSizes
Assumingthatyouhavecomputedeitherdorr,howshoulditbeinterpreted?Cohen(1988)providesaruleofthumbforinterpretingthepracticalimportanceofboth
dandrasfollows:
smalleffect

d=.2

r=.1

mediumeffect

d=.5

r=.3

largeeffect

d=.8

r=.5

Cohenprovidesresearchexamplesofsmall,mediumandlargeeffectstosupportthesuggesteddandrvalues.Mostresearcherswouldnotconsideracorrelation(r)
of.5tobeverystrong.However,Cohenarguesthatadof.8(andanrof.5,whichheshowsaremathematicallyequivalent)are''grosslyperceptibleandtherefore
largedifferences,as(forexample)themeandifferenceinheightbetween13and18yearoldgirls"(p.27).
Wediscusseffectsizesagaininchapter23.Inthatchapterweshowhoweffectsizescanbecombinedinametaanalysistoaccumulateknowledge.
Summary
Aconceptualviewofinferentialstatisticswasprovided,alongwithanexample.Inferentialstatisticsinvolvesmakinginferencesfromsamplestatistics,suchasthe
samplemeanandthesamplestandarddeviation,topopulationparameterssuchasthepopulationmeanandthepopulationstandarddeviation.Hypothesistesting,a
primaryprincipleofstatistics,wasintroduced.Wedefinedthenullhypothesesanddemonstratedthreedifferentwaystosetupthealternativehypothesis,
nondirectional,directionalpositive,anddirectionalnegative.Asignificantoutcomewasshowninourexample,alongwithtypeIandtypeIIerrors.AtypeIerrorwas
definedasrejectingatruenullhypothesis.AtypeIIerrorwasdefinedasfailuretorejectafalsenullhypothesis.
Hypothesistestinghasbeenundercriticismforseveralyears,morenowthanever.Reasonsforthiscriticismincludedincorrectinterpretationofstatisticalsignificance.
Alternativesandadditionstosignificancetestingwerediscussed,includingconfidenceintervalsandeffectsizes.Becauseastatisticallysignificantfindingmaybesmall
andoflittlepracticalimportance,itisdesirabletoknowthesizeoftheeffect.Unfortunately,foravarietyofreasons,re

Page179

searchershistoricallyhavenotreportedeffectsizes.Becauseastatisticallysignificantfindingmaybesmallandoflittlepracticalimportance,itisdesirabletoknowthe
sizeoftheeffect.Unfortunately,foravarietyofreasons,researchershistoricallyhavenotreportedeffectsizes.
StudyAids
Concepts
Alternativehypothesis
Confidenceintervals
Directionalhypothesis
Effectsize
Nondirectionalhypothesis
Nullhypothesis
Populationparameters
Samplestatistics
Statisticallysignificantdifference
TypeIerror
TypeIIerror
Distinctions
Directionalhypothesisversusnondirectionalhypothesis
Nullhypothesisversusalternativehypothesis
Samplestatisticsversuspopulationparameters
TypeIversusTypeIIerrors
Testsofsignificanceversusconfidenceintervals
ApplicationProblems
Forproblems1to4.Providenondirectionalanddirectionalalternativehypotheses.
1.Thereisnodifferencebetweenreformteachingmethodsandtraditionalteachingmethodsinstudents'mathematicsachievementdata.
2.Thereisnodifferencebetweensupportedemploymentandshelteredworkinsuccessfulcommunityparticipation.
3.Thereisnodifferencebetweenexerciseandnoexerciseincardiovascularhealth.
4.Thereisnodifferencebetweenstudentswhoperformwellandstudentswhodonotperformwellonteacherevaluations.
Forproblems5to8,describeinwords,theType1Error,theType2Error,andthetwocorrectdecisions.
5.Astudyisperformedtodetermineifreformteachingmethodsarebetterthantraditionalteachingmethods.

Page180

6.Astudyisperformedtodetermineifpeopleinsupportedemploymentparticipatemoreinthecommunitythanpeopleinshelteredworkshops.
7.Astudyisperformedtodetermineifthosewhoexercisehavelowerrestingheartratesthanthosewhodonotexercise.
8.Astudyisperformedtodetermineifstudentswithhighgradesgivebetterteacherevaluationsthanstudentswithlowgrades.
9.AcommonpracticeinresearchistoestablishtheprobabilityofaType1Errorat0.05andaType2Errorat0.20.Whydoyouthinkresearchersaremorewilling
toacceptaType2Error?
10.Aresearcherperformsastudycomparingtwodifferentgroups.Thereare26participantsineachgroup.Sheanalyzesthedatausingattest.Hertvalueis2.53.
Shouldsherejectthenullhypothesis?Why?
11.Aresearcherperformsastudycomparingthreedifferentgroups.Thereare17participantsineachgroup.HeanalyzesthedatausingasinglefactorANOVA.His
Fvalueis3.23.Shouldherejectthenullhypothesis?Why?
12.Whataretwogeneralproblemswithnullhypothesissignificancetesting?Howcantheseproblemsbealleviated?
13.Aresearcherdecidestodisplayhisresultsasaconfidenceinterval.Hecannotdecidebetweena95%confidenceintervalora99%confidenceinterval.Whichof
thetwointervalsdoyouthinkwillbelargerandwhy?
14.Whatarethetwodifferentmethodsofcomputingeffectsizes?

Page181

Chapter12
GeneralDesignClassifications
Introduction
GeneralDesignClassifications
BetweenGroupsDesigns
WithinSubjectsorRepeatedMeasuresDesigns
MixedDesigns
MoreDesignConsiderations
NumberofIndependentVariables
TypeofIndependentVariable
ThreeCasesofWithinSubjectsDesignsWithanAttributeIndependentVariable
ChangeOverTime(orTrials)asanIndependentVariable
DiagramingDesigns
BetweenGroupsDesigns
WithinSubjectsDesigns
MixedDesigns
DescribingtheVariousTypesofDesign
SingleFactorDesigns
BetweenGroupsFactorialDesigns
WithinSubjectsFactorialDesigns
MixedDesigns
ClassificationofSpecificExperimentalDesigns
SolomonFourGroupDesign
PretestPosttestNonEquivalentControlGroupDesign
WithinSubjectsRandomizedExperimentalDesign
Summary

Page182

StudyAids
Concepts
Distinctions
ApplicationProblems
Introduction
Inchapter7wediscussedspecificresearchdesigns,suchastheposttestonlyrandomizedexperimentaldesignandthenonequivalentpretestposttestcontrolgroup
design.Thesespecificresearchdesignshelpusvisualizetheoperationsofastudy,especiallyforinternalvalidity.Inthepresentchapter,weconsidergeneraldesign
classifications.Generaldesignclassificationisespeciallyimportantfordeterminingtheproperstatisticalapproachtobeusedindataanalysis.Withintherandomized
experimental,quasiexperimental,andalsocomparativeapproaches,alldesignsmustfitintooneofthreecategoriesorlabelsthatwecallgeneraldesignclassification.
GeneralDesignClassifications
BetweenGroupsDesigns
Betweengroupsdesignsaredefinedasdesignswhereeachparticipantintheresearchisinoneandonlyoneconditionorgroup.Forexample,inastudy
investigatingtheeffectsofteachingstyleonstudentsatisfaction,theremaybethreegroups(orconditionsorlevels)oftheindependentvariable,teachingstyle.These
conditionscouldbetraditional,reform,andacombinationofthetwo.Inabetweengroupsdesign,eachparticipantreceivesonlyoneofthethreeconditionsorlevels.
Iftheinvestigatorwishedtohave20participantsineachgroup,then60participantswouldbeneededtocompletetheresearch.

Page183

WithinSubjectsorRepeatedMeasuresDesigns
Withinsubjectsdesigns,thesecondtypeofgeneraldesignclassification,areconceptuallytheoppositeofbetweengroupsdesigns.Inthesedesigns,eachparticipant
intheresearchreceivesorexperiencesalloftheconditionsorlevelsoftheindependentvariabletocompletethestudy.Intheinvestigationoftheeffectsofthe
independentvariable(teachingstyle)onthedependentvariable(studentsatisfaction),therestillwouldbethreeconditionsorlevelstotheindependentvariable.These
conditionsagainaretraditionalteachingstyle,reformteachingstyle,andacombinationofthetwo.Inawithinsubjectsdesign,eachparticipantwouldexperienceand
bemeasuredforstudentsatisfactiononallthreeconditionsorlevelsoftheindependentvariable.Iftheresearcherwishedtohave20participantsforeachcondition,
only20participantswouldbeneededtocarryouttheresearch,becauseeachparticipantundergoesallthreeconditionsoftheindependentvariableintheresearch.
Becauseeachparticipantisassessedmorethanonce,thatis,foreachcondition,thesedesignsarealsoreferredtoasrepeatedmeasuresdesigns.
Withinsubjectsdesignshaveappealbecauseofthereducednumberofrequiredparticipantsandthereducederrorvariancethatresultswheneachparticipantishisor
herowncontrol.However,oftenthesedesignsmaybelessappropriatethanbetweengroupsdesignsbecauseofthepossibilityofcarryovereffects.Ifthepurposeof
thestudyistoinvestigateconditionsthatmayresultinalongtermorpermanentchange,suchaslearning,itisnotpossibleforaparticipanttobeinonecondition,and
then"unlearn"thatconditiontoreturntothepreviousstatetostartthenextcondition.Withinsubjectsdesignsmaybeappropriateiftheeffectsoforderofpresentation
arenegligiblewhenparticipantsareasked,forexample,toevaluateseveraltopics.Ordereffectscanbecontrolledbypresentingtheconditionstoparticipantsin
differentorders(e.g.,inrandomordersorcounterbalancedsothat,forexample,halfreceiveconditionAfirstandhalfreceiveconditionBfirst).Whenthereisa
pretestandaposttest,wehaverepeatedmeasuresandawithinsubjectdesignforoneoftheindependentvariables.
MixedDesigns
Theprevioustwoclassificationshaveonlyoneindependentvariable.Amixeddesignhasatleastonebetweengroupsindependentvariableandatleastonewithin
subjectsindependentvariablethus,ithasaminimumoftwoindependentvariables.1Abetweengroupsindependentvariableisanyindependentvari
1

Therearesomeintroductoryresearchdesigntexts(e.g.,Cosby,1989)thatdescribeamixeddesignasadesignthathasatleastoneactiveindependentvariableandoneattribute
independentvariable.Theproblemwiththischaracterizationofamixeddesignisthatitcouldbeconfusedwiththemixeddesignasdefinedinthisbook,andthenitwouldbe
incorrectlyanalyzedstatisticallybecausebothindependentvariablesarebetweengroupsvariables,whichrequireadifferenttypeofANOVAthanamixed(betweenandwithin)
design.Toavoidconfusion,abetweengroupsdesignwithoneactiveindependentvariableandoneattributeindependentvariablecouldbereferredtoasageneralized
randomizedblocksdesign(Kirk,1982).However,theproperdataanalysisdoesnotdistinguishbetweenactiveorattributeindependentvariables,onlythattheyarebetween
groupsindependentvariables.

Page184

ablethatsetsupbetweengroupsconditions.Awithinsubjectsindependentvariableisanyindependentvariablethatsetsupwithinsubjectsconditions.Letusreturnto
ourexampleofinvestigatingtheeffectoftheindependentvariable,teachingstyle,onthedependentvariable,studentsatisfaction.Ifteachingstyleisawithinsubjects
independentvariable,asinthesecondexampleearlier,wewouldadditionallyneedasecondindependentvariablethatisabetweengroupsindependentvariableto
completethecriteriaforamixeddesign.Thesecondindependentvariableforthisexamplecouldbethetypeofstudentintheclass.Studenttypeisabetweengroups
independentvariable,withtwolevels,traditionalandnontraditional.Therefore,thisexamplesatisfiesthecriteriaforamixeddesign:twoindependentvariables:onea
withinsubjectsvariable(teachingstyle)andtheotherabetweengroupsvariable(studenttype).
MoreDesignConsiderations
NumberofIndependentVariables
Amixeddesignmusthaveaminimumoftwoindependentvariables,oneabetweengroupsindependentvariableandtheotherawithinsubjectsindependentvariable.
Betweengroupsdesignsandwithinsubjectsdesignsalsomayhavemorethanoneindependentvariable(usuallynomorethanthree),althoughtheminimum
requirementforeachofthesedesignsisonlyoneindependentvariable.Iftheresearcherdecidestousemorethanoneindependentvariableineitherabetweengroups
designorawithinsubjectsdesign,theseadditionalindependentvariablesalsomustbebetweengroupsindependentvariables(inabetweengroupsdesign)andwithin
subjectsindependentvariables(inawithinsubjectsdesign).Otherwise,thedesignwouldbecalledamixeddesign.
TypeofIndependentVariable
Previouslyallindependentvariablesweredescribedasactive(i.e.,theindependentvariableismanipulatedorgiventoonegroupbutnottoasecondgroup)or
attribute(theinvestigatorisinterestedinaqualitythatisacharacteristicofonegroupofpeoplethatisnotcharacteristicofasecondgroupofpeople).Inabetween
groupsdesign,theindependentvariablemaybeeitheranactiveoranattributevariable.Thus,betweengroupsdesignscanusetherandomizedexperimental,quasi
experimental,orcomparativeapproach.Forexample,theworkshops,newcurricula,andinterventionsdescribedinchapters4and5wereactive,andthedesigns
werebetweengroups.Theexamplesofgender,giftedness,andtypeofdisabilitywereattributeindependentvariablesusedinbetweengroupsdesigns.
Ontheotherhand,inawithinsubjectsdesign,theindependentvariableisusuallyactive.Thus,theapproachisusuallyrandomizedexperimental(iftheorderofthe
conditionsisrandomized)orquasiexperimental.Thisbecomesclearifweconsideranexampleofstudentswithlearningdisabilities,anattributeindependentvariable.
Supposethatweareinterestedinthereadingspeedofstudentswhoarelearningdisabledandthosewhoarenotlearningdisabled.

Page185

Astudentcannotbebothlearningdisabledandnotlearningdisabledatthesametime.Likewise,apersoncannotbebothfemaleandmale.However,insomecases,
therecanbeawithinsubjectsdesignandanattributeindependentvariable.
ThreeCasesofwithinSubjectsDesignswithanAttributeIndependentVariable
Thesecasesusethecomparativeapproach.Onecaseinvolvesasituationwhereparticipantsareassessedonseveralpartsofaparticularinstrument,suchasthe
AssessmentofMotorandProcessSkills(A.G.Fisher,1993).Thisinstrumentprovidesseparatescoresformotorandprocessskills.Iftheinvestigatorisinterestedin
comparingthemotorscorestotheprocessscores,thedesignbecomesawithinsubjectsdesignwithtwolevels.Theindependentvariableistypeofskill,anattribute
withtwolevels.Asimilarexampleofawithinsubjectsdesignwithanattributevariable(comparativeapproach)isinaquestionnairestudywheretheparticipantsare
askedtorateseveralaspectsoftheirpersonality,motivation,orattitudes,andthentheseaspectsarecompared.Forexample,eightaspectsofworkers'perceptionsof
theimportanceof"margininlife"werecomparedbyHanpachern,Morgan,andGriego(1998).However,iftheindependentvariableisastimulus,suchasaproduct
oradesign,theindependentvariablewouldbeconsideredactivebecausethespecificdesignsorproductsusedare"given"bytheexperimenterandthefeaturescan
bemanipulated.Thentheapproachwouldbeexperimental.
Asecondcaseinwhichtheindependentvariableinwithinsubjectsdesignsisnotactiveinvolvesmatchingparticipants.Matchingreferstoasituationinwhich
participantsarecombinedintopairs(ortriads)tomakeeachmemberofthepairasmuchalikeaspossibleonsomemeasurerelevanttothedependentvariable.For
example,aresearcherisinterestedinqualityoflifeissuesforpersonswithdevelopmentaldisabilities.Specifically,heisinterestedindeterminingifpeoplewhoworkin
supportedemploymenthaveahigherqualityoflifethandopeoplewhoworkinshelteredemployment.However,previousresearchhasindicatedthatthereisa
relationshipbetweenintelligenceandqualityoflife(thedependentvariableforthestudy).Therefore,priortothestudy,theresearcherdeterminestheintelligencelevel
foralloftheparticipants(N=20).Thenhematches(formspairsofparticipants)theirintelligencelevel.Theparticipantswiththetwohighestintelligencelevelswould
formthefirstpair.Participantswiththethirdandfourthhighestintelligencelevelswouldformthenextpair.Thematchingprocesswouldcontinueuntilall20
participantshadformed10pairsofparticipants.Nowtheresearchercanformtwoequalgroupsbystartingwiththefirstpairofparticipantsandrandomlyassigning
onememberofapairtothesupportedemploymentgroupandtheothermemberofthepairtotheshelteredworkgroup.Thiscontinuesuntilall10pairsof
participantshavebeenrandomlyassignedtooneofthetwogroups.
Theimportantconsiderationforresearchdesignsthatusematchingisthattheychangeintothecategoryofwithinsubjectsdesigns.2Althoughpartici
2

Designsthatinvolvematchingsubjectsintopairs(ortriads)andthenrandomlyassigningonememberofeachpairtoaparticulargrouparecalledrandomizedblocksdesigns.
However,statisticallythesedesignsareanalyzedlikewithinsubjectsdesigns.

Page186

pantsareinone,andonlyonegroup,asdemonstratedinthequalityoflifestudy,thedesignisnotabetweengroupsdesign,becausethegroupsarenotindependent.
Theinvestigatormatchedtheparticipantsbeforeassigningthemtogroups.Tounderstandmatchingconceptually,rememberthedefinitionofawithinsubjectsdesign
eachparticipantundergoesallconditionsofthestudy.Inthematchingdesign,wearetryingtomakeeachpairofparticipantsasthoughtheywerethesameparticipant
bymatchingonacriterionrelevanttothedependentvariable.Forthefirstpairofparticipants,oneparticipantundergoesthesupportedemploymentconditionandthe
otherparticipantundergoestheshelteredworkcondition.However,fromastatisticalstandpoint,itisasthoughthesameparticipantunderwentbothsupported
employmentandshelteredworkconditions.(Thelackofstatisticalindependencewouldbeobviousifthepairsofparticipantsweretwinsseethenextsection.)
Althoughwedonotusuallyrecommendmatchingofparticipantsasacommonresearchstrategy,therearecertaincircumstanceswheretheinvestigatormaywishto
matchpairsofparticipants.Thesesituationsusuallytakeplacewhenthesamplesizeisrelativelysmallandheterogeneousforthedependentvariable.
Asimpliedearlier,thereisathirdwithinsubjectsdesigncaseinwhichtheindependentvariableisnotactive.Thisoccurswhenthemembersofthegroupstobe
comparedarerelatedinanimportantway.Thedesignissaidtobearelatedsamplesorpairedsamplesdesign.Obviously,identicaltwinsshouldbetreated
statisticallyasiftheywerethesameperson,soonewoulduseawithinsubjectsanalysisforthem.Perhapslessobviously,thesamewouldbetrueforcouples,parent
andchild,andteacherandstudent.Theseexampleswouldbetreatedstatisticallyaswithinsubjectsdesigns.
ChangeoverTime(orTrials)asanIndependentVariable
Inwithinsubjectsdesignstherecanbeathirdtype(neither,active,orattribute)ofindependentvariable,changeovertimeortrials.Thisthirdtypeofindependent
variableisextremelyimportantinrandomizedexperimentalandquasiexperimentaldesignsbecausepretestandposttestaretwolevelsofthistypeofindependent
variable.Longitudinalstudies,inwhichthesameparticipantsareassessedatseveraltimeperiodsorages,areanotherimportantcasewherechangeovertimeisthe
independentvariable.
Considerthefollowingstudythatusesapretestposttestcontrolgroupdesign(chap.7).Participantsarerandomlyassigned(R)tooneoftwogroups,anintervention
group(E),whichreceivesanewcurriculum,andacontrolgroup(C),whichreceivestheoldcurriculum.Participantsaremeasuredpriortotheintervention(O1)and
aftertheintervention(O2),perhapsattheendofthesemester.Thedesigncanbeviewedasfollows:
R

E:

O1

O2

C:

O1

~X

O2

Itisamixeddesignbecausetherearetwoindependentvariables,oneabetweengroupsindependentvariable,andtheotherawithinsubjectsindepend

Page187

entvariable.Theindependentvariable,typeofcurriculum,isabetweengroupsindependentvariablebecauseeachparticipantexperiencesonlyoneofthetwo
curriculums.Theotherindependentvariableinthisstudy,changeovertime,isawithinsubjectsindependentvariablebecauseparticipantswithineachgroupwere
measuredmorethanonceinthestudy.Thisindependentvariableisreferredtoaschangeovertimebecausethesecondmeasurementperiodtookplaceatalatertime
thanthefirstmeasurementperiod.Changeovertimeisconsideredathirdtyperatherthananactiveindependentvariablebecauseyoucannotactivelymanipulate
changeovertimetheposttestalwayscomesafterthepretest.
DiagramingDesigns
Betweengroups,withinsubjects,andmixeddesignscanbediagramedtohelpvisualizewhatishappeningintheresearch.Inaddition,themethodofdiagramingthat
werecommend(Winer,1962)depictshowthedataareenteredintothecomputerforfuturestatisticalanalyses.
BetweenGroupsDesigns
Thesedesignsalwayshavethedataforasinglesubjectorgroupplacedhorizontallyintoarowonthepage.Supposethatwehaveabetweengroupsdesignwithtwo
independentvariables,teachingstyleandgender.Eachindependentvariablehastwolevels(teachingstyle,traditionalorreformandgender,maleorfemale).Notice
thatwehavesimplifiedthediagrambyincludingthenamesofthelevelsbutnotthevariablename,aswedidinchapter4.Therefore,thedesignwouldbeasfollows,
assuming40participantswereassignedtothefourgroups:

Thefourgroupsareformedbytheconditions[TraditionalFemale],[TraditionalMale],[ReformFemale],and[ReformMale].Inthisexample,eachparticipant
ineachgroupisobservedormeasuredonce(O)onthedependentvariable,perhapssomemeasureofachievement.
Youmightaskwhywedonotusuallydisplaythediagraminblocksform,asfollows:

TeachingStyle

Traditional

Grp1(N=10)

Grp2(N=10)

Reform

Grp3(N=10)

Grp4(N=10)

Gender
Female

Male

Page188

Onereasonwedonotusetheblockdiagrammethodisthatthismethodworkswellonlyaslongastherearenomorethantwoindependentvariables.Whenthereare
morethantwoindependentvariables,thethirdindependentvariablewouldbevisualizedonathirddimension.Moreimportantly,theblockdiagramdoesnotrepresent
thewaythedatawouldbeenteredintothecomputerforproperanalysis.Thefollowingdiagrampartiallyillustratesthewaythatthesedatawouldbepreparedfor
enteringintothecomputer.(Onlythefirstandlastsubjectineachgroupof10areshown.)Noticethesimilaritytothefirstdiagram.
Group

ParticipantNo

TeachingStyle

Gender

Achievement

53

10

75

11

67

20

77

21

82

30

75

31

86

40

92

Let'saddathirdbetweengroupsindependentvariable,age,withtwolevels,youngandold.Because8groupsareneededtocompletethedesign,wewouldneed80
participantsforthestudy.Thisbetweengroupsdesignwiththreeindependentvariablescanbediagramedasfollows:

WithinSubjectsDesigns
Incontrasttobetweengroupsdesigns,withinsubjectsdesignsarealwaysdiagramedbyusingcolumns,andthedataareenteredforanalysisthatway.Supposewe
haveastudythatusesawithinsubjectsdesign.Therearetwoindependentvariables,bothwithinsubjectsindependentvariables.Thefirstindependentvariableis
changeovertime,withtwolevels,pretestandposttest.

Page189

Thesecondindependentvariableisourteachingstyleindependentvariable,withtwolevels,authoritativeandparticipatory.However,becausebothindependent
variablesarewithinsubjectsindependentvariables,eachparticipantmustundergoallconditionsoftheexperiment.
Notethatthedependentvariable(O)scoresarewhatgoineachcolumn.Thedesignisdiagramedasfollows:

Pretest
Reform
(condition1)

(n=10)

Posttest
Reform
(condition2)
O

Pretest
Traditional
(condition3)
O

Posttest
Traditional
(condition4)
O

Inthisdesign,only10participantsareneededtocompletethestudy.However,eachparticipantmustundergoallfourconditions.3
MixedDesigns
Thistypeofdesignisdiagramedbycombiningthebetweengroupsdesignandthewithinsubjectsdesign.Acommonexampleofamixeddesignwouldbearesearch
studytoevaluatetheeffectsofanewcurriculum.Thebetweengroupsindependentvariablewouldbethecurriculum,withtwolevels,newcurriculumandold
curriculum.Thewithinsubjectsindependentvariablewouldbetime,withtwolevels,beforetheevaluationandaftertheevaluation.Becausethediagramisrelatively
simple,wehaveincludedthevariablenameaswellasthelevels.

Typeof
curriculum

Pretest

Posttest

(Grp1,n=
10)

(Grp2,n=10)

Noticethateachparticipantisinonlyonegroup,butallparticipantsineachgrouparemeasuredbeforetheinterventionandaftertheintervention.
DescribingtheVariousTypesofDesign
Withinthemethodssectionofaresearchpaper,oftenthereisasubsectiondesignatedDesignorDesign/Analysis.Thepurposeofthissectionistoidentifythe
independentvariable(s),dependentvariable(s),anddesigninrandomizedexperimental,quasiexperimental,andcomparativestudies.Mostjournalswillnotallow
spacetodiagramthedesign,sotheappropriateprocedureistodescribethedesign.Designsareusuallydescribedintermsof(a)thegeneraltypeofdesign(between
groups,withinsubjects,ormixed)(b)thenumberofindependentvariablesand(c)thenumberoflevelswithineachindependentvariable.
3

Insomecasestheposttestforthefirstlevel(reform)servesasthepretestforthesecondlevel(traditional)ofoneindependentvariable,necessitatingonlythreeobservations.

Page190

SingleFactorDesigns
Ifthedesignhasoneindependentvariableonly(eitherabetweengroupsdesignorawithinsubjectsdesign),thenitshouldbedescribedasasinglefactordesign.
(Factorisanothernameforindependentvariable.)Forexample,abetweengroupsdesignwithoneindependentvariableandfourlevelswouldbedescribedasa
singlefactordesignwithfourlevels.Ifthesamedesignwasawithinsubjectsdesignwithfourlevels,thenitwouldbedescribedasasinglefactorrepeatedmeasures
designwithfourlevels.Notethat''betweengroups"isnotstateddirectlyinthefirstexample,butisimpliedbecausethereisnomentioninthatexampleofrepeated
measures.
BetweenGroupsFactorialDesigns
Whenthereismorethanoneindependentvariable,thenthelevelsofeachindependentvariablebecomeimportantinthedescriptionofthedesign.Forexample,
supposeadesignhasthreebetweengroupsindependentvariables,andthefirstindependentvariablehastwolevels,thesecondindependentvariablehasthreelevels,
andthethirdindependentvariablehastwolevels.Thedesigniswrittenasa2 3 2factorialdesign(factorialmeanstwoormoreindependentvariables).Again,
betweengroupsisnotexplicitlymentioned,butisimpliedbecausethereisnomentionofrepeatedmeasures,asinawithinsubjectsdesigndescription.Becausethe
designisabetweengroupsdesign,thenumberofgroupsneededtocompletethestudyis2 3 2,or12groups.
WithinSubjectsFactorialDesigns
Ontheotherhand,ifthedesignisawithinsubjectsdesignwithtwoindependentvariables,eachwithtwolevels,thenitisdescribedasa2 2withinsubjectsdesign
or,morecommonly,a2 2factorialdesignwithrepeatedmeasuresonbothfactors.
MixedDesigns
Suchadesignmighthavetwobetweengroupsindependentvariableswiththreeandfourlevels,respectively,andhaveonewithinsubjectsindependentvariablewith
twolevels.Itwouldbedescribedasa342factorialdesignwithrepeatedmeasuresonthethirdfactor.
Remember,whendescribingadesign,thateachindependentvariableisgivenonenumber,thenumberoflevelsforthatvariable.Thusadesigndescriptionwiththree
numbers(e.g.,2 4 3)hasthreeindependentvariablesorfactors,whichhave2,4,and3levels,respectively.Asinglefactordesignisspecificallyclassifiedor
describedinwords,notwithnumeralsandmultiplicationsigns.Notethatthedependentvariableisnotpartofthedesigndescription,soisnotconsideredinthis
section.Table12.1providesexamplesfor

Page191
TABLE12.1
ExamplesofDesignClassifications
SingleFactor

OneIndependentVariable

Between

Singlefactordesignwith___levels

Within

Singlefactorrepeatedmeasuresdesignwith___levels

Mixed

NA

TwoFactor

TwoIndependentVariables

Between

___ ___Factorialdesign

Within

___ ___Designwithrepeatedmeasuresonbothfactors

Mixed

___ ___(Mixed)Designwithrepeatedmeasuresonthelastfactor

ThreeFactor

ThreeIndependentVariables

Between

___ ___ ___Factorialdesign

Within

___ ___ ___Designwithrepeatedmeasuresonallfactors

Mixed

___ ___ ___Designwithrepeatedmeasuresonlast(orlasttwo)factors

Note.Thedependentvariableisnotpartofthedesignclassificationand,thus,isnotmentioned.The
numberoflevelsforanindependentvariableisinsertedineachblank.

describingthebetween,within,andmixeddesignsforstudieswithone,two,andthreeindependentvariables.
ClassificationofSpecificExperimentalDesigns
Althoughspecificexperimentaldesigns(seechap.7)areimportantforassessinginternalvalidity,theydonothelptodetermineselectionoftheproperstatistical
analysis.However,anyspecificresearchdesigncanbeexpressedasageneraldesign.Weprovidethreeexamplesofhowspecificresearchdesignsfitintogeneral
designclassifications.TheseexamplesaretheSolomonfourgroupdesign,thepretestposttestnonequivalentcontrolgroupdesign,andawithinsubjectsrandomized
experimentaldesign.
SolomonFourGroupDesign
Ofparticularinterestforthisdesignishowitfitsintoourgeneraldesignclassificationofbetweengroups,withinsubjects,ormixeddesigns.Afirstguessisthatitisa
mixeddesignbecauseatleasttwoofthegroupsreceiveapretestandaposttest.However,closerexaminationofthisdesignindicatesthattheinvestigatorisreallynot
interestedinthepretestscores,onlyintheeffectsthattakingapretesthasontheposttest.Therefore,thedesignisactuallyabetweengroupsdesignwithtwo
independentvariables.Morespecifically,thedesignisa22factorialdesign.Thetwoindependentvariablesarepretest(yesorno)andintervention(yesorno),
eachwithtwolevels.Thedesigncanbeviewedschematicallyasfollows:

Page192

PretestPosttestNonEquivalentControlGroupDesign
Thisspecificresearchdesignfitsthegeneraldesignclassificationofamixeddesign.Therearetwoindependentvariables.Oneindependentvariableistypeof
intervention,abetweengroupsindependentvariablewithtwolevels,treatmentandnotreatment.Thesecondindependentvariableischangeovertime,awithin
subjectsindependentvariablewithtwolevels,pretestandposttest.Thedesigncanbeseenasfollows:

Pretest

I.V.

Posttest

Treatment

NoTreatment

~X

Noticethatthepretestposttestcontrolgroupdesign,adesignthatfitstherandomizedexperimentalapproach,hasthesamegeneraldesignclassification,mixed,asthe
pretestposttestnonequivalentcomparisongroupdesign,aquasiexperimentalapproach.
WithinSubjectsRandomizedExperimentalDesign
Thethirdexampleisonethatwedidnotexplicitlydescribeinchapter7becausewehadnotyetdiscussedwithinsubjectsdesigns.Inthesimplestcase,thisdesignhas
twolevelsandcanbeshownasfollows:

Condition1

Test

Condition2

Test

RGroup1

~X

RGroup2

~X

TheparticipantsarerandomlyassignedtoeitherGroup1,whichreceivestheexperimentalconditionfirstandthenthecontrolcondition,orGroup2,whichreceives
thecontrolconditionandthentheexperimental.Ofcourse,thisdesigncanhaveproblemsiftherearecarryovereffectsfromtheexperimentalcondition.Theapproach
isconsideredrandomizedexperimentaliftheorderinwhichparticipantsreceivetheconditionsisrandomized.Ifnot,theapproachisquasiexperimental.Thistypeof
designisfrequentlyusedinstudieswhereparticipantsareaskedtoevaluatestimuli(e.g.,foods)andinevaluatingdiets,exercise,andsimilareventsassumednotto
havelongcarryovereffects.
Table12.2describeshoweachspecificresearchdesignfitsthegeneraldesignclassification(lastcolumn).Thisclassification(betweengroups,withinsubjects,or
mixed)determines,ingoodpart,theappropriatetypeofdifference

Page193
TABLE12.2
ClassificationofSpecificDesignsforExperimentsandQuasiExperiments
PoorQuasiExperimental
Designs

Assign.

Grp

OneGroupPosttestOnly
Design

NR

E:

NR

E:

NR

E:

NR

Pre

I.V.

Post.

Class.

None

Within

Between

C:

~X

OneGroupPretestPosttest
Design

PosttestOnlyDesignwith
NonequivalentGroups
QuasiExperimentalDesigns

PretestPosttest
NR
Nonequivalent
NR
ComparisonGroupDesigns

E:

C:

~X

Mixed

SingleGroupTimeSeries
Designs
withTemporaryTreatment

NR

E:

OOO

OOO

Within

withContinuousTreatment NR

E:

OOO

XOXO

XOXO

Within

MultipleGroupTimeSeries
Designs
withTemporaryTreatment

NR

E:

OOO

OOO

NR

C:

OOO

~X

OOO

withContinuousTreatment

NR

E:

OOO

XOXO

XOXO

NR

C:

OOO

OO

OO

RandomizedExperimental
Designs

PosttestOnlyControlGroup R
Design

E:

C:

~X

Mixed

Mixed

Between

PretestPosttestControl
GroupDesign

E:

C:

~X

Mixed

SolomonFourGroupDesign

El:

E2:

C1:

C2:

Between

2factor

~X

~X

RandomizedExperimental
Designwith
Matching

MR

E:

MR

C:

~X

Within

Abbreviationsare:Assign.=assignmentofsubjectstogroups(NR=nonrandom,R=random,MR=matched
thenrandomlyassigned).Grp.=grouporcondition(E:=experimental,C:=controlorcomparison).Pre=pretest(O
=anobservationormeasurementablankmeanstherewasnopretestforthatgroup).I.V.=activeindependent
variable(X=intervention,~X=controlorothertreatment).Post=posttest(O=aposttestobservationor
measure).Class.=classification(between,within,ormixed).

inferentialstatistictouse.ThewithinsubjectsdesignswouldbeanalyzedusingthecorrelatedorpairedsamplesttestorrepeatedmeasuresANOVA(ifthereare
morethantwolevelsormeasuresofthedependentvariable),asdis

Page194

cussedinchapter15.ThesinglefactorbetweengroupsdesignswouldbeanalyzedwithanindependentsamplesttestoronewayANOVA,asdiscussedinchapter
14,andthetwofactorSolomonfourgroupdesignwouldbeanalyzedwithatwowayANOVA,asdiscussedinchapter17.
Summary
Thischapterdescribedthegeneraldesignclassificationsofbetweengroups,withinsubjects,andmixeddesigns.Rememberthatinbetweengroupsdesigns,each
participantisinonlyonegrouporcondition.Inwithinsubjectsorrepeatedmeasuresdesigns,ontheotherhand,eachparticipantreceivesalltheconditionsor
levelsoftheindependentvariable.Inmixeddesigns,thereisatleastonebetweengroupsindependentvariableandatleastonewithinsubjectsindependentvariable.
Inclassifyingthedesign,donotconsiderthedependentvariable(s).
Thediagrams,classifications,anddescriptionspresentedinthischapterarefordifferencequestionsthatusetherandomizedexperimental,quasiexperimental,and
comparativeapproachestoresearch.Appropriateclassificationanddescriptionofthedesigniscrucialforchoosingtheappropriateinferentialstatistic,usuallysome
typeofanalysisofvariance.
StudyAids
Concepts
Betweengroupsdesigns
Carryovereffects
Changeovertime
Matching
Mixeddesigns
Solomonfourgroupdesign
Withinsubjectsdesigns
Distinctions
Activeversusattributeversuswithinsubjectsversusmixeddesigns
Betweengroupsdesignsversuswithinsubjectsdesignsversusmixeddesigns
Singlefactorversusfactorialdesigns
ApplicationProblems
1.Explainwhytheindependentvariable(s)forawithinsubjectsdesignarenotusuallyattributeindependentvariables.
2.Withinsubjectsdesignsusuallyemployoneoftwotypesofindependentvariables.Onetypeistheactiveindependentvariable.Whatistheothertype,andhowisit
used?

Page195

3.Giveanexampleofawithinsubjects(repeatedmeasure)design,anddiagramit.
Forquestions47answerthefollowing:
a.Identifytheindependentvariable(s).Foreach,statewhetheritisactive,attribute,orchangeovertime.
b.Identifythedependentvariable(s).
c.Diagramthedesign.
d.Identifythedesignclassification(e.g.,44factorial).
4.Aresearcherwantedtoknowiftypeofexerciseandtypeofindividualinfluencesaperson'swillingnesstostayinanexerciseprogram.Theresearcherrecruited
300participants.Thestudyincludedpeopleconsideredyoung(2035),middleaged(3650)andabovemiddleaged(5170).Thestudyalsoincluded150menand
150women.Additionally,ofthe300participants,100wereAfricanAmerican,100werewhite/NonHispanic,and100wereHispanic.Theparticipantswere
randomlyassignedtothreedifferentexerciseregimes:runningincirclesaroundatrack,swimminglapsatanindoorpool,orridingabikeintheRockyMountains.The
regimeslastedfor2months.Attheendofthe2months,theparticipantsallcompletedtheWillingnesstoContinueExercisingRegimeScale.
5.Ahumanitiesprofessorwhowasgoingtoleadayearlongstudyabroadprogramwonderediftravelexperiencehadanyimpactonstudents'abilitytounderstand
andembracediversityinothers.AtthebeginningoftheschoolyeartheprofessorgaveallthestudentstheMorganMulticulturalAcceptanceScale.Thisscalewasalso
givenattheendoftheyearwhenthestudentsreturnedfromabroad.
6.Adogtrainerwasinterestedinknowingwhetherhernewaversiveapproachtoobediencetrainingwaseffective.Shedividedhernewclienteleintothreedifferent
groups.Thefirstgroupreceivedtraditionaldogtraining,whereingoodbehaviorisrewardedwithpraiseandtreats.Thesecondgroupreceivedthenewaversive
training,whereinnonconformingbehaviorwaspunishedwithremovedwaterandfood,slapsonthenose,andloudyellingbythetrainerandtheowner.Thethird
groupwasthecontrolanddidnotgetanytrainingfortheirdogs.Beforethetrainingandthreemonthslater,thetrainerratedthedogsfromallthreegroupsonadog
obediencescale.
7.Aninvestigatorwasinterestedintwodifferentcuesthatmightbeusedinthereproductionofmovement:(a)theinitialpositionofthemovementandsameor
different,(b)thespeedofthemovement,fastorslow.Inaddition,shewasalsointerestedinhowageaffectsreproductionofmovement.Threegroupsofparticipants
(40participantspergroup)wereinthestudy.Thesethreegroupswereeither7yearolds,11yearolds,oradults.Withineachgroupsubjectswererandomlyassigned
tooneoffourconditions.Theseconditionswereafastmovementwiththeinitialpositionthesame,aslowmovementwiththeinitialpositionthesame,afastmovement
withtheinitialpositiondifferent,andaslowmovementwiththeinitialconditiondifferent.Theresearchermeasuredthedistanceerrorfromthetargetandtheangle
error.

Page197

Chapter13
SelectionofStatisticalMethods
ReviewofConceptsNecessaryforSelectingInferentialStatistics
ResearchApproachesandQuestions
DifferenceQuestions
AssociationalQuestions
IndependentandDependentVariables
NumberofIndependentVariables
NumberofLevelsoftheIndependentVariable
DesignClassifications
BetweenGroupsVersusWithinSubjectsSingleFactorDesigns
ClassificationinFactorialDesigns
LevelsorScalesofMeasurement
OtherAssumptions
SelectionofAppropriateInferentialStatistics
HowtoUsetheStatisticalSelectionTables
BasicDifferenceStatistics
BasicAssociationalStatistics
ComplexDifferenceStatistics
ComplexAssociationalStatistics
TheGeneralLinearModel
Summary
StudyAids
Concepts
Distinctions
ApplicationProblems

Page198

Choosingtheproperstatisticalanalysismayseemlikeadifficulttask,consideringthelargenumberofpossiblechoices.However,thistaskshouldbeeasierbecause
youarenowfamiliarwithindependentanddependentvariables,researchapproaches,designclassifications,andscalesorlevelsofmeasurement.Thischapter
presentsaseriesofdecisionstepsandfourtablesthatwillhelpyouchooseanappropriateinferentialstatistic.However,beforepresentingthedecisiontreeand
describinghowtousethestatisticalselectiontables,wereviewtheconceptsthatarenecessaryforselectinginferentialstatistics.
ReviewofConceptsNecessaryforSelectingInferentialStatistics
ResearchApproachesandQuestions
Inchapter5,wediscussedfiveresearchapproachesandthreetypesofresearchquestions.Figure13.1(whichisthesameasFig.5.3)isthekeyfigurethatpresents
therelationshipsamongthefivespecificapproaches,thethreetypesofresearchquestions,andthethreetypesofstatistics:differenceinferential,associational
inferential,anddescriptive.
DifferenceQuestions
Thefirstthreeapproaches(randomizedexperimental,quasiexperimental,andcomparative)allcomparegroupsandtestdifferencequestionsorhypotheses,asinour
womengraduatestudents'exerciseandheartrateexample(chap.11).Thesethreeapproachesusuallyusethesametypesofstatistics,whichwecalleddifference
inferentialstatistics.Rememberthatdifferencestatisticsandquestionsareusedtocompareafewgroups(e.g.,malesversusfemales,experimentalversuscontrol,or
threecurriculums)foreachgroup'saveragescoresonthedependentvariable(e.g.,anachievementmeasure).
AssociationalQuestions
Wehopeyourememberthatassociationalquestionsusetheassociationalapproachtoresearchandwhatwecallassociationalinferentialstatistics.Thestatisticsinthis
groupexaminetheassociationorcorrelationbetweentwoormorevariables.Ifthereisapositiveassociation,personswhohavehighscoresononevariabletendto
havehighscoresonthesecondvariablethosewithlowscorestendtobelowonbothvariables.Thatis,highscoresareassociatedwithhighscores,lowwithlow,
andmediumwithmedium.Ontheotherhand,ifthereisanegativeassociationbetweenthetwovariables,thosewithlowscoresonthefirstvariabletendtohavehigh
scoresonthesecondvariableandviceversa.Thatis,lowscoresareassociatedwithhighscores.Ifthereisnoassociation,youcannotpredictaperson'sscoreonthe
secondvariablefromknowingthefirst.Peoplewhoscorehighonthefirstvariablemightbehigh,orlow,ormediumonthesecondvariable.
Descriptivequestionsandstatisticswerediscussedinchapter9,sowillnotbediscussedinthischapter.Itisworthnotingthatinseveralcasestheremaybemorethan
oneappropriatestatisticalanalysis.Onemightassume,becausethestatisticalformulasareprecisemathematically,thatthissameprecisiongener

Page199

alizestothechoiceofastatisticaltest.Asweshallsee,unfortunately,thisisnotalwaystrue.
IndependentandDependentVariables
Wediscussedvariablesindepthinchapter4.Rememberthattheindependentvariableisapresumedcauseofchangesinthedependentvariable,althoughthe
moderateandweakquasiexperimental,comparative,andassociationalapproachesdonotprovidegoodevidenceaboutcauses.Wedistinguished,inchapter4,
betweenactiveormanipulatedindependentvariablesandattributeindependentvariables.Althoughthisdistinctionisimportantfordecidingwhethertheindependent
variableisacause,itisnotrelevantfordecidingwhichinferentialstatistictouse.Thus,wewillnotmentionactiveandattributeindependentvariablesagaininthis
chapter.Whatisrelevantforselectingstatisticsisthenumberofvariables,especiallythenumberofindependentvariablesandlevelswithintheseindependent
variables.

Fig.13.1.
Schematicdiagramshowinghowthegeneraltypeofstatisticandhypothesesor
questionusedinastudycorrespondstothepurposesandtheapproach.

Page200

NumberofIndependentVariables
Firstyouneedtodecidewhetherforthisresearchquestionthereisone,ormorethanone,independentvariable.Ifthereisonlyone,wecallthedesignbasic(or
singlefactordesign,ifansweringadifferencequestion).Ifthereismorethanoneindependentvariable,thestatisticsarecalledcomplex(orfactorialinthecaseof
differencequestions).
NumberofLevelsoftheIndependentVariable
Adifferencequestionisindicatedwhentheindependentvariablehasafew(i.e.,twotofour)levels.Thatis,domalesandfemalesorexperimentalandcontrolgroups
differonthedependentvariable?However,iftheindependentvariablehasmorethanfourunordered(nominal)levels,onewouldusuallystillaskadifferencequestion
andcomparethegroups.Forexample,dosixethnicgroupsdiffer?Rememberthattherehavetobeatleasttwolevelsorthereisnotavariablerather,thereisa
constant.
Whentheindependentvariablehasfiveormoreorderedlevels,youusuallyaskanassociationalquestionanduseassociationalinferentialstatistics.Thus,ifthe
independentvariableiscontinuous(aninfinitenumberoforderedlevelswithinarange)orapproximatesacontinuousvariable(ourruleofthumbisfiveormore
orderedlevels),associationalstatisticsareused.However,onecanalsoaskanassociationalquestionwhentheindependentvariableisnominal.Itshouldbenoted
thatthetwovariableassociationalinferentialstatistics(e.g.,Pearsoncorrelation)arebidirectional,sostatisticianswouldsaythatthereisnoindependentvariable.
However,becauseresearchersusuallyhaveacausalrelationshipinmind,wesuggestidentifyingoneofthevariablesastheindependentvariable.
Thedependentvariableisalsoimportantfortheappropriatechoiceofaninferentialstatistic.Theprimaryissueisthelevelofmeasurementofthedependentvariable,
whichwediscussfollowing.
DesignClassifications
Ourdiscussionofdesignclassificationsinchapter12isimportantbackgroundforselectinganappropriatestatistic.Thekeyissueforselectinganappropriatestatistic
iswhethertheclassificationisbetween,within,ormixed.Theseclassificationsapplyonlytotherandomizedexperimental,quasiexperimental,andcomparative
approaches(i.e.,todifferencequestions).
BetweenGroupsVersuswithinSubjectsSingleFactorDesigns
Withoneindependentvariable,thedesignmustbeeitherbetweenorwithinbecauseittakesatleasttwoindependentvariablestohaveamixeddesign.Tousebasic
differencestatistics,youneedtoknowwhetherthetwoormoregroupsorlevelsoftheindependentvariableareindependentofeachother(abetweengroupsdesign)
orrelated(awithinsubjectsorrepeatedmeasuresdesign).1In
1

Notethatinthissentencethewordindependenthastwodifferentmeanings.Thesecondusage,meaningseparatefrom,notrelatedto,ornotinfluencedby,isakeytermin
statisticsandisanassumptionofmanystatisticaltests.AppendixAcontraststheseveralmeaningsoftermslikeindependent,random,andvalidity,whosemeanings,
unfortunately,dependonthecontext.Wehavetriedtobeclearaboutthecontext.

Page201

betweengroupsdesigns,eachparticipantisinonlyonegroup,andparticipantsareneithermatchedinpairs,triads,andsoon,norrelatedascouples,motherandchild
(ren),orteacherandstudent(s).Inwithinsubjectsorrepeatedmeasuresdesigns,theparticipantsareeitherassessedtwoormoretimes(repeatedmeasures),orelse
two(oreventhreeormore)ofthemarematchedorpairedupinameaningfulway.Forstatisticalpurposes,theirscoresarenotindependent(i.e.,theyaresaidtobe
relatedorcorrelatedsamples).Thesewithinsubjectsdesignsusedifferentstatisticsthanthebetweengroupsdesigns,aswewillsee.
ClassificationinFactorialDesigns
Whenyouhavetwoormoreindependentvariables,therearethreepossibledesignclassifications:allbetweengroups,allwithinsubjects,andmixed(betweenand
within).Thus,tochoosetheappropriatecomplexdifferencestatistic,youhavetounderstandthisdistinction.Inbetweengroupsdesigns,thegroupsareindependent
eachparticipantisassessedonlyonceonanygivendependentvariable.Inwithinsubjectsdesigns,eachpersonisassessedineveryconditionandsohasascorein
everycellinthedesign.Inmixeddesigns,suchasthepretestposttestcontrolgroupdesign,thereisatleastonebetweengroupsvariableandatleastonewithin
subjectsvariable.
LevelsorScalesofMeasurement
Forappropriatestatisticalselection,levelofmeasurementisalsoimportant.Rememberthatnormallydistributeddatawasthehighestleveldiscussedinchapter9.It
isalsoanassumptionofparametricstatisticssuchasthettest,ANOVA,andPearsoncorrelation.Ordinaldatahavethreeormorelevelsorderedfromlowtohigh
(oftenranks),butwithunequalspacesbetweenlevelsand,moreimportantlyforstatisticalselection,theyarenotnormallydistributed.Incontrast,nominaldatahave
threeormoreunorderedlevelsorcategories.
Fordifferencestatistics,thevariablewhoselevelofmeasurementmattersisthedependentvariable.Theindependentvariablecanbenominal(e.g.,ethnicgroups)or
ordered(e.g.,low,medium,andhigh),butusuallyhasfewerthanfiveorderedlevels.Forassociationalstatistics,thelevelofmeasurementforbothorallvariables
needstobedetermined.
Dichotomousvariablesformaspecialcase,asdiscussedinchapter9.Rememberthatalthoughdichotomousvariablesareinmanywayslikenominalvariables,they
canbeused,especiallyasindependentorpredictorvariablesinmultipleregression,asiftheywerenormallydistributedvariables.
OtherAssumptions
Everystatisticaltestisbasedoncertainassumptions.Ingeneral,theparametricstatistics(ttest,ANOVA,Pearsoncorrelation,multipleregression)havemore
assumptions.Theparametricstatisticshavenormalityofadistributionasoneoftheassumptions.Often,thisassumptioncanbeviolatedquiteabitbeforetheresults
aredistortedthus,thevariablesusedinparametricanalysesonlyhavetobeapproximatelynormallydistributed.

Page202

Equalityofvariancesisanothercommonassumptionofparametrictests.Again,thiscanbeviolatedtoadegree,andsomestatisticalprograms(e.g.,SPSS)have
builtincorrectionsforsomeparametricstatistics.However,ifoneorbothoftheseorotherassumptionsaremarkedlyviolated,thentheequivalentordinal
nonparametrictestshouldbeused.
Itshouldbeapparentfromtheprecedingdiscussion(andthattofollow)thatselectionofanappropriatestatisticis,tosomeextent,amatterofjudgmentandbest
practice.Evenso,thereareseveralguidelinesthatmustbefollowed.Wewillpointtheseoutasweencounterthem.
TheassumptionofindependencewasmentionedearlierunderDesignClassifications.Thisassumptionmustnotbeviolated,forexample,byusingabetweengroups
statisticwhenyourdataarereallywithinsubjects.
SelectionofAppropriateInferentialStatistics
HowtoUsetheStatisticalSelectionTables
Figure13.2,aswellasthefollowingtext,willhelptoorganizeyourthinkingwhenselectinganappropriatestatisticaltest.First,decidewhetheryourresearchquestion
orhypothesisisadifferenceone(i.e.,comparesgroups)oranassociationalone(relatesvariables).Ourruleofthumbisthatiftheindependentorpredictorvariable
hasfiveormoreorderedlevelsorcategories,thequestionshouldbeconsideredanassociationalone.2Iftheindependentvariablehastwotofourcategoriesitis
usuallybettertotreatthequestionasadifferenceone.ThelatterleadsyoutoTables13.1or13.3andtheformertoTables13.2or13.4.
Second,decidehowmanyvariablesthereareinyourquestion.Ifthereisonlyoneindependentvariable,useTable13.1or13.2,dependingonhowyouansweredthe
firstquestion.Ifthereismorethanoneindependent(ordependent)variabletobeusedinthisanalysis,useTables13.3or13.4,dependingonwhethertheresearch
questionisadifferenceorassociationalquestion.
BasicDifferenceStatistics
Ifyourquestioninvolvesabasic,singlefactor,differencequestion,useTable13.1.Todosoyoumustdetermine(a)thelevelofmeasurementofthedependent
variableandwhetherassumptionsaremarkedlyviolated,(b)howmanylevelsorgroupsorsamplesthereareinyourindependentvariable,and(c)whetherthedesign
isbetweengroupsorwithinsubjects.TheanswerstothesequestionsleadtoaspecificlocationandstatisticinTable13.1.Noticethatyouareaskedtodecide
whethertheindependentvariablehastwoversustwoormorelevelsoftheindependentvariable.Youmightaskwhywebothertohaveaseparatecategoryfortwo
levelswhen''twoormore"includestwo.Onepartoftheansweristhatthepopularttestcanonlybeusedwhentherearetwolevelsthesecondpartisthatthettest
canbeusedwithadirectional(onetailed)hypothesis,whereasthealternativestatistic,
2

Theexceptionisthatifyouwanttoassessthestrengthoftheassociationbetweentwonominalvariables(ororderedvariableswithafewlevels,youwouldusetheappropriate
nominalassociationalstatistic(i.e.,PhiorCramer'sV).

Page203

Fig.13.2.
Adecisiontreetodecidehowtoselecttheappropriatestatistic.

onewayANOVA,isalwaystwotailed.Thereismoreonthistopicinchapter14.AllthestatisticsinTable13.1arediscussedinchapters14and15.
Rememberthatifassumptionsoftheparametrictestaremarkedlyviolatedoneshouldusetheequivalent,ordinalnonparametricstatistic(e.g.,MannWhitneyinstead
oftheindependentsamplesttest).Theseordinal,nonparametricalternativesarelistedrightbelowtheparametrictest.Littlepowerislostinusingthesetests,soitis
probablywisetousethemwhenassumptionsaremarkedlyviolated.Itwouldalsobelegitimatetousethestatisticsinthebottomrow(e.g.,chisquare)ifonehad
ordinalornormaldata,butthereisamajorlossofpowerindoingthat,soitisnotagoodpractice.AprincipleinusingTables13.1and13.2isthatitisacceptableto
useastatisticfartherdownagivencolumn.Youlosealittlepowergoingfromthetoptothesecond(ordinal)row,butyoulosealotofpowergoingfromthesecond
tothethirdrowinthecolumn.Itisaseriouserrortousethewrongcolumn,thatis,withininsteadofbetweenorviceversa.Anotherabsoluteviolationthatwill
producemeaninglessresultsistouseastatisticfromthetoptworowsinTables13.1and13.2(e.g.,attestorMannWhitneyU)whenonehasanominaldependent
variable.Thatisdefinitelywrong!
BasicAssociationalStatistics
Ifyouaskabasic,twovariable,associationalquestion,useTable13.2.Whichrowyouusedependsonbothvariables.Ifbothareatleastapproximatelynormally
distributed(andotherassumptionsmet),youwouldusethePearsonproductmomentcorrelation.Ifbothvariablesareatleastorderedandparametricassumptionsare
markedlyviolated,

Page204
TABLE13.1
SelectionofanAppropriateInferentialStatisticforBasic,SingleFactor,DifferenceQuestionsorHypotheses(forExperimental,QuasiExperimental,and
ComparativeApproaches)

OneFactororIndependentVariableWith
2CategoriesorLevels/Groups/Samples

ScaleofMeasurementof
DependentVariable
Parametric
Statistics

Dependentvariable
approximatesnormal
distributiondataand
assumptionsnot
markedlyviolated

Compare
Means

IndependentSamples
orGroups
(Between)

OneIndependentVariable2orMore
CategoriesorLevelsorGroups

RepeatedMeasures
RepeatedMeasuresor
orRelatedSamples IndependentSamples
RelatedSamples
(Within)
orGroups(Between)
(Within)

Independentsamplest Pairedsamples
testorOneway
ttest
ANOVA

OnewayANOVA

RepeatedMeasures
ANOVA

Chap14

Chap15

Chap14

Chap15

MannWhitney

Wilcoxonorsigntest KruskalWallis

NonParametiric
Statistics

Dependentvariables
clearlyordinal(ranked)
dataorassumptions
markedlyviolated

Mediansor
ranks

Friedman

Chap15
Chap14

Chap14

Chap15

Dependentvariable
nominal(categorical)
data

Counts

ChiSquareorFisher
exacttest

McNemar

ChiSquare

CochranQtest

Chap14

Chap15

Chap14

Chap15

Note.Toselecttheappropriatestatistic,locateaboxbasedona)thetypeofquestion,b)thedesign,andc)thescaleofmeasurement.Itisacceptabletouse
statisticsthatareinthebox(es)belowtheappropriatestatistic,butthereisusuallylossofinformationandpower.Itisnotacceptabletousestatisticsabovethe
appropriatebox.
Relatedsamplesdesignsarealsocalledrepeatedmeasures,matchedorpairedgroups,andwithinsubjects.
ChiSquare:Insteadofdifferencequestions,chisquaretestsfortheindependenceoftwovariables.FrequencydataorcountsofthenumberofSsineachcell
orcategoryareusedratherthanrawscoresandmeans.
ANOVAisAnalysisofVarianceorF.

Page205
TABLE13.2
SelectionofanAppropriateInferentialStatisticforBasic,TwoVariable,Associational
QuestionsorHypotheses(forAssociationalApproach)
Level(scale)ofMeasurement
ofBothVariables

Relate

TwoVariablesorScoresforthe
SameorRelatedSubjects

Variablesarebothnormaldataandother
assumptionsnotmarkedlyviolated

Scores

Pearson(r)
Chap16

Bothvariablesatleastordinaldata

Ranks

Spearman(rho)
Chap16

Oneorbothvariablesarenominaldata

Counts

PhiorCramer'sV
Chap16

TABLE13.3
SelectionoftheAppropriateComplex(MorethanOneIndependentand/orDependentVariable)
StatistictoAnswerDifferenceQuestionsorHypotheses(fortheExperimental,
QuasiExperimental,and/orComparativeApproaches)
TwoorMoreIndependentVariables
DependentVariable(s)

AllBetweenGroups

AllWithinSubjects

Mixed(Between&
Within)

Onenormallydistributed FactorialANOVAChap FactorialANOVAwith FactorialANOVAwithrepeated

dependentvariable
17
repeatedmeasuresonall measuresonlastorlast2factors
factors
Chap18
Ordinaldependent
variable

Nonecommon

Nonecommon

Nonecommon

Nominaldependent
variable

Loglinear

Nonecommon

Nonecommon

Severalnormally
distributeddependent
variables

MANOVAChap19

MANOVAwithrepeated
MANOVAwithrepeated
measuresonallfactors
measuresonlastorlastseveral
factorsChap19

TABLE13.4
SelectionoftheAppropriateComplexAssociationalStatisticforthePurposeofPredictingaSingleDependent
orOutcomeVariablefromSeveralIndependentVariables
OneDependent
Variable
Normallydistributed
(continuous)
Dichotomous

SeveralIndependentVariables
AllNormally
Distributed
Multipleregression
Chap19
Discriminantanalysis
Chap19

SomeNormalSome
Dichotomous
Multipleregression
Chap19
Logisticregression
Chap19

AllDichotomous
Multipleregression
Chap19
Logisticregression
Chap19

Page206

usetheSpearmanrankordercorrelation,Rho.Ifoneorbothofthevariablesarenominal,usePhi(ifbothvariableshavetwolevels,a22crosstabulation)or
Cramer'sVforalargercrosstabulation.Table13.2showsonlytwo(PhiandCramer'sV)ofmanyassociationalstatisticsthataskaboutthestrengthofthe
associationbetweentwovariables,whenoneorbotharenominalvariables,forexample,ethnicgroupandvotingpreference.Theuseofthesestatisticsisrelatively
uncommonintheliteraturesowedonotdiscussthemindetail,butPhiandCramer'sVarediscussedbrieflyinchapter16.ThePearsonandSpearmancorrelations
alsoarediscussedinchapter16.
ComplexDifferenceStatistics
Ifyouaskacomplexdifferencequestion(threeormorevariables),appropriatestatisticsareidentifiedinTable13.3.Toselecttheappropriatestatistic,youfirst
decidewhetherthedesignclassificationisbetweengroups,withinsubjects,ormixed.Thenifthereisonedependentvariable,anditisapproximatelynormally
distributed,youareledtooneofthreefactorialANOVAs.TheseANOVAsaresimilarbuthavedifferentformulas,soitisimportanttoknowwhichonetodo.The
betweengroupsfactorialANOVAisdiscussedinchapter17,andthemixedANOVAisdiscussedinchapter18,alongwiththeanalysisofcovariance(ANCOVA).
WedonotdiscussthetwofactorrepeatedmeasuresANOVA,becauseitisnotcommonineducationorinthealliedhealthdisciplines.
Noticethat,unfortunately,therearenocommonordinalstatisticsthatareequivalenttothefactorialANOVAs.Loglinearanalysisissometimesseenintheliterature,
butisnotdiscussedinthisbook.ItissimilartoafactorialANOVAfornominalorcategoricaldataandissimilartoacomplexchisquare.
ThebottomrowofTable13.3showsthreemultivariateanalysesofvariance(MANOVAs)thatparallelthethreefactorialANOVAsbutareusedwhenonewantsto
analyzeseveralnormallydistributeddependentvariablestogetherinsteadofoneatatime.MANOVAcanalsobeusedinsteadofseveralonewayANOVAswhen
thereisoneindependentvariable(singlefactordesign)andseveraldependentvariablesthatyouwanttoanalyzeinoneanalysisratherthanseparately.Chapter19has
abriefdiscussionofMANOVA,itsuse,andassumptions.
ComplexAssociationalStatistics
Ifyouaskacomplexassociationalquestion(twoormoreindependentvariables),appropriatestatisticsareidentifiedinTable13.4.Theseandothercomplex
associationalstatisticsarediscussedinchapter19.NoticethatthelefthandcolumnofTable13.4isdifferentfromtheotherthreetablesinthatordinalandnominal
levelsofmeasurementarenotlisted.Therearenocommonordinalstatisticssimilartothese,but,discriminantanalysescanbeusedwithanominal(threeunordered
levels)dependentvariable.Thatmakesinterpretationmuchmorecomplex.Thetoprowlistsmultipleregression,whichisusedforcasesinwhichtwoormore
independentvariablespredictanormallydistributeddependentvariable.Noticethatmultipleregressioncanbeusedwhenthedependentvariablesarenormally
distributedandwhentheyaredichotomous.Actually,theassumptionof

Page207

normalityformultipleregressionismorecomplexthanindicated.WhentousediscriminantanalysisandlogisticregressionisindicatedinTable13.4.
Occasionallyyouwillseearesearcharticleinwhichadichotomousdependentvariablewasusedinattest,ANOVA,orPearsoncorrelation.Becauseofthespecial
natureofdichotomousvariables,thisisnotwrong,aswouldbetheuseofanominal(threeormoreunorderedlevels)dependentvariablewiththeseparametric
statistics.However,wethinkthatitisabetterpracticetousethesamestatisticswithdichotomousvariablesthatyouwouldusewithnominalvariables.Theexception
isthatitisappropriatetousedichotomous(dummy)independentvariablesinmultipleregressionandlogisticregression(seeTable13.4).
Notshownarefourcomplexassociationalstatisticsthatyoumayseeintheliterature.Themostcommonisfactoranalysis,whichisusuallyusedtoreducearelatively
largenumberofvariablestoasmallernumberofgroupsofvariables.Thesenewcompositevariablesarecalledfactorsorcomponents.Factoranalysisisdiscussed
brieflyinchapter19andalsoinchapter20.
Becausetheyareveryadvancedstatistics,theotherthreearenotdiscussedinthisbook,buttheyarementionedhere.Canonicalcorrelationisacorrelationofa
linearcombinationofseveralindependentvariableswithalinearcombinationofseveraldependentvariables.Pathanalysisisamultivariateanalysisinwhich"causal"
relationshipsamongseveralvariablesarerepresentedbyfiguresshowingthe"paths"amongthem.Structuralequationmodels(SEM)aremodelsthatdescribe
causalrelationshipsamonglatent(unobserved)variables.PathanalysisandSEMarerelatedbothprovidetestsoftheaccuracyoftheproposedmodelandbothare
saidbyproponentstoprovideevidenceofcausallinkagesfromnonexperimentaldesigns.However,theAPATaskForceonStatisticalInferencestatesthat"theuse
ofcomplicated'causalmodeling'softwarerarelyyieldsresultsthathaveanyinterpretationascausaleffects"(Wilkinson&TheTaskForceonStatisticalInference,
1999,p.600).
TheGeneralLinearModel
SomethingthatisnotobviousfromTables13.1and13.2isthatthebroadquestionofwhetherthereisarelationshipbetweenvariablesXandYcanbeansweredtwo
ways.Ifboththeindependentvariableanddependentvariableprovideapproximatelynormallydistributeddatawithfiveormorelevels,theobviousstatistictouse
(fromFig.13.1andTable13.2)isthePearsoncorrelation,andthatwouldbeourrecommendation.However,someresearcherschoosetodividetheindependent
variableintotwoorseveralcategoriesorgroupssuchaslow,medium,andhighandthendoaonewayANOVA.Conversely,otherswhostartwithanindependent
variablethathasafewcategories(saytwothroughfourorderedcategories)maychoosetodoacorrelationinsteadofaonewayANOVA.Althoughthesechoices
arenotwrong,wedonotthinktheyarethebestpractice.Wesaythisbecause,inthefirstexample,informationislostbydividingacontinuousindependentvariable
intoafewcategories.Inthesecondexample,therewouldbearestrictedrange,whichtendstodecreasethesizeofthecorrelationcoefficient.

Page208

Intheprecedingexamples,werecommendedoneofthechoices,butthefactthattherearetwochoicesraisesabiggerandmorecomplexissuethatwehavehintedat
inearlierchapters.Statisticianspointout,andcanprovemathematically,thatthedistinctionbetweendifferenceandassociationalstatisticsisanartificialoneandthat
onewayANOVAandPearsoncorrelationaremathematicallythesame,asarefactorialANOVAandmultipleregression.Thus,thefullrangeofmethodsusedto
analyzeonecontinuousdependentvariableandoneormoreindependentvariables,eithercontinuousorcategorical,aremathematicallyrelated(Keppel&Zedeck,
1989).ThemodelonwhichthisisbasediscalledtheGeneralLinearModelitis"general"inthatthekindofindependentvariableisnotspecified.Theideaisthat
therelationshipbetweentheindependentanddependentvariablescanbeexpressedbyanequationwithtermsfortheweightedvaluesofeachoftheindependentor
predictorvariablesplusanerrorterm.
Whatthismeansisthatifyouhaveacontinuous,normallydistributeddependentoroutcomevariableandfiveorsolevelsofanormallydistributedindependent
variable(notderivedbydividingacontinuousvariable),itwouldbeappropriatetoanalyzeitwitheitheracorrelationoraonewayANOVA.Youshouldgetthe
sameanswerforthesignificancelevel.However,youwouldneedalargesampletohaveenoughparticipantsineachgroupfortheANOVAcomparisonsifyouhave
manymorethanfourlevelsoftheindependentvariable.
Althoughwerecognizethatourdistinctionbetweendifferenceandassociationalparametricstatisticsisasimplification,westillthinkitisusefulheuristically.Wehope
thatthisglimpseofanadvancedtopicisclearandhelpful.Ifnot,youcanprobablyskipthissectionandstillhaveagoodbasicunderstandingofhowtochoose
appropriateinferentialstatistics.
Summary
Thischapterservesasanintroductiontotheselectionofappropriatestatisticalmethods.Inthenextsixchapterswediscussconceptually,andinmoredepth,the
statisticalmethodsshowninTables13.1to13.4.Wetakeexamplesfromjournalsthatpublishresearchinappliedsettingsanddemonstratewhytheauthor(s)selected
aparticularstatisticalmethod.Ourapproachshowsthatthechoiceofaparticularstatisticalmethodisdirectlyrelatedtothegeneraldesignclassification,andthelevel
ofmeasurement.Inaddition,wediscusshowtheresultsofthestatisticalmethodwereinterpreted.
Selectionofanappropriatestatisticrequiresjudgmentaswellasfollowingthedecisionrules.Thisprocesscanbetough,butthisoverviewshouldhavegivenyoua
goodfoundation.Youshouldreviewthischapterwhenyouneedtodecideonastatistictouse.Ifyoudoso,youwillhaveagoodgraspofhowthevariousstatistics
fittogetherandwhentousethem.

Page209

StudyAids
Concepts
Generallinearmodel
Normalityornormallydistributed
Statisticalassumptions
Statisticalindependence
Distinctions
Basic(twovariables)versuscomplex(threeormorevariables)inferentialstatistics
Comparegroupsversusassociatevariables
Dichotomousversusnominalversusordinalversusnormallevelofmeasurement
Differencequestionsversusassociationalquestions
Numberofindependentvariablesversusnumberoflevelsofthevariable
Parametricversusnonparametricstatistics
Withinsubjectsversusbetweengroupsversusmixeddesignclassification
ApplicationProblems
1.Howshouldyoudecideifyourresearchisadifferencequestionoranassociationalquestion?
2.Howshouldyoudetermineifyoushouldusebasicorcomplexstatisticalanalyses?
3.Adecisiontocomparemeans,mediansorcountsisbasedon?Forproblems4to10,createanexampleandthenusethetablesinchapter13toarriveatthe
properstatisticalanalysis.
4.Oneindependentvariable,threelevels,betweengroups,oneordinalnonnormallydistributeddependentvariable.
5.Twobetweengroupsindependentvariables,eachwiththreelevels,onenormallydistributeddependentvariable.
6.Onebetweengroupsindependentvariable,onerepeatedmeasuresindependentvariable,eachwithtwolevels,onenormallydistributeddependentvariable.
7.Oneindependentvariable,twolevels,repeatedmeasures,onenominaldependentvariable.
8.Oneindependentvariable,fourlevels,betweengroups,onenominaldependentvariable.
9.Threenormallydistributedandonedichotomousindependentvariables,onenormallydistributeddependentvariable.
10.Ateacherrankedthe25studentsinherAlgebra1classfrom1=highestto25=lowestintermsoftheirgradesonseveraltests.Afterthenextsemester,she
checkedtheschoolrecordstoseewhatgradethestudentsreceivedfromtheirAlgebra2teacher.Theteacherwantedtoknowifherfinalrankingofthestudentsin
Algebra1hadinfluencedtheirgradeinAlgebra2.

Page211

PARTIV
INTEGRATINGDESIGNSANDANALYSES:
INTERPRETINGRESULTS

Page213

Chapter14
SingleFactorbetweenGroupsDesigns:
AnalysisandInterpretation
Overview
AnalyzingSingleFactorDesignswithParametricStatistics
tTestforIndependentSamples
ConceptualViewofthetTestandOtherParametricTests
UnderstandingthetValue
DegreesofFreedom
DescribingtheResultsofthetTest
AssumptionsUnderlyingthetTest
ASecondExampleofSelectinganIndependentSamplestTest
ConfidenceIntervals
SingleFactorAnalysisofVariance(ANOVA)
MultipletTests?
SourceTablefortheOneWayANOVA
InterpretingtheFValue
ExampleofaPostHocTestFollowingaSingleFactorANOVA
AnalyzingSingleFactorDesignswithNonparametricStatistics
MannWhitneyUTestforIndependentSamples
ExamplesoftheMannWhitneyUTest
KruskalWallisOneWayAnalysisofVariancebyRanks
ExamplesoftheKruskalWallisAnalysisofVariancebyRanks
ChiSquare( 2)TestforIndependencewithNominalData
Summary

Page214

StudyAids
Concepts
Distinctions
ApplicationProblems
Overview
Theexamplesofanalysesforthischapterincludestatisticalteststhatareusedforadesignwithoneindependentvariablethathastwoormorelevels,andall
participantsareinoneandonlyonecondition(betweengroups).Thesinglefactordesignwouldincludestudiesthatusethequantitativerandomizedexperimental,
quasiexperimental,andcomparativeapproaches.Itisimportanttoremember(seeFig.5.2)thattheappropriatestatisticaltestisthesameforeachofthese
quantitativeapproaches.Forexample,considerthreedifferentquantitativeapproaches.
Therandomizedexperimentalapproachusesadesignwhereparticipantsarerandomlyassignedtointerventionandcontrolgroups.Theinterventiongroupreceivesa
formoftreatment,andthecontrolgroupeitherreceivesnotreatment,oratraditionaltreatment.Attheendoftheinterventionperiod,bothgroupsaremeasuredand
comparedonadependentvariable(s).Inthisdesign,thereisoneindependentvariablewithtwolevels.Participantsareinoneandonlyoneconditionorgroup.
Nowconsiderthequasiexperimentalapproachthatusesadesignwhereparticipantsselfassignthemselvesintotheinterventiongroup,andaseparatecontrolgroupis
selected.Attheendoftheinterventionperiod,bothgroupsaremeasuredandcomparedonadependentvariable(s).Again,thereisoneindependentvariablewithtwo
levelsandparticipantsareinoneandonlyoneconditionofthestudy.

Page215

Athirdexampleisthecomparativeapproachthatcomparesparticipantsonanattributeindependentvariable.Onecouldcompareagroupofphysicallyhandicapped
participantswithagroupofparticipantswhoarenotphysicallyhandicappedonamotorskilldependentvariable.Again,thereisoneindependentvariablewithtwo
levelsandparticipantsareinoneandonlyoneconditionofthestudy.
Allthreeoftheseexamplesofquantitativeapproachesfallundertheheadingofsinglefactordesigns.Theinterpretationoftheresultsfromeachoftheseexamples
wouldbedifferentbecausetherigor(relativeinternalvalidity)ofeachdesign,butthestatisticalanalysiswouldbethesame,becauseeachexamplewasasingle
factorbetweengroupsdesignwithtwolevels.Althoughthepreviousexampleshadonlytwoconditions,singlefactordesignsalsoincludeapproacheswithone
independentvariableandmorethantwolevels.Therefore,anyoftheaboveexamplescouldhavehadthreeormorelevelsandstillbeclassifiedasasinglefactor
design.Forexample,astudywithtwotreatmentconditionsandacontrolconditionisstillasinglefactordesign.
Wehavedividedthischapterintoanalysesthatapplytosinglefactordesignsthatuseparametricstatisticsandsinglefactordesignsthatusenonparametricstatistics.
Forthemostpart,thedecisionismadefromthelevelofthedependentvariable.Parametricstatisticsareappropriatelyappliedtodatafromstudieswherethe
dependentvariableisnormallydistributed(seechap.9).Nonparametricstatisticsareusuallyappliedtodatafromstudieswheretheassumptionsforusingparametric
statisticsareviolated,orthescaleofthedependentvariableisordinalornominal.
AnalyzingSingleFactorDesignswithParametricStatistics
Westartouranalyseswithsinglefactor,betweengroupsdesignsperformedondependentvariablesthatareinterval,orratioscale(i.e.,useparametricstatistics),or
both.Figure14.1demonstratesthedecisiontreefortheseanalyses.Wediscusstheindependentsamplesttestfirstandthenthesinglefactor,oroneway,analysisof
variance(ANOVA).
tTestforIndependentSamples
Doesthemethodofquestioninginclassaffecthowstudentsprepareforclass?McDougallandGranby(1996)conductedastudytodetermineifstudentswho

Fig.14.1
Selectingparametricstatisticsforsinglefactorbetweengroupsdesigns.

Page216

werecalledoninclassrandomlywouldprepareforclassinadifferentmannerthanstudentswhoseparticipationwasstrictlyvoluntary.Theresearchersconductedthe
studyinanundergraduatestatisticsclasswith40participants.Themodeofparticipationintheclasswasvoluntaryuntilthemiddleofthesemester.Then,
approximatelyhalfoftheclass(interventioncondition)wasrandomlyassignedtoaconditionofrandomoralquestioning(n=18).Theotherhalfoftheclass(control
condition)continuedinthevoluntaryparticipationmode(n=22).Attheendofthenextclass,allstudentswereaskedtocompletea12iteminstrumentthatmadeup
thedependentvariables.Forourpurposes,wewilllookatoneofthedependentvariables:completionofassignedreading.Thismeasurewasdefinedasthenumberof
pagesfromtheassignedchapterthatstudentshadreadpriortocomingtoclass.Thedesignforthisstudywasaposttestonlycontrolgroupdesign,because
participantswererandomlyassignedtooneandonlyonegroup,andmeasuredjustonce.Thus,theresearchapproachwouldbecharacterizedasrandomized
experimental.TheresultsofthestudycanbeseeninTable14.1.
McDougallandGranbyperformedattestforindependentsamplestodetermineifthetwoconditionsdifferedsignificantlyonthemeasureofnumberofassigned
pagesread.Attestwasselectedtoanalyzethedataforthefollowingreasons(Fig.14.1).First,therewasjustoneindependentvariable,typeofintervention.Second,
therewereonlytwolevels,randomparticipationandvoluntaryparticipation.Third,thedependentvariable,numberofpagesoftheassignedchapterread,wasratio
data.Theresultsofthettestwereexpressedas,''Thedifferencesbetweengroupmeansattainedstatisticalsignificance:t(38)=2.67,p<.02."(p.49).Whatdoes
thismean?
ConceptualViewofthetTestandOtherParametricTests
Thettest(andthedifferentanalysesofvariance[Ftest])isaratioofthevariancebetweengroupsorconditionstothevariancewithinthegroupsorconditions:

Ifthisratioislarge(i.e.,thevariancebetweengroupsisseveraltimesgreaterthanthevariancewithingroups),thenitislikelythattheresultwillbestatistically
significant.Whatdowemeanbythevariancebetweengroupsandthevariancewithingroups?Inthettest,thevariancebetweengroupsisdeterminedfromthe
differencebetweenthemeanoftheinterventiongroupandthe
TABLE14.1
DescriptiveStatisticsfortheMcDougallandGranby(1996)Study
SampleDescription
Numberofparticipants

Intervention
Condition

ControlCondition

18

22

Mean

16.47

9.84

Standarddeviation

5.82

9.09

Page217

meanofthecontrolgroup.IntheMcDougallandGranby(1996)study,themeanoftheinterventiongroupwas16.47pagesread,andthemeanofthecontrolgroup
was9.84pagesread.Therefore,thedifferencebetweenthemeanswas6.63pagesinfavoroftheinterventioncondition.Onewouldexpectthisdifferencebetween
thetwomeanstobelargeifinfacttheinterventionwaseffective.Betweengroupsvarianceisoftenreferredtoastreatmentvariance.Thevariancewithingroupsor
conditionsisthevariabilityordifferencesamongtheindividualparticipantswithineachgroup.Onewouldexpecttheretobedifferencesamongparticipantswithin
groupsbecausetheyaredifferentindividuals.Otherdifferencesmightbeduetoerrorsmadeinmeasurement.Thesizeofthevariancewithingroupscanbeestimated
bycalculatingthestandarddeviationwithineachgroup.Thestandarddeviationsareusedaspartofthecalculationofthewithingroupsvariance,asseeninEquation1
following.Iftherearelargedifferencesamongparticipants,thestandarddeviationwillbelarge.Ontheotherhand,iftherearesmalldifferences,thestandarddeviation
willbesmall.Variancewithingroupsisoftenreferredtoaserrorvariance.McDougallandGranbyreportedthestandarddeviationsforbothoftheirconditions
(Table14.1).
Westatedthatthetstatisticisaratioofbetweengroupsvariancetowithingroupvariance.However,wehaveaspecificequationforthisstatistic.Noticethatinthe
numeratorofEquation1,themeanofthecontrolgroupissubtractedfromthemeanoftheinterventiongroup,yieldingadifferencebetweenmeans.Thisisan
expressionofbetweengroupsortreatmentvariance.Inthedenominatoroftheequation,eachstandarddeviationisdividedbythesamplesize,andthensummed,
yieldingatermcalledthestandarderrorofthedifferencebetweenmeans.Thisisanexpressionoferrororwithingroupsvariance.
Theequationforthetstatisticwhensamplesizesareequalis:

andwhenthesamplesizesareequalornotequalis:

McDougallandGranby(1996)reportedthevalueoftintheirstudyas2.67.TheyarrivedatthisvaluebyusingEquation2becausethesamplesizesintheirtwo
conditionswerenotequal(theinterventionconditionhad18participantsandthecontrolconditionhad22participants):

Page218

Whatdoesatvalueof2.67mean?Howdotheyknowtheresultisstatisticallysignificant?Toanswerthisquestion,weneedtounderstandwhatwemeanby
statisticalsignificance.Whenweusehypothesistesting,wephraseouroutcomeintermsofthenullhypothesis.Specifically,westate,Ifthenullhypothesiswere
true,whatisthelikelihoodthatthisoutcomecouldhappen?Ifthelikelihoodissmall,lessthan5timesin100forexample,wewouldrejectthenullhypothesisand
acceptthealternative,orresearchhypothesis.
UnderstandingthetValue
WhatifMcDougallandGranbyhadrandomlyassignedtheirparticipantstothetwodifferentconditions,buthadnotintervenedinthetreatmentcondition?Thetwo
groupswouldbereceivingthesameconditions.Now,supposetheyperformedatttestonthedata.Wewouldnotexpecttoseeasignificantdifferencebetweenthe
twogroups.Wewouldexpectthevalueofttobezerointhissituation,becausenothingdifferenthasbeendonetotheinterventiongrouptoseparateitfromthe
controlgroupandthegroupsshouldbeequivalentinallrespectsasaresultoftherandomassignment.Therefore,wewouldexpectthemeansofthetwogroupstobe
equal,andifwecomputethedifferencebetweenthem(asinthenumeratorofthetequation),avalueofzerowouldbeexpected.However,anytimewecompare
meansfromtwodifferentgroups,eventhoughthegroupsareequivalentandwehavedonenothingtoeithergroup,themeansprobablywillnotbeexactlyequal
becausethereareindividualdifferencesamongtheparticipants.Thus,whenwecalculateanytstatistic,theoutcomewillbeavaluedifferentfromzero,butusually
closetozero(usuallylessthanone).Whennothingisdonetoeithergroup,theprobabilityishighthatthetvaluewillbeclosetozero,supportingthenullhypothesisof
nodifference.However,whenatvalueislarge,suchasthatfoundintheMcDougallandGranbystudy,theprobabilityislowthattherewasnothingdonetoeither
group.Inthiscase,theprobabilitywaslessthan.02.Placedinstatisticaljargon,theprobabilityislessthan.02thattheoutcomeoftheMcDougallandGranbystudy
couldhappenifthenullhypothesiswereactuallytrue.Therefore,theyrejectedthenullhypothesis.
DegreesofFreedom
Remember,McDougallandGranby(1996)statedtheirresultast(38)=2.67,p<.02.Thenumberinparentheses,38,wasthedegreesoffreedom.Degreesof
freedomreferstothenumberofindependentpiecesofinformationfromthedatacollectedinthestudy.Inthettest,wefindthedegreesoffreedomfromthenumber
ofparticipantsineachgroupminus1.Intheinterventiongroup,therewere18participants,sowesubtract1from18toget17degreesoffreedom.Inthecontrol
group,therewere22participants,sowesubtract1from22toget21degreesoffreedom.Then,weadd17and21toget38degreesoffreedom,asstatedinthe
study.

Page219

Whydowesubtractonefromthenumberofparticipantsineachgrouptoarriveatdegreesoffreedom?Degreesoffreedomareindependentpiecesofinformation.In
eachgroup,eachparticipant'sscoreisanindependentpieceofinformation.Thedatacollectedfromoneparticipantarenotdependentonthedatacollectedfroma
secondparticipant.Thedatafromthetwoparticipantsareindependent.Therefore,intheMcDougallandGranbystudy,itappearsthatwehave40independent
piecesofinformation.However,whenwecomputethetvalue,asinEquation1orEquation2,weusetheinformationfromthestandarddeviationofeachgroup
(chap.9).Tocomputethestandarddeviation,wesubtractthemeanofeachgroupfromeachindividualscoreinthegroup.Thisgivesusadeviationscore(deviation
fromthemean).Ifweweretoaddthesescores,theywouldequalzero.Becausethedeviationscoresaddtozero,ifweknowthemeanofagroup,andallbutoneof
thescoresinthatgroup,wecouldfigureoutthevalueofthatmissingscore.Whenweusethettesttocalculateatvalue,oneofthescoresineachgroupisnot
independent.Therefore,wesubtract1fromthesamplesizeineachgrouptodetermineourdegreesoffreedom(Equation3),following:

DescribingtheResultsofthetTest
McDougallandGranby(1996)concludedfromtheresultsoftheirstudythattheindependentvariable(methodofinclassquestioning)affectsthedependentvariable
(preparationforclass).Specifically,theynotedthat"studentswhoexpectedrandomoralquestioningalsoreportedcompletingmoreoftheassignedreadingsbefore
class"(p.51).Althoughtheyusedastrongexperimentaldesign,theauthorsdidnotrefertocauseintheirdiscussion,buttheycouldhave,aswediscussedinchapters
5and7.
AssumptionsUnderlyingthetTest
Therearethreemajorassumptionsunderlyingtheuseofthettestforindependentsamples.ThesethreeassumptionsalsoapplytothebetweengroupsANOVA.
Otherassumptionswillbeaddedwhenwediscussrepeatedmeasuresdesigns.
1.Normality:Thedependentvariableshouldbenormallydistributedforeachofthepopulationsfromwhichthesampleswereselected.
2.Homogeneityofvariance:Thevariancesofthedependentvariablesofthepopulations(

and

A2

)underlyingthesamplesshouldbeequal.

B2

3.Independence:Measuresrecordedfromparticipantsmustbeindependentofeachother.Inotherwords,theperformanceofoneparticipantisnotaffectedbythe
performanceofanotherparticipant.
HowimportantaretheseassumptionsforuseofparametricstatisticssuchastandFtests?Althoughthereissomedisagreementamongauthorsinthefieldof
statistics,wetendtoagreewithKerlinger(1986,p.267),whosuggestedthatviolationoftheassumptionsofnormalityandhomogeneityofvarianceare

Page220

overrated.Violationoftheindependenceassumptionisseriousandshouldnotbetolerated,becauseindependenceunderliestherationaleofmoststatisticaltests.In
fact,manyresearchersappeartopaylittleattentiontotheseviolations.Whenthefirsttwoassumptionsunderlyingparametrictestsaremoderatelyviolated,the
researchercanignoretheviolationsbecausetandFare"robust"(i.e.,notaffectedbyviolations).However,iftheseassumptionsaremarkedlyviolated,youshould
selectanonparametrictestormakeanadjustment.Oneproblemwithchoosinganonparametrictestisthatthesetestsarelessflexibleandgenerallylesspowerfulthan
theirparametricequivalents.Powerisdefinedastheprobabilityofrejectingafalsenullhypothesis,whichiswhatwehopetodo.
ASecondExampleofSelectinganIndependentSamplestTest
Doeschronicdisabilityaffectmemoryperformance?Tun,Tun,andWingfield(1997)conductedastudybyusingyoungandoldadultswithlongtermspinalcord
injury(SCI).Theauthorsusedbothobjectiveandsubjectivemeasurestoassessmemoryperformance.OneofthesubjectivemeasureswastheDividedAttention
Questionnaire(DAQ).Onthisquestionnaire,participantsratedfrom1(veryeasy)to5(verydifficult)theirabilitytodivideattentionbetweenconcurrentactivities.
Therewere15combinationsofactivities.Theparticipantsratedeachactivityondifficulty,change,andfrequency.Wewilllookatthedifficultypartofthescale.
Fortysixparticipantswithspinalcordinjurywereincludedinthestudy.Tunetal.(1997)dividedtheparticipants,accordingtoage,intotwogroupsof23participants
each.Themeanoftheyoungergroupwas39.9years,andthemeanoftheoldergroupwas67.1years.ThedataforthedifficultymeasureoftheDAQforthisstudy
canbeseeninTable14.2.
Tunetal.(1997)selectedattestforindependentsamplestoanalyzetheirdata.Therationalebehindthisdecision(seeFig.14.1)wasthatthestudyinvolvedone
betweengroupsindependentvariablewithtwolevels.Thedependentvariablewasscaledattheintervallevel.Noticethatthisstudywouldfitintothecategoryofthe
comparativeresearchapproachbecausetheindependentvariable,age,wasanattributeindependentvariable.However,eventhoughthisstudyusedthecomparative
approachandtheMcDougallandGranby(1996)studyusedarandomizedexperimentalapproach,bothstudieswereanalyzedwithanindependentsamplesttest.
Tunetal.usedtheformulaforthettestforindependentsampleswithequalsamplesizes(seeEquation1):
TABLE14.2
DescriptiveStatisticsfortheTunetal.(1997)Study
SampleDescription

YoungSCIParticipants

OldSCIParticipants

Numberofparticipants

23

23

Meandifficultyrating

2.3

2.5

Standarddeviation

1.0

.7

Page221

Tunetal.(1997)reported,"Therewasnoagedifferenceinscores...ontheDifficultyScaleoftheDividedAttentionQuestionnaire,t(44)<1,indicatingthatolder
adultswerenotmoreconcernedthantheyoung...withproblemsindividingattention"(p.171).
ConfidenceIntervals
RecallthestudybyMcDougallandGranby(1996).Thedataobtainedfromthatstudyweresubjectedtoanalysisbyattestforindependentsamples.Thettest
demonstratedthatthenullhypothesisshouldberejected.Statisticallyspeaking,wecouldsaythatthetwogroupscamefrompopulationswithdifferentmeans,onea
populationofpeoplewhoreceivedtherandomoralquestioningconditionandtheotherapopulationofpeoplewhoreceivedthevoluntaryparticipationmode.
Practicallyspeaking,wemightsaythatwecanbeconfidentthatthoseintherandomoralquestioningconditionreadmorepagesofthechapteroutsideofclassthan
didthoseinthevoluntaryparticipationcondition.
Whenwespeakofsomethingasbeingsignificantlydifferentfromsomethingelse,oronegroupperformingsignificantlybetterthananothergroup,thereisadanger
fromaninterpretationstandpointthatsignificanceindicatesallornothingoralargedifference.Furthermore,weoftenforgetabouttheactualdataandhowmuchofa
differenceactuallyexists.Theuseofstatisticsimpliesprobabilities.Whenthenullhypothesisisrejectedatan levelof.05,wearesayingthattheprobabilityisless
than5timesin100,givenatruenullhypothesisthatthiscouldhavehappenedbychance.Rejectingthenullhypothesisdoesnotmeanthedifferencecouldnothave
happenedbychance.Ratherthanexpressingtheresultsofastudyasasignificancestatement(i.e.,rejectingornotrejectingthenullhypothesis),statisticiansoften
expressresultsasaconfidenceintervalaroundthedifferencebetweensamplemeans.Ratherthansayingthatwerejectthenullhypothesis,wesaythatwith95%
confidencethetruepopulationmeandifferenceiswithintheconfidenceinterval.Whatthistellsusisthatifweweretoselect100samples,andestablishaconfidence
intervalforeachofthesamples,thetruepopulationmeandifferencewouldbein95oftheconfidenceintervalsbutwouldnotbein5oftheconfidenceintervals.
WecanusethedatafromMcDougallandGranby(1996)toestablisha95%confidenceinterval.Wecanconstructthisintervalbyusinginformationprovidedtous
fromperformingthettest.Toconstructconfidenceintervalsweneedtoknowthedifferencebetweenthesamplemeans(numeratorofthetequation),theestimated
standarderrorofthedifferencebetweenmeans(denominatorofthetequation),andthetvalueassociatedwiththetwotailedsignificancelevelthatwehaveselected
at =.05(thisvalueisoftenreferredtoas

Page222

thecriticaltvalueandisobtainedfromattable).Alloftheinformationneededfordeterminingconfidenceintervals(withtheexceptionofthetwotailedsignificance
level)wasobtainedfromperformingthettest.The95%confidenceintervalfortheMcDougallandGranby(1996)studyisbetween1.6pagesand11.6pages.
Practicallyspeaking,wecanbeconfidentthattherandomparticipationtechniquewillincreasestudents'readingoutsideofclassbybetween1.6and11.6pages.
SingleFactorAnalysisofVariance(ANOVA)
Inourpreviousexamples,wehadoneindependentvariablewithtwolevels.Wenowintroducethesinglefactoranalysisofvariance(ANOVA).Thisanalysisismostly
usedinbetweengroupdesignswithoneindependentvariableandmorethantwolevels.Itshouldbenoted,however,thatthesinglefactoranalysisofvariancealso
canbeusedinsinglefactorbetweengroupsdesignswithonlytwolevels,similartothettest(Fig.14.1).Thedatashouldbescaledattheintervalorratiolevel.One
normallywouldchoosethettestifadirectionalhypothesisismade,becausetgivestheresearchermorepower(probabilityofrejectingafalsenullhypothesis).Ifa
nondirectionalhypothesisisselectedforthisdesign,theneitherattestorasinglefactorANOVAcouldbeusedbecauset 2=Fwhenthereareonlytwolevels.Our
presentationofthesinglefactorANOVAwillbewithmorethantwolevelsorgroups.
OneoftwostudiesperformedbyTuckman(1996)investigatedincentivemotivationincollegestudents.Tuckmancomparedanincentivemotivationconditiontoa
learningstrategyconditiontodetermineifincentivemotivationwouldincreaseperformanceona50itemmultiplechoicetestthatmeasuredcomprehensionratherthan
recall.Tuckmanusedthreedifferenteducationalpsychologyclassescomposedofjuniorandseniorstudentswhowantedtobecometeachers.Tuckmandetermined
thatthethreeclasseswerenotdifferentwithrespecttoage,gender,andscoresonareadingsubtest.Thenherandomlyassignedtheclassestooneofthreeconditions
(astrongquasiexperimentaldesign).Thethreeconditionswereanincentivemotivationconditionthatinvolvedtakingaquizpriortoeachclasssession.Thequiz
gradeswouldcountaspartoftheirgradefortheclass.Thesecondconditionwasalearningstrategycondition.Forthiscondition,thestudentsperformedhomework
assignmentstoidentifykeytermsinthetextbook.Thethirdclasswasacontrolgroup.Theyreceivedstrictlylecturesonthechapters.Attheendof5weeks,the
studentstookthe50itemmultiplechoicetest.Thedependentvariablewasthepercentageofcorrectresponsesonthetest.Thismeasurewasscaledattheintervalor
ratiolevel.Tuckmananalyzedthedatabyusingasinglefactoranalysisofvariance.Thereasonforthischoice(Fig.14.1)wasthattherewasoneindependentvariable
withmorethantwolevels.Theindependentvariablewasabetweengroupsindependentvariable.Thedependentvariablewasscaledattheintervalorratiolevel.
MultipletTests?
Howshouldweproceedifwehaveabetweengroupsdesign,adependentvariablethatisscaledattheintervalorratiolevel,andonein

Page223

dependentvariablewiththreelevels?Thissituationisnotuncommoninresearch.Actually,weinfrequentlyhavetwolevelswithinoneindependentvariable.One
answertotheanalysisquestioncouldbetoperformthreettests.Wecouldcomparetheincentivemotivationgrouptothelearningstrategygroup,thelearningstrategy
grouptothecontrolgroup,andtheincentivemotivationgrouptothecontrolgroup.Allpossiblecomparisonswouldbeconsidered.Theproblemisthatthethreet
testsarenotindependentfromeachother.Inotherwords,oncewedeterminethattheincentivemotivationgroupisgreaterthanthelearningstrategygroup,andthe
learningstrategygroupisgreaterthanthecontrolgroup,thenthethirdpieceofinformation,thattheincentivemotivationgroupisgreaterthanthecontrolgroup,is
alreadyknown.Inthefieldofstatistics,independenceisheldinhighregard.TheresultofdoingmultiplettestsisthattheprobabilityofatypeIerror,whichisset
at.05foreachhypothesis,increasessubstantially.
Whenweexaminethesinglefactoranalysisofvariance,wearecomparingthevariabilitybetweengroups(numerator)tothevariabilitywithingroups(denominator).
Previously,whendiscussingthettest,ourvariabilitywithingroupshadbeenexpressedinstandarddeviationunits(estimatedstandarderrorofthedifferencebetween
means),andourvariabilitybetweengroupshadbeenexpressedasadifferencebetweenmeans.Nowwearegoingtoworkwithvariances.(Rememberthatvariance
isthesquareofthestandarddeviation.)TheANOVAisusedtocomparethevariabilitybetweengroups(treatmentvariance+error)tovariabilitywithingroups
(error).
SourceTablefortheOneWayANOVA
Althoughtheresultsofattestareusuallysummarizedinonesentenceaspreviouslyreported,itiscommonfortheresultsofanANOVAtobesummarizedinasource
table.Westartwiththesummaryorsourcetable(Table14.3).Thistableshowshowthevarianceanddegreesoffreedomareassignedintheanalysisofvariance.It
alsodemonstrateshowtheFstatisticisdetermined.TheFstatisticisthesummarystatisticforalloftheanalysesofvariance.Forthedegreesoffreedom,k=the
numberoflevels(groups)oftheindependentvariable,andN=totalnumberofparticipants.
TheprocedureforcalculatingtheFforthesinglefactoranalysisofvarianceistocalculateatotalvarianceforthestudy.Thisvarianceisthendividedintovariancethat
resultsfromthetreatment,referredtoasbetweengroupsvari
TABLE14.3
AnalysisofVarianceSourceTable
SumsofSquares(SS)

Degreesof
Freedom(df)

SourceofVariation
Betweengroups

SSbetweengroups.

Withinsubjects

SSwithinsubjects

Total

SSTotal

k1

MeanSquares
(MS)
MSbetweengroups

MSbetweengroups

MS.withinsubjects

Nk

MSwithinsubjects

N1

Page224

ance,andvariancethatresultsfromerror,referredtoaswithinsubjectsvariance.Tomakethecalculationseasier,sumsofsquaresarecalculatedfirst.Thenthis
valueisdividedbythedegreesoffreedom.Thenewvalues,meansquares,arethevarianceestimatesforbetweengroupsandwithinsubjects.Themeansquare
betweengroupsisthendividedbythemeansquarewithinsubjects,yieldingtheFvalue.Let'sexaminethesourcetableofasinglefactoranalysisofvariancebefore
returningtotheTuckman(1996)study.
Durr,Guglielmino,andGuglielmino(1995)wereinterestedintherelationshipbetweendifferentoccupationalcategoriesatalargemanufacturingfirmandreadinessfor
selfdirectedlearning.TheymeasuredreadinessforselfdirectedlearningusingtheSelfDirectedLearningReadinessScale(SDLRS),a58itemsummatedLikert
scale.Therewerenineoccupationalcategoriesatthefactory.Thisisanexampleofthecomparativeresearchmethodology,becausetheindependentvariable,typeof
occupationalcategory,isanattributeindependentvariable.Thesecategories,thenumberofparticipantsineachcategory,andthemeanscoreontheSDLRSforeach
categorycanbeseeninTable14.4.
Durretal.(1995)performedasinglefactoranalysisofvarianceonthedatabecausetheyhadoneindependentvariable,typeofoccupationalcategory,withnine
levels.Thedesignwasbetweengroupsbecauseparticipantscouldbeclassifiedintooneoccupationalcategoryonly.Thedependentvariable,intheformofaLikert
scale,wasassumedtobescaledattheintervallevel.TheresultsofthisanalysiscanbeseeninTable14.5.
TABLE14.4
MeansandSampleSizesfortheDurretal.(1995)Study
OccupationalCategory

NumberofParticipants

MeanScoreonSDLRS

Manufacturing/factory

16

220

Clerical/administration

107

223

Engineers

114

233

Sales

98

242

Supportprofessionals

43

231

Manufacturingmanagers

237

Supportmanagers

63

238

Engineeringmanagers

81

236

Salesmanagers

44

246

TABLE14.5
SourceTableforComparisonofNineOccupationalCategoriesintheDurretal.(1995)Study
SumsofSquares
(SS)

DegreesofFreedom
(df)

MeanSquares
(MS)

Betweengroups

28,855

3,607

6.39

Withinsubjects

325,045

576

564

Total

353,901

584

SourceofVariation

Page225

Thesumsofsquaresforbothbetweengroupsandwithinsubjectsarecalculatedfirst.Thenthedegreesoffreedomaredetermined.Noticethatthereare8degreesof
freedomassociatedwiththebetweengroupsvariance.Thisisbecausetherewereninegroupsinthestudy,andfromTable14.3,degreesoffreedomforbetween
groupsvariancearedeterminedbysubtractingthenumberofgroupsminusone.Thedegreesoffreedomforwithinsubjectsvariancearedeterminedbysubtractingthe
numberofgroupsfromthetotalnumberofparticipants.Therewere585participantsandninegroups,whichyields576degreesoffreedom.Themeansquaresare
determinedbydividingeachsumsofsquaresbytheaccompanyingdegreesoffreedom.ThelaststepistodeterminetheFvaluebydividingthemeansquarebetween
groups(treatmentvariance)bythemeansquarewithinsubjects(errorvariance).ThisyieldsanFvalueof6.39.
InterpretingtheFValue
WhatdoesanFof6.39mean?Durretal.(1995)foundthisvaluetobesignificantataprobabilityof.000.Inotherwords,theprobabilitythattheoutcomeofthe
study(theninedifferentmeanvalues)couldhappenbychance,assumingatruenullhypothesisislessthan1in1,000,orhighlyunlikely.Therefore,Durretal.rejected
thenullhypothesisofnodifferenceamongsamplemeans.Theyconcludedthat,becausethehighestscoresinthemanagementcategorywerethoseofsalesmanagers,
thesescoresweresignificantlyhigherthanalloftheothercategories.Thisconclusionisnotnecessarilycorrectstatistically.AsignificantFvaluefromananalysisof
varianceonlytellsyouthatthereisatleastonesignificantdifferenceamongallofthelevelsorgroups.Itdoesnotinformtheresearcherwhetherthereareother
significantdifferences.TodeterminewhichgroupsorconditionsaresignificantlydifferentfromeachotherfollowingasignificantF,aposthoctestmustbedone.The
onlyexceptiontothisruleisadesignwithoneindependentvariableandonlytwolevelsorgroups.
Therearenumerousposthoctestalternativesfromwhichtochoose.SomeofthemostcommonareFisher'sleastsignificantdifferencetest(LSD)(R.A.Fisher,
1935),Tukey'shonestlysignificantdifferencetest(HSD)(Tukey,1953),NewmanKeulstest(seeKirk,1982),Scheffe'stestforallcomparisons(Scheffe,1953),
andDunnett'stest(Dunnett,1955).Huck,Cormier,andBounds(1974)rankposthoccomparisonsfromliberaltoconservative,dependingonhowdifficultitisto
findasignificantdifferencebetweensamplemeans.TheFisherLSDtestisthemostliberal,whereastheScheffetestisthemostconservative.Werecommendagainst
usingtheLSDtest(isthisthesixties?),asitistooliberal(increasedprobabilityofmakingatypeIerror).TheTukeyHSDtestandtheNewmanKeulstestare
consideredmiddleoftheroadtestsbetweenliberalandconservative.TheDunnetttest(Dunnett,1955)isaposthoctesttobeusedwhencomparingacontrolgroup
toeachoftheothertreatmentgroups.Figure14.2showsthestepstobefollowedindecidingwhetheraposthoctestisneeded.
ExampleofaPostHocTestFollowingaSingleFactorANOVA
LetusreturntotheTuckman(1996)study.Therewerethreegroupsinthisstudy,anincentivemotivationgroup,alearningstrategygroup,andacontrolgroup.The

Page226

dependentvariablewasthepercentageofitemscorrectona50itemachievementtest.ThedataforthisstudycanbeseeninTable14.6.
Tuckman(1996)performedasinglefactorANOVAonthedatafromhisstudyandreportedtheoutcomefromthisanalysisas,''Theanalysisofvariance(ANOVA)
forthedifferencebetweenthethreegroupmeansyieldedF(2,106)=21.69,p<.001"(p.202).First,noticethatTuckmandidnotpresentasourcetable,but
reportedhisanalysisinthetextofthearticle.Thisiscommon,especiallywhenreportingtheresultsfromasinglefactorANOVA.Inhisstatement,thedegreesof
freedomareinparentheses,similartothereportingofthettest,excepttherearetwodifferentdegreesoffreedomforthesinglefactorANOVA.Inthecurrentstudy,
thereweretwodegreesoffreedomforthebetweengroupsvariance(numberofgroupsinthestudyminusone)and106degreesoffreedomforthewithinsubjects
variance(numberofparticipantsinthestudyminusthenumberofgroups).Theresultwassignificantataprobabilityoflessthan.001,indicatingaverylowlikelihood
thatthisresultcouldhappenby

Fig.14.2.
SchematicrepresentationofwhentouseposthocmultiplecomparisonswithaonewayANOVA.
TABLE14.6
DescriptiveStatisticsfortheTuckman(1995)Study
SampleDescription

IncentiveMotivation

LearningStrategy

Control

Numberofparticipants

36

35

38

Mean

82.8%

71.6%

66.9%

Standarddeviation

9.3

9.4

12.6

Page227

chance.AsignificantFindicatesthatthereisatleastonesignificantdifferenceamongthegroupsinthestudy.However,todeterminewhichgroupsweresignificantly
different,aposthoctestmustbeperformed.
Tuckman(1996)performedaposthoctestbyusingaNewmanKeulsprocedure.Theresultsofthistestrevealedthattheincentivemotivationgroupperformed
significantlybetterthaneitherthelearningstrategygrouporthecontrolgroup,butthelearningstrategygroupwasnotsignificantlydifferentfromthecontrolgroup.
AnalyzingSingleFactorDesignswithNonparametricStatistics
MannWhitneyUTestforIndependentSamples
TheMannWhitneyUtest,anonparametrictest,isdonewhenthedesignisabetweengroupsdesignwithoneindependentvariableandtwolevels.However,either
therehasbeenaviolationoftheassumptionsunderlyingthettest,mostoftentheassumptionofnormality,orthedataareordinalandnotnormallydistributed.
Nonparametricanalysesoftenarereferredtoasdistributionfreeanalyses.ParametrictestssuchasthetandtheFrequirethatthedatabenormallydistributed.
Nonparametrictestsarefreeofthenormaldistribution.Actually,eachnonparametricanalysishasitsownsamplingdistribution.Thewordnonparametricindicates
thattheanalysesdonotusetheparameters and (populationmeanandstandarddeviation)fromthenormaldistribution.
Ineducation,alliedhealth,andappliedsocialsciences,therearenumerousexamplesofscales,suchasindependencescales,whereitwouldbepresumptuousto
assumethatthedataareinterval.Forexample,ifwerateindependenceonascalefrom1to3,anddescribeascoreof1asbeingabletoperformnoactivities,a
scoreof2asbeingabletoperformafewactivities,andascoreof3asbeingabletoperformalotofactivities,theintervalsbetween1and2,and2and3donot
seemtobeequal.Therefore,itwouldbemoreappropriatetousenonparametricanalyses.Theonesituationwhereanonparametrictestshouldneverbeusediswhen
aparametrictestalreadyhasbeenusedandtheoutcomehasfailedtoleadtoadecisiontorejectthenullhypothesis.Theuseofanonparametrictestinplaceofa
parametrictestwillnotincreasepower,ortheprobabilityofrejectingafalsenullhypothesis.Nonparametrictests,likeparametrictests,havetheirproperplace,but
typicallytheyarenotaspowerful.
Althoughwedonotgointodetailaboutthenonparametricteststhatwemention(seeSiegelandCastellan,1988),mostofthetestsusearankingprocedureforthe
dataandthenapplyaparticularformulatothedata.Toperformtherankingprocedure,allofthedatafromallofthegroupsareconvertedtoranks,byorderingfrom
thesmallestnumbertothelargestnumber,regardlessoftheparticulargrouporcondition.Oncethedataareranked,therankingscanbeusedinanequation.Usually
thisentailssummingtherankingsfromeachgroup.Asonemightexpect,ifthesumsoftherankingsareverydifferentbetweenoramonggroups,thentheyare
probablysignificantlydifferent.Wecanlookuptheresultoftheequationsinatableanddrawaconclusionbyusingthesamelogicasthettest.

Page228

ExamplesoftheMannWhitneyUTest
AcommonsituationwheretheMannWhitneyUtestisappliedoccurswhenthereareasmallnumberofparticipants,andthesamplesizesarenotequal.Janelle
(1992)studiedthelocusofcontrolin13nondisabledand8congenitallydisabledadolescentsbyusingtheNowickiStricklandLocusofControlScaleforChildren.
Thetypeofquantitativeapproachforthisstudywascomparative,becausetheindependentvariable,presenceofadisability,wouldbeconsideredanattribute
independentvariable.Thedesignforthisstudyhadoneindependentvariable,presenceofdisability,withtwolevels.Also,itwasabetweengroupsdesign.Therefore,
attestforindependentsamplesoraMannWhitneyUtestcouldhavebeenselectedbecausethedatawereinterval.Janelleselectedanonparametrictest,
presumablyasaresultofthesmallsamplesizeandtheunequalsizesofthetwogroups(Fig.14.3).Shereportedherfindingsas,"TheMannWhitneyUtest(U=
58.50,p=.635),howeverdidnotindicateastatisticallysignificantdifferencebetweenthetwogroups"(p.337).Janelleconcludedthatdisabilityisnotasignificant
factorinexternalityinadolescents.
TheMannWhitneyUtestisalsousedwhenthedependentvariabledatadonotappeartobenormallydistributed.Czerniecki,Deitz,Crowe,andBooth(1993)
comparedboyswithgirlsaged18to23monthsonfourmeasuresofattendingbehavior,totaltimeattending,totalnumberofactivitiesattendedto,averageattending
timeperactivity,andlongesttimeattendingtooneactivity.Again,thisstudyisanexampleofthecomparativequantitativeapproachbecausetherewasoneattribute
independentvariable,gender,withtwolevels.Theauthorsstatedthat,"becausevisualinspectionoftheresultingtablesandgraphsrevealedthatmuchofthedatawere
notnormallydistributed,theMannWhitneyU,anonparametricstatistic,wasemployedtotestthehypothesis"(p.712).Ofinterestinthisstudywasthatasecond
hypothesiswasalsotested,comparingchildren18to20monthswiththose21to23months.Again,aMannWhitneyUtestwasselected.
KruskalWallisOneWayAnalysisofVariancebyRanks
TheKruskalWallisOneWayAnalysisofvariancebyranksisthenonparametricanalogueofthesinglefactorbetweengroupsANOVAwiththreeormorelevels.It
isusedwhenthereisoneindependentvariablewithmorethantwolevelsparticipantsareinoneandonlyonegroupandeithertherehasbeen

Fig.14.3.
Selectionoftheappropriatenonparametricstatisticforasinglefactor
betweengroupsdesignwithanordinalornominaldependentvariable.

Page229

aviolationoftheassumptionsforparametricstatistics,orthedataareordinalandnotnormallydistributed,orboth.ThepreliminarystepsintheKruskalWallis
analysisofvariancearesimilartothatoftheMannWhitneyUtest.Dataarerankedfromsmallesttolargestwithoutrespecttogroup.Thenthegroupsaresummed
andappliedtotheequation.SimilartoasinglefactorANOVA,asignificantKruskalWallistestmustbefollowedbyaposthoctestbecausethereareatleastthree
levels.
ThelogicunderlyingtheKruskalWallisANOVAisthatifyouhadthreeidenticaldistributionsofscoresandyouselectedthreegroupsatrandom,onefromeach
distribution,youwouldexpecttheirrankstobedistributedequallyunderthenullhypothesis.However,iftheranksareverydifferentforatleastoneofthegroups,
thenthenullhypothesiswouldberejected.
ExamplesoftheKruskalWallisAnalysisofVariancebyRanks
MorganandGriego(1998)provideanexamplefromthe"highschoolandbeyond"dataoftherelationshipbetweenfathers'educationandmathematicsachievement
scores(p.196).Therewerethreelevelsoffathers'education(highschoolgraduateorless,somecollege,orBachelorofSciencedegreeormore).Thescaleofthe
dependentvariable,mathematicsachievement,wasconsideredtoberatio.Therefore,thedesignwasasinglefactordesignwiththreelevels.Thedatacanbeseenin
Table14.7.
Normally,thecorrectstatisticalselectionwouldbeasinglefactoranalysisofvariance(Fig.14.1).Notice,however,thatthesamplesizesaredifferent.Thestandard
deviationsalsoaredifferent,withthoseintheBachelorofSciencedegreeconditionhavingastandarddeviationalmosttwicethatofthoseinthesomecollege
condition.MorganandGriego(1998)determinedthatthedataviolatedtheassumptionofhomogeneityofvariancebyusingaprocedurecalledLevene'stest.
Therefore,theyperformedaKruskalWallisOneWayAnalysisofVariancebyRanks,andfoundtheresultstobestatisticallysignificant(p=.001).
AlthoughitismorecommontousetheKruskalWallisstatisticaltestbecauseatleastoneoftheassumptionsunderlyingparametrictestsisviolated,thetestalsocan
beusedwhenthedataforasinglefactordesignareordinal.Tate(1997)conductedastudytodetermineifpeppermintoilreducedpostoperativenauseain
gynecologicalsurgicalpatients18patientswhounderwentmajorsurgerywereassignedtooneofthreegroups,acontrolgroupthatreceivednotreatment,aplacebo
group,andapeppermintoilgroup.Eachparticipantfilledoutascalethattheauthorsdescribedasa"standardizeddescriptiveordinalscaletocollectthesubjective
patientselfreporteddata"(p.545).The
TABLE14.7
ComparisonofMathAchievementScoresforThreeFathers'EducationGroups
SampleDescription

HighSchoolGraduate
orLess

SomeCollege

BachelorofScience
DegreeorMore

Numberofparticipants

38

16

19

Mean

10.09

14.40

16.35

Standarddeviation

5.61

4.66

7.41

Page230

scalecanbeseeninFig.14.4.Notethatmanyresearcherswouldconsiderthisratingscaletobeatleastapproximatelyintervalbecausethecategories(04)arenot
onlyordered,butatleastapproximatelyequallyspaced.Ifso,theappropriatestatisticwouldhavebeenaonewayANOVA.However,KruskallWallisisa
reasonableandconservativechoice.
Theauthorsfoundastatisticallysignificantdifferenceamongthethreegroups(p=.048).Theauthorsperformedaposthoctesttodeterminewherethesedifferences
occurred,asmustalsobedoneinasignificantsinglefactoranalysisofvariancewithmorethantwolevels.TheyusedtheMannWhitneyUtestforthethreeposthoc
comparisonsandfoundthatthesignificantdifferenceswerebetweenthepeppermintoilgroupandtheplacebogroup.TheMannWhitneyisappropriateasaposthoc
testbutisalsoliberal,beinganalogoustotheFisherLSDtestmentionedearlier.
Chisquare(c 2)TestforIndependencewithNominalData
Inthispartofthechapterweareworkingwithcategoricaldata.Specifically,wearedetermininghowmanypeople(thefrequencycount)fallintoaparticularcategory,
relativetoadifferentcategory.Weexaminedata(nominaldata)thatidentifiesapersonasbelongingtoaparticulargroup.Forexample,wemaybeinterestedinthe
relationbetweengenderandpoliticalaffiliation.OneparticipantmaybefemaleandbelongtotheDemocraticParty.Thenextparticipantmaybemaleandbelongto
theDemocraticParty.ThenextparticipantmaybefemaleandbelongtotheRepublicanParty.Intheseexamples,itwasdeterminedthroughmeasurementthatno
personhadmoreorlessofsomethingthananotherperson.Eachparticipantwasonlyidentifiedasbelongingtoaparticulargrouporcombinationofgroups.
Whenmeasuringatthenominalorcategoricallevel,theamountofinformationgainedislessthanwhenmeasuringattheintervalorordinallevels.Forexample,
supposethatyouaretakingacourse,Statistics1A,attheuniversityandyou(withtheinstructor'sapproval)decidetobegradedpass(P)orfail(F).Let'sassumethat
youpassedtheclass.Now,supposethatareviewcommitteeexaminesatranscriptofyourcoursework.Becausetheyareinterestedinyourbackgroundinstatistics,
theylookforastatisticscourseonyourtranscript.ThegradeofPonlyshowsthatyoupassedthecourse.YoucouldhavedoneA,B,C,orDwork.Therefore,even
ifyoudidAlevelwork,thatinformationhasnotbeentransmitted.Thefinerthegradationsonthemeasurementscale,themoreinformationthatistransmitted.We
recommendthatyounotdivideyour
0

Iamnot
experiencingany
nausea

Ifeelslightly
nauseated

Ifeelmoderately
nauseated

Ifeelextremely
nauseated

IamsonauseatedIfeelI
amabouttovomit

Fig.14.4.
Anordinalratingscale.From"PeppermintOil:ATreatmentforPostoperativeNausea,"byS.Tate,1997,
JournalofAdvancedNursing,26,p.545.Copyright1997byBlackwellSciencesLtd.AdaptedwithPermission.

Page231

dataintotwoorafewcategoriesifthedataarecontinuousorhaveanumberoforderedcategories.However,sometimesyourdataonboththeindependentand
dependentvariablesarenominalorcanbestberepresentedbyafewcategories.Thatiswhenyouuseachisquaretest.
Thechisquaretestforindependenceteststheassociationbetweentwovariables,anindependentvariableandadependentvariable.Testingarelationshipbetween
anindependentvariableandadependentvariableisnodifferentwhenusingachisquaretestthanitiswhenusingattest,exceptthatthedependentvariableis
categoricalratherthancontinuous.Underthenullhypothesis,theindependentanddependentvariablesareassumedtobeindependentofeachother.
Therequirementsforachisquaretestofindependenceareoneindependentvariablewithatleasttwolevelsorcategoriesandonedependentvariablethatisalso
categorical.Thedata(frequencies)aremeasuredatthenominalorcategoricallevel.Thedesignisbetweengroupssothateachparticipantcanbeinonlyonegroup.
Gliner,Haber,andWeise(1999)evaluatedacurriculumchangeinoccupationaltherapy(OT)designedtoincreaseawarenessaboutappropriateemployment
placementforpersonswithdevelopmentaldisabilitiesbyusingacomparativequantitativemethodology.Glineretal.(1999)comparedprofessionalprogramOT
studentswithpreOTstudentsandbusinessstudentsbyusingvignettes.Thestudentsweregivendescriptionsofapersonwithadevelopmentaldisabilityand
descriptionsofdifferenttypesofemployment.Eachstudentcould"hire"thepersonwithadevelopmentaldisabilityand"assign"thatpersontoasegregatedwork,
supportedwork,orcompetitiveworksetting.Thedependentvariablewasthenumberofstudents(frequency)whoselectedaparticulartypeofwork.Glineretal.
(1999)selectedachisquaretestbecausetherewasonebetweengroupsindependentvariable,typeofstudent,withthreelevels,andthedatawerenominal(Fig.
14.3).ThedataareshowninTable14.8.Theequationforthechisquaretestofindependenceisasfollows:

Theequationlooksalotworsethanitisbecausesummationsignsalwaysscarepeople.ForeachoftheninecellsinTable14.8,wemustsubtracttheexpected
frequenciesfromourobservedfrequencies(ouractualdata),squarethisnum
TABLE14.8
CrosstabulationTableofFrequencyDatafromtheGlineretal.(1999)Study
Group

SegregatedWork

SupportedWork

Businessstudents

20

42

14

76

PreOTstudents

43

55

OTstudents

49

17

68

Totals

28

134

37

199

CompetitiveWork

Totals

Page232

ber,andthendividethisnumberbyourexpectedfrequencies.Theexpectedfrequenciesarethefrequencieswewouldexpectiftheindependentvariablewasnot
relatedtothedependentvariable.Weusethissameequationforeachofthecells(rowstimescolumns)inthestudyandthenaddthenumberforeachcelltoobtain
ourchisquarevalue.Inourpresentexample,wehaveninecells,sowewillneedtocarryoutthisprocedureninetimes.
NoticethatinTable14.8wehaveincludedthetotalsfortherowsandthecolumns.Thesearecalledmarginaltotals.Inaddition,wehavethetotalforallofthe
participants(N=199).WeperformtheanalysisbyusingEquation4.Toobtainourexpectedfrequencies,wemustmultiplythemarginalsassociatedwitheachcell
anddividebyN.
Westartbyfindingtheexpectedfrequencyforeachcell.Thereareninecells.Wedothisbymultiplyingthecorrespondingtotalforeachrowandcolumn,anddividing
bythetotal.
Forourexample,theexpectedfrequencyforthecellbusinessstudentsbysegregatedworkis:

TheexpectedfrequencyforthecellOTstudentsbysupportedworkis:

Wedothisforallninecells.Thenweusethechisquareequationtomake
thecalculations,whichareasfollows:

Glineretal.(1999)reported,"Achisquaretestofindependencedemonstratedasignificantrelationshipbetweenthetwovariablesofemploymentdecisionandclass,
2(4)=18.79,p=.001."Thedegreesoffreedomforthisexamplearefour,asstatedintheparentheses.ThiswasdeterminedbyusingEquation5,asfollows:

wherer=numberofrows,and
wherec=numberofcolumns.

Page233

TherewerethreelevelsintheindependentvariablefromtheGlineretal.(1999)study.Todeterminewhichofthegroupsweredifferent,Glineretal.performed
followupchisquareteststomorepreciselyestimatedifferencesamongconditions.TheyfoundthatthesignificantdifferencewasbetweentheprofessionalOT
studentsandthebusinessstudents.
Similartotherequirementsdiscussedforthettestandsinglefactoranalysisofvariance,therearetworequirementsthatshouldbemettosatisfytheassumptionsfor
anychisquareanalysis:Yatescorrectionandminimumexpectedfrequencies.1
Sometimesitisdifficulttodecidewhichvariableistheindependentvariable.Forexample,considertwovariables,attitudetowardguncontrolandpoliticalaffiliation.
Onemightaskwhichvariableseemstocauseorpredicttheother.Davis(1985)hasspecifiedrulesofcausalorderthatarehelpfulindeterminingwhichvariableisthe
causalvariable.Forthemostpart,theserulesarebasedontheindependentvariableprecedingthedependentvariableintime.However,althoughsomeresearchers
seethechisquaretestasamethodoftestingtherelationshipbetweentwovariableswithoutspecifyingwhichvariableistheindependentvariable,wethinkthat
researchersshouldhaveinmindapositionaboutwhichistheindependent(presumedcause)anddependent(outcomeorresult)variablepriortothestudy.
Summary
Inthischapterwediscussedtheselectionandapplicationofappropriatestatisticalmethodsusedinsinglefactorbetweengroupsdesigns.Wedividedthechapterinto
aselectionofappropriateparametricandnonparametricstatisticsfromthescaleofthedependentvariableanddependingoncertainassumptionshadbeensatisfied.In
thesectiononparametricstatistics,wediscussedthettestforindependentsamplesandthesinglefactoranalysisofvariancewithaccompanyingposthoctests.Thet
testismorecommonlyusedwhenthereisoneindependentvariablewithtwolevelsbecauseitgivestheresearchertheoptionoftestingadirectionalhypothesis.When
theindependentvariablehasmorethantwolevels,thesinglefactorANOVAistheprocedureofchoice.
Thenonparametrictestsforsinglefactor,betweengroupsdesignsthatwediscussedweretheMannWhitneyUtest,theKruskalWallisanalysisofvarianceby
ranks,andthechisquaretest.TheMannWhitneyUtestandtheKruskalWallistestareusedwithordinaldatathatarenotnormallydistributed,ortoconvert
intervaldatatoranksbecauseofviolationofassumptionsunderlyingparametrictests.Thechisquaretestisusedwithnominaldataandcanbeusedwithtwoormore
thantwoconditionsorlevels.Nonparametric
1

Statisticalopiniononmeetingeitheroftheserequirementsforachisquaretestisnotunanimous.CamilliandHopkins(1978)usedcomputergeneratedreplicationsandfound
thatthechisquaretestwasrobustwithexpectedfrequencieslessthanfive.Inaddition,theyrecommendednotusingYatescorrection,becauseitdecreasedthetypeIerrorrate
tofarlowerthanthevalueofalpha,hencedecreasingpower.Bradley,Bradley,McGrath,andCutcomb(1979)alsofoundthatusingexpectedfrequenciesoflessthanfivedidnot
increasetypeIerror.

Page234

testsareusedlessfrequentlythanparametricanalysesandusuallyarelesspowerful.
StudyAids
Concepts
Chisquaretest
Degreesoffreedom
Homogeneityofvariance
Independence
KruskalWallisanalysisofvariancebyranks
MannWhitneyUtest
Normality
Posthoccomparisons
SinglefactorANOVA
Sourcetable
ttest
Distinctions
Parametricversusnonparametricstatistics
ttestversussinglefactorANOVA
ttestversusMannWhitneyUtestversusc2test
SinglefactorANOVAversusKruskalWallisANOVAversusc2test
ApplicationProblems
Questions1to4.Forthefollowingpassagesselecttheproperstatisticalanalysisbasedon(a)whetherthedesignisbetweengroups,withinsubjects,ormixed(b)
numberoflevelsoftheindependentvariable(s)(c)thescalelevelofmeasurementofthedependentvariable(s)and(d)whetherassumptionsunderlyingparametric
testsareviolated.
1.Atherapistwasinterestedindeterminingtheeffectivenessofanewtreatmentforchildrenwithcerebralpalsy.Thetherapisthad40suchchildreninherclinic.She
randomlyassignedparticipantstooneoftwogroups(20ineachgroup).Thetreatmentgroupreceivedthenewtreatmenttherapy.Thecontrolgroupreceiveda
traditionaltherapy.Afterfourmonths,all40childrenperformedamotorcoordinationtask.Ifthechildcompletedthetask,treatmentwasconsideredtobesuccessful.
Ifthechildwasunabletocompletethetask,treatmentwasconsideredtobeunsuccessful.
2.AprofessorwhotaughtstatisticswascurioustoknowaboutdifferentmethodsofcalculatingthePearsonProductMomentCorrelationCoefficient.Specifically,he
wonderedwhichwaywasquicker,thedeviationmethodortherawscoremethod.Itjustsohappenedthathehadaclassof31graduatestudents.Herandomly
assigned16studentstothedeviationmethod,and15studentstotherawscoremethod,andaskedthestudentstokeeptrackofhowlongittook(tothenearest
minute)todeterminethecorrelationcoefficientfortheproblem.

Page235

3.AtaclinicintheRockyMountains,ahandtherapistwasinterestedindeterminingthefunctionalrecoveryofjointreplacementsurgeryasopposedtotwoother
moreconservativetreatments,steroidsandsplinting,onpersonswithrheumatoidarthritis.Thirtyparticipantswererandomlyselectedfromapopulationofhand
therapypatients.Theparticipantswererandomlyassignedtooneofthreegroups(10participantsineach).Thegroupswerethesurgerycondition,drugcondition,and
thesplintcondition.After6months,allthreegroupsweremeasuredonasubtestoftheGlinerOccupationalHandRecoveryIndex,anintervalscale.
4.Aninvestigatorisinterestedincomparingsuccessfulemploymentduetodifferentservicedeliverysystemsforpersonswithtraumaticbraininjury.Onesystem(n=
10)wasreferredtoasthecognitivedeliverysystem(C).Asecondsystem(n=10)wastheemotionaldeliverysystem(E).Athirdsystem(n=10)wasthecase
managementdeliverysystem(CM).Theinvestigatorrankedthe30subjectsfrom1to30onhowsuccessfultheywereontheirfirstjobafterrecovery.
5.Atherapistwantedtoknowifhisspecialsplintwouldincreasetherangeofmotion(ROM)inthewristafteratraumaticinjury.Hehadeightpatientswearthe
specialsplint,andeightpatientswearthestandardsplint.HepredictedthathisspecialsplintwouldincreaseROM(indegrees)attheendoftherecoveryperiod.He
performedattestforindependentsamplesonthedataandfoundatvalueof1.82.Hewenttoattableandfoundthatthisvaluewasgreaterthanthecriticalvaluefor
atwith14degreesoffreedom(onetailed).Heconcludedthathisspecialsplintwasstatisticallysignificantlybetterthanthestandardsplint.Whatdidhemeanby
statisticalsignificance?
6.Youhavethreeindependentgroups,with10participantsineachgroup(n=30).ThesegroupsarelabeledA1,A2,andA3.Themeansofthethreegroupsare10,
14,and20respectively.Youwonderifthereisasignificantdifferenceforanyofthepossiblecomparisons.
a.Howmanycomparisonsarethere?
b.Ifyouuseattesttotesteachcomparison,whatdangerdoyouruninto?
c.Youdecidetoperformananalysisofvarianceonthedata.Theresultsareasfollows,fillintherestofthetable:
Sourceofvariation

SS

df

MS

Betweengroups

320

k1

Withinsubjects

1080

nk

d.YoufindthattheFisstatisticallysignificant.Howdoyoudeterminewhichofthegroupsaredifferentfromeachother?
7.Thedesignisasinglefactor(betweengroups)designwithtwolevels.Thedataarenormallydistributed.Therearetwoanalysesthatcanbeperformed.
Whatarethetwodifferenttypesofanalysesthatcanbeusedinthissituation?
Whenshouldeachanalysisbeused,andwhy?

Page237

Chapter15
SingleFactorwithinSubjectsDesigns:
AnalysisandInterpretation
Overview
AdvantagesofwithinSubjectsDesigns
DisadvantagesofwithinSubjectsDesigns
AnalyzingSingleFactorwithinSubjectsDesignswithParametricStatistics
tTestforPairedorRelatedSamples
SingleFactorAnalysisofVariancewithRepeatedMeasures
AnalyzingSingleFactorwithinSubjectsDesignswithNonparametricStatistics
WilcoxonMatchedPairsTest
FriedmanTwoWayAnalysisofVariancebyRanks
McNemarTest
CochranQTest
Summary
StudyAids
Concepts
Distinctions
ApplicationProblems

Page238

Overview
Theanalysesthatarediscussedinthischapterareusedinadesignwithoneindependentvariableandtwoormorelevelsorconditions,andparticipantsaremeasured
underallconditions.Thesedesignsarereferredtoaswithinsubjects,dependentsamples,orrepeatedmeasuresdesigns.Thismeansthatparticipantsundergoall
conditionsofthestudyorparticipantsarematchedonavariablerelatedtothedependentvariable,indicatingthatthetwogroupsarerelated.
Similartotheapproachesdescribedinthepreviouschapter,thetypeofresearchapproachesusedwithsinglefactorwithinsubjects(repeatedmeasures)designsare
randomizedexperimental,quasiexperimental,andcomparative.Forexample,RumrillandGarnette(1997)studiedaninterventiondesignedtoincreaseconsumer
participationintheAmericanswithDisabilitiesAct(ADA)TitleIaccommodationrequestprocess.Therewere36participantsinthestudy.Theydividedthese36
participantsinto18matchedpairsbasedongender,occupationalstatus,andage.Thentheyrandomlyassignedonememberofeachpairtoaninterventioncondition
andtheothermemberofeachpairtoacontrolcondition.Thedesignforthisstudyisdescribedasasinglefactorrepeatedmeasuresdesignwithtwolevels.Thisisan
exampleoftherandomizedexperimentalapproachappliedtosituationswithrepeatedmeasures.
AnotherexampleoftherandomizedexperimentalapproachthatappliestorepeatedmeasuressituationsisastudybyHsieh,Nelson,Smith,andPeterson(1996).
Theywereinterestedintheeffectofpurposefulactivityonbalanceforpeoplewithhemiplegia.Theyhad21participantswhounderwentallthreeconditionsofthe
study.Becausetheyrandomlyassignedparticipantstooneofthreedifferentordersofexperiencingtheconditions,theapproachwasrandomizedexperimental.The
designforthisstudywasasinglefactorrepeatedmeasuresdesignwiththreelevels.
Singlefactorrepeatedmeasuresdesignsalsocanfitintothequasiexperimentalapproach.ConsiderastudybyLuftandPizzini(1998),whowereinterestedinthe
influenceofvisitingaProblemSolvingDemonstrationClassroom(PSDC)onaproblemsolvingmodel(Search,Solve,Create,andShareProblemModel).Theyhad
seventeachersintheirstudy.Priortovisitingthedemonstrationclassroom,theteacherscollecteddatafromtheirclassroomswithanassessmentinstrumentthatwas
developedtomeasurechangesinbehaviorofbothstudentsandteachersrelatedtoproblemsolvingimplementation.Thesechangesweremeasuredonaone(highest)
tofive(lowest)score.Examplesofcategoriesmeasuredincludedtimeincooperativegroups,groupcohesiveness,andstudents'activelyparticipating.Afterfourvisits
tothePSDC,datawerecollectedagainbyusingthesameinstrument.Thespecificdesignforthisstudywasasinglegrouppretestposttestdesign.Thegeneraldesign
classificationisasinglefactorrepeatedmeasuresdesignwithtwolevels,pretestandposttest.
Kingetal.(1997)usedasimilarrepeatedmeasuresapproachtostudysocialskillstraining.Theystudieda10weeksocialskillstrainingprogramforchildrenwith
physicaldisabilities.ThesestudentsweremeasuredonaSchoolSocialSkillsscalepriortotheintervention(pretest),aftertraining(posttest),and24weeksafterthe
endoftraining(followup).Thedesignforthisstudyisa

Page239

singlefactorrepeatedmeasuresdesignwiththreelevels.Becauseallparticipantsunderwentthesameorder(time),theapproachisquasiexperimental.
Thecomparativeapproachcanbeusedwithrepeatedmeasuresdesigns.Forexample,Webster,Plante,andCouvillion(1997)examinedtheeffectsofdisordered
speechonphonologicalawareness,verbalworkingmemory,andletterknowledge.Theymatchedpairsof15childrenwithphonologicimpairmentwith15children
withnormalphonologyonthebasisofmentalageandgender.Phonologyisanattributeindependentvariable,sotheapproachiscomparative.Thedesignforthis
studyisasinglefactorrepeatedmeasuresdesignwithtwolevels.
AdvantagesofwithinSubjectsDesigns
Anobviousadvantageofusingawithinsubjectsorrepeatedmeasures,designisthatfewerparticipantsareneededinthestudy.Thissavestimeinrecruitmentof
participants,especiallyifparticipantshavecertaincharacteristicsthatarenotcommontomostofthepopulation.However,amoreimportantreasonforselectinga
repeatedmeasuresdesignisthatvariabilityamongparticipantsisreduced.Recallfromchapter14thatthestatisticalanalysisofbetweengroupsdesignsisusuallyat
testorananalysisofvariance,wherewearecomparingthevariabilitybetweengroups(treatmentvariance)tothevariabilitywithingroups(errorvariance).Whenwe
performarepeatedmeasuresdesign,eachparticipantisundergoingalloftheconditions.Therefore,itisexpectedthatanychangesfromconditiontoconditionresult
fromthenatureofaparticularcondition(treatment),andnotvariabilityamongparticipants(error),becausethesameparticipantisexperiencingeachoftheconditions.
ThisreductioninerrorvarianceincreasesthesizeofthetorFratioandresultsinagreaterprobabilityoffindingastatisticallysignificantdifferenceifoneisactually
there.
DisadvantagesofwithinSubjectsDesigns
Althoughwithinsubjectsorrepeatedmeasures,designsareadvantageousinreducingerrorvariance,therearetwodistinctdisadvantagesofusingrepeatedmeasures
designs.Thefirstdisadvantageisthatrepeatedmeasuresdesignscannotbeusedinsituationswherealastingeffectmighttakeplacewithinanyoftheconditions.This
problemisreferredtoascarryovereffects.Forexample,ifaninvestigatorisinterestedintheeffectofatreatmentorintervention,shecouldnotusearepeated
measuresdesign,becauseonceparticipantsexperiencedthetreatmentcondition,theycouldnotbeexpectedto''unlearn"thetreatmentandthenundergoacontrol
condition.Becauseofcarryovereffects,repeatedmeasuresdesignsareconsideredlessflexiblethanbetweengroupsdesignsandareseenlessfrequentlyinthe
literature,especiallyinjournalsemphasizingappliedresearch.Onemethodofcircumventingcarryovereffectsandstillgainingtheadvantageofreducingerrorvariance
istouseamatchingprocedure.Inamatchingprocedure,participantsaregroupedintopairs(dyads)ortriplets(triads)basedonacharacteristicthatshouldbe
relatedtothedependentvariable.Anexamplemightbeintelligence.Afterparticipantsaregrouped,theyareassigned(randomlyisoptimum)togroupsandthenthe
studyiscarriedout.Conceptually,theideaofmatchingistomakeeachmemberof

Page240

thepairortriadasthoughheorshewerethesameparticipantundergoingallconditions.Therefore,designsthatusematchingareconsideredtoberepeatedmeasures
designsandusesimilarstatisticalprocedures.
Aseconddisadvantageofrepeatedmeasuresdesignsisthatthedegreesoffreedominthestudyisreduced.Ifonedidarepeatedmeasuresstudywithtwoconditions,
itwouldtakehalfasmanyparticipantstogatherthesameamountofdata,becauseeachpersonwasmeasuredtwice.Althoughareducednumberofparticipantsmay
makeiteasiertoconductthestudy,italsoreducesstatisticalpower.Forexample,considerastudythatcomparesaninterventionconditiontoacontrolcondition,
wherethereare20participantsineachcondition.Ifabetweengroupsdesignisusedtocompletethestudy,thereare40participants,or38degreesoffreedom
(rememberdf=[n1]+[n1]foranindependentsamplesttest).Ontheotherhand,supposethatarepeatedmeasuresdesignwasusedtocarryoutthestudy.
Therewouldbe20participantsineachcondition,butbecauseeachparticipantundergoesbothconditions,therewouldonlybe20participantsinthestudy.The
degreesoffreedomwouldben1,or19.Therefore,statisticalpowerwouldbereduced.Thedecisiontouseeitherabetweengroupsdesignorarepeatedmeasures
designstemsfromtheissueofanincreasedsamplesizeversusareductioninerrorvariance.
AnalyzingSingleFactorwithinSubjectsDesignswithParametricStatistics
Wewilldiscusstwodifferentstatisticsthatareusedwithsinglefactorwithinsubjectsorrepeatedmeasures,designs.Thesetwostatisticsarethettestforcorrelated
samplesandthesinglefactorrepeatedmeasuresanalysisofvariance.WhenthesestatisticsshouldbeselectedisdepictedinFig.15.1.
tTestforPairedorRelatedSamples
Thettestforrelatedorpairedsamplesisusedwhenthereisoneindependentvariable,withtwolevels,andtheparticipantsundergobothconditions,orpairsof
participantshavebeenmatchedonarelevantvariable.Thedependentvari

Fig.15.1.
Decisiontreeforselectingtheappropriateparametric
singlefactorwithinsubjects(repeatedmeasures)statistic.

Page241

ablemustbeintervalorratiodata.Theadvantageofthisstatisticaltestoverthesinglefactoranalysisofvariancewithrepeatedmeasuresistheoptiontousea
directionalhypothesis,whenappropriate.
AnexampleofastudythatusedthettestforrelatedsamplesisthestudybyWebsteretal.(1997)mentionedearlier.Theymatched15childrendiagnosedas
phonologicallyimpairedwith15childrenwithnormalphonologyonmentalageandgender.Thentheymeasuredthechildrenondependentvariablesofsentence
memory,rhyme,alliteration,pseudowordsegmentation,andletteridentification.Weexamineoneanalysisofthedependentvariable,sentencememory.Participants
hadtorepeatsentencesthatrangedinwordlengthfrom3to13words.Apercentagecorrectscorewascomputed.Table15.1showsthedatafromtheWebsteret
al.(1997)study.
Websteretal.usedattestforpairedorcorrelatedsamplestocomparetheperformanceofthematchedpairsonsentencememory.Thistestwasselectedbecause
therewasoneindependentvariablewithrepeatedmeasures(thegroupswerematched),twolevels,andthedependentvariablewasaratiolevelscale.
TheequationforthettestforcorrelatedsamplesisshowninEquation1:

Theformulaforthettestforcorrelatedsamplesisdifferentfromthettestforindependentsamples(chap.14),andmucheasiertouse.Wedonotneedtocompute
themeans,standarddeviations,andstandarderrorsforeachgroup.Instead,foreachpairofparticipants,wesubtractthephonologicallyimpairedscorefromthe
phonologicallynormalscore,yieldingadifferenceorchangescoreforeachpairofparticipants.Then,weobtainthemean,standarddeviation,andstandarderrorof
thedifferencebetweenmeansonthesechangescores.Thus,oncewehaveobtainedthechangescores,weperformthesestatisticalcalculationsononlyonegroupof
scores.ThettestforcorrelatedsamplescanbedemonstratedwiththedatafromTable15.1(Websteretal.,1997):
TABLE15.1
MeansandStandardDeviationsfromtheWebsteretal.(1997)
StudyfortheVariableofSentenceMemory
SampleDescription

PhonologicallyNormal

Phonologically
Impaired

Numberofparticipants

15

15

Mean

84.50

62.00

Standarddeviation

11.92

21.38

Differencebetweenmeans

22.50

Standarddeviationofthe
differencebetweenmeans

25.39

Page242

Websteretal.(1997)concludedthat"childrenwithnormalphonologyoutperformedthosewithimpairedphonologyonsentencememory,t(14)=3.43,p<.004.
"(p.369).Noticethatthenumberinparentheses,14,isthedegreesoffreedomforthisstudy.Aswepointedoutpreviously,themeanandstandarddeviationare
performedononlyonesetofscores,thedifferencescores.Thereare15differencescoresandonlyonedegreeoffreedomislostfromredundancy.Therefore,the
degreesoffreedomforthisstudyis14.Thenotationp<.004indicatesthattheprobabilitythatthisoutcomeresultsfromchance,assumingatruenullhypothesis,is
lessthan4timesin1,000,arareevent.Thenullhypothesisofnodifferencebetweenconditionsisrejected.
SingleFactorAnalysisofVariancewithRepeatedMeasures
Themostcommonsituationwherethesinglefactoranalysisofvariancewithrepeatedmeasuresisperformedoccurswherethereisoneindependentvariable,twoor
morelevelsthedependentvariableisintervalorratiolevelandtheparticipantsundergoallconditionsorlevelsofthestudy.Wepreviouslymentionedastudyby
Hsiehetal.(1996),whoinvestigatedtheeffectofpurposefulactivityonbalanceforpeoplewithhemiplegia.Alloftheirparticipantsunderwentthreedifferent
conditions,althoughtheorderofundergoingeachconditionwasrandomlyassigned.Thedesignfortheirstudywasasinglefactorrepeatedmeasuresdesignwiththree
levels.Thedependentvariables,frequencyanddurationofexercise,wereratio.Hsiehetal.(1996)performedasinglefactoranalysisofvariancewithrepeated
measurestodetermineifthereweredifferencesamongthethreeconditions(seeFig.15.1).
Asecondsituationwherethesinglefactorrepeatedmeasuresanalysisofvarianceisperformedoccurswhenmatchinghasoccurredasapartofthedesignofthe
study.Whenparticipantsarematched,thedesignbecomesarepeatedmeasuresdesign.GlinerandSample(1993)conductedastudytogatherbaselinedataon
qualityoflifeamongthreedifferentgroupspriortofurtherintervention.Thethreegroupswereashelteredworkgroup(SW),asupportedemploymentgroup(SE),
andajobmatchedcontrolgroup(JM).ParticipantswerematchedfromsupportedemploymenttoshelteredworkonIQandage,andfromsupportedemploymentto
jobmatchedcontrolonplaceofworkandage.Thesematchesresultedin18triadsfrom54subjects.ThedependentvariablewasEnvironmentalControl,asubscale
oftheQualityofLifeIndex(Schalock,Keith,Hoffman,&Karan,1989).
Likeallanalysisofvariancestatistics,theresultsfromasinglefactorrepeatedmeasuresorwithinsubjectsanalysisofvariancearereportedinasourcetable(see
Table15.2).Noticethatbecauseourdesignisarepeatedmeasuresorwithinsubjectsdesign,thereisnobetweengroupscomponent.Thetreatmenteffect

Page243

comesfromthewithinsubjectscomponent.Thisisdifferentfromthesinglefactorbetweengroupsdesignthatwediscussedinchapter14(seeTable15.3).
Thefirstsourceofvariationreportedinthesourcetableofthesinglefactorrepeatedmeasuresanalysisofvarianceisthebetweensubjectssumsofsquares.This
sourceofvariationisdeterminedfromthedifferencesamongparticipants.Iftheparticipantsdifferamongeachotheronthedependentvariable,thenthissourceof
variationwillberelativelylarge.If,ontheotherhand,therearesmalldifferencesamongparticipantsonthedependentvariable,thenthebetweensubjectssumsof
squareswillbesmall.Thelargerthebetweensubjectssumsofsquares,thelargerthereductioninerrorvariancetherefore,itisdesirableforparticipantstodifferon
thedependentvariableinarepeatedmeasuresdesign.Oncethebetweensubjectssumsofsquaresiscomputed,itissubtractedfromthetotalsumsofsquares.The
differenceisthewithinsubjectssumsofsquares.Fromhere,thecalculationsareidenticaltothesinglefactorbetweengroupsanalysisofvariancediscussedinchapter
14.Theresultisatreatmentmeansquaresandanerrororresidualmeansquares.Itisimportanttorememberthatthereisatradeoffwhenusinganyrepeated
measuresanalysisofvariance.Thistradeoffisbetweenthereductioninerrorvariancefromtheparticipantsbeingdifferentonthedependentvariable,andthe
reductionindegreesoffreedombecausethedesignisrepeatedmeasures.Ifparticipantsarenotdifferentonthedependentvariable,thenpowerwillbelost,because
degreesoffreedomarelessthanforabetweengroupsdesign.
Let'sreturntotheGlinerandSample(1993)study.ThedatacanbeseeninTable15.4.GlinerandSample(1993)selectedasinglefactoranalysisrepeatedmeasures
analysisofvariancetodeterminedifferencesamongthethreeemploymentconditionsinthiscomparativeresearchapproach.Therationaleforselectionofthisanalysis
wasthattherewasoneindependentvariablewiththreelevelstheparticipantswerematched,formingtriadsandthedependent
TABLE15.2
SourceTableforaSingleFactorWithinSubjectsDesign
Source

SS

MS

Betweensubjects

n1

df

WithinsubjectsTreatment

k1

MSTreatment

MSResidual

ResidualorError

(n1)(k1)

TABLE15.3
SourceTableforaSingleFactorBetweenGroupsDesign
Sourceof
Variation

SumsofSquares DegreesofFreedom
(SS)
(df)

Betweengroups

Withinsubjects

k1

Nk

MeanSquares
(MS)

MSbetweengroups

MSbetweengroups

MSwithinsubjects

MSwithinsubjects

Page244

variablewasintervallevel.ThesourcetablethatdemonstratestheresultsfromtheGlinerandSamplestudycanbeseeninTable15.5.
GlinerandSamplereportedtheirresultsasfollows.Asignificantdifferencewasobservedamongthethreeemploymentconditionsonthemeasureofenvironmental
control,F(2,34)=19.21,p<.001.Therefore,theprobabilityofthisresultoccurringbychance,assumingatruenullhypothesis,islessthan1in1,000.Therefore,
GlinerandSamplerejectedthenullhypothesisofnodifferenceamonggroups.However,findingasignificantFafterperformingananalysisofvariancewithmore
thantwogroupsonlytellsyouthatatleasttwoofthegroupsaresignificantlydifferent.Itdoesnottellyouwhichtwoaredifferentorifthereismorethanonepairof
groupsthataredifferent.Therefore,GlinerandSampleperformedaTukeyHSDposthoctesttodeterminewheredifferencesexisted.Theresultsfromthattest
showedthatthejobmatchedcontrolwassignificantlygreateronenvironmentalcontrolthanboththeshelteredworkandsupportedemploymentconditions.In
addition,thesupportedemploymentconditionwassignificantlyhigheronenvironmentalcontrolthantheshelteredworkcondition.
TheexamplethatweprovidedforthesinglefactorANOVAwithrepeatedmeasuresusedmatchinginsteadofrepeatedmeasuresonthesameparticipants.The
reasonfortheformeristhattherearefewinstanceswherearepeatedmeasuresdesignwithoneindependentvariablecouldbeusedinaninternallyvalidstudy.Can
youthinkofanexample?
AnalyzingSingleFactorwithinSubjectsDesignswithNonparametricStatistics
Nonparametricstatisticsareusedwithwithinsubjects,orrepeatedmeasures,designsaswithbetweengroupsdesigns,whenthedataarescaledattheordinal(andnot
normallydistributed)ornominallevel,oroneoftheassumptionsunderlyinguseofparametricstatisticshasbeenviolated.Wediscussbrieflyfour
TABLE15.4
DescriptiveStatisticsfortheGlinerandSample(1993)Study
EmploymentCondition

NumberofParticipants

MeanScoreonEnvironmentalControl

Shelteredwork

18

33

Supportedemployment

18

37.11

Jobmatchedcontrol

18

42

TABLE15.5
SourceTablefortheGlinerandSample(1993)Study
Source

SS

df

MS

Betweensubjects

595

17

35

Withinsubjects

Treatment

730

365

Residual

646

34

19

19.21

Page245

differentnonparametricstatisticstobeusedwithsinglefactorrepeatedmeasuresdesigns(seeFig.15.2).Itshouldbenotedthattheuseofthesestatisticsinapplied
settingsarerelativelyrarecomparedtotheirparametriccounterpartsdiscussedpreviously.
WilcoxonMatchedPairsTest
TheWilcoxonmatchedpairstestistobeusedinadesignwherethereisoneindependentvariable,withtwolevels,andtheparticipantsundergobothconditions,or
pairsofparticipantshavebeenmatchedonarelevantvariable.Thedependentvariabledataareordinal(andnotnormallydistributed)ortherehavebeenviolationsof
assumptionsofthettestforcorrelatedsamples.
Aswasthecaseinothernonparametricteststhatwediscussed,intheWilcoxontestweconvertthescorestoranksandthenuseaspecificequationtodetermine
statisticalsignificance.Inthenonparametricproceduresforbetweengroupsdesigns,therankingofscorestakesplacewithineachgroup.IntheWilcoxontest,
differencesbetweenconditionsforeachparticipant(orpairifparticipantsarematched)areobtainedfirst(maintainingthepositiveornegativesign)andthenthese
differencesareranked.Next,theranksaretotaled.Thefartherthesumoftheranksisfromzero,thehighertheprobabilityofasignificantdifferencebetween
conditions.
Chiara,Carlos,Martin,Miller,andNadeau(1998)usedaWilcoxontestaspartofasetofanalysestodeterminehowpatientswithmultiplesclerosis(MS)would
reacttoacoldtreatment.FourteenparticipantsdiagnosedwithMSunderwenttwotreatments,restatroomtemperatureandrestinacoldwaterbath.Chiaraetal.
describethedesignas"arepeatedmeasuresdesignwithrandomassignmentof(theorderof)experimentalandcontrolconditions"(p.523).Therefore,thisresearch
approachisrandomizedexperimental.Thegeneral

Fig.15.2.
Decisiontreeforselectingtheappropriatenonparametric
singlefactorwithinsubjectsstatistic.

Page246

classificationofdesignisasinglefactorrepeatedmeasuresdesignwithtwolevels.OneofthedependentvariablesselectedbyChiaraetal.waslevelofspasticity,as
measuredbytheModifiedAshworthScale,anordinallevelscale.AWilcoxontestwasusedtoexaminethespasticitydatabecausetherewasonerepeatedmeasures
independentvariablewithtwolevels,andthedependentvariablewasmeasuredonanordinalscale.Theyfoundasignificantdifferencebetweenconditionsand
reportedtheirresultsas,"spasticitywassignificantlylower(Z=2.309,p<.05)immediatelyafterAT(roomtemperature)thanafterCT(coldtemperaturebath)."
Chiaraetal.concluded,"OurfindingofaCTinducedincreaseinspasticitydoesnotagreewithotherstudies,whichhavereportedadecrease.Thereasonsforthis
discrepancyareunclear,butmayincludetheuseoflowerbathtemperaturesinotherstudiesortheuseofsubjectivemeasuresofspasticityinpreviousstudies"(p.
527).
FriedmanTwoWayAnalysisofVariancebyRanks1
TheFriedmantestisusedinarepeatedmeasuresdesignwhenthereisoneindependentvariable,threeormorelevels,andthedependentvariableisscaledatthe
ordinallevel(andnotnormallydistributed)orthedataareintervalbutviolationsofassumptionshaveoccurred,especiallythoseofnormality.TheFriedmantestis
similartotheWilcoxonsignrankstestinthatscoresareobtainedforeachparticipantacrossconditionsandthenconvertedtoranks.IntheWilcoxontest,differences
wereobtainedbetweenthetwoconditionsandthenranked,whereasintheFriedmantest,scoresaresummedacrossconditionspriortoconvertingtoranks.
Studieshaddemonstratedthatrunningonnaturaloroffroadsurfacesdemandedmoreenergythanrunningonroad.Creagh,Reilly,andLees(1998)wereinterestedif
thisrunningoffroadcausedbiomechanicalalterationstostridecharacteristics.Theysetupastudythatusedarepeatedmeasuresdesigntotesttheirhypothesisinnine
femalerunners.Therunnerswererandomlyassignedtotheorderofexperiencingthreedifferentterrainconditions.Theseconditionswereapath(controlcondition),
shortgrass,andlonggrass.Therunnerswerevideotapedwhileexperiencingeachoftheseconditions.Creaghetal.measured10differentstridecharacteristics.
Althoughallofthesemeasureswerescaledattheratiolevel,notallwerebelievedtobenormallydistributed.AccordingtoCreaghetal.,"Thosevariablesshownto
benonparametric(maximumupperlegangleandmaximumflexionangularvelocityofknee)werecomparedusingFriedman'stest"(p.1031).Therationalefor
selectingFriedman'stestisonerepeatedmeasuresindependentvariable,withmorethantwolevels,andthedependentvariablewasnotassumedtobenormally
distributed.Creaghetal.proposedposthoctestsbyusingtheWilcoxontesttodeterminespecificdifferencesamonggroupsiftheFriedmantestwasstatistically
significant.Note,however,thatusingtheWilcoxontestaftertheFriedmantestisanalogoustousingtheFisherLSDtestafteranANOVAitissomewhat"liberal."
1

Note,partofthename(twoway)ofthisanalysisismisleadingbecause,inourconceptualization,thereisonlyoneindependentvariableinthisanalysis.ThesecondFisfor
"betweensubjects"(seeSourceTable15.2),butwearenotinterestedinthatF.

Page247

McNemarTest
TheMcNemartestisusedinadesignwherethereisoneindependentvariable,withtwolevels,andtheparticipantsundergobothconditions,orpairsofparticipants
havebeenmatchedonarelevantvariable.Thedependentvariableisnominalorcategoricaldata.TheMcNemartestissimilartothechisquaretestinthatfrequencies
aretheunitofmeasurement,andtheycanbevisualizedinacrosstabulationtable.2Furthermore,achisquaretableisusedtodeterminestatisticalsignificance.
However,becauseeachparticipantundergoesbothconditionsofthestudy,thereareimportantdifferencesfromthechisquaretestforindependence.
CochranQTest
TheCochranQtestisusedinadesignwherethereisoneindependentvariable,withthreeormorelevels,andtheparticipantsundergoallconditions,orparticipants
havebeenmatchedonarelevantvariable.Thedependentvariableisnominalorcategoricaldataandcanhaveonlytwocategories.TheCochrantestisanextension
oftheMcNemartestinthatfrequenciesaretheunitofmeasurement,butinthistesttheparticipantsundergomorethantwoconditions.Achisquaretableisusedto
determinestatisticalsignificance.
AstudybyTarrandBishop(1992)providesexamplesfortheuseofboththeMcNemartestandtheCochranQtest.Theywereinterestedinthedifferent
instructionalapproachestoincreasemotorperformanceinchildrenwithdevelopmentaldisabilities.Thethreeinstructionalapproacheswerecooperative,competitive,
andindividualistic.Theparticipantswerefiveboyswithdevelopmentaldisabilitiesenrolledinaremedialmotorbehaviorprogram.Theparticipantsunderwentsix
observationalperiodsforeachinstructionalapproach.Theorderofinstructionalapproacheswasrandomlyassigned,sothisisconsideredtobearandomized
experiment.Duringeachobservationperiod,eachchildperformedthreemotorskillsinarandomlyassignedorder.Thedependentvariablewasthefrequencyof
successfulmotorperformancebytheparticipantsforeachinstructionalapproach.Thisisadichotomousnominaldependentvariable,wheretheparticipantisjudgedas
successfulornotsuccessful.TarrandBishopselectedaCochranQtesttoanalyzethedifferencesamonggroups.Thisanalysiswasselectedbecausetherewasone
repeatedmeasuresindependentvariable(typeofinstruction),thereweremorethantwoconditions,andthedependentvariablewasnominalanddichotomous.The
authorsreported,"Eventhoughthemagnitudeofthedifferencebetweenthecompetitive,cooperative,andindividualisticapproachesissmall,theCochranQtest
yieldedasignificantdifferencebetweentheinstructionalapproaches,(CochranQ[2]=9.418,p<.05)"(p.20).TheCochranQtest,similartootheranalysesthat
comparemorethantwogroupsorconditions,onlyindicatesatleastonesignificantdifference.Anadditionaltestneedstobeperformedtodeterminespecific
differencesamongconditions.
2

Iftheexpectedfrequenciesarelessthanfivepercell,thebinomialtestisrecommended(Siegel&Castellan,1988).

Page248

TarrandBishop(1992)selectedtheMcNemartestasafollowuptestforthesignificantCochranQtest.ThereasonaMcNemartestwasselectedwasthatthe
followupcomparisonswouldbebetweentheconditionsofcompetitiveversuscooperative,cooperativeversusindividualistic,andcompetitiveversusindividualistic.
Foreachofthesecomparisons,thereisonerepeatedmeasuresindependentvariable,twolevels,andanominaldependentvariable.AfterperformingtheMcNemar
testforeachofthethreecomparisons,TarrandBishopconcluded,"BasedontheMcNemartest,therewasasignificantdifferencebetweenthecompetitiveandthe
individualisticapproaches( 2[1,N=5]=7.018,p<.05)thecompetitiveapproachwasmoreeffectiveinproducingsuccessfulmotorperformance"(p.20).Neither
oftheothercomparisonswasstatisticallysignificant.
Summary
Inthischapter,wediscussedtheselectionofappropriatestatisticaltestsforsinglefactorwithinsubjects(repeatedmeasures)designsandprovidedexamplesfromthe
literature.Aswedidintheprecedingchapter,wedividedtheanalysesintoparametricandnonparametrictests.Theparametricteststhatwediscussedforsinglefactor
repeatedmeasuresdesignswerethettestforcorrelatedsamplesandthesinglefactorrepeatedmeasuresANOVA.Whilebothanalysescanbeusedinsinglefactor
repeatedmeasuresdesignswithtwolevels,thettestshouldbeselectediftheresearcherplansontestingadirectionalhypothesis.Whenthedesignisrepeated
measures,buthasmorethantwolevels,thesinglefactorrepeatedmeasuresanalysisofvarianceshouldbeusedandfollowedbyappropriateposthocprocedures,if
theFissignificant.
Wealsointroducedfournonparametricteststobeusedinsinglefactorwithinsubjectsorrepeatedmeasures,designs.TheWilcoxontest(appliedwhentherearetwo
levelsoftheindependentvariable)andtheFriedmantest(appliedwhentherearemorethantwolevelsoftheindependentvariable)areusedwithordinaldata,orto
convertintervaldatatoranksbecauseofviolationofassumptionsunderlyingparametrictests.TheWilcoxontestalsomaybeusedasaposthoctesttoasignificant
Friedmantest,butismoreliberalthantheTukeytest.TheMcNemartest(appliedwhentherearetwolevelsoftheindependentvariable)andtheCochranQtest
(appliedwhentherearemorethantwolevelsoftheindependentvariable)areusedwithnominaldata.TheMcNemartestmaybeusedasaposthoctestfollowinga
significantCochranQtest,butit,too,isliberal.Nonparametricstatisticalproceduresarerelativelyrareintheliteratureforsinglefactorrepeatedmeasuresdesigns.
StudyAids
Concepts
Carryovereffects
CochranQtest

Page249

Friedmantest
Matching
McNemartest
SinglefactorrepeatedmeasuresANOVA
ttestforpairedsamples
Wilcoxonmatchedpairstest
Distinctions
Betweengroupsdesignsversuswithinsubjectsdesigns
SinglefactorrepeatedmeasuresANOVAversusFriedmantestversusCochranQtest
Matchingversuswithinsubjectsdesigns
ttestforpairedsamplesversussinglefactorrepeatedmeasuresANOVA
ttestforpairedsamplesversusWilcoxonmatchedpairstestversusMcNemartest
ApplicationProblems
Forthefollowingsixpassagesselecttheproperstatisticalanalysisbasedon(a)whetherthedesignisbetweengroupsorwithinsubjects,(b)numberoflevelsofthe
independentvariable(s),(c)scaleofmeasurementofthedependentvariable(s),and(d)whetherassumptionsunderlyingparametrictestsareviolated.
1.Agraduateseminarclasshas10students.Thestudentsareexposedtofourdifferentinstructors,eachinstructorrepresentingadifferentteachingstyle.Attheendof
thesemester,eachstudentisaskedtorankthefourinstructorsfromonetofouronclasschallenge.Aretheresignificantdifferencesamongteachingstyles?
2.Aresearcherhypothesizedthatapplyingsplintsovera3monthperiodwouldsignificantlyincreaserangeofmotioninpatientswhowerequadriplegic.Arandom
sampleof16patientswiththisdisabilitywasselected.Thepatientswerethenmatchedoninitialrangeofmotiontoformeightpairs.Thenoneparticipantofeachpair
wasrandomlyassignedtotheinterventiongroup(n=8),andtheotherrandomlyassignedtothecontrolgroup(n=8).Theinterventiongroupwassplintedfor3
months,whilethecontrolgroupwasnotsplinted.After3months,rangeofmotion(whichwasnormallydistributed)wasmeasuredforeachgroupandtheywere
compared.
3.Aneducatorisinterestedincooperativelearninggroups.Shewondersifactiveparticipationincreasesifgroupsarefacilitatedbysomeoneinthisarea.Astudyis
conductedduringtwodifferentclassperiods.Inoneclassperiod,participantsengageincooperativegroupswithoutafacilitator.Duringasecondclassperiodshe
introducesafacilitatorintothegroups.Onememberofeachgroup,unknowntoothermembers,keepstrackofactiveparticipation.Aftereachclass,participantsare
dividedintowhethertheyactivelyparticipatedordidnotactivelyparticipate.
4.Astudyiscarriedouttodetermineifahandsonentrepreneurialcurriculumforhighschoolstudentswillincreaseentrepreneurialskills.Twentystu

Page250

dentswerematchedintotenpairsbasedongenderandprevioushighschoolgrades.Onememberofeachpairwasassignedtotheinterventioncondition,thehands
onentrepreneurialcurriculum.Theothermemberofthepairwasassignedtothetraditionalbusinessclass,wheremostoftheactivitieswerestudentsreading
simulationsandclassdiscussions.Attheendofthesemester,eachstudentwasgivenanentrepreneurialskillscore,onaonetofivescale,withfivebeingalways,four
beingmostofthetime,threebeingsometimes,twobeingrarely,andonebeingnever.
5.Aresearcherwasinterestedindetermininghowtogetpeoplewitharthritistousejointprotectiontechniques.Sheobserved20peoplewitharthritisintheirhomefor
onemorningandfoundthat6outof20usedjointprotectiontechniques.Shethengaveademonstrationonjointprotectiontoeachparticipant.Onemonthlatershe
observedeachoftheparticipantsagainforonemorning.Shefoundthat16outof20peopleusedjointprotectiontechniques.
6.Agraduateseminarclasshas10students.Thestudentsareexposedtofourdifferentinstructors,eachinstructorrepresentingadifferentteachingstyle.Attheendof
thesemester,eachstudentisaskedtojudgeeachteachingstyleaschallengingornotchallenging.Aretheresignificantdifferencesamongteachingstyles?
7.Asdescribedinthischapter,Websteretal.(1997)performedastudythatmatchedparticipantsinpairsandthenperformedattestforpairedsamples.Whatare
theadvantagesanddisadvantagesofmatchinginthissituation(Hint:considerdegreesoffreedom)?
8.Asdescribedinthischapter,GlinerandSample(1993)performedarepeatedmeasuresANOVAtocomparethreematchedgroups.TheyalsoperformedaTukey
HSDposthoctestinthestudy.Explainwhy.

Page251

Chapter16
BasicAssociationalDesigns:
AnalysisandInterpretation
AnalyzingContinuousSingleFactorDesignswithParametricStatistics
PearsonProductMomentCorrelation
StatisticalSignificanceandthePearsonProductMomentCorrelation
TheCorrelationMatrix
UsingNonparametricAssociationalStatisticstoAnalyzeSingleFactorDesigns
SpearmanRankOrderCorrelation
PhiandCramer'sV
LinearRegression
Summary
StudyAids
Concepts
Distinctions
ApplicationProblems

Page252

Inthischapterweexaminetheassociationbetweentwovariables,anindependentvariableandadependentvariable,butinthiscase,bothvariablesarecontinuousor
atleasthavemanyorderedlevelsand,thus,maybeapproximatelycontinuous.Manymethodologistschoosetodescribeacorrelation,suchasthatresultingfroma
Pearsonproductmomentcorrelation,asarelationbetweentwovariableswithoutdesignatingoneastheindependentvariableandoneasthedependentvariable.
Otherschoosetodescribetherelationasoccurringbetweentwodependentvariables.Ouremphasisondesignatingonevariableastheindependentvariableandthe
othervariableasthedependentvariablefollowsfromthepreviouschaptersondesignandthenaturalleadintolinearregression.WediscussthePearsonproduct
momentcorrelation,(r),thenonparametricequivalent,Spearmanrankordercorrelation(rS),andalsointroducelinearregression.Inourpreviousdiscussionof
statisticalanalysesinchapters14and15,theindependentvariablehasbeencategorical,andtypicallynotmorethanthreeorfourcategories.Theanalysesthatwe
discussedconformedtotheresearchapproachesofrandomizedexperimental,quasiexperimental,andcomparative.Nowwewouldlikeyoutoconsideran
independentvariablethatiscontinuous.Acontinuousindependentvariableisalmostalwaysanattributeindependentvariable.Therefore,whentheindependent
variableandthedependentvariablearecontinuous,theresearchapproachisassociational.
AnalyzingContinuousSingleFactorDesignswithParametricStatistics
PearsonProductMomentCorrelation
AtlerandGliner(1989)wereinterestedintherelationshipbetweencertainpsychosocialvariablesandpoststrokeactivity.Theyhypothesizedthatdifferentmarital
relationshipcharacteristics,attitudesexpressedtowarddisability,andsocialsupportnetworksofspousesofstrokepatientswouldresultindifferentactivitylevelsand
differentsatisfactionofactivitylevels.Theindependentvariablesinthisstudyweremaritalrelationshipcharacteristics,attitudesexpressedtowarddisability,andsocial
supportnetworksofspouses.Thedependentvariableswereactivitylevelsandsatisfactionwithactivitylevels.Inthisstudy,boththeindependentvariablesandthe
dependentvariablesweremeasuredbyusingdifferentscalesthatresultedincontinuousmeasures.Forexample,totestoneofthehypothesesofthisstudy,Atlerand
Glinerexaminedtherelationshipbetweentheindependentvariableofsocialsupportandthedependentvariableofsatisfactionofactivitylevel.AtlerandGliner
selectedaPearsonproductmomentcorrelationtotesttherelationshipbecausetherewasoneindependentvariable,levelofsocialsupporttheindependentvariable
hadnumerouslevels(consideredtobeapproximatelycontinuous)andthedependentvariablewasscaledattheintervallevel.
ThePearsonproductmomentcorrelationteststheassociationorrelationshipbetweentwocontinuousvariables.Wedescribeitinthepresentcontexttotestthe
relationshipbetweenacontinuousindependentvariableandacontinu

Page253

ousdependentvariable,asdescribedintheassociationalapproach.However,thePearsonproductmomentcorrelationisalsowidelyusedinotherareas,especially
measurementreliabilityandvalidity(chap.20),whereonemighttesttherelationshipbetweentwoadministrationsofthesameinstrument(testretestreliability)orthe
relationshipbetweenaninstrumentandsomeexternalcriterion(predictivevalidity).
ThePearsonproductmomentcorrelationisexpressedasacoefficient,r,whichgivestheassociation,orrelationship,betweenthetwovariables.Thiscoefficienthasa
rangeof1to+1.Pearsonrvaluesthatarecloseto+1,suchasvaluesabove.7,areconsideredtobestrongpositiverelationshipsbetweenthetwovariables.A
positiverelationshipmeansthatasscoresononevariableincrease,scoresontheothervariablealsoincrease.Ontheotherhand,rvaluesthatarecloseto1,suchas
valuesbelow.7,areconsideredtobestrongnegative,orinverse,relationshipsbetweenthetwovariables.Aninverserelationshipmeansthathighscoresononeof
thevariablesareassociatedwithlowscoresforthesamepersonontheothervariable,andviceversa.Whenthervalueisnearzero,itindicatesthatthereisno
relationshipbetweenthetwovariables.Anexampleofnorelationshipoccurswhenhighscoresontheindependentvariableareassociatedwithhigh,medium,andlow
scoresonthedependentvariable.Alowcorrelationmeansthatyoucannotpredictthedependentvariablebyknowingthescoresontheindependentvariable.
WhenreportingtheresultsofastudythatusesaPearsonproductmomentcorrelation,thevalueofrisexpressedalongwiththedegreesoffreedomandthe
significancelevel.Forexample,AtlerandGliner(1989)reportedthat''apositivecorrelationbetweensocialsupportscoresandsatisfactionwasfound...(r=.32,df
=28,p<.05)"(p.21).Thervalueof.32fromthisstudyisconsideredtobearelativelyweakpositiverelationship.Thedegreesoffreedomof28indicatethatthere
were30participantsinthestudy,becausethedegreesoffreedomforaPearsoncorrelationcoefficientisthenumberofparticipantsinthestudyminustwo.
StatisticalSignificanceandthePearsonProductMomentCorrelation
Aswithanyinferentialstatistic,onemustbecautiousabouttheinterpretationofstatisticallysignificantcorrelationcoefficients.Thereisaninverserelationshipbetween
thenumberofparticipantsinthestudy,andthesizeofthecoefficientneededtoobtainstatisticalsignificance.Inotherwords,studieswithalargenumberof
participantsmightfindstatisticallysignificantcorrelationcoefficients,buttheymaybetrivial.IfweexamineatableofcriticalvaluesforthePearsonproductmoment
correlation,wewillfindthat,ifthereare100participantsinastudy,acorrelationofabout.20isallthatisneededtoobtainstatisticalsignificance,givena
nondirectionalhypothesis.Therefore,itisusefulwithcorrelationcoefficientstoobtainanindexofeffectsize.
Astatisticallysignificantoutcomegivesanindicationoftheprobabilitythattheresultcouldhavehappenedbychance.Itdoesnotdescribethestrengthofthe
relationshipbetweentheindependentanddependentvariables.Aneffectsizedescribesthestrengthofarelationshipbetweenanindependentandadependent
variable,inotherwords,howmuchoftheoutcomecanbepredicted

Page254

fromtheindependentvariable.Onecancalculateaneffectsizeforeverystatistic.ItisespeciallyeasytoperformthisoperationforaPearsonproductmoment
correlationbecauseonemeasureoftheeffectsizeisr2.Thisdescribestheamountofsharedvarianceoroverlapbetweentheindependentanddependentvariables.
Forexample,astudymightshowthattherelationshipbetweenhighschoolachievementtest(ACT)scores(independentvariable)andfirstyearcollegegrades
(dependentvariable)wasr=.7.Ther2wouldequal.49.Thiswouldindicatethatapproximatelyhalfofthevariancebetweenthetwomeasureswascommontoboth,
suchasintelligence.Ontheotherhand,halfofthevarianceisunexplained.
Althoughmanyinvestigatorsdonotreporttheeffectsizewiththeirresults,mostjournalsarenowrequiringsomeindexofthesizeofeffectinadditiontoa
determinationofstatisticalsignificance.Forexample,intheAtlerandGliner(1989)study,theyreportedthatthesharedvariance(r2)betweensocialsupportscores
andactivitylevelswasonly9.6%.Therefore,eventhoughtheyobtainedasignificantcorrelationbetweenthetwomeasures,the"effect"oftheindependentvariable
wassmall.
CorrelationMatrix
ItisrelativelyraretoseeasinglecorrelationcoefficientoreventwoorthreecorrelationcoefficientsinastudysuchasthatfoundinAtlerandGliner(1989).Instead,it
ismuchmorecommontoreportacorrelationmatrix.Acorrelationmatrixisatableofcorrelationcoefficientsthatshowshowallvariablesarerelatedtoeachother.
Forexample,KirchnerandHolm(1997)conductedastudytopredictacademicandclinicalperformanceofoccupationaltherapystudents.Intheirstudy,theyhadsix
independentvariables(undergraduategradepointaverage[GPA],GRE(GradRecordExamination)verbal,GREquantitative,GREanalytic,ascorefromthree
referenceletters,andascorefromastudentessay)andthreedependentvariables(graduateGPA,clientattendance,andclientoutcomes).Table16.1showsthe
correlationmatrixfortheirstudy.
TABLE16.1
IntercorrelationsBetweenDependentandIndependentVariables
Variables

1.UndergraduateGPA

.18

.08

.11

.15

.10

.24

.16

.02

2.GREverbal

.27

.48

.02

.10

.23

.20

.16

3.GREquantitative

.49

.09

.08

.34

.25

.19

4.GREanalytical

.04

.00

.25

.17

.23

5.Reference

.20

.14

.09

.12

6.Essay

.24

.27

.15

7.GPAinO.T.courses

.17

.12

8.Clientoutcomes

.28

9.Clientattendance

Note.|r| .20,p .05

|r| .27,p .01

Page255

Atabledisplayingacorrelationmatrixhasthevariablesorderedhorizontallyacrossthetoprowofthetable,andverticallydownthefirstcolumnofthetable.Notice
thatallofthecorrelationcoefficientsareintheupperrightcornerofthetable.Thevaluesalsocouldbeinthelowerleftcorerofthetable,butthiswouldberedundant
becausethesamevalueswouldbepresent.Noticethatineachrowthereisonecellwithanunderline.Thisdenotesthatavariableisbeingcorrelatedwithitself.Allof
theunderlinestakentogetherarereferredtoasthediagonal.Tointerpretacorrelationmatrix,onereadsdownthefirstcolumntofindthevariableofinterest.Next,
youproceedacrosstofindtheothervariableofinterest.Wherethesetwovariablesintersectisthecorrelationcoefficientforthetwovariables.Forexample,ifweare
interestedintherelationshipbetweentheindependentvariableundergraduateGPAandthedependentvariableclientoutcomes,weproceedacrosstherowmarked
undergraduateGPAuntilitintersectsthenumber8,whichisclientoutcomes.Thecorrelationcoefficientforthisrelationshipis.16,indicatingaweak,notstatistically
significant,positiverelationship.Belowthetable,wehavenotedthecriticalvaluesthatacorrelationcoefficientmustexceedtobesignificantat.05or.01.Wehave
placedthesecriticalvalueswithinanabsolutevaluessign.Thismeansthatacorrelationcoefficientof.20or.20issignificantatthe.05level.
Itiscommontoseecorrelationmatricesinstudiesthatareinterestedintherelationshipamongmanyvariables.However,formanyofthesestudies,thecorrelation
matrixisjustthefirststepinthedataanalysis.Inthesestudies,amultipleregressionwillbedone.Wediscussthiscommonmultivariateprocedureinchapter19.
Wheneveracorrelationmatrixisthefirststepinamorecomplexanalysis,thecorrelationcoefficientsthataredisplayedinthematrixarereferredtoaszeroorder
correlations.
UsingNonparametricAssociationalStatisticstoAnalyzeSingleFactorDesigns
SpearmanRankOrderCorrelation
Similartosinglefactordesignsdiscussedinchapters14and15,thesinglefactordesignwithmanyorderedlevelsshouldbeanalyzedwithanonparametricstatisticif
eithertheindependentordependentvariablesaremeasuredonanordinalscale(andnotnormallydistributed)orthemeasurementofoneofthevariablesviolatesthe
assumptionsunderlyingparametricstatistics.Thenonparametricstatisticmostcommonlyusedwhenassessingtherelationshipbetweenorderedindependentand
dependentvariablesistheSpearmanrankordercorrelation(rho).
MurphyandGliner(1988)wereinterestedintheperceptualandmotorprocessesofchildrenwhohaddifficultylearningmotorskills.Theyfeltthatpoormotorskills
(dependentvariable)wereduetoproblemsinperception(independentvariable).Toconductthisstudy,theymeasured38studentsonperceptualskillsandthen
correlatedtheseresultswithmeasuresofmotorskills.Theapproachtothisstudywasassociationalbecauseboththeindependentanddependentvariableshadmany
orderedlevelsandtheindependentvariablewas

Page256

anattributevariable.However,accordingtoMurphyandGliner,"Foranalysispurposes,thedatawereconceivedofasrepresentinganordinalscale,suggestingthe
useofnonparametricstatistics"(p.96).Furthermore,theypointedout,"Thepurposeoftheseanalyseswastodeterminewhethertherewasamonotonicrelationship
betweenthescoresobtainedonthemotortasksandthescoresobtainedontheperceptualtasks"(p.98).Noticethattheauthorsrefertotherelationshipbetweenthe
twovariablesasmonotonic,whichindicatesthatasonevariableincreases,theothervariablealsoincreases,butnotnecessarilybythesameamount.Hadthisbeena
Pearsonproductmomentcorrelation,therelationshipwouldhavebeendescribedaslinear.However,withaSpearmancorrelation,therelationshipbetweenthetwo
variableswillnotbelinearbecausethedataareordinal.Anexampleofordinaldatainthisstudyisasfollows.Scoringononeofthemeasuresofmotorskillswastwo
ormoreerrors:inthesequence=0points,onecardoutofsequence=1point,allcardsinsequence=2points.MurphyandGlinerselectedtheSpearmanrho
correlationtoanalyzethedatabecausetherewasonecontinuousindependentvariable,onemultilevelordereddependentvariable,andthedatawereconsideredtobe
ordinalonatleastoneofthetwovariables.
Burleigh,Farber,andGillard(1998)performedastudytodetermineifthereisarelationshipbetweencommunityintegrationandlifesatisfactionaftertraumaticbrain
injury.Usingtheassociationalapproach,Burleighetal.correlatedcommunityintegration(anattributeindependentvariable)andlifesatisfactionwiththeSpearman
correlation.ASpearmancorrelationwasselectedbecausetheinstrumentsusedtomeasuretheindependentanddependentvariableswereordinalscales.They
correlatedlifesatisfactionwithtotalcommunityintegration,anditssubscales,homeintegration,socialintegration,andproductivityintegration.Theresultsoftheir
analysesappearinTable16.2.Theyconcluded,"Spearmanrankordercorrelationsusedtoanalyzetherelationshipoflifesatisfactiontohome,social,and
productivityintegrationshowedalow,butsignificant,correlationonlybetweensocialintegrationandlifesatisfaction(rho=.37,p=.047)"(p.48).
PhiandCramer'sV
Whentheindependentanddependentvariablesarenominal,theappropriateassociationalinferentialstatisticisPhiorCramer'sV.Thesestatisticsdescribe
TABLE16.2
CorrelationofLifeSatisfactionIndexAWithTotalCommunityIntegration
QuestionnaireandItsSubscales

LifeSatisfactionIndexA

CommunityIntegrationQuestionnaire

rho

Totalcommunityintegration

.06

.77

Homeintegration

.16

.39

Socialintegration

.37

.047

Productivityintegration
Note.N=30.

.28

.13

Page257

thestrengthoftheassociationorrelationshipbetweentwonominalvariables.Theybeginwithacrosstabulationtablethatisthesameasthetableswepresentedin
chapter14whendiscussingchisquare.Phiistheappropriatestatisticfora2 2table,thatis,whentheindependentanddependentvariableseachhavetwolevels.
Cramer'sVisusedwhentherearemorethantwolevelsofoneorbothvariables.
LinearRegression
Notonlyisitimportanttoknowthatthereisarelationbetweentwovariables,butifthisrelationissufficientlystrong,itallowsustopredictonevariablefromanother
variable.Whenwediscussprediction,weareattemptingtopredictthedependentvariablefromtheindependentvariable.Thisiswhywehavediscussedthe
associationalapproachintermsofindependentanddependentvariables.Whenwehavejusttwovariables,anindependentvariableandadependentvariable,andwe
attempttopredictonefromtheother,thisiscalledlinearregression.Itshouldbenotedthattherearemanydifferenttypesofregression,butherewearetalking
aboutusingdataobtainedfromthePearsonproductmomentcorrelation.
Ifweknowthecorrelationcoefficient(r)betweenanindependentvariable(X)andadependentvariable(Y),andalsoknowthemeanandstandarddeviationofeach
ofthevariables,wecancreateanequationwhichallowsustopredictthedependentvariablefromtheindependentvariable.Oftenwhendiscussinglinearregression,
theindependentvariableisreferredtoasthepredictorvariableandthedependentvariableisreferredtoasthecriterionvariable.Thelinearregressionequationis
usedtodetermineastraightlineandisexpressedasfollows:
StartwiththeequationY'=a+bX(theequationforastraightline),where
Y'=YvaluepredictedfromaparticularXvalue.
a=YcoordinateofthepointatwhichtheregressionlineintersectstheYaxis.
b=Slopeoftheregressionline.
Let'sstartwithanexample.Frone(1998)wasinterestedinpredictingworkinjuriesamongemployedadolescents.Froneselectedanumberofindependentor
predictorvariablesthatwouldbestpredictthedependentvariableworkinjuries.Someofthesepredictorvariablesweredemographicvariables,suchasageor
gender.Otherpredictorvariableswerejobrelated,suchasjobtenure,physicalhazards,andsupervisormonitoring.Inhisstudy,Froneactuallyhad20predictor
variables.Thiswasbecausehewasattemptingtodeterminethebestcombinationofthesevariablestopredictworkinjuries.Thisprocedureiscalledmultiple
regression,andwediscussitsuseinchapter19.However,forourexamplewewillselectjustoneofFrone'spredictorvariables,workhazards,andhisdependent
variable,workinjuriestoestablishalinearregressionequation.Fronedefinedphysicalhazardsas,"theextenttowhichindividualsareexposedtodangerous
equipment,unsafeworkingconditions,andpoorenvironmentalcontrols"(pp.569570).Theindependentvariable,physicalhazards,wasmeasuredfromtheaverage
ofsevenitems.Thedependentvariable,

Page258

workinjuries,wasassessedfromaselfreportofseventypesofworkinjuriesexperiencedduringtheprevious9months.Thedataforthetwovariablescanbeseenin
Table16.3.Thecorrelationcoefficient,r,betweenthetwovariableswas.50.
Theslope,b,isfoundasfollows:

Theyintercept,a,isfoundasfollows:

Therefore,ifweknowaparticularXvalue,wecanpredicttheYvalue.Or,ifweknowaparticipant'sphysicalhazards,wecanpredictworkinjuries.Forexample,ifa
participanthadaphysicalhazardratingofthree,wewouldpredictthathewouldhaveaninjuryratingof

Ontheotherhand,ifourparticipanthadaphysicalhazardratingofone,wewouldpredictthathewouldhaveaninjuryratingof

Wecouldalsographourresults(Fig.16.1).
TABLE16.3
DescriptiveStatisticsfromtheFrone(1998)Study
SampleDescription

PhysicalHazards

WorkInjuries

Numberofparticipants

319

319

Mean

2.02

1.62

Standarddeviation

.78

.70

Page259

Fig.16.1.
Linearregressionofphysicalhazardsandinjuryrating.

Todisplaytheregressionline,wesubstituteintoourequationtofindatleastthreepoints,anddrawalinebetweenthepoints.Forourexample,theequationisas
follows:

WesubstitutethreevaluesforX.Thefirstvalueshouldbe0.Therefore,whenX=0,Y'=.73.Thenweplotthosetwocoordinates.Thesecondvaluemightbe2.
WhenX=2,Y'=1.61.Thenweplotthesetwocoordinates.Thethirdvaluemightbe4.WhenX=4,Y'=2.51.Thenweplotthesetwocoordinates.Nowwedraw
alinethroughthethreepoints.Wemustbecautiouswhenapplyinganylinearregressionequationbecausethecorrelationcoefficient,r,isnevergoingtobe1.0.Inour
presentexample,rwasonly.5.Therefore,r2,whichisanindexofthestrengthoftherelationshipbetweenthetwovariables,isonly25%.Thismeansthat75%ofthe
variancebetweenthetwovariablesisunexplained.Hadwegraphedtheactualdatapointsfromthe319participantsinthestudyforthetwovariables,mostofthe
pointswouldnotfitalongourline.Thefartherthepointsareawayfromtheline,thelowerthevalueofthecorrelationcoefficient.
Sometimes,weareinterestedinapplyingalinearregressionequationthatwasobtainedfromanotherstudy.Asecondreasontobecautiouswhenapplyingalinear
regressionequationobtainedfromanotherstudyisthatthesamplefromwhichtheequationwasobtainedmustbesimilartoyourparticipantsofinterest.Wereturnto
linearregressioninchapter19,whenweaddressmultipleregressionandothermultivariatemethods.
Summary
Inthischapterwediscussedtheselectionandapplicationofappropriatestatisticalmethodsusedintheassociationalresearchapproach.Thesestatistical

Page260

methodsareusuallyusedinsinglefactordesignswithmanyorderedlevelsoftheindependentvariable.Themostcommoncorrelationcoefficientusedtodescribethe
relationshipbetweenacontinuousindependentandadependentvariableisthePearsonproductmomentcorrelation,r.Thisvaluecanbesquaredtodeterminethe
effectsizeofacorrelationcoefficient.Whenthemeasurementoftheindependentvariableisnotattheintervallevel,orthereareviolationsprecludingtheuseof
parametricstatistics,theSpearmancorrelationismostoftenapplied.Wealsodemonstratedlinearregressionasamethodtopredictthedependentvariablewhen
thereisarelationshipbetweentheindependentanddependentvariables.
StudyAids
Concepts
Correlationmatrix
Cramer'sV
Linearregression
Pearsonproductmomentcorrelation
Phicoefficient
Spearmanrankordercorrelation
Squaredcorrelationcoefficient
Distinctions
Correlationcoefficientversussquaredcorrelationcoefficient
Correlationversuslinearregression
PearsonproductmomentcorrelationversusSpearmanrankordercorrelation
ApplicationProblems
1.Atherapistwasinterestedinpredictingsuccessonthejobfollowingtherapy.Thetherapistfeltthatagoodoutcomemeasureoftherapyatdischargewasthe
numberofhoursapersoncouldspendinasimulatedworktask.Thebestmeasureofsuccessonthejobwasnumberofmonthsincurrentemployment.Thetherapist
gathereddatafromfilesofpreviouspatients.Thedatawereasfollows:

Page261
S#

HoursSimulatedTask

MonthsCurrentEmployment

12

7.5

10

10

4.5

10

Threepatientshavejustbeendischarged.Theirworksimulationresultswere3.5,6.5,and7respectively.Howlongwouldyoupredicteachpersonwillwork,
assuminganrof0.94fromtheabovedata?
2.Astudywasperformedtodetermineifhighschoolteachingperformanceintheclassroomcouldbepredictedfromscoresonalicensureexamination.Both
measurementtoolsarescaledattheintervallevel.Thecorrelation,r,betweenteachingperformanceandlicensureexaminationwas0.5.Whatproblemsmightresultin
predictingthisteachingperformance?
3.Istherearelationshipbetweenteachingevaluationsandcoursegrades.Thehypothesiswastestedinagraduateseminarwith10students.Thestudents'gradesin
thecourserangedfromC(2)toA(4).Thecourseevaluationsrangedfromneutral(3)toverygood(5).Whatanalysisshouldbeperformedtotestthisrelationship,
andwhy?
4.Aninvestigatorperformsastudyforaninsurancecompanytodeterminetherelationship,ifany,betweenhandstrengthaftersurgeryandlengthoftimeintreatment.
Theinvestigatormeasureseachpatientaftersurgerytodeterminehandstrength.Thenshedividesthepatientsintolow,medium,andhighlengthoftimeeachpatient
spentintreatment.Inordertodetermineiftherearedifferencesamongthethreegroups,sheperformsasinglefactorANOVAwithhandstrengthasthedependent
variable.Howcouldsheusetheassociationalapproachinthisstudy?Whatwouldbetheadvantages?

Page263

Chapter17
AddingIndependentVariables
ComplexbetweenGroupsDesigns:
AnalysisandInterpretation
Overview
ReasonsforAddingaSecond(orMore)IndependentVariable(s)
TwoFactorDesignsandQuantitativeResearchApproaches
MainEffectsinMoreDetail
InteractionEffectsinMoreDetail
AnalysisofTwoFactorDesigns
SourceTableforaTwoFactorAnalysisofVariance
QuestionsAnsweredintheTwoFactorAnalysisofVariance
DescribingtheResultsintheText
InterpretationoftheResultsfromaTwoFactorANOVASourceTable
StatisticalProceduresforInterpretationofanInteractionEffect
StatisticalProceduresforInterpretationofSignificantMainEffects
Summary
StudyAids
Concepts
Distinctions
ApplicationProblems

Page264

Overview
Thethreepreviouschaptersdiscussedthedesignandanalysesofsinglefactorstudies,wheretherewasonlyoneindependentvariable.However,itisuncommonto
findincurrentliteraturethecasewhereonlyoneindependentvariableispresent.Inthischapterweintroduceasecondbetweengroupsindependentvariable,discuss
theadvantagesofhavingmorethanoneindependentvariable,anddemonstratehowthesedesigns(complexdesigns)areanalyzedandinterpreted.Inchapter18,we
continueourdiscussiononcomplexdesignsbyintroducingarepeatedmeasuresindependentvariableasthesecondindependentvariable,creatingamixeddesign.
Remember,thatinchapter12,wediscussedthethreegeneraldesignclassifications(betweengroups,withinsubjects,andmixed)andnotedthattheydeterminethe
specificstatisticstouseindataanalysis.Althoughweareaddingasecondindependentvariableinthecurrentchapter,thedesignsarestillbetweengroupsbecause
bothindependentvariablesarebetweengroupsindependentvariables.However,mostoftheexperimentaldesignsinchapter18aremixeddesigns,whichby
definitionhaveatleasttwoindependentvariables,onebetweengroupsandonewithinsubjects(repeatedmeasures).Thesemixed,twofactordesignsareoften
thosewithapretestandaposttest(withinsubjects)andwithexperimentalandcontrolgroups(betweengroups).
ReasonsforAddingaSecond(orMore)IndependentVariable(s)
Therearetwomajorreasonsforaddingasecondindependentvariableinastudy.Thefirstreasonisthataddingasecondindependentvariablegivestheresearcher
moreinformation.Whenwehavetwoindependentvariablesinasinglestudy,wecandeterminehoweachindependentvariableworksbyitselfanddeterminehowthe
twoindependentvariablesworktogetherorinteract.Howanindependentvariableworksbyitselfisreferredtoasamaineffect.Howtwoindependentvariables
interactinastudyisreferredtoasaninteractioneffect.Inastudywithtwoindependentvariables,therewillbetwomaineffects(oneforeachindependentvariable)
andaninteractioneffect.Wewouldliketoemphasizethatthetermeffectcanbemisleadingbecauseitseemstoimplyacausalrelationship.Asnotedinearlier
chapters,thisinferenceisnotjustifiediftheindependentvariableisanattribute,andmaynotbejustifiedeveniftheindependentvariableisactive.Thusyoushouldbe
cautiouswheninterpretingresults.
ConsiderastudybyRobinson,Katayama,Dubois,andDevaney(1998),whowereinterestedintheeffectsoftwoindependentvariablesonconceptapplication.The
twoindependentvariableswere(a)whetherthestudentsreviewedinformationimmediatelyordelayeddoingso,and(b)theuseofastudyaid(textonly,textplus
outline,textplusgraphicorganizer).Inthisstudy,therewasonemaineffectforwhenstudentsreviewedtheinformation,andasecondmaineffectfortheuseofa
studyaid.Therewasalsoaninteractioneffectbetweenthereviewofinformationandtheuseofastudyaid.Robinsonetal.randomlyassignedstudentstoall
conditions,thusbothindependentvariableswereactive.

Page265

Itisimportanttorecognizethatwhenindependentvariablesareaddedtoincreaseinformation,themaineffectsofbothindependentvariablesareusuallyofinterest.
Therefore,theseindependentvariablesareusuallyactive.
Thesecondreasonforusingatwofactordesigninsteadoftwosinglefactordesignsisthaterrorvarianceismorepreciselyestimated.Toexplainthisconceptually,
thenatureoferrorvariancemustbeunderstood.Errorvarianceisvariabilitythatisattributedtoindividualdifferencesamongparticipants.Sometimesthesedifferences
areduetothefactthattestsoftendonotmeasureaparticularconstructreliably.Othertimesthesedifferencesareduetothingssuchasageorgenderdifferences
amongparticipants.Itisthelattertypeoferrorthatwearetryingtoremove.Ifwecanintroduceasecondindependentvariable,suchasageorgender,thenthepart
oftheerrorvariancethatresultedfromtheseattributeswouldberemovedanddistributedasasecondindependentvariable.Theerrorvariancewouldbesignificantly
reduced.Glineretal.(1999)conductedastudytoinvestigatehiringdecisionsamongundergraduatestudents.Specifically,Glineretal.wereinterestedinthe
employabilityratingsstudentswouldgivetohiringapersonintoeitherasegregatedemploymentsetting,asupportiveemploymentsetting,oracompetitiveemployment
setting.Employmentsettingwastheindependentvariableofinterest,andstudentswererandomlyassignedtorateoneofthethreeemploymentsettings.Glineretal.
selectedstudentsfromapreoccupationaltherapyprogram,anoccupationaltherapyprogram,andabusinessprogram.Therefore,thesecondindependentvariable,
studentprogram,wasanattributeindependentvariable.SimilartothoseintheRobinsonetal.(1998)studymentionedearlier,thereweretwomaineffectsandan
interactioneffectintheGlineretal.study.Theseeffectsweretypeofemploymentcondition(maineffect),typeofstudentprogram(maineffect),andemployment
conditionbystudentprogram(interactioneffect).WhatisespeciallyimportantaboutthisstudywasthatGlineretal.wereprimarilyinterestedintheactiveindependent
variable,typeofemploymentsetting.Theotherindependentvariable,studentprogram,wasnotimportantbyitself(maineffect),butifitwassignificant,itcouldreduce
thevariabilityinthestudy.Therefore,whenattemptingtoreduceerrorbyaddingasecondindependentvariable,thesecondindependentvariableisusuallyanattribute
independentvariable.
TwoFactorDesignsandQuantitativeResearchApproaches
Twofactordesignsimplythatbothindependentvariablesarebetweengroupsindependentvariables.Inaddition,weoutlineddifferentquantitativeapproaches,
includingrandomizedexperimental,quasiexperimental,comparative,associational,anddescriptive.Thesequantitativeapproacheswerebased,forthemostpart,on
characteristicsoftheindependentvariable.Whenweaddasecondindependentvariableinastudy,dowestillhavethesamequantitativeapproaches?Theanswerto
thisquestionisyes.Webaseouransweronthecharacteristicsoftheprimaryindependentvariableinthestudy.

Page266

WestartwiththeexampleofthestudybyRobinsonetal.(1998),describedearlier.Thisstudyhastwoactiveindependentvariables,withparticipantsrandomly
assignedtotheconditionsofbothindependentvariables.Therefore,thisstudywouldfittherandomizedexperimentalapproach.NowconsiderastudybyBergin
(1994),whowasinterestedinthetypeofgoalsituationonanumberoflearningindices.Specifically,heinvestigatedthedifferencebetweenamasterygoalsituation
andacompetitivegoalsituationoninterest,learningstrategies,testachievement,andfreerecallachievement.Herandomlyassignedstudentstoeitherthemasterygoal
situationorthecompetitivegoalsituation.Asecondindependentvariablewasalsointroducedinthestudy,studentGPA.Thissecondindependentvariablehadtwo
levels,loworhigh.Wealsoclassifythisstudyintothequantitativeresearchapproachofrandomizedexperimental,becausetheprimaryindependentvariablewas
activeandstudentswererandomlyassignedtoconditions.TheGlineretal.(1999)studywouldalsofitintotherandomizedexperimentalapproach,becausethe
independentvariableofprimaryinterestwasactiveandparticipantswererandomlyassignedtoconditions.
AstudybyTuckman(1995,Experiment2)providesanexampleofatwofactordesignthatfitsthequasiexperimentalapproach.Inthisstudy,Tuckmanwas
interestedincomparinganincentivemotivationconditionwithalearningstrategyconditiononthreedifferentachievementtestsandallthreetestscombined(four
dependentvariables).Thelearningstrategyconditionwasprovidedtooneintactclassandtheincentivemotivationconditionwasprovidedtoasecondintactclass.A
secondindependentvariable,studentgradepointaverage(GPA)withthreelevels(low,medium,andhigh),wasalsoadded,similartotheBergin(1995)study.This
studyisconsideredtobeaquasiexperimentalapproachbecausetheprimaryindependentvariablewasactive,butparticipantswerenotrandomlyassignedto
classrooms.Instead,theclassroomswereselectedfromthemethodusedbytheinstructor,makingamediumlevelquasiexperimentalapproach.
AstudybyRichardsonandFergus(1993)demonstratesthetwofactorcomplexcomparativeapproach.Theywereinterestedinwhetherhighschoolstudentswho
werejudgedtobeofhigherabilitydifferedfromthosewhowerejudgedtobeoflowerabilityonlearningstyle,asmeasuredbytheInventoryofLearningProcesses.
Inaddition,RichardsonandFerguswereinterestedinwhethertheindependentvariablegenderalsoinfluencedlearningstyle.Noticethatinthisstudybothindependent
variablesareattributeindependentvariables,sothequantitativeapproachiscomparative.1
MainEffectsinMoreDetail
Tosetthestageforthetwofactoranalysisofvariance,weneedtoexaminemaineffectsandinteractioneffectsinmoredetail.Therefore,wereturntothestudyby
1

Althoughthisexampleoftwoattributeindependentvariablesfitsourcomparativeapproach,cautionmustbetakenwhenanalyzingthedatafromthistypeofstudy.Theproblem
isthatwhenbothindependentvariablesareattributeindependentvariables,thepossibilitythatthetwoindependentvariablesarecorrelatedishigh.Ifthisisthecase,the
outcomefromtheanalysiswillbebiased(seePedhazur&Schmelkin,1991,pp.537538.)

Page267

Tuckman(1996,Experiment2).RecallthatTuckmanhada2 3factorialdesign.Thetwolevelsofthefirstindependentvariablewereincentivemotivationversus
learningstrategy.ThethreelevelsofthesecondindependentvariablewereGPA:low,medium,andhigh.Wewillusejustoneofhisdependentvariables,thefirst
achievementtest,whichwasa65itemmultiplechoiceexamination.Theaveragescoresonthefirstachievementtest(percentcorrect)canbeseeninTable17.1.
Noticethatwehaveincluded,alongwiththeaveragetestscoresforeachofthesixgroups,columnaverages,rowaverages,andanoverallaverage.Thesecolumnand
rowaveragesarehelpfulwhenexaminingmaineffects.Forexample,intheTuckmanstudy,thedesignwasa3 2factorialdesign.Becausethereweretwo
independentvariables,thereweretwomaineffects,oneforeachindependentvariable.Onemaineffectisthetypeofcondition,incentivemotivationorlearning
strategy.Noticethatthemeanfortheincentivemotivationconditionis72.9(row1)andthemeanforthelearningstrategyconditionis72.5(row2).Whenweanalyze
ourdataforthisstudybyusingatwofactoranalysisofvariance,oneofthequestionsthatwillbeanswerediswhetherthemaineffectfortypeofconditionis
statisticallysignificant.Todeterminethisanswer,themeanforincentivemotivationcondition(72.9)willbecomparedwiththemeanofthelearningstrategycondition
(72.5).Thesecondmaineffectforthisstudywasgradepointaverage.Todeterminethismaineffect,wewillcomparetheaverageforthehighGPA(76.8),withthe
averageforthemediumGPA(72.4),withtheaverageofthelowGPA(68.8).
InteractionEffectsinMoreDetail
Inadditiontodeterminingifeachmaineffectisstatisticallysignificant,wearealsointerestedindeterminingiftheinteractioneffectisstatisticallysignificant.The
interactioneffectisdeterminedbyexaminingtheindividualmeanswithinthecellsofTable17.1.Tounderstandaninteractioneffect,weneedtoexaminepatterns
amongtheseindividualcells.Forexample,ifwelookacrossthethreecellsofrowone(incentivemotivationcondition)ofTable17.1,weseethatthethreemeansare
almostequal(73.7,72.2,72.7).Ontheotherhand,ifwelookacrossthethreecellsofrowtwo(learningstrategycondition)ofTable17.1,weseethatthemeansare
decreasingfromhighGPA(79.9)tomediumGPA(72.6)tolowGPA(64.9).
Thefirststepintheexaminationofcomplexstudies,wheremorethanoneindependentvariableisinvolved,istographthedata.Whensettingupanygraph,always
placethedependentvariableontheY(vertical)axis.Whentherearetwoindependentvariablesinvolved,aruleofthumbistoplacetheattributeindependentvariable
ontheX(horizontal)axis,andgraphtheactiveindepend
TABLE17.1
AverageTestScoresforTest1,Experiment2FromTuckman(1995)Study
Condition

HighGPA

MediumGPA

LowGPA

Average

Incentivemotivation

73.7

72.2

72.7

72.9

Learningstrategy

79.9

72.6

64.9

72.5

Average

76.8

72.4

68.8

72.7

Page268

entvariablewithseparatelines,aswehavedoneinFig.17.1.Figure17.1demonstratesadisordinalinteraction.Inadisordinalinteraction,thelinesonthegraph
cross.IfwehadnotincludedthelowGPAcondition,butonlythemediumandhighGPAconditions,theinteractionwouldbecalledordinal.Aninteractionissaidto
beordinalifthelinesareclearlynotparallelbutdonotcrosswithinthegraph.Whenthereisnointeraction,thelinesusuallyrunparalleltoeachother.Sometimesitis
helpfultousebargraphstoexamineinformationfromtwofactorexperiments.Figure17.2isanexampleofthesamedataplottedasabargraph.ForthehighGPA
group,weseethatthelearningstrategyconditionisabout6percentagepointshigherthantheincentivemotivationcondition.WhenwelookatthemediumGPA
group,weseethatthetwoconditionshavealmostthesamepercentagepoints.Ontheotherhand,whenwelookatthelowGPAgroup,weseeareverseofthehigh
GPAgroup.Theincentivemotivationconditionisalmosteightpercentagepointshigherthanthelearningstrategycondition.AccordingtoKerlinger(1986),
''Interactionoccurswhenanindependentvariablehasdifferenteffectsonadependentvari

Fig.17.1.
Plotoftheinteractionofconditionwithgradepointaverage(GPA).

Fig.17.2.
Bargraphoftheinteractionofconditionwithgradepointaverage(GPA).

Page269

ableatdifferentlevelsofanotherindependentvariable"(p.230).Whatthismeansisthatwhenweobserveasignificantinteractioneffect,wewillnoticethatatone
levelofanindependentvariable,mediumGPAinourexample,thereisnosignificantdifferencebetweenthetwoconditionsoftheotherindependentvariableincentive
motivation =64.9).Althoughwehaveexaminedthedatavisually,theonlywaytoknowwhichconditionsaredifferentstatisticallyistoperformtheproperanalysis,
whichforthepresentexampleisatwofactorANOVA,withappropriatefollowuptestsifneeded.
AnalysisofTwoFactorDesigns
Regardlessofthetypeofquantitativeresearchapproach,twofactordesignsareanalyzedbytwofactoranalysisofvarianceaslongasbothindependentvariablesare
betweengroupsindependentvariables,andthedataapproximateanintervalscaleandotherANOVAassumptions(seechap.14)arenotmarkedlyviolated.For
thosestudieswithtwoindependentvariablesthathavedatathatareordinalorcategorical,therearenotmanyoptions.Thesestudiesarefrequentlyanalyzedby
nonparametrictechniquesappliedtooneindependentvariableatatime,buttheinteractioneffectislost.Therearemoresophisticatedtechniques,suchasloglinear
analysis,whicharebeyondthescopeofthistext.
SourceTableforaTwoFactorAnalysisofVariance
Inchapter14wediscussedthesinglefactoranalysisofvariance.Allanalysisofvarianceprocedureshaveanaccompanyingsourcetable.Inchapter14,thissource
tablewasdepictedinTable14.3.Forthesinglefactoranalysisofvariance,thevarianceisdividedintothatduetotreatment,usuallyreferredtoasbetweengroups,
andthatduetoerror,usuallydescribedaswithinsubjects.ThegeneralizedsourcetableforthetwofactorANOVAispresentedinTable17.2.Similartotheoneway
ANOVA,thetwowayANOVAstartsbydividingthesumsofsquaresintoatreatmentcomponentandanerrorcomponent.Next,thetreatmentcomponentis
dividedintoasumsofsquaresforindependentvariableA,asumsofsquaresforindependentvariableB,andtheremainderistheinteractionsumsofsquaresAB.
Eachofthesesumsofsquaresisdividedbydegreesoffreedomtoobtainmeansquares.Therewillbefourmeansquares.Thedegreesoffreedomforindependent
variableAareq(thenumberoflevelsofindependentvariableA)minusone.ThedegreesoffreedomforindependentvariableBarer(thenumberoflevelsof
independentvariableB)minusone.TheinteractiondegreesoffreedomarecomputedbymultiplyingthedegreesoffreedomforindependentvariableAtimesthe
degreesoffreedomofindependentvariableB.Thedegreesoffreedomfortheerrortermarecomputedbymultiplyingthe

Page270

numberoflevelsofindependentvariableAtimesthenumberoflevelsofindependentvariableBtimesthesamplesizeforanycell(n)minusone.Therewillbefour
meansquares(MS)inthesourcetable:ameansquareforindependentvariableA,ameansquareforindependentvariableB,ameansquareinteractionAB,anda
meansquareerror.Oncethemeansquareshavebeencalculated,theFvaluesareobtained.
RememberthatinatwofactorANOVAtherearethreeFsandthreequestionsthatwecananswer,oneabouteachmaineffectandtheinteraction.Eachofthethree
FvaluesisobtainedbydividingtheMSforthatsourceofvariationbytheMSforerror,asseeninthelastcolumninTable17.2.
TounderstandtheroleofthesourcetableinatwowayANOVA,let'sexaminethedatafromthestudybyTuckman(1995,Experiment2).Earlierwedescribedthe
meansforeachofthegroupsinthisstudyinTable17.1.NowweexaminethesourcetablefromthatstudyinTable17.3.
QuestionsAnsweredintheTwoFactorAnalysisofVariance
Inaonewayorsinglefactordesign,onehypothesisistested,theeffectofthatindependentvariable,forexample,thetreatment.Inthistwowayortwofactordesign,
threehypothesesweretested:(a)thereisadifferencebetweenthetwoconditionsofindependentvariableA(incentivemotivationversuslearningstrategy)(b)thereis
adifferenceamongthethreeGPAgroupingsofindependentvariableBand(c)thereisaninteractionofindependentvariablesAandB.Inthisexample,thefirst
hypothesiswasnotsupported.Therewasnosignificantdifferencebetweentheincentivemotivationconditionandthelearningstrategycondition,asevidencedbya
verysmallFvalueof.40.Thesecondhypothesiswassupported.TherewasasignificantdifferenceamongthethreeGPAgroupings,withanFvalueof4.71,which
wassignificantatthe.05level.Thethirdhypothesis,theinteractionbetweenthetwoindependentvariables,alsowassupported.TheFvalueof3.46wassignificantat
the.05level.
DescribingtheResultsintheText
AlthoughwehavepresentedthedatafromtheTuckman(1995,Experiment2)twowayANOVAinasourcetable,itisnotuncommonformanyjournalstohave
authorsreporttheirresultsintextformtosavespace.ThedatafromTable17.3wouldbereportedasfollows.
TABLE17.2
TwoFactorAnalysisofVarianceSourceTable
SourceofVariation

SS

IndependentvariableA

df
q1

MS

MSA

Mserror

r1

IndependentvariableB

MSB

Mserror

InteractionA B
Errororwithinsubjects

(q1)(r1)

MSAB

qr(n1)

MSerror

Page271

Therewasnosignificantdifferencebetweentheincentivemotivationconditionandthelearningstrategycondition,F(1,109)=.40,NS.Therewasasignificantmain
effectforGPA,F(2,109)=4.71,p<.05.Therealsowasasignificantinteractioneffect,F(2,109)=3.46,p<.05.Whenpresentingtheresultsintextform,the
degreesoffreedomareplacedinparentheses.Thedegreesoffreedomreportedarethosefortheparticularsourceofvariation,suchasaparticularmaineffect,
dividedbythedegreesoffreedomoftheerrorterm.Iftheoutcomeisnotsignificant,theabbreviationNSisoftenreported.However,becausemostanalysesthese
daysareperformedbyastatisticalpackageonacomputer,theactualpvalueshouldbereported.
InterpretationoftheResultsfromaTwoFactorANOVASourceTable
Table17.3demonstratedthattherewasasignificantmaineffectforGPAandasignificantinteractioneffectbetweenthetwoconditionsandGPA.Whenthereisa
statisticallysignificantinteractioneffect,itshouldbeinterpretedfirst,beforeanysignificantmaineffect.Thereasonforgivingmoreweighttointerpretingtheinteraction
effectisthatitprovidesthesimplest(mostparsimonious)explanation.Italsogivesmoreaccurate,lesspotentiallymisleadinginformation.Intheexampleearlier,the
maineffectsuggeststhattherearesignificantdifferencesamongthethreeGPAgroups.However,acloserexaminationofthemeansprovidedinTable17.1showsthat
thesedifferencesareprimarilyduetothelearningstrategycondition.Infact,thedifferencesamongthethreeGPAgroupsfortheincentivemotivationconditionare
small.Thereisasecondreasonforinterpretingasignificantinteractioneffectbeforeasignificantmaineffectinthepresentstudy.Theauthorwasnotparticularly
interestedinGPAbyitself.Aswementionedearlier,oftenwhenthesecondindependentvariableisanattributeindependentvariable,thepurposeistoreduceerror
variation.Inthepresentstudy,Tuckman(1996)referredtoGPAasamoderatorvariable.Inotherwords,GPAwouldaffecttheincentivemotivationcondition
differentlythanthelearningstrategycondition.
Theinformationprovidedinthesourcetable,thatthereisasignificantinteractioneffect,isnotsufficientforinterpretationofthatinteractioneffect.Instead,followup
statisticalproceduresneedtobeperformedtopinpointtheinteractioneffect.
TABLE17.3
TwoFactorANOVASourceTablefromTuckman(1995,Experiment2)UsingtheFirstTestDataScores
astheDependentVariable
Source
Condition
GPA
ConditionxGPA
Withinsubjects(error)
*p<.05.

SS

df

MS

4.82

4.82

.40

112.46

56.23

4.71*

82.62

41.31

3.46*

1302.55

109

11.95

Page272

StatisticalProceduresforInterpretationofanInteractionEffect
Wheneverasignificantinteractioneffectisobtained,furtherstatisticalproceduresmustbeimplemented.Thesestatisticalproceduresarereferredtoassimplemain
effectsanalysesandposthoccomparisons.Simplemaineffectsanalysisisastatisticalprocedurethattakesadvantageoftheinformationalreadycompiledfrom
computingthetwofactorANOVA.PerformingsimplemaineffectsissimilartoperformingsinglefactorANOVAsoneithertherowsorthecolumnsinTable17.1.2If
simplemaineffectswereperformedforthecolumnconditions,therewouldbethreesimplemaineffects,highGPA,mediumGPA,andlowGPA.Eachsimplemain
effectwouldbetestedtodetermineiftherewasasignificantdifference.Inthepresentstudy,itappearedthattherewasnosignificantsimplemaineffectforthehigh
GPAconditionorthemediumGPAcondition,buttherewasasignificantsimplemaineffectforthelowGPAcondition.Tuckman(1996)concluded,"Eachsignificant
interactionresultedfromlowGPAstudentsdoingmuchbetterintheincentivemotivationconditionthaninthelearningstrategycondition,whereasmiddleGPAand
highGPAstudentsdidequallywellineachtreatmentcondition"(p.204).
Simplemaineffectsanalysiscouldhavebeenperformedforthetworowconditionsinsteadofthethreecolumnconditions.However,iftherewerestatistically
significantdifferencesforeitheroftherowconditions,followupposthocanalyseswouldhavetobeperformed,similartoaonewayANOVA(chap.14)because
therearethreelevelsineachoftherowconditions.
StatisticalProceduresforInterpretationofSignificantMainEffects
Earlier,westatedthatinterpretationofasignificantinteractioneffectshouldalwaysprecedeinterpretationofsignificantmaineffects.However,itisnotuncommonto
findstudieswheretherearenosignificantinteractioneffects,butoneorbothmaineffectsaresignificant.Table17.4showsdatafromTuckman(1996,Experiment2)
forhisthirdtest(dependentvariable).AtwofactorANOVAperformedonthedatarevealedthefollowingsourcetable(Table17.5).
Noticethatwhereasbothmaineffectsarestatisticallysignificant,theinteractioneffectisnotsignificant.Thefollowupstatisticalproceduresforinterpretationofmain
effectsarestraightforward.Ifthemaineffectissignificantandhasonlytwolevels,thennofurtheranalysisisneeded.Becausethereisasignificantdifference,the
differencemustbebetweenthemeansofthetwoconditions.Examinationofthemeansshouldrevealwhichconditionperformedbetter.Thisisevidencedbythemain
effectforcondition,whichteststhedifferencebetweentheincentivemotivationconditionandthelearningstrategycondition.DatafromTable17.4demonstratethat
studentsperformedsignificantlybetterintheincentivemotivationcondition( =71.7).AsignificantmaineffectalsowasfoundforGPA.However,becausethere
werethreeconditionsofGPA,afollowuporposthoctest,suchasaTukeyHSD,needstobeperformedtodeterminewherethethreeGPAconditionsaredifferent.
2

Althoughitispossibletoperformsimplemaineffectsonboththerowsandthecolumns,theinformationfromperformingbothis,forthemostpart,redundant.

Page273
TABLE17.4
AverageTestScoresforTest3,Experiment2,FromtheTuckman(1995)Study
Condition

HighGPA

MediumGPA

LowGPA

Average

Incentivemotivation

80.2

72.7

75.4

76.1

Learningstrategy

78.7

70.9

65.5

71.7

Average

79.4

71.8

70.5

73.9

TABLE17.5
TwoFactorANOVASourceTableFromTuckman(1996,Experiment2)ThirdTestData
Source

SS

df

MS

Condition

51.91

51.91

4.03*

GPA

182.80

91.40

7.10**

ConditionxGPA

34.42

17.21

1403.92

109

12.88

Withinsubjects(error)

1.34

*p<.05.**p<.01

Summary
Weintroducedcomplexdesignsinthischapter,startingwithcomplexbetweengroupsdesigns,ormorespecifically,twofactordesigns.Complexdesignsaredesigns
withmorethanoneindependentvariable.Thetwomajorreasonsforaddingasecondindependentvariableinastudyaretoprovidetheresearcherwithmore
informationandtoreduceerrorvariance.Eventhoughthedesignsarecomplex,theystillfallintooneofthequantitativeapproachesdescribedinchapter5.We
describedmaineffectsandinteractioneffectsandprovidedexamplesofeach.Thetwofactoranalysisofvariancewasintroducedastheproperanalysisforthetwo
factordesign.Thisanalysisanswersthreedifferentquestions,oneforeachmaineffectinthestudyandonefortheinteractioneffect.Followupposthoctestsfollowing
significantmaineffectsweredescribed.Simplemaineffectsanalysisfollowingsignificantinteractioneffectswereexplained.Emphasiswasplacedontheproper
interpretationofinteractioneffects.
StudyAids
Concepts
Disordinalinteraction
Interactioneffects
Maineffects
Ordinalinteraction

Page274

Simpleeffects
TwofactorANOVA
Distinctions
Maineffectsversusinteractioneffects
Ordinalversusdisordinalinteraction
Simpleeffectsversusposthoccomparisons
ApplicationProblems
1.Whatisafactorialdesign?Whywouldyouuseafactorialdesign?
2.Whatisthedifferencebetweenamaineffectandaninteractioneffect?
3.Whywouldyoudooneanalysis(factorialANOVA)insteadoftwoseparateanalyses(e.g.,ttests)whenyouhavetwoindependentvariables?
4.Whyisitimportanttolookfirstatinteractioneffects?
5.Afacultymemberconductedastudytodeterminewhoperformedbetterinhisresearchclass,thosestudentsineducationorthosestudentsinoccupationaltherapy.
Inaddition,hefeltthatgendercouldalsomakeacontribution.Therefore,heconductedaretrospectivestudy(i.e.,wentbackintopreviousrecords)andformedthe
followingfourgroupsoftenstudentsineachgroup:Maleeducation(ME),femaleeducation(FE),maleOT(MOT),femaleOT(FOT).Dr.Gthencalculatedthe
meantestscoresforeachgrouptheywereasfollows:
ME=81
FE=93
MOT=89
FOT=84
a.Graphtheabovedata.
b.Thesumsofsquaresfortheresultsareinthefollowingsourcetable.Completeit.
c.IfanFof4.11isrequiredforstatisticalsignificance,interprettheresults.
SourceTable
Source

SS

Major

420

Gender

250

Majorbygender

600

Withinsubjects(error)

5200

df

MS

6.Aninvestigatorwasinterestedintheeffectofteachingstyleonstudents'perceptionofcredibilityoftheirinstructor.Onestyleofinterestwastheparticipatoryaction
styleinwhichthestudentstookresponsibilityformuchoftheclassmaterial.Theotherstyleofinterestwaslabeledthetraditionalstyle,inwhichdeliveryofmaterial
wasbylecture.Twoinstructorsfromthesamede

Page275

partmenttaughtthesameclassoneinstructorwasskilledintheparticipatorylearningstyle,whiletheotherwasskilledinthetraditionalstyle.Theinvestigatoralso
thoughtthattheageofstudentsmightaffectthisresearchprojectduetodifferentexpectationsamongstudents.Theinvestigatordecidedtoselectthreedifferentage
groupsfortheproject:young,middle,andolder.TheinvestigatorperformsatwofactorANOVA.Whatshouldtheinvestigatordoundereachofthefollowing
situations:
a.Theinvestigatorfindsasignificantmaineffectforteachingstyle,butnosignificantmaineffectforage,orteachingstylebyageinteraction.
b.Theinvestigatorfindsasignificantmaineffectforage,butnosignificantmaineffectforteachingstyleandnoteachingstylebyageinteraction.
c.Theinvestigatorfindsnosignificantmaineffectsforteachingstyleorage,butasignificantteachingstylebyageinteraction.
d.Theinvestigatorfindssignificantmaineffectsforteachingstyleandage,andateachingstylebyageinteraction.

Page277

Chapter18
MixedFactorialDesigns:
AnalysisandInterpretation
Introduction
TheTrueMixedDesign
AnalyzingtheTrueMixedDesign
SourceTablefortheMixedANOVA
ExampleSourceTable
ThePretestPosttestDesign
AnalyzingthePretestPosttestDesign
TheMixedANOVAApproach
TheGainScoreApproach
TheAnalysisofCovarianceApproach
ANoteontheGeneralUseofANCOVA
Summary
StudyAids
Concepts
Distinctions
ApplicationProblems

Page278

Introduction
Inthischapterwediscusshowtoanalyzeandinterpretthemixedfactorialdesign.Previously,wedefinedthemixeddesignasadesignthathasaminimumoftwo
independentvariables.Oneoftheindependentvariablesmustbeabetweengroupsindependentvariable.Theotherindependentvariablemustbeawithinsubjects
independentvariable.Sometimesmixeddesignshavemorethantwoindependentvariables.Forexample,amixeddesignmighthavethreeindependentvariables,with
twobetweengroupsindependentvariablesandonewithinsubjectsindependentvariable.Or,amixeddesignmighthavethreeindependentvariables,withone
betweengroupsindependentvariableandtwowithinsubjectsindependentvariables.However,theminimumrequirementstillholds:theremustbeatleastone
betweengroupsindependentvariableandonewithinsubjectsindependentvariableforthedesigntobeamixeddesign.Becausemixeddesignsarethemostcomplex
designsthatwehavecovered,welimitourdiscussiontothesimplestmixeddesign,adesignwithonebetweengroupsindependentvariableandonewithinsubjects
independentvariable.Thisdesignisoftenreferredtoasatwofactordesignwithrepeatedmeasuresonthesecondfactor.
Similartotheapproachesfortheotherdesignsthatwehavediscussedpreviously,thedifferentquantitativeapproachesapplytothemixeddesign.Forthemostpart,
theseapproachesarerandomizedexperimentalandquasiexperimental.Comparativeapproachesthatusethemixeddesignarelesscommon,butdooccur.Welabel
thedesignasoneofthesethreeapproaches,dependingonthehighestapproachused,usuallynotedfromthebetweengroupsindependentvariable.Forexample,if
participantswererandomlyassignedononeoftheindependentvariables,thentheapproachisrandomizedexperimental.Jongbloedetal.(1989)randomlyassigned
participantswhohadexperiencedastroketotwodifferenttreatmentconditions(betweengroupsindependentvariable),andthenmeasuredthemovertime(within
subjectsindependentvariable).LanandRepman(1995)hadtwobetweengroupsindependentvariables,modelingandgradelevel,andonewithinsubjects
independentvariable,context.Becauseparticipantswererandomlyassignedtooneoftheindependentvariables,modeling,theapproachisconsideredtobe
randomizedexperimental.
Louth,McAllister,andMcAllister(1992)wereinterestedintheeffectsofcollaborativewritingtechniques(activeindependentvariable)onfreshmanwriting.Students
werealreadyinsectionsoffreshmancomposition(intactgroups)sotheapproachwasquasiexperimental.Sectionswererandomlyassignedtooneofthreewriting
conditions(betweengroupsindependentvariable)andmeasuredovertime(withinsubjectsindependentvariable).Becausethebetweengroupsindependentvariable
involvedrandomassignmentoftreatmentstointactgroups,thestudywasconsideredtobeastrongquasiexperimentalapproach.GilfoyleandGliner(1985)were
interestedintheimpactofaneducationprogramonattitudestowardhandicappedchildren.Threeschoolsparticipatedinthestudy.Twoschoolsreceivedthe
interventionandthethirdschoolservedasthecontrolcondition.Studentsintheschoolswereassessedovertime.The

Page279

quantitativeapproachforthestudywasconsideredtobequasiexperimentalofmediumstrengthbecausestudentswereinintactgroups(schools)ratherthanbeing
randomlyassignedtotheinterventionorthecontrolcondition,andthetreatmentwasnotrandomlyassignedtothegroups.
AmixeddesignexamplethatusedthecomparativeapproachcanbeseeninastudybyChambers(1994).Chamberswasinterestedinassessingdifferenttypesof
leadersinthefieldofeducation.Hesentaquestionnairetothreedifferenttypesofleadersineducation.Theywerestudentaffairsleadershipeducators,academic
affairsleadershipeducators,andcommunitybasedleadershipeducators.Theseleadershiptypesformedthebetweengroupsindependentvariablewiththreelevels.
Thisindependentvariablewasanattributeindependentvariable.Eachparticipantinthestudyratedtheimportanceinevaluatingcollegestudentleadershipprograms
onthecriteriaofprogramstructuring,methodology,programadministration,andconsequences.Thus,thesecondindependentvariable,awithinsubjectsindependent
variable,waslabeledevaluationcategorywithfourlevels.Therefore,thedesignfortheChambersstudywasamixeddesign,andbecausebothindependentvariables
wereattributes,theresearchapproachwasconsideredtobecomparative.
Forourdiscussion,wedividemixeddesignsintotwodifferentcategories.Onecategorywerefertoastruemixeddesigns.Inthiscategory,thebetweengroups
independentvariablemaybeactive(usually)orattribute,butthewithinsubjectsindependentvariablemustbeactive.Welabeloursecondcategoryofmixeddesigns
PretestPosttestdesigns.Again,thebetweengroupsindependentvariableisusuallyactive,butcouldbeanattributeindependentvariable.However,thewithin
subjectsindependentvariableinpretestposttestdesignsisneitheractivenorattribute,butreferredtoaschangeovertime.Whilethisisaspecialcircumstance,itis
commoninallareasofappliedresearch.Whenwedescribeeithercategoryofdesign,thetermmixedortwofactordesignwithrepeatedmeasuresonthesecond
factorappliestoboth.However,althoughtheanalysisofthetruemixeddesignisalmostalwaysatwofactorANOVAwithrepeatedmeasuresonthesecondfactor,
thereareatleastthreedifferentmethodstoanalyzethepretestposttestdesign.
TheTrueMixedDesign
Thetruemixeddesignhasonebetweengroupsindependentvariablethatisalmostalwaysactiveandonewithinsubjectsvariablethatisalwaysactive.
AnalyzingtheTrueMixedDesign
Toanalyzedatafromadesignwithatleastonebetweengroupsindependentvariableandonewithinsubjectsindependentvariable,theproperanalysisisamixed
analysisofvariance(ANOVA).Inthepresentchapter,forexplanatorypurposes,weonlydiscussmixeddesignswithonebetweengroupsindependentvariableand
onewithinsubjectsindependentvariable.Therefore,whenwearereferringtothemixedANOVAinthischapter,itcouldalsobecalledatwofactorANOVAwith
repeatedmeasuresonthesecondfactor.ThemixedANOVAanswersthesamethreequestionsthatwereansweredbyatwofactor

Page280

ANOVAdiscussedinchapter17.Thethreequestionsare(a)Isthereasignificantmaineffectforthefirstindependentvariable(inthiscase,thebetweengroups
independentvariable)?(b)Isthereasignificantmaineffectforthesecondindependentvariable(inthiscase,thewithinsubjectsindependentvariable)?and(c)Isthere
asignificantinteractioneffectbetweenthetwoindependentvariables?WhilethequestionsaskedinthetwofactormixedANOVAaresimilartothoseofthetwo
factorANOVA,theactualanalysisisdifferent.Weseethiswhenweshowthesourcetable.Let'sstartwithanexampleofatruemixeddesignaboutchildren's
persistence.
LanandRepman(1995)wereinterestedinelementaryschoolagedchildren'sresponsestosuccessandfailureonamathematicstask.Specifically,theyinvestigated
theindependentvariablesofmodeling,collaboration,andgradelevelonthedependentvariable,persistence.Tosimplifyourdiscussion,wewillleaveoutthe
independentvariable,gradelevel.Thebetweengroupsindependentvariablewasthemodelingcondition,withtwolevels,modelingandnomodeling.Inthemodeling
condition,participantsreadastoryonacomputerpriortoproblemselectionaboutaboywho,eventhoughexperiencingfailure,chosedifficultmathematicsproblems
andimprovedinmathematicsachievement.Thenonmodelingconditionparticipantsdidnotreceivethestory.Therepeatedmeasuresindependentvariablewas
collaboration,withtwolevels,performingindividually,andperformingcollaborativelywithanotherparticipant.Theorderofparticipationinthesetwoconditionswas
randomlyassigned.Therefore,thedesignforthestudybyLanandRepman(aswereportit)isatwobytwofactorialdesignwithrepeatedmeasuresonthesecond
factor.Oneofthedependentvariablesforthestudywaspersistence,whichwasdeterminedbytheparticipants'choiceoflevelofproblemsafterfailingaproblemata
particularlevel.Thisscorerangedfrom0to2.Table18.1showstheaveragepersistencedataforthestudy.
SourceTableforMixedANOVA
Table18.2showsthesourcetableforthetwofactorANOVAwithrepeatedmeasuresonthesecondfactor.Notice,inthelastcolumn,thatthreedifferentFvalues
areobtained,similartothoseobtainedinthetwofactorANOVA.Furthermore,eachFvalueanswersquestionssimilartothoseinthetwofactorANOVA.For
example,thereisanFvaluecomputedforthemaineffectforindependentvariableA.Inthepresentcase,independentvariableAisalwaysthebetweengroups
independentvariable.ThereisalsoanFvalueforindependentvariableB,which,inthemixedANOVA,isalwaystherepeatedmeasuresindependentvariable.Last,
thereisanFvaluefortheinteractionbetweenthetwoindependentvariables.Thema
TABLE18.1
DataFromtheLanandRepman(1995)Study
Condition

Individual

Collaborative

Average

Modeling

1.25

1.16

1.21

NoModeling

1.11

1.23

1.17

Average

1.18

1.20

1.19

Page281

jordifferencebetweenthetwofactorANOVAandthetwofactorANOVAwithrepeatedmeasuresonthesecondfactor(mixedANOVA)isthattheFvaluesare
calculateddifferently.InthemixedANOVA,therearetwoerrorterms,oneforthebetweengroupsindependentvariable(calledAerror),andonefortherepeated
measuresindependentvariable(calledBerror).ThebetweengroupsindependentvariableFisdeterminedbyusingtheerrortermfromthebetweengroupspartofthe
analysis.TherepeatedmeasuresindependentvariableFandtheinteractioneffectFaredeterminedbyusingtherepeatedmeasureserrorterm.
ExampleSourceTable
Let'slookatthesourcetablefromtheLanandRepman(1995)studyonpersistencedata.Wehavemodifiedtheirsourcetablefromtheoriginalarticlebecausewe
leftoutthethirdindependentvariable,gradelevel.ThesourcetablecanbeseeninTable18.3.ThedatadescribedinthesourcetablefromtheLanandRepman
(1995)studyrevealnosignificantmaineffectsfortheindependentvariablesofmodeling,F(1,136)=1.08,p=.30orcontext,F(1,136)=.29,p=.57,butathere
issignificantmodelingbycontextinteractioneffect,F(1,136)=10.43,p<.01.Figure18.1demonstratesthemodelingbycontextinteractioneffect.
TABLE18.2
SourceTablefortheMixedANOVA
SourceofVariation

SS

Betweengroups

df

Aerror

q(n1)

(r1)

AB

(q1)(r1)

Berror

(q1)

Withinsubjects

MS

q(n1)(r1)

MSA

MSAerror

MSB

MSBerror

MSAB

MSBerror

TABLE18.3
ModifiedSourceTableonPersistenceDataFromLanandRepman(1995)
SourceofVariation
Betweensubjects

SS

Modeling

.14

Errorbetween

17.98

Withinsubjects

df

MS

.14

136

.13

1.08

Context

.02

.02

.29

Modelingcontext

.73

.73

10.43*

8.93

136

.07

Errorwithin
*p<.01.

Page282

Fig.18.1.
PlotofaninteractiondrawnfromthedatainLanandRepman(1995).

Theinteractioneffectisinterpretedasfollows.Whenparticipantsperformintheindividualcondition,thoseinthemodelinggrouparemorepersistentthanthoseinthe
nomodelinggroup.Ontheotherhand,whenparticipantsareinthecollaborationcondition,thoseinthemodelinggroupseemtoperformworsethanthoseintheno
modelingcondition.LanandRepman(1995)testedthisinterpretationoftheinteractioneffectbyusingsimpleeffectsanalysis(chap.17).Interestingly,theyfound
significantdifferencesbetweenthemodelingandnomodelinggroupswhentheparticipantsperformedthetaskintheindividualsituation.However,whenparticipants
performedthetaskinthecollaborativesituation,therewerenosignificantdifferencesbetweenthemodelingandnomodelinggroups.Thisstudyshowstheneedto
followupsignificantinteractioneffectsfromanyfactorialANOVAwithsimplemaineffectsanalysis.Thus,eventhoughthegraphofthedataappearedtodemonstrate
significantdifferencesunderbothconditions,statisticalanalysisrevealedsignificantdifferencesunderonlyoneoftheconditions.
ThePretestPosttestDesign
AnalyzingthePretestPosttestDesign
Becauseoftheextensiveuseofthepretestposttestdesign,wheretypeofinterventionisthebetweengroupsindependentvariableandchangeovertimeisthe
repeatedmeasuresfactor(mixeddesign),muchhasbeenwrittenontheanalysis(Huck&McLean,1975Reichardt,1979).Wediscussthreedifferentmethodsused
toanalyzethemixeddesign,themixedANOVAapproach,thegainscoreapproach,andtheanalysisofcovariance(ANCOVA)approach.Asanexamplefor
thethreedifferentapproaches,considerthestudybyLouthetal.(1992),whoexaminedtheeffectsofcollaborativewritingtechniquesonfresh

Page283

manwritingandattitudes.Therewerethreetreatmentgroupsinthisstudy,consistingofsixsectionsoffreshmancompositionstudents.Twosectionswereinstructedto
writeinteractively,twosectionswereinstructedtousegroupwriting,andtwosectionswereinstructedtowriteindependently.Thedesignforthestudywasa3 2
factorialdesignwithrepeatedmeasuresonthesecondfactor.Thisisamixeddesign.Thebetweengroupsindependentvariableistreatment,withthethreewriting
conditionsasthethreelevels.Thewithinsubjectsorrepeatedmeasuresindependentvariableischangeovertime,withtwolevels,pretestandposttest.Thedifferent
sectionswererandomlyassignedtothetreatmentconditions(strongquasiexperimentalapproach).Allstudentswrotea50minutepretestessaybeforethestudy
began.After8weeks,allstudentsweregivenaposttestbyusingthesameessayinstructionsasinthepretest.Englishfacultyscoredbothpretestsandposttestsnaive
tothetreatmentconditionsofthestudents.Theessayswerescoredona6pointscale.ThedataforthestudycanbeseeninTable18.4andisalsographedinFig.
18.2.
TheMixedANOVAApproach
WediscussedthemixedANOVAindetailearlierinthischapter.Whenusedwiththepretestposttestdesign,theanalysisisthesame,generatingthesamesource
table.Likeanytwofactoranalysisofvariance,theoutcomeresultsinanswerstothreeseparatequestions.Arethere
TABLE18.4
MeanWritingScoresFromLouthetal.,1992

TestTime

Intervention

Pretest

Posttest

Independent

44

3.00

3.07

Interactive

45

2.37

3.01

Group

47

2.27

2.84

Fig.18.2.
PretestPosttestplotdrawnfromtheLouthetal.(1992)data.

Page284

significantdifferencesamongthelevelsofthefirstindependentvariable(maineffectforthebetweengroupsindependentvariable)?Aretheresignificantdifferences
betweenthelevelsoftherepeatedmeasuresindependentvariable(maineffectforthewithinsubjectsindependentvariable)?Aretheresignificantinteractioneffectsas
aresultofthetwoindependentvariables?Let'sexaminethemixedANOVAapproachfortheexampledemonstratedinFig.18.2.
WhenweexaminetheexampleinFig.18.2,weseethattheonlyquestionofinteresttousistheresultsoftheinteractioneffect.Why?First,thinkaboutthemaineffect
forthebetweengroupsindependentvariable,typeofwritingcondition.Whatwouldasignificantmaineffectforwritingconditiontellus?Itwouldmeanthatthereisat
leastonedifferenceamongthethreewritingconditions,averagedacrossthepretestandtheposttestmeasurementperiods.Inotherwords,itwouldnotdemonstrate
whichinterventionconditionactuallyworkedbecausethepretestscoresarecombinedwiththeposttestscores.Therefore,thebetweengroupsmaineffectinthe
mixedANOVAapproachtoanalysisofthepretestposttestdesignprovideslittlemeaningfulinformation.Nowlet'sexaminethesecondmaineffect,thewithin
subjectsmaineffect.Whatwouldasignificantdifferencetellus?Asignificantmaineffectfortimetellsusthatthereisachangefrompretestscorestoposttestscores
averagedacrossallthreegroups.Itdoesnottelluswhichgroupschangedsignificantly,onlythat,ontheaverage,allthreegroupsorconditionschanged.Itshouldbe
notedthatthisisnotuncommonintherapeuticinterventions,wherepatientsareapttogetbetterovertime,regardlessoftreatment.Thisisoftenreferredtoasthe
internalvaliditythreatofmaturation.Thus,asignificantwithinsubjectsmaineffect,time,alsodoesnotprovideuswiththeinformationofinterest.Nowwewillexamine
themeaningofasignificantinteractioneffectinthepretestposttestdesign.
IfwestudyFig.18.2,weseethatthetrendsofthethreedifferentinterventionsfrompretesttoposttestarenotsimilar.Thesectionsthatwererequiredtowrite
independentlyremainalmostthesamefrompretesttoposttest.Ontheotherhand,theinteractivewritingsectionsandthegroupwritingsectionsincreasedfrompretest
toposttest.Louthetal.(1992)foundastatisticallysignificantinteractionbetweenthewritingconditionsandtime,whichtheyreportedas,'Therewasasignificantmain
effectfortesttime,F(1,133)=23.33,p<.001,whichwasqualifiedbyasignificanttesttimeconditioninteraction,F(2,133)=4.50,p<.013"(p.219).To
interpretthesignificantinteractioneffect,simplemaineffectsanalysismustbeperformed,asdescribedinourdiscussioninchapter17.Louthetal.performedsimple
maineffectsanalysisandreportedtheirfindingsas,"Itestedthesimpleeffectoftesttimewithineachcondition.Asexpected,therewasasignificantincreasefrom
pretesttoposttestforboththeinteractivecondition,F(1,133)=12.65,p<.001,andthegroupcondition,F(1,133)=20.26,p<.001.However,therewasno
significantincreasefortheindependentgroup,F(1,133)=.14,p<.710"(pp.219).
OneoftheproblemswithperformingamixedANOVAtoanalyzethepretestposttestdesignistheinterpretationoftheinteractioneffect.Theproblemresultsfrom
theinteractionofthetreatmentandthepretest.Logically,itisdif

Page285

ficulttocomprehendhowthiscouldbepossible,becausethetreatmentisnotintroduceduntilafterthepretesthasbeengiven.Inapretestposttestdesign,the
treatmentcanonlyinteractwiththeposttest(Huck&McLean,1975).
Insummary,themixedANOVAapproachtoanalysisofthepretestposttestdesignisdependentonasignificantinteractioneffectbetweenthetreatmentconditions
andtime.Iftheinteractioneffectisstatisticallysignificant,testsofsimplemaineffectsmustbeperformed,similartothosedonebyLouthetal.(1992),tointerpretthe
outcome.
TheGainScoreApproach
ThegainscoreapproachtotheanalysisofthepretestposttestdesignisasimplermethodthanthemixedANOVAapproach.Thegainscoreapproachreducesthe
mixeddesigntoasinglefactordesign.Tounderstandthegainscoreapproach,let'sreexaminetheexampleofthestudybyLouthetal.(1992).Thefirststepinthe
gainscoreapproachistosubtractpretestscoresfromposttestscoresforeveryparticipantinthestudy.Therefore,intheLouthetal.study,wewouldstartby
subtractingthepretestscorefromtheposttestscoreforeachofthe44participantsintheindependentwritingcondition.Thiswouldgiveus44gainscoresforthe
writingcondition.Wewoulddothesameforeachofthe45participantsintheinteractivewritingconditionandthe47participantsinthegroupwritingcondition.The
resultisseeninTable18.5.Noticethatthedesignhasbeenchangedfromamixeddesign(twofactordesignwithrepeatedmeasuresonthesecondfactor)toasingle
factordesignwiththreelevels.Thethreelevels,orconditions,aretheindependentwritingcondition,theinteractivewritingcondition,andthegroupwritingcondition.
Thedependentvariableisnolongerthescoresona6pointscale,butitisthedifferenceinthesescoresfrompretesttoposttest.Becausethedesignisnowa
singlefactordesignwiththreelevels,theappropriateanalysisisthesinglefactorANOVA,asdiscussedinchapter14.IfasinglefactorANOVAisperformedon
thesedata,andtheresultisstatisticallysignificant,aposthoctestwouldbeperformedtointerpretthedifferencesamongconditions.
Whatinformationisobtainedfromperformingagainscoreanalysisforamixeddesign?Thegainscoreanalysisdeterminesdifferencesintheamountgained(orlost)
amongthethreeconditions.Thisisusuallytheinformationthatoneisinterestedinwhenimplementingastudyusingthepretestposttestdesign.Let'stakealookatthe
threeconditionsintheLouthetal.(1992)study.Theindependentwritingconditionisreallyacontrolcondition,becausethisiswhathasbeeninexistencepreviously.
Liketypicalcontrolgroupsinapretestposttestdesign,thiscontrolgroupmaynotstaythesameoveraperiodoftime,andcouldchangeasaresultofpracticeor
othervariables.However,what
TABLE18.5
MeanGainScoresFromLouthetal.,1992
Intervention

AverageGainScore

Independent

44

.07

Interactive

45

.64

Group

47

.57

Page286

isimportantisthattheothertwoconditionsofthestudymustbecomparedtothecontrolgrouptodeterminetheeffectivenessoftheirinterventions.Thegainscore
approachprovidesthegainoreffectivenessofeachconditionovertime.Bycomparingthegainscoresoftheinteractivewritingconditionandthegroupwriting
conditionwiththeindependentwritingcondition,aconclusioncanbereachedconcerningtheeffectivenessofeachcondition.Wementionedpreviouslythatapplying
themixedANOVAtothepretestposttestdesignpresentedproblemsbecauseoftheinterpretationoftheinteractioneffect.Interestingly,theFvalueforthe
interactioneffectinthemixedANOVAisidenticaltotheFvalueofasinglefactorANOVAperformedonthegainscores(ort 2ifattestisperformed).
Aweaknessofthegainscoreapproachisthatitdoesnotprovideinformationaboutdifferencespriortotheintervention(pretestscores).Forexample,intheLouthet
al.(1992)study,itappearedthattheindependentwritingconditionwasgreaterthantheothertwoconditionspriortotheintervention.Infact,Louthetal.confirmed
thisfindingbyusingsimpleeffectsanalysisintheirmixedANOVAapproach.Thegainscoreapproachonlydemonstrateschangesfrompretesttoposttest,butdoes
notprovideinformationaboutthepretestscores.
TheAnalysisofCovarianceApproach
Theanalysisofcovariance(ANCOVA)isastatisticalmethodusedtoreduceerrorvariance.Whenusedintheanalysisofthepretestposttestdesign,theANCOVA
alsochangesthedesignfromamixeddesigntoasinglefactordesign.Therefore,inthiscase,theANCOVAisasinglefactorANCOVA.However,theANCOVA
makesuseofdifferencesinthepretestscoresamongconditionstoadjusttheposttestscores.Oncetheseadjustmentshavebeenmadeontheposttestscores,thenthe
analysisisappliedonlytotheposttestscores.1
Again,let'slookattheLouthetal.(1992)studytoapplytheANCOVAapproach.Figure18.2showsthedatafromthisstudy.Noticethatthepretestscoresare
differentamongthethreeconditions,especiallyfortheindependentwritingcondition.TheANCOVAapproachtakesthesepretestscoredifferencesintoaccountby
adjustingtheposttestscoresfromthesedifferences.Howaretheseadjustmentsmade?TheANCOVAproceduretakesintoaccountthecorrelationbetweenthe
pretestandposttestscoresforeachgroup.OneoftheassumptionsoftheANCOVAisthattherelationshipbetweenthecovariate(pretestscores)andthedependent
variable(posttestscores)issignificantandlinear.Theposttestscoresareadjustedbyusingtheaveragelinearregression(chap.16)betweenpretestscoresand
posttestscoresforallconditionsofthestudy.Asecondassumptionoftheanalysisofcovarianceisthattheregressionslopesforeachcovariate(pretest)variate
(posttest)relationshipmustbeparallel.
WhattypeofadjustmentwouldbemadetotheLouthetal.(1992)data?AlthoughwedonothavetheoriginaldatafromtheLouthetal.study,wecangeneratedata
thatwouldgiveusthesamepretestandposttestmeans.Wewillmakeupdatafor10participantsfromeachofthethreeconditions.Thethreegroups
1

AnotheruseofthesinglefactorANCOVAinthepretestposttestdesignistousethepretestscoresasthecovariate,butusethegainscores,ratherthantheposttestscores,as
thedependentvariable.

Page287

of10participantsineachgrouphavethesamepretestandposttestmeans,asseeninTable18.4.ThesinglefactorANCOVAdoestwothings.First,itadjuststhe
posttestscoresfrompretestdifferences.Table18.6demonstratesthisadjustmentofposttestscores.Noticethedifferencesbetweentheposttestmeansandthe
adjustedposttestmeans.Asweexpected,theindependentwritingconditionwasadjustedmuchlower,whereastheothertwoconditionswereadjustedhigher.In
additiontoadjustingposttestscores,thesinglefactorANCOVAalsoperformsastatisticalanalysissimilartothesinglefactorANOVA.Infact,thesourcetablefor
thesinglefactorANCOVAisalmostthesameasthesinglefactorANOVA,exceptthatintheANCOVA,onedegreeoffreedomislostintheerrorsourceof
variation.Thus,powerisdecreasedslightlywhenanANCOVAisperformed.Table18.7showsthesourcetableforthesinglefactorANCOVAforourmadeup
data.TheresultsofourANCOVAshowasignificantdifferenceamongthethreegroups,F(3,26)=7.23,p<.01.Therefore,weperformedaTukeyHSDposthoc
testtointerprettheresults.Wefoundthattheadjustedposttestmeansweresignificantlyhigherforboththeinteractivewritingconditionandthegroupconditionwhen
comparedtotheindependentwritingcondition,buttheydidnotdifferfromeachother.
TheadvantageofusingasinglefactorANCOVAtoanalyzedatafromthepretestposttestdesignisthatitiseasiertounderstandthanthemixedANOVAandoften
moreprecisethanthegainscoremethod.ThesinglefactorANCOVAsimplifiestheanalysisbyconvertingthemixeddesignintoasinglefactordesign,asdoesthe
gainscoreanalysis.TherearetwodisadvantagesoftheANCOVAapproachtotheanalysisofthepretestposttestdesign.Thefirstdisadvantageisthattousethe
ANCOVA,theassumptionsoflinearityandparallelslopesshouldbesatisfied.Becausethepretestscoresandtheposttestscoresarefromthesamepeople,these
assumptionsareusuallymet.Thesecond,andperhapsmoreimportant,cautionwhenapplyingthesinglefactorANCOVAtotheanalysisofthepretestposttest
designistheissueoftypeof
TABLE18.6
PosttestScoreAdjustmentforMeanWritingScoresfromMadeUpData

TestTime

Intervention

Pretest

Posttest

AdjustedPosttest

Independent

10

3.00

3.07

2.653

Interactive

10

2.37

3.01

3.177

Group

10

2.27

2.84

3.09

TABLE18.7
SourceTableforSingleFactorANCOVA
SourceofVariation

SS

df

MS

Writingconditions

.693

.347

7.23*

Errorbetween

1.238

26

.048

*p<.01.

Page288

quantitativeapproach.Ifparticipantsarerandomlyassignedtoconditionspriortothestudy(randomizedexperimentaldesign)thentheANCOVAisthe
recommendedanalysis.Ontheotherhand,whenparticipantsarenotrandomlyassignedtoconditions,asinthequasiexperimentalandcomparativeapproaches,then
onemustbecautiouswhenusingtheANCOVAtoanalyzethepretestposttestdesign.Reichardt(1979)discussesextensivelythepositivesandnegativesofusing
ANCOVAtoadjustposttestscoresfrompretestdifferencesasaresultofgroupsthatwereintactpriortotheintroductionofthetreatment.Forthemostpart,
ANCOVAistheproperanalysis.However,theprecisionofANCOVAinthissituationisalmostalwayslessthaninthesituationwhereparticipantshavebeen
randomlyassignedtogroups.
Inconclusion,whenthemixeddesignhasonlytwolevelsofthewithinsubjectsindependentvariable,suchaspretestandposttest,thefollowingshouldapply.We
stronglyrecommendtheuseofthesinglefactorANCOVAtoanalyzethepretestposttestdesignwhenparticipantshavebeenrandomlyassignedtoconditionsprior
tothestudy.However,whenparticipantsarealreadyinintactgroups,thegainscoreapproachmaybejustassensitive.WedonotrecommendthemixedANOVA
approachtotheanalysisofthepretestposttestdesignbecause,forthemostpart,thegainscoreapproachprovidesthesameinformationandisclearer.However,
whenthewithinsubjectsindependentvariablehasmorethantwolevels,themixedANOVAapproachmaybethebestchoice,althoughtheanalysisofcovariancecan
alsobeusedinthissituation.AsintheLouthetal.(1992)example,theresultsofthethreemethodsoftenturnouttoproducethesamegeneralinterpretation.
ANoteontheGeneralUseofANCOVA
WehavediscussedtheANCOVAasastatisticalproceduretoanalyzethePretestPosttestdesign.However,theANCOVAprocedurehasmanyuseswithother
typesofdesigns,especiallysingleandtwofactorbetweengroupsdesigns.ANCOVAiscommonlyusedinsituationswhentheinvestigatorsuspectsthatathird
variablemayberesponsiblefortherelationshipbetweentheindependentanddependentvariables.Onewaytoeliminatethe''third"variableproblemistomatch
participantsonthatvariable.Inchapter14,wediscussedanexamplebyGlinerandSample(1993)thatmatchedparticipantsonanumberofvariables,including
intelligence.Insteadofmatching,theseauthorscouldhaveusedintelligenceasacovariateandperformedasinglefactorANCOVA.Thiswouldeliminatethepossible
confoundingthatintelligencelevelhasonenvironmentalcontrol.Usedinthisexample,theANCOVAprocedurewouldadjustscoresonthedependentvariable,
environmentalcontrol,onthebasisoftherelationshipbetweenthisvariableandthecovariate,intelligence.ForanindepthexaminationofANCOVAseeKeppel
(1991).
Summary
Inthischapter,wedefinedthemixeddesignashavingatleastonebetweengroupsindependentvariableandonewithinsubjectsindependentvariable.

Page289

Themixeddesignmayhavedifferentquantitativeapproaches,usuallyrandomizedexperimentalorquasiexperimental.Wedividedthemixeddesignintotwo
categories,thetruemixeddesignandthepretestposttestdesign.ThetruemixeddesignisanalyzedbyusingamixedANOVA.Thepretestposttestdesignmaybe
analyzedbythreedifferentstatisticalmethods:themixedANOVAanalysis,thegainscoreanalysis,andtheanalysisofcovariance.Wepointedoutthestrengthsand
weaknessesofeachofthesemethods,andwheretheyaremostappropriate.
StudyAids
Concepts
ANCOVA
Gainscoresanalysis
MixedANOVA
Pretestposttestdesign
Truemixeddesign
Distinctions
MixedANOVAversusgainscoresanalysisversusANCOVA
TruemixeddesignversusPretestPosttestdesign
ApplicationProblems
1.SomeresearchersarguethatANCOVAisthemostappropriatestatistictousewithPretestPosttestdata.Explain.
2.Aresearcherwasinterestedintheeffectsofdifferenttreatmentsforbackinjuryonreturntowork.Specifically,shewasinterestedinwhetherthosepeoplewho
weretreatedforbackinjurynoninvasively,forexamplerestandexercise,wouldperformdifferentlyfromthosewhoweretreatedthroughsurgery.Tomodeltheback
toworkexperience,avibrationmachinewasusedtosimulatedrivingoflargemachinery.Apersonwhosufferedabackinjurymightbeabletositforlongperiodsof
timeiftherewasnovibration,butnotunderconditionsofvibration.Therefore,eachparticipantinthestudyunderwentthreeconditions,drivingwithvibration,driving
withoutvibration,andjustsitting.Therewereeightpatientsintheexercisegroupandeightpatientsinthesurgerygroup.Thedependentvariablewaspainperception,
a10pointscalewhere1wasnopainand10wasintolerablepain.Themeanswereasfollows:

Condition
Sitting

Drivingwithoutvibration

Drivingwithvibration

Exercise

.33

1.33

Surgery

.33

3.67

Treatment

Page290

a.Describethedesign(e.g.,2 2factorial).
b.Graphtheabovedata.
Thesourcetableisasfollows:
SourceofVariation
BetweenSubjects

SS

df

MS

Treatment

28

Error(between)

Condition

80

Treatmentby

30

15

WithinSubjects

ConditionError(within)

c.Completethesourcetable.
3.Aphysicsprofessorwasinterestedindemonstratingthathisnewmethodofteachingwassuperiortothetraditionalteachingmethodinchangingtheattitudeof
studentstowardphysics.Priortothestartofthesemester,studentswererandomlyassignedtooneoftwobeginningphysicssections,thenewmethodandtheold
method.Duringthefirstdayofclass,allstudentstooktheAttitudeTowardPhysicstest.Attheendofthesemesterthestudentstookthistestagain.Foreachofthe
followinganalyses,explainwhatoutcomewouldbeusedtosupporttheprofessor'sargument.
a.MixedANOVA
b.Gainscorettest
c.Analysisofcovariance

Page291

Chapter19
InterpretationofComplexAssociationalAnalysesandMANOVA
AnalysesBasedontheComplexDifferenceQuestions
MultivariateAnalysisofVariance(MANOVA)
Purpose
ConditionsforUseofMANOVA
ComplexAnalysesBasedontheAssociationalApproach
MultipleRegressionAnalysis
Purpose
SimultaneousMultipleRegression
HierarchicalMultipleRegression
StepwiseRegression
ConditionsforUseofMultipleRegressionAnalysis
DiscriminantAnalysis
Purpose
ConditionsforUseofDiscriminantAnalysis
LogisticRegressionAnalysis
Purpose
ConditionsforUseofLogisticRegressionAnalysis
FactorAnalysis
Purpose
ConditionsforUseofFactorAnalysis
SummaryofMultipleVariableStatisticalMethods
StudyAids

Page292

Concepts
Distinctions
ApplicationProblems
Mostofthemethodspresentedbrieflyinthischapterarecomplexandneedtobeperformedwithacomputer,yetyouprobablywillencountersomeoftheminyour
readings.Infact,inresearchbyourgraduatestudents,allofthesemethodshavebeenused.Amajorlimitingrequirementofthesemethodsisthatalargenumberof
participantsisneededrelativetothenumberofvariablesselected.
Allofthemethodswehavediscussedinpreviouschaptersareconsideredtobeunivariatestatisticsbecausethereisonlyonedependentvariableintheanalysis.
Generally,statisticsarecalledmultivariateiftheyhavemorethanonedependentvariable.However,mostbooksonmultivariatestatisticsincludeothercomplexor
multiplevariablestatistics,suchasmultipleregression,becausethereisnotaconsensusabouthowthetermmultivariateshouldbeused(seeGrimm&Yarnold,
1994).Inthischapter,wediscussfivemultiplevariable(morethanoneindependentvariable,ordependentvariable,orboth)methodsthatareusuallyincludedin
multivariatestatisticsbooks.Westartwiththemultivariateanalysisofvariance(MANOVA),becausethedesignsthatareusedforthisprocedurearesimilartomany
oftheunivariatedesignsthatwealreadyhavediscussedinchapters14,15,17,and18.
AnalysesBasedontheComplexDifferenceQuestions
MultivariateAnalysisofVariance(MANOVA)
Purpose
MANOVAcomparestwoormoregroupsbymeansofalinearcompositedependentvariable.ThetermMANOVAappearsfrequentlywhenreadingjournalarticles.
MANOVAcanbeusedwithanyoftheANOVAsthatwehavediscussedinchapters14,15,17,and18whentherearemultiplede

Page293

pendentvariablesormeasures.Forexample,Stone,Lemanek,Fishel,Fernandez,andAltemeier(1990)wereinterestedindifferencesamongchildrenwithcertain
disabilitiesonplaybehaviorandmotorimitation.Theyhadfivedifferentgroupsofchildrenintheirstudy:autistic,hearingimpaired,languageimpaired,mentally
retarded,andnonhandicapped.Theyobservedeachchildthroughaonewaymirrorinafreeplaysituationfor8minutesandtheninaoneononeimitation
assessmentfor5minutes.Therewerefivedependentvariablesinthestudy:numberoftoysused,percentoftimeplayingwithtoys,percentoftimeplaying
appropriatelywithtoys,numberoffunctionalplayacts,andimitationscore.Thedesignforthisstudywasasinglefactordesignwithfivelevels.Theindependent
variablewasthetypeofdisabilitygroup.Thefivelevelswerethefivedifferentgroupsstudied.Becausetheindependentvariablewasanattributeindependentvariable,
thequantitativeapproachwasconsideredtobecomparative.
NoticethattherewerefivedifferentdependentvariablesintheStoneetal.(1990)study.Asweincreasethenumberofdependentvariables,theprobabilitythatthese
dependentvariablesarerelatedincreases,especiallybecauseweareoftentryingtousemultiplemeasuresofasingleconstruct.Thedangeristhatifweweretoapply
aseparateFtestforeachofthedependentvariables,andinfacttheywererelated,wemightgetsignificantoutcomesforeachdependentvariable.However,itis
probablethattheredundantinformationsharedamongthedependentvariablescontributestostatisticalsignificance.Ifweusemultipledependentvariablesthatare
relatedindesignstypicallyanalyzedbytorFstatistics,theMANOVAistheappropriatechoice.Therefore,Stoneetal.decidedtoperformasinglefactor
MANOVAonthedata.
TheMANOVAcombinestheseveraldependentvariablesinsuchawaythatanewvariable(linearcomposite)isformed.ThemultivariateFteststoseeifthegroups
(basedontheindependentvariable)differforthisnewcompositevariable.Thus,MANOVAprovidesadditionalinformationnotavailablefromcomputingseveral
separateANOVAs.ThereismorethanonesignificanceteststatisticfortheMANOVA.ThemostcommonstatisticsareWilk'slambda,andHotellingT2.Othertest
statisticsfortheMANOVAseen,forexample,onanSPSSprintout,arePillai'straceandRoy'slargestroot.AllofthesestatisticsareconvertedtoanFstatisticto
determinestatisticalsignificance(seeJ.Stevens,1986formoreinformationaboutchoiceofaparticularmultivariatestatistic).IftheMANOVAisstatistically
significant,thenappropriatefollowupproceduresareperformed,similartoposthoctestsafterasignificantF.Thesefollowupproceduresareoftenselectedforthe
purposeofthestudy.Forexample,ifthepurposeofthestudyistodeterminetherelativecontributionsofthedifferentdependentvariablestotheindependentvariable,
thenadiscriminantanalysiscouldbeused.Inthiscase,eachofthedependentvariableswouldbeusedasapredictorvariableandtheindependentvariablewouldbe
usedasthecriterionvariable.Thediscriminantanalysiswouldyieldthebestlinearcombinationofvariables(dependentvariables)thatwouldpredicttheindependent
variable.Stoneetal.(1990)performedadiscriminantanalysisasafollowupprocedureintheirstudy.ForaconceptualviewofotherMANOVAfollowup
procedures,seeWeinfurt(1997).

Page294

OnemightaskifaMANOVAshouldbeperformedifmorethanonedependentvariableisincludedinthestudy,butthedependentvariablesarenotrelated.
Previously,itwasthoughtthatusingaMANOVAwouldprotectagainstmakingatypeIerror,whichcouldoccurifoneweretoperformmultipleANOVAsorttests.
Inotherwords,itwasthoughtthatMANOVAwouldkeepthetypeIerroratthesamelevelandnotinflateitfromthemultiplecomparisons(similartoperforming
multiplettests).Therefore,itwascommonpracticetofirstperformaMANOVA,andthen,ifastatisticallysignificantresultwasfound,toperformunivariate
ANOVAsorttests.However,itappearsthatMANOVAdoesnotprotectagainstmakingatypeIerrorformultipledependentvariables.Currentthinkingisthatif
morethanonedependentvariableisincludedinthestudy,andthesedependentvariablesarenotrelated,donotdoMANOVA.Instead,doseveralANOVAsort
testsbutdividethesignificancelevelbythenumberofdependentvariablestoprotectagainstatypeIerror(Dunn,1961).
ConditionsforUseofMANOVA
1.Oneorafewcategoricalindependentvariables.
2.Twoormoreapproximatelynormallydistributeddependentvariables.
3.Thedependentvariablesmustbecorrelated.
ComplexAnalysesBasedontheAssociationalApproach
Thefirstthreedataanalysismethodsinthissectionareusedtopredictadependentorcriterionvariablefromseveralindependentorpredictorvariables.Thefirst
statisticwediscuss,multipleregression,includesseveraldifferentformsortypesofanalyses.Themostcommonoftheseissimultaneousmultiple(linear)regression.
Othertypesofmultipleregressionincludehierarchicalmultipleregressionandstepwisemultipleregression.Inmultipleregressionanalysis,thecriterionvariableshould
beapproximatelynormallydistributed,usuallyhavingmanyorderedvalues.Twoothermethodsusedtopredictacriterionvariablefromseveralpredictorvariablesare
discriminantanalysisandlogisticregression.However,intheselattertwomethods,thecriterionvariableiscategorical.Thefourthcomplexanalysisthatisbasedonthe
associationalapproach,factoranalysis,alsoappearsfrequentlyintheresearchliterature.Themostcommonpurposeoffactoranalysisistoreducelargenumbersof
dependentvariablestoasmallernumbercalledfactors.
MultipleRegressionAnalysis
Purpose
Multipleregressionisusedtopredictanormallydistributedcriterion(dependent)variablefromacombinationofseveralnormallydistributed,ordichotomous
predictors(independentvariables),orboth.
Previously(chap.16),wediscussed(a)therelationbetweentwovariables,(b)howthisrelationcouldbecalculatedwithaPearsonproductmomentcorrelation
coefficient,and(c)howthisrelationcouldbeusedtoformanequationtopredictonevariablebyknowingtheother(regression).Ifweweretodeter

Page295

minetherelationbetweenGREscoresandgraduatestudentperformance,asmeasuredbygradesingraduateschool,youmightfindaPearsonproductmoment
correlation,r,of0.30.Anadditionalstepmightbetoformanequationsothatyoucouldpredictfuturesuccessingraduateschool(asmeasuredbyGPA)fromcurrent
GREscores.Therefore,ifyouknewastudent'saverageGREscore,youcouldpredictperformanceingraduateschool.Howwell?Ther2givesoneindication.Ther2
inthisexamplewouldbe0.09.Howdoweinterpretr2inthissituation?
Ther2istheamountofsharedvariancebetweenthetwovariables.WecouldsaythatthereisanunderlyingvariableofcognitiveabilitythatiscommontobothGRE
scoresandgradesingraduateschool,suchasbrainpowerorIQ,thatexplainsabout9%ofthevariance.AnotherwayoflookingattheproblemistofocusontheY
variableorgradesingraduateschool.WecallYthedependentvariable,orforthispurpose,thecriterionvariable.GREscoresbeforeenteringgraduateschoolcould
nowbereferredtoastheindependentvariableorpredictorvariable.Fromourdata,wecanconcludethatGREscoresaccountforabout9%ofthevarianceof
gradesingraduateschool.Lookingatitfromanotherdirection,wecouldsaythat91%ofthevarianceinpredictinggradesingraduateschoolisunexplained.Howcan
weimproveourprediction?Thisleadsustomultipleregression.
Tocomputemultipleregression,westartbyconstructingacorrelationmatrixamongallofthevariables,includingthedependentvariable.(Thismatrixisusedfor
almostallmultipleregressionandmultivariateprocedures.)Acorrelationmatrixisjustatableofallpossiblecorrelationsamongvariables.Then,operationssimilarto
linearregressionanalysis(chap.16)areperformed.WearriveatanequationtopredictY'.Thisequationisthebestlinearcombinationofindependentorpredictor
variablestopredictthecriterionvariable.Eachvariableintheequationhasaweightassignedtoittomaximizeprediction.Theseweightsarecalledpartialregression
coefficients.Partialregressioncoefficients,takenbythemselves,demonstratehowmuchofanindividualcontributioneachvariableismakingtothepredictionofthe
dependentorcriterionvariable.Statisticalsignificanceisdeterminedforeachofthesepartialregressioncoefficients.Thesignificancestatementdetermineswhethera
particularpredictorvariableissignificantlydifferentfromzero.Iftheseregressioncoefficientsarestandardizedbyconvertingthemtozscores(betaweights),thenthe
coefficientscanbecomparedtodeterminetherelativestrengthamongeachofthevariables.Inaddition,avarianceexplainedmeasure,R2,isusedtodeterminethe
totalvarianceaccountedforinthecriterionvariablebythecombinationofpredictorvariables.Itshouldbenotedthatjustbecauseapredictorvariableisnot
statisticallysignificantdoesnotmeanthatvariableshouldbedroppedfromtheequation.ThevariablecouldstillbemakingacontributiontotheoverallR2.
Furthermore,itispossible,butnotcommon,tohaveasignificantR2,evenifnoneoftheindividualpredictorvariablesarestatisticallysignificant.
SimultaneousMultipleRegression
Themostcommonmethodofmultipleregressioniscalledsimultaneousmultipleregression.Inthismethod,allofthe

Page296

predictorvariablesareenteredsimultaneously.Thebestlinearcombinationofvariablesisdeterminedbyusingaleastsquaresfitsimilartolinearregressiondiscussed
inchapter16.Anexampleofasimultaneousmultipleregressionisasfollows.MorganandGriego(1998)wantedtodeterminehowwellmathematicsachievement
couldbepredictedfromacombinationofsevenvariables:motivation,competence,pleasure,gradesinhighschool,father'sandmother'seducationrevised,and
gender.Table19.1showsthemultipleRandR2valuesforthesimultaneousmultipleregression.ThemultipleRtellsusthatastrongrelationshipexistsbetweenthe
predictorvariablesandthecriterionvariable,mathematicsachievement.TheadjustedR2indicatesthat38%ofthevarianceinmathematicsachievementwaspredicted
fromthecombinationofpredictorvariables.Table19.2showsthecoefficientsforthesamesimultaneousmultipleregressionproblem.Table19.2shows,foreach
variableinthemultipleregression,theraw(unstandardized)coefficients,thestandardizedcoefficients(betaweights),thetvalues,andthesignificancelevelforeachof
thevariables.Notethatbothgradesinhighschoolandgenderwereconsideredtobestatisticallysignificant,assuminganalphalevelof.05.Inaddition,wecouldform
theequationforpredictionofmathematicsachievementfromtheunstandardizedcoefficientsasfollows:

TABLE19.1
MultipleCorrelationandAdjustedR2
R

AdjustedR2

.666

.379

TABLE19.2
MultipleRegressionCoefficients

StandardizedCoefficients
(Beta)

UnstandardizedCoefficients
Variables

Constant

S.E.

4.293

5.15

Motivationscale

1.684

1.23

.158

1.373

Competencescale

.117

1.41

.011

.083

Pleasurescale

.833

.874

1.78

.083

.742

Gradesinhighschool

1.762

.48

.416

3.680**

Father'seducationrevised

1.477

1.03

.186

1.439

Mother'seducationrevised

1.027

1.27

.104

.810

3.359

1.39

.253

2.409*

Gender
**p<.01.*p<.05.

Page297

Thisequationthencouldbeusedinthefuturetopredictmathematicsachievementofindividualstudents,assumingasimilarsampleofparticipants.TheXscorrespond
toanindividual'sscoreoneachofthepredictorvariablesi.e.,X1=motivationscalescore,X2=competence,andsoforth.
HierarchicalMultipleRegression
Thismultipleregressionprocedurehasrecentlybecomepopular.Conceptually,ithassimilaritiestothestepwiselinearregressionapproach,describedinthenext
section.Variablesareenteredinsteps,andthechangeinR2isexaminedateachstep.However,inhierarchicalregression,theordertoentereachvariableintothe
equationisdecidedaheadoftimebytheinvestigator.Usuallythesedecisionsarebasedonacarefulconceptualizationoftheproblemandresultinthetestingof
particularhypotheses.Hierarchicalmultipleregressionalsoallowstheresearchertocontrolfororeliminatetheeffectsofaparticularvariablebyenteringthatvariable
intotheequationfirst.Also,morethanonevariablecanbeenteredintotheequationatonce.TheprobabilityofcommittingatypeIerrorisusuallyreducedin
hierarchicalregressionbecausefewerstepsareusedtoentervariablesintotheequation.Table19.3(Morgan&Griego,1998)showsthehierarchicalmultiple
regressionanalysisfromthesameexampleasthatusedforthesimultaneousmultipleregression.Inthisexample,thevariablegenderwasenteredfirsttocontrolforits
effects.Alloftheotherpredictorvariableswereenterednextinablock.Genderaccountedforapproximately6%ofthevariance.Theadditionoftheotherpredictor
variablesaccountedforabout32%ofthevariance.
StepwiseRegression
Thestepwiseapproachissimilartohierarchicalmultipleregression,butthecomputerdecidestheorderandhowmanyofthepotentialpredictorsareused.The
stepwiseregressionproceduredescribeshowmuchmoreeachindependentorpredictorvariablehascontributedtothepredictionfromtheprecedingpredictor
variable(s).Theconceptualviewofstepwiselinearregressionisasfollows.
Eachphaseoftheprocedureiscalledastep.AtStep0,acorrelationmatrixisformedamongallvariablesincludingthecriterionordependentvariable.AtStep1,the
predictorvariablethatcorrelatesthehighestwiththecriterionvariableisenteredintotheequation.(Note:Atthisstep,andanyfollowingstep,thesizeofthe
correlationmustreachadesignatedsignificancelevelforthepredictorvariabletobeentered.)AtStep2,thepredictorvariablethataddsthemostnewvariancetothe
firstpredictorvariableisenteredintotheequation.AtStep3,thepredictorvariablethataddsthemosttothetwopredictorvariablesalreadyselectedisenteredinto
theequation.Thiscontinuesuntileitherallvariablesareenteredintotheequationorsignificanceisnotreachedforanyoftheremainingvariables.
Table19.3
MultipleCorrelationandR2forHierarchicalMultipleRegression
R

R2

AdjustedR2

StandardErroroftheEstimate

Gender

.272

.074

.060

6.46

Allothervariables

.666

.443

.379

5.25

Variables

Page298

Whilestepwiselinearregressionmakesalotofsenseconceptually,thereareseveralproblemsthathavebeenassociatedwiththisapproach,andoneshouldprobably
usethisapproachonlyasanexploratoryprocedure(seeThompson,1995,foracriticalreviewofthisprocedure).Oneoftheproblemswiththisapproachisthat
becauseofthepotentiallylargenumberofpredictorvariablesthatcouldbeenteredintotheequation,theprobabilityofatypeIerrorisconsiderablylargerthanthe
usualalphaof.05.Thesecond,andperhapsmoreimportant,objectionwiththestepwiseapproachisthatthecomputer,ratherthantheresearcher,makesthe
decisionaboutwhichvariablesshouldbeincludedintheequation.Thisisespeciallythecasewhenoneentersalargenumberofpredictorvariablesintothestepwise
analysiswithlittlethoughtgiventoparticularhypothesesortheory.Manystatisticiansassociatethisapproachwiththeterms''datamining"or"snooping."Third,the
approachtakesadvantageofpossiblesmalldifferenceswhenenteringvariablesand,thus,islikelynottoreplicatewithanothersample.
ConditionsforUseofMultipleRegressionAnalysis
1.Thedependentvariableshouldbenormallydistributeddata.
2.Theindependentvariables,forthemostpart,alsomustbenormallydistributeddata.Iftheindependentvariablesrepresenteitherordinal(nonnormal)ornominal
(categorical)data,thendichotomous"dummy"variablesmustbecreatedbyrecoding.
DiscriminantAnalysis
Purpose
Discriminantanalysisisusedtopredictacategoricalcriterionvariablefromacombinationofseveralnormallydistributedindependentvariables.Specifically,the
methodcanbeusedtopredictgroupparticipation,forexample,passorfail.
Discriminantfunctionanalysisisusedinsituationssimilartomultipleregressionanalysis,exceptthatthecriterionvariableiscategoricalratherthancontinuous.A
discriminantfunctionpredictionequationisderivedinasimilarwayastheequationonmultipleregression.However,therearetwootherimportantdifferences
betweendiscriminantfunctionanalysisandmultipleregressionanalysis.Thefirstdifferenceisthatthecoefficients(weights)inthediscriminantfunctionanalysisare
selectedfromhowwelltheyclassifyparticipantsintogroups.Theseconddifferencebetweendiscriminantfunctionanalysisandmultipleregressionanalysisisthatifthe
criterionvariableindiscriminantfunctionanalysishasmorethantwocategories,thenmorethanonediscriminantfunctionequationislikely.
Auseofdiscriminantfunctionanalysiswouldbeifourcriterionvariablerepresentinggraduatestudentsuccesswasdividedintostudentswhograduatedornot,rather
thanbeingbasedontheirgradepointaverage.Likewise,inanemployabilityexample,ifsuccessversusfailure,ratherthandegreeofsuccess,wasthedependent
variable,discriminantanalysiswouldbeappropriate.Youmayreadarticlesthatusemultipleregressionwithadichotomousdependentvariable.Suchananalysisisnot
necessarilywrong,butwewouldarguethatitisnotthemostappropriateforadatasetwithadichotomouscriterion.

Page299

AnexampleoftheuseofdiscriminantfunctionanalysisistakenfromMorganandGriego(1998).Inthisexample,predictorvariablesofmotivation,competence,
pleasure,father'sandmother'seducationrevised,andgenderwereusedtopredictifstudentswouldtakealgebra2.Table19.4showsthediscriminantfunction
coefficients.Thesecoefficients,orweights,producethebestlinearcombinationofthevariablesthatmaximallydiscriminatesordifferentiatesthosewhotookalgebra2
fromthosewhodidnot.Thediscriminantfunctionequationfortheabovepredictorvariablesiswrittenas:

Itcouldbeusedtopredictinanewsamplewhetherastudentwouldtakealgebra2.
Table19.5demonstratestheclassificationresultsforthecurrentsamplefromthediscriminantfunctionequation.Theclassificationtabledescribeshowwellthe
discriminantfunctionequationpredictsgroupmembership(takenalgebra2ornot).Inthisexample,theoverallcorrectclassificationwas72.5%andwasstatistically
significant.
Earlierinthischapter,wementionedthatthediscriminantanalysisisacommonfollowupprocedureforastatisticallysignificantMANOVA.ThestudybyStoneetal.
(1990)providesanexampleofthisfollowupprocedure.Recallthattheyhadoneindependentvariable(typeofhandicapcondition)withfivelevels.Therewerefive
dependentvariables.FollowingasignificantMANOVA,theyperformedadiscriminantanalysis,byusingthedependentvariablesasthepredictorvariables.Because
theywereprimarilyinterestedindiscriminatingbetweenautisticchildrenandotherchildren,theircriterionvariablewasdichotomous,eitherautisticchildrenortheother
handicapping
TABLE19.4
StandardizedCanonicalDiscriminantFunctionCoefficients
PredictorVariables

Function

Father'seducationrevised

.577

Mother'seducationrevised

.330

Gender

.197

Motivationscale

.151

Pleasurescale

.320

Competencescale

.530

TABLE19.5
DiscriminantAnalysisClassificationTable

NotTaken

Taken

Total

Frequency

Nottaken

28

36

Taken

11

22

33

Nottaken

77.8

22.2

100

Taken

33.3

66.7

100

Percent

Algebra2in
HighSchool

PredictedGroupMembership

Sample
Description

Page300

conditions,excludingthenonhandicappedgroup.Theyfoundthatthreeofthepredictorvariables,appropriatetoyplay,functionalplay,andimitation,combinedto
significantlydiscriminatebetweenthetwogroups(autisticandnonautistic).Furthermore,theyreportedthat82%oftheautisticchildrenand100%ofthenonautistic
childrenwereclassifiedcorrectly.
ConditionsforUseofDiscriminantAnalysis
1.Thedependentvariablemustbecategorical.Itmayhavemorethantwocategoriesbutthatmakestheinterpretationmuchmorecomplex.
2.Theindependentvariablesmustbenormallydistributed.Ifnominalindependentvariablesmustbeused,alogisticregressionwouldbetheappropriateanalysis
(Grimm&Yarnold,1994).Thus,intheMorganandGriego(1998)example,itwouldhavebeenbettertoomitthegendervariableortouselogisticregression.
LogisticRegressionAnalysis
Purpose
Logisticregressionisusedtopredictadichotomouscriterionvariablefromacombinationofseveralindependentvariables.
Logisticregressionalsoinvolvesasituationwherethecriterionvariableiscategoricalhowever,forthisanalysis,thecriterionvariablemayhaveonlytwocategories.In
addition,logisticregressionanalysisisconsideredtobeanonlinearanalysis.Therefore,ifyoususpectthattherelationbetweenyourpredictorsandthecriterion
variableisnotlinear,logisticregressionisagoodchoice,assumingyouhaveatwocategorycriterionvariable.Inaddition,logisticregressionanalysisisusedwhen
someorallofthepredictorvariablesarecategorical.
Forexample,onemightbeinterestedinpredictingwhichstudentswoulddropoutversusfinishorgraduatefromatrainingprogramonthebasisofacombinationof
genderandethnicgroup(nominalorcategoricalpredictorvariables)andatest(acontinuousorintervalpredictorvariable).
ConditionsforUseofLogisticRegressionAnalysis
1.Thecriterionvariablemustbecategoricalandislimitedtoonlytwocategoriesthatis,itmustbedichotomous.
2.Thepredictorvariablescanbedichotomousornormallydistributed.
FactorAnalysis
Purpose
Factoranalysisismostoftenusedtocombineanumberofitemsorvariablestoformasmallernumberofcompositevariablesorfactors.Itisalsousedtolookfor
latentorunobservablevariablesorconstructsfromobservedvariables.
Factoranalysisisperhapstheoldestandbestknownofthemultivariatemethods.Factoranalysishasbeenusedbyourgraduatestudentsprimarilytoreducethe
numberofvariablesinastudybyexaminingwhichvariablesarecloselyrelated.Oncetheoriginalvariablesarereducedtoasmallernumberofcompositevariables
(calledfactorsorcomponents),otheranalysesmaybe

Page301

performed.Factoranalysis,likeothermultivariatemethods,beginswithacorrelationmatrix.Thepurposeofthismatrixistoidentifywhichvariables(oftentestor
ratingscaleitems)correlatewitheachotherbutdonotcorrelatewithothervariablesoritems.Theideaistoidentifyalimitednumberoffactorsthatarerelatively
independentofeachother.Ifyouaredevelopingatest,youmightstartwith25questions.However,thereisastrongpossibilitythatall25questionsarenot
independentofeachother.Afactoranalysiscouldbeusedintheexploratorysensetodetermineunderlyingclustersorgroupingsofquestionsthatcouldbeexplained
byafewfactors.EachclusterorfactorshouldbetestedwithCronbach'salphatobesurethatinternalconsistencyreliabilityisadequate(seechap.20).
Inaddition,afactoranalysiscouldbeusedtoconfirmconceptualizationsthathavealreadybeenpostulatedbutnotfullytested.Inthisuse,factoranalysisisoneofthe
methodsofdemonstratingconstructvalidityandissometimescalledfactorialvalidity(seechap.20).
AstudybyRobnettandGliner(1995)demonstratestheuseoffactoranalysistoreducethenumberofdependentvariablesinastudytoasmallernumberoffactors.
RobnettandGlinerwereinterestedinthequalityoflifeofpersonswithdisabilities.Specifically,theywereinterestedinthequalityoflifeofpeoplewithdisability
conditionsthatwereprogressiveanddisabilityconditionsthatwerestable.Theyselectedmultiplesclerosis(MS),asanexampleoftheformer,andforthelatter,spinal
cordinjury(SCI).Asacomparisongroup,theyaddedoccupationaltherapists(OT).Todeterminequalityoflife,theydevelopedaqualityoflifeinventory.There
were80questionsinthisinventory.Aftertheinventoryhadbeengiventoparticipantsfromallthreegroups(MS,SCI,andOT),theresearchersperformedafactor
analysistoreducethenumberofquestionstoafewfactors.Therewere405participantswhoreturnedthequalityoflifeinstrument.Table19.6demonstratesa
portionoftheresultsofthefactoranalysis(theactualfactoranalysishadsevenfactors).Table19.6showshoweachofthequestionsrelates(loads)oneachofthe
factors.Inthistable,allfactorloadingsthatwerelessthan.30wereomittedanddisplayedasdashestomakeiteasiertovisualizetheresults.Tointerpretthistable,
startwithFactor1.NoticethatQuestions22,11,16,and18allhaverelativelyhighloadingsonFactor1andlowloadingsonFactors2and3.However,Question
20hadloadingsonbothFactors1and2.Thismeansthatquestion20wasrelatedtobothfactors.BecauseRobnettandGlinerwereinterestedinusingthefactorsfor
furthercomparison,thisitemwasdroppedfromthestudy.Questions61and34alsohadmultiplefactorloadingsandwereeliminatedfromthestudy.Factors2and3
wereexaminedinthesamemannertheresearcherslookedforquestionsthatloadedhighoneachofthosefactors,butlowontheothertwofactors.
Factoranalysisdoesnotnamethefactors,anditisuptotheinvestigatorstocomeupwithnamesforthefactorsthatwouldbeagreedonbyothersinthefield.Inthe
RobnettandGliner(1995)study,thefirstthreefactorswerecalledclosenesstoothers,selfachievement,andhealth/activitylevel.Whenperformingfactor
analysis,itisimportanttodescribehowmuchofthetotalvarianceisaccountedforbyeachfactor.Table19.7providesthenamesofthesevenfactorsfromthe
RobnettandGliner(1995)studyandtheirpercentage

Page302
TABLE19.6
FactorLoadingPatternsforFirstThreeFactorsinRobnettandGliner(1995)Study
Question

FactorI

Factor2

Factor3

22

.759

11

.699

16

.695

18

.633

20

.593

.376

19

.580

24

.524

59

.666

65

.588

61

.571

.316

60

.562

34

.522

.346

30

.761

27

.704

25

.630

26

.619

14

.552

.543

.515

TABLE19.7
FactorNamesandPercentageVariancefromRobnettandGliner(1995)
Factor

FactorName

PercentageVariance

Closenesstoothers

7.7

Selfachievement

6.8

Health/Activitylevel

6.5

Positivefeelings

6.3

Outlook

5.3

Spirituallife

3.1

Routinebehavior

2.3

variance.Therearenumerousproceduresfordetermininghowmanyfactorsshouldberetainedfromthefactoranalysis.Thesemethodsareoftenreferredtoas
stoppingrules(seeGrimm&Yarnold,1997).However,itisimportanttorememberthatthereshouldbetheoreticalreasoninginvolvedinthedevelopmentofthe
instrument,andsubsequentfactorretentionshouldbe,atleastinpart,basedonthisreasoning.
RobnettandGliner(1995)usedthesevenfactorsasdependentvariablesforfurthercomparisons,whichinvolvedsevensinglefactor(oneway)ANOVAs,to
determineifthereweredifferencesinqualityoflifeamongthethreegroupsintheirstudyoneachofthesecompositevariables.Notethatthetermfactor

Page303

hastwodifferentmeaningsintheprecedingsentence.Bothtermsrefertovariables,butsinglefactorANOVAreferstothenumberofindependentvariablesinthe
study(inthiscaseone,typeofperson:MS,SCI,orOT).Thesevenfactorsfromthefactoranalysisarethedependentvariablesinthiscase.
ConditionsforUseofFactorAnalysis
1.Anumberofnormallydistributedvariables,usuallydependentvariables.
2.Itisarbitraryanddependsonthestudyastowhetherthesevariablesareindependent(predictors)ordependent(criterion)variables.
SummaryofMultipleVariableStatisticalMethods
Table19.8isprovidedasasummaryofthefivestatisticalmethodswehavepresentedinthischapter.Thetableprovidesthepurposeandinformationaboutthe
numberandscaleortypeofindependentanddependentvariablesforeachofthemethods.Wealsohaveadded,atthebottomofthetable,factorialANOVA(see
chap.17and18)becauseitfitswithinthegroupofmethodsthatwehavecalledcomplexormultiplevariable(i.e.,thosewithmorethanoneindependentvariable,or
dependentvariable,orboth)eventhoughitisnotusuallyconsideredtobea"multivariatestatistic."
StudyAids
Concepts
Discriminantanalysis
Factoranalysis
Logisticregression
MANOVA
Multipleregression
Distinctions
FactoranalysisversusfactorialANOVA
MANOVAversusunivariateANOVAs
Multipleregressionversusdiscriminantanalysisversuslogisticregression
Simultaneousregressionversushierarchicalregressionversusstepwiseregression
ApplicationProblems
1.TheDirectorofSpecialEducationinasuburbanschooldistrictwantedtocomparetwoschoolsintermsofhowEnglishasaSecondLanguage(ESL)students
wereperformingintheirrespectiveschools.Theindependentvariablewasschool,withseverallevels.DependentvariablesincludedESLstu

Page304
TABLE19.8
SummaryofthePurposesandConditionsforUseofSixMultipleVariableStatistics
ComplexAssociationalorRelationalQuestions
Purpose:Predictacriterionordependentvariablefromseveralindependentvariables.Lookforacombination
ofindependentvariablestooptimallypredictthedependentvariable.
Statistic
Multipleregression

Discriminant

Logisticregression

IndependentVariableNumber
andType

DependentVariableNumberand
Type

Several

One

Normaland/ordichotomous

Normallydistributed

Several

One

Normal

Categorical

Several
Normaland/ordichotomous

One
Dichotomous

Purpose:Reducethenumberofvariablestoasmallernumberofcompositevariablesorfactors.Alsolookfor
latentvariables.
Statistic

IndependentVariable

Factoranalysis

None(however,sometimesthe
severalvariablesareindependent
variables)

DependentVariable
Severalnormallydistributed

ComplexDifferenceorGroupComparisonQuestions
Purpose:Lookforsignificantdifferencesamongthelevelsofeachindependentvariableandforasignificant
interaction.
Statistic

IndependentVariable

DependentVariable

FactorialANOVA

Twoorafewcategorical

One(atatime)normally
distributed

Purpose:Comparegroupmeans(i.e.,lookforsignificantdifferences)onalinearcombinationofthe
dependentvariablesandalsoeachdependentvariableseparately.
Statistic

IndependentVariable

DependentVariable

MANOVA

Oneorafewcategorical

Twoormorenormallydistributed

Note.Categoricalvariablescanbeunordered(nominal)orordered,butusuallyhaveasmallnumberoflevels
orvalues.AdaptedfromTabachnick&Fidell(1989).

dents'standardizednationaltestscoresineachofthefoursubjectareas.Whattypeofstatisticshouldbeusedinthisstudyandwhy?
2.Aconsortiumofresearcherswantstolookatsomeoftheimpactsofwelfarereformonindividualsandfamilieswhohavenotreceivedpublicassistancefortwo
years.Theyhavealargemultistatesample.Foreachcaseaneconomicwellbeingscorewascomputedastheoutcomevariable.Theresearcherswereinterested
intheirabilitytopredicteconomicwellbeingfrompriorlevelofeducation,yearsofworkexperience,transportationavailability,trainingreceivedwhileonwelfare,
andtherelativehealthofthelocaleconomy(alldichotomousorintervalscalemeasures).
a.Whattypeofanalysiswouldbeappropriate?Explain.

Page305

b.Theresearcherswantedtoknowwhatcombinationoftheabovefactorspredictedwhetheraformerwelfarerecipientwillhavebeenemployedornotsince
leavingwelfare.Whatanalyticalapproachshouldtheyuse?Forquestions3to6,matchtheanalysiswiththeparticularandexplainwhy.
a.Multivariateanalysisofvariance
b.Multipleregression
c.Factoranalysis
d.Discriminantanalysis
3.YouhaveaPretestPosttestcontrolgroupdesign.Yourdependentvariableisa50itemquestionnairewhichwasgivento250participantsintheinterventiongroup
and250participantsinthenoninterventiongroup.Yournextstepistoreducethenumberofquestionstoasmallernumbercomposite/summatedvariables.
4.Youareinterestedinpredictingifpeoplearesuccessfulorunsuccessfulwhentheyreturntowork.Yourpredictorvariablesarestrength,rangeofmotion,I.Q.,and
theGlinerGreedScale.
5.Youhaveoneindependentvariablewiththreelevels.Youhavefivedependentvariablesthatcouldberelated.
6.YouaretryingtopredictgradepointaverageingraduateschoolinanEnglishdepartment.YourpredictorvariablesareundergraduateGPA,GREverbalscores,
scoreonaninterestinventory,andage.

Page307

PARTV
MEASUREMENT,INSTRUMENTS,ANDPROCEDURES

Page309

Chapter20
MeasurementReliabilityandValidity
MeasurementReliability
AnExample
TestScores
CorrelationCoefficient
MethodstoAssessReliability
TestRetestReliability
ParallelFormsReliability
InternalConsistencyReliability
SplitHalfMethods
KuderRichardson20
Cronbach'sAlpha
StandardErrorofMeasurement
Interrater(Interobserver)Reliability
PercentageAgreementMethods
IntraclassCorrelationCoefficients(ICC)
Kappa
SummaryofReliabilityMethods
MeasurementValidity
FaceValidity
ContentValidity
CriterionRelatedValidity
PredictiveEvidence
ConcurrentEvidence
ConstructValidity
ConvergentEvidence
DiscriminantEvidence

Page310

FactorialEvidence
Examples
Summary
StudyAids
Concepts
Distinctions
ApplicationProblems
Inthischapterwediscusstwoimportantconceptsforperformingappliedresearch.Theseconcepts,measurementreliabilityandmeasurementvalidity,arepartof
overallresearchvalidity,orthequalityofthewholestudy.Inchapters6and10,wegaveabriefoverviewofthefourcomponentsofresearchvalidity.Thosefour
componentswereinternalvalidity,measurementreliabilityandstatistics,measurementvalidityandthegeneralizabilityoftheconstructs,andexternalvalidity.
Measurementreliabilityisacomponentofmeasurementreliabilityandstatistics.Measurementvalidityisacomponentofmeasurementvalidityandthegeneralizability
oftheconstructs.Wediscussmeasurementreliabilityandvalidityindepthinthischapter.Then,inchapter23,wediscusshowbotharepartofresearchvalidity.
MeasurementReliability
Whatisreliability?Whenapersonissaidtobereliable,wehavecertainconceptionsaboutthatperson.Forexample,thepersonalwaysshowsupformeetingson
timetherefore,heisareliableperson.Or,thepersonalwaysgetsthejobdonetherefore,sheisareliableperson.Whenweusetestsorotherinstrumentstomeasure
outcomes,wealsoneedtomakesurethattheseinstrumentsarereliable.Cronbach(1960)saidthatreliability''alwaysreferstoconsistency

Page311

throughoutaseriesofmeasurements"(p.127).Theimportanceofreliabilityforresearchmethodscannotbeoverstated.Ifouroutcomemeasureisnotreliable,then
wecannotaccuratelyassesstheresultsofourstudy.Hence,ourstudywillbeworthless.
AnExample
Tounderstandtheimportanceofmeasurementreliabilityanditsunderpinnings,itisbesttostartwithanexample.Aresearcherisinterestedindeterminingifqualityof
lifeforpersonswithcognitivedisabilitiescanbeincreasedthrougharecreationalsupportprogram.Todetermineiftheintervention(recreationalsupportprogram)
works,hedesignsarandomizedexperimentbyusingaPretestPosttestcontrolgroupdesign,whereonegroupreceivestheintervention(X)for6months,andthe
othergroupdoesnotreceivetheintervention(~X).Bothgroupsreceivethepretestwithaninstrumentthatmeasuresqualityoflife,andthenafterthe6monthperiod,
theyreceivethesameinstrumentontheposttest.Asinearlierchapters,thedesigncanbeshownasfollows:
R

O1

O2

O1

~X

O2

Theresearcherwillmeasurequalityoflifewithaparticularmeasurementtool(dependentvariable),whichwewillcalltheQualityofLife(QOL)inventory.Hewill
measureboththeinterventionandcontrolgroupspriortotheinterventionontheQOLandthenagainaftertheperiodoftheintervention.Therefore,eachparticipantin
thestudywillobtainascoreontheQOLpriortotheinterventionandaftertheinterventionperiod.IftheQOLinventoryhasarangebetween0and100,theneach
participantwillreceivebothapretestscoreandaposttestscorewithinthisrange.Theresearcherhopesthattheposttestscoresintheinterventiongrouparehigher
thanthoseofthecontrolgroup.Becauseoftherandomassignment,thegroupsshouldbeequivalentinitially.Aswehaveseeninchapters6and7,thisdesignisa
strongoneintermsofinternalvalidity.However,itispossiblethatthestudyisweakinotherrespects.Forexample,populationexternalvaliditycouldbelowifthe
participantswereunrepresentativeofthetheoreticalpopulation.TheissuehereiswhethertheQOLinventorywillmeasurequalityoflifeconsistently(reliably)inthis
studywiththisgroupofparticipants.
TestScores
Wecallanyscorethatweobtainfromanyindividualonaparticularinstrument,anobservedscore.IfJonesscores49onthepretestoftheQOL,thenJones'
observedscoreis49.IfweweretogiveJonestheQOLasecondtime,hisobservedscoreprobablywillbedifferentfrom49.Itmightbe53or43.Ifwegavethe
QOLtoJonesathirdtime,thescoreprobablywillbedifferentfromeitherofthescoresreceivedfrompreviousadministrationsofthetest.BecauseJones'scorewill
notbethesameeachtimewegivetheQOL,andbecausewemustgiveJonesasecondQOLaftertheintervention,howwillweknowifthechangeinJones'score
frompretesttoposttestisduetotheintervention,orperhapsduetosomethingelse?Statedanotherway,howdowe

Page312

knowwhetherthechangeinJones'scoreisduetosystematicvariation(variationasaresultoftheintervention)orunsystematicvariation(variationasaresultofother
factors).Tounderstandourproblem,wemustconsiderclassicaltesttheory,andtrueanderrorscores.
Accordingtoclassicaltesttheory,anobservedscoreconsistsofatruescoreanderror.Ifwecouldsubtractthetruescorefromtheobservedscore,wecould
determinehowmuchofthescoreisduetoerror.Weneveractuallyknowtheamountoftheobservedscorethatisduetothetruescore,andtheamountofthe
observedscorethatisduetoerror.Ifweweretomeasurethepersonthousandsoftimes,andtaketheaverageofallofthosemeasurements,thentheaveragescore
wouldbeveryclosetotheindividual'struescore.Unfortunately,werarelymeasureapersonmorethanacoupleoftimesonanygiveninstrument.
Becausewerarelymeasureapersonmultipletimeswithanyinstrument,theresearchermayhavetroublewithhisstudy.Again,theproblemisthatifheistryingto
assessthechangethatresultsfromhisintervention,hewillneedtomeasureeachparticipantmorethanonetime.SupposeJonesisintheinterventioncondition,andhis
QOLscoreincreasesfrom49(pretest)to53(posttest).HowdoweknowwhetherthisincreaseisduetoanincreaseinJones'truescore(systematicvariation)or
merelyduetoerror(unsystematicvariation)?Thesolutiontotheproblemistochooseatestwithhighreliability.Wehavenotconsideredspecificmethodsof
determiningreliabilityatthispoint,butwehavestatedthatreliabilityisameasureofconsistency.Howdoesreliabilityrelatetoobservedscoresandtruescores?
Measurementreliabilityisexpressedasacoefficient.Thereliabilitycoefficientistheratioofthevarianceoftruescorestothevarianceofobservedscores(Ghiselli,
Campbell,&Zedeck,1981).Inotherwords,thehigherthereliabilityofaninstrument,thecloserthattruescoreswillbetoobservedscoresforthatinstrument.Now,
givenwhatweknowaboutobservedscores,truescores,anderror,weshouldconsidercorrelationcoefficients.
CorrelationCoefficient
Wecandiscussreliabilityconceptuallyasaformofconsistency.However,whenevaluatinginstruments,itisimportanttobeabletoexpressreliabilityinanumerical
form.Thisallowsustocomparedifferentinstrumentsonpropertiesofreliability.Themeasuremostoftenselectedtoevaluatereliabilityisreferredtoasacorrelation
coefficient.Asdiscussedinchapter16,acorrelationcoefficientisusuallyexpressedastheletterr,andindicatesthestrengthofarelation.Thevaluesofrrange
between1and+1.Avalueof0isviewedasnorelationbetweentwovariablesorscores,whereasvaluescloseto1or+1areviewedasverystrongrelationships
betweentwovariables.Astrongnegativerelationship,oftenreferredtoasaninverserelationship,indicatesthatthehigherthescoreisononevariableortest,the
lowerthescoreisonasecondvariableortest.Ontheotherhand,astrongpositiverelationshipindicatesthatpeoplewhoscorehighononetestalsowillscorehigh
onasecondtest.Althoughthecorrelationcoefficientrcanvaryfrom1to+1,tosaythatameasurementisreliable,onewouldexpectacoefficientbetween+.7and
+1.0.Othershavesuggestedevenstrictercriteria.Anastasi(1988),whendiscussingareliability

Page313

coefficientof.72stated,"Nevertheless,theobtainedcorrelationissomewhatlowerthanisdesirableforreliabilitycoefficientswhichusuallyfallinthe.80s
and.90s"(p.115).Nunnally(1978)statedthatreliabilitycoefficientsof.8areacceptableforresearch,butthat.9isnecessaryformeasuresthatwillbeusedtomake
decisionsaboutindividuals,byusinginstrumentssuchasIQtests,GREs,SATs(ScholasticAptitudeTests),andthoseforpersonneldecisions.However,itiscommon
toseepublishedjournalarticlesinwhichoneorafewreliabilitycoefficientsarebelow.7,usually.6orabove.Notethat,althoughcorrelationsof.7to1.0indicatea
strong(negative)correlation,theyaretotallyunacceptableforreliability.Suchahighnegativecorrelationwouldindicatethatpersonswhoinitiallyscorehighonthe
measurelaterscorelowandviceversa.Anegativereliabilitycoefficientprobablyindicatesacomputationalerrororterribleinconsistency.
MethodstoAssessReliability
TestRetestReliability
Testretestreliabilityisoneofthemostcommonformsofreliability(Daniel&Witta,1997).Cronbach(1960)referstothiscoefficientasacoefficientofstability.
Testretestreliabilityiseasytounderstand.Ifatestisreliable,andifitisgivenmorethanoncetothesameperson,thatperson'sscoresshouldbeveryclosetoone
another,ifnotequal.IftheresearcherwantstoobtaintestretestreliabilityonhisQOLinstrument,hewouldfindasampleofpersonswhowerenotparticipantsinthe
experimentbutwhowouldfithistargetpopulation.HewouldadministertheQOLtothissample,andatalaterdate(atadatethatwouldapproximatetheintervalof
theintervention),hewouldadministertheQOLtothesamesample.Thenhewoulddeterminethereliabilitycoefficientfromthescoresofthetwoadministrationsby
usingacorrelationbetweenthesetsofscores.Ifthereliabilitycoefficientisrelativelyhigh,forexampleabove.80,hewouldbesatisfiedthattheQOLhasgoodtest
retestreliability.Ontheotherhand,ifthereliabilitycoefficientisbelow.70,thenhemayneedtoreconsidertheQOLasareliablemeasureofqualityoflife.
Certainconsiderationsmustbetakenintoaccounttodeterminetestretestreliability.Thefirstpointisthattestretestreliabilityisnotestablishedduringastudy.The
reliabilitycoefficientmustbeestablishedaheadoftime,priortothestudy,byusingaperiodoftimewhenlittlerelatedtothesubstanceoftheinstrumentshouldbe
happeningbetweenthetwoadministrationsoftheinstrument.1Inmostcases,testretestreliabilityhasalreadybeenestablishedfortheinstrumentofchoice,sothe
investigatorneednotworryabouthavingtodeterminereliabilityforthestudy.Inmanycases,morethanonereliabilitycoefficienthasbeenobtainedforaparticular
instrument.However,theinvestigatorneedstomakesureofthefollowingcriteriawhenselectingtheinstrument:
1.Thereliabilityoftheinstrumentishigh(e.g.,above.80)oratleastmarginallyacceptable(e.g.,above.60).
1

Thisisespeciallyimportantforexperimentsandforareassuchaschilddevelopment,whererapidgrowthduringtheintervalbetweenthetwoadministrationsoftheinstrument
couldaltertestretestreliability.

Page314

2.Thelengthoftimethathadbeenusedtoestablishthetestretestreliabilityissimilartothelengthoftimetobeusedinthestudy.Itshouldbenotedthatasthelength
oftimeincreasesbetweenadministrations,thereliabilitydecreases.
3.Thesamplethathadbeenusedtodeterminereliabilityoftheinstrumentissimilartothesamplethatwillbeusedinthecurrentstudy.
ParallelFormsReliability
Oneoftheproblemsofusingthesameinstrumentforthepretestandtheposttestofastudyisthatparticipantsmayusetheknowledgegainedonthepretesttoalter
theposttestscore.Thisproblem,oftenreferredtoastesting(orcarryover)effects,createssignificantproblemsfortheinvestigatorbecauseitbecomesimpossibleto
determineifthechangeinscoresisduetotheinterventionortoknowledgeobtainedonthepretest.Onewayofavoidingthepretestproblemistocreateadesign
withoutapretest,forexample,theposttestonlycontrolgroupdesign.However,thatdesignonlycanbeusediftheinvestigatorcanrandomlyassignparticipantsto
groups.Aquasiexperimentalapproachismorelikelyinappliedsettings,wheretheinvestigatorwillneedtouseapretest.
Tocounteractthetestingproblem,sometestshaveasecondorparallelformthatcouldbeusedasaposttestinplaceoftheinstrumentusedforthepretest.Parallel
formsreliability(coefficientofequivalence,Cronbach,1960)involvesestablishingtherelationshipbetweenthetwoformsofthesametest.Thistypeofreliabilityis
easytoestablish,becauseitinvolveshavingasampleofparticipantstakethetwoformsofthesameinstrumentwithverylittletimeelapsedbetweenthetwo
administrations.Thenacorrelationcoefficientisdeterminedforthetwosetsofscores,similartothatdoneintestretestreliability.Again,areliabilitycoefficientofat
least.80wouldbeexpectedforgoodparallelformsreliability.
InternalConsistencyReliability
Often,inadditiontoobtainingtestretestreliability,orparallelformsreliability,theresearcherwantstoknowthattheinstrumentisconsistentamongtheitems,thatis,
theinstrumentismeasuringasingleconcept,orconstruct.Ratherthancorrelatedifferentadministrationsofthesameinstrument,theinvestigatorcanusetheresultsofa
singleadministrationoftheinstrumenttodetermineinternalconsistency.Themostcommonmethodsofdetermininginternalconsistencyarethesplithalfmethod,the
KuderRichardson(KR20)method,andCronbach'salpha.Thelasttwomethodsareoftenreferredtoasinteritemreliability.Allthreecanbeusedonlywhenone
hasdatafromseveralitemsthatarecombinedtomakeacompositescore.
SplitHalfMethods
Thesemethodsofobtaininginternalconsistencyreliabilityinvolvecorrelatingtwohalvesofthesametest.Thetermsplithalfisageneraltermtodescribeanumberof
differentmethodsofcorrelatingonehalfofthetestwiththesecondhalfofthetest.Forexample,onecouldcorrelatethefirsthalfofthetestwiththesecondhalfofthe
test,orcorrelatetheodditemswiththe

Page315

evenitems.Athird,andhighlyrecommendedmethod,istorandomlysamplehalfoftheitemsofthetestandcorrelatethemwiththeremainingitems.
Oneoftheproblemswithobtainingsplithalfreliabilityisthatwhendividingthetestintotwohalves,thenumberofitemsisreducedby50%,comparedtoatestretest
reliabilityoralternativeformsreliability.Thisreductioninsizemeansthattheresultingcorrelationcoefficientwillprobablyunderestimatereliability(Suen,1990).
Therefore,oncethereliabilitycoefficientisestablishedbycalculatingthecorrelationcoefficient,r,itisnecessarytoadjustthesizeoftherbyusingtheSpearman
Brownformula,2asfollows:

whereIratio=theratioofthenumberofitemsinthedesiredtestdividedbythenumberofitemsintheoriginaltest,and
r=thecorrelationcoefficientofthetwohalvesofthetest.
However,becauseweareusingtheSpearmanBrownformulainaspecificinstance,splithalfreliability,Iratiowillalwaysbe2(twicethesizeofthesplittest).
Therefore,theformulaforadjustingthereliabilitycoefficientforsplithalfreliabilityisasfollows:

Therefore,ifyoucomputethecorrelationcoefficientbetweenthefirstandsecondhalvesofyourtest,andfindthatitequals.7,theSpearmanBrownformula
adjustmentwouldchangetherfrom.7to.82.
KuderRichardson20
Iftheinstrumentthatyouareusingisintendedtomeasureasinglethemeortrait,youmaywishtodeterminehowalloftheitemsarerelatedtoeachother.Ifeachitem
isscoreddichotomously,suchaspass/fail,true/false,right/wrong,thenKR20istheappropriatemethodofdetermininginteritemreliability.Tousethismethod,the
followingequationisused:

whereI=totalnumberofitemsonthetest,s2=totalvarianceofthetestscores,p=proportionofparticipantswhopassanitem,andq=proportionofparticipants
whodonotpassanitem.
2

TheSpearmanBrownformulaismostcommonlyusedtodeterminereliabilityofatestifmoreitemsaretobeaddedorsubtracted.

Page316

Cronbach'sAlpha
Ifeachitemonthetesthasmultiplechoices,suchasaLikertscale,thenCronbach'salphaisthemethodofchoicetodetermineinteritemreliability.Cronbach'salpha
currentlyisthemostcommonlyusedindexofreliabilityintheareaofeducationalandpsychologicalresearch(Daniel&Witta,1997).TocompleteCronbach'salpha
thefollowingequationisapplied:

whereI=totalnumberofitemsonthetest,s2=totalvarianceofthetestscores,and

=sumofthevariancesforeachitem.

Itshouldbenotedthatmeasuresofinteritemreliability,especiallyCronbach'salpha,areoftenseenwhenreadingaresearcharticle.Thereasonforthis,asstated
earlier,isthatittakesonlyoneadministrationoftheinstrument.Moreimportant,though,isthatalphaisrelatedtoconstructvalidity.Oneoftheproblemswith
Cronbach'salphaisthat,althoughitisameasureofinternalconsistency,itdoesnotnecessarilymeasurehomogeneity,orunidimensionality.Inotherwords,people
oftendetermineCronbach'salpha,andassumethatbecauseitisatahighlevel,forexample,.85,thetestismeasuringonlyoneconceptorconstruct.Unfortunately,as
pointedoutbySchmitt(1996b)andSpector(1992),eventhoughtheoverallitemcorrelationsmayberelativelyhigh,theycouldbemeasuringmorethanonefactoror
dimension.Thiscanleadtoproblems,becauseoneoftheassumptionsofusingCronbach'salphaasanindexofreliabilityisthatitismeasuringonlyoneconstruct.We
cautionthat,whenreportingreliability,ifonlyCronbach's isprovidedwithoutinformationindicatingthatthereisonlyoneunderlyingdimension,oranotherindexof
reliability,thenreliabilityhasnotbeenadequatelyassessed.
StandardErrorofMeasurement
Whenselectingatest,oneofthemostimportantquestionstoask,inadditiontoreliabilityandvalidityinformation,iswhattypeofvariabilityofperformancemightwe
expect.Inthepreviouschapter,wediscussedthestandarddeviationasanindexofvariabilityandalsointroducedthenormalcurve.Wewillneedboth,inadditionto
informationonreliability,whenweconsiderthestandarderrorofmeasurement.Thestandarderrorofmeasurementallowsustoestablisharangeofscores
(confidenceinterval)withinwhichshouldlieaperformer'struescore.First,let'slookattheequationforthestandarderrorofmeasurementthenweprovidean
exampletohelpexplaintheconcept.Theequationforthestandarderrorofmeasurementisasfollows:

wheres=standarddeviationofthetest,andr=thereliabilitycoefficient.

Page317

SupposethatyouhaveanintelligencetestliketheWechslerAdultIntelligenceScale(WAIS),thathasaknownstandarddeviationof15.Let'ssaythatthereliability
coefficientis,onaverage,.92.Whatwillbethestandarderrorofmeasurement?Fittingthesenumbersintotheequation,thestandarderrorofmeasurement(Sm)=
4.24.Agivenindividualtakesthetestandscores110.Fromourearlierintroductiontoclassicaltesttheory,thisobservedscoreisequaltoatruescorepluserror.
Wedonotknow(norwillweeverknow)theindividual'struescore.Therefore,wemustestimatethetruescorefromasingletest.Todothis,weusethestandard
errorofmeasurement,andsetaconfidenceintervalaroundtheobservedscore.Thesizeofthisconfidenceintervalwilldependonhowsurewewanttobethatthe
truescorefitswithinthisinterval.Inmostcases,wewanttobeatleast95%sure(twostandarddeviations).Therefore,wesetupa95%confidenceintervalaround
theobservedscore.Todothis,wemultiplyourstandarderrorofmeasurementtimesthezscorethatrepresentstwostandarddeviationsfromthemeanonanormal
curve.Thiszvalueis1.96(seeFig.9.2).Therefore,4.24times1.96givesusavalueof8.32.Wecanconcludethatourtruescorefallswithinthe95%confidence
intervalof110 8.32orbetween101.68and118.32.Or,ifthepersontookthetest100times,95timesthetruescorewouldfallwithinthatinterval.
Thestandarderrorofmeasurementillustratestheimportanceofthereliabilitycoefficient.Supposethatinthisexample,ourreliabilitycoefficientwas.65insteadof.92.
Ourstandarderrorofmeasurementwouldnowbe12.09.Wemultiplythisvaluetimes1.96toestablishourconfidenceinterval.Ourconfidenceintervalis110
23.70orbetween86.3and133.7.Theprecisionofourestimateofthetruescorehasdecreasedsubstantiallyasaresultofalowreliabilitycoefficient.
Interrater(Interobserver)Reliability
Thepreviousmethodstoestablishreliabilitywereaccomplishedbyexaminingscoresonaninstrument.However,sometimesthemeasurementtoolisobservation
performedbyjudges(usuallythecasewithqualitativeresearch).Whenobservationistheinstrument,thenreliabilitymustbeestablishedamongthejudgestomaintain
consistency.Thistypeofreliabilityisreferredtoasinterraterreliability.Althoughtherearenumerouswaystodeterminethisformofreliability,thecommonthemeis
thattwoormorejudges(observers)scorecertainepisodesofbehaviorandaformofcorrelationisperformedtodeterminethelevelofagreementamongthejudges.
PercentageAgreementMethods
Thesemethodsinvolvehavingtwoormoreraters,priortothestudy,observeasampleofbehaviorsthatwillbesimilartowhatwouldbeobservedinthestudy.
SupposethatraterAobserves8occurrencesofaparticularbehaviorandraterBobserves10occurrencesofthesamebehavior.Apercentageisthencomputedby
dividingthesmallernumberofobservationsbythelargernumberofobservationsofthespecificbehavior.Inthiscase,thepercentageis80.Oneoftheproblemswith
thismethodisthat,al

Page318

thoughbothobserversmayagreethatabehaviorwaselicitedaparticularnumberoftimes,thisdoesnotmeanthateachtimethebehavioroccurredbothjudges
agreed.Forexample,supposethatthebehaviorofcooperationwasthedependentvariableforastudy.Priortothestudy,twojudgesweretoobserveaclassroomof
studentsforparticularinstancesofcooperation.Oneobserver(judge)saidthattherewere8examplesofcooperation.Asecondobserversaidthattherewere10
examplesofcooperation.Thepercentageagreementwouldbe8dividedby10or80%.However,itispossiblethatthe8instancesobservedbyonejudgewerenot
thesameinstancesobservedbythesecondjudge.Thepercentagewouldbeinflatedinthisparticularinstance.Ottenbacher(1986)suggestedusingapointbypoint
basisofestablishinginterraterreliability.Inthismethod,eachbehaviorwouldberatedasanagreementordisagreementbetweenjudges.Thepointbypointmethod
wouldbeeasiesttoperformifthebehaviorisonatapethatcouldbeplayedforthejudges.Tocalculatepercentageagreementinthepointbypointmethod,the
numberofagreementsbetweenthetwojudgeswouldbedividedbythetotalnumberofresponses(agreementsplusdisagreements).Aproblemwiththismethodis
thatitignoreschanceagreementswhenfewcategoriesareused(Bartko&Carpenter,1976).Anadditionalproblemwiththesepercentageagreementmethodsisthat
theyaremostsuitedtosituationswithonlytworaters.
IntraclassCorrelationCoefficients(ICC)
Often,whenperformingastudybyusingobservationsofbehaviorasthedependentvariable,morethantwoobserversareneeded.Intraclasscorrelationcoefficients
allowtheresearchertocalculateareliabilitycoefficientwithtwoormorejudges.(ForanexcellentreviewofICCtypemethods,includingKappa,seeBartko&
Carpenter,1976.)Onecriterionthatmustbesatisfiedtousetheintraclasscorrelationcoefficientisthatthebehaviortoberatedmustbescaledatanintervallevel.For
example,eachratermightberatinginstancesofcooperationona15scale.TheseICCsarecomputedbyusinganalysisofvariancemethodswithrepeatedmeasures
toanalyzeinterraterreliability.3(Wediscussedrepeatedmeasuresanalysisofvarianceinchap.15.)AsecondadvantageoftheICCmethodofcomputinginterrater
reliabilityisthatifthejudgesareselectedrandomly,thentheresearchercangeneralizetheinterraterreliabilitybeyondthesampleofjudgeswhotookpartinthe
reliabilitystudy.
Kappa
AmethodofcalculatingintraclasscorrelationcoefficientswhenthedataarenominalistheKappastatistic.SimilartoICC,Kappacanbecomputedwithtwoormore
raters.Kappaalsocorrectsforchanceagreement.AlthoughthedataforusingKappaareoftendichotomous,forexample,presentorabsent,itisnotuncommonto
havemorethantwonominalcategories.
3

Althoughitappearsthatintraclasscorrelationcoefficientsareusedmostcommonlyforinterraterreliability,especiallyinrehabilitationliterature,thesemethodscanbeusedfor
testretestreliabilityandinternalconsistencyreliability(Shavelson,1988).

Page319

SummaryofReliabilityMethods
Wehavediscusseddifferentmethodsofassessingreliability.Althougheachmethodprovidesameasureofconsistency,notallprovidethesamemeasureof
consistency.Itisuptotheconsumertobeawareofhowreliabilitywasestablishedbeforeusingaparticularinstrument.Tosaythataninstrumentisreliablehas
relativelylittlemeaning.Eachstatementofreliabilitymustspecifythetypeofreliabilityandthestrengthofthereliabilitycoefficient.
Typically,ifonedoesnotcreatetheinstrument,butusesaninstrumentalreadypublished,thenreliabilityindicesshouldhavebeenestablished.Themostcommon
placestofindstudiesofthereliabilityoftheinstrumentareintheinstrumentmanual,whichisoftenreferredtointheoriginaljournalpublicationthatintroducedthe
instrument.Theinstrumentationsectionofanyresearcharticlethatusedtheparticularinstrumentshouldalsoprovideinformationaboutthereliabilityofthatinstrument.
MeasurementValidity
Earlyinthischapter,wementionedthatmeasurementreliabilityandmeasurementvaliditywerepartoftheoverallconceptofresearchvalidity.Measurementvalidityis
partofthecomponentthatwehavelabeledmeasurementvalidityandgeneralizabilityoftheconstructs.
Thecurrentviewofvalidityisthatitisanevaluationofscoresonaparticulartestandofhowthesescoreswillbeinterpreted.Thus,whenweaddresstheissueof
validityforaparticulartest,weareaddressingtheissueofthevalidityofthescoresonthattestforaparticularpurpose,andnotthevalidityofthetestorinstrument
perse.Therefore,anyparticulartestmightbeusedforanumberofpurposes.Forexample,specialtyareascoresonthegraduaterecordexaminationmightbeusedto
predictfirstyearsuccessingraduateschool.However,theyalsocouldbeusedasamethodtoassesscurrentstatusinaparticularundergraduatemajor.Althoughthe
sametestisusedinbothinstances,thepurposeofthetestiscompletelydifferentforeachsituation.Or,asacolleaguehaspointedout,achainsawis"valid"fortree
surgerybutnotbrainsurgery.Therefore,priortousingthetest,validitywouldneedtobedeterminedforeachpurposeoruse.
Validity,or,perhapsmoreimportantly,theevaluationofvalidity,isconcernedwithestablishingevidencefortheuseofaparticularinstrumentinaparticularsetting.In
thenextsectionweillustratedifferentmethodsofgatheringevidencetosupportvalidity.Itshouldbenotedthatonlyonetypeofevidence,nomatterhowstrong,is
insufficientforestablishingvalidity.Instead,allofthedifferentmethodsshouldbeappliedtowardevaluationofthevalidityofaparticulartestorinstrument.
Eventhoughaninstrumentmaybeconsistent(highreliability),itmaynotbevalid.Forexample,onecouldconstructadeviceformeasuringalengthof12inches.
However,supposethatthedeviceactuallymeasures13inches.Thedevicewillbeconsistentbutnotvalidbecauseitdoesnotmeasurewhatitissupposedto
measure.Althoughthecorrelationcoefficientismostoftenusedto

Page320

describemeasurementreliability,thereisnoonetypeofstatistictodescribemeasurementvalidity.However,thecorrelationcoefficientisusedtodescribeonetypeof
measurementvalidity,criterionvalidity.Whenonereadsaresearcharticle,thereisusuallymoreinformationaboutthereliabilityoftheinstrumentthanaboutthevalidity
oftheinstrument.Thereasonforthisisthatmeasurementvalidityisusuallymoredifficulttoobtain.Wediscussfourdifferenttypesofevidenceforvalidity:face
validity,contentvalidity,criterionvalidity,andconstructvalidity.Note,however,thatmanyresearchersdonotconsiderfacevaliditytobeascientificallyrecognized
typeofmeasurementvalidity.
FaceValidity
Aninstrumentissaidtohavefacevalidityifthecontentappearstobeappropriateforthepurposeoftheinstrument.Thekeywordisappears.Facevaliditydoesnot
actuallydescribethecontent.Forexample,facultyareoftendelugedbybooksalespersonstryingtogetthemtousetheirtextforaclass.Sometimes,theinstructorwill
selectthebookfromthetableofcontents,becausethetopicsappeartobecovered,eventhoughtheinstructorhasnoideahowwellthetopicsarecovered.Similarly,
supposeaninstructoristeachingaclassinstatistics.Themidtermexamcontainsquestionsthatappeartobeaboutstatisticsthatwerecoveredinclass.Studentsmay
feelthatthistesthasgoodfacevalidity(atleastbeforethemidterm),eventhoughtheproblemsarenotsolvableand,thus,notagoodmeasureofstatisticsknowledge.
Sofacevalidityisnotenough,butifnothingelse,facevalidityisasellingpointforaninstrument.
ContentValidity
Contentvalidity,asopposedtofacevalidity,referstotheactualcontentoftheinstrument.Specifically,oneasksifthecontentthatcomprisestheinstrumentis
representativeoftheconceptthatoneisattemptingtomeasure.Forexample,A.G.Fisher(1995)hasconstructedaninstrumentcalledtheAssessmentofMotorand
ProcessSkills(AMPs).Oneofthemostimportantcontributionsoftheinstrumentisthatithasecologicalsoundness.Fisherhasparticipantschoosetoperform
everydaytasksfromalistofpossibletasksthatrequiremotorandprocessskills.IfFisheraskedparticipantstostackblocksorperformotherartificialtypesofmotor
tasks,thenhertestwouldnothavestrongcontentvalidity,eventhoughtheartificialtasksinvolvedmotoractivity.Hertesthasstrongcontentvaliditybecausethetasks
notonlyinvolvemotorandprocessactivity,butalsobecausetheyarerepresentativeofthetypesoftasksthatapersonwoulddoineverydaylife.
Thereisnostatisticthatdemonstratescontentvalidity.Instead,theprocessofestablishingcontentvalidityusuallystartswithadefinitionoftheconceptthatthe
investigatorisattemptingtomeasure.Asecondsteptocontentvalidityisaliteraturesearchtoseehowthisconceptisrepresentedintheliterature.Next,itemsare
generatedthatmightmeasurethisconcept.Gradually,thislistofitemsisreducedtoformthetest.Oneofthemethodsofreducingitemsistoformapanelofexpertsto
reviewtheitemsforrepresentativenessoftheconcept.

Page321

CriterionRelatedValidity
Whenpeoplementionmeasurementvalidity,theyareusuallyreferringtocriterionvalidity.Criterionvalidityreferstovalidatingtheinstrumentagainstsomeformof
externalcriterion.Thisvalidationprocedureusuallyinvolvesestablishingacorrelationcoefficientbetweentheinstrumentandtheexternaloroutsidecriterion.Thekey
tocriterionvalidityisbeingabletoestablishanoutsidecriterionthatismeasurable.Commonexamplesofcriterionvalidityinvolveinstrumentsthatareintendedto
selectparticipantsforaschooloraprofession.Therearetwotypesofevidenceforcriterionvalidity:predictiveevidenceandconcurrentevidence.
PredictiveEvidence
Whenwetrytodeterminehowsomeonewilldointhefutureonthebasisofaparticularinstrument,weareusuallyreferringtopredictiveevidence.Testssuchasthe
ScholasticAptitudeTest,theGraduateRecordExamination,andtheLawSchoolAptitudeTestareexamplesofinstrumentsthatareusedtopredictfuture
performance.Forexample,theScholasticAptitudeTest(SAT)isoftenrequiredforstudentsattemptingtoentercollege.IftheSAThasgoodpredictiveevidence,
thenstudentswhoscorehighonthistestwillperformbetterincollegethanthosewhodonotscorehigh.Thecriterioninthiscasewouldbeameasureofhowwellthe
studentperformsincollege,usuallygradesduringthefirstyear.
Toestablishpredictiveevidenceinthisexample,highschoolstudentswouldtaketheSAT.Then,whentheyarefinishedwiththeirfreshmanyearofcollege,
correlationswouldbeestablishedbetweentheirhighschoolSATscoresandcollegegrades.Ifthecorrelationishigh,thenpredictiveevidenceisgood.Ifthe
correlationislow,thenthetesthasproblemsforpredictionoffutureperformance.Aproblemwithpredictiveevidenceisthatoftennotalloftheparticipantswhowere
evaluatedontheoriginalinstrumentcanbeevaluatedonthecriterionvariable.Thisisespeciallythecaseinselectionstudies.Forexample,wemayhaveSATscores
forawiderangeofhighschoolstudents.However,notallofthesestudentswillattendcollege.Therefore,ourcriterionvariableoffirstsemestercollegeGPAwillnot
onlyhavefewerparticipantsthanourpredictorvariable,butwillrepresentamorehomogeneousgroup(thoseselectedintocollege).Therefore,therangeofscoresof
thosewhocouldparticipateinthestudyonboththepredictorandcriterionvariablesisrestricted,leadingtoasmallercorrelationcoefficient(Anastasi,1988).
Aseconddrawbackwithpredictiveevidenceisthat,toestablishvalidity,theresearchermustwaituntilthosewhoweretestedinitiallycanbemeasuredonthe
criterion.Sometimesthiswaitcouldtakeyears.Therefore,asecondtypeofcriterionvaliditywasdevelopedtodealwiththisproblem.
ConcurrentEvidence
Similartopredictiveevidence,concurrentevidencealsoexaminestherelationshipbetweenaninstrumentandanoutsidecriterion.However,asmentionedearlier,
sometimesitistooexpensivetowaitbetweenthetimethatthetestwastakenandthemeasurementofthecriterion.Forexample,supposethatwewereinterestedto
seeiftheSATtakeninhighschoolwasagoodpredictoroffreshmangradesincollege.However,wedonotwish

Page322

towaitthetimeittakesforthehighschoolstudentstobecomefreshmen.Todetermineconcurrentevidence,wecouldtakecurrentfreshmenandhavethemtakethe
SATandseeifitcorrelateswiththeirgrades(presentgradesbecausetheyarenowfreshmen).Ifthereisahighcorrelation,wecanhaveconfidenceinusingthe
instrumentasapredictorforsuccessincollege.However,concurrentevidenceisnotthesameaspredictiveevidence,andonemaynotwishtoplaceasmuch
confidenceinthisprocedure.Also,restrictedrangeproblemssimilartothosepointedoutunderpredictiveevidencearepresent.Inthisexample,wemustmakethe
assumptionthattherewaslittlechangebetweenhighschoolstudentsandcollegestudents,becausethetargetfortheSATishighschoolstudents.Iftherearelarge
changesbetweenhighschoolandcollege,thevalidityoftheinstrumentshouldbequestioned.Thebestsituationwouldbetoobtainbothpredictiveandconcurrent
evidence,althoughCronbach(1960)suggeststhatthisrarelyoccurs.
Concurrentevidencealsocanbeobtainedbysubstitutinganotherinstrumentforthecriterion,especiallyifitisdifficulttomeasurethecriterion.Forexample,the
AssessmentofMotorandProcessSkills(A.G.Fisher,1995)(discussedearlier)wascomparedtotheScalesofIndependentBehaviorinadultswithdevelopmental
disabilities(Bryze,1991).However,Cromack(1989)pointsoutthattheinstrumentthatissubstitutedforthecriterioncanneverbemorevalidthanthecriterion.One
mustbecautiouswhensubstitutinganinstrumentforacriterion,becauseinmanycasesthesubstitutedinstrumenthasnotbeenvalidatedagainstthecriterionofinterest.
Thisisoftenthecasewiththerapeuticoreducationaloutcomes.Perhapsmoreimportant,ifanotherinstrumentissubstitutedforthecriterion,whatsizecorrelation
wouldbeexpected?Ifthecorrelationcoefficientisquitehigh,forexample,.8or.9,thenyourinstrumentisnotprovidingdifferentinformationfromthecriterion
instrument.Ifthecorrelationistoolow,thenyourinstrumentismeasuringadifferentconstructthanthecriterioninstrument.
Themajordrawbacktocriterionvalidityistheproblemoffindingasuitablecriterionandthenbeingabletomeasurethatcriterion.Forexample,gainingadmissionto
occupationalandphysicaltherapyprogramsintheUnitedStatesisverydifficultbecauseofthehighnumberofapplicantsforthelimitednumberofpositions.Toselect
thesuccessfulapplicants,criteriasuchasgradesandachievementtestsareoftenused.Students(especiallythosewhoarenotadmitted)oftencomplainthathigh
gradesdonotmakethepersonagoodtherapist.Couldonecreateanadmissiontestthatwouldpredictbecomingagoodtherapist?Considertheproblemsofdefining
andmeasuringthecriterionof''whatmakesagoodoccupationalorphysicaltherapist?"
ConstructValidity
Thelasttypeofmeasurementvalidity,andcertainlythemostcomplex,isconstructvalidity.Constructsarehypotheticalconceptsthatcannotbeobserveddirectly.
Intelligence,achievement,andanxietyareallconstructs.Althoughwecannotobserveaconstructdirectly,mostofusagreethattheseconstructsexistthroughdifferent
observablebehaviors.Forexample,wecannotdirectly

Page323

observeanxiety,butundercertaincircumstanceswemayobserveanxiousbehaviorssuchassweatingorpacingthatarespecifictoaparticularcontext,suchas
immediatelybeforeanimportantexamination.Therefore,itiscommontocreateinstrumentstomeasureparticularconstructs,suchasatestthatmeasuresstateanxiety,
oratestthatmeasuresintelligence.
Whenapplyingconstructvaliditytoaninstrument,thereisarequirementthattheconstructthattheinstrumentismeasuringisguidedbyanunderlyingtheory.Often,
especiallyinapplieddisciplines,thereislittleunderlyingtheorytosupporttheconstruct.AsCronbach(1960)pointsout,"Sometimesthetestisusedforalongtime
beforeanytheoryisdevelopedaroundit"(p.121).Constructvalidationisa(relativelyslow)processwheretheinvestigatorconductsstudiestoattemptto
demonstratethattheinstrumentismeasuringaconstruct.Threeprocessesthatareimportantforachievingconstructvalidityareconvergentevidence,discriminant
evidence,andfactorialevidence.
ConvergentEvidence
Thisisdeterminedbyobtainingrelativelyhighcorrelationsbetweenyourscaleandothermeasuresthatthetheorysuggestswouldberelated.Todemonstrateconstruct
validity,onedevelopshypothesesaboutwhattheinstrumentshouldpredict(convergentevidence,orvalidity)ifitisactuallymeasuringtheconstruct.
DiscriminantEvidence
Thisisprovidedbyobtainingrelativelylowcorrelationsbetweenyourscaleandmeasuresthatthetheorysuggestsshouldnotberelatedtoit(Lord&Novick,1968).
Discriminantevidencecanalsobeobtainedbycomparinggroupsthatshoulddifferonyourscaleandfindingthattheydo,infact,differ.
FactorialEvidence
Thistypeofevidenceisprovidedwhenaconstructiscomplex,andseveralaspects(orfactors)ofitaremeasured.Iftheclusteringofitems(usuallydonewithfactor
analysis)supportsthetheorybasedgroupingofitems,factorialevidenceisprovided(seechap.19forabriefdiscussionoffactoranalysis).
Examples
Thefirstexample,fromtheareaofrehabilitation,demonstratesthesepoints.Harveyetal.(1992)developedaninstrumentcalledthePatientEvaluationand
ConferenceSystem(PECS)topredictclinicallybasedadmissiondecisions.Theypredictedthattheirinstrument,thePECS,coulddiscriminateamongpatientsin
hospital,intermediate,anddayprogramsintermsofthefunctionalindependenceandimpairment.Theyfoundthat,overall,theirinstrumentcorrectlyclassified75%of
thepatientsforeachofthecategories.Thissamegroupofinvestigatorsalsoprovidedfurtherevidenceofconstructvaliditybyperformingfactoranalysistoreduce
itemsthatwereeitherredundantornotrelatedtotheconstruct(Silversteinetal.,1992).
AsecondexampleconcernstheAssessmentofMotorandProcessSkills(AMPs)instrumentdevelopedbyA.G.Fisher(1995).BernspangandA.G.Fisher(1995)
predictedthatAMPsscoreswoulddiffersignificantlybetweenpersonswithCerebralVascularAccident(CVA)whencomparedtoanondisabledagegroupcontrol.
Ontheotherhand,theyalsopredictedthat

Page324

therewouldbenodifferencesbetweenpersonswhohadaCVAinthelefthemispherecomparedtothosewhohadaCVAintherighthemisphere.Thislastprediction
isveryimportantfortheAMPs,becausethetasksperformedforthisinstrumentarethosedoneeveryday.TheresultssupportedBernspangandFisher'spredictions
andprovidedbothdiscriminantandconvergentevidenceforthevalidityoftheAMPs.
Athirdexampleisfromthemasterymotivationliterature(Morganetal.,1993).TheDimensionsofMasteryquestionnaire(DMQ)wasdesignedtomeasurefive
aspectsordomainsofmasterymotivation:persistenceatobjectrelatedtasks,grossmotortasks,socialmasterymotivationwithpeers,socialmasterymotivationwith
adults,andmasterypleasure.Factoranalysissupportedthegroupingofitemsintothesefiveclustersandprovidedfactorialevidence.
Constructvalidityisneveractuallyachieved.Itisacontinuingprocessofexperimentationandmodificationleadingtotherefinementoftheinstrumentthatmeasuresthe
construct.
Summary
Measurementreliabilityandvalidityareexceptionallyimportantissuesforresearchinappliedsettings.Manyoftheissuesarebeyondthescopeofthepresenttext.For
thoseinterestedinmeasurementreliabilityandvalidityinmoredepth,especiallyforconstructinganinstrument,werecommendthetextsofAnastasi(1988),Cronbach
(1970),andGhisellietal.(1981).Table20.1providesasummaryoftheconceptscoveredintheprecedingsections.
TABLE20.1
MeasurementReliabilityandValidity
StabilityorConsistency

CorrectnessorAccuracy

Theparticipantgetsthesameorverysimilarscorefrom
atest,observation,orrating.Thereisreliabilityofthe
following:

Thescoreaccuratelyreflectsormeasureswhatitwas
designedorintendedtomeasure.

a.Participants'responses

a.Facevaliditytheitemslookvalid.

1.Testretestreliabilitystabilityovertime.

b.Contentvalidityallaspectsoftheconstructare
representedinappropriateproportions.

2.Parallelformsreliabilityconsistencyacross
presumablyequivalentversionsoftheinstrument.

c.Criterionrelatedvalidity1.Predictivethetest
predictssomecriterioninthefuture.2.

3.Internalconsistencyitemsthataretobe
combinedarerelatedtoeachother.

Concurrenttestandcriterionaremeasuredatthe
sametime.

b.Observers'responses
1.Interraterreliabilitydifferentobserversorraters
givesimilarscores.

d.Constructvalidity
1.Convergentbasedontheory,variables
predictedtoberelatedarerelated.

2.Discriminantvariablespredictednottobe
relatedarenotrelated.

3.Factorialfactoranalysisyieldsatheoretically
meaningfulsolution.

Page325

StudyAids
Concepts
Cronbach'salpha
Correlationcoefficient
Error
Intraclasscorrelationcoefficients
Kappastatistics
KuderRichardson20
Observedscore
Percentageagreementmethods
Splithalfreliability
Standarderrorofmeasurement
Truescore
Distinctions
Convergentversusdiscriminantversusfactorialevidenceforconstructmeasurementvalidity
Faceversuscontentversuscriterionversusconstructevidenceformeasurementvalidity
Predictiveversusconcurrentevidenceforcriterionrelatedmeasurementvalidity
Measurementreliabilityversusmeasurementvalidity
Measurementvalidityversusresearchvalidity
Testretestversusparallelformsversusintervalconsistencyversusinterraterevidenceformeasurementreliability
ApplicationProblems
1.Aresearcherisinterestedindeterminingiftherapistsinterruptfemaleclientsmorethanmaleclients.Hehasobtainedvideotapesof30therapysessions,andplansto
countthenumberoftimesthetherapistinterruptsfemaleandmaleclients.Theresearcherhiresagraduatestudenttocounttheoccurrenceofinterruptionsonallthe
tapes.Theresearcherthenhiresanothergraduatestudenttocounttheoccurrenceofinterruptionson12ofthetapes.Whydidtheresearcherhirethesecondgraduate
student?Whattypeofevidenceforvalidityorreliabilityistheresearcherconcernedwith?Whatstatisticalproceduremighttheresearcherbestusetodeterminethis
typeofreliabilityorvalidity?
2.Aresearcherisinterestedintheinfluenceofmaritalequalityonmaritalsatisfaction.Inreadingtheliterature,shelearnsthatmanyvariables(orfactors)havebeen
usedtooperationallydefineequality,suchasshareddecisionmakingpowerbetweenspouses,fairdivisionoflabor,andequalaccessto

Page326

finances.Shedecidestoincludeseveralofthesefactorsinhermeasureofequality.Istheresearcherconcernedwithreliabilityorvalidityasshemakesdecisionsabout
thebestwaytomeasureequality?Whatkindofreliabilityorvalidityissheprincipallyconcernedwith?Inanalyzingherdata,howmightshedetermineifthevariables
shemeasuredwererelatedtoherindependentvariable?
3.Aresearcherhasdevelopedameasureofanxiety.Sheplanstousethemeasureforthefirsttimetodetermineiflearningaboutstressreductiontechniqueswill
influenceanxietylevels.Shegivesheranxietymeasuretoherundergraduatepsychologyclass,andteachesthemstressreductiontechniques.Thenextday,shegives
heranxietymeasureagain.Sheisexcitedtoreportthathertestretestreliabilityisveryhigh(0.98).Yourealizethatshedoesnotunderstandtestretestreliabilityfully.
Whatrecommendationswouldyoumaketoherabouthowtoappropriatelyestablishtestretestreliability?
4.Whatistheappropriatemethod(ifany)fordetermininginternalconsistencyreliabilityforthefollowingmeasures.
a.A10itemmeasureoflocusofcontrolscoredwithTrueorFalse.
b.An80itemmeasureofintimacyscoredwitha5pointLikertscale.
c.Aoneitemmeasure("Whosecareerisgivenmoreprioritybetweenyouandyourspouse?")ofrelativecareerprioritybetweenspouses.
5.ResearchersatalargeelementaryschoolinColoradowouldliketomeasurechangeinaptitudeonscientificconceptsforallfifthgradersbeforeandafteranew
experientialsciencecurriculumhasbeenintroduced.Anexamwouldbegivenasapretestmeasure.However,researchersareconcernedaboutcarryovereffects,so
willuseanotherformoftheirinstrumentastheirposttest.Whatisthecoefficientofequivalence?Explainhowtheywillcheckthereliabilityoftheirinstruments.
6.ResearchersarepresentedwithanewformofintelligencetesttousewithelementaryagechildrenintheUnitedStates.Thetesthasbeenpilottestedwithgreat
excitementinseveralwesternstates.Coloradoresearcherswouldliketohavemoreinformationbeforepilotingtheinstrument.Theyhavebeeninformedthatthe
standarddeviationis15andthereliabilitycoefficientis0.74,ontheaverage.Whatisthestandarderrorofmeasurement?Whyisitusefultoknowthis?Howwould
thisbemeasured?
7.AnInstrumentofSupportwasusedtomeasureperceivedsupportfromcoworkersinamentalhealthinstitution.Participantsrespondedtofouritemsona7point
Likertlikescale.Cronbach'salphaforthe(support)scalewas0.79.Whatdoesthismean?
8.Glinerhasdevelopedamultiplechoicetestcalled,"Iwanttogetintogradschoolrealbad,"tomaketheselectionprocesseasier.(Also,ifenoughotherschoolsare
interested,hemightmakesomemoney.)Firsthewantstodetermineevidenceforreliability.Describeatleastthreemethodsthathecouldusetoassessreliability.
a.Afterstudiesonreliabilityhavebeenperformed,Glinerconcludes,"The'Iwanttogetintogradschoolrealbad'testisreliable."What,ifanythingiswrong
withthisstatement?

Page327

b.Next,Glinerconductsapredictivevaliditystudy.Hegiveshistesttoallstudentsadmittedtothegraduateprogramintheyear1988.Fiveyearslater,he
sendseachstudentaoneitemquestionnaire.Thequestionasked,"Howmuchmoneydoyoumakeperyear?"ThecorrelationbetweenscoreontheGliner
testandsalarylevelwas0.70.Therefore,Glinersuggestedthatthetestbeusedinthefutureforapplicants.
c.WhataresomeoftheproblemsencounteredwiththewayGlinerestablishedvalidityevidence?
d.HowcouldGlinerhaveobtainedvalidityinformationusingconcurrentvalidity?
e.Whencomparedtopredictivevalidity,whataretheadvantagesanddisadvantagesofconcurrentvalidity?

Page329

Chapter21
TypesofDataCollectionTechniques
Overview
TypesofDataCollectionTechniques
DirectObservation
NaturalnessoftheSetting
DegreeofObserverParticipation
AmountofDetail
BreadthofCoverage
StandardizedVersusInvestigatorDeveloped(OneStudy)Instruments
StandardizedTests
AchievementTests
PerformanceandAuthenticAssessments
AptitudeTests
StandardizedPersonalityInventories
AttitudeScales
Summated(Likert)AttitudeScales
SemanticDifferentialScales
QuestionnairesandInterviews
Questionnaires
MailedQuestionnaires
DirectlyAdministeredQuestionnaires
TypesofQuestionnaireItems
Interviews
FocusGroups
Summary
StudyAids
Concepts

Page330

Distinctions
ApplicationProblems
Overview
Therearemanytypesoftechniquesandinstrumentsusedtocollectdata.Someresearchmethodsbookshaveanumberofchapters,eachfocusingonadifferent
techniqueortoolsuchasinterview,questionnaire,projectivetechniques,tests,orobservations.Becausethisbookfocusesonresearchdesignandtheresultingdata
analysis,wehavechosentodeemphasizeourtreatmentofdatacollectiontechniques.Inaddition,thisbookisdesignedforabroadaudienceofstudentsinthemany
disciplinesrelatedtoeducation,appliedhealthsciences,andappliedsocialsciences.Becauseeachofthesefieldshasitspreferreddatacollectiontechniques,wehave
focusedonwhatiscommonacrossthesedisciplines,namelydesignandanalysis.Whatwedointhischapterisprovideabroadcontextforthinkingaboutdata
collectiontechniquesandsourceswhereyoumaygotolearnmoreaboutthespecificsofdevelopingorevaluatingaquestionnaire,interview,orotherdatacollection
technique.
Aspointedoutinchapter1,weconceptualizetheresearchapproachesanddesignsasbeingapproximatelyorthogonaltothetechniquesofdatacollection,andthus,in
theoryatleast,anytypeofdatacollectiontechniquecouldbeusedwithanyapproachtoresearch.Itistruethatsometypesofdatacollectionaremorecommonly
usedwiththerandomizedexperimentalorquasiexperimentalapproaches.Othersaremorecommonwithcomparativeorassociationalapproaches,andstillothers
aremorecommoninqualitativeresearch.
Table21.1approximateshowcommoneachofseveraldatacollectiontechniquesarewithineachofthesethreemajorgroupsofresearchapproaches(listedonthe
lefthandsideofthetable).Notethatwehaveorderedthedatacollectiontechniquesalongadimensionfromobserverreporttoselfreport

Page331
TABLE21.1
DataCollectionTechniquesUsedbySpecificResearchApproaches
(Symbolsinthetableindicatelikelihoodofuse)

ResearchApproach

Qualitative
Research

QuantitativeResearch

DataCollectionTechniques
Researcherreportmeasures

Comparative,
Experiments&Quasi
Associational,&
Experiments
DescriptiveApproaches

Physiologicalrecordings

++

Physicaltracemeasures

Codedobservations

++

++

Narrativeobservations

++

Participantobservations

++

Othermeasures

Standardizedtests

++

Archivalmeasuresor
documents

++

Contentanalysis

++

Selfreportmeasures

Summatedattitudescales

++

Standardizedpersonalityscales

++

Questionnairesorsurveys

++

Interviews

++

++

Focusgroups

++

++Quitelikely
+Possibly
Notlikely

measures.Theobserverreportendincludesobservationsandphysiologicalrecordingsthatareassumedtobelessinfluencedbytheparticipants'desiretolookgood
ortoanswerinasociallydesirableway.Ofcourse,eventhesemeasuresarenotfreeoftheeffectsofsuchfactorsif,asisusuallythecase,theparticipantsrealizethat
theyarebeingobservedorrecorded.Attheotherendofthescalearemeasuresbasedonselfreportsoftheparticipants,suchasinterviews,questionnaires,focus
groups,andattitudeandpersonalityscales.Inthesecases,theresponsesareclearlyfilteredthroughtheparticipants'eyesandareprobablyheavilyinfluencedby
factorssuchassocialdesirabilityandansweringinacceptableways.Inthemiddle,wehaveputseveraltypesofmeasuresthatareundoubtedlyinfluencedbythe
participants'consciousorunconsciousneedtolookgoodbutare,perhaps,lesssusceptibletosuchfactorsbecausetheidea(instandardizedachievementandaptitude
tests,forinstance)isforpeopletodoaswellastheycaninfiguringoutthecorrectanswer.Witharchivaldocumentsandcontentanalysis,thedataaregatheredfrom
recordsmadeforanotherpurposesotheremaybelessbuiltinbias.
Theconcernaboutthefilteringofparticipants'answersthroughperhapsfaultymemoriesorintermsofsociallydesirableresponseshasledquantitative

Page332

researchers,especiallythosewhotendtousetherandomizedexperimentalandquasiexperimentalapproaches,tobesuspiciousaboutthevalidityoftheselfreport
instruments.Thus,whenusingselfreportmeasures,youshouldalwaysbepreparedtoprovideevidencesupportingtheirvalidity,asdiscussedinthepreviouschapter.
Ofcourse,someselfreportinformation,suchasgenderandothersimplequestionsoffactthatarenotsensitiveorcontroversial,areusuallyacceptedatfacevalue.
Ontheotherhand,observerreportsarenotnecessarilyvalidmeasuresofwhattheyareintendedtoassess.Oneissuethatisoftenpointedoutbyqualitative
researchersisthatculturalbiasesmayleadobserverstointerprettheirobservationsininappropriateorethnocentricways.
Recommendationsforfurtherreadingaboutdatacollectiontechniquesareprovidedinthereferences.Ingeneral,itisadvisabletoselectinstrumentsthathavebeen
usedinotherstudiesiftheyhavebeenshowntobereliableandvalidwiththetypesofparticipantsandforthepurposethatyouhaveinmind.Tests(1991)andTests
inPrint(1994)providereferencestothousandsofpublishededucational,psychological,andbusinessinstrumentsthatareavailableforpurchaseoruse.TheMental
MeasurementsYearbooksprovidesummariesandreviewsofalargenumberofpublishedinstruments,includingaptitude,intelligence,andachievementtests,andalso
personalityandvocationalinventoriesorscales.Similarly,TestCritiquesannuallypublishesnorms,reliability,andvaliditydata,andpracticalapplicationsinauser
friendlystyle.Itcoversthemostfrequentlyusedpsychological,educational,andbusinessrelatedinstruments.Notethattheuseofthetermtestsinthosebooksis
broaderthanthewayitisusedinthisbook.Tests,asinTestCritiques,refertoabroadrangeofdatacollectiontechniques,notjustthosewithcorrectanswers,and
aresimilartoourtermstandardized.Textbooksontestingandmeasurement(e.g.,Anastasi,1988Thorndike&Hagen,1991)alsoprovideinformationonawide
varietyoftypesofstandardizedinstruments.TherelevantresearchliteratureandERICareothergoodsourcesforinstrumentsthatonemightuse.
Ofcourse,youmaynotbeabletofindaninstrumentthatsuitsthegoalsofyourresearch.Thisisespeciallylikelyifyouareinterestedinattitudes,orknowledge,or
bothaboutaspecifictopic,issue,orprogram.Inthiscase,youmaydecidetoconstructaquestionnaireorinterviewtoassesswhatyourparticipantsknowabout,or
howtheyperceivethetopic,orboth.Dillman(1978),Fowler(1993),SalantandDillman(1994),andSudmanandBradburn(1989)provideusefuladviceabout
developingandusinginterviewsandquestionnaires.
TypesofDataCollectionTechniques
DirectObservation
Asnotedearlier,manyresearchersprefersystematic,directobservationofbehaviorasthemostaccurateanddesirablemethodofrecordingbehavior,especiallythe
behaviorofchildren.Thefollowingdiscussionofobservationsdealswithwhatisoftencalleddirectobservationinwhichtheinvestigatortrainsobserverstoobserve
andrecordthebehaviorsoftheparticipantsinthestudy.In

Page333

directobservationsareusedwhentheinvestigatorinterviewsorotherwisequestionsuntrainedobservers,suchasparentsorteachers,aboutparticipants(e.g.,
children)thattheyknowwell.Indirectobservationcouldalsoincludequestionnairesorinterviewsbecausetheparticipantsoftenareaskedtoreportabouttheirown
behavior.Nowwewilldiscussseveralotherdimensionsonwhichobservationaltechniquesvary.
NaturalnessoftheSetting
Thesettingfortheobservationscanvaryfromnaturalenvironments(suchasaschool,playground,park,orhome)throughmorecontrolledsettings(suchasa
laboratoryplayroomthatisdesignedtolooklikealivingroom)tohighlyartificiallaboratorysettings(suchasusedinahospitalorphysiologicallaboratory).Inchapter
10,wediscussedtheissueofecologicalvalidity,oneaspectofwhichwasthenaturalnessofthesetting.Althoughnaturalsettingshaveecologicalvalidity,theyusually
sacrificeadegreeofcontrolandtheopportunitytopresentstimuliinasystematicway.Furthermore,equipmentsuchasvideocamerasandcomputerbased
observationalaidesaremuchmoredifficulttouseinanaturalsetting.Notethatqualitativeresearchersobservealmostexclusivelyinnaturalsettings.Quantitative
researchersusethewholerangeofsettings,butsomepreferlaboratorysettings.
DegreeofObserverParticipation
Thisdimensionvariesfromsituationsinwhichtheobserverisaparticipant(preferredbyqualitativeresearchers)tosituationssuchaspublicplacesinwhichthe
observerisentirelyunobtrusive.Mostobservations,however,aredoneinsituationswheretheparticipantsknowthattheobserverisobservingthemandhaveagreed
toit.Itiscommonforsuchobserverstoattempttobeasunobtrusiveaspossiblebysittingofftoonesideorobservingfrombehindaonewaymirrorinalaboratory.
AmountofDetail
Observationsalsovaryonthisdimension,whichgoesfromglobalsummaryinformation(suchasoverallratingsbasedonthewholeobservationperiod)tomoment
bymomentrecordsoftheobservedbehaviors.Obviously,thelatterprovidesmoredetail,anditrequiresconsiderablepreparationandtrainingofobservers.Moment
bymomentobservationsmayusecodesforvariousbehaviorsthatcanberecordedeitherwithpaperandpencilorwithsomeaid,suchasacomputerordictating
machine.Detailedrecordsalsocanbenarrativerecordsinwhichtheobserverdictatesorattemptstowritedowneverythingthathappensinsequentialorder.
BreadthofCoverage
Thisdimensionvariesfromobservationalschemesthatattempttorecordmanythingsthataregoingoninaperson'senvironmenttoveryspecificobservationsofone
orafewtypesofbehavior,suchasaggressiveincidentsortaskdirectedbehaviors.
StandardizedversusInvestigatorDeveloped(OneStudy)Instruments
Standardizedinstrumentsarethosethathaveresultedfromcarefulpreparationandcovertopicsofbroadinteresttoanumberofinvestigators.Theyareusually
publishedand,thus,aredescribedandreviewedintheMentalMeasurement

Page334

YearbookorTestCritiques.Theseinstrumentsusuallyhaveamanualthatincludesnormsusedtomakecomparisonswithabroadersamplethanisusuallyusedina
singlestudy,andtheyhaveinformationaboutreliabilityandvalidity.
Investigatordevelopedmeasuresarethosedevelopedbyaresearcherforuseinoneorafewstudies.Suchinstrumentsalsoshouldbecarefullydeveloped,andthey
shouldprovideatleastbasicevidenceofreliabilityandvalidityinthearticleorreportofthestudyinwhichtheywereused.However,thereusuallyisnoseparate
manualormaterialsavailableforotherstobuyoruse.
Thenextseveralsectionsusethisdistinction.Althoughsometests,personalitymeasuresandattitudemeasures,forinstance,aredevelopedbyinvestigatorsorteachers
foronetimeuseinaspecificstudy,therearemanystandardizedmeasuresavailableand,ingeneral,itiswisetousethemiftheyhavegoodreportedreliabilityand
validityandcovertheconceptthatyouintendtomeasure.Questionnairesandinterviewsareusuallydevelopedbyaninvestigatorforonetimeuseinaparticularstudy
onaspecifictopic.However,somequestionnairesandinterviewsareusedinseveralstudies,oftentoassessthesameissueatdifferenttimesforexample,thereisan
annualsurveyofenteringcollegefreshmenthathasaskedmanyofthesamequestionsforanumberofyears.
StandardizedTests
Althoughthetermtestisoftenusedbroadlytorefertoawiderangeofaptitude,personality,andattitudemeasures,wedefinethetermmorenarrowly.Byatest,we
meanasetofproblemswithrightorwronganswers.Thescoreisbasedonthenumberofcorrectanswersthatthepersonhad.Instandardizedtests,thescoresare
usuallytranslatedintoanormedscorethatcanbeusedtocomparetheparticipantswithotherswhohavetakenthetest.Thesetestsarereferredtoasnorm
referencedtests.Thescoresmaybeintermsofpercentileranksorawellestablishedmetricinwhichthemeanandstandarddeviationareknown.Forexample,the
GraduateRecordExam(GRE)scoreswereoriginallynormedsothat500wouldbethemeanand100wouldbethestandarddeviation.IQtestswerenormedso
that100wasthemeanand15wasthestandarddeviation.Analternativetonormreferencedtestsiscriterionreferencedtests.Thesetestsexaminehowwellthe
studentorparticipanthaslearnedaspecificskill(thecriterion).Suchtestsmeasureastudent'sachievementwithoutcomparingittothescoresofotherstudents.This
kindoftestisoftenusedinschools,butislesscommonlyusedinresearch.
Moststandardizedtestsaresaidtobeobjectivebecausethereislittledisagreementaboutthescoresobtainedfromthem.Theremaybedisagreementabouthowto
interprettheresults,butifamachineoranuntrainedassistantcanscorethetestorothermeasure,themeasureissaidtobeobjective.Multiplechoicetestsandrating
scalesaresaidtobeobjectiveessaytestsandprojectivetechniquesarelessobjectivebecausethescoresareinfluencedbythejudgmentofthescorers.
AchievementTests
Mostresearchabouttheeffectivenessofinstructionalmethodsusesachievementasthedependentoroutcomevariable.Thus,

Page335

achievementtestsarewidelyusedineducationalresearchaswellasinschools.Suchtestsmeasurethemasteryorachievementofstudentsinanarearelatedtowhat
theyshouldhavelearnedinschool.Achievementtestsareavailableforindividualschoolsubjectssuchasbiologyorhistory,andtheyarealsoavailablein
comprehensivebatteriesthatmeasurebroadareasofachievementsuchasverbalorquantitativeachievement.Forexample,theCaliforniaAchievementTest(CAT)
containstestsintheareaofreading,language,andarithmetic.Youneedtobecarefulwhenselectinganachievementtestthatitisreliableandappropriatefor
measuringtheaspectofachievementinwhichyouareinterested.Thetestalsomustshowreliabilityandvalidityevidencefortheparticipantstobeincludedinthe
study.Thus,ifyouareusingaparticularethnicgrouporstudentswithdevelopmentaldelays,youneedtobesurethatthetestisappropriateforthatsample.Ifthese
criteriaaremet,thenthereareadvantagesintheuseofastandardizedinstrument.Inadditiontosavingtimeandeffort,theresultsofyourstudycanbecomparedto
thoseofothersbyusingthesameinstrument.
Whentheavailablestandardizedtestsarenotappropriatefortheobjectivesofyourstudy,youmayhavetoconstructyourowntest.Itisbettertodosothantousean
inappropriatestandardizedtestjustbecauseitisavailable.Ifyoudodevelopyourowntest,youshouldbecarefulinpreparingitsothatyoudetermineitsreliability
andvaliditybeforeusingit.Refertothebooksontestsandmeasurementmentionedearlier(e.g.,Thorndike&Hagen,1991)ifyoudecidetodevelopyourown
achievementtest.
PerformanceandAuthenticAssessments
Althoughmostcommonachievementtestsarepaperandpenciltestsofthetypejustdescribed,aresearchermaywanttomeasureactualperformance,thatis,what
anindividualcandoratherthanwhatheorsheknows.Performanceassessmenthasbecomeapopularalternativetotraditionaltests.Insuchanassessment,the
investigatorobservesanindividual'sperformanceonacertaintaskandthenjudgestheproductonthebasisofcertaincriteria.Performanceassessmentsarecommon
insuchareasasart,music,orscience,wheretheindividualisexpectedtobeabletodoorproducesomethingsuchasapainting,arecital,oraresearchreport.
Someperformanceassessmentsarereferredtoasauthenticassessments,butnotallperformanceassessmentsareauthenticinthesensethattheyare''reallife"
assessments.Tobeconsideredauthentic,thetasksshouldbehighonecologicalvalidity,asdiscussedinchapter10,thatis,theymightincludesuchthingsasanactual
jobinterview,anindividualorgroupresearchproject,orareport.Performanceandauthenticassessmentsprovideawaytomeasureabilitiesandskillsthatarenot
easilyassessedbypaperandpenciltests.However,theytakemuchmoretimeandexpensetoadministerandscore.
AptitudeTests
Aptitudetestsinthepastwereoftencalledintelligencetests,butthistermislessoftenusednowbecauseofcontroversyaboutthedefinitionofintelligenceandto
whatextentitisinherited.Performanceonsuchaptitudetestsispartlydependentongeneticbackgroundandpartlyonenvironmentandschooling.Aptitudetests,in
contrasttoachievementtests,areintendedtomeasuremoregeneralperformanceorproblemsolvingability.Thesetestsat

Page336

tempttomeasuretheparticipant'sabilitytosolveproblemsandapplyknowledgeinavarietyofsituations.Researchersandeducatorshavefoundaptitudeteststobe
generallyusefulforthepurposeofpredictingschoolsuccessandasanindependentvariablethatmustbecontrolledineducationalstudies.Themanyaptitudeteststhat
areavailablecanbedividedintothosethatmustbeadministeredindividuallyandthosethatcanbeusedwithgroups.
ThemostwidelyusedindividualintelligencetestsaretheStanfordBinetandtheWechslertests.TheStanfordBinettestproducesanintelligencequotient(IQ),
whichisderivedbydividingtheobtainedmentalage(MA)bytheperson'sactualorchronologicalage(CA).TheStanfordBinetgivesageneralmeasureof
intelligenceandprovidesmeasuresofseparateabilities.ThereareseveralageversionsoftheWechslerintelligencescaleseachprovidestwoscoresforeachperson,
verbalandnonverbalIQ.Atrainedpsychometricianmustgivetheseindividualintelligenceteststoonepersonatatime,whichisexpensiveinbothtimeandmoney.
Groupaptitudetests,ontheotherhand,aremorepracticalforuseinschoolsystemsandinresearchwheregroupaveragesaretobeused.Therearenowmanygroup
aptitudetestsavailable,identifiedintheMentalMeasurementsYearbookorTestCritiques.
StandardizedPersonalityInventories
Personalityinventoriespresenttheparticipantwithacollectionofstatementsdescribingbehaviorsorpatternsofbehaviors.Theparticipantsarethenaskedtoindicate
whetherthestatementischaracteristicoftheirbehavior,bycheckingyesornoorbyindicatinghowtypicalitisofthem.Usuallythereareanumberofstatementsfor
eachcharacteristicmeasuredbytheinstrument.Someoftheseinventoriesassessonlyonetraitforexample,authoritarianismismeasuredbytheCaliforniaFScale
andanxietyismeasuredbytheStateTraitAnxietyScales.Otherpersonalityinventories,suchasCattell's16PersonalityFactorQuestionnaire,measurea
numberoftraits.Someinventoriesmeasurecharacteristicsthatonemightnotstrictlyconsidertobepersonality.Forexample,theStrongInterestInventoryisused
primarilytoassessvocationalinterests.Otherinventoriesmeasuretemperament(e.g.,ChildTemperamentInventory),behaviorproblems(e.g.,ChildBehavior
Checklist),ormotivation(e.g.,DimensionsofMasteryQuestionnaire).Noticethatthesepersonalityinstrumentshavevariouslabels:scale,inventory,questionnaire,
andchecklist.
Thesemeasuresaresaidtobestandardizedbecausetheyhavebeenadministeredtoawidevarietyofrespondentsandinformationaboutthesenormgroupsand
aboutthereliabilityandvalidityevidenceofthemeasuresisusuallyprovidedinthemanualfortheinventory.Itisalsopossibleforaninvestigatortodevelopameasure
ofanaspectofpersonalityspecificallyforaparticularstudy.Aswithstandardizedmeasures,reliabilityandvalidityneedtobeaddressed.
Paperandpencilinventorieshavetheadvantagesofbeingrelativelyinexpensivetoadministerandobjectivetoscore.However,therearedisadvantagesmostly
relatedtotheproblemofvalidity.Weshouldmentionherethattheva

Page337

lidityofapersonalityinventorydependsnotonlyonrespondents'abilitytoreadandunderstandtheitemsbutalsoontheirunderstandingofthemselvesandtheir
willingnesstogivefrankandhonestanswers.Althoughpersonalityinventories,especiallythemorecarefullydevelopedandstandardizedinventories,canprovideuseful
informationforresearch,thereisclearlythepossibilitythattheymaybesuperficialorbiased.
Anothermajortypeofpersonalityassessmentistheprojectivetechnique.Thesemeasuresarenotoftenusedineducationalandsocialscienceresearch,because
theyrequireanextensivelytrainedpersontoadministerandscore.Thus,theyareexpensive.Projectivetechniquesasktheparticipanttorespondtounstructured
stimulilikeinkblotsorambiguouspictures.Theyarecalledprojectivebecauseitisassumedthattherespondentwillprojecttheirpersonalityormotivationintotheir
interpretationofthestimulus.
AttitudeScales
Summated(Likert)AttitudeScales
Manypersonalityinventoriesusethesamesummatedmethodtobedescribedhere,butLikert(1932)initiallydevelopedthismethodasawayofmeasuringattitudes
aboutparticulargroups,institutions,orconcepts.Researchersoftendeveloptheirownscalesformeasuringattitudesorvalues,buttherearealsoanumberof
standardizedscalestomeasurecertainkindsofattitudeslikesocialresponsibility.Thereareseveralapproachestomeasuringattitudes.Wedescribeonlythe
summatedLikertscalesandthesemanticdifferentialscales.ThetermLikertscaleisusedintwoways:forthesummatedscaletobediscussedinthissectionandfor
theindividualitemsorratingscalesfromwhichthesummatedscaleiscomputed.Likertitemsarestatementsaboutaparticulartopicandtheparticipantsareaskedto
indicatewhethertheystronglyagree(SA),agree(A),areundecided(U),disagree(D),orstronglydisagree(SD).ThesummatedLikertscaleisconstructedby
developinganumberofstatementsaboutthetopic,usuallysomeofwhichareclearlyfavorableandsomeofwhichareunfavorable.Thesestatementsareintendedto
providearepresentativesampleofallpossibleopinionsorattitudesaboutthesubject.Thesestatementsarethenpresentedtoagroupofparticipantswhoareasked
torateeachstatementfromstronglydisagreetostronglyagree.Tocomputethesummatedscalescore,eachtypeofanswerisgivenanumericalvalueorweighting,
usually1forstronglydisagree,upto5forstronglyagree.Whencomputingthesummatedscale,thenegativelywordedorunfavorableitemsneedtobereversedin
termsoftheweighting.Inthiscase,stronglydisagreeisgivenaweightof5andstronglyagreeisgivenaweightof1.Considerthefollowingthreeitemsfromasocial
responsibilityscale(Berkowitz&Lutterman,1968):

1.Everypersonshouldgivesomeofhistimefor
thegoodofhistownorcountry.

SD

SA

2.Lettingyourfriendsdownisnotsobad
becausewecan'tdogoodallthetime.

SD

SA

3.Itisthedutyofeachpersontodohisjobthe
verybesthecan.

SD

SA

Page338

ApersonwithahighlyfavorableattitudetowardsocialresponsibilitymightcircleSAforthefirstitem,SDfortheseconditem,andAforthethirditem.Hisorher
summatedscorewouldbe5forthefirstitem,5fortheseconditem,and4forthethirditem,or14.Youshouldbeabletoseethatthesummatedscorescouldrange
from3forsomeonewhoisverylowonagreementwiththeattitudeofsocialresponsibilitytoamaximumof15forsomeonewhoismosthighlypositiveintermsofthis
attitude.
Summatedratingattitudescales,likealltheotherdatacollectiontoolsdiscussedinthischapter,needtobeinvestigatedforreliability,asdiscussedinchapter20.
Internalconsistencywouldbeindicatedifthevariousindividualitemscorrelatewitheachother,indicatingthattheybelongtogetherinassessingthisattitude.Validity
wouldbeassessedinthewaysdetailedinchapter20,byseeingifthissummatedscalecandifferentiatebetweengroupsthoughttodifferonthisattitudeorby
correlationswithothermeasuresthatareassumedtoberelatedtothisattitude.Theconstructionofsummatedscales(forattitudeorpersonalitymeasurement)is
discussedindepthbySpector(1992).
SemanticDifferentialScales
AnotherapproachtomeasuringattitudesisthesemanticdifferentialscaledevelopedbyOsgood,Suci,andTannenbaum(1957).Thismeasureisbasedonthe
assumptionthatconceptsorobjectshavewhatiscalledconnotativeinadditiontodenotative(ordictionary)meaningforindividuals.Connotativemeaninghastodo
withsurplusmeaningorwhattheconceptorobjectsuggestsorconnotestotheparticipant.
Semanticdifferentialscalesareadaptableandrelativelyeasytoconstruct,ifonewantstoknowhowparticipantsfeelaboutconceptssuchas"sitebased
management,""ADArequirements,"or"organizedreligion.''Participantsareaskedtoratetheconceptoneachofasetofbipolaradjectivepairs,whichOsgoodetal.
(1957)foundformedthreeclustersorfactors:evaluative,withadjectivepairssuchasgoodbadorvaluableworthlesspotencypairssuchasstrongweakorlarge
smallandactivitypairssuchasactivepassiveorfastslow.Theevaluativeclusterisusedmostofteninresearch.Thesemanticdifferentialscalesarescoredmuchlike
thesummatedratingscalesjustdiscussed.Theratingforeachitemisgivenascore,usuallyfrom1to7.Ifthepositivelyconnotedtermisontheleft,thescorewould
bereversed.Ifthepositivetermisontheright,noreversalwouldbedone.Thenthescoreforeachitemonascale(e.g.,evaluative)wouldbeadded.
QuestionnairesandInterviews
Thesetwobroadtechniquesaresometimescalledsurveyresearchmethods,butwethinkthatismisleadingbecausequestionnairesandinterviewsareusedinmany
studiesthatwouldnotmeetthedefinitionofsurveyresearch.Insurveyresearch,asampleofparticipantsisdrawn(usuallybyusingoneoftheprobabilitysampling
methodsdiscussedinchap.10)fromalargerpopulation.Thissampleisaskedaseriesofquestionsrelatedtoatopicaboutwhichtheyshouldhavesomeknowledge
orattitude.Theintentofsurveysistomakeinferencesdescribingthewholepopulation,sothesamplingmethodandreturnrateareimportantconsiderations,as
discussedinchapter10.

Page339

Questionnairesandinterviewsusedinsurveysareusuallydevelopedbytheinvestigatorforonetimeuseinaparticularstudy.However,sometimesthesameorsimilar
questionsareaskedonanumberofoccasionstoassesschangesinattitudes,productpreferences,orvotingpreferencesovertime.Questionnairesareanygroupof
writtenquestionstowhichparticipantsareaskedtorespondinwriting,oftenbycheckingorcirclingresponses.Interviewsareaseriesofquestionspresentedorally
byaninterviewerandareusuallyrespondedtoorallybytheparticipant.Bothquestionnairesandinterviewscanbehighlystructuredwithcloseendedquestionsin
whichthepossibleanswersarespecifiedandtheparticipantsmerelypickoneoftheprovidedresponses.However,itiscommonforinterviewstobemoreopen
ended,allowingtheparticipanttoprovidedetailedanswerstoquestionsthatdonotlendthemselvestoshortanswers.
Questionnaires
Therearetwobasicwaystogatherinformationwithaquestionnaire:mailedquestionnairesanddirectlyadministeredquestionnaires.
MailedQuestionnaires
Inthiscase,namesandaddressesofpersonsinthepopulationmustbeassembled.Then,asamplefromthispopulationisselectedbyusingoneofthetechniques
describedinchapter10.Whentheaccessiblepopulationissmall,allpersonsmaybesampled.Thisgroupisthenmailedaquestionnairewithacoverletteranda
stamped,returnaddressedenvelope.Reminderpostcards,orduplicatecopiesofthequestionnaire,orbothareoftensenttononrespondentsor,ifrespondentsare
notspecificallyidentified,toallpersonswhoinitiallyreceivedthequestionnaire.Comparedtointerviews,mailedquestionnairesarerelativelycosteffectivebecause
theyrequirelittletimetoadministeronthepartoftheinvestigatorsanddonotrequirethehiringofpersonstoadministertheinstrument.Informationcanbeobtained
relativelyrapidly,thatis,inafewweeks,butapoorresponserateisoftenobtainedbecauseoftheimpersonalityandlikelylackofrapportwiththeinvestigator.
DirectlyAdministeredQuestionnaires
Inthistechnique,thequestionnaireisusuallyadministeredtoagroupofpeoplewhoareassembledinacertainplaceforaspecificpurposesuchasaclassoraclub
meeting.Itisalsopossibletodirectlyadministeraquestionnaireinaonetoone,facetofacesituation,suchasgivingaquestionnairetothemotherofayoungchild
whiletestingthechild,butthisisrelativelyuncommon.Themainadvantageofdirectlyadministeredquestionnairesisthatahighresponserateisusuallyobtained,
especiallyiftheparticipantsareexpectedtobeinthatlocationanyway.Ontheotherhand,thesampleisunlikelytobeaprobabilitysamplefromadesiredtarget
population,inpartbecausesomepercentageofpotentialparticipantsprobablywillnotattendtheclassormeeting.Thiscanbeaseriousproblemincollege
classrooms.Thistechniquecanbecosteffectiveifitonlyrequiresoneorfewadministrationsofthequestionnaireandiftheadministrator'stimeisnotconsideredor
doesnothavetobepaid.
TypesofQuestionnaireItems
SalantandDillman(1994)provideanexcellentsourceforpersonswhowanttodevelopandconducttheirownquestion

Page340

naireorstructuredinterview.Theydescribefourtypesofquestionstructureforquestionnaireandinterviewitems:openended,partiallyopenended,closeended
unorderedchoices,andcloseendedorderedresponses.Eachofthesetypesofitemshasadvantagesanddisadvantages,asdiscussedbrieflyinthefollowing
paragraphs.
Openendedquestionsdonotprovidechoicesfortheparticipantstoselect.Instead,eachparticipantmustformulateananswerinhisorherownwords.Although
thistypeofquestionrequirestheleastefforttowrite,ithasseveralmajordrawbacks.Openendedquestionsaredemandingfortheparticipants,especiallyifthe
responseshavetobewrittenoutorareonissuesthatthepersonhasnotconsideredrecentlyoratall.Openendedquestionstendtoproducemanydifferent
responseswithonlyafewmentionsofeachtopic.Thistypeofquestiondoesnotprovidecomparableinformationacrossasample,becausepeoplewhodidnotthink
tomentionananswermighthavedonesoiftheyhadbeengivenchoicesfromwhichtoselect.Finally,theresponsestoopenendedquestionsrequireconsiderable
timetocodeandprepareforentryintoacomputer.However,thereareanumberofadvantagesthatmakethemusefulincertaincircumstances,especiallyifthe
investigatordidnothaveenoughknowledgebeforethestudytomakegoodcloseendedquestions.Sometimesopenendedquestionsrequireasimplestraightforward
answersuchastheperson'sdateofbirthorfavoriteclass.Inthesecases,developingalistofpossibleresponsesiswastefulofspace.Openendedquestionsaremore
oftensuccessfullyusedininterviewsthaninquestionnaires.
Partiallyopenendedquestionsusuallyprovideseveralpossibleanswersandthenhaveaspaceforotherresponsesorcomments.Thiscanbeuseful,butour
experienceisthatparticipantsusuallydonotusethespaces,andnotmuchadditionalinformationisprovided.
Closeendedunordereditemsarecommonlyusedwhenanswerstoaquestionfitnominalcategoriesthatdonotfallonacontinuum.Participantsareaskedto
chooseamongthesediscretecategoriesandselectwhichonebestreflectstheiropinionorsituation.Insomecases,thepersonisallowedtocheckallcategoriesthat
apply,butthenthequestionactuallybecomesaseriesofyes/noquestionswitheachresponsecategorybeingscoredlaterasifitwereaseparatequestion.Ifitisnot
possibletohaveacompletelistofpossibleanswers,apartiallyopenendedquestionmaybeused.
Closeendedquestionswithorderedchoicesarecommononquestionnairesandareoftensimilartotheindividualitemsinapersonalityinventoryorasummated
attitudescale.ThesequestionsmayinfactbesingleLikerttypeitemsinwhichastatementismadeandtherespondentisaskedtorateoneoraseriesofitemsfrom
stronglydisagreetostronglyagree.Anumberofothertypesofitemswithorderedchoicesarepossible(seeSalant&Dillman,1994).
Interviews
Twomaintypesofinterviewsaretelephoneandfacetoface.Telephoneinterviewsarealmostalwaysstructuredandusuallybrief,thatis,lessthanhalfanhour.This
techniqueiscommonlyusedbysurveyresearcherstoobtainaquick,geographicallydiverseornationalsample.Facetofaceinterviews,on

Page341

theotherhand,canvaryfromwhatamountstoahighlystructured,oralquestionnairewithcloseendedanswerstoindepthinterviews,whicharepreferredby
qualitativeresearcherswhowanttogetdetailedresponsesfromtheparticipants.Telephoneandstructuredfacetofaceinterviewsareusuallycodedonthespot.The
categoriesareoftencloseendedsothattheintervieweronlyneedstocirclethechosenresponseorfillinabriefblank.Indepthinterviewsareusuallytaperecorded
andtranscribedlatersothattheparticipant'scommentscanbecodedlater.Alltypesofinterviewsarerelativelyexpensivebecauseoftheironetoonenature.In
depthinterviewsareevenmoreexpensivebecauseoftraining,transcription,andcodingcosts.
FocusGroups
Focusgroupsarelikeinterviews,butrelativelysmallgroupsof8to10peopleareinterviewedtogether.Suchgroupsmaystimulatepeoples'thinkingandelicitideas
aboutaspecifictopic.Theyhavebeenusedbybusinessestolearnhowcustomerswillreacttonewproducts,andhavebeenusedbypoliticalcampaignstotestvoter
opinionsaboutatopic.Nonprofitagenciesmayalsousefocusgroupstoidentifytheperceptionsandideasofpotentialoractualparticipantsinaprogramoraservice.
Focusgroupscanprovideaninitialideaaboutwhatresponsespeoplewillgivetoacertaintypeofquestion.Thiscanbehelpfulindevelopingmorestructured
questionnairesorinterviews.
Summary
Thischapterprovidesanoverviewofmanyofthetechniquesusedintheappliedbehavioralsciencestogatherdatafromhumanparticipants.Mostofthemethodsare
usedbybothquantitative(positivist)andqualitative(constructivist)researchersbuttodifferentextents.Qualitativeresearchersalsoprefermoreopenended,less
structureddatacollectiontechniquesthandoquantitativeresearchers,butthisdistinctionisnotabsolute.Directobservationofparticipantsbytheresearcheris
commonamongexperimentalresearchersandqualitativeresearchersitislesscommonamongsocalledsurveyresearcherswhouseselfreportinterviewsand
questionnairesextensively.Itisimportantthatinvestigatorsuseinstrumentsthatarereliableandvalidforthepopulationandpurposeforwhichtheywillbeused.
Standardizedinstrumentshavemanualsprovidingnormsandindexesofreliabilityandvalidity.However,ifthepopulationsandpurposesonwhichthesedataare
basedaredifferentfromyours,itmaybenecessaryforyoutodevelopyourowninstrumentor,atleast,toprovidenewevidenceofreliabilityandvalidity.
StudyAids
Concepts
Directobservation
Focusgroup

Page342

Naturalnessofthesetting
Participantobservation
Performanceandauthenticassessment
Reliabilityandvalidityofthemeasures
Semanticdifferentialscales
Standardizedtests
Standardizedpersonalityinventories
Summated(Likert)attitudescales
Distinctions
Achievementtestsversusaptitudetests
Datacollectiontechniques(methods)versusresearchapproaches
Normreferencedversuscriterionreferencedtests
Openendedversusclosedendedquestions
Questionnairequestionoritemversusresearchquestion
Questionnaireversusinterview
Researcherreportmeasuresversusselfreportorparticipantmeasures
ApplicationProblems
1.Aresearcherdesignedameasureofworksatisfaction.Partofthismeasureisincludedbelow.
Shownbelowarepairsofwordsthatindicatehowpeoplefeelabouttheirwork.Considereachofthewordpairsandcirclethenumberthatbestindicateshowyou
feelaboutyourjobworkingeneral.
Boring

Interesting

Enjoyable

Miserable

Useless

Worthwhile

etc.
Whatkindofattitudescaleisthis?Howwouldyouscoreitifapersoncircled5,2,and6?
2.Table21.1givesanapproximationofhowcommoneachoftheseveraldatacollectiontechniquesarewithineachmajorgroupingofresearchapproaches.Why
wouldphysiologicalrecordingsbemostcommonforexperimentsandquasiexperiments?Whywouldselfreportmeasuresbemostcommonlyusedwithcomparative,
associational,anddescriptiveapproaches?Whyisitthatstandardizedtests,summatedattitudescales,andstandardpersonalityscalesareunlikelytobeusedin
qualitativeresearch?
3.Indicatewhetherthefollowingquestionsareopenended,orpartiallyopenended,orcloseendedordered,orcloseendedunordereditems.Discusstheprosand
consofformattingquestionsasshown.
a.Whatisyourdateofbirth?________________.

Page343

b.Isthereanyoneforwhomyouprovidespecialcareduetoillness,ahandicap,oroldage?
No_____
Yes_____
Pleaseexplain:__________________________________________________
c.Forwhichofthefollowingareasofexpendituredoyouhavethehighestpriority?
Defense_____
Education_____
HealthandWelfare_____
Other.Pleasespecify:_____________________________________________
d.Whattypeofworkschedulebestdescribesyourworksituation?
_____Standardfulltime(85)
_____Flexibleworkhours
_____Compressedweek
e.Whichbestdescribesthekindofbuildinginwhichyoulive?
_____Amobilehome
_____Aonefamilyhousedetachedfromanyother
_____Aonefamilyhouseattachedtoatleastoneotherhouse
_____Anapartmentbuilding
f.Pleasedescribethequalitiesofyourfavoriteteacher.
4.Yourcolleagueisinterestedinlearningifparentingstyleinfluencesadolescentdelinquency.Heasksforyouropinionaboutwhetherheshoulduseaquestionnaireor
interviewformattocollecthisdata.Whatdoyoutellhimaretheprosandconsofeach?
5.Aresearcherisinterestedinthedegreetowhichtherapeuticalliance(or,thestrengthoftherelationshipbetweenclientandtherapist)affectsthetherapeutic
outcome(or,thesuccessoftherapy).IftheresearcherobservesthesessionsfrombehindaonewaymirrorandratestherapeuticallianceonaLikertscale,whatkind
ofmeasureisthis?IftheresearcheraskstheclienttoreporttheirperceptionofallianceusingaLikertscale,whatkindofmeasureisthis?Whatarethebenefitsand
drawbacksofeach?
6.What'sthedifferencebetweenaresearchquestionandaquestionnaireoritem?Providetwoexamplesofeach.

Page345

Chapter22
StepsintheResearchProcess:
PracticalandEthicalIssues
StepsintheResearchProcess
PlantheSampleSelection
CooperatingAgencies
ResponseRate
Dropouts
PlantheDesignandAnalysis
Deception
Debriefing
ExperimentalResearch
NonexperimentalandQualitativeResearch
AnimalResearch
SelectorDeveloptheInstruments
PlantheProcedureforDataCollection
Confidentiality
ObtainApprovalfromtheInstitutionalReviewBoard
IRBsandHowTheyWork
TheResearchProtocol
PotentialProblemsWithResearchProtocols
Collect,Analyze,andInterprettheData
IntegrityoftheData
IntegrityofDataAnalysis
InterpretationoftheResults
DisseminatetheResults
Plagiarism

Page346

MultiplePublications
Authorship
CitingPublicationsinYourRsum
OtherPracticalandEthicalIssues
ReviewsandReviewers
ConflictsofInterest
MisconductandtheStructureofScience
Summary
StudyAids
Concepts
Distinctions
ApplicationProblems
StepsintheResearchProcess
Throughoutthisbook,wehavebeendiscussingtheprocessofappliedbehavioralresearch.Now,wereviewthesestepsbrieflywithanemphasisonpractical
considerationsandadiscussionofavarietyofethicalissuesrelatedtothevariousstepsintheprocessofdoingresearch.
PlantheSampleSelection
Inchapter10,wedescribedtheprocessofselectingasampleofpotentialparticipantsfromwhatisusuallyamuchlargertheoreticalpopulation.Severalstrategiesfor
selectingthesampleandseveralobstaclestoobtainingarepresentativesamplewerediscussed.Wepointedoutthatexternalpopulationvaliditydependsonthe
representativenessoftheaccessiblepopulationandalso

Page347

ontherepresentativenessoftheactualsampleofthoseparticipantswhoagreedtoparticipateandcompletedthestudy.
CooperatingAgencies
Toobtainabroadandrepresentativeaccessiblepopulation,itisoftennecessarytomakearrangementswithotheragenciesorinstitutionssuchasschooldistrictsor
clinics.Theseorganizationsmustbeconvincedoftheimportanceandbenefitsoftheresearchandthatanypotentialrisksareminimal.Ifithasaninstitutionalreview
board(IRB),thatIRBwillneedtoreviewtheproject,ormaydecidetoexemptit.IftheorganizationdoesnothaveanIRB,ameanstoassureyourownIRBthatthe
projectisacceptabletothemisneeded.Forexample,apersonauthorizedtoobligatetheagencycouldwritealettertoyourIRBstatingsupportfortheprojectand
theextentofanyassistance.Developingandmaintainingthesecontactscanbeatimeconsumingaspectofresearchthatneedstobeplannedandbudgeted.Thereare
alsoethicalissuestobeconsideredforcollaboratingagencies.Whatbenefitswilltheyandtheirstudentsorclientsgain?Willtheagencybenefitbutthestudentsor
clientsbeexposedtoapotentialriskorlossofprivacy?YourIRBwillnodoubtconsidertheseissuesandpossibleconflictsofinterest.
Avariantofthisiswhatiscalledbrokereddata.Inthiscase,theresearcherlacksaccesstoagivenpopulationandthebroker(e.g.,schoolprincipalorclinicdirector)
maynotallowtheresearchertoactuallycollectthedatabecauseofconcernaboutprivacy.Theagencymaybewillingtocollectthedatafortheresearcheroratleast
tohandoutanonymousquestionnairestotheirclientsandaskthemiftheywouldbewillingtorespond.Itisconsideredabreachofpatientproviderconfidentialityto
allowanoutsideresearcherfullaccesstomedicalfilesorevenalistofpatientstocontactdirectly.Contactorfilereviewshouldbedonebythehealthcareprovideror
school.Becauseclinicsandschoolsarebusy,theymaynothavetimetocontactclientsorreviewfiles.Thishasledtoafairamountoftensionbetweentheprinciples
ofrecruitmentethicsandthedesiretoobtaincompletedataandarepresentativesample.Alowresponserateand,likely,anunrepresentativesamplewillbecreatedif
theclinicorschoolannouncesthestudyandleavesituptopotentialparticipantstocontacttheresearcher.
ResponseRate
Anotherissueaboutresponserateistheneedtobalanceobtainingahighresponseratewithrespectforpersonswhodecidenottotakepart.Itisacceptabletotryto
convincepotentialparticipantsoftheimportanceandvalueoftheircontributionyoumayremindthemthattheyforgottoansweramailedsurvey.Youcanalsooffer
incentives,butyoumuststopbeforebecomingcoerciveoroffensive.Thismayespeciallybeaproblemwithtelephonesurveys,inpartbecausehiredinterviewersmay
gotoofarunlessproperlytrained.Rememberthatwellconstructedshortquestionnairesaremorelikelytoberespondedtothanpoorlyworded,long,oropenended
ones.
Dropouts
Inmultisessionandlongitudinalresearch,thereistheadditionalissueofmaintainingtheconsentandcooperationoftheparticipants.Inthesekindsofresearch,itis
importantthatparticipantsdonotdropoutofthestudyunnecessarily.Anycoerciontocontinueisunacceptable.Therefore,develop

Page348

inggoodrapportandmaintaininggoodsensitiverelationshipswiththeparticipantsandtheirneedswilloftenforestallsuchdropouts.Iftheparticipantsaretobe
rewardedfortheirparticipation,itmaybepossibletoarrangeproratedpaymentstowardtheendofthestudysothatitscompletionisrewarded.However,theIRB
willhavetoapproveanysucharrangementsandtheymustnotseemcoerciveorunfairtoparticipantswhodesiretoleavethestudymidway.
PlantheDesignandAnalysis
Itisimportanttocarefullyplantheresearchdesignandalsothedataanalysisbeforedatacollectionbegins.Statisticiansarefrequentlyfrustratedwhenan
inexperiencedresearchercomestothemwithapileofdataandasksforittobeanalyzed.Alltoooftenthedesignorinstrumentswerenotcarefullyplannedand,thus,
theappropriateanalysiscannotbeperformed.
Ontheotherhand,qualitativeresearcherssaythattheirdesignisemergentratherthanpreplanned.Webelievethatthisapparentdichotomybetweenqualitativeand
quantitativeresearchparadigmsisrelative,moreoneofdegreethanabsolute.Qualitativeresearchersneedtohaveagoodideaabouttheirresearchquestionsandat
leastagoodindicationoftheliteraturerelatedtothosequestions.Theywouldbeunwisetoembarkonamajorstudywithoutagoodideaabouthowtheyweregoing
toanalyzethedata.Itistruethatafterdoingafewinterviewsorobservationstheymaydiscoverthattheiroriginalresearchquestionswerenotthemostinterestingor
didnotelicittheinformationtheysought.Thentheymaydecidetoreformulatethequestionstoaskfutureparticipants.Thisisalsotrue,toalesserextent,of
quantitativeresearch.Allgoodresearchshouldbeginwithpilottestingtoensurethatthedesignandinstrumentsareappropriateandwillworkwelltoanswerthe
researchquestions.Ifitisdiscoveredthattheproceduresorquestionsarenotthemostappropriate,anewsampletoassessthenewquestionsshouldbeobtained.
Deception
Certainethicalissuesaremorelikelytoarisewithsometypesofdesignthanwithothers.Forexample,deceptionismorelikelytooccurinexperimentalresearch,but,
aswesawinchapter3(Humphreys,1970),therecanbedeceptioninqualitativeandsurveyresearchiftheparticipantsarenotfullyinformedoftheresearcher's
purposesandprocedures.Deceptioninvolvesamisrepresentationoffactsbycommission,whichoccurswhentheresearchergivesfalseinformationaboutthestudy.If
theinvestigatordoesnotfullyinformthesubjectsabouttheimportantaspectsofthestudyoritsgoals,omissionorconcealmenthasoccurred.
Untilrecentyears,socialpsychologicalresearchreliedheavilyondeceptiononthepremisethatinformationaboutcertaintopics,suchasconformityorobedience,
wouldbeunobtainablewithoutdeceptionbecauseofparticipants'defensiveness,embarrassment,orfearofreprisal.Wediscussed,inchapter3,theMilgram(1974)
studiesonobedience.Therewouldbetwoproblemswithrepeatingthatresearchtoday.First,it'snowtypicalforresearchparticipants,especiallycollegestudents,to
assumethatdeceptionwilloccurandforthemtoaltertheirbehaviorbasedonthatassumption.Second,IRBsprobablywouldnotallowitbecausedeceptionshould
notinvolveparticipantsinwaysthatthey

Page349

wouldfindunacceptable.Sieber(1992)statesthat''Anindefensiblerationalefordeceptionistotrickpeopleintoresearchparticipationthattheywouldhavefound
unacceptableiftheycorrectlyunderstoodit"(p.64).IRBsemphasizetrulyinformedconsentandrespectforautonomy.
DeceptionisallowableundercertaincircumstancesbutisrestrictedbyIRBsrecently.Aretherealternativestodeceptioninresearch?Simulations,whicharemock
situations,arebeingusedeffectivelytoexploresocialbehavior.Ethnographicorparticipantobservationmethodsareusedincreasinglytostudyrealbehavior,oftenina
communitybasedsetting.Ethicalandpracticalconsiderationshaveledsuchresearcherstoprovidefullyinformedconsentproceduresandtorelyonrapportandtrust
ratherthanclevernessordeceptionaswasthecaseintheMilgram(1974)obediencestudiesandtheHumphreys(1970)tearoomsexstudydescribedinchapter3.
Also,theparticipantsmaybeaskedtoconsenttotheresearcherconcealingimportantpartsoftheprocedure.Thereisnowevidencethatmostsubjectswillparticipate
inresearchwiththeunderstandingthatsomedetailsmustbewithhelduntilafterthestudy.Ofcourse,theyareguaranteedafulldebriefing.Often,afterthedebriefing,
participantsareofferedanopportunitytowithdrawtheirdatafromthestudy.Iftheparticipantstrusttheresearchertokeeptheirdataconfidential,fewarelikelyto
withdrawatthispoint.
Therearedeepdifferencesamongthemembersoftheresearchcommunityabouttheethicsofdeception.Somearestronglyagainstitandothersbelievethatitisthe
onlyviablewaytostudycertaintypesofsocialbehavior.Sieber(1992)makestwopointsonwhichshehopesallcanagree.First,someimportantbehaviorsvanish
underobviousscrutiny,and,thus,concealmentordeceptionissometimesnecessary.Second,themoreobjectionableformsofdeceptionareunnecessaryanddonot
needtobeused.
Debriefing
Debriefingisagoodpracticeformoststudiesandisalmostalwaysnecessaryfordeceptionstudies.Inadditiontodiscussingthegoalsofthestudyandreasonsforthe
deception,itisdesirabletoprovideevidenceaboutthedeception.Inthecaseoffalsefeedbackabouttestperformance,participantscouldbegiventheirown
unscoredtestsinasealedenvelopejustastheyhadsubmittedthem.Itisimportanttotrytoeliminateanyresidueofgeneralizedmistrustonthepartoftheparticipants.
Iftheresearcherdetectsanyundesirableemotionalresultsoftheresearch,heorsheshouldattempttorestoreparticipantstoaframeofmindatleastaspositiveas
thatwithwhichtheyenteredthestudy.
However,therearecertaincaseswhereitmightbebetternottodehoaxordebrieftheparticipantbecauseitmaybeharmful.Forexample,ifaresearcherwereto
studydishonesty,itmaybebetternottopointouttoparticipantsthattheirbehaviorduringthestudywasdishonest.Atanyrate,debriefingshouldbedonewithout
demeaningtheparticipants'behaviororattitudes.
ExperimentalResearch
Itisthenatureofexperimentaldesigns(randomizedexperimentalandquasiexperimentalapproaches)thatsomeoralloftheparticipantsaregivenaninterventionor
treatmentthatmaybemedical,psychological,oreducational.Withtheseinterventions,thereisalwaysthepossi

Page350

bilityofpotentialharm.Physicalharmismuchmorelikelywithmedicalinterventionsthanwitheducationalorpsychologicalinterventions,butlesstangibleharmis
possiblewithallinterventions.Forexample,theparticipantsinthenewcurriculumgroupmaylearnlessthantheywouldhaveiftheyhadstayedinthetraditional
curriculum.Orcertainkindsoftrainingmayrequiretheparticipantsintheinterventiongrouptobemoreopenandselfdisclosingthantheymightotherwiseprefer.If
thereispotentialriskofharmasaresultoftheintervention,itshouldbeasminor,reversible,ofshortduration,andnegatedaspossible.
Wehavehintedatdifficultissuesaboutthecontrolgroupinearlierchaptersaboutexperimentaldesigns.Forexample,ifanewtreatmentisfoundtobehighly
advantageous,itmaybeunethicaltowithholditfromthecontrolgroup.Itwouldbedesirabletoofferittothecontrolgroup.Insomecases,thiscanbedoneby
havingawaitlistcontrolgroupthatreceivesthetreatmentafteraperiodofdelaypresumablyequaltothetimethattheinterventiongroupwasgiventhetreatment.It
maybenecessaryforaninvestigatortobudgetthecostsofprovidingtreatmenttothecontrolgroupatalaterdate.
Inearlierchapters,wediscussedthedesignadvantagesofhavinganointerventionorplacebocontrolgroup.Ifaplacebocontrolgroupwereused,a"naturalstate
argument"wouldneedtobemadetotheIRB.Thereasoningisthatuntreatedparticipantsarenotbeingdeniedabenefittheyalreadyhave,butaremerelybeingleftin
theirnaturalstate.Thisargumentisseverelyundercutifthecontrolgrouphasadiseaseorhascomeinforanddoesnotreceivetreatment.
NonexperimentalandQualitativeResearch
Asmentionedearlier,ethicalproblemsarenotconfinedtoexperimentalresearch.Forexample,surveyresearchhaspotentialethicalissuesrelatedtocoercingsubjects
toparticipate.Inaddition,certaintypesofinformationobtainedfromsurveyscoulddistressparticipantsandbedetrimentaltothemiftheywereidentifiedbytheir
employersorbyotherpersonswithpower.Socaremustbetaken.Thisissueappliestoqualitativeresearchaswell.Infact,longquotesgatheredinqualitativestudies
maybeidentifiablebecausetheymayincludeuniqueorpersonalinformationrecognizablebyothers.Inthesecases,suchinformationwouldhavetobealteredor
deletedfromtheresearchreport.
AnimalResearch
Thereisaseparatesetofissuesrelatedtoresearchwithnonhumananimals.Becausethisbookdealsalmostexclusivelywithhumanresearch,weonlydiscussanimal
researchherebriefly.ItisimportanttonotethattheNationalInstitutesofHealthhaspublishedinformationaboutappropriateuseofanimalsinresearch,andmost
universitieshaveaseparateIRBtoconsideruseofanimalsinresearch.Principlesinanimalresearchinvolve:thetrainingofthepersonnelconductingtheresearchand
handlingtheanimals,thenatureoftheresearchandprocedures,thefacilitiesusedtofeedandhousetheanimals,themethodsusedtotransportthem,andthe
justificationforthenumberandspeciestobeused.Clearly,experimentsshouldbeconductedinsuchawayastoavoidallunnecessarysufferingandharmtothe
animals.

Page351

SelectorDeveloptheInstruments
Asdiscussedinthelastchapter,itisnecessaryforavalidandethicalstudytohavehighqualitydatacollectioninstruments.Therefore,selectingordeveloping
instrumentswithstrongevidenceofreliabilityandvaliditydataisbothapracticalandanethicalissue.Ingeneral,aninexperiencedresearchershouldusealready
developed,standardizedinstrumentswheneverthereareappropriateinstrumentsavailable.Rememberthatreliabilityandvaliditydonotresideintheinstruments
themselves,butintheiruseforcertainpurposeswithcertaintypesofparticipants.Itmaybenecessarytoquestionwhetheracommonlyusedinstrumentisappropriate
ifyourpopulationisanunusualorvulnerableone.Evenwellestablishedinstrumentsshouldbepilottestedtobesurethattheinstrumentisappropriateanddoesnot
raiseethicalissuesaboutprivacy,forexample.
PlantheProcedureforDataCollection
IRBsaresensitivetoissuessurroundingtheprocedurethatisusedfordatacollection.Wehavealreadydiscussedtheissueofdeceptionitshouldbefullyexplained
andjustifiedifitisnecessary.Theproceduresectionofaproposalorahumanresearchprotocolshouldalsodescribetheproceduresthatwillbeusedtoobtain
consentfromtheparticipants.TheIRBwillwanttoseerecruitingmaterialssuchasflyersoradvertisementsandalsocoverletterstobeusedaspartofquestionnaire
research.Iftheparticipantsareseenfacetofaceoronthetelephone,theIRBmaywanttoseeacopyofthescriptthatyouplantousetointroducetheprojectand
informparticipantsabouttheprocedures.
Whenplanningtheproceduresfordatacollection,considertokensofappreciationsuchasatoyforchildrenorapenciloradollarforotherparticipants.Such
inducementsaredesignedtoincreasetheresponserate,asarepaymentstoparticipants.
Confidentiality
Animportantpartoftheprocedureistoplanforensuringconfidentialityforeachparticipant.Thisisatwopartissue:a)onlythoseontheresearchteamcanmatchthe
participants'identitieswiththeirresponse,ifthatisnecessary,andb)theidentityofspecificparticipants,ifknown,isnotrevealed.Thisproscriptionappliesnotjustto
writtenreportsbutalsomeansthattheteamwillnottalkaboutspecificparticipantsinpublic,forexample,intherestroom,lunchroom,orhall.Focusgroupspose
specialconfidentialityproblemsbecauseeventhoughtheresearchercautionsotherparticipantsinthegroupaboutconfidentiality,theymaynotheedit.
Confidentialityalsomaybeimportantforthegroups(e.g.,school,hospital,company)fromwhichthesampleisdrawn.Itiscommonpracticeandoftennecessarythat
theidentityofsuchgroupsbedisguisedinareport.Infact,someNativeAmericantribesrequirethattheyonlybereferredtobygeneralgeographicregiontoavoid
stigmatizingtribalmembers.
Usuallytheissueofconfidentialityariseswhentheresearcherisawareoftheparticipants'identitiesandhasagreedtokeepthemconfidential.Certainprocedures
eliminateorminimizethelinkbetweenidentifiersandthedataand,

Page352

therefore,helptoassureconfidentiality.Forexample,onecanassurethatparticipants'namesarenotputontranscriptionsofaudiotaperecordings,questionnaires,or
dataforms.Participantscanbeidentifiedbyacode(nottheirsocialsecuritynumbers)thatiskeptlockedinadifferentplacefromthedata.Ifvignettesorother
descriptionsareprovidedinawriteup,characteristicssuchasoccupation,city,ethnicbackground,andsoonshouldbechanged.Audioorvideotapesshouldbe
storedinalockedplaceandonlyviewedinplacesthatprovideprivacyfromunintendedvisitors.Tapesandmasterlistsofnamescanbedestroyedafterthereporthas
beenacceptedforpublicationorthethesisapproved.Themethodsusedtopreserveconfidentialityshouldbeidentifiedintheconsentprocesssothattheprospective
participantscanbeassuredthatinformationwillbekeptconfidential.
Incaseswheretheresearchdataareanonymoustotheinvestigators,theissueisdifferent.Forexample,ifdemographicorotherpotentiallyidentifyingdataare
obtainedfromananonymoussurvey,theresearcherneedstobecarefulthatresultsarenotpresentedinawaythatsomeonefamiliarwiththeinstitutionsfromwhich
participantsweredrawnwouldbeabletodeducetheidentityofparticipants.Forexample,ifacompanyhadonlyoneorafewminorityworkers,theconfidentialityof
theirresponseswouldbejeopardizediftheaverageoftheirresponseswaspresentedinareport.Ensuringthatthereportdoesnotunintentionallyrevealidentitiesis,of
course,importantinallresearch.
ObtainApprovalfromInstitutionalReviewBoard
IRBsandHowTheyWork
Aninstitutionalreviewboardorhumansubjectscommitteeisacommitteethatreviewsproposalsforstudieswithhumanparticipantsbeforetheresearchcanbegin.
Sieber's(1992)book,PlanningEthicallyResponsibleResearch:AGuideForStudentsandInternalReviewBoards,isahelpfulguideonwhichwehavebased
muchofthissection.Thecommitteeismandatedbyfederalregulationstoprotecthumansubjectsandtodecidewhethertheresearchplanhasadequatelydealtwith
ethicalissuesrelatedtotheproject.IRBsweretheresultofthekindsofethicalproblemsthatwementionedinchapter3.Theyconsistoffiveormorememberswho
havevaryingbackgroundstheyincludemembersofthebroadercommunityaswellasscholarsfromavarietyofareaswithintheuniversityorresearchinstitution.The
committeemeetsperiodically,oftenmonthly,toreviewresearchprotocolsforprojectsproposedbyscholarsandstudentsattheinstitution.
Allresearchattheinstitutionthatsystematicallycollectsdataandisintendedtodevelopgeneralizableknowledgemustbereviewed,unlessitmeetstheexemption
criteriathatsomeinstitutionsallow.Inpractice,thismeansthatanyresearchprojectthatisintendedtobepublishedinajournalorbook,or,asadissertationorthesis
mustbereviewed.Datagatheredforadministrativepurposesandclassroomdemonstrationsarenotreviewed.Manyinstitutionsdonotreviewresearchdonefor
coursesthatisnotintendedtobepublished,buttheinstructorandstudentshouldfollowtheethicalprincipalsdescribedinthisandchapter3.Thegovernmentalso
allowscertaintypesofresearch,forexample,anonymousquestionnairesonnoncontroversialtopicsandresearchdealingwith

Page353

methodsofinstructioninschools,tobeexempt.However,manyuniversityIRBsrequirethatallresearchbesubmittedtothemandthentheydecidewhetheritwillbe
exempt.Often,exemptstatusonlymeansthatthereisalessintensivereview,whichwillnothavetowaituntilthenextcommitteemeeting,buttheresearchprotocol
andperiodicreportsonprogressmayberequired.
Usuallypilottesting,whichinvolvestryingoutproceduresorfinetuningaquestionnairewithafewacquaintancesorknowledgeablepersonsinthefielddoesnot
requireIRBreview.However,pilotstudiesinwhichformaldataarecollectedandanalyzeddorequireIRBreview.
IRBshavebeencontroversialwithsomeresearcherswhoviewthemasobstaclestogoodscientificresearch.Thisispartlyduetopressurestomeetdeadlines,which
canleadtomiscommunicationandmisunderstandings.ItalsomightbeduetoanIRBthatdoesnotprovidehelpfulassistancetoresearchers.Thefederalregulations
requireinstitutionstodeveloppoliciesinkeepingwiththeregulations,butwhichreflectcommunitystandards.Thus,itislikelythateachinstitutionmayhavedifferent
policiesandcouldmakedifferentdecisionsaboutthesameprotocol.Forthesereasons,itisdesirableforstudentstodiscusstheirresearchwithknowledgeable
peopleattheirinstitution,suchasexpertsinthecontentareaorotherresearchers,tobeawareofpotentialethicalissues.Itisalsodesirabletotalkwithpeopleabout
theproceduresofthelocalIRBandwhetherfeedbackcanbeobtainedinadvance.Thismaysaveconsiderabletimeandfrustration.Studentsshouldbeawareof
policiesandproceduresoftheIRBattheiruniversity.Oftentheadministratoroftheboardiswillingtodiscussaprojectwiththeresearcherbeforetheprotocolis
submitted.
TheResearchProtocol
Theresearchprotocolisashortversionofyourresearchproposalfocusingontheresearchproblemorobjectives,theparticipants,procedurestobefollowed,risks,
benefits,consentprocedures,andconfidentiality.TheIRBprobablywillprovideadetailedlistofquestionsthattheywanttohaveansweredaspartoftheprotocol.
Usuallybriefbutspecificanswerstothesequestionswillbethetextoftheprotocol.Althoughsomeoftheanswersmaybecondensedversionsofyourproposal,
others(e.g.,statementsofrisksandbenefits)mayhavetobeexpandedfromwhatyouhaveinyourproposal.Inaddition,youwillprobablyhavetoincludeseveralof
thefollowingattachments:
Advertisementsorposters
Telephonescriptsorotherrecruitmentscripts
Consentforms,includingparentalpermissionandchildassent,orcoverlettersifwrittenconsentisnotrequired.MostIRBshaveasampleconsentform,which
indicatesnecessaryandsuggestedwording
Letter(s)ofagreementoranIRBapprovalfromcooperatingorganizations,perhapsontheirletterheadwithoriginalsignatures
Instruments(evidenceofpermissionforusemayberequirediftheinstrumentiscopyrighted)
Debriefingmaterials
Principalinvestigator'srsum
Acopyofthefullresearchproposaloratleastthemethodsection

Page354

TheprotocolandattachmentsaresubmittedtotheIRBfortheirconsiderationand,onehopes,approval.Theprotocolshouldremindtheresearcheroftheelements
thatareessentialtoscientificallyandethicallysoundresearch.
Institutionsarelegallyresponsibleforresearchconductedbyfacultyandstudents,andsoaretheresearchersandadvisors.Thustheprotocolmustreflectwhatis
actuallydoneintheresearch.Iftheresearcherdecidestochangetheprocedureortheinstruments,approvalmustbeobtainedfromtheIRB.
Inadditiontoacompletediscussionoftherisksandbenefits,includinginducements,andananalysisoftherisk/benefitratio,thereshouldbeacompletediscussionof
thecharacteristicsoftheparticipantsandtheconsentandconfidentialityprocedures.Intermsoftheparticipants,informationabouttheirethnicbackground,gender,
age,andstateofhealthshouldbegivenand,ifvulnerablepopulationsareincluded,theiruseshouldbejustified.Ifcooperatingorganizationsorinstitutionsareusedto
obtainparticipants,writtenapprovalmustbeobtained.Itisdesirabletoprovidearationaleforthenumberofparticipantstobeincludedbyusingananalysisofpower,
asdiscussedinchapter23.
Theconsentproceduresandmethodsusedtoassureconfidentialityneedtobedescribedintheprotocol.Theproceduresshouldindicatehow,where,andbywhom
informedconsentwillbenegotiatedandhowdebriefingwillbeconducted.Theactualconsentformshouldbeattachedtotheprotocol.Ifconsentisimpliedby
returningamailedquestionnaireoritisverbal,asinthecaseofatelephoneinterview,thecoverletterorscriptdetailingtheproceduresmustbeprovided.
PotentialProblemswithResearchProtocols
Sieber(1992)discussesanumberofproblemsthatIRBshaveencountered,andsheprovidestipsonhowtoovercomethem.Sometimesstudentsorinexperienced
researcherswillnothaveadequatehelpinpreparingtheprotocol.Ifthatisyourcase,youshouldconsultotherexperiencedresearchers,ortheIRBadministrator,or
both.Watchfortrainingclassesthattheinstitutionmightprovide,andcheckotherinformationalresourcessuchastheIRBWebpage.
Someprotocolsdevotemuchspacetotheimportanceoftheresearchbutfailtodescribethemethodsandproceduresinenoughdetailorspecificity.Forexample,the
samplingprocedureneedstobeoutlinedclearly,asdoestheresearchdesignandthelocationoftheresearch.
SomeresearchersplaydownorignorerisksthattheIRBmayidentify.Inadditiontophysicalrisks,therecanberiskstoemployment,advancement,reputation,and
financialstanding.Emotionaldistressalsocanbeasignificantrisk.Researchersneedtobeclearthattheyaresensitivetotheissuesofcoercionandwhatiscalled
dualrolerelationships,whenaresearcherisalsotheteacherorsupervisorofthepotentialparticipants.Theresearcher,whoseintentionistohelppersonswitha
problemorhandicapbyusinganintervention,alsoneedstobesensitivetothepossibilitythatidentifyingthemasparticipantsintheinterventionmayinfactstigmatize
them.Everyeffortshouldbemadetobesensitivetothissortofsituationandtoinsuretheprivacyofsuchindividuals.

Page355

Collect,Analyze,andInterprettheData
Aswiththeotherstepsintheresearchprocess,anumberofethicalissuesariseduringthedatacollectionandanalysisphasesofresearch.Someoftheminvolvethe
treatmentofparticipantsandhavealreadybeendiscussed,forexample,sensitivitytoparticipants'privacyconcerns,confidentiality,anddebriefing.Anothersetof
ethicalissueshastodowiththeintegrityofthedatacollection,recording,andanalysis.Weturnnowtotheseissues.
IntegrityoftheData
Itshouldbeobvious,asstatedinthePublicationManualoftheAmericanPsychologicalAssociation(APA,1994),thatresearchers"donotfabricatedataor
falsifyresultsintheirpublications"(p.293).And,ifresearchersdiscoversignificanterrorsintheirpublisheddata,theycorrectthem.Unfortunately,suchscientific
misconducthasoccurredtoooften.AltmanandHernon(1997)describeover60publiclydiscussedcasesofpublicationsthatinvolvedfabricated,falsified,or
plagiarizeddata.AltmanandHernonstatethattheydiscussonlyafractionofthecasesinwhichscientificmisconductwasdetermined.Theynotethatalthough
medicinehasmostcases,theproblemisspreadacrossmanydisciplines,includingpsychology,history,andchemistry,andthatwholeissuesofjournalsinsociology,
business,andmedicinehavebeendevotedtomisconductandprofessionalethics.
Fabrication(makingupthedataorresults)andfalsification(changingdataorresults)areclearlyunacceptablebut,onehopes,relativelyrare.However,thereareother
behaviorsthatmayresultfromcarelessness,bias,oranunwisedecisionthesebehaviorscauseproblemsfortheintegrityofthedataandtheinferencesthatcanbe
made.Someerrorinobserving,recording,andenteringdatamaybeaninevitablebyproductofusinghumans(versuselectronicrecordingdevices)intheseroles,but
goodresearchminimizessucherrors.Carefultrainingofobserversandotherassistantscanhelp.Checkingdatatobesuretheyarerecordedandenteredcorrectlycan
helpas,insomecases,cantheuseofcomputerstoreducepossibleerrorsintranscribingdata.Thus,carefulnessisasimportantashonestyifthecollecteddataareto
bemeaningful.
Qualitativeresearchershaveargued,asdiscussedinchapter2,Axiom5,thatinquiryisalwaysvalueladenandisnevercompletelyobjective.Thus,theperspectives
thatonebringstotheresearchareboundtoinfluencenotonlytheselectionoftheproblems,variables,andmethodsused,butalsothecodingorcategorizationofthe
dataandhowtheyareinterpreted.Althoughthisistrue,muchcanandshouldbedonetominimizetheeffectsoftheresearcher'sbiases.First,onecanacknowledge
biasesandtrytofigureouthowtheymightinfluencethedatacollection,coding,andanalysis.Checkingthereliabilityofcodingisdesirablebutnotenough,especiallyif
theothercodershavesimilarbiasesorweretrainedbytheresearcher.Again,caretominimizetheeffectsofavoidablebiasmaybeasimportantashonestyforgood
research.
Therearemanychoicestobemadeinconductingresearch.Financialandotherconstraintsresultinnecessarychoicesthatweakenastudyinsomewayand
strengthenitinotherways.Forexample,thereisalmostalwaysatradeoffbetweeninternalandexternalvaliditythatis,goodcontroltendstomake

Page356

thingsartificial.However,researchersalsomakeunnecessarybadchoices,someofwhichareethicallyquestionable.Meltzoff(1997)providesseveralexamplesof
baddecisionsthatmightoccurduringthedatacollectionphaseofthestudy.Oneexampleistheinvestigatorwhoeliminatesparticipantsfromthestudyforunexplained
reasons.Aswediscussedearlier,participantsmustbegivenpermissiontowithdrawatanytime,sothatitisavalidreasonforthedatatobeexcluded.But,if
participantsdonotperformintheexpectedmanner,thatisnotavalidreason.Anotherexamplewouldbechangingthelengthofthestudyoroverrulingsupposedly
independentraters.Anysuchchangesneedtobejustifiedandshouldnotbecausedbythedataturningoutthe"wrong"way.
IntegrityofDataAnalysis
Aportionofthisbookhasbeenaboutappropriateuseandinterpretationofstatistics.Itisimportanttopointoutthattherearemanylegitimatedisagreementsamong
statisticiansandresearchers.Wehavenotedsomeofthesedifferencesinearlierchapters.Clearly,alteringthedataordeliberatelyreportinganincorrectpvalueis
unethical.Wealsohavepointedoutanumberofthingsaresearchermightdoinanalysisorinterpretationthatarewrongbutnotunethical,unlessdonetodeceive
deliberately.Forexample,usinganinappropriatestatistic,suchasattest,withathreeormorecategorynominaldependentvariableiswrong.Manyotherchoices
aboutstatisticsarenotthebestpractice,oftenbecausetheresearcherisrelativelyinexperiencedorunknowledgeableaboutstatistics.Forexample,nottestingfor
assumptionscouldleadtothewrongconclusionsiftheassumptionsweremarkedlyviolated.
Inothercases,reportsofdataanalysismayatleastraisesuspicionsofunethicalbehavior.Meltzoff(1997)providesseveralexamples.Oneisthecaseinwhich
participantsseemtobedividedarbitrarilyafterthefactintogroups(suchashighandlow)whentherewasacontinuousindependentvariable.Didtheresearchertry
manycutoffpointsuntilfindingonethatwasstatisticallysignificant?Thisconcernisonereasonwerecommendusingacorrelationwhentheindependentvariableis
continuousorhasmanyorderedcategories.
Manystatisticiansthinkthatnullhypothesissignificancetesting(NHST)isonlyappropriatewhentheresearcherhasoneorafewwellthoughtouthypothesestotest.
Theyareskepticalofastudywithmanysignificancetests.However,mostwouldsupportexploratorydataanalysis,withoutNHST(seeTukey,1977).
InterpretationoftheResults
Authorsneedtobecareful,inwritingtheirresultsanddiscussionsections,toavoiddistortingthefindingsortheirimplications.Forexample,inearlierchapterswe
discussedthemistakeofinferringcausationfromcomparative,associational,orevenquasiexperimentalstudies.Therefore,careshouldbetakennottostatethatthe
independentvariable"caused,""determined,"or''impacted"thedependentvariableunlessthestudywasawellcontrolledrandomizedexperiment.Indiscussingresults
fromnonexperimentalstudies,thesetermsshouldnotbeusedorshouldonlybeusedwithqualifierssuchas"maycause,"or"appeartoinfluence."Likewise,one
shouldbecarefulaboutgeneralizationstobroaderpopulationsfromsamplesthatmaybeunrepresentativeofthepopulation.

Page357

DisseminatetheResults
Publishingisanessentialpartoftheresearchprocess,whichisnotcompleteuntiltheresultsaredisseminatedtootherinterestedresearchers.Publishingisthemeans
bywhichtheresultsaremadepublic.Considerabledetailisprovidedinthepublicationabouttheproceduresanddataanalyses.Thismakestheresearcher'swork
availableforscrutinybythescholarlycommunity.Thispublicscrutinyhelpsmakescienceobjectiveandisonereasonwhypublishingissoimportant.Publicationsare
alsoanimportantfactorinthecareerofaresearcher.Theyareusedtoevaluatethecapabilityandperformanceoftheresearcherorprofessor,andareanimportant
aspectoftenureandpromotionatauniversity.Therefore,thereisconsiderablepressure,especiallyonyoungfaculty,whichleadstoanumberofpotentialethical
problems.
Plagiarism
Plagiarismispresentingasubstantialportionoftheworkofanotherasifitwereone'sown.Paraphrasing,whichinvolvessummarizingandrearrangingsentences,is
acceptableifcreditisgiveninthetext.Plagiarismrefersnotjusttowordsbutalsototheideasanddataofanotherperson.Becauseliteraturereviewsandtextbooks
arebasedheavilyontheworkofothers,itisdifficulttoprovideappropriatecreditwithoutoverusingquotations.Also,itisoftenhardtoknowexactlywhichthoughts
camefromwhichsource.InexperiencedwritersandthoseforwhomEnglishisnottheirfirstlanguagefinditespeciallychallengingtoparaphrasebecausetheoriginal
authormayappeartohavewrittenthingsinthe"best"way.
MultiplePublications
TheAmericanPsychologicalAssociation'sethicalprinciples(APA,1992)statesthatresearchers"donotpublish,asoriginaldata,datathathavebeenpreviously
published.Thisdoesnotprecluderepublishingdatawhentheyareaccompaniedbyproperacknowledgment"APA,1994(p.293).TheAPApublicationmanual
(APA,1994)goesontosaythatduplicatepublicationdistortstheknowledgebaseandwastesscarceresources,thatis,journalpages.Authorsmustnotsubmittoa
journalamanuscriptthathasalreadybeenpublishedinsubstantiallythesameform.ManuscriptspreviouslyincludedintheERIC(EducationResourcesInformation
Centers)system,publishedasanabstractorsummary,orcirculatedasauniversityorlimitedcirculationdocumentcanbepublishedinfulllater.However,manuscripts
publishedinfullinproceedingsorinbookchaptersshouldnotbepublishedlaterinajournal.Thereisalwaysanissueabouthowsimilarthecurrentmanuscriptisto
theoriginalandthesimilarityoftheaudience.Itisnotuncommon,butperhapsethicallyquestionable,forresearcherstorewriteapaperforajournalwithadifferent
audience.Journalarticlesaresometimesrevisedforpublicationasachapterinabook.Thisisacceptableaslongastheoriginalsourceiscitedandpermissionto
adaptorreprintisobtainedfromthecopyrightholder.ProblemsofduplicatepublicationalsomayariseifthematerialisfirstpublishedontheInternetorthroughthe
massmedia.
Articlesmustnotbesubmittedtomorethanonejournalatatime.Onlyafterrejectionorwithdrawalofthemanuscriptisitappropriatetosubmitthesamearticleto
anotherjournal.TheAPApublicationmanual(APA,1994)states:

Page358
theauthormustinformaneditoroftheexistenceofanysimilarmanuscriptsthathavealreadybeenpublishedoracceptedforpublicationorthatmaybesubmittedforconcurrent
considerationtothesamejournalorelsewhere.Theeditorcanthenmakeaninformedjudgmentastowhetherthesubmittedmanuscriptincludessufficientnewinformationto
warrantconsideration.(p.296)

Itiscommon,butinsomewaysundesirable,forseveralsubstantivelydifferentarticlestobepublishedfromthesamedissertationorlargestudy.Pressuresonauthors
tohavealargenumberofpublicationsandlimitationsbyeditorsonspaceoftenleadtomultiplepublicationsfromonestudy.
Authorship
Therehasbeenconsiderablediscussioninrecentyearsaboutwhoshouldbelistedasanauthorandevenwhetherthewholeconceptofauthorshipshouldbescrapped
infavorofsomeothersystem.Forexample,Rennie,Yank,andEmanuel(1997)haveproposedthatinsteadofauthorseacharticleprovidealistofcontributors
indicatingtheirspecificcontribution(s),forexample,designedthestatisticalanalyses,conceptualizedthedesign,wrotetheresultsanddiscussion.Partofthereasonfor
thisproposalistoidentifyresponsibilityoraccountabilityforpartsofthearticle.Anotherreasonisthattherehavebeendisagreementsaboutwhoshouldbeanauthor
andinwhatorder.
Ageneral,butnotuniversally,agreedonpolicyisthatauthorshipisreservedforthosewhomakeasubstantialprofessionalcontributiontothestudy,andthatorderof
authorshipshouldbedeterminedbytheimportanceoftheircontribution.TheAPApublicationmanual(APA,1994)statesthat"substantialprofessionalcontributions
mayincludeformulatingtheproblemorhypothesis,structuringtheexperimentaldesign,organizingandconductingthestatisticalanalysis,interpretingtheresults,or
writingamajorportionofthepaper"(p.294).Themanualgoesontosaythatlessercontributions,whichmaybeacknowledged,includesupportingfunctionssuchas
designingtheapparatus,conductingthestatisticalanalysis,collectingorenteringdata,andrecruitingparticipants.Notethattheselattercontributionsareoftenthoseof
undergraduateorgraduatestudentvolunteersorpaidassistants,whomaythinkthattheydeserveauthorship.
Twotypesofproblemsresultwhendeterminingauthorship.Ontheonehandthereare"guest"authorssuchaslabdirectorsorcolleagueswho"need"another
publication.Theyarepersonswhohavenotmadeasignificantprofessionalcontributiontotheprojectbutaregivenauthorshipasafavororasa"right"asaresultof
theirstatusinadepartmentorlaboratory,orbecausewiththeirnamesonanarticletheprobabilityofacceptanceisincreased.Ontheotherhand,thereare"ghost"
authorswhodidmakeasignificantprofessionalcontributionbutwerenotincludedasauthors.Theseincludejuniorresearchersandstudents.Sometimespersonsin
powersimplytakeadvantageoflesspowerfulordepartedcolleaguesorstudents.However,theissuesarenotalwaysclear.Oftendifficultiesarisewhenaperson
losesinterestorleavestheareaafterplayinganimportantpartintheinitialaspectsofthestudy.Perhapsthepersonevenwroteathesisthat,inretrospect,turnsoutto
beanearlydraftofthe

Page359

finalacceptedpaper.Theissueiswhatkindofcreditshouldbegiventosuchapersonwhenanarticleisrejected,reanalyzed,andfullyrewrittenwithouttheassistance
oftheinitialcontributor.
FineandKurdek(1993)presentanumberofexamplesofissuesthatcanarisewhenfacultyandstudentscollaborateonresearch.Thesituationisfrequentlysimilarto
theexampleinthepreviousparagraphthatis,thestudent'sthesisordraftarticleisnotadequateforpublication,sothefacultymembermustreviseitextensively.In
general,wethinkthatifanarticleisbasedonastudent'sthesisordissertation,thestudentshoulddefinitelybeanauthor,evenifheorshedoesnotparticipateinthe
revisions.1Inmostcases,wethinkthestudentshouldbethefirstauthor.However,FineandKurdekprovideseveralexamplesinwhichtheythinkitmaynotbe
appropriateforstudentstobethefirstauthorbecausetheylostinterestintheprojectanditwasuptotheadvisortoreanalyzeandrewritethepaperortocollectnew
datatomakeitpublishable.Somescholarsthinkthatnottakingtheleadinwritingupadissertationforpublicationisindicativeofpoorscholarshiponthepartofthe
studentandthepapershouldnotbepublished.
Anotherissueiswhetherthefacultyadvisorshouldbecoauthoronapublicationfromastudent'sdissertationorthesis.Theanswer,itseemstous,isnotunlessthe
advisormadeasignificantcontributiontothedesignofthestudyortothewritingofthefinalarticle.Readingandprovidingextensivefeedbackonathesisor
dissertationiswhatisexpectedofafacultymemberandisnotsufficientforauthorship.
Agoodpracticeisforthecollaboratorstomeetatthebeginningoftheprojectandagreeonwhoshouldbeauthorsandtheorderofauthorship.Itisalsonecessary
fortheseauthorstokeepincontactandtorenegotiateauthorshipifthecircumstanceschangesignificantly.Eachperson'scontributionshouldbedocumentedand
updatedasnecessary(seeRennieetal.,1997).
Finally,therearetwootherissuesrelatedtoauthorship.First,theirconsentshouldalwaysbeobtainedbeforeincludingsomeoneasanauthoror,accordingtothe
APApublicationmanual(1994),evenbeforeincludingtheminanoteofacknowledgment.Asageneralrule,thisrecommendationseemsextremetous,butifitis
statedorimpliedthattheacknowledgedpersonagreeswithorsupportstheconclusions,theirconsentshouldsurelybeobtained.Second,allauthorsshouldreviewthe
manuscriptbeforeitissubmittedbecausetheirnamesasauthorsimplythattheytakeresponsibilityforthearticle.Someeditorsrequirethat,whenapaperisaccepted,
eachauthormustsignaformacceptingresponsibility.However,withmultipleauthoredarticlesitisprobablyunrealistictoassumethatallauthorsareknowledgeable
andshouldberesponsibleforallaspectsofthepaper.ThiswasoneofthereasonsfortheproposalbyRennieetal.(1997)tolistspecificcontributionsratherthan
authors.
CitingPublicationsinYourRsum
Studentssometimesaskwhenandhowtociteresearchpapersintheircurriculumvitae(C.V.)orrsum.Thisisanim
1

NotethattheAPApublicationmanual(APA,1994)statesthatallauthorsshouldreadandapprovethefinalmanuscriptandacceptresponsibilityforit.Thiscouldbedifficultif
theprofessorhaslostcontactwiththestudent.

Page360

portantissuebecauseincompletecitationscanleadtoconcernsaboutsophisticationandinaccuratecitationscanleadtoaccusationsofmisconduct.Fortheexact
formatofcitations,youshouldconsultthepublicationmanualusedinyourdiscipline(e.g.,APA,1994).Ifthereisn'tsuchamanual,theformatusedbyjournalsin
yourfieldisagoodmodeltoemulate.Inallfieldsitisimportanttolistalloftheauthorsintheorderthattheyappearinthepublication,thedateofpublication,the
exacttitleofthearticle,thepublicationtitle(ifthearticleappearsinabookorjournal),thevolume,ifany,andthepagenumbers.
OtherPracticalandEthicalIssues
ReviewsandReviewers
Mostgrantproposalsandproposedjournalarticlesarereviewedbypeerswhomustbecarefulnottousetheideasoftheoriginalauthorsuntiltheyarepublishedand
thengivethemcredit.Editorsandreviewersmustnotquoteproposalstheyhavereviewedunlessgivenexplicitpermissionbytheauthor.
Theprocessrequiresagooddealoftrustandintegritybythereviewersforittoworkfairlyandnotbeexploitative.Problemsrelatedtofairnessofreviewsare
relativelycommonandmostfundingagenciesandjournalshavespecificpoliciestodealwiththem.Usuallyreviewers'identitiesarenotrevealedtotheauthorsonthe
assumptionthatthiswillmakereviewsmorecandidandnegativereviewslessopentoreprisal.However,ontheotherhand,othershavearguedthatreviewsmightbe
moreresponsibleandmeasurediftheidentityofthereviewerwasknown.Infact,insmallfields,applicantscanoftenguesstheidentityofthereviewer.Amasked,
formerlycalledblind,reviewoccurswhentheauthor'sidentityisnotgiventothereviewer.Informationabouttheauthorsisremovedfromthemanuscript.Thistypeof
reviewiscommonformanuscriptsbutisunusualforgrantproposals.Theargumentforanonymousormaskedreviewisthatitgivesabetterchancetoanewscholar
becausetheworkisjudgedsolelyonitsmeritsratherthanonthestatusoftheauthors.Again,insmallfields,itmaynotbepossibletodisguisethemanuscriptsofwell
knownresearchers.
Onceanarticleorbookispublished,adifferentkindofreviewtakesplace,notjustinpublishedbookreviewsbutalsoinliteraturereviewsandmetaanalyses(see
chap.23).Inmetaanalyses,thereviewersoftenexcludestudiesjudgednottobeofhighquality,eventhoughpublished.Orthereviewermaydecidetoweightstudies
intermsoftheirmerit,sosomecountmorethanothers.Althoughthesepracticesareanecessarypartofthescientificprocess,theyprovidetheopportunityfor
potentialabuseand,attheleast,hurtfeelings.
ConflictsofInterest
Althoughscholarsdotheirresearchforavarietyofreasons,includingcuriosityandaltruism,otherfactors,suchasfame,fortune,andtenurearealsomotivatorsfor
doingresearch.Aproblemoccurswhenthereisarealorapparentconflictbetweenpersonalgainandobligationstotheuniversityorscientificcommunity.Onetypeof
conflictisrelatedtocompetitionamongscholars.

Page361

Thiscouldleadtoreviewerstreatingtheircompetitorsunfairlyortothewithholdingofinformationfromcolleagues.Becauseoriginalityandpriorityaresoimportant,
thereisoftenaninherentconflictofinterestthatmayrestrictcollaborationandcooperation.Ontheotherhand,itisusuallyconsideredaconflictofinteresttoreview
grantsorpapersfromclosecolleaguesorpersonsfromthesameinstitutionbecauseofpotentialloyalty.Conflictsofinterestareofconcerninthewritingstageaswell
asinearlierstagesoftheresearchandshouldbeacknowledged.Forexample,ifresearchontheeffectsorvalueofaproductisfundedbytheproducerofthat
product,thefundingshouldbeacknowledgedinthenotestothearticle.
Conflictsofinterestarenotthesameasmisconductbutthelattercanresultfromunacknowledgedconflictsofinterest,whichneedtoberecognizedanddisclosed.
Theresearcherneedstobalancetheprioritiesofcollaborationandthesearchfortruthwithpotentialconflictsfrompersonalgain,loyalty,andfunding.Conflictsof
interestareinevitableandnotinherentlybad,butnotdisclosingthemisaproblem.Eventheappearanceofconflictsshouldbedisclosed.
MisconductandtheStructureofScience
Inacontroversialarticle,WoodwardandGoodstein(1996),professorsofphilosophyandphysics,maketheargumentthat"manyplausiblesoundingrulesfordefining
ethicalconductmightbedestructivetotheaimsofscientificinquiry"(p.479).Examplesofthetypesofplausibleethicalprinciplesorrulesthattheyargueshouldnot
governthebehaviorofscientistsareasfollows:scientistsshouldneverbemotivatedbypersonalgain,neverbelievedogmaticallyinanideanoruseexaggerationin
promotingit,andscientistsmustleanoverbackwardstopointoutevidencecontrarytotheirhypothesis.Theyaskthequestionofhowfraudcanbereducedwithout
losingthepositiveeffectsofcompetitionandreward.
WoodwardandGoodstein(1996)arguethatacertainamountofexaggerationofthevalueofone'sapproachandneglectingorplayingdownofcontraryevidence
maybenecessary,especiallyintheearlystagesofaproject.Thisiswhatpsychologistscallbeliefperseverance.Giventhis,WoodwardandGoodsteinsaythat"an
implicitcodeofconductthatencouragesscientiststobeabitdogmaticandpermitsacertainmeasureofexaggeration...andthatdoesnotrequireanexhaustive
discussionofitsdeficienciesmaybeperfectlysensible"(p.485).Theyarguethatpartoftheresponsibilityofscientistsistoprovidethebestpossiblecasefortheir
ideas.Itisuptootherstopointoutdefectsandlimitations.WoodwardandGoodsteinstatethatthisis,infact,whatmostscientistsdo.Thereare,ofcourse,reallimits
here,and"exaggeration"isprobablynotthebestword.Advocacyisappropriate,butanyfactualmisstatementisunethical.
Thetypicalresearchpaperlooksasifthewritershadamuchclearerconceptualizationoftheproblemandresultsthanisactuallythecase.Variousblindalleysusually
arenotinthefinalpublication.Althoughthismayappeartobedeceptivetotheuninitiatedandisclearlynotinagreementwiththeprincipleofleaningoverbackwards
topointoutcontraryevidence,"Nevertheless,the

Page362

practiceisvirtuallyuniversal,becauseitisamuchmoreefficientmeansoftransmittingresultsthananaccuratehistoricalaccount"(Woodward&Goodstein,1996,p.
488).Again,althougharesearchreportdoesnothavetobehistoricallyaccurate,itmustnotdistorttheresultsbyleavingoutimportantnegativeresultsormisstating
whatwasfound.
WoodwardandGoodstein(1996)concludebystatingthattheythinkeachofthe15reasonablesoundingprinciplesorrulestheylistedisdefectiveasauniversalrule
ofscientificconduct.However,that"doesnotmeanthatitisimpossibletorecognizedistinctivescientificmisconduct"(p.489).Thereisadifferencebetweenscientific
misconductandothertypesofmisconduct,suchasstealingorplagiarismthatareseriousmisdeeds,whichhaveestablishedproceduresfordealingwiththem.They
arguethatthetypesofmisconductthatarespecificallyscientificmisconductrequirethatapanelofpeersassesstheissue.Sometimestheissueinvolvessteppingover
thelinebetweenadvocacyanddeception.Fabricationandunwarrantedmanipulationofthedataareexamplesofthekindofdeceptionthatcannotbetolerated.
Expertjudgmentbypeersisrequiredtodecidewhattypesofdatamanipulationareunwarranted.Thepointhereisthatwhatmayseemlikesimple,obviousrules
aboutmisconductareoftenlessclearinthespecificcase.
Howisthestudentornewresearchertoknowwhatisacceptableadvocacyandwhatcrossestheline?Wementionedearlierinthesectionsonintegrityofthedata
andintheanalysisafewexamplesthatappeartocrosstheline,aswellasclearexamplesofmisconduct.Peerjudgmentisrequiredtodecidewhetheraresearcher's
proceduresforselectingparticularparticipantsorselectivelydiscardingdataareappropriateorinvolvemisconduct.Juniorresearcherscanlearnaboutthe
complexitiesofappropriatebehaviorintheirfieldbestbyobservinganddiscussingissueswithseniorscholarsormentorsintheirfield.However,youshouldbecareful
whomyouemulatebecause,aspointedoutinchapter3andearlier,notallseniorresearchersaregoodrolemodels.Wehopethatthischapterhasconveyednotonly
thecomplexityoftheissuespresentedbutalsosuggestionsforaction.
Summary
Thischapterextendsthediscussioninchapter3ofethicalproblemsandprinciples.Wemovedstepbystepthroughtheresearchprocess,discussingbothpractical
andethicalissuesthatdealwithsampling,planningthedesignandanalysis,selectinginstruments,planningtheprocedure,obtainingapprovalfromtheIRB,collecting
andanalyzingthedata,andwritingthereport.Inaddition,wediscussedissuesaboutrsumcitations,reviewers,conflictsofinterest,andadvocacyascontrastedto
misconduct.

Page363

StudyAids
Concepts
Confidentiality
Conflictofinterest
Consentforms
Datafabricationandfalsification
Debriefing
Deception
Dropoutsorattrition(alsocalledexperimentalmortality)
Ghostauthors
Guestauthors
IRB(InstitutionalReviewBoard)
Masked(blind)review
Multiplepublications(ofthesamearticle)
Nointervention(orplacebogroup)
Peerreview
Professionalcontribution(toaresearchproject)
Researchprotocol
Responserate
Scientificmisconduct
Distinctions
Confidentialversusanonymous
Pilottestingversuspilotstudies
Plagiarismversusparaphrasing
ApplicationProblems
1.Ifyouwerewritingapaperondatacollectiontechniquesandwantedtoincludethemainideafromthefollowingsentencefromchapter21ofthistext,whataretwo
waysthatyoumightdosoappropriately?
"Theconcernaboutthefilteringofparticipants'answersthroughperhapsfaultymemoriesorintermsofsociallydesirableresponseshasledquantitativeresearchers,
especiallythosewhotendtousetherandomizedexperimentalandquasiexperimentalapproaches,tobesuspiciousaboutthevalidityofselfreportinstruments."(p.
331332)
2.Aresearcherhasworkedforseveralmonthstodesignherquestionnaire,andwantstoobtainfeedbackontheaveragetimeittakestocompletethequestionnaire
andifanyquestionsareunclear.Sheasksseveralofherfriendsandcolleaguestotakethequestionnaire,timethemselves,andcommentonanythingthatseemed
unclear.Whatisthisprocedurecalled?WouldtheresearcherberequiredtoobtainInstitutionalReviewBoard(IRB)approvalbeforegivingthequestionnairetoher
friendsandcolleagues?

Page364

3.Anotherresearcherwantstotryherinterviewquestionswithteenparentswhowillnotbeinherstudytoassesstheeffectivenessofherinterviewquestions.She
recruitsteenparentsfromherlocalcommunitytointerview.Whatisthiscalled?DoesthisproceduresrequireIRBapproval?
4.Aresearcherinterviewedindividualswhohadexperiencedasexualassault.Sheassuredhersubjectsthattheiridentitywouldbekeptconfidential.Howmightshe
dothis?
5.Ifaresearcherwantedtoassureparticipantsthattheywouldbeanonymous,howmightshecollectherdata?
6.Scientistsareinterestedinthecausesofviolentbehavior.Whydosomeindividualswhoappeartohaveexperiencedanormalchildhoodexhibitveryviolent
behavior,withlittleornoremorse?Theresearchershypothesizedinnatebrainphysiologydifferences.Thewardenagreedthatallmaleprisonersfromahighsecurity
prisoninasouthernstatewhohadcommittedaviolentcrimeandwhohadnoevidenceofchildhoodabuseorneglectwouldbeincludedinthestudy.A
demographicallysimilar(age,ethnicity,familybackground,etc.)sampleofmaleswasselectedfromthatstate'spopulationtoserveasacomparisongroup,40%of
themagreedtoparticipate(andwerecompensated$200).Therewere28individualsintheprisonergroupand30inthenoncriminalcommunitygroup.Brainscans
weredoneonallparticipantsandthencomparisonsweremadebetweenthetwogroups.
a.Discusstheethicalissuesinvolvedintheabovestudy.
b.Discusspopulationvalidityissuesfromtheinformationprovided.
c.Discussissuesofecologicalvalidity.
7.Bobhasjustcompletedamanuscriptforpublication.Althoughhehaddevelopedtheroughoutlinesoftheprojectonhisown,heowesmuchtootherindividuals.
Theassistancehereceivedincludesthefollowing:
AfriendofhisprovidedBobwithadviceonhowtoobtainhissample.
ThedirectorofthestatlabgaveBobadviceandalsoassistedinwritingtheresultssection.
Agraduatestudentcollectedmostofthestructuredinterviewdataanddidthecomputerdataentry.
a.Whatkindofattributionshouldbegiventoeachoftheseindividuals?Forexample,whoshouldberecognizedasanauthorandwhoshouldreceive
anacknowledgmentinthepaper?Whodoesnotmeritformalrecognition?Explain.
b.Atwhatpointintheprocessofone'sresearchshoulddecisionsconcerningauthorshipandacknowledgmentsbemade?
c.AredecisionsconcerningattributionentirelyBob'sresponsibility?Explain.

Page365

PARTVI
RESEARCHVALIDITY,REPLICATION,ANDREVIEW

Page367

Chapter23
ResearchValidityandReplication
TwoMoreDimensionsofResearchValidity
MeasurementReliabilityandStatistics
ReliabilityofMeasures
AppropriatenessofPower
TheConceptofPower
HowCanWeGainPower?
EvaluatingPower
AppropriatenessofStatisticalTechniques
AppropriatenessofInterpretation
MeasurementValidityandGeneralizabilityoftheConstructs
ResearchValidity:Summary
TheRelativeImportanceofDifferentValidityCategories
ResearchReplication:MetaAnalysis
SignificanceTestingImpedestheGoalsofScience
EffectSizeandMetaAnalysis
IssuesWithUsingMetaAnalysis
PerformingaMetaAnalysis
Summary
StudyAids
Concepts
Distinctions
ApplicationProblems

Page368

TwoMoreDimensionsofResearchValidity
Wenowreturntoexaminethelasttwodimensionsofresearchvalidity.Rememberresearchvalidityisthevalidityofawholestudyincontrasttothevalidityofasingle
measureorinstrument,whichiswhatwediscussedinchapter20.Inchapter6wediscussedinternalvalidityandinchapter10wediscussedexternalvalidity,each
withtwomainsubdimensionsandratingscalestouseinevaluatingthevalidityofastudy.
MeasurementReliabilityandStatistics
CookandCampbell(1979)calledthisdimensionstatisticalconclusionvalidity,butwehavemodifiedthenametoemphasizetheimportanceofmeasurementreliability.
Wehaveconcentratedonwhatwebelievearethekeyissues.Wethinkthattherearefourimportantissuesunderlyingthisdimensionthatstudentsmustkeepinmind
whendesigningorevaluatingresearch.Alloftheseissuesshouldbeconsideredwhenmakingtheoverallratingofthisdimensionacontinuumfromlowthroughmedium
tohigh(seeFig.23.1).
ReliabilityofMeasures
Thefirstissueiswhetherthevariables(asagroup)aremeasuredreliably.Ajudgmentismadebasedontheoverallratingofthereliabilityoftheinstruments.A
principleoftenemphasizedinmeasurementclassesisthatatestormeasurecannotbevalidifitisnotreliable.Likewise,astudy'sresearchvalidityisreduced
considerablyifoneormoreofthekeyvariablesisnotmeasuredreliably.Anotherconsiderationiswhetherseveralorallofthetypesofreliabilitywerereported.
Chapter20discussedmeasurementreliabilityindetail.
AppropriatenessofPower
TheConceptofPower
Powerfromastatisticalpointofviewrelatestotheprobabilityofrejectingafalsenullhypothesis.Powerimpliesacorrectdecision,soitshouldbemaximized.Inother
words,ifwesetouralphalevelatthe

Fig.23.1.
Evaluatingthestatisticsandmeasurementreliabilityofthefindingsofastudy.

Page369

conventional.05orthelessconventional.01,whatistheprobabilityofrejectingthenullhypothesis,assumingitisfalse?
Togetabetterfeelforthisconcept,studyFig.23.2(adaptedfromLoftus&Loftus,1982,p.225).Figure23.2showstwonormalcurvesrepresentingthepopulation
distributionifthenullhypothesis(H0)istrueandthepopulationdistributionifthealternativehypothesis(H1)istrue(thenullhypothesisisfalse).Thenormalcurveinthe
rightofthefigure,thepopulationdistributionassumingthatthenullhypothesisistrue,hasan levelestablishedat.05or.01(lefttailofthisdistribution).Remember
that isequaltotheprobabilityofatypeIerror,ortheprobabilityofrejectingatruenullhypothesis.Anyoutcometotheleftofthatvalue(i.e.,atvalueexceeding
thecriticalvalueestablishedpriortothestudy)willresultinrejectingthenullhypothesisandacceptingthealternativehypothesis.Noticeinthefigurethatatthecutoff
pointof =.05,thelineisextendedthroughthealternativehypothesisdistribution(thenormalcurveintheleftofthefigure).Theshadedareaofthealternative
hypothesisdistribution(thenormalcurveontheleft)establishedbythecutoffpointlineat =.05istheprobabilityofatypeIIerror,or (notrejectingthenull
hypothesiswhenitisfalse).Theareaofthedistributionofthealternativehypothesisthatisnotshadedistheprobabilityofrejectingafalsenullhypothesis,orpower
(1 ).Whatmostresearchersreallywanttoknowishowmuchpowerisintheirstudyandhowtoincreasepower.Ideally,powerissetatavalueof.80.1

Fig.23.2.
Relationshipsamong , ,andpower.FromEssenceofStatistics(p.225),byG.R.
LoftusandE.F.Loftus,1982,Monterey,CA:Brooks/Cole.Adaptedwithpermission.
1

Keppel(1991)suggeststhatmostmethodologistsinthebehavioralsciencesappeartoagreeonthislevelofpower,assumingthattypeIerrorsaremoreseriousthantypeII
errors.Avalueof.80wouldyieldatypeIIerrorof.20,whichisfourtimesaslargeasatypeIerrorof.05.

Page370

Wecandeterminehowmuchpower(1 )ispresentinastudybyusingpowerchartsifweknowthesizeofthesample,thesignificancelevel( ),andanestimateof

theeffectsizeofthestudy.Forexample,supposeastudyisproposedtodeterminetheeffectofreformteachingonmathematicsachievementincollegestudents.A
facultymemberhasvolunteeredtoteachtwodifferentsectionsoflinearalgebra.Shewillteachonesectioninareformmannerandtheothersectioninthetraditional
manner.Theclasssizeofthereformsectionis24studentsandtheclasssizeofthetraditionalsectionis28students.Shesetsheralphalevelpriortothestudyat.05
foradirectionalhypothesis.Nowsheknowshersamplesizeandalphalevel.Whatistheestimatedeffectsizeofherstudy?Rememberthatearlierwediscussedtwo
differentindicesofeffectsize.Hereweareinterestedintheeffectsize,d,instandarddeviationunits.Wecouldestimatethiseffectsizebyusingthefollowingformula:

Wewouldhavetoestimatebothpopulationmeansandtheweightedaverageofthepopulationstandarddeviation.Instead,theinvestigatorcanspecifyaneffectsize
assmall,medium,orlarge.Forexample,asmalleffectsizewouldbearound.2standarddeviationsbetweentheinterventionandcontrolmeans,amediumeffectsize
wouldbearound.5standarddeviationsbetweentheinterventionandcontrolmeans,andalargeeffectsizewouldbeabout.8standarddeviationsbetweenthe
interventionandcontrolmeans.Howdoestheresearcherknowwhateffectsizetoestimate?Thebestestimatewouldbefrompreviousstudiesinthisarea.Ifprevious
studiesinvestigatingtherelationshipbetweenreformteachingandachievementdemonstratemediumeffectsizes,thenanestimateofeffectsizeforherstudymight
be.5.Nowthattheresearcherknowssamplesize,estimatedeffectsize,andsignificancelevelforherstudy,shecanuseFig.23.3todetermineherpower(from
Lipsey,1990).Noticethatthisparticularpowerchart(Fig.23.3)isforanindependentttestwithanaof.05twotailed(nondirectionalhypothesis).OntheXaxisof
Fig.23.3arethesamplesizesforeachgroup.Becauseourtwogroupsarenotequal,wewilltaketheaverageofthetwo,whichis26.Theclosestvalueto26onthe
tableis25.Wewillusethatasanestimateofoursamplesizeineachgroup.Oneachofthecurvesisaneffectsize(ES).Welocatetheeffectsizecurveof.5,the
estimateofourpredictedeffectsize.Wefindwheretheeffectsizecurveof.5crossesthesamplesizelineof25.Atthispoint,wereadthepowerontheYaxis.Inour
example,thepowerwouldbeabout.40.Thisislessthanidealpower.Howcantheresearcherincreasepowerinherstudy?
HowCanWeGainPower?
Toincreasepowerinourresearchsituation,weneedtodecidewhichthingswecancontrolandwhichwecannotcontrol.Controlinthissituationisrelativeratherthan
dichotomous.Ourorderofpresentationofpowerisfromwhatisusuallyleastcontrollabletomostcontrollable.
Ourfirstelementingainingpoweris .InFig.23.2itcanbeseenthatasalphaissetatalowersignificancelevel,forexample,.01(thecutoffpointis

Page371

movedtotheleft),powerisdecreased.Ontheotherhand,if issetatahigherlevel,forexample,.10,thenpowerisincreased.However,asweincreasealpha,we
alsoincreasetheprobabilityofatypeIerror.Moreimportantly,wereallyshouldnotsetalphaatalevelhigherthan.05.Thereasonforthisismostlyconvention.Few
researchjournalsand,hence,ourowncolleagueswillacceptaresearchpublicationwithalphaestablishedhigherthan.05.
Asecondmethodtoincreasepowerinvolvesformulationofhypotheses.Rememberthatwhenweuseattest,wehavetheoptionofformulatingadirectionalor
nondirectionalhypothesis.Choiceofadirectionalhypothesiswillincreasepowerbecauseyouareincreasingthealphalevelfrom,forexample,.025to.05.Similarly,
whenonehasmorethantwogroupsinasinglefactordesign,theoptionofusingplannedcomparisonsasopposedtoasinglefactoranalysisofvarianceisanotherway
toincreasepowerthroughtheuseofhypothesisformulation(seeKeppel,1991).
Thenexttwoelementsimportantforincreasingpowercanbeclarifiedifwerememberconceptuallytheformulaforthettest.Intheformula,variabilitybetween
groups,referredtoastreatmentvariance,isdividedbyvariabilitywithingroups,referredtoaserrorvariance.Ifwedecreaseourerrorvariance,andtreatment
varianceremainsthesame,wewillincreasepower.
Onemethodofdecreasingerrorvarianceandincreasingpoweristoreducethestandarddeviation, .Figure23.4(adaptedfromShavelson,1988,p.301)
demonstratesthisconcept.Asyoucansee,reducing .(rightsideofFig.23.4)changestheshapeofthepopulationdistributionunderthenullandalternative
hypothesisbymakingitnarrower.Ifwereducethevalueofthedenominator

Fig.23.3.
Powertable.FromDesignSensitivity:StatisticalPowerforExperimentalResearch
p.91),byM.W.Lipsey,1990,NewburyPark,CA:Sage.Reprintedwithpermission.

Page372

(byreducing ors)inourtequation,andallothervaluesremainthesame,thevalueoftwillincrease.Howdowedecreases?Onestrategyfordecreasingsisto
makesurethatthegroupsarehomogeneous.Whenworkingwithspecialpopulationsthisisnotalwayseasytoachieve.Forexample,personswhohavehadcerebral
vascularaccidentsareaveryheterogeneousgroup.Toreducetheheterogeneityoftheparticipants,criterianeedtobespecifiedinadvance.Thesecriterianeedtobe
relevanttothepurposeofyourstudy(e.g.,Atler&Gliner,1989Dittmar&Gliner,1987Lafferty&Gliner,1991).
Asecondstrategytodecreasesistomakesurethatthedependentmeasurehasahighlevelofreliability.Wheneverpossible,useameasurementinstrumentthathas
beenstandardizedandhasshownevidenceofgoodreliability.Measuresthathavelowevidenceofreliabilityincreasewithingroupvariability,ors.
Finally,samplesizeistheelementoverwhichwehavethemostcontroltowardincreasingpower.Again,anincreaseinsamplesize(N)reducesvariability.Returning
toourformulafort,thedenominatorinvolvesdividingthestandarderrorofthemeanbythesquarerootofthesamplesize,N.Therefore,thelargerthevalueofN,the
smallerthedenominatorwillbe.ReturningtoFig.23.3,ourpowerchart,wecanseethatanincreaseinparticipantswillincreasepower.Ifwedoubleoursamplesize
to50participantsineachgroup,ourpowerwillbe.70.Itisimportanttorememberallofthemethodstoincreasepowerbecausetherearemanysituations,suchas
programevaluations,wherethereare

Fig.23.4.
Increasingpowerbydecreasingvariability.FromStatisticalReasoning
FortheBehavioralSciences(2nded.)(p.301),byR.J.Shavelson,
Copyright1988,Boston:AllynandBacon.Adaptedwithpermission.

Page373

limitsonobtainingparticipants.Also,theremaybediminishingreturnsafteracertainsamplesizeisachieved.
EvaluatingPower
Thesecondissueinjudgingmeasurementreliabilityandstatisticsiswhetherpowerisappropriate.Poweristheabilitytodetectastatisticallysignificantdifferenceor
theabilitytorejectafalsenullhypothesis.Wehavedefinedpoweranddiscussedseveralmethodsofincreasingpower,butwefocushereonsamplesize.Asmall
samplemaynotproduceenoughpowertodetectafalsenullhypothesis,soitmayberatedlowonappropriatenessofpower.CookandCampbell(1979)raisea
secondsidetotheissueofpowerthatinvolveshavingtoomuchpower,especiallywithrespecttothenumberofparticipantsinastudy.Forthemostpart,theproblem
ariseswhenanexceptionallylargesamplesize(e.g.,1,000)yieldsastatisticallysignificant,butveryweak,relationshiporaverysmalldifference(i.e.,onewecanbe
almostcertainwasnotduetochancebutmaylackpracticalimportance).
AppropriatenessofStatisticalTechniques
Athirdissueunderlyingmeasurementreliabilityandstatisticsinvolvestheselectionoftheproperstatisticalmethodtoassesswhetherarelationshipbetweenthe
independentanddependentvariableactuallyexists.Inchapters13to19,weexaminedindetailtheappropriatechoiceofstatisticalprocedures.Wedemonstrated
differentmethodsofselectingstatisticalproceduresbasedprimarilyonthenumberofindependentvariables,thenumberoflevelswithintheindependentvariable,the
typeofdesign,andthescaleofmeasurementofthedependentvariable.Inaddition,knowledgeofassumptionsunderlyingparametricanalyseswasalsoimportantfor
selectionofalternativestoparametricstatistics.
Sometimesresearchersselectthewrongstatistic,suchasattestoraPearsonproductmomentcorrelationwithanominaldependentvariable.However,asCook
andCampbell(1979)pointedout,moreoftenproblemsinvolveviolationofassumptionsunderlyingstatisticaltestsorproblemsinmakingmultiplecomparisonswithout
adjustingthealphalevel.SuchproblemsmostoftenresultinincreasingtheprobabilityofmakingatypeIerror.Ourownexperiencesuggeststhatnotadjustingthe
alphalevelismorecommonthanselectionofaninappropriatestatistic.
AppropriatenessofInterpretation
Thefourthissueunderlyingmeasurementreliabilityandstatisticsinvolvesmakingtheproperinterpretationofthestatisticalanalysis.Withineachofourchapterson
interpretingstatisticalmethods(chap.1319),wespenttimedescribingtheproperinterpretationoftheresults.Sometimesthecorrectstatisticisselected,butthe
investigatormisinterpretsthefindings,concludingmorefromthedatathanisactuallygiven.
OnecommonexampleistointerprettheresultsfromasignificantsinglefactorANOVAwithatleastthreelevelswithoutperformingtheappropriate

Page374

posthoccomparisons.Sometimesaconclusionfollowsthatallgroupsorlevelsaresignificantlydifferentfromeachother.AsignificantFimpliesonlythatthereisa
significantdifferencesomewhere.
FailuretotakeintoaccounttheinteractioneffectsandinsteadattendingonlytothemaineffectsinafactorialANOVAisasecondexampleofimproperinterpretation
ofdataanalysis.Someresearchersfailtoacknowledgethatinalmostallfactorialdesigns,asignificantinteractionbetweentwoindependentvariablesalmostalways
yieldsmoreinformationthanresultsfromeitherindependentvariablebyitself.
Themostcommoninterpretationproblemistheomissionofanindicationofthesizeoftheeffect.Mostpublishedstudieshavemerelynotedwhetheraresultwas
statisticallysignificant.Aswediscussedearlier,inthesectionofAppropriatenessofPower,astatisticallysignificantfindingfromaverylargesamplemaybeoflittle
practicalimportance.Ameasureofeffectsizeisnecessarytoknowwhethertheeffectwassmall,mediumorlarge.Refertochapter11orCohen(1988)fora
discussionofhowtointerpretsizeoftheeffect.
MeasurementValidityandGeneralizabilityoftheConstructs
CookandCampbell(1979)calledthisaspectofresearchvalidityconstructvalidityofputativecausesandeffects,butwethinkthatthislabelisconfusing.What
doesputativemean?Moreimportantly,thisaspectofresearchvalidityrelatestocontentandcriterionrelatedvalidityaswellastoconstructvalidity.Thus,we
suggestthatabetterlabelismeasurementvalidityandgeneralizabilityoftheconstructs.Thisdimensionhastodowithwhetherthevariablesareappropriately
definedandrepresentativeoftheconceptsorconstructsunderinvestigation.Researcharticlesdealwiththisquestionpiecemealastestormeasurementvalidity(see
chap.20foranextendeddiscussion).Herethetaskistomakeanoveralljudgmentofthemeasurementvalidityandthegeneralizabilityoftheoperationaldefinitions
oftheseveralkeyconstructsinthestudy(seeFig.23.5.)
Akeyissueiswhethertheseoperationaldefinitionsarerepresentativeoftheintendedconstructs.Anexampleisanissueabouttheoperationaldefinitionofmastery
motivationusedinseveraloftheearlystudiesonthatconcept.Youngchildrenweregivenarangeoftoystoplaywithfreely.Theassumptionwasthattheamountof
playorthenumberofdifferentobjectsexploredwasameasureofmasterymotivation.Others(e.g.,Morganetal.,1990)havearguedthat,althoughsuchbehavior
doesreflectatypeofmotivation,itseemsinappropriatetolabelitmasterymotivationbecauseinfreeplayyoungchildrentendtoplaywithtoysandinwaysthatthey
havealreadymastered.Morganetal.arguedthatsome(moderate)degreeofchallengeshouldbeapartofanydefinitionandassessmentofmasterymotivation.
Consideranotherexamplewhereaninvestigatormaybeinterestedintestingtheeffectsoftwodifferentteachingmethods,traditionalandreformed,onthe
developmentofselfconfidenceinstudents.Toperformthisstudy,theinvestigatormustdefinetheindependentanddependentvariablesfortheoperations

Page375

Fig.23.5.
Evaluatingthemeasurementvalidityandgeneralizabilityofastudy.

neededtoconductthestudy.Thetraditionalapproachmaybedefinedasasituationwheretheteacherlecturestothestudentsanddiscussionorquestionsinclassare
discouraged.Thereformapproachmightbedefinedasasituationwherethestudentsareactivelyinvolvedinthelearningprocessandtakeresponsibilityforlearning.
Thedependentmeasure,selfconfidence,couldbedefinedasascorederivedfromselfreportsonaparticularpsychometricinventory.Theissueformeasurement
validityandgeneralizabilityoftheconstructsiswhethertheseoperationaldefinitionsarerepresentativeoftheintendedconstructs.Unfortunately,oftenthese
operationaldefinitionscouldbeinterpretedasmeasuringsomeotherconstruct,whichsetsupaconfoundedstudy.
CookandCampbell(1979)provideanexcellentexampleofconfoundingbyusingtheconstructofsupervision.Inastudy,supervisionisoperationallydefinedasthe
supervisorbeing10feetorlessfromtheworker.However,theoperationaldefinitioncouldalsoberelevanttotheconstructofstressbecausehavingasupervisorthat
closecouldleadtoincreasedstressamongworkers.Therefore,aretheinvestigatorsassessingtheeffectsofsupervisionortheeffectsofstress?
ResearchValidity:
Summary
Thissectionreviewsandsummarizestheinformationonresearchvalidityfromchapters6,10,20,andthischapter.Wehavedividedresearchvalidityintofourmain
dimensions,eachofwhichcanberatedonacontinuumfromlowthroughmediumtohigh.Thetextofthechapterslistednexttoeachdimensionelaboratesonhowto
usethedimensionstoevaluateanystudy:
1.MeasurementReliabilityandStatistics(seechap.20andthischapter)
2.InternalValidity(seechap.6)
a.Equivalenceofgroupsonparticipantcharacteristics
b.Controlofextraneousexperienceorenvironmentvariables
3.MeasurementValidityandGeneralizabilityoftheConstructs(seechap.20andthischapter)

Page376

4.Externalvalidity(seechap.10)
a.Populationvalidity
b.Ecologicalvalidity
Notethatinternalandexternalvalidityeachhavetwokeydimensions.Thesedimensions,showninFig.23.6inthelowerrighthandquadrantunderresearch(or
study)validity,provideacomprehensiveevaluationofthevalidityofawholestudy.
Figure23.7showsgraphicallytherelationshipamongtheseconcepts.Itshouldbeclearthattheresearchvalidityofastudydependstosomeextentonthereliabilityof
thespecificinstrumentsusedinthestudy.Thatiswhywehavelabeledonedimensionofresearchvaliditymeasurementreliabilityandstatistics.Furthermore,as
statedearlier,measurementvaliditydepends,inpart,onthereliabilityofthemeasureandinturnresearchvaliditydepends,inpart,onthevalidityoftheseveral
measuresused.

Fig.23.6.
Similaritiesanddifferencesbetweenmeasurementandresearchreliabilityandvalidity.

Page377

Fig.23.7.
Schematicdiagramshowinghowtheoverallvalidityofastudydependson
thefourmajoraspectsordimensionsofresearchvalidityandinturn
onthemeasurementreliabilityandvalidityoftheseveralvariables.

TheRelativeImportanceofDifferentValidityCategories
Itwouldappearimpossibletoachievehighratingsforeachcategoryofvalidityinasinglestudy.Typically,somethingissacrificedinonecategorytoobtainstrengthin
anothercategory.Forexample,investigatorswhoconductstudiesintightlycontrolledsituations,ofteninalaboratory,areusuallywillingtosacrificeameasureof
externalvalidityinfavorofstronginternalvalidity.Ontheotherhand,researcherswhoconductstudies''inthefield"usuallygiveupadegreeofinternalvalidityinfavor
ofstrongexternalvalidity,evenifrandomassignmentofparticipantstointerventionsispossible.Shouldastudybejudgedmoreorlessharshlycontingentona
weaknessinacertainvaliditycategory?
CampbellandStanley(1966)didcommentontheproblem."Bothtypesofcriteria(internalandexternalvalidity)areobviouslyimportant,eventhoughtheyare
frequentlyatoddsinthatfeaturesincreasingonemayjeopardizetheother.Theselectionofdesignsstronginbothtypesofvalidityisobviouslyourideal"(p.5).We
agreewithCookandCampbell's(1979)statementwhenplanningresearch,"Theartofdesigningcausalstudiesistominimizetheneedfortradeoffsandtotryto
estimateinanyparticularinstancethesizeofthegainsandlossesininternalandexternalvaliditythatareinvolvedindifferenttradeoffoptions"(p.90).Wealsothink
thatinastudy,thevalidityconsiderationsarenotrandomundertakings,butareconstrainedbyothervaliditycategoriesoncecertainprioritiesareselected.
Therefore,wethinkthatallaspectsofinternalandexternalvalidityareimportant.Noneshouldbeconsideredaprioriashavingmoreimportancethanothers.Instead,
wethinkthatcertainvaliditycharacteristicswillbeconstrainedbyothers,similartotheprecedingexample.Wediscussthistopicinmoredepthinchapter24.
Finally,wewouldliketopointoutthatthelowerleftsideofFig.23.6indicatesthatthereliabilityofthefindingsofastudyisimportant.Ifthefindingscannotbe
replicatedbyotherresearchers,thefindingscannotbevalid.Thisissueisdealtwith,inpart,intheremainderofthischapterunderthetopicofmetaanalysis.

Page378

ResearchReplication:
MetaAnalysis
SignificanceTestingImpedestheGoalsofScience
Applieddisciplines,likenonapplieddisciplines,havethegoalofaccumulatingknowledge.Mostresearcherswouldacknowledgethatitisnotpropertorecommendan
interventionoratreatmentbasedonasinglestudy.Instead,itisadvisabletoexaminethecumulativeeffectofnumerousstudiesbeforeapplication.Buthowdowe
learnaboutthecumulativeeffectsofaparticularareaofstudy?Usuallythistakesplacebyexaminingresearchthatappearsinjournalsofthatparticulardiscipline.
Mostofthearticlesthatappearindisciplinejournalspresentstatisticallysignificantfindings.However,notallstudiesresultinstatisticallysignificantfindings,andthis
createsaproblem.
Often,whenperformingresearchinappliedsettings,thesamplesizeissmall.Sometimesthislimitationappliestoeducationalsettings,suchasworkingwithintact
classrooms,whereclassroomsizeisoftensmall.Otherexamplesmightincludeclinicalsettingsorworkingwithpersonswithaparticulardisability.Whenthesizeofthe
sampleissmall,theprobabilityofrejectingafalsenullhypothesis(power)isreduced.Thelowerprobabilityoffindingastatisticallysignificantdifferenceinyourstudy
whenitis,infact,thereleadstomanyproblems,theleastimportantofwhichmightbethecommissionofatypeIIerror,andthemostimportant(atleasttomost
researchers)ofwhichisthefailuretopublishthefindingsinareputable,orperhapsnotsoreputable,journal.Mostjournaleditorswouldhaveresearchersintheir
variousdisciplinesbelievethatanystudythataddressesameaningfulquestion,hastheproperresearchdesign,analyzesthedatacorrectly,andmakestheproper
conclusions,butfailstorejectthenullhypothesishasthesamechanceofbeingpublishedasastudythatdidalloftheabove,butrejectedthenullhypothesis.Thisis
simplynottrue.Studieshavebeenperformedthatdemonstrateahigherprobabilityofacceptanceofarticlesthatpresentstatisticallysignificantdifferences(Rosenthal,
1979).Whatdoesallofthismeanforapplieddisciplinesinthefuture?
Itmeansthatwhenresearcherssearchtheliteraturetodeterminewhetheraparticulartherapyworked,oraparticularclassroominterventionwassuccessful,thesize
oftheeffectwillbeoverestimated.Thereasonisthat,withtheoveremphasisonstatisticalsignificancebyvariousjournals,onlythosestudiesshowingarelativelylarge
differencewillbepublished.Hence,ifoneweretoaveragethesizeoftheeffectsonlyfrompublishedstudies,ratherthanfrompublishedandunpublishedstudies,a
misleadingresultwouldoccur.
EffectSizeandMetaAnalysis
Previouslywediscussedoneoftheproblemswithnullhypothesissignificancetesting(NHST),themistakeofequatingastatisticallysignificantrelationshipwiththe
strengthofthatrelationship.Infact,manystatisticallysignificantrelationshipsarenotverystrong,butoftenareacquiredwithalargesamplesize.Toexaminethe
strengthofarelationship,orspecifically,theeffectoftheinde

Page379

pendentvariableonthedependentvariable,differentstatisticalmethodsmustbeapplied.Theseproduceanindexofthestrengthoftherelationshipbetweenthe
independentanddependentvariables,calledaneffectsize.Effectsizeshavebeenusedforyearshowever,mostresearchersdidnotpaymuchattentiontothese
indicesbecausejournalsweremostinterestedinstatisticallysignificantresults.
Recently,effectsizeindiceshavebecomepervasiveinallareasofresearch.Tounderstandtherationalebehindthem,considerpreviousscholarlyarticlesthatfocused
onreviewofaresearchtopicthatmightappearinajournalsuchasReviewofEducationalResearchRevieworPsychologicalBulletin.Inthepast,thesereview
articleswouldsynthesizetheliteraturebyusingascorecardmethodtodeterminethenumberofstudiesthatsupportedaparticularfindingandthenumberofstudies
thatdidnotsupportaparticularfinding.Theauthorofthearticlewouldattempttorationalizethefindingsandmakealegitimateconclusion.Anexampleofthistypeof
researchmightbetherelationshipbetweenclasssizeandstudentachievement.Supposethatthosearticlesthatdemonstrateastatisticallysignificantrelationship
betweenclasssizeandstudentachievementformacategorycalledsupportforstudentachievement,andthosearticlesthatdonotshowastatisticallysignificant
relationshipformacategorycallednosupport.Oneeitherhastoassumethatthereisarelationshipbetweenclasssizeandstudentachievementorthereisnota
relationship.
Now,ratherthanexaminewhetherornotastudydemonstratedastatisticallysignificantrelationship,wedecidetoanalyzeeachstudyfortheeffectsizeofthestudy,
thatis,thesizeoftherelationshipbetweenclasssizeandstudentachievement.Thisisdoneregardlessofwhetherthestudywasstatisticallysignificant.Inshort,we
takeeachstudyandcalculateaneffectsizethenweaveragetheeffectsizesforallofthestudies.Thisresultgivesusanoverallindexofthestrengthoftherelationship
betweentheindependentvariableanddependentvariable,orinthiscasebetweenclasssizeandstudentachievement.Thistypeofanalysisisreferredtoasmeta
analysis.Amajorstrengthofmetaanalysisisthatmanystudiesthathavebeenpublishedpreviouslyhavenotdemonstratedstatisticalsignificancebecausetheyhave
beenunderpowered,thatis,thesamplesizewastoosmall.However,theeffectsizeofthestudyisrelativelyindependentofsamplesize(asopposedtoajudgmentof
statisticalsignificance).Therefore,studiesthatdidnotresultinstatisticalsignificancestillcanbeincludedintheoverallmetaanalysis.(Itshouldbenotedthateach
effectsizecanbeweightedforthenumberofparticipantsinthestudy.)Furthermore,thereisagreaterchanceformorestudiesthatdidnotshowstatisticalsignificance
tobepublishedbecausetheirusefulnessinfuturemetaanalysiswouldbeconsidered.Foranexcellenttreatmentonthisargument,seeSchmitt(1996a).
Ifonebelievesthatscienceisanaccumulationofknowledge,thenmetaanalysisisthetypeofresearchthatwecanexpectinthefuture.Sincetheearlierworkby
Glassandcolleagues(Glassetal.,1981),therehavebeenanumberoftextsinthisareaincludingtwothatwerecommend:MethodsofMetaanalysis:Correcting
ErrorandBiasinResearchFindings(Hunter&Schmidt,1990)andTheHandbookofResearchSynthesis(Cooper&Hedges,

Page380

1994).Inaddition,journalssuchasthosementionedearliernowincludeahighproportionofarticlesthatusethemetaanalysisframework.
IssueswithUsingMetaAnalysis
Althoughmetaanalysishasnotbeenwithoutcritics,wewillfocusonthreeissuesthatareoftenencounteredwhenusingmetaanalysis.Thefirstissueforusingthis
methodisthatcontentareasoftenareheterogeneous.Forexample,supposethatoneisinterestedindemonstratingthatreformorinquirybasedteachingproduces
higherachievementthandoestraditionalteaching.Assumingthatwecouldlabeltraditionalteachingasalecturemethodwithlittleinterchangebetweenfacultyand
students,howwouldwedeterminereformteaching?Therearemanydifferentmethodsofreformteaching.Somebelievethatreformteachinginvolvescooperative
learning.Othersbelievethatnewtechnologymaybethekeytoreformteaching.Onemightarguethatanyteachingmethodthatisnotlectureshouldbeconsidered
reformteaching.Therefore,theresearcherisconfrontedwiththedecisionofhowexclusionarytobewhenconsideringwhatentailsareformedcourse.Alongwitha
suitabledefinitionofreformteachingistheissueofcontentareas.Shouldallreformedcoursesbeincluded,oronlythoserelatedtomathematicsorscience?Thelatter
isstillaheterogeneouscategory.Therefore,wecanseethatifthereisdifficultyindefiningtheindependentordependentvariable,thenthestrengthoftherelationship
betweenthetwowillbeweakenedanddifficulttointerpret.Themorerestrictivethatwebecomeindefiningourvariables,thehighertheprobabilityofdemonstratinga
largeoveralleffectsize.Ontheotherhand,aswebecomemorerestrictiveinourinclusioncriteriaforacceptablestudies,welimitthegeneralizabilityofourfindings.
Asecondissuethatisoftenraisedwhenapplyingmetaanalysisconcernsthetypeofmethodologicalapproachusedtoconducteachstudy.Forexample,supposethat
inourmetaanalysis,somestudiesusedanexperimentaldesign,whereasotherstudiesusedeitheraquasiexperimentaldesignoracomparativeapproach.Lipseyand
Wilson(1993)foundlittledifferenceinaverageeffectsizesbetweenrandomizedandnonrandomizedstudies.Ontheotherhand,HeinsmanandShadish(1996)found
thattherewerelargedifferencesbetweenrandomizedandnonrandomizedstudieswheninvestigatinganumberofdifferenttopics.HeinsmanandShadish(1996)
suggestedtwowaystoimprovethedesignofnonrandomizedstudiesformetaanalysis.Thefirstconcernsselectionofparticipantsintogroups.Theyemphasizedthat
wheneverpossible,participantsshouldnotbeallowedtoselectthemselvesintogroups.However,sometimesitisimpossibletoruleoutselfselectionbiasof
participantsintocertainsettings,suchascoursesinauniversity.TheothermethodtoimprovenonrandomizeddesignsformetaanalysissuggestedbyHeinsmanand
Shadish(1996)istoreducepretestdifferencesbetweentwogroupsthroughmatchingoranalysisofcovariance.
Athirdissueencounteredinmetaanalysisconcernstheacceptablenumberofstudiesforametaanalysis.Mostmetaanalysesincludelargenumbersofstudies.For
example,inMetaAnalysisforExplanation(Cooketal.,

Page381

1992),allfourcontentchaptersinvolvingmetaanalysisincludedatleast150studies.Journalarticlesthatinvolvemetaanalysistypicallyhavefewerstudies,butusually
includeatleast30differentstudies.Isthereaminimumnumberofstudiesforinclusionasametaanalysis?Rosenthal(1985)pointsoutthat"metaanalyticprocedures
canbeappliedtoasfewastwostudiesbutwhenthereareveryfewstudies,themetaanalyticresultsarerelativelyunstable"(p.185).
PerformingaMetaAnalysis
Performingametaanalysisinvolvesanumberofsteps,similartoanymethodologicalprocedure.Althoughnotallpublishedarticlesusingmetaanalysisfollowidentical
procedures,wewillpointoutsomeofthemorecommonsteps:
1.Literaturesearch.Theliteraturesearchwouldinvolvedecidingwhattypeofstudiestoinclude.Forexample,publishedarticles,dissertationabstracts,and
presentationsatnationalmeetingswouldmeetthecriteriaforrelevantliterature.Inaddition,thetypeofliteraturesearchesmightincludePsychologicalAbstracts,Index
Medicus,andvariousmanualsearchesofparticularjournals.
2.Inclusioncriteria.Thecriteriaforinclusionintothemetaanalysismustbestated.Forexample,ifonewereconsideringtherelationshipbetweenreformteaching
andstudentachievement,someoftheinclusioncriteriamightbe(a)onlymathematicscourses,(b)standardizedmeasuresofachievement,(c)studiesthatwereeither
experimentalorquasiexperimental,and(d)studieswherethecomparisongroupusedatraditionalmethodofinstruction.
3.Codingofstudies.Studiesarethenreviewed,andthosewhichmeetthecriteriaforinclusionintothemetaanalysisarethencodedoncertaincharacteristicssuch
as(a)demographics,(b)typeofdesign,(c)methodofreformteaching,(d)lengthofcourse,(e)typeofachievementtest,andsoon.
4.Computationandanalysisofeffectsizes.Effectsizesarethencomputedforeachstudy,althoughonestudymayhavemorethanoneeffectsize.Eacheffectsize
iscomputedintermsofeitheranrvalueoradvalue(seechap.11).Effectsizesarethenaveraged,takingintoaccountthenumberofparticipantsineachstudy
(weightedmean).Ifoneisusingrvaluesforeffectsizes,anoveralleffectsizeof.5isconsideredlarge,aneffectsizeof.3isconsideredmedium,andaneffectsizeof.1
isconsideredsmall,ifoneisusingdvalues,.2issmall,.5ismedium,and.8islarge(Cohen,1988).
Summary
Thischapterwasdividedintotwodifferenttopics,researchvalidityandresearchreplication.Thetwoarenotindependent.Weincludedtwomoredimensionsof
researchvalidityunderourgeneraltopicofresearchvalidity.Thesedimensionsaremeasurementreliabilityandstatistics,andmeasurementvalidityandgeneralizability
ofconstructs.Undereachdimensionwepointedoutthecriteriathatshouldbeusedtomakeaproperjudgmentwhenevaluatingaresearchstudy.Thesetwonew
dimensions,alongwiththosepreviouslyintro

Page382

duced,internalvalidityandexternalvalidity,comprisethefourdimensionsofresearchvalidity.Althoughallfourcriteriaareusedtoevaluatearesearchstudy,itisnot
likelythatastudywouldbehighonallfourdimensions.Typically,somethingissacrificedinonecategorytoobtainstrengthinanothercategory.Wethinkthatnoneof
thedimensionsshouldbeconsideredaprioriashavingmoreimportancethanothers.Underthetopicofresearchreplication,wediscussedmetaanalysis,andwhyitis
needed.Inaddition,wepointedouthoweffectsizesareaccumulated.Last,weintroducedsomeoftheimportantissueswhenusingmetaanalysis.
StudyAids
Concepts
Effectsize
Measurementreliabilityandstatistics
Measurementvalidityandgeneralizabilityofconstructs
Metaanalysis
Reliability
Researchvalidity
Distinctions
Measurementreliabilityversusmeasurementvalidity
Measurementreliabilityversusresearchreliability
Measurementvalidityversusresearchvalidity
Metaanalysisversusresearchsynthesis
Researchreliabilityversusresearchvalidity
ApplicationProblems
Forquestions1,2,and3useFig.23.3.
1.Aresearcherfeelsthatcertainmodificationstohertreatmentwillresultinaddedbenefitstopatients.Astudyissetuptocomparethemodifiedtreatment
(interventiongroup)totheoriginaltreatment(controlgroup).Previousresearchusingtheoriginaltreatmenthasdemonstratedeffectsizesofabout0.70.The
researcheriswillingtoacceptpowerof0.60.Howmanyparticipantswillsheneedineachgroup?
2.Acolleaguehasjustperformedastudy.Attesthadfailedtodemonstrateasignificantdifferencebetweenhistreatmentandcontrolgroups.Theeffectsizewas0.4.
Hehad15participantsineachgroup.Howmuchpowerdidhehaveinthisstudy?WhatwastheprobabilityofaTypeIIError?
3.Agraduatestudentisplanningherstudy.Shehasthecooperationofenoughundergraduatestoformtwogroupsof30studentsineachgroup.Shewouldliketo
havepowerof0.70.Inordertoobtainastatisticallysignificantoutcome,howlargeofaneffectsizewillsheneed?

Page383

4.Onemethodofgainingpoweristoreduceerrorvariance.Howisthisaccomplishedwithoutincreasingsamplesize?
5.Youhavebeenaskedtoevaluateaprogramthatadvocatesjointprotectiontechniquesforpersonswitharthritis.Theprogramisrelativelysmall,andyouonlyhave
18participantsineachgroup.Whatreasonablestepsshouldyoutaketomaximizepowerwithoutincreasingsamplesize?
6.Youaredesigningastudytodemonstratethatnontraditionalmethodsofinstructioninmathematicsproducebetterreasoningskillsamonghighschooljuniors.You
haveaccesstoonehighschoolthatappearsrepresentativeofmosthighschoolsinyourcity.Shouldyoudesignyourstudytofocusoninternalvalidityorexternal
validity?Why?
7.Itisoftenstatedthatresearchstudiesinthesocialsciencesareunderpowered?Howcanmetaanalysisimprovethesituation?

Page385

Chapter24
EvaluatingEmpiricalResearchStudies:
ASynthesis
Overview
PeerReview
IdentifyingorAnalyzingtheKeyVariablesandDesign
VariablesandTheirMeasurementScales
ResearchHypothesesorQuestions,Approaches,andDesign
MeasurementReliabilityandValidity
EvaluatingResearchValidity
MeasurementReliabilityandStatistics
InternalValidity
EquivalenceofGroupsonParticipantCharacteristics
ControlofExtraneousExperienceorEnvironmentVariables
MeasurementValidityandGeneralizabilityoftheConstructs
ExternalValidity
PopulationValidity
EcologicalValidity
IssuesofInterpretation
AnExampleofHowtoUseThisFramework
ConcludingComments
Summary
StudyAids
ApplicationProblems

Page386

Overview
Inthischapterwereview,stepbystep,theprocessofanalyzinganarticleandthenevaluatingitbyusingtheresearchvalidityscalesdescribedinearlierchapters.As
wemovethroughthisprocess,wewillmentionkeyconceptsanddiagramsfromearlierchaptersthatwehavefoundusefulinunderstandingresearchandtheprocess
ofevaluatingit.Thisevaluationisbasedontheanswersto16questions(Table24.1)thatprovidetheinformationneededtoanalyzeandevaluatethefouraspectsof
researchvalidity.Thequestions,andyourcritique,willfocusonthemethodsectionofanarticlebutwillalsoincludequestionsabouttheremainderofthearticle,
includingthetitleandconclusions.Thisevaluationdrawsonandsynthesizesmanypartsofthisbook,sowewillrefertotheappropriatechapters.
Ideally,astudyshouldberatedhighoneachcriterion.However,therearealwaystradeoffsandfew,ifany,studiesarehighonallcriteria.Furthermore,theweight
thatresearchersgivetoeachofthecriteriavaries.Forexample,humanandanimalexperimentalresearchersgivemoreweighttointernalvalidity,whilesurveyand
qualitativeresearchersgiverelativelymoreweighttoaspectsofexternalvalidity.Ourexperienceindicatesthatstudiesusuallycompromiseoneormoreaspectsof
externalvaliditytoachievehighinternalvalidityorviceversa.
PeerReview
TheevaluationbeginswithQuestion1,whichisaboutthesourceinwhichthestudyispresentedorsummarized.Asaconsumerofresearch,youwilloftenread
newspaperornewsletterarticlessummarizingresearchstudies.Thesearticlesmaynotgivemuchdetailaboutthemethodused,buttheywillusuallyprovide
informationaboutthesourcefromwhichthesummaryarticlewastaken.Ifthesourceisascholarlyjournal,thechancesarethatthepeerreviewwasatleast
moderatelyextensiveandstrict.Bypeerreview,wemeanthatthearticlewasevaluatedbyotherexperts(peers)inthefield,usuallywithoutthemknowingwhothe
authorofthearticlewas(maskedreview).Iftheassociationthatpublishesthejournalconsistsprimarilyofpractitionerswhoareonlysecondarilyinterestedin
research,thepeerreviewofthedesignandanalysisislikelytobelessstrictbecausepractitionerreviewsfocusmoreonapplicationandimplications.
Presentationsatprofessionalmeetings,evenresearchorientedmeetings,areusuallylessstrictlyreviewed,especiallyifthejudgmenttoacceptwasbasedona
summaryorabstractofthepaper.Furthermore,presentationstononscholarlyaudiencesorateventslikepressconferencesareevenlesslikelytobereviewedby
independentscholarsorresearchersand,thus,donothavepeerreview.Finally,studieswhosemainorsolesourceisdisseminationinapopulararticleoranarticlein
apopularmagazineornewspaperwouldnothavehadpeerreview.However,manynewspaperarticlesarebasedonpublishedpeerreviewedarticlesor
presentationsatprofessionalmeetings,sotheyhavehadsomesortofpeerreview.Butthejournalistmayhaveleftoutimportantdetails.

Page387

IdentifyingorAnalyzingtheKeyVariablesandDesign
VariablesandTheirMeasurementScales
Thesecondquestionasksyoutoidentifythekeyindependentvariables.Foreach,state(a)whetherthevariableisactive(i.e.,manipulated),attribute,orchange
overtime,(b)thenumberoflevelsorcategories,and(c)thescaleofmeasurementoftheindependentvariable.Thesedistinctionsarediscussedinchapters4,9,and
12.
Thethirdquestionasksyoutoidentifythekeydependentvariable(s)andtheirlevelsofmeasurement.Figure9.1andTables9.1and9.2shouldbeusefulinlearning
orrememberinglevelsorscalesofmeasurement.Thelevelofmeasurementfortheindependentand,especially,thedependentvariableshelpsdeterminethe
appropriatenessofthestatisticsusedinthestudy.
ResearchHypothesesorQuestions,Approaches,andDesign
Thefourthquestionasksyoutoidentifythemainresearchhypothesesorquestions.Moststudieshaveseveralquestionsorhypotheses,oftenoutlinedinthe
introduction,orthemethodsectionofthearticle,orboth.
Webelievethatitispossibleandusefultodescribeallstudiesasusingoneormoreofthefiveresearchapproaches(descriptive,associational,comparative,quasi
experimental,orrandomizedexperimental,oracombinationoftheseapproaches),asdescribedinTable5.1andinFig.5.1.Question5asksyoutostatewhich
approacheswereusedinthisstudy.Generally,eachresearchquestionorhypothesisusesoneoftheapproaches,soinastudywithanumberofquestions,morethan
oneapproachmaybeused.Itiscommonforastudytohaveseveraldescriptiveresearchquestions,especiallyaboutthecharacteristicsofthesampleand,perhaps,
aboutsomeofthedependentvariables.Thosepartsofthemethodandresultssectionsofanarticleusedescriptivestatisticsandwhatwecallthedescriptive
approach.However,almostalltheses,dissertations,andpaperspublishedinquantitativejournalsgobeyondthepurelydescriptiveapproachtocomparegroups,or
relatevariables,orboth.Thus,moststudiesalsowilluseone(ormore)oftheotherfourapproaches.Identifyingtheresearchapproachisimportantbecauseit
influencestheinternalvalidityofastudyandwhatcanbeinferredaboutwhethertheindependentvariablecausedanydifferenceinthedependentvariable.Ingeneral,
therandomizedexperimentalapproachproducesthebestevidenceforcausation.Neitherthecomparativenortheassociationalapproachesarewellsuitedto
providingevidenceaboutcauses.Quasiexperimentationisusuallyinbetween.
Question6statesthatifthestudyhasexperimental,quasiexperimental,orcomparativeresearchquestion(s),thenoneshouldidentifythedesignclassification(see
chap.12).Thisrequiresknowing(a)thenumberoffactors(independentvariables),(b)thenumberoflevelsorvaluesofeachfactor,and(c)whetherthedesignis
betweengroups,withinsubjects(repeatedmeasures),ormixed.Forexample,adesignmightbedescribedasa3 2(mixed)factorialdesignwithrepeated
measuresonthesecondfactor.Thismeansthatthereare

Page388

twoindependentvariables,thefirstwiththreelevelsorgroupsandthesecondwithtwolevelsor,inthiscase,measuredattwotimesbecausetherearerepeated
measures.Wethinkthatthistypeofanalysisofdesignsappliestonotonlytherandomizedexperimentalandquasiexperimentalapproaches(whichistypical),butalso
tocomparativeapproachquestions,wherethereisnoactiveormanipulatedindependentvariable.Notethattheabove3 2designcouldbethetypicalexperimental
orquasiexperimentalpretestposttestdesignwiththreegroups(e.g.,twotreatmentsandacontrol)oritcouldbealongitudinal(twoages)designcomparingthree
typesofparticipants(e.g.,securelyattachedversusavoidantversusdisorganized)overtime.
Becausetheclassificationofthedesign,inthissense,couldapplytocomparativeaswellasexperimentalapproaches,itdoesnotprovideinformationaboutcausality.
However,thedesigniscriticaltodeterminingtheappropriatestatisticalanalysis,which,likewise,doesnottellusaboutcausation.Weallknowthephrase,''correlation
doesnotindicatecausation."However,thesameistrueforanalysisofvariance(ANOVA),iftheapproachwascomparativethatis,thetypeofanalysisorstatistic
doesnotdeterminecausality.Thus,thetypicalANOVAterminologyofmain"effects"canbemisleading.Ingeneral,similarexperimental,quasiexperimental,and
comparativedesignsareallanalyzedwiththesametypeofanalysisofvariance(orttest).Thespecifictypeofdataanalysisdependsonthedesignclassification,and
thelevelofmeasurementofthedependentvariableasshowninTables13.1and13.3.
Question7asksyoutonamethespecificrandomizedexperimentalorquasiexperimentaldesign,ifonewasused.Table7.2providesaschematicdiagramofmostof
thecommondesignsandtheirnames.
Notethatiftheresearchquestionorhypothesisandapproachareassociational,theanalysiswillusuallybedonewithacorrelationormultipleregression,and
questions6and7arenotapplicable.
MeasurementReliabilityandValidity
Questions8and9requireanevaluationbasedontheprinciplethatinagoodstudyeachkeyvariableshouldbemeasuredreliablyandvalidly.Therefore,weaskyou
toevaluatetheseaspectsofeachkeyvariableseparately(seechap.20).
Question8askswhattypesofreliability(e.g.,testretest,internalconsistency,interrater)wereobtainedandhowstrongistheevidenceforthereliabilityofthe
measure?Thiswasdiscussedindetailinchapter20.Notethatactiveindependentvariablesseldomhaveinformationaboutmeasurementreliability(orvalidity).
However,forattributeindependentvariables(exceptveryobjectiveorobviousoneslikeageandgender)andfordependentvariables,themethodsectionwillusually
reportsomethingonmeasurementreliability.Rememberthatareliabilitycoefficientof.70orhigherisusuallyconsiderednecessaryforameasuretohaveacceptable
reliability,butinacomplexstudy,afewreliabilitycoefficientsinthe.60sarecommon.Asindicatedfollowing,weviewgoodmeasurementreliabilityaspartofone
aspectofresearchvaliditythepartwecallmeasurementreliabilityandstatistics.Inevaluatinginformationaboutreliabilityinanarticle,examinethemethodsection.
Wereany

Page389

reliabilitycoefficientsreported?Wasmorethanonetypereportedforameasure?Iftheinstrumentshadbeenusedbefore,theauthormayonlyrefertoanotherstudy
andnotprovideactualcoefficients.
Question9isaboutthevalidityofeachmeasure.Again,youshouldcommentbothonwhattypesofvalidity(i.e.,face,content,criterion,construct)werereportedand
howstrongtheevidencewasforvalidity.Thiswasdiscussedindetailinchapter20.Authorsoftenonlycitepreviousstudiesthatusedtheinstrumentwithoutproviding
detailsabouttheevidenceforvalidity.
Insummary,foreachkeymeasureorvariable,oneshouldcommentaboutitsreliabilityandvalidity.
EvaluatingResearchValidity
Nowyouarereadytoevaluatethefourkeycriteriaforthevalidityofastudy.Thesecriteriaarebasedonthewriting(discussedinearlierchapters)ofCampbelland
Stanley(1966)andCookandCampbell(1979),asmodifiedbyus.Ahighqualitystudyshouldhavemoderatetohighratingsoneachofthedimensionsofresearch
validity,asshowninFig.24.1andFig.24.2.Youareaskedtoratethestudyasawholeoneachofthescales,byusingthecriterialistedinthefiguresanddiscussed
inthefollowingsections.
MeasurementReliabilityandStatistics
Thefirstdimensionemphasizestheimportanceofmeasurementreliability.Question10requestsanoverallratingofthestudyfromlowthroughmediumtohigh,thatis
basedonallfourofthefollowingissues.
First,thereistheissueofwhetherthevariablesasagrouparemeasuredreliably.YouareaskedinQuestion10toconsideranoverallratingofthereliabilityofallthe
instruments.Aprincipleoftenemphasizedinmeasurementclassesisthatatestormeasurecannotbevalidifitisnotreliable.Likewise,astudy'svalidityisreducedif
oneormoreofthekeymeasuresarerelativelyunreliable.
Second,canastatisticallysignificantrelationshipbedetected,assumingthatsucharelationshipexists?Theabilitytodetectastatisticallysignificantdifferenceismost
commonlyreferredtoaspowerortheabilitytorejectafalsenullhypothesis.Althoughadequatepowerisbased,inpart,onhavingenoughsubjectsinthestudy,there
areothermethodsofincreasingpower(Lipsey,1990).Someofthesemethodsincludedecreasingvariabilityandincreasingreliabilityofthedependentvariableor
increasingthestrengthandconsistencyofadministeringtheindependentvariable.Chapter23discussesthesemethodsinmoredetail.CookandCampbell(1979)
raiseasecondsidetotheissueofpower,whichinvolveshavingtoomuchpower,especiallyforthenumberofparticipantsinastudy.Forthemostpart,theproblem
ariseswhenanexceptionallylargesamplesizeyieldsastatisticallysignificant,butperhapstrivial,relationship.Thetrendtowardestimatingeffectsizesincurrent
researchisonewayofresolvingthisproblem.
Athirdissuetoconsiderinvolvestheselectionoftheproperstatisticalmethodtoassesswhetherarelationshipbetweentheindependentanddependentvariable
actuallyexists.Wehavediscussedselectionandinterpretationof

Page390

Fig.24.1.
Ratingscalestoevaluatethemeasurementreliability
andstatisticsandtheinternalvalidityofthefindingsofastudy.

Page391

Fig.24.2.
Ratingscalestoevaluatethemeasurementvalidityandgeneralizability
oftheconstructsandtheexternalvalidityofthefindingsofastudy.

Page392

statisticsinchapters13to19.Sometimesresearchersselectthewrongstatistic,suchasattestorcorrelationwithanominaldependentvariable.AsCookand
Campbell(1979)pointout,moreoftenproblemsinvolveviolationofassumptionsunderlyingstatisticaltestsorproblemsinmakingmultiplecomparisonswithout
adjustingthealphalevel.SuchproblemsmayresultinatypeIortypeIIerror.Ourownexperiencesuggeststhatnotadjustingthealphalevelismorecommonthan
selectionofaninappropriatestatistic.
Thefourthissuetoconsiderinvolvesmakingtheproperinterpretationofthestatisticalanalysis.Sometimesthecorrectstatisticisselected,buttheinvestigator
misinterpretsthefindings,concludingmorefromthedatathanisactuallygiven.Forexample,ifonehasasignificantinteractionfromtheanalysisofafactorial
ANOVA,oneshouldexaminethesimpleeffectsratherthanthemaineffects,whichmaybemisleading.Theseissuesarediscussedinmoredetailinchapters13to19.
InternalValidity
EquivalenceofGroupsonParticipantCharacteristics
Thereareanumberofspecificthreatstointernalvalidity,severalofwhicharesubjectfactors,thatcouldleadtoalackofequivalenceoftheparticipantsinthetwo(or
more)groups,and,thus,influencethedependentvariable.YouareaskedinQuestion11toevaluatethisaspectofinternalvalidity.
Intherandomizedexperimental,quasiexperimental,andcomparativeapproaches,akeyquestioniswhetherthegroupsthatarecomparedareequivalentinall
respectsexcepttheindependentvariableorvariables,beforetheproceduresofthestudytakeplace.Theusualwaytoassureequivalenceisbyrandomlyassigningthe
participantstogroups.Randomlyassigningtreatmentstointactgroups(strongquasiexperiments),matching,analysisofcovariance,orcheckingforpretestequalityof
groupsafterthefactareallmethodsofachievingamediumlevelofthisaspectofinternalvalidityifrandomassignmentisnotpossible.
Alsoincludedinthisaspectofinternalvaliditywouldbeparticipantcharacteristicsthatappeartobeequalinitially,butleadtodifferentdevelopmentaltrajectoriesover
timeand,thus,groupdifferencesthatarenotduetotheindependentvariable.Differencesbetweengroupsinmaturationrates,geneticpredispositions,orinfactors
leadingtosubjectattritionareexamples.Regressiontothemeanofextremegroupsisanexampleofwheredifferencesinparticipants,whichwereinitiallyobvious,
disappearovertimebecausetheinitialscorewasnotthetruescore.
Iftheapproachisassociational,thereisonlyonegroup.Inthatcase,thisaspectofinternalvaliditycomesdowntothequestionofwhetherparticipantswhoscorehigh
ontheindependentvariableofinterestareequivalenttothosewhoscorelow,intermsofotherattributesthatmaybecorrelatedwiththedependentvariable.Ifitis
likelythatthehighscorersarenotequivalenttothelowscorersforvariablessuchasage,socialstatus,education,andsoforth,partialingoutthevariablesonwhichthe
highandlowsubjectsareunequalisonemethodofachievingadegreeofthisaspectofinternalvaliditywithintheassociationalresearchapproach.Theseissuesare
discussedinmoredetailinchapter6.

Page393

ControlofExtraneousExperienceorEnvironmentVariables
Similarly,wehavegroupedseveralother"threats"tointernalvalidityunderacategorythatdealswiththeeffectsofextraneous(thosevariablesnotofinterestinthis
study)experiencesorenvironmentalconditionsduringthestudy.Question12asksyoutoevaluatetheinternalvalidityscalecontrolofextraneousexperienceand
environmentvariables.Thefirstaspecthastodowithwhetherextraneousvariablesoreventsaffectonegroupmorethantheother(s).Intheassociationalapproach,
theissueiswhethertheexperiencesoftheparticipantswhoarehighontheindependentvariablearedifferentfromthosewhoarelowontheindependentvariable.
Asecondissueiswhethersomethingotherthantheindependentvariableisaffectingthedependentvariableforbothorallgroups.Thisproblemis,ofcourse,more
likelytobeseriouswhenthetreatmentcontinuesoveranextendedtimeperiodandinlongitudinalresearch.Historicaleventsandmaturationcouldaffectbothgroups
equally,sogroupdifferencesmaynotoccur.However,theresultscouldbemisinterpretedif,forexample,oneconcludedthatsignificantpretesttoposttestgains(or
losses)intwoexperimentalgroupswereduetothetreatmentsratherthantoanuncontrolledcommon(e.g.,Hawthornetype)experienceormaturation.Ofcourse,in
longitudinalstudies,maturationisthevariableofinterest.Thisissueisquitecomplex.
Inrandomizedexperimentaldesigns,theseexperientialandenvironmentalvariablesareusuallywellcontrolled,butinquasiexperimentaldesigns,andespeciallyinthe
comparativeandassociationalapproaches,suchexperiencesmaybeinadequatelycontrolled.Ingeneral,thereisatradeoffbetweenhighcontrolofextraneous
variablesandhighecologicalvalidity.Itishardtohaveboth.Ifastudyisratedlowonthesedimensionsofinternalvalidity,theauthorsshouldnotusetermssuchas
effect,impact,anddeterminethatimplycauseandeffect.Phrasessuchasmayaffect,presumedcause,possibledeterminantseemappropriatelycautious.
MeasurementValidityandGeneralizabilityoftheConstructs
TheratingrequestedinQuestion13hastodowithwhethertheactiveindependentvariables(treatments),attributeindependentvariables,anddependentvariablesare
measuredvalidlyandareappropriatelydefinedsothattheyrepresenttheconceptsunderinvestigation.Youhavebeenaskedtodealwiththisquestionpiecemealin
Question9,above,onmeasurementvalidity,andwediscusseditmorefullyinchapter20.Nowyourtaskistomakeanoveralljudgmentofthevalidityand
generalizabilityoftheoperationaldefinitionsoftheseveralkeyvariablesinthestudy.
Theissueiswhethertheseoperationaldefinitionsarerepresentativeoftheintendedconceptsandconstructs.Inchapter23,weprovidedthreeexamplesofproblems
forthisdimension:masterymotivationinyoungchildrenasmeasuredbytheamountoffreeplay,thedefinitionsoftwodifferentteachingmethods(traditionaland
reform),andsupervisionasmeasuredbybeingwithin10feetoftheworker.Theissueiswhethertheseoperationaldefinitionsarerepresentativeoftheintended
constructs.Unfortunately,oftentheseopera

Page394

tionaldefinitionscouldbeinterpretedasmeasuringsomeotherconstruct,whichwouldsetupaconfoundedstudy.
ExternalValidity
PopulationValidity
Thisaspectofexternalvalidityisaparticipantselectionproblemthatinvolveshowparticipantswereselectedtobeinthestudy.Weretheyrandomlyselectedfroma
particularpopulation,orwerevolunteersused?Mostquantitativestudiesinthesocialscienceshavenotusedrandomselectionofparticipants,buttheissueof
populationexternalvalidityismorecomplexthanwhethertherewasarandomsample,asdiscussedinchapter10andpresentedinFig.10.1.
Question14asksforanoverallratingofwhethertheactualsampleofparticipantsisrepresentativeofthetheoreticalortargetpopulation.Tomakethisevaluation,first
identify(a)thetheoreticalpopulation,(b)theaccessiblepopulation,(c)thesamplingdesignandselectedsample,and(d)theactualsampleofparticipantswho
participatedinthestudy.Itispossiblethattheresearchercouldusearandomorotherprobabilitysamplingtechnique,buthaveanactualsamplethatisnot
representativeofthetheoreticalpopulationeitherbecauseofalowresponserateorbecauseoftheaccessiblepopulationnotbeingrepresentativeofthetheoretical
population.Thelatterproblemseemsalmostuniversal,inpartasaresultoffundingandtravellimitations.Exceptinnationalsurveyresearch,wealmostalwaysstart
withanaccessiblepopulationfromthelocalschooldistrict,community,clinic,animalcolony,andsoon.
Figure10.4helpsreinforcetheimportantdistinctionbetweenrandomsampling(orselectionofsubjectsfromthepopulation),whichinfluencespopulationexternal
validity,andrandomassignment(ofsubjectstogroups),whichinfluencestheparticipantequivalenceaspectofinternalvalidity.
EcologicalValidity
Thesecondaspectofexternalvalidity,calledecologicalvalidity,hastodowiththeconditionsorsettings,testers,procedures,andtimes.Question15asksyouto
evaluatetheseaspectsforhowrepresentativetheyareofreallife,and,thus,whethertheresultscanbegeneralizedtoreallifeoutcomes.Obviously,fieldresearchis
morelikelytobehighonecologicalexternalvaliditythanlaboratoryprocedures,especiallyiftheyarehighlyartificial.Wewouldratemostoftheselfreportmeasures,
especiallyquestionnaires,tobeartificialbecausetheyarenotdirectmeasuresoftheparticipant'sactualbehaviorinatypicalenvironment.
Inchapter10wediscussedtwoexamplesofproblemswithecologicalvalidity.First,itwastypicalinthe1960stotest6to12montholdinfantsinanatleast
somewhatunnaturalsetting(labplayroom)withamalestrangerwhoapproachedandquicklypickedupthebabyinashortseriesofpredeterminedsteps.Inthename
ofexperimentalcontrol,noattemptwasmadetohavetheexperimenterorstranger'sbehaviorbecontingentonthebaby's.Thisprocedureandeventheexistenceof
fearofstrangers,wascriticizedasbeinglowonecologicalvalidity.Theresearcherhadtradedecologicalvalidityforbettercontroloftheenvironmentandthe
independentvariable.

Page395

Theotherexampleinvolvedaneducatorinterestedintheeffectofaparticularteachingstyleonstudentparticipation.Tobehighonecologicalvalidity,theclassroom
shouldbesimilartothatofanormalclassroom.Similarly,iftheinvestigatoraskedstudentstocomeatnightforthestudy,butthesestudentsnormallyattendedclass
duringtheday,thenthereisaprobleminecologicalexternalvalidity.Theinvestigatormustaskifarepresentativemethodwasusedforselectionofthesettingand
time,orwasanunrepresentativemethodused?
IssuesofInterpretation
Finally,thelastquestionisakindofsummaryquestionthatasksyoutoevaluatethetitleandespeciallythediscussionandconclusionsforindicationsofinaccuracyor
misleadingstatements,givenyourpreviousanalysisofthestudy.Ofteninpopulararticles,theeditororwriterwilloverstatethefindingstomakethemmoreimpressive
ormoreeasilyunderstoodbythepublic.Thus,inapopulararticlereportingonastudywithrelativelylowinternalvaliditybecauseoflackofcontrolsorlackof
equivalenceofgroups,thewritermayreportorimplythattheindependentvariablecausedthedependentvariable,hadanimpacton,ordeterminedtheoutcome.
Theseoverstatementsmaynothavebeenmadeintheactualarticlebytheresearcher,whomayhavepresentedtheconclusionsmorecautiouslyandappropriately,
giventheinternalvalidityofthestudy.Likewise,astudythatinvolvedratsoranunrepresentativesampleofpeopleorononegendermaybeovergeneralized,perhaps
withoutanymentionofthetypesofsubjectsused,orimplyingthatthereisnoprobleminmakingmoregeneralstatements.Theastuteconsumershouldbecomeaware
ofthesepossibleoverinterpretationsandevaluatethearticleappropriately.
AnExampleofHowtoUseThisFramework
ToillustratehowtousethisframeworkandFig.24.1and24.2toevaluateresearch,considerastudybyGlinerandSample(1996).Thepurposeofthestudywasto
increasequalityoflifeforpersonswithdevelopmentaldisabilitieswhowereemployedinshelteredworkorsupportedemployment,byusinganinterventionof
communitylifeoptions.Thestudyattemptedtoachievehighinternalvaliditybyrandomlyassigningparticipantstoeitheracommunitylifeoptionsinterventionortotheir
presentsituation.Thestudyalsoattemptedtoachievehighexternalvaliditybycarryingouttheconditionsintheactualsetting.However,obtaininggoodresearch
validityonallsixdimensionscouldnotbeaccomplished.AsummaryexampleofhowawrittenevaluationofthesixdimensionsfortheGlinerandSample(1996)
articlemightlookisprovidedinthenexttwoparagraphs.
Measurementreliabilityandstatisticswerejudgedoveralltobemediumhigh.Reliabilityofthemeasureswasgood.However,statisticalpowerwasconstrained
becausetherewerealimitednumberofpersonswhofitthecriteriatobeinthestudy(personswithdevelopmentaldisabilitieswhowereemployedinsupportedor
shelteredwork).Thus,theabilitytodetectarelationshipwasreduced.Thechoiceofstatisticsandtheirinterpretationwasjudgedtobehigh.

Page396

Internalvalidity:equivalenceofgroupsonparticipantcharacteristicswasratedhighbecauseparticipantswererandomlyassignedtointerventionconditions.
However,acautionarynotecouldberaisedbecauserandomassignmentofparticipantstoconditionsmaynotmakethegroupsequivalentwithsmallnumbers.
Controlofextraneousexperienceorenvironmentvariableswasconstrainedbyanemphasisonecologicalvaliditysoitwasjudgedtobemediumtolow.Ina
communitysetting,wherechoicewasexperienceddifferentlybydifferentparticipants,itwasdifficulttoinsurethattheexperiencesofeachgroupwerenotinfluenced
byoutsidevariables.
Intermsofmeasurementvalidityandgeneralizabilityoftheconstructs,thedependentvariable,qualityoflifeforpersonswithdevelopmentaldisabilities,hadbeen
usedseveraltimeswiththispopulationtomeasurequalityoflifeamongindividualswhohadmovedoutofinstitutionalizedsettingsintocommunitysettings.However,
theinstrumentmaynothavebeenappropriateformeasuringchangesfollowinginterventioninonlyonelifearea.Inaddition,theinstrumentmayhavebeenintendedfor
lowerfunctioningparticipants.Intermsoftheindependentvariable,theinterventionseemsappropriatelynamedandgeneralizable.Overallthisdimensionwasrated
medium.Populationexternalvaliditywasconsideredtobemediumlowbecausethesamplewaslimitedtopersonsinonecity,andtherewasnotarandom
selectionofappropriateparticipantsevenfromthatcity.Instead,thesamplewasoneofconvenience.Thus,theaccessiblepopulationmightnotrepresentallpersons
withdevelopmentaldisabilities.Becausetheinterventionwasarealoneandtookplaceinanactualcommunitysetting,ecologicalexternalvaliditywasjudgedtobe
high.
ConcludingComments
Therearemanyotherquestionsthatcouldbeaskedaboutaresearcharticle,suchasitsreadability,clarity,andsoforth,butwebelievethatwehavediscussedthe
majordimensionsand,thus,wehavenottriedtobetotallycomprehensiveinourcoverage.
Summary
Thischapterprovidesanintegratedreviewofmanyoftheimportantconceptsmadethroughoutthebook.Thus,therearenonewconceptsordistinctionstopresentat
theendofthischapter.Answerstothe16questionspresentedinTable24.1provideacompleteanalysisandevaluationofaresearchstudy,especiallyofitsmethod.
Todosuchananalysis,youmustidentifythekeyvariablesandtheircharacteristics(typeandlevelofmeasurement).Youmustalsoidentifytheresearchquestions,
approaches,anddesign.Intermsofevaluation,wehaveprovidedratingscales,andrubricsforratingthem,toevaluatethefourmaindimensionsofresearchvalidity.
Thesedimensionsareasfollows:
1.Statisticsandmeasurementreliability
2.Internalvalidity
a.Equivalenceofthegroupsonparticipantcharacteristics

Page397
TABLE24.1
AnalysisandEvaluationofQuantitativeResearchStudies
Foreachstudy,answerthefollowingquestions,focusingmostofyourevaluativecommentsonquestions8
16.
EvaluationoftheEditorialReview
1.Ratetheamountandstrictnessofthepeerreviewfromlowtohigh.Explain.
VariablesandTheirMeasurementScales(seechapters4and9).Iftherearemanyvariables,pick
510keyormajorones,whichmaybecompositevariables.
2.Identifythekeyindependentorantecedentorpredictorvariable(s).Foreach,statea)whetheritisan
active,attribute,orchangeovertimeindependentvariable,b)thenumberoflevelsorcategories,andc)the
levelofmeasurement.
3.Namethekeydependentoroutcomevariable(s)andtheirlevel(s)ofmeasurement.
ResearchQuestions,Approach(es),andDesign(seechapters5,7,and12)
4.Identifythemainresearchquestionsorhypotheses.
5.Namethespecificresearchapproach(i.e.,descriptive,associational,comparative,quasiexperimental,
and/orrandomizedexperimental)foreachquestion(seechapter5).
6.Identifythedesignclassification(forexample,22factorialwithrepeatedmeasuresonthesecondfactor),
iftheapproachisrandomizedexperimental,quasiexperimental,orcomparative(seechapter12).
7.Identifythespecificdesignname(e.g.,posttestonlycontrolgroupdesign)iftheapproachisrandomized
experimentalorquasiexperimental(seechapter7).
MeasurementReliabilityandValidity(seechapter20).Forthemeasureofeachkeyvariable,evaluate:
8.Thereliabilityofthemeasure(notusuallyreportedfordemographicsoractiveindependentvariables).
Whattype(s)weredone?Ratethestrengthoftheevidencefromlowtohigh.
9.Thevalidityofthemeasure(notusuallyreportedfordemographicsoractiveindependentvariables).What
type(s)werereported?Ratethestrengthoftheevidencefromlowtohigh.
EvaluationoftheValidityoftheFindingsofaStudy(refertoFig.24.1andFig.24.2)
10.Rateandcommentonthemeasurementreliabilityandstatisticsoverall(seechapters20and23).Baseyour
ratingon:
a)Reliabilityoftheinstrumentsormeasures
b)Appropriatenessofpower
c)Appropriatechoiceoruseofstatistics
d)Appropriateinterpretationofstatisticalresults.
11.Rateandcommentontheequivalenceofthegroupsonparticipantcharacteristics(seechapter6).
12.Rateandcommentonthecontrolofextraneousexperienceorenvironmentvariables(seechapter6).
13.Rateandcommentonthemeasurementvalidityandgeneralizabilityoftheconstructs(seechapters20and
23).
14.Rateandcommentonthepopulationvalidityoverall(seechapter10).Beforedoingso,answerthe
following:
a)Identifythetheoretical,ortarget,andaccessiblepopulationsandtheselectedandactual
samples.
b)Howrepresentativeistheaccessiblepopulationofthetheoreticalpopulation?
c)Howrepresentativeistheselectedsampleoftheaccessiblepopulation?
d)Howrepresentativeistheactualsamplevisvistheselectedsample?Thatis,howgoodwasthe
responserate?
15.Rateandcommentoneachofthefiveaspectsoftheecologicalvalidity(seechapter10).
EvaluationoftheConclusions
16.Commentonwhetherthetitle,conclusions,andsoforthwereoverstatedormisleading,givenyour
evaluationoftheaspects.

Page398

b.Controlof(extraneous)experience/environmentvariables
3.Measurementvalidityandgeneralizabilityoftheconstructs
4.Externalvalidity
a.Populationvalidity
b.Ecologicalvalidity
StudyAids
ApplicationProblems
Foreachofthefollowingproblems,evaluatethefouraspectsofresearchvalidity.Ifnotenoughinformationisprovided,statewhatyouwouldneedtoknowtomake
anevaluation.
1.Researchersinalargemetropolitanschooldistrictwithadiversemultiethnicstudentpopulationhaveimplementedastudyregardingthepossibleeffectsoftypeof
curriculumandtypeofcounseloronstudentleadershiplevelsmeasuredwithaninstrumentusingasummatedLikertscale.Theresearcherswereabletochoosea
randomsamplefromtheentireseniorclass.Thesestudentswerethenrandomlyassignedtotwogroups,eithertheexperimentalmultiethnicortraditionalleadership
curriculum,taughtbyacounselorwithextensivetraininginmulticulturalissuesandatraditionallytrainedcounselor,respectively.Inotherregards,theseindividualsare
verysimilarineducationandexperience.
2.Aresearcherisinterestedinstudyingtheeffectofsleepdeprivationonteenagers'mathperformance.Hehasalimitedresearchbudget,sohedecidestostudy
studentsatthelocalcollege.Heobtainedalistofallthestudentsforeachlevel(freshmentosenior).Herandomlysamples10studentsfromeachofthelevels.All40
agreetobeinthestudy.TheyansweraquestionnaireabouttheamountofsleeptheyhadduringfinalsweeklastsemesterandtheirSATmathscores.
3.Aresearcherplanstodoalaboratoryexperimentalstudyofsleepdeprivationonmathperformance.Herandomlyassignsstudentsfromhissampletotwogroups
of20each.Onegroupiskeptawakeallnightstudyingandgivenamathtestinthemorning.Theothergroupisencouragedtosleepaslongastheywantbeforethey
takethemathtestinthemorning.
4.APh.D.studentaskedarandomsampleoffacultyatacollegetoansweraquestionnaire50%responded.Thefacultymembersclassifiedtheirdepartmentheads
asoneoffourtypesofleader(A,B,C,orD)basedonanswerstoabriefleadershipinventory.Facultymemberswereaskedtheirownage,classifiedasyounger(<
35),middle(3549),orolder(50+).Theresearcherwantedtoknowiftheabovecharacteristicsseemtoinfluencetheirjobsatisfaction,ratedona9pointLikert
scale.
5.Astudywasundertakentodeterminethebacktoworkeffectsoftwotypesoftreatmentonpostsurgicalcarpaltunnelsyndromepatients.TreatmentFullused
splintsonafulltimebasis,whereastreatmentPartusedsplintsonaparttimebasis.Inaddition,theinvestigatoralsowasinterestedinwhetherpatientswhoscored
highonthepersonalityvariableofcodependencywould

Page399

doworsethanpatientswhoscoredlowonthevariableofcodependency.Fivehundredpostsurgicalpatientsfromalargemetropolitanareavolunteeredforthis
study.All500patientsweregiventheGlinerCodependencyPersonalityInventory(testretestreliabilityr=0.88predictivevalidityr=.66).Fromthissampleof
500carpaltunnelsyndromepatients,the20patientswiththehighestcodependencyscores(H)andthe20patientswiththelowestcodependencyscores(L)were
selectedtocontinueinthestudy.Fromthesetwogroups,HandL,halfofthepatientswererandomlyassignedtothefulltimesplintgroupFandhalfofthepatients
wererandomlyassignedtotheparttimesplintgroupP.Priortotheinterventions,all40participantsweregiventheGlinerCarpalTunnelSyndromepainInventory
(notcurrentlyfoundintheBurosMentalMeasurementsYearbooks).Ahighscoreonthisinventorymeantmuchpainandlittlesuccess.Afterthreemonthsof
intervention,all40participantsweretestedagainontheGlinerCarpalTunnelSyndromepainInventory.

Page401

APPENDIXA
CONFUSINGTERMS1ANDGLOSSARY
DifferentTermsforSimilarConcepts
Variables
Independentvariable=antecedent=predictor=presumedcause=factor=#way
Activeindependentvariable=manipulated=intervention=treatment
Attributeindependentvariable=measuredvariable=individualdifferencevariable
Dependentvariable=outcome=criterion
Levels(ofavariable)=categories=values=groups=samples(fromthevariable)
Designs
Betweengroups=independentsamples=uncorrelatedsamples
Withinsubjects=relatedsamples=pairedsamples=repeatedmeasures=matchedgroups=correlatedsamples
Singlefactordesign=oneindependentvariable=basic
Factorialdesign=twoormoreindependentvariables=complexdesign
Randomassignmenttogroups=howsubjectsgetintogroups=randomizeddesign highinternalvalidity
1

Thedefinitionsofmost''confusing"termsarefoundintheglossary.

Page402

Randomizedexperiment=trueexperiment
Statistics
Associatevariables=relate=predict correlation
Comparegroups=testdifferences ANOVA
ANOVA=F=analysisofvariance
RepeatedmeasuresANOVA=withinsubjectsANOVA=randomizedblocksANOVA
MixedANOVA=SplitplotANOVA=(sometimescalledrepeatedmeasuresANOVA)
MannWhitneyUtest=WilcoxonMannWhitneytest
Sampling
Randomselection=randomsampling highexternalvalidity
Theoreticalpopulation=targetpopulation
Accessiblepopulation=samplingframe
Measurement
Alternateforms=equivalentforms=parallelformsreliability
Intervalscale=numeric=continuousvariable=quantitative=scalevariable
Nominalscale=categorical=unorderedvariable=qualitative=discretevariable
Ordinalscale=unequalintervalscale
Normal=normallydistributedvariable=intervalorratiovariable
SimilarTermsforDifferentConcepts
Measurementvalidityresearchvalidity
Randomassignmentrandomselectionrandomorder
Randomassignmentofparticipantstogroupsrandomassignmentoftreatmentstogroups
Cronbach'salphaalpha(significance)level
Randomsamplespairedorrelatedsamplesindependentsamples
Factorialdesignfactorialevidenceforconstructvalidity
Factor(i.e.,independentvariable)factoranalysis
Discriminantanalysisdiscriminantevidenceforconstructvalidity
Independentsamplesindependentvariables
Measurementscalearatingscalesummatedorcompositescalesemanticdifferentialscale
Researchquestionquestionnairequestionoritem
Theoreticalresearchtheoreticalpopulation

Page403

KeyDistinctions
(seeglossaryandtheindicatedchapters)
Achievementtestsversusaptitudetests(chap.21)
Activeversusattributeindependentvariable(chap.4,5)
Anonymousversusconfidential(chap.3)
Basic(twovariables)versuscomplex(threeormorevariables)statistics(chap.13)
Betweengroupsdesignsversuswithinsubjectsdesignsversusmixeddesigns(chap.12,13,15)
Comparegroupsversusassociatevariables(chap.13)
Confidentialversusanonymous(chap.3,22)
Controlgroupversuscomparisiongroup(chap.7)
Convergentversusdiscriminantversusfactorialevidenceforconstructmeasurementvalidity(chap.20)
Correlationcoefficientversussquaredcorrelationcoefficient(chap.16)
Correlationversuslinearregression(chap.16)
Datacollectiontechniques(methods)versusresearchapproaches(chap.21)
Dichotomousversusnominalversusordinalversusnormallevelsofmeasurement(chap.9,13)
Differenceversusassociationalversusdescriptiveresearchquestionsandstatistics(chap.5,6,13)
Directionalhypothesisversusnondirectionalhypothesis(chap.11)
Equivalenceofgroupsonparticipantcharacteristicsversuscontrolofextraneousexperienceorenvironmentvariables
Experimentalversusindividualdifferenceresearchapproaches(chap.5)
Faceversuscontentversuscriterionversusconstructevidenceformeasurementvalidity(chap.20)
FactoranalysisversusfactorialANOVA(chap.19)
Identifiablecausesversusmutualsimultaneousshaping(chap.2)
Independentversusdependentversusextraneousvariable(chap.4)
Internalvalidityversusexternalvalidity(chap.6,10,23,24)
Laboratoryversusfieldresearch(chap.1)
Levelversusmeanlevel(chap.8)
Levelversustrendversusslope(chap.8)
Levelsofonevariableversusasetorgroupofvariables(chap.4)
Maineffectsversusinteractioneffects(chap.17)
MANOVAversusunivariateANOVAs(chap.19)
Meanversusmedianversusmode(chap.9)
Measurementreliabilityversusmeasurementvalidity(chap.20,23)

Page404

Measurementreliabilityversusresearchreliability(chap.23)
Measurementvalidityversusresearchvalidity(chap.20,23)
Metaanalysisversusresearchsynthesis(chap.23)
MixedANOVAversusgainscoresanalysisversusANCOVA(chap.18)
Multiplebaselineacrosssubjectsdesignversusmultiplebaselineacrossbehaviorsdesignversusmultiplebaselineacrosssettingsdesign(chap.8)
Multipleregressionversusdiscriminantanalysisversuslogisticregression(chap.19)
Nominalversusordinalversusinterval/ratioscalesofmeasurement(chap.9)
Nomothetic/generalizableknowledgeversusidiographicknowledge(chap.2)
Normreferencedversuscriterionreferencedtests(chap.21)
Nullhypothesisversusalternativehypothesis(chap.11)
Numberofindependentvariablesversusnumberoflevelsofthevariable(chap.13)
Openendedversusclosedendedquestions(chap.21)
Orderedversusunorderedornominalvariables(chap.4,9)
Ordinalversusdisordinalinteraction(chap.17)
Parametricversusnonparametricstatistics(chap.13,14)
PearsonproductmomentcorrelationversusSpearmanrankordercorrelation(chap.16)
Pilottestingversuspilotstudies(chap.22)
Plagiarismversusparaphrasing(chap.22)
Poorquasiexperimentalversusbetterquasiexperimentalversusrandomizedexperimentaldesigns(chap.7)
Populationversusecologicalexternalvalidity(chap.10)
Populationversussample(chap.10)
Positivistapproachversusconstructivistapproachtoresearchorparadigmortheoreticalframework(chap.1,2)
Predictiveversusconcurrentevidenceforcriterionrelatedmeasurementvalidity(chap.20)
Probabilityversusnonprobabilitysampling(chap.10)
Producingknowledgeversusunderstandingresearchasaconsumer(chap.1)
Quantitativeversusqualitativedataanalysis(chap.1)
Quantitativeversusqualitativedataanddatacollection(chap.1)
Questionnairequestionoritemversusresearchquestion(chap.21)
Questionnaireversusinterview(chap.21)
Quotaversuspurposiveversusconveniencesampling(chap.10)
Randomassignmentofparticipantstogroupsversusrandomassignmentoftreatmentsto(intact)groups(chap.5,7)
Randomassignmentofparticipantstogroupsversusrandomselectionorsamplingofparticipantstobeincludedinastudy(chap.5,10)
Randomassignmentversusnonrandomassignmentofparticipantstogroups(chap.7)

Page405

Randomizedexperimentalversusquasiexperimentalversuscomparativeversusassociationalversusdescriptiveapproachtoresearch(chap.5)
Relationshipsbetweenoramongvariablesversusdescriptionofavariable(chap.5)
Researchproblemversuspersonalorsocietalproblem(chap.4)
Researchreliabilityversusresearchvalidity(chap.23)
Researcherandparticipantsareindependentversusresearcherandparticipantsalwaysinfluenceeachother(chap.2)
Researcherreportorobservationalmeasuresversusselfreportmeasures(chap.1,21)
Reversaldesignsversusmultiplebaselinedesigns(chap.8)
Risksversusbenefits(chap.3)
Samplestatisticsversuspopulationparameters(chap.11)
Selectedsampleversusactualsample(chap.10)
Simpleeffectsversusposthoccomparisons(chap.17)
Simplerandomversussystematicrandomversusstratifiedrandomversuscluster(random)sampling(chap.10)
Simultaneousregressionversushierarchicalregressionversusstepwiseregression(chap.19)
SinglefactorANOVAversusKruskalWallisANOVAversusc2test(chap.14)
SinglefactorrepeatedmeasuresANOVAversusFriedmantestversusCochranQtest(chap.15)
Singlefactordesignsversusfactorialdesigns(chap.12)
Singlesubjectdesignsversustraditionalgroupdesigns(chap.8)
Singletangiblerealityversusmultipleconstructedrealities(chap.2)
Stratifiedsamplingwithequalversusdifferentialproportions(chap.10)
ttestforpairedsamplesversussinglefactorrepeatedmeasuresANOVA(chap.15)
ttestforpairedsamplesversusWilcoxonmatchedpairstestversusMcNemartest(chap.15)
ttestversusMannWhitneyUtestversus 2test(chap.14)
ttestversussinglefactorANOVA(chap.14)
Testretestversusparallelformsversusintervalconsistencyversusinterraterevidenceformeasurementreliability(chap.20)
Testsofsignificanceversusconfidenceintervals(chap.11)
Thevariable(itself)versuslevelsorcategoriesofthevariable(chap.4)
Theoreticalversusappliedresearch(chap.1)
Theoreticalortargetpopulationversusaccessiblepopulation(chap.10)
Truemixeddesignversuspretestposttestdesign(chap.18)
TypeIversusTypeIIerrors(chap.11)
Valuefreeversusvalueladeninquiry(chap.2)

Page406

Weakquasiexperimentsversusmoderatestrengthquasiexperimentsversusstrongquasiexperiments(chap.7)
Glossary
A
AccessiblePopulationThegroupofparticipantstowhichtheresearcherhasaccess.Theaccessibleportionofthetheoreticalpopulationoftenreferredtoasthe
surveypopulationorsamplingframe.Seealsoconveniencesample.(chap.10)
ActiveIndependentVariableSometimescalledamanipulatedvariable.Aninterventionormanipulatedindependentvariableasopposedtoonethatisanattribute
oftheparticipantsortheirongoingenvironment.Avariableonelevelofwhichisgiventoonegroupofparticipantsbutnottoanother,withinaspecifiedperiodof
time.Thus,apretestandposttestarepossiblebutnotalwaysdone.(chap.4)
ActualSampleThesampleofsubjectsthatcompletethestudyandwhosedataareactuallyusedinthedataanalysisandreportofthestudy'sresults.(chap.10)
AlternateFormsReliabilityAlsocalledequivalentorparallelformsreliability.Ameasureoftheconsistencyorcorrelationbetweentwosupposedlyequivalent
versionsofatestorinventory.(chap.20)
AlternativeHypothesisAstatementpredictingthatarelationshipexistsbetweenthevariablesbeingresearched.Theconverseofthenullhypothesis.Ifthenull
hypothesisisrejected,thealternativehypothesisissupported.(chap.11)
AnalysisofVariance(seeANOVA)
ANOVAAstatisticalmethodthatusesthesumsofthesquaresofthedeviationsfromthemeanstotestthedifferencesamongtwoormoregroupsinmostcasesit
allowsthetotalvariancetobeseparatedandattributedtodefinedsourcesthesimplestcaseANOVAcomparesthevariabilitybetweengroups(treatmentvariance
pluserror)tovariabilitywithingroups(error).(chap.13,14,15,17,18)
AssociationWhenthesameorpairedparticipantshavescoresontwovariables,thestrengthoftheirassociationcanbeanalyzedbyusingtechniquessuchas
correlation(fortwovariables)ormultipleregression(formorethanoneindependentvariable)orcanonicalcorrelation(formorethanoneindependentandmorethan
onedependentvariable).(chap.5,16,19)
AssociationalInferentialStatisticsAgrouportypeofstatisticsthatanalyzestheassociationsorrelationshipsamongvariablesandtestsofthestatistical
significanceoftherelationships.Includescorrelationandmultipleregression.Seealso:association.(chap.5)
AssociationalResearchApproachSometimescalledcorrelationalapproach.Asinglegroupofsubjectsormatchedpairsareusedtoexaminetherelationship
betweenanindependentvariableandadependentvariable.Usuallyvariablesmeasuredarecontinuousorhavefiveormoreorderedlevelsifso,correlation
coefficient(Pearson'sorSpearman's)showingthestrengthoftheassociationorrelationshipbetweenvariablescanbecomputed.Iftheindependentanddependent
variablesarenominal,phiorCramer'sV(ifmorethantwolevelsofeithervariable)istheappropriatemeasureofassociation.(chap.5,13,16,19)

Page407

AssociationalResearchHypothesesorQuestionsAtypeofresearchquestionthatattemptstoassesstherelationbetweenvariables,usuallyinasinglegroupof
subjectstheassociationalapproachtestsassociationalhypothesesorquestions.(chap.5)
AttributeIndependentVariableAvariablethatisacharacteristicoratraitofaparticipantoroftheirongoingenvironment,whichtheresearchersdidnot
manipulatebutonlymeasured.Avariablethatcannotbemanipulatedorgiven,yetisamajorfocusofthestudy,forexample,gender,age,ethnicgroupordisability.
(chap.4)
B
BasicAssociationalQuestionUsedinstudiesthatexamineassociationsorrelationshipsbetweentwovariables:onedependentvariableandoneindependent
variable.Seealso:complexassociationalquestion.(chap.5,13,16)
BasicDifferenceQuestionUsedwithrandomizedexperimental,quasiexperimental,andcomparativeapproachesexaminesdifferencesbetweengroupsor
differencesbetweentwoormoremeasuresofthesameparticipant(withinsubjectsdesign).Thesequestionshaveasingleindependentandasingledependent
variable.(chap.5,13,14,15)
BetweenGroupsDesignsDesignswhereeachsubjectparticipatingintheresearchisinoneandonlyoneconditionorgroup.(chap.12)
BiasAconsistent,nonrandomerrorinmeasurementwithapredictablepatternacrossvariables,whichresultsinanoverorunderestimationofpopulationvalues.
(chap.10)
C
CalculatedValueAvalueornumberobtainedaftersubjectingcollecteddatatoastatisticalformula.(chap.11)
CanonicalCorrelationAcorrelationbetweenlinearcompositesformedfromseveralindependentandseveraldependentvariables(chap.13)
CasesPersons,groups,oreventsthatareofinterestinaparticularstudy.(chap.10)
CategoricalVariableAvariablethatdistinguishesamongsubjectsbyputtingthemintoalimitednumberoflevels,categories,orgroups,indicatingtypeorkind,as
genderdoesbycategorizingpeopleintomaleorfemale.Alsocalleddiscreteandnominalvariable,butnotallcategoricalvariableshaveunorderedcategories.For
example,low,medium,andhighIQisacategoricalvariablebutnotnominal.(chap.4,9)
CauseWhenachangeinonevariableproducesachangeinanothervariable,thefirstvariableissaidtocausethesecond.Thereismuchdisagreementamong
researchersabouthowandevenwhethercausescanbeidentified.(chap.2,5,6)
ChisquareTestforIndependenceAnonparametric,inferentialstatisticaltestthatcomparesanexpectedfrequencywithameasuredfrequency(observedcases).
Resultsshowwhetherornottheresultscouldhaveoccurredbychancewithinthepopulation.Usedtodetermineasignificantdifferencebetweentwoormoregroups
whenthereisoneindependentvariablethatiscategorizedontwoorafewlevels,andthedependentvariableisalsocategorical.(chap.13,14)
ClusterAcollectionorgroup(e.g.,townsorschools)ofpotentialsubjectsthatdonotoverlap.Usuallygeographicallygroupedtogether.(chap.4)

Page408

ClusterSamplingTwostagesamplingprocedurethatfirstrandomlyselectsspecificgroups(clusters)ofsubjects,andthenselectsallsubjects(orafixed
proportionrandomly)fromthosegroups.(chap.10)
CochranQTestAnonparametricstatisticusedinadesignwheretheindependentvariablehasthreeormorelevelsandtheparticipantsundergoallconditionsor
havebeenmatchedontherelevantvariable.Thedependentvariableisnominalorcategorical.Frequenciesaretheunitofmeasurement.Achisquaredistributionis
usedtodeterminestatisticalsignificance.(chap.15)
CoefficientofEquivalenceComputedwhentestingforequivalentorparallelformsofreliability,establishingarelationshipbetweenthetwoformsofthesametest.
(chap.20)
CoefficientofStabilityComputedwhentestingfortestretestreliability.(chap.20)
ComparativeResearchApproachExaminesthedifferencebetweengroupsthatarenonequivalent.Eachgrouphasdifferingcharacteristicsbutisthesamein
termsofsomeattributeindependentvariables.Thenumberofgroupscorrespondstothenumberoflevelsoftheattributeindependentvariable.Therearealwaysat
leasttwoandusuallynotmorethanfourlevels.Theinvestigatorcannotrandomlyassignsubjectstogroupsbecausesubjectsaregroupedaccordingtooneindividual
attributethattheinvestigatorcannotalterormanipulate(e.g.,gender,eyecolor,politicalaffiliation).(chap.5)
ComparisonGroupAgroupofparticipantsinaquasiexperimentwhodonotreceivethetreatmentorintervention.Alsoagroupinarandomizedexperimentthat
receivesanalternativetreatment.(chap.7)
ComplexAssociationalQuestionUsedinastudythatexaminesrelationshipsamongvariableswithinasinglegroup.Hastwoormoreindependentvariablesused
topredictonedependentvariable.(chap.13,19)
ComplexAssociationalStatisticAstatisticdesignedtopredictasingledependentoutcomevariablefromseveralindependentvariables.(chap.13,19)
ComplexDifferenceQuestionUsedinrandomexperimental,quasiexperimental,orcomparativestudiesthatexaminedifferencesbetweengroupsorbetween
measuresonthesamegroup(withinsubjectsdesigns).Therearetwoormoreindependentvariables.(chap.13,17,18)
ConcurrentEvidenceforCriterionValidityAlsocalledconcurrentvalidity.Examinestherelationshipbetweenameasurementandanoutsidecriterion,asdoes
predictivevalidity,butassessesthetesttobevalidatedandthecriterionatroughlythesametime.(chap.20)
ConfidenceIntervalArangeofvalueswithinwhichthereisapredeterminedprobability(95%)thatthepopulationparametersmayfall.(chap.14)
ConstructSomethingthatexiststheoreticallybutisnotdirectlyobservable.Aconceptdeveloped(constructed)fordescribingrelationsamongphenomenaorfor
otherresearchpurposes.(chap.2)
ConstructValidityOneaspectofmeasurementvalidity.Aslowprocesswheretheinvestigatorconductsstudiestodemonstratethataninstrumentismeasuringa
construct(hypotheticalconceptsthatcannotbedirectlyobserved).Threetypesof

Page409

evidenceareusedtosupportconstructvalidity:convergent,discriminant,andfactorial.Seealsoeachtype.(chap.20)
ConstructivistParadigmAssertsthatrealityandtrutharesubjectiveanddependentoneachobserver.Usuallyinvolvesqualitativedatacollectionandonly
descriptiveornostatisticalanalyses.Frequentlyappropriateforhumansciencesresearch,whichmayinvolveconsiderationofvariablesthataredifficulttoquantifyor
toholdconstant.Tendstobeinductiveinnature.Alsocalledqualitativeandnaturalistparadigm.(chap.1,2)
ContentValidityAnaspectofmeasurementvalidity.Thecontentoftheinstrumentisrepresentativeoftheconceptthatisbeingmeasured.(chap.20)
ContinuousVariableAvariablethathasaninfinite(orverylarge)numberofscoreswitharange.Forexample,heightandgradepointaveragearecontinuous
variables.Person'sheightscouldbe69.38inches,69.39inches,oranythinginbetween.(chap.4)
ControlGroupInrandomizedexperimentalresearch,asamplethatreceivesnotreatmenttocomparemeasurementswiththesamplethatreceivestreatment.(chap.
5,7)
ConvenienceSampleAsampledrawnfromanapproachableoreasilyobtainablesubsetofthegeneralpopulation.Participantsareselectedonthebasisof
convenienceratherthanmakingaseriousattemptbeforehandtoselectparticipantsthatarerepresentativeofthetheoreticalpopulation.(chap.10)
ConvergentEvidenceforConstructValidityOthermeasuresthatarepredictedtheoreticallytoberelatedtotheconstructbeingvalidatedare,infact,related.
Onetypeofevidencetosupportconstructvalidity.(chap.20)
CorrelationAstatisticthatindicatestheassociationorrelationshipbetweenscoresontwovariables.Correlationmaybepositive,negative,orzero.Positive
correlationreflectsadirectrelationshipbetweentwovariables:asthevalueofonevariableincreases,thevalueoftheothervariablealsoincreases.Negative
correlationreflectsaninverserelationshipbetweentwovariables:Asthevalueofonevariableincreases,thevalueoftheothervariabledecreases.Ifnosystematic
relationshipexistsbetweentwovariables,thecorrelationissaidtobezero.(chap.16)
CorrelationCoefficientAnumberbetween1.0and+1.0thatexpressesthestrengthofanassociationorrelationship.Usuallysymbolizedasr.(chap.16)
Cramer'sVAstatisticaltestusedinassociationalresearchapproacheswhereboththeindependentanddependentvariablesarenominalorcategorical.(chap.13,
16)
CredibleTobetrustworthyorbelievable.Qualitativeresearchfrequentlyestablishescredibilitybyrelyingonotherresearcherstoreanalyzecollectedresearchdata
andcorroboratetheoriginalresearchconclusions.(chap.2)
CriterionRelatedValidityOneaspectofmeasurementvalidity.Validatingtheinstrumentagainstaformofexternalcriterion,usuallyinvolvingcomputinga
correlationcoefficientbetweentheinstrumentandtheexternaloroutsidecriterion,eitherconcurrentlyorpredictively.Seealsoconcurrentandpredictiveevidencefor
validity.(chap.20)

Page410

CriticalValueTheprecisenumericalvaluesthatdefinethelimitsofrejectionforanullhypothesis.Valuesusedintestingastatistictodetermineifitissignificant,
thatis,ifitexceedstheparametersofthenullhypothesis.(chap.11)
Cronbach'sAlphaAmethodtodetermineinteritemreliabilityorinternalconsistencyformeasureswithseveralitems(eachwithorderedresponses)thatwillbe
summedtomakeacompositescale.(chap.20)
CrossSectionalDesignAdesigninwhicheachparticipantisassessedatoneage,butotheryoungerorolderpersonsarealsotestedtomakeageormaturation
comparisons.Seealsolongitudinaldesigns.(chap.12)
D
DecisionTreesApictorialmethod,resemblingaflowchart,forbreakingdownthestatisticalselectionprocess.(chap.13,14,15,17,18)
DeductiveReasoningUsingthelogicpatternofapplyinggeneralprinciplestoderiveconclusionsaboutspecificcases.Usedinquantitativestudiestotestatheory
bydeducinghypotheses,whicharetestedusingstatistics.(chap.2)
DegreesofFreedom(df)Thenumberofdatavaluesthatarefreetovarywhencalculatingastatistic.Thenumberofobservations(n)minusthenumberof
restrictionsplacedonthedatadfisusedtolookupcriticalvaluesformanystatisticaltests.(chap.11)
DependentVariableThedependentvariableisthepresumedoutcomeorcriterion.Itisassumedtomeasureorassesstheeffectoftheindependentvariable.
(chap.4)
DescriptiveResearchApproachThisbasicresearchapproachconsidersonlyonevariableatatimesothatnocomparisonsorrelationshipsaremade.
Descriptiveresearchapproachesuseonlydescriptivestatisticssuchasaverages,percentages,histograms,andfrequencydistributions.(chap.5)
DescriptiveStatisticsProceduresforsummarizing,organizing,graphing,and,ingeneral,describingquantitativeinformation.Oftencontrastedwithinferential
statistics,whichareusedtomakeinferencesaboutapopulationbasedoninformationaboutasampledrawnfromthatpopulation.(chap.9)
DichotomousVariableAlsocalleddummyvariable.Avariablewithtwolevelsorcategories(e.g.,maleandfemale).Canbeusedinmultipleregressionanalyses
asanindependent,predictorvariable.(chap.9,19)
DifferenceHypothesisorQuestionsPredictivestatementsabouttherelationshipbetweengroupsformedfromtheindependentvariables.Researchquestionsare
similartohypothesesbutareinquestionformat.Differencehypothesesandquestionsareusedwhentheresearchapproachisrandomizedexperimental,quasi
experimental,orcomparative.(chap.5)
DifferenceInferentialStatisticsStatisticsthatanalyzethedifferencesbetweentwoormoregroupsorlevelsofavariable.Usedwithrandomizedexperimental,
quasiexperimental,andcomparativeresearchapproaches.Differencesbetweentwoindependentgroupscanbeassessedusingatwosamplettest(orthe
nonparametricMannWhitneyorchisquaretest).Ifthedesigniswithinsubjectsandtherearetwomeasuresorconditions,userelatedsamplesttest(orthe
nonparametricWilcoxonorMcNemartests).Differencesbetweenseveralindependentgroupsformedfromoneindependentvariablecanbetestedusinga

Page411

onewayANOVA.SeveralrepeatedassessmentscanbetestedusingrepeatedmeasuresANOVA.(chap.5,13)
DirectionalHypothesisAresearchhypothesisthatpredictsthedirectionoftherelationshipbetweenvariables.Becauseitbasesrejectionofthenullhypothesison
aonetailedtest,itisamorestatisticallystringentmeasurethanthenondirectionalhypothesis.(chap.11)
DisciplinedInquiryAsystematicinvestigationofamatterofpublicinterestsothatalogicalargumentcanbecarefullyexamined.Evidentialtestsandverification
arevalued.Adispassionatesearchfortruthisvaluedoverideology.(chap.1)
DiscriminantAnalysisAcomplexassociationalstatisticusedtopredictacategoricaloutcome(e.g.,passversusfail)fromseveralnormallydistributedindependent
variables.Usedtoclassifyparticipantsintogroups.(chap.19)
DiscriminantEvidenceforConstructValidityEvidencetosupportconstructvalidityisprovidedwhengroupsthatarepredictedfromtheoryorliteratureto
differonthemeasurementbeingvalidated.(chap.20)
DisordinalInteractionIngraphingtheresultsofafactorialANOVA,whendifferencesbetweencellmeanswithinagrouparestatisticallysignificant,andwhenthe
linesthatconnectthecellmean(withingroupdifferences)fortwoseparategroupsintersect,theinteractionisdisordinal.Whentheydonotintersectbutaresignificant,
theinteractionisordinal.Seealsoordinalinteraction.(chap.17)
DummyVariableAdichotomous(twolevels),variable(suchasgender)thatisassignedbinaryvaluesforpurposesofcomputation(i.e.,male=1,female=2).
Seedichotomousvariable.(chap.9,19)
E
EcologicalExternalValidityTheextenttowhichtheresearchisgeneralizableonthebasisofwhethertheresearchenvironmentissimilartothe''realworld"or
naturalenvironment.Thereareseveralaspectsofecologicalvalidity:naturalnessofthesettingandprocedures,rapportwiththeresearcher,andlackoftime
boundnesstothequestionsorproblem(chap.10,24)
EffectSizeAnyofseveralmeasuresofthestrengthofarelation.Bycontrast,testsofthenullhypothesisonlyallowyoutoconcludethatarelationissignificantly
largerthanzero,buttheydonottellyoubyhowmuch.Thus,theeffectsizeisanestimateofthedegreetowhichaphenomenonispresentinapopulation,orthe
extenttowhichthenullhypothesisisfalse,orboth.(chap.11,23)
Efficacy(ofatechniqueortreatment)Inresearch,appliestoevaluatingthedegreetowhichanexperimentaltreatmentorinterventionhasthedesiredor
predictedeffect.(chap.6,7)
ElementsObjectsorpersonsofinteresttobesampledinaparticularstudy,usuallytheindividualparticipants.(chap.10)
EmpiricalDataInformationbasedonobservationsorexperience.Oftencontrastedwiththeoreticalinformation.(chap.4)
EpistemologyStudyortheoryofhowknowledgeisattainedandthelimits,validityandreliabilityofknowledge.Varianceexistsindefinitionsofwhatknowledgeis
andhowitiscreatedordiscovered.Seechapter2foradiscussionoftwopointsofview:thepositivistandconstructivist.

Page412

ErrorVarianceInanANOVA,thepurposeofanalyzingvarianceistofactorouttheamountofvariancebetweensubjectswithintheirowngroupstomore
accuratelydeterminethevariancebetweengroups.Theamountofvariancewithinthegroupistheerrorvariance.(chap.14,15,17,18)
ExperimentalApproachesIncludesbothrandomizedexperimentalandquasiexperimentalresearchdesigns.Theyaregenerallystrongininternalvalidity,soone
canbemorecertainthanwithanyotherapproachesaboutattributingcausetotheindependentvariables.However,quasiexperimentaldesignsvarygreatlyinstrength
and,thus,internalvalidity.(chap.5,7)
ExternalValidityThisaspectofresearchvalidity,thevalidityofawholestudy,dealswiththequestionofgeneralizability.Ithastwoaspects,population(sample
representativeness)andecological(naturalnessoftheresearchsettingandprocess).(chap.10,24)
ExtraneousVariablesThesevariablesarenotofinterestinaparticularstudybuttheycouldinfluencethedependentvariable,forexample,environmentalfactors,
timeofday,andsoon.(chap.4)
F
FStatistic(Fratio)IntheANOVAsourceofvariationtable,computinganFvalueisthefinalstep.Itisthevariancebetweengroups(treatmentpluserror)
dividedbythevariancewithineachgroup(error).Thisratioisusedtoassesswhetheradifferencecouldhaveoccurredbychance.(chap.14,15,17,18)
FaceValidityOneaspectofmeasurementvalidity.Thecontentoftheinstrumentappearstobeappropriateforthepurposeofmeasuringtheintendedconstruct.
Notacceptedasascientificmethodofvalidation,butoftenhelpfulinconvincingthepublic.(chap.20)
FactorAnothertermforindependentvariable(e.g.,singlefactordesign=designwithoneindependentvariable).(chap.14,15)
FactorAnalysisAcomplexassociationalstatisticusedprimarilytoreducearelativelylargenumberofitemsorquestionstoasmallernumberofcomposite
variables.(chap.19)
FactorialANOVAAnanalysisofvariancewithtwoormoreindependentvariables.Seealsofactorialdesign.(chap.17)
FactorialDesignAcomplexdesignwithmorethanoneindependentvariable.Designswithtwovariablesarecalledtwofactororfactorialdesigns.(chap.17)
FactorialEvidenceforConstructValidityOnetypeofevidencetosupportthemeasurementvalidityofaconstruct.Factoranalysissupportstheproposed
theoreticalorganizationoftheaspectsofaconstruct.(chap.20)
FieldStudyinanaturalsetting,notinalaboratory.Anexampleisaschoolclassroom.(chap.1)
FrequencyDistributionTabulationthatindicatesthenumberoftimesagivenscoreorgroupofscoresoccurs.(chap.9)
FrequencyPolygonsAfigureorchartthatdiagramsthefrequencydistributionandconnectsthepointsbetweenorderedcategories.(chap.9)

Page413

FriedmanTestAnonparametricanalysisofvariancebyusingrankswhenthereisoneindependentvariablewiththreeormorelevels,andarepeatedmeasuresor
withinsubjectsdesign.(chap.13,15)
G
GeneralizeSeeexternalvalidity.(chap.10,24)
H
HawthorneEffectAspecifickindofthreattoresearchvalidity.ThenameistakenfromtheHawthorneelectricplantatwhichateamofresearchersintendedto
experimentwithhowchangesintheworker'senvironment(betterlighting,morecoffeebreaks)mightcontributetoimprovedproductivity.Withtheseindependent
changes,productivityincreased.Theincreasewasattributedtotheworker'sknowledgethatthetreatmentwasintendedtoresultinapositiveresponsefromthe
subjects.Ironically,subsequentstudiesinwhichaproposedtreatmentwashypothesizedtodecreaseproductivity(poorerlightingorfewercoffeebreaks)alsoresulted
inimprovedproductivity.Thusthetermisnowusedtodescribeanypositiveeffectonadependentvariablethatcanbeattributedtotheparticipant'sknowledgeofthe
treatment,whetherornotthetreatmentisimplementedandwhetherornottheintendedtreatmentisfavorable.Oppositeof"demoralizationeffect."(chap.2,6)
HomogeneityofVariance(Assumptionof...)Althougheachsamplewillhaveindividualmembersthatdifferfromoneanotherintheirmeasuredcharacteristics
(variance),itispresumedthatonesampledoesnothavemorevariationinitsmembersthandoesanothersampleofequalsize.Whenthesamplesareequal,attestof
thedifferencebetweenthemeansofthesamplesisanappropriatemeasureofcomparison.Whensamplesizesdifferbetweengroups,thisassumptionislikelytobe
violatedandanadjustedttest(Welch's)mustbecalculatedtoaccountforthedifferingsizes.Thisisoneofthethreeunderlyingassumptionsthatmustbemettousea
ttestorANOVA.Seealsonormalityandindependence.(chap.13,14,15)
HypothesisApredictivestatementabouttherelationshipbetweentwoormorevariablesthataresearcherintendstostudy.(chap.4)
I
Independence(Assumptionof...)Assumesthat,withineachsample,scoresforthedependentvariableareindependentofeachother.Theperformanceofone
participantisnotaffectedbytheperformanceofanotherparticipant.ThisisoneofthethreeunderlyingassumptionsthatmustbemettouseattestorANOVAand
manyotherstatisticaltests.Seealsohomogeneityofvarianceandnormality.(chap.13,14)
IndependentSamplesSamplesorgroupsthatmeettheassumptionofindependence.Thus,thetwosamplettestisalsocalledtheindependentsamplesttest.
(chap.14)
IndependentVariableThepresumedcauseinastudy.Alsoavariablethatcanbeusedtopredictthevaluesofanothervariable.(chap.4)
InductiveReasoningUsingthelogicofapplyingspecificinstancestoderivegeneralconclusions.Constructivistresearchersarguethattheirapproachisinductivein
contrasttothedeductivenatureoftheclassicalpositivistscientificmethod.Seealsodeductivereasoning.(chap.2)
InferentialStatisticsStatisticsthatallowonetodrawconclusionsorinferencesfromdata.Usuallythismeanscomingtoconclusionsaboutapopulationonthe
basisofdatathatdescribeasample.(chap.11)

Page414

InstrumentValiditySeemeasurementvalidity.(chap.20)
InstrumentationAformordeviceusedtocollectdata.Examplesincludeinterviews,questionnaires,andobservations.(chap.21,AppendixB)
InteractionEffectAlsocalledconditioningeffect,jointeffect,andmoderatingeffect.Thejointeffectoftwoormoreindependentvariablesonadependent
variable.Interactioneffectsoccurwhenindependentvariableshavecombinedeffectsonadependentvariable.Interactioneffectsoccurwhentherelationbetweentwo
variablesdiffersdependingonthelevelofanothervariable.(chap.17,18)
InteritemReliabilityAlsocalledinternalconsistencyreliability.Howalloftheitemsofaninstrumentarerelatedtoeachother.(chap.20)
InternalConsistencyReliabilityWhentheinstrumentisconsistentamongitems.Cronbach'salphaisthemostcommonmeasureofthistypeofreliability.(chap.
20)
InternalValidityOneaspectofresearchvalidity,thevalidityofawholestudy.Theextenttowhichtheresultsofastudycanbeattributedtothetreatmentsrather
thantoflawsintheresearchdesign.Inotherwords,thedegreetowhichonecandrawvalidconclusionsaboutthecausaleffectsofonevariableonanother.(chap.
6,24)
Interrater(Interobserver)ReliabilityThemeasureofreliabilityorconsistencyamongjudges(observers)intoolswhereobservationisperformed.Usuallytwo
ormorejudgesscorecertainepisodesofbehaviorthentheresearchercomputesameasureofagreementamongjudges(e.g.,Cohen'sKappa,correlation,interclass
correlationcoefficient).(chap.20)
IntervalScaleofMeasurementMutuallyexclusivecategoriesthatareorderedfromhightolow.Thecategorieshaveequalintervalsbetweenthem.(chap.9)
InterventionSeetreatment.
InterviewAdatacollectiontechniqueinwhichtheresearcherorinterviewerasksquestionsorallyoftheparticipantwhoanswersorally.Thereareseveralvariants.
(chap.21)
IntraclassCorrelationCoefficient(ICC)Acoefficientusedtoallowtheresearchertocalculateareliabilitycoefficientwithtwoormorejudges.Acriterionfor
usingtheICCisthatthebehaviorbeingratedmustbescaledatanintervallevel.(chap.20)
InverseRelationshipAnegativerelationshipwherepersonswithhigherscoresononevariableortesthavelowerscoresonthesecondvariableortest,andvice
versa.(chap.16)
K
KappaStatisticAmethodofcalculatinganintraclasscorrelationcoefficientwhenthedataarecategoricalornominal.(chap.20)
Kendall'sTauAstatisticaltestusedinassociationalresearchapproacheswhereboththeindependentanddependentvariablesareatleastordinaloroneofthe
assumptionsismarkedlyviolated.(chap.13)
KruskalWallis(onewayANOVA)AnalternativetoaparametriconewayANOVA,anonparametricanalysisofvarianceusedforstudieswithone
independentvariablewithtwoormorelevels,andadependentvariablethatisordinalorranksorwhenANOVAassumptionsaremarkedlyviolated.(chap.13,14)

Page415

KuderRichardson20(KR20)Amethodofdetermininginteritemreliabilityontestswhereitemsarescoreddichotomously.(chap.20)
LaboratoryResearchAnyofseveralmethodsofisolatingsubjectssoastocontrolextraneousvariables.Sometimesconsideredtobesynonymouswith
experimentalresearch,butlabresearchcanuseanyapproach.Subjectscanbeisolatedfromcontextsthatmightinfluencetheirbehaviorhence,theresearchcan
focusonthoseindependentvariablesofinterest.(chap.1)
LawsofProbabilityTheprobabilityofanevent,p(X),istheratioofthenumberofoutcomesthatincludeXtothetotalnumberofpossibleoutcomes.(chap.11)
LevelsofaVariableThedifferentvalues,categories,orconditionsofavariable,usuallyanindependentvariable(e.g.,interventionversuscontrolormarried
versuswidowedversusdivorced).Alsothevaluesofthedependentvariableusuallytherearemany(e.g.,scoresonatest).Thetermleveldoesnotnecessarilyimply
thatvaluesareordered.(chap.4)
LevelsofMeasurementSimilartoscalesofmeasurement,butmakingdistinctionsthataremoreimportantfortheselectionofstatistics.Thelevelsare
dichotomous,nominal,ordinal,andnormalornormallydistributed.Seeeachterm.(chap.9)
LikertTypeRatingScaleIndividualitemswhoseresponsesindicatethedegreeofagreement:usuallystronglyagree,agree,neutral,disagree,stronglydisagree.
(chap.21)
LinearRegressionBasedonanequationtopredictthedependentvariablefromknowledgeoftheindependentvariable,providingthatthecorrelationbetweenthe
twovariablesisknown.Linearregressionisalsousedtoexaminethelinearrelationshipbetweentheindependentanddependentvariable.Seecorrelation.(chap.16)
LiteratureSeeresearchliterature.
LogicalPositivistParadigmAlsocalledthequantitativeordominantparadigm.Intheextreme,assertsthatrealityandtrutharesingularandknowable.The
researchapproachthatadherestostrictscientificmethodologyinvolvingdeductivehypothesistestingthroughexperimentation,quantitativedatacollection,and
statisticalanalysesbasedontheapproachofnaturalscienceresearch.(chap.2)
LogisticRegressionAcomplexassociationalstatisticaltechniqueusedtopredictadichotomousdependentoroutcomevariablefromacombinationofseveral
independentvariables,someorallofwhichcanbedichotomous.(chap.19)
LongitudinalDesignOneinwhichparticipantsareassessedattwoormoreagestodeterminematurationortheeffectofsometreatmentovertime.Seealsocross
sectionaldesigns.(chap.12)
M
MainEffectInatwoormorefactorial(ANOVA),itindicatesthe"effect"ofeachindependentvariableseparately.Thisisthe"maineffect"ofeachindependent
variablebyitselfonthedependentvariable.(chap.14,15,17,18)
MannWhitneyUTestAnonparametrictesttodetermineifasignificantdifferenceexistsbetweentwogroups.Commonlyusedinsteadofattestforindependent
groupsorsampleswhenthedependentvariabledataisordinalortheassumptionofnormalitydoesnotapply.(chap.13,14)

Page416

McNemarTestAnonparametrictestusedinadesignwherethereisoneindependentvariable,withtwolevels,andtheparticipantsundergobothconditions,or
pairsofparticipantshavebeenmatchedonarelevantvariable.Thedependentvariableisnominalorcategoricaldata.Frequenciesaretheunitofmeasure.(chap.13,
15)
MeanAlsocalledarithmeticaverage.Derivedbysummingallscores,thendividingbythenumberofallscores.Mostcommonmeasurementofcentraltendency.
(chap.9)
MeanSquare(MS)AnessentialpartofanANOVAsourcetable,calculatedbydividingthesumofsquaresbythedegreesoffreedom.(chap.14,15,17,18)
MeasurementTheassignmentofnumeralstoobjectsoreventsaccordingtorules.(chap.9)
MeasurementReliabilityTheextenttowhichameasurementisconsistent.Therearefourtypesofreliabilityindexes:testretest,parallelforms,internal
consistency,andinterrater.Eachprovidesdifferentinformationaboutthereliabilityorconsistencyofameasureortest.(chap.20)
MeasurementScalesAccordingtoStevens,therearefourscalesorlevelsofmeasurement:nominal,ordinal,interval,andratio.Seeeachterm,alsolevelsof
measurement.(chap.9)
MeasurementValidityAnassessmentofhowwellaninstrumentmeasuresaconstructforagivenpurposeinagivenpopulation.Contrastwithresearchvalidity.
(chap.20)
MeasuresofCentralTendencyMeasuresofthecenterofafrequencydistribution.Seemean,median,andmode.(chap.9)
MedianThemiddlescore.Inadistributionofscores,thepointbelowwhich50%ofallscoresfall,derivedbysimplycountingthenumberofscoresaboveand
belowthatpoint.(chap.9)
MetaAnalysisQuantitativeproceduresforsummarizingorintegratingthefindingsobtainedfromanumberofrelatedstudies.Themetaanalystusestheresultsof
individualresearchprojectsonthesametopic(perhapsstudiestestingthesamehypothesis)byusingeffectsizestocombinetheresults.(chap.23)
MixedDesignsMixeddesignshaveatleastonebetweengroupsindependentvariableandatleastonewithinsubjectsindependentvariable.Theyhaveaminimum
oftwoindependentvariables.(chap.12)
ModeThemostfrequentlyoccurringscoreinadistribution,derivedbysimplycountingwhichscoreoccursmostoften.(chap.9)
MultipleRegressionAcomplexassociationalstatisticusedtopredictanormallydistributedoutcomeordependentvariablefromseveralnormallydistributedor
dichotomousindependentpredictionvariables.(chap.19)
MultivariateAnalysisofVariance(MANOVA)Acomplexdifferencestatisticusedtoconsiderseveralrelateddependentvariablesatatime,whenthereare
oneormoreindependentvariables.(chap.19)
N
NaturalistParadigmOftencalledqualitative.Seeconstructivistparadigm.(chap.1,2)

Page417

"NoTreatment"EffectInstudiesinwhichtheeffectsofoneormoretreatmentsareobserved,acontrolgroupthatdoesnotreceiveatreatmentisusefulto
determinewhetherornotachangemayoccurwith"notreatment"atall.Addingacontrolgroupmayincreaseinternalvaliditybyincreasingthecontrolofexperience
orenvironmentalvariables.(chap.5)
NominalScaleNumeralsassignedtoeachcategorystandforthenameofthecategory,buthavenoimpliedorderorvalue.Allmembersofacategoryare
consideredthesamebutdifferentfromthoseinothercategories.Examples:Gender,schoolcurriculum,haircolor,maritalstatus.(chap.9)
NonDirectionalHypothesisAresearchhypothesisthatdoesnotpredictthedirectionoftherelationshipbetweenvariables.Usesatwotailedtesttorejectthe
nullhypothesis.(chap.11)
NonparametricStatisticsAlsocalleddistributionfreestatistics.Inferentialstatisticsthatdonotassumethesamplecomesfromapopulationthatisnormally
distributed.Thechisquaretestisprobablythebestknownexample.Seeparametricstatistics.(chap.13,14,15,16)
NonprobabilitySamplingSelectingparticipantsfromthegeneralpopulationsuchthateachmemberofthegeneralpopulationhasanunknown(andusually
unequal)chanceforinclusioninthestudysample.Conveniencesamplesarecommonexamplesofnonrandomselection.Canaffectexternalvalidityofthestudy,but
doesnotnecessarilyaffectinternalvalidity.(chap.10)
NormalCurveAtheoreticalcontinuousprobabilitydistributioninwhichthehorizontalaxisrepresentsallpossiblevaluesofavariable.Thetotalareaunderthe
curveis1.0.Thescoresonthisvariableareclusteredaroundthemeaninasymmetrical,unimodalpatternknownasthebellshapedornormalcurve.Inanormal
distribution,themean,medianandmodeareallthesame.(chap.9)
NormalLevelofMeasurementAvariablewithorderedlevelsorcategoriesonwhichscoresaredistributedapproximatelylikethenormal(bellshaped)
distributioninthepopulation.(chap.9)
Normality(Assumptionof...)Assumesthatthescoresofthedependentvariablearenormallydistributedineachofthepopulationsfromwhichsamplesare
selected.OneofthethreeunderlyingassumptionsthatshouldbemettouseattestorANOVA.Seealsohomogeneityofvarianceandindependence.(chap.14,15,
17,18)
NullHypothesisThehypothesisthattwoormorevariablesarenotrelatedorthattwoormoremeansarenotdifferentinthepopulation.Inaccumulatingevidence
thatthenullhypothesisisfalse,theresearcherindirectlydemonstratesthatthevariablesarerelatedorthatthemeansaredifferent.Thenullhypothesisiscentralto
hypothesistesting.(chap.11)
O
ObjectiveSeeobjectivity.
ObjectivityTreatingordealingwithfactswhilevoidofpersonalfeelingsorprejudices.(chap.1,2,21)
ObservedScoreAscoreobtainedfromanysubjectonaparticularinstrument.Accordingtoclassicaltesttheory,theobservedscoreiscomposedofthetrue
scoreanderror.(chap.20)

Page418

OneGroupPosttestOnlyDesignInthispoorquasiexperimentaldesign,thetreatmentisgiventoonegroupofsubjects,andthenaposttesttakesplace.The
problemwiththisdesignisthatitdoesnotsatisfyeventheminimumconditionforaresearchproblem,whichistoinvestigatearelationshiporcomparison.Thereis
onlyoneleveloftheintervention,soitisnotavariable.Thevalueofthisdesignisthatitmayserveaspilotdataforafutureexperiment.(chap.7)
OneGroupPretestPosttestDesignApoorquasiexperimentaldesign.Thisdesignprovidesanobservationintheformofapretest,thentheinterventionis
given,andlast,asecondobservationintheformofaposttestisrecorded.Thisdesignisbetterthantheonegroupposttestonlydesignbecauseyoucancomparethe
preandposttest,butwithoutacontrolorcomparisongroup,youcannotknowifthedependentvariablewouldhavechangedwithouttheintervention.(chap.7)
OneWayANOVAAnanalysisofvariancetousewhenyouhaveonebetweengroupsindependentvariablewithtwoormorelevels.(chap.14)
OperationalDefinitionDescribesordefinesavariableintermsoftheoperationsusedtoproduceitorthetechniquesusedtomeasureit.(chap.4)
OperationalizeTodefineaconceptorvariableinsuchawaythatitcanbemeasuredoridentified(or"operatedon").Whenyouoperationalizeavariable,you
answerthequestions:HowwillIknowitwhenIseeit?HowwillIrecordormeasureit?(chap.4)
OrderedVariableVariablethathasasetofvaluesthatvaryfromlowtohighwithinacertainrange.(chap.4)
OrdinalInteractionIngraphingtheresultsofafactorialANOVA,whenaninteractionFissignificantandwhenthelinesthatconnectthecellmeansfortwo
separategroupsdonotintersect,theinteractionisordinal.Whentheyintersectandaresignificant,theinteractionisdisordinal.Seedisordinalinteraction.(chap.17)
OrdinalScaleAlsocalledordinallevelofmeasurement.Categoriesareorderedfromlowtohighinmuchthesamewayonewouldranktheorderinwhichhorses
finisharace(first,second,third,...last),andtheintervalsbetweenthecategoriesarenotequal.Examples:1stplace,2ndplace,lastrankedpreferences.(chap.9)
OrthogonalStatisticallyindependentandperpendicular.Thefactorsorcomponentsinafactoranalysis.(chap.19)
OutcomeTheeffectonthedependentvariableoftheindependentvariable.(chap.4)
P
pThelikelihoodthatastatisticalresultwouldhavebeenobtainedbychancealone.Thisactualprobabilityvalue(pvalue)iscomparedbyaresearcherwitha
presentcriterion,alpha,todeterminewhethertheresulthas"statisticalsignificance."Ifthepvalueissmallerthanalpha(usually.05),theresultisstatisticallysignificant.
(chap.11)
PairedSamplesAlsocalledrelatedsamplesandwithinsubjects.Referstoatypeofdesigninwhichtwoormoremeasuresofthesamevariableareobtainedon
onesampleofsubjects,orsimilarmeasuresareobtainedfrompaired(e.g.,twins,couples,sibs)ormatchedsubjects.(chap.15)
ParadigmAphilosophicalormentallensthroughwhichtheworldisviewed,basedonthesharedbeliefsofacommunity.Agivenparadigmisawayofthinking
about

Page419

andconductingresearch.Itwillinfluencewhatresearchquestionsorhypothesesaregenerated,howtheywillbeexamined,andhowgathereddatawillbe
interpreted.Thetwoparadigmsexaminedinthetextarepositivistandconstructivist.(chap.2)
ParallelFormsReliabilityAlsocalledequivalentoralternateformsreliability.Reliabilityisassessedwithtwopresumablyequivalentversionsofthesame
instrument.(chap.20)
ParametricStatisticsTechniquesdesignedforusewhendatahavecertaincharacteristicsusuallywhentheyapproximateanormaldistributionandare
measurablewithintervalorratioscales.(chap.13,14,15,16)
ParticipantsPeopleofinterestinaparticularstudyformerlycalledsubjects.
PearsonProductMomentCorrelation(r)Degreeoflinearrelationshiporassociationbetweentwovariablesthatarenormallydistributedandmeetother
assumptions.Forexample,relationshipbetweenheightininchesandweightinpounds.(chap.16)
PercentageAgreementMethodsDeterminesinterraterreliabilitybyhavingtwoormoreratersobservebehaviorsandthencomputeapercentage.(chap.20)
PhiCoefficientAtypeofcorrelationcoefficientthatisusedwhenbothvariablesaredichotomous.(chap.13,16)
PopulationThegroupofinterest,whichisusuallylargerthanthesample.Seetheoreticalpopulationandaccessiblepopulation.(chap.10)
PopulationParameterAmeasureofavariablewithinatheoreticalpopulation.(chap.11)
PositivistParadigmOftencalledthequantitativeframework,theinvestigatorestablishesadetailedplanpriortothestudy,whichincludesthesubjectsofthe
investigation,howtheyareselected,thetreatment,andhowthetreatmentismeasured.(chap.1,2)
PostHocTestLiterally,aLatinphrasemeaning"afterthis,"aposthoctestisanecessaryfinaltestinmanyanalyses.Whenanoveralloromnibusstatisticis
identifiedasbeingsignificant,aposthoctesthelpsidentifywherethesignificanceoccurs(e.g.,aTukeyHonestlySignificantDifferenceposthoctestmakesevery
possiblecomparisonofgroups,twoattime).(chap.14)
PosttestonlyControlGroupDesignOneoftherandomizedexperimentaldesigns.Becausethereisarandomassignmentofsubjectstogroups,onecanassume
thatthetwogroupsareequivalentpriortotheintervention.Therefore,iftherearedifferencesonthedependentmeasurefollowingtheintervention,andallother
variablesareequalorcontrolled,itcanbeassumedthatthedifferencesareduetotheinterventionandnotduetodifferencesinsubjectcharacteristics.(chap.7)
Power(...ofaStatistic)Poweristheabilityofastatistictodetectatruerelationship.Parametricstatisticaltestsaremore"powerful"thannonparametrictests
becausethereisahigherprobabilitythatparametrictestresultswillleadtheresearchertocorrectlyrejectthenullhypothesiswhenitshouldberejected(thatis,when
itreallyisfalse).ThelesslikelytheprobabilityofmakingatypeIortypeIIerror,themorepowerfulthetest.(chap.23)

Page420

PredictiveEvidenceofCriterionValidityTheextentthatonecanpredicthowasubjectwilldoonthecriterionmeasureinthefuturebasedonascoreonthe
instrumenttobevalidated.(chap.20)
ProbabilityThelikelihoodthataparticulareventorrelationshipwilloccur.Theproportionalexpectationofagivenoutcome.Valuesforstatisticalprobabilityrange
from1.0to0.(chap.11)
ProbabilitySamplingOneofseveralmethods(e.g.,simplerandom)ofobtaininganonbiasedselectionofsubjectswhereeverysubjectorelementinthe
populationhasaknown,nonzerochanceofbeingselected.(chap.10)
PurposiveSamplingSubjectsarehandpickedfromtheaccessiblepopulationandarejudgedtobeappropriateorinformativeforthepurposeofthestudy.(chap.
10)
Q
QualitativeDataSubjectiveobservationsthataredifficulttoscoreorclassify.Suchdataaretypicallycollectedthroughopenendedinterviews,observations,and
documents.(chap.1,2)
QualitativeResearchReferstoaparadigmorapproachtoresearchthatiscontrastedwithquantitativeresearch.Seealsoconstructivist.(chap.1,2)
QuantitativeDataObservationsofphenomenon,attributesorbehavior,whichcanbenumericallyscored,rated,orscaled.(chap.1,2)
QuantitativeResearchResearchwithinthepositivistframeworkthatisusuallyhandlednumerically.(chap.1,2)
QuasiexperimentalApproachAtypeofresearchapproachforconductingstudiesinfieldorreallifesituationswherethereisanactiveindependentvariablethat
theresearchermaybeabletomanipulate,buttheresearchercannotrandomlyassignsubjectstocomparisonandexperimentalgroups.(chap.5,7)
QuestionnaireAgeneraltermforadatacollectiontechniqueinwhichrespondentsansweraseriesofquestionsinwriting.Therearemanyvariantsofthetypesand
waysthatquestionsareasked(orallyversusinwriting)andthetypesofresponsesrequested(openended,multiplechoice,etc.).Thistermincludessingleusesurveys
aswellasstandardizedinventories.(chap.21)
QuestionnaireItemAsinglequestionoritemonthequestionnaire.(chap.21)
QuotaSamplingAnattempttomakethesamplerepresentativebyassigninginterviewersquotasforeachofseveraldemographicvariables(e.g.,gender,race,
censustract).Acertaindegreeoflatitudeisgiventointerviewerabouthowtheyfillthequotas.(chap.10)
R
RandomAssignmentofParticipantstoGroupsPuttingsubjectsintoexperimentalandcontrolgroupsinsuchawaythateachindividualineachgroupisassigned
entirelybychance.Eachsubjecthasanequalprobabilityofbeingplacedineachgroup.Usingrandomassignmentreducesthelikelihoodofbias.(chap.5)
RandomAssignmentofTreatmentstoGroupsInquasiexperiments,onewaytocompensateforinabilitytorandomlyassignparticipantstogroupsisto
randomlyassigntheinterventionandcontrolconditionstointactgroups.(chap.7)
RandomOrderofConditionsorTreatmentstoEachParticipantAwithinsubjectsdesignisconsideredrandomizedexperimentalifthetreatmentorderis
randomized.(chap.12)

Page421

RandomSelectionorSamplingSelectingagroupofsubjects(asample)forstudyfromalargergroup(population)sothateachindividual(orotherunitof
analysis)ischosenentirelybychance.Arandomsampleisnotthesamethingasahaphazardoraccidentalsample.(chap.5,10)
RandomizedExperimentalResearchApproachMembersofasamplearerandomlyassignedtotwoormoregroupsinwhichanindependentvariableis
manipulatedanditseffectonadependentvariableisobserved.Ifwelldesigned,itprovidesthebestevidenceaboutcause(s)ofthedifferencesonadependent
variable.Theindependentvariablemustbeactive,usuallymanipulatedbytheresearcher.Thegreateststrengthisusuallyhighinternalvalidity.Thegreatestweakness
maybeexternalvalidity,bothpopulationexternalvalidity,ifthesampleisnotrepresentative,andecologicalexternalvalidity,ifthesettingisartificial.Includesseveral
specificdesigns.(chap.5,7)
RankPutinorderoffirst,second,third,fourth.Notethedifferencefromrate.(chap.9)
RateToindicateextentoramount(e.g.,ofagreementsuchasstronglyagree,agree,neutral).(chap.9)
RatingScaleAmeasuringinstrumentusedtoobtainselfreportsorobserverratingsoftheamountorextentofabehavior,attitude,etc.SeeLikerttyperating
scale,summatedratingsscale,semanticdifferentialscale.(chap.21)
RatioScaleCategoriesthathaveequalintervalsandhaveatruezero.Examples:Length,weight,dollarsandmanyotherphysicalmeasures.(chap.9)
RealityAperceptionofwhatisnotartificialorimaginarytoaparticularperson.PositivistviewThereisasingletangiblerealityfragmentableintoseparate
variablesandprocesses.ConstructivistviewTherearemultipleconstructedrealitiesthatcanbestudiedonlyholistically.(chap.2)
RelatedSamplesSeepairedsamples.
ReliabilitySeemeasurementreliability.Consistencythroughoutaseriesofmeasurements.Totheextentatestcontainsunsystematicvariation,thelowerreliability.
(chap.20)
ReliabilityCoefficientAccordingtoclassicaltesttheory,theratioofthevarianceoftruescorestothevarianceofobservedscores.(chap.20)
RepeatedMeasuresDesignsSeewithinsubjectsdesigns.
RepresentativeSampleAsamplethatrepresentsthepopulation(i.e.,asmallreplicaofthepopulation)withallthekeyvariablesinthesameproportionsasthe
wholepopulation.(chap.10)
ResearchSystematicanddisciplinedmethodofgainingnewinformationoransweringquestions.Design,methods,andconclusionsmustadheretologicaland
verifiablestandards.Duplicationoftheresearchbyothersshouldyieldconsistentconclusions.(chap.1)
ResearchLiteratureThescholarlywriting(usuallypublished)aboutagivenresearchabletopic.Includesempiricalresearchreports,theoreticalarticles,and
reviewsofresearch.(chap.4)

Page422

ResearchProblemTheguidinginterrogativestatementthatdetermineswhatissueswillbeexaminedindesigningaresearchprojectandthathelpstoidentifythe
dependentvariable(s).Directstheresearchertowardacriticalreviewofconceptsandtheoriesandaclearidentificationofrelatedmajorresearchissues.Helps
developclearandlogicalrationalesforaformalhypothesis.(chap.4)
ResearchQuestionStatingtheresearchproblemintheformofaquestion.Researchquestionsaremorespecificthanresearchproblemsandtheremaybeseveral
questionstohelpdetermineananswertoasingleresearchproblem.(chap.4,5)
ResearchValidityThequalityormeritofthewholestudy.(chap.6,10,23,24)
ResponseRateThenumberofpeopleinterviewedorrespondingdividedbythenumberofpeoplesampled.(chap.10)
S
SampleAselectedpartofalargergroupofparticipantstakenwiththeintentofgeneralizingfromthesmallergrouporsampletothetheoreticalpopulation.(chap.
10)
SampleStatisticsStatisticsderivedfromanalyzingdatafromasample.Inferentialstatisticsusethesenumberstomakegeneralizationsaboutthetheoretical
population.(chap.11)
SamplingTheprocessofselectingpartofalargergroupofsubjectswiththeintentofgeneralizingfromthesamplegroup,calledthesample,tothepopulation.
(chap.10)
SamplingDesignTheprocessbywhichtheselectedsampleischosen.(chap.10)
SamplingDistributionofMeansThedistributionofthemeanvaluesofallthepossiblesamplesofnsizedrawnfromthetheoreticalpopulation.(chap.11)
ScalesofMeasurementSeemeasurementscales.(chap.9)
ScientificMethodFromtheLatin''scientia"=knowledgeandGreek"methodos"=way,aviewheldbypositiviststhatknowledgecanbeattainedthrougha
processoflogicalinquiry.Theclassicfivestepsare(a)observetherepeatedoccurrenceofaphenomenonandformulateatheorythatexplainsit,(b)fromthetheory,
deviseageneralhypothesisthatpredictsthenecessaryconditionsunderwhichthespecificphenomenonwilloccur,(c)fromthehypothesis,deduceother
consequences(predictotherspecificoccurrences),(d)deviseexperimentstotestwhetherornottheseconsequencesoccur.Iftheconsequencesoccur,thenthe
hypothesisisconsideredtrueandthetheoryisvalid.Earlyinthetwentiethcentury,alogicalfallacywasrecognized(byPopper)inthefinalstepofthismethod:the
theoryhasnotbeenproven,rather,ithasonlybeennotdisproved.Inotherwordsithasonlybeenshowntoholdtrueforthecasesobserved.Logicrequiresthatif
onecounterexamplecanbegenerated,thentheconclusioncannotbetrue.Therefore,constructivistscanobjectthatageneralizationcannotbemadeuntilallpossible
cases,notjustthoseexamined,havebeenproven,whichisusuallyimpracticalorimpossible.Thishasledtoamorecarefulapproachtoexternalvalidationby
positivistsandtoavarietyofalternativeresearchapproachesbyconstructivists.(chap.2)
ScreeningExcludingsubjectsthatdidnotmeetthestatedpurposeofthestudy.(chap.10)
SelectedSampleGroupofsubjectswhoareselectedbytheresearcherandaskedtoparticipateinthestudy.(chap.10)

Page423

SelfAssignmentorSelectionOnceasampleisselectedfromthegeneralpopulation,subjectsareallowedtoassignthemselvestothestudyortreatmentgroupin
whichtheychoosetoparticipate.Compromisesinternalvalidity.Seerandomassignment.(chap.5,6)
SelfReportParticipantsinastudyreporttotheresearcher(inwritingororally)abouttheirattitudes,intentions,orbehavior.(chap.1,21)
SemanticDifferentialScaleSubjectsrateconceptssuchas"myjob"onsetsofbipolaradjectivepairs.TheseincludeActivitypairs,suchasactivepassive,fast
slowEvaluativepairs,suchasgoodbad,valuableworthlessPotencypairs,suchasstrongweak,largesmall.(chap.21)
SignificanceLevel(alphalevel)TheprobabilityofcommittingatypeIerror.Theprobabilityofrejectingatruenullhypothesis.(chap.11)
SimpleRandomSamplingAsampleinwhichallsubjectsorelementshaveanequalandindependentchanceofbeingincludedinthesample.(chap.10)
SingleFactorDesignThedesignofastudywithoneindependentvariable.(chap.4,5,14,15)
SnowballSamplingBuildingasamplefromthereferencesofafewsubjectsthatmeetthecriteriaofthestudy.(chap.10)
SourceTable(SourceofVarianceTable)Inanalysisofvariance,atableisgeneratedthatenumeratesthesumsofsquareddeviations(SS)fromthemeanofeach
group,thedegreesoffreedom(df)foreachgroup,themeansquares(MS)foreachgroup,andfinally,anFratioiscalculated.(chap.14,15,17,18)
SpearmanRankOrderCorrelationAnonparametrichypothesistestingprocedurethatusecontinuousdatathathasbeenranked.ToperformtheSpearman
correlationcoefficient,rankthedataasinothernonparametrictestsandthenperformaPearsonproductmomentcorrelationontheranks.Symbol:r(s)Seealso
correlationandPearsonproductmomentcorrelation.(thap.13,16)
SplitHalfReliabilityProcessofdetermininginternalconsistencybycorrelatingonehalfofinstrumentwiththeotherhalfofinstrument.(chap.20)
StandardDeviationAmeasureofthevariance(variability)orspreadofscoresaroundthemeanwithinadistribution.(chap.9)
StandardErroroftheMeanThestandarddeviationofthedistributionofthesamplemeans.(chap.11)
StandardizationTomakeuniform.Usuallyperformedonaninstrumentbyobtainingnormsandproducingamanual.(chap.21)
StatisticallySignificantDifferenceTheprobabilitythatthedifferencebetweenthepopulationparameterandthesamplestatisticoccurredbychanceatalevel
lessthanthepredeterminedsignificancelevel.(chap.11)
StatisticsAsetofproceduresandtheorythatareusedtoanalyzedata.(chap.9)
StrataVariablesusedtodividethepopulationsintosegments.(chap.10)
StratifiedRandomSamplingGroupingthesampleonthebasisofkeyvariablestoensurethatthesamplecontainstheexactproportionofsubjectswiththesekey
variables,asinthepopulation.(chap.10)

Page424

StratifiedSamplingWithDifferentialProbabilitiesofSelectionOversamplingunderrepresentedgroupstohavesimilarsizedgroupstomakereasonable
comparisons.(chap.10)
StrengthofAssociationThedegreeofrelationshipbetweentwo(ormore)variables.Oftentheproportionofthevariabilityinadependentvariableexplainedby
oraccountedforbytheindependentvariable(s).Etasquared,r2,andomegasquaredarecommonmeasuresofstrengthofassociation.(chap.2,11)
StrongPositiveRelationshipThosescoringhighononetestorvariablewillalsoscorehighonthesecondtestorvariableandviceversa.(chap.16)
SummatedRatingScaleEachsubject'sresponsestoseveralratingsareaddedtogether.(chap.21)
SumsofSquares(SS)AnessentialpartoftheANOVAsourcetable.Thesumofsquareddeviationsforallscores.(chap.14)
SystematicRandomSamplingRandomlyselectingthesample,butfirstrandomlyselectingthestartingpointandthensystematicallyselectingeachnthsubject
(suchaseverytenthone).(chap.10)
SystematicVariationVariationfromtheindependentvariableorintervention.(chap.10)
T
ttestforCorrelatedorPairedSamplesAcommonparametricstatisticthatcomparestwolevelsorconditionswithinthesamegroupofparticipants.(chap.13,
15)
ttestforIndependentSamplesorGroupsAcommonparametricstatisticthatcomparestwoseparategroupsbycomputingtheratioofthevariancebetween
groupstothevariationwithingroups.(chap.13,14)
TheoreticalorTargetPopulationAllofthesubjectsofthetheoreticalinteresttowhichtheresearcherwouldliketogeneralize.(chap.10)
TheoryAsetofinterrelatedconstructs(concepts),definitions,andpostulationsthatpresentasystematicviewofphenomenabyspecifyingrelationsamong
variables,withthepurposeofexplainingandpredictingthephenomena.(chap.1,2,4)
TimeSeriesAnalysisAnalysisofchangesinvariablesovertime.Anyofseveralstatisticalproceduresusedtotellwhetherachangeintimeseriesdataisdueto
somevariablethatoccurredatthesametimeorisduetocoincidence.(chap.7)
TimeSeriesDesignAsetofmeasuresofavariablerecordedperiodically,beforeandafteraneventortreatment.(chap.7)
TreatmentAlsocalledinterventionandexperimentalgroup.Thestudygrouporsubjectsforwhichanactiveindependentvariableismanipulatedissaidtohave
receivedatreatment,thatis,theindependentvariablediffersforthatgroupandtheeffectsonthedependentvariableareobserved.(chap.4,5ff)
TrueExperimentalDesignsSeeRandomizedExperimentalDesigns.
TrueScoreTheaverageofaninfinitenumberofobservedscoresforoneindividual.(chap.20)
TukeyHSDApairwiseposthoctestforasignificantoverall/omnibusFtest.Seeposthoctest.(chap.14)
TwobyTwoFactorialDesign(2 2design)Astudycraftedtoexaminetheeffectsoftwoindependentvariablesonasingledependentvariable.Allowsthree
hypothe

Page425

sestobetested(differencebetweentwolevelsofindependentvariableA,differencesbetweentwolevelsofindependentvariableB,andtheinteractionofAand
B.)(chap.17)
TwoFactorANOVAAlsocalledatwowayanalysisofvariance.Aninferentialstatisticthatisusedtocomparegroupmeans(ofgroupsformedfromtwo
independentvariables)ononedependentvariable.ItissimilartoaonewayANOVA,addingasecondindependentvariable.SeeANOVA.(chap.17)
TwoFactorDesignsSeeFactorialDesign.
TypeIErrorAnerrormadebywronglyrejectingatruenullhypothesis.Thismightinvolveincorrectlyconcludingthatthetwovariablesarerelatedwhentheyare
not,orwronglydecidingthatasamplestatisticexceedsthevaluethatwouldbeexpectedbychance.Thisisalsocalledalphaerror.(chap.11)
TypeIIErrorAnerrormadebywronglyretainingorfailingtorejectafalsenullhypothesis.Thisisalsocalledabetaerror.(chap.11)
U
UnsystematicVariationVariationthatresultsfromfactorsotherthantheinterventionorindependentvariable.(chap.20)
V
ValidityReferstothequalityofameasureorobservation(i.e.,measurementvalidity)andalsotothequalityofawholestudy(i.e.,researchvalidity).(chap.2,6,
10,20,23,24)
VariabilityDifferencesbetweenscoresinadistribution.Ifallthescoresinadistributionarethesame,thereisnovariability.Ifthescoresinadistributionareall
differentandwidelyspaced,thevariabilitywouldbehigh.(chap.9)
VariableAcharacteristicoftheparticipantsorsituationofagivenstudythathasdifferentvaluesusedinthatstudy.(chap.4)
VarianceAmeasureofvariabilityordifferencesbetweenscoresinadistribution.Thestandarddeviationsquared.(chap.9)
W
WithinSubjectsDesignsAlsocalledrepeatedmeasuredesigns.Eachparticipantisassessedonallconditionsorlevelsoftheindependentvariable.(chap.12,15)
YatesCorrectionAcorrectionusedinthechisquarecomputationtoadjustforinstancesinwhichthereisonly1degreeoffreedom(22crosstabulation).
Effectivelydecreasesthechisquarevalue.(chap.14)

Page427

APPENDIXB
ANATOMYOFARESEARCHARTICLE
Thereisnouniqueformatthatisusedbyalljournalsindisseminatingresearchinformation.Eachdisciplinehassomepeculiaritythatiscommontothatdiscipline.In
addition,researchformatsmaydifferbythetypeofresearchreported.Theformatforqualitativeresearchstudiessuchasethnographiesorcasestudiesisdifferent
fromthatforexperimental,quasiexperimental,orsurveyresearch.TheorganizationofthisappendixisprimarilystyledafterthePublicationManualofthe
AmericanPsychologicalAssociation(APA,1994)andWilkinsonandtheTaskForceonStatisticalInference(1999).Forthemostpart,APAformatisacceptable
forresearchjournalsandrequiredformanypublicationsinthesocialsciencesandeducation.Theresearchformatpresentedherehassevenparts:title,abstract,
introduction,method,results,discussion,andreferences.
Title
Thetitleshouldbebrief(APA,1994,recommendsatitlelengthbetween12and15words),yetdescribewhatyouhavestudied.Thetitleisalsoasellingpointforthe
article.Fewresearchershavethetimeorenergytoreadeveryarticleinjournalstowhichtheysubscribe.Therefore,iftheyarelikeus,onreceivingthejournalthey
skimthetableofcontentstoseeifthereareanyarticlestheywishtoread.Atthispoint,thetitleistheonlysellingfeatureofyourarticle.Therehavebeensome
memorabletitles,like"RememberThatOldTheoryofMemory,WellForgetIt"and"BeaglesandLocks."Evenstatisticianshavehadagoatitwithsuchtitlesas"On
theSignificanceofEffectsandtheEffectsofSignificance''(Cooper,1981)"TheReligionofStatisticsasPracticedinMedicalJour

Page428

nals"(Salsburg,1985)McNemar'sfamousaddress,"AtRandom:SenseandNonsense"(1960)andCohen's(1994)"TheEarthisRound(p<.05)."Thelatterwas
influentialinmotivatingtheAPATaskForcetoproducetheirrecentreport,whichguidedmuchofthisappendix.Ourfavoriteis,"TheUnicorn,theNormalCurve,and
OtherImprobableCreatures"(Micceri,1989).Forthemostpart,however,youcannotgowrongbybeingbriefandtothepoint.
Abstract
Oncethetitlecatchestheconsumer'sinterest,theabstractisthenextpartofthearticlethatusuallygetsread.Theabstractfollowsthetitleandprovidesasummaryof
thepaper.Liketitles,abstractsarelimitedinlength.APA(1994)suggestsanabstractofnomorethan120wordsforanempiricalstudyandnomorethan100words
forareviewortheoreticalarticle.Anabstractdescribesbrieflythepurpose,design,andresults(inasentenceortwo)ofthestudy.Likethetitle,theabstractisalsoa
sellingpointforthepaper.Wehavereviewedmanyresearchstudiesthatresultedincommentstotheauthor(s)suchas,"Noonewillreadthepaperbecausethe
abstractisnotrepresentativeofwhatthestudyfound,"or"thewrongcontenthasbeenhighlighted."Itisimportanttocorrectlyrepresentthepaperintheabstract.
Introduction/LiteratureReview
Inthissectionwedescribewhatshouldbeintheintroductorysectionofaresearcharticle.Howonegetsthatinformation(e.g.,fromalibrarysearch)isnotthe
purposeofthissection.Thereareanumberofgoodbooksonhowtodoaliteraturesearch,includingFink's(1998)ConductingResearchLiteratureReviews:
FromPapertotheInternet.NoticethatwehaveputaslashbetweenIntroductionandLiteratureReview.Inmostjournals,theintroductionandliteraturerevieware
togetherinonesection,whichisuntitledinAPAformat.However,someofthejournalsyoureadseparatetheintroductionandtheliteraturereview.Mostmaster's
thesesanddoctoraldissertationshaveseparatechapterstitledIntroductionandLiteratureReview.
Whatmaterialshouldgointotheintroductionofajournalarticle?Thefirstparagraphoftheintroductionshouldbeageneralpurposestatementofwhatistobe
accomplished.TheAPAPublicationManualsuggeststhatthefirstparagraphshouldintroducetheproblem.Thereisnothingworsethanstartingajournalarticleby
reading,forinstance,thatSmithandJones(1977)foundonething,andthenUpandDown(1982)foundsomethingelse,andthenHillandDale(1983)supported
SmithandJones(1977),andsoon.Afteraboutthreeorfourpages,youwonderwhattheauthor(s)aredoingthatis,ifyouarestillawake.(Don'teverreada
researcharticlelateatnight,unlessyoucan'taffordsleepingpills.)
Aftertheintroductoryparagraph,startyourliteraturereview.Althoughadegreeofchronologicalorderisnecessary,itshouldonlyhappenaftertheliteratureis
organized,forexample,intostudiesthatsupportyourcase(hypothesis)andstudiesthatcontradictthesefindings.Let'ssaythatweproposeageneralhypothesisthat
studentswhotakeacourseinresearchdesignpriortoa

Page429

courseinstatisticswillbecomebetterresearchersthanthosestudentswhotakethecoursesinthereverseorder.Wewouldtrytoformtwoorthreecategoriesof
articles.Thosearticlesthatsupportourhypothesiswouldformonecategory.Thosearticlesthatopposeourhypothesiswouldformasecondcategory.Athird
categorymightbethosearticlesinwhichtheauthorsfoundnodifferenceintheirresults.Withineachcategorymightbearticlesthatarerelevanttothetopicandothers
thatmerelytouchonthetopic.Thelattershouldonlybelistedassupportingornotsupportingyourhypothesis.Thosearticlesthatarerelevanttothetopicshouldbe
exploredindepth,especiallyforwhyorwhynottheysupportedyourhypothesis.Youareleadinguptowhyyourstudywillmakeadifferenceinrelationtothepast
literature.Whatisitaboutyourstudythatisdifferentfrompreviousresearch?Insummary,onegoodapproachtoaliteraturereview,intheintroductionsectionofa
journalarticle,startsbycategorizingstudies(citingthem)thatsupportordonotsupporttheresearchhypothesis,andthendescribesindepthafewselectstudiesthat
arerelevanttothepresentstudy.Forstudiesontopicsthatarenotwellresearched,anothertechniquewouldbeused:forexample,chronological,bykeyvariables,or
byresearchquestion.
Thelastpartoftheintroductioninaresearcharticleisaformalstatementofthehypothesesorresearchquestions.Thesestatementsshouldbepresentedinoperational
termssothatthereaderknowsexactlywhattheresearcherisattemptingtostudy.
Method
Themethodsectionformostresearcharticlesisdividedintofoursubsections.Theultimatepurposeofthemethodsectionistoinstructthereaderaboutexactlywhat
wasdoneinthestudyandtoallowthereadertoreplicatethestudyunderidenticalconditions.AccordingtotheAPAManual,thesubsectionsincludeParticipants,
Apparatus(orInstruments/Materials),andProcedure.WeliketoaddDesign/Analyses.TheAPATaskForce(Wilkinson&TheTaskForce,1999)recommends
amorecompletedescriptionofthemethodthanhascommonlybeenthecaseinpublishedarticles.Iftheguidelinesareadoptedandimplemented,itmaymeanthat
journalswillpublishfewerbutlonger,morefullydocumentedarticles.
Participants
Thissubsectionshouldcontainathoroughdescriptionoftheparticipants.Wheredidtheparticipantscomefrom?Howdidyoucontactthesample?Didthe
participantsvolunteerfortheproject?Asentenceortwoabouthowinformedconsentwasobtainedneedstobeincluded.
Howdidyousample(select)thesubjects?Weretheyaconveniencesample,ordidyouusearecognizedprobabilitymodeofselection(random,systematic,cluster,
orstratified)?Becausetheinterpretationoftheresultsdependsonthecharacteristicsofthepopulation,itisimportanttodefinethepopulationclearly.Unfortunately,
thisisnotalwaysdone.Thedescriptionofthesamplingproceduresshouldcompriseinclusionorexclusioncriteria,fullinformationabouthowthesamplewasstratified
(ifitwas),andthesamplesizeforeachsubgroup.Aconveniencesampleshouldbeclearlyidentifiedassuch.Some

Page430

timesthecaseforitsrepresentativenesscanbestrengthenedbyshowinghowyoursamplecomparestothepopulationonkeyvariables.
Youshouldprovideinformationabouttheprocessthatledtoyoursamplesizedecision.Apoweranalysis(chap.23)shouldbedonebeforethedataarecollected.
Describetheassumptionsusedandhowthepoweranalysiswasdone.
Afteradescriptionofhowthesamplewasobtained,thenextpartofthemethodsectiondealswithcharacteristicsofthesample.Thesecharacteristicsshouldinvolve
age(averageandrangeorstandarddeviation)andgender.Ethnicgrouping,typeofdisability,socioeconomicstatus,levelofeducation,oracombinationof
characteristicsshouldalsobeincluded.
Apparatus/Instrumentation/Materials
Thissectiondescribesindetailalloftheinstrumentsorteststhatwereusedintheresearch.Ifanapparatuswasinvolved,thentheresearchermustdescribethetypeof
equipment(includingbrandname),accuracy,specificationsrelativetothesubject,andpossiblecalibrationinformation.Iftests,questionnaires,orsurveyswereusedin
thestudy,informationonreliabilityandvaliditymustbeincluded.Wastheinstrumentstandardized,orwasitdevelopedforthisstudy?Iftheinstrumentwas
standardized,wasitusedforpurposessimilartothoseinthisstudyandwerethestandardizationsamplessimilartothoseinthisstudy?Iftheinstrumentwasdeveloped
forthepresentstudy,wasapilotstudyperformed?Iftheinstrumenthasanumberofitems,hasitbeenfactoranalyzed?Samplequestionstodemonstratethecontent
oftheinstrumentshouldbeincluded.Also,thelevelofmeasurementofthedatashouldbegiven.
Eachkeyvariableshouldbecarefullyandexplicitlydefined.Showhowsuchvariablesarerelatedtothegoalsofthestudyandhowtheyaremeasured.The
measurementsshouldfitthelanguageusedintheIntroductionandDiscussion.Naminganabstractvariableisalmostasimportantashowitismeasured,andshouldbe
doneconsistently.Wediscussedthistopicinchapter23asmeasurementvalidityandthegeneralizabilityoftheconstructs.
Procedure
Thissectionisa"blowbyblow"descriptionofhowthestudywasconducted,andisespeciallyimportantforreplicationtooccur.Alsoincludedinthissectionishow
theparticipantswereassignedtothedifferentgroupsunderstudythatis,weretheyrandomlyassignedorweretheyalreadyinanintactgroup?Itisespecially
importantinthissectiontoreportanyinstructionsthatweregivenbytheresearchertotheparticipants.
WilkinsonandTheTaskForce(1999)andthistextemphasizethedistinctionbetweentherandomassignmentofparticipantstointerventionandcontrol(or
comparison)groups,and,ontheotherhand,therandomselectionofparticipantsfromthepopulation.Forresearchintendedtomakecausalinferences,random
assignmentiscriticalbecause"itallowsthestrongestpossiblecausalinferences...'Random'doesnotmean'haphazard.'Randomizationisafragilecondition,easily
corrupted"(Wilkinson&TheTaskForce,1999,p.595).TheAPATaskForcealsorecommendsadescriptionofhowyourandomized,preferablybyusing
publishedtablesofrandomnumbersratherthantrustingacoin,slipsofpaperinahat,orphysicaldevices.

Page431

Iftheparticipantscannotberandomlyassignedtogroups,youshoulddescribehowyouattemptedtocontrolforinitialgroupdifferences.Youshouldtrytodetermine
therelevantcovariatesorconfounds,anddescribeanymethodsusedtoadjustforthem.Alsodescribemethodstoreducebias,includingminimizingdropouts,
noncompliance,missingdata,andexperimenterbias.
Design/Analysis
Thissectionisusuallythelastsubsectionofthemethodsection,butsometextsplacethissectionearlierinthemethodsectionofthearticle,andsometimesthedesign
isdescribedintheintroduction.Inthedesignparagraphofthissubsection,theresearcherfirstnamestheindependentvariableorvariablesandthenumberoflevels
withineachvariable.Thenextpieceofinformationisthetypeofdesign(betweengroups,withinsubjects,ormixedorassociational).Next,theresearcherdiscusses
whethertheindependentvariable(s)isanactive(manipulated)orattributevariable.Thelastpartofthedesignsectionincludesthespecificationofthedependent
variable(s)andthelevel(scale)ofmeasurement.
TheAPATaskForcereport(Wilkinson&TheTaskForce,1999)says:
Makeclearattheoutsetwhattypeofstudyyouaredoing.Donotcloakastudyinoneguisetotrytogiveittheassumedreputationofanother.Forstudiesthathavemultiple
goals,besuretodefineandprioritizethosegoals.
Therearemanyformsofempiricalstudies...includingcasereports,controlledexperiments,quasiexperiments,statisticalsimulations,surveys,observationalstudies,andstudies
ofstudies(metaanalyses).Somearehypothesisgenerating....Somearehypothesistesting....Somearemetaanalytic...Occasionallyproponentsofsomeresearchmethods
disparageothers.Infact,eachformofresearchhasitsownstrengths,weaknesses,andstandardsofpractice.(p.594)

Theanalysisparagraphofthissubsectionspecifiesthetypesofanalysesthatweredoneintheresearch.Theseanalysesaredeterminedbyalloftheinformation
providedinthedesignparagraph.Thecomputerprogram(system)thatwasusedtocarryouttheanalysesmaybespecified,forexample,SPSS(StatisticalPackage
fortheSocialSciences).
WilkinsonandTheTaskForce(1999)recommendsusingrelativelysimplestatisticalanalyses,suchasthosedescribedinchapters14to19ofthisbook,iftheyare
reasonableforyourresearchproblem.TheAPATaskForcereportgoesontostatethat
Theenormousvarietyofmodernquantitativemethodsleavesresearcherswithanontrivialtaskofmatchinganalysisanddesigntotheresearchquestion.Althoughcomplex
designsandstateoftheartmethodsaresometimesnecessarytoaddressresearchquestionseffectively,simplerclassicalapproachesoftencanprovideelegantandsufficient
answerstoimportantquestions.Donotchooseananalyticmethodtoimpressyourreadersortodeflectcriticism.(p.598)

Page432

Results
TheAPATaskForcereport(Wilkinson&TheTaskForce,1999)states:
Assoonasyouhavecollectedyourdata,beforeyoucomputeanystatistics,lookatyourdata.Datascreeningisnotdatasnooping.Itisnotanopportunitytodiscarddataor
changevaluesinordertofavoryourhypothesis.However,ifyouassesshypotheseswithoutexaminingyourdata,youriskpublishingnonsense.(p.597)

TheAPATaskForcealsorecommendsthatyoureportcomplications,missingdata,attrition,andresponserates.Theyalsorecommendthatyouusetechniquesto
assurethatthereportedresultsarenotduetoanomaliesinthedatasuchasoutliersandattrition.TheTaskForcealsorecommendsgraphicalinspectionofdataby
usingscatterplots,boxplots,andotherexploratorytechniquestodetectproblemsanderrorsinthedata:
thetypeof"atheoretical"searchforpatternsthatwearesometimeswarnedagainstingraduateschoolcansaveusfromthehumiliationofhavingtoretractconclusionswemight
ultimatelymakeoncontaminateddata.Wearewarnedagainstfishingexpeditionsforunderstandablereasons,butblindapplicationofmodelswithoutscreeningourdataisafar
gravererror.(Wilkinson&TheTaskForce,1999,p.598)

Theserecommendationsareconsistentwithoursearlierinthebook,anditwouldbedesirabletodiscusstheseissuesinathesisordissertation.However,wewonder
whethermanyeditorswillallowspaceforpresentationofthesetopics.
Theresultssectionisasummaryoftheanalysesofthedatacollectedinthestudy.Aproblemformoststudentsisthelevelofdescriptionintheresults.J.A.G.likesto
tellastorythathappenedtohimwhileingraduateschool."Isubmittedanarticletoaveryprestigiousjournalinthefieldofexperimentalpsychology.SinceIhadjust
completedtwoyearsofgraduatelevelstatistics,ImadesurethatIincludedeverypossiblestatistic,degreesoffreedom,andprobabilitylevel,eventhoughmostofthe
findingswerenotstatisticallysignificant.Eventhoughthearticlewasrecommendedforpublication,majorrevisionswererequested.Oneoftherequestswastoreduce
thesizeoftheresultssectionbyeliminatingmostofthestatisticaldescriptions.Asthesectioneditorputit,'Usethetraumaticoculartest.Ifithitsyoubetweentheeyes
useit.Otherwisegetridofit.'"Thiseditor'scommentiscommon,eventoday,butdoesnotseemconsistentwiththerecommendationsoftheAPATaskForce.
Apictureisworthathousandwords.Werecommendputtingafigureoratablenearthebeginningoftheresults,andthenbrieflydescribingthesignificantresults.
WilkinsonandTheTaskForce(1999)statethat"figuresattractthereader'seyeandhelpconveyglobalresults...itoftenhelpstohavebothtablesandfigures"(p.
601).TheAPATaskForcereportrecommendsthatfiguresbekeptsmallandrelativelysimple,andthattableshavenumbersroundedtoafew

Page433

significantdigitswithnodecimalsunlessreallyneeded.Ontheotherhand,theTaskForcereportdiscussesindetailthefailureofmanypublishedfigurestoinclude
essentialinformation,usuallyabouttheshapeordistributionofthedata.Again,aproblemfacedbypersonsattemptingtopublisharticlesisthateditorshistorically
havepreferredtokeepfigurestoaminimum.Thispotentialrestrictionshouldnotlimitgraphicsintheses,however.
Withregardtostatisticalanalyses,theAPATaskForceguidelinesadmonishresearcherstoassurethattheunderlyingassumptionsarereasonable,giventhedata.
Theypointoutthatstatisticalprogramsprovidetestsformanyassumptions,buttheywarnthatsuchtestscanbeimpracticallysensitive,especiallywithlargesamples
androbusttestssuchasANOVA.
Inaccordwithourrecommendationinchapter11,theAPATaskForcestatesthateffectsizesshouldalwaysbepresentedforprimaryoutcomes,andthatithelpsto
addbriefcommentstoputtheeffectsizesincontext.Theyalsosaythatitisalmostalwaysbettertoreporttheactualpvalueratherthanmerelytosaywhetherthe
resultwasstatisticallysignificantornot.Itisbetterstill,theysay,toreportconfidenceintervals.Also,neverusetheexpression"acceptthenullhypothesis."TheTask
ForcerecommendsCohen's(1994)article,"TheEarthisRound(p<.05)"foradiscussionoftheissueofsignificancetesting.
Aprobleminwritingtheresultssectioniswhethertoincludematerialthatmightbemoreappropriateforthediscussionsection.Iftheoutcomeyouaredescribing
pertainsstrictlytotheanalysis,thenitbelongsintheresultssection.However,ifyouarerelatingyouroutcometootherstudiesthatwerepreviouslydiscussedinthe
introduction,thenthismaterialbelongsinthediscussionsection.
Discussion
Wesuggeststartingthediscussionsectionwithabriefreview(nomorethanoneparagraphinanarticle,probablyafewpagesinathesis)ofthehypotheses,and
whetherornottheywereconfirmed.Themajorpurposeofthediscussionsectionistorelatetheresultstotheresearchhypothesesorquestionswithinthecontextof
theliteraturepreviouslycited.Sometimesanoutcomefromthestudyistotallyunexpectedandthediscussionsectionbranchesintoawholenewliteraturereviewand
hypothesis,indicatingthattheauthorforgotabouttheoriginalintentofthestudyandtheoriginalhypotheses.Don'tletthishappentoyou.Eachhypothesisshouldbe
discussedwithcontinuedreferencetopreviousfindingsfromtheliteraturereview.
TheAPATaskForcereportstatesthatwhenyouinterpretyourresultsthinkaboutwhethertheyarecredible,giventheresultsofpreviousstudiesandtheory.Dothe
designandanalysissuggestthattheresultsaregeneralizable?Thereportthenstatesthat:
Noviceresearcherserreitherbyovergeneralizingtheirresultsor,equallyunfortunately,overparticularizing.Explicitlycomparetheeffectsdetectedinyourinquirywiththeeffect
sizesreportedinrelatedpreviousstudies.(Wilkinson&TheTaskForce,1999,p.602)

Page434

Aswehavestressedthroughoutthebook,youshouldbecautiousaboutinferringcausationfromapproachesthatarenotrandomizedexperimental.Evenwith
randomizeddesigns,yourstatementsshouldbeconsideredcarefully.WilkinsonandTheTaskForce(1999)supportourconclusionwithastrongrecommendation
that"inferringofcausalityfromnonrandomizeddesignsisariskyenterprise.Researchersusingnonrandomizeddesignshaveanextraobligation...toalertthe
readertoplausiblerivalhypotheses"(p.600).Or,wewouldsay,donotmakecausalstatementsatall.
SomediscussionsectionshaveaseparateheadingcalledLimitations.Ourbiasisthatitisnottheroleoftheresearchertoattempttoreviewhisorherownpaper
(exceptinathesisordissertation).Usuallysomeofthelimitationscomeoutwhendiscussingwhyhypotheseswereconfirmedornotconfirmed.Inshort,wedonot
thinkthatthisadditionalsectionisnecessary.However,theAPATaskForcereportstatesthat:
Speculationmaybeappropriate,butuseitsparinglyandexplicitly.Notetheshortcomingsofyourstudy.Remember,however,thatacknowledginglimitationsisforthepurposeof
qualifyingresultsandavoidingpitfallsinfutureresearch.Confessionshouldnothavethegoalofdisarmingcriticism.Recommendationsforfutureresearchshouldbethoughtful
andgroundedinpresentandpreviousfindings.(Wilkinson&TheTaskForce,1999,p.602)

Althoughaconclusionisnotnecessary,theresearchershouldattempttodescribe,inthelastparagraphorsoofthediscussionsection,whatthenextstepinthislineof
researchshouldbe.Obviously,thisnextstepwillbecontingentonwhatwasfoundinthepresentstudy.However,aparagraphofthisnatureleavesthereaderwitha
feelingofwheretheresearcherfeelsfutureresearchisheaded,andalsomaygivethereaderresearchideas.
References
Thereferencesshouldfollowaparticularformat.Forourpurposes,APAformatisused(seeAPA,1994).Onlyreferencesthatwerecitedinthetextareincludedin
thereferencesection.Thismeansthatifmaterialwasreadbutnotincludedinthetext,donotciteit.Notethatinthisbookwehaveusedthepublished,orfinal
manuscript,formofAPAstyleinourreferencelist.WhensubmittingmanuscriptstoanAPAjournaleditor,doublespaceeverythingandindentthefirstlineofa
reference,insteadofusing"hangingindents"aswehave,anduseunderlinesinsteadofitalicsinthereferencesandtext.

Page435

APPENDIXC
WRITINGRESEARCHPROBLEMSANDQUESTIONS
FrameworksforStatingResearchProblems
Althoughacommondefinitionofaresearchproblemisthatitisastatementthataskswhatrelationshipexistsbetweentwoormorevariables,mostresearchproblems
aremorecomplexthanthisdefinitionimplies.Theresearchproblemshouldbeabroadstatementthatcoverstheseveralmorespecificresearchquestionstobe
investigated,perhapsbyusingtermsthatstandforseveralvariables.Onewaytostatetheproblem,whichcanhelpyoudeterminetheindependentanddependent
variables,isasfollows(italicsindicatethatyoufillintheappropriatenameforthevariableorgroupofvariables).
Format
Ifyourstudyisrandomizedexperimentalorstrongquasiexperimental,youcouldphrasetheproblemasfollows:
Theresearchproblemistoinvestigatethepresumedeffectof(putindependentvariable1orgroupofvariableshere),(independentvariable2,ifany,here),and
(independentvariable3,ifany),on(dependentvariable1,here)and(dependentvariable2,ifany)in(population).
Exceptinatotallydescriptivestudy,therealwaysmustbeatleastoneindependentandonedependentvariable.However,therecanbe,andoftenare,twoormoreof
eachvariable.Inthestatementoftheproblem,incontrasttotheresearchquestionorhypotheses,itisdesirabletousebroaddescriptorsforgroupsofsimilar
variables.Forexample,demographicsmightcoverfourvariables:gender,mother'seducation,father'seducation,andethnicity.Spatial

Page436

performancemightincludeamosaicorpatternscoreandavisualizationscore.Likewise,gradesandmathematicsattitudescouldrefertomorethanonevariable.
Conceptssuchasselfesteemorteachingstylehaveseveralaspectsthatusuallyresultinmorethanonevariable.Thefirstexamplethatfollowsiswritteninaformat
thatismostappropriateforstudiesthatusetherandomizedexperimentalandstrongquasiexperimentalapproaches.Thesecondandthirdexamplesare
suggestedvariationswhentheapproachismoderatetoweakquasiexperimental,comparative,orassociational.
Examples
1.Theresearchproblemistoinvestigatetheeffectofanewcurriculumongrades,mathattitudes,andquantitativeorspatialachievementinhighschoolstudents.
Ifyoudonothaveanactiveormanipulatedindependentvariableand,thus,willusethecomparativeorassociationalapproach,youshouldphraseasample
problemasfollows:
2.Theproblemistoinvestigatetherelationshipsamongdemographics,grades,andmathematicsattitudeswithquantitativeandspatialachievementinhighschool
students.
Ifyouhavemorethantwoorthreeindependentvariablesitmaybebesttosaythefollowing:
3.Theproblemistoinvestigatewhetherdemographics,mathematicscourses,grades,spatialscores,andmathematicsattitudesarepredictorsofmathematics
achievement.Moregenerally,wecouldsaytheproblemistoinvestigatethefactorsthatpredictorseemtoinfluencemathematicsachievement.
Thislatterformatisespeciallyusefulwhentheapproachisacomplex(manyindependentvariables)associationalone.
FrameworkforStatingResearchQuestionsorHypotheses
Althoughitisfinetophrasearandomizedexperimentalresearchproblem(intheformatofExample1)asa"studyoftheeffectof...,"wethinkitisbesttophrase
yourresearchquestionsorhypothesessothattheydonotappeartoimplycauseandeffectthatis,phrasethemasdifferenceorassociationalquestions
(hypotheses)orasdescriptivequestions.Theformerareansweredwithinferentialstatisticsandthelatterwithdescriptivestatistics.

Page437

DescriptiveQuestions
BasicDescriptiveQuestions
Thesequestionsareaboutanaspectofonevariable.Descriptivequestionsaskaboutthecentraltendency,frequencydistribution,percentageineachcategory,
variability,orshapeofthedistribution.Somedescriptivequestionsareintendedtotestassumptions.Somequestionssimplydescribethesampledemographics.
Examplesofdescriptivequestionsareasfollows:
1.Whatpercentageofparticipantsareofeachgender?
2.Whatisthemean,mode,andmedianofthemathematicsachievementscores?
3.Ismathematicsachievementdistributedapproximatelynormally?
ComplexDescriptiveQuestions
Thesequestionsdealwithtwoormorevariablesatatime,butdonotinvolveinferentialstatistics.Crosstabulationsoftwocategoricalvariables,factoranalysis,and
measuresofreliability(e.g.,Cronbach'salpha)areexamples.
Anexampleofacomplexdescriptivequestionisasfollows:
1.Whatisthereliabilityofthevisualizationscore?
DifferenceQuestionsorHypotheses
BasicDifferenceQuestions
Theformatforaskingresearchquestionsisasfollows:
Aretheredifferencesbetweenthe(insertnumber)levels(youcouldnamethelevelshere)of(puttheindependentvariablenamehere)inregardtotheaverage
(putthedependentvariablenamehere)scores?
Anexampleisasfollows:
1.Aretheredifferencesbetweenthethreelevels(high,medium,andlow)offather'seducationinregardtotheaveragemathematicsachievementscoresofthe
students?
ComplexDifferenceandInteractionQuestions
Whenyouhavetwocategoricalindependentvariablesconsideredtogether,youwillhavethreeresearchquestionsorhypotheses.Thereareadvantagesofconsidering
twoorthreeindependentvariablesatatime.Seechapter17forhowtointerprettheinteractionquestion.Sampleformatsforasetofthreequestionsansweredby
one2wayANOVAareasfollows:
1.Isthereadifferencebetweenthelevelsofthe(insertindependentvariable1)inregardtotheaverage(putdependentvariable1here)scores?
2.Isthereadifferencebetweenthelevelsofthe(insertindependentvariable2)inregardtotheaverage(dependentvariable1)scores?
3.Isthereaninteractionof(independentvariable1)and(independentvariable2)inregardtothe(dependentvariable1)?
(Repeatthesethreequestionswiththeseconddependentvariable,ifthereismorethanonedependentvariable.)
Anexampleisasfollows:

Page438

1.Isthereadifferencebetweenstudentshavinghighversuslowmathgradesinregardtotheiraveragemathematicsachievementscores?
2.Isthereadifferencebetweenmaleandfemalestudentsinregardtotheiraveragemathachievementscores?
3.Isthereaninteractionbetweenmathematicsgradesandgenderinregardtomathachievement?
Notethatthefirstquestionstatesthelevelsorcategoriesofthefirstindependentvariablethatis,itstatesthegroupsthataretobecompared(highvs.lowmathgrade
students).Thesecondquestiondoesthesameforthesecondindependentvariablethatis,itstatesthelevels(maleandfemale)tobecompared.However,inthethird
(interaction)question,itisaskedwhetherthefirstvariableitself(mathematicsgrades)interactswiththesecondvariable(gender).Nomentionismade,atthispoint,of
thevaluesorlevelsorgroups.
Associational(Relationship)QuestionsorHypotheses
BasicAssociationalQuestions
Whenboththeindependentanddependentvariablesareorderedandessentiallycontinuous(i.e.,havefiveormoreorderedcategories)theapproachandresearch
questionareconsideredtobeassociational.Theformatforassociationalquestionsisasfollows:
Isthereanassociationbetween(independentvariable1)and(dependentvariable1)?
Ifthereismorethanoneindependentvariable,ordependentvariable,orboth,whichiscommon,andeachpairofvariablesisassociatedseparately,youcanhavea
seriesofquestionsthataskwhetherthereisanassociationbetweeneachindependentvariableandeachdependentvariable.Ifitisarbitrarywhichvariablesare
independentorantecedent(predictors)andwhicharedependent(outcome),onemightaskwhethereveryvariableisrelatedtoeveryothervariable.Thiswould
produceacorrelationmatrix.
Anexampleforasingleassociationorrelationshipisasfollows:
1.Isthereanassociationbetweengradesinhighschoolandmathematicsachievement?
Anexamplethatwouldproduceacorrelationmatrixisasfollows:
2.Arethereassociationsamongthethreemathematicsattitudescalescores?
Notethatwhatissaidtobeassociatedinthesequestionsisthevariableitselfnomentionismadeofthelevelsorvalues.
ComplexAssociationalQuestions
Intheassociationalapproach,whentwoormoreindependentvariablesareconsideredtogether,ratherthanseparatelyasinthebasicformat,yougetanewkindof
question.Theformatcanbephrasedsomethinglikethefollowing:
Isthereacombinationoffactors(listtheseveralspecificindependentvariableshere)thatpredict(putdependentvariablehere)betterthananyonealone?

Page439

Anexampleofacomplexassociationalquestionisasfollows:
1.Isthereacombinationofthenumberofmathematicscourses,gender,father'seducation,mathematicsgrades,andmotivationformathematicsthatpredicts
mathematicsachievementbetterthananyonepredictorvariablealone?

Page441

REFERENCES
Altman,E.,&Hernon,P.(1997).Researchmisconduct:Issues,implications,andstrategies.Greenwich,CT:Ablex.
AmericanPsychologicalAssociation.(1992).Ethicalprinciplesofpsychologistsandcodeofconduct.AmericanPsychologist,47,15971611.
AmericanPsychologicalAssociation.(1994).PublicationmanualoftheAmericanPsychologicalAssociation(4thed.).Washington,DC:Author.
Anastasi,A.(1988).Psychologicaltesting(6thed.).NewYork:Macmillan.
Ary,D.,Jacobs,L.C.,&Razavieh,A.(1996).Introductiontoresearchineducation(5thed).FortWorth,TX:HarcourtBraceCollegePublishers.
Atler,K.E.,&Gliner,J.A.(1989).Poststrokeactivityandpsychosocialfactors.PhysicalandOccupationalTherapyinGeriatrics,7,2833.
Bakan,D.(1966).Thetestofsignificanceinpsychologicalresearch.PsychologicalBulletin,66,129.
Bakshi,R.,Bhambhani,Y.,&Madill,H.(1991).Theeffectsoftaskpreferenceonperformanceduringpurposefulandnonpurposefulactivities.AmericanJournalof
OccupationalTherapy,45,912916.
Bambara,L.,&Ager,C.(1992).Usingselfschedulingtopromoteselfdirectedleisureactivityinhomeandcommunitysettings.TheJournaloftheAssociationfor
PersonswithSevereHandicaps,17,6776.
Bartko,J.J.,&Carpenter,W.T.(1976).Onthemethodsandtheoryofreliability.TheJournalofNervousandMentalDisease,163,307317.
Beatty,W.W.,&Gange,J.J.(1977).Neuropsychologicalaspectsofmultiplesclerosis.TheJournalofNervousandMentalDisease,164,4250.
Bergin,D.A.(1995).Effectsofamasteryversuscompetitivemotivationsituationonlearning.JournalofExperiementalEducation,63,303314.
Berkowitz,L.,&Lutterman,K.(1968).Thetraditionallysociallyresponsiblepersonality.PublicOpinionQuarterly,32,169185.
Bernspang,B.,&Fisher,A.G.(1995).DifferencesbetweenpersonswithrightorleftCVAontheAssessmentofMotorandProcessSkills.ArchivesofPhysical
MedicineandRehabilitation,76,11441151.
Bradley,D.R.,Bradley,T.D.,McGrath,S.G.,&Cutcomb,S.D.(1979).Type1errorrateofthechisquaretestofindependentinRXCtablesthathavesmall
expectedfrequencies.PsychologicalBulletin,86,12901297.

Page442

Bryze,K.A.(1991).Functionalassessmentofadultswithdevelopmentaldisabilities.Unpublishedmaster'sthesis,UniversityofIllinoisatChicago,Chicago.
Burleigh,S.A.,Farber,R.S.,&Gillard,M.(1998).Communityintegrationandlifesatisfactionaftertraumaticbraininjury:Longtermfindings.AmericanJournalof
OccupationalTherapy,52,4552.
Camilli,G.,&Hopkins,K.D.(1978).Applicabilityofchisquareto22contingencytableswithsmallexpectedcellfrequencies.PsychologicalBulletin,85,163
167.
Campbell,D.T.,&Stanley,J.C.(1966).Experimentalandquasiexperimentaldesignsforresearch.Chicago:RandMcNally.
Caracelli,V.J.,&Greene,J.C.(1993).Dataanalysisstrategiesformixedmethodevaluationdesigns.EducationalEvaluationandPolicyAnalysis,15,195207.
Chambers,T.(1994).Criteriatoevaluatestudentleadershipprograms:Whatleadershipeducatorsconsiderimportant.NASPAJournal,31,225234.
Chiara,T.,Carlos,J.,Martin,D.,Miller,R.,&Nadeau,S.(1998).Coldeffectonoxygenuptake,perceivedexertion,andspasticityinpatientswithmultiple
sclerosis.ArchivesofPhysicalMedicineandRehabilitation,79,523528.
Cohen,J.(1988).Statisticalpoweranalysisforthebehavioralsciences(2nded.).Hillsdale,NJ:LawrenceErlbaumAssociates.
Cohen,J.(1990).ThethingsIhavelearned(sofar).AmericanPsychologist,45,13041312.
Cohen,J.(1994).Theearthisround(p<.05).AmericanPsychologist,49,9971003.
Cook,T.D.,&Campbell,D.T.(1979).Quasiexperimentation:Designandanalysisissuesforfieldsettings.Boston:HoughtonMifflin.
Cook,T.D.,Cooper,H.,Cordray,D.S.,Hartmann,H.,Hedges,L.V.,Light,R.J.,Louis,T.A.,&Mosteller,F.(1992).Metaanalysisforexplanation.New
York:RussellSageFoundation.
Cooper,H.M.(1981).Onthesignificanceofeffectsandtheeffectsofsignificance.JournalofPersonalityandSocialPsychology,41,10131018.
Cooper,H.,&Hedges,L.(1994).Thehandbookofresearchsynthesis.NewYork:RussellSageFoundation.
Cortina,J.,&Dunlap,W.P.(1997).Onthelogicandpurposeofsignificancetesting.PsychologicalMethods,2,161172.
Cosby,P.C.(1989).Methodsinbehavioralresearch(4thed.).MountainView,CA:Mayfield.
Creagh,U.,Reilly,T.,&Lees,A.(1998).Kinematicsofrunningon'offroad'terrain.Ergonomics,41,10291033.
Creswell,J.W.(1994).Researchdesign:Qualitativeandquantitativeapproaches.ThousandOaks,CA:Sage.
Cromack,T.R.(1989).Measurementconsiderationsinclinicalresearch.InC.B.Royeen(Ed.),Clinicalresearchhandbook.Thorofare,NJ:SlackInc.
Cronbach,L.J.(1960).Essentialsofpsychologicaltesting(2nded.).NewYork:Harper&Row.
Cronbach,L.J.(1970).Essentialsofpsychologicaltesting(3rded.).NewYork:Harper&Row.
Czerniecki,J.G.,Deitz,J.C.,Crowe,T.W.,&Booth,C.L.(1993).Attendingbehavior:Adescriptivestudyofchildrenaged18through23months.American
JournalofOccupationalTherapy,47,708716.
Daniel,L.G.,&Witta,E.L.(1997,March).Implicationsforteachinggraduatestudentscorrectterminologyfordiscussingvalidityandreliabilityona
contentanalysisofthreesocialsciencemeasurementjournals.PaperpresentedatAmericanEducationResearchAssociation,Chicago,Il.
Davis,J.A.(1985).Thelogicofcausalorder.NewburyPark,CA:Sage.
Dickerson,A.E.,&Fisher,A.G.(1993).Agedifferencesinfunctionalperformance.AmericanJournalofOccupationalTherapy,47,686692.
Dillman,D.A.(1978).Mailandtelephonesurveys:Thetotaldesignmethod.NewYork:Wiley.
DiPasqualeLehnerz,P.(1994).OrthoticinterventionfordevelopmentofhandfunctionwithC6quadriplegia.AmericanJournalofOccupationalTherapy,48,
138144.
Dittmar,C.M.,&Gliner,J.A.(1987).Bilateralhandperformancewithdividedattentionfollowingcerebralvascularaccident.AmericanJournalofOccupational
Therapy,41,96101.
Drew,C.(1980).Introductiontodesigningandconductingresearch.(2nded.).St.Louis,MO:C.V.MosbyCompany.

Page443

Dunlap,G.,FosterJohnson,L,Clarke,S.,Kern,L.,&Childs,K.(1995).Modifyingactivitiestoproducefunctionaloutcomes:Effectsontheproblembehaviorsof
studentswithdisabilities.TheJournaloftheAssociationforPersonswithSevereHandicaps,20,248258.
Dunn,O.J.(1961).Multiplecomparisonsamongmeans.JournaloftheAmericanStatisticalAssociation,56,5264.
Durr,R.,Guglielmino,L.,&Guglielmino,P.(1995).Selfdirectedreadinessandoccupationalcategories.HumanResourceDevelopmentQuarterly,7,351357.
Dunnett,C.W.(1955).Amultiplecomparisonprocedureforcomparingseveraltreatmentswithacontrol.JournaloftheAmericanStatisticalAssociation,50,
10961121.
Edgington,E.(1992).Nonparametrictestsforsinglecaseexperiments.InT.Kratochwill&J.Levin(Eds.),Singlecaseresearchdesignandanalysis(pp.1540).
Hillsdale,NJ:LawrenceErlbaumAssociates.
Fine,M.A.,&Kurdek,L.A.(1993).Reflectionsondeterminingauthorshipcreditandauthorshiporderonfacultystudentcollaborations.AmericanPsychologist,
48,11411147.
Fink,A.(1998).Conductingresearchliteraturereviews:FrompapertotheInternet.ThousandOaks,CA:Sage.
Fisher,A.G.(1993).TheassessmentofIADLmotorskills:AnapplicationofmanyfacetedRaschanalysis.AmericanJournalofOccupationalTherapy,47,319
338.
Fisher,A.G.(1995).Assessmentofmotorandprocessskills.FortCollins,CO:ThreeStarPress.
Fisher,A.G.,Liu,Y.,Velozo,C.A.,&Pan,A.W.(1992).Crossculturalassessmentofprocessskills.AmericanJournalofOccupationalTherapy,46,876
885.
Fisher,R.A.(1935).Thedesignofexperiments.Edinburgh,Scotland:Oliver&Boyd.
Fowler,F.J.,Jr.(1993).Surveyresearchmethods(2nded.).NewburyPark,CA:Sage.
Frone,M.R.(1998).Predictorsofworkinjuriesamongemployedadolescents.JournalofAppliedPsychology,83,565576.
Gerst,J.(1994).Anethnographyofthemedicallyindigentwithinthecontextofprenatalcare.Unpublisheddoctoraldissertation,ColoradoStateUniversity,
FortCollins.
Ghiselli,E.E.,Campbell,J.P.,&Zedeck,S.(1981).Measurementtheoryforthebehavioralsciences.SanFrancisco:W.H.Freeman.
Gilfoyle,E.M.,&Gliner,J.A.(1985).Attitudestowardhandicappedchildren:Impactofaneducationalprogram.PhysicalandOccupationalTherapyin
Pediatrics,5,2741.
Glass,G.V.,McGaw,B.,&Smith,M.L.(1981).Metaanalysisinsocialresearch.BeverlyHills,CA:Sage.
Gliner,J.A.(1994).Reviewingqualitativeresearch:Proposedcriteriaforfairnessandrigor.TheOccupationalTherapyJournalofResearch,14,7892.
Gliner,J.A.,Bedi,J.F.,&Horvath,S.M.(1979).Somaticandnonsomaticinfluencesontheheart:Hemodynamicchanges.Psychophysiology,16(4),358362.
Gliner,J.A.,Bunnell,D.E.,&Horvath,S.M.(1982).Hemodynamicandmetabolicchangespriortospeechperformance.PhysiologicalPsychology,10,108
113.
Gliner,J.A.,Haber,E.,&Weise,J.(1999).Useofcontrolledvignettesinevaluation:Doestypeofresponsemethodmakeadifference?EvaluationandProgram
Planning,22,313322.
Gliner,J.A.,&Sample,P.(1993).Participatoryactionresearch:Anapproachtoevaluatecommunityintegrationforpersonswithdevelopmental
disabilities.PaperpresentedattheAmericanEvaluationAssociation,Dallas,TX.
Gliner,J.A.,&Sample,P.(1996).Amultimethodapproachtoevaluatetransitionintocommunitylife.EvaluationandProgramPlanning,19,225233.
Griffin,M.M.(1992).Dostudentgeneratedrationalsetsofexamplesfacilitateconceptacquisition?JournalofExperimentalEducation,61(2),104115.
Grimm,L.G.,&Yarnold,P.R.(1997).Readingandunderstandingmultivariatestatistics.Washington,DC:AmericanPsychologicalAssociation.
Hanpachern,C.,Morgan,G.A.,&Griego,O.V.(1998).Extensionofthetheoryofmargin:Aframeworkforassessingreadinessforchange.HumanResource
DevelopmentQuarterly,9,339350.
Harter,S.(1985).Manualfortheselfperceptionprofileforchildren.Unpublishedmanuscript,UniversityofDenver,CO.

Page444

Harvey,R.F.,Silverstein,B.,Venzon,M.A.,Kilgore,K.M.,Fisher,W.P.,Steiner,M.,&Harley,J.P.(1992).Applyingpsychometriccriteriatofunctional
assessmentinmedicalrehabilitation:III.Constructvalidityandpredictinglevelofcare.ArchivesofPhysicalMedicineandRehabilitation,73,887892.
Heinsman,D.T.,&Shadish,W.R.(1996).Assignmentmethodsinexperimentation:Whendononrandomizedexperimentsapproximateanswersfromrandomized
experiments?PsychologicalMethods,1,154169.
Heller,J.(1972,July26).SyphilisvictimsinU.S.studywithouttherapyfor40years.TheNewYorkTimes,pp.1,8.
Hendricks,B.,Marvel,M.K.,&Barrington,B.L.(1990).Thedimensionsofpsychologicalresearch.TeachingofPsychology,17,7682.
Howe,K.R.(1985).Twodogmasofeducationalresearch.EducationalResearcher,44,1018.
Hsieh,C.,Nelson,D.L.,Smith,D.A.,&Peterson,C.Q.(1996).Acomparisonofperformanceinaddedpurposeoccupationsandroteexercisefordynamic
standingbalanceinpersonswithhemiplegia.AmericanJournalofOccupationalTherapy,50,1016.
Huck,S.W.,Cormier,W.H.,&Bounds,W.G.,Jr.(1974).Readingstatisticsandresearch.NewYork:Harper&Row.
Huck,S.W.,&McLean,R.A.(1975).UsingarepeatedmeasuresANOVAtoanalyzethedatafromaPretestPosttestdesign:Apotentiallyconfusingtask.
PsychologicalBulletin,32,511518.
Humphreys,L.(1970).Tearoomtrade:Impersonalsexinpublicplaces.Chicago:Aldine.
Hunter,J.E.,&Schmidt,F.L.(1990).Methodsofmetaanalysis:Cumulatingresearchfindingsacrossstudies.BeverlyHills,CA:Sage.
Iverson,A.M.,Iverson,G.L.,&Lukin,L.E.(1993).Frequent,ungradedtestingasaninstructionalstrategy.JournalofExperimentalEducation,62(2),93101.
Janelle,S.(1992).Locusofcontrolinnondisabledversuscongenitallyphysicallydisabledadolescents.AmericanJournalofOccupationalTherapy,46,334342.
Jones,J.H.(1982).Badblood.NewYork:FreePress.
Jongbloed,L.,Stacey,S.,&Brighton,C.(1989).Strokerehabilitation:Sensorimotorintegrativetreatmentversusfunctionaltreatment.AmericanJournalof
OccupationalTherapy,43,391397.
Kazdin,A.(1982).Singlecaseresearchdesigns.NewYork:OxfordUniversityPress.
Keppel,G.(1991).Designandanalysis:Aresearcher'shandbook(3rded.).NewYork:PrenticeHall.
Keppel,G.,&Zedeck,S.(1989).Dataanalysisforresearchdesigns.NewYork:W.H.Freeman.
Kerlinger,F.(1986).Foundationsofbehavioralresearch(3rded.).NewYork:Holt,Rinehart&Winston.
King,G.A.,Specht,J.A.,Schultz,I.,WarrLeeper,G.,Redekop,W.,&Risebrough,N.(1997).Socialskillstrainingforwithdrawnunpopularchildrenwith
physicaldisabilities:Apreliminaryevaluation.RehabilitationPsychology,42,4760.
Kircher,M.A.(1984).Motivationasafactorofperceivedexertioninpurposefulversusnonpurposefulactivity.AmericanJournalofOccupationalTherapy,38,
165170.
Kirchner,G.L.,&Holm,M.B.(1997).Predictionofacademicandclinicalperformanceofoccupationaltherapystudentsinanentrylevelmaster'sprogram.
AmericanJournalofOccupationalTherapy,51,775779.
Kirk,R.E.(1982).Experimentaldesign:Proceduresforthebehavioralsciences(2nded.).Belmont,CA:Wadsworth,Inc.
Komesaroff,P.A.(1986).Objectivity,scienceandsociety.London:Routledge&KeganPaul.
Kraemer,H.C.,&Thiemann,S.(1987).Howmanysubjects:Statisticalpoweranalysisinresearch.NewburyPark,CA:Sage.
Krathwohl,D.R.(1993).Methodsofeducationalandsocialscienceresearch.NewYork:Longman.
Kratochwill,T.,&Levin,J.(Eds.)(1992).Singlecaseresearchdesignandanalysis.Hillsdale,NJ:LawrenceErlbaumAssociates.
Kuhn,T.S.(1970).Thestructureofscientificrevolutions(2nded.).Chicago:UniversityofChicagoPress.
Laconte,M.A.,Shaw,D.,&Dunn,I.(1993).Theeffectsofarationalemotiveaffectiveeducationprogramforhighriskmiddleschoolstudents.Psychologyinthe
Schools,30,274281.

Page445

Lafferty,A.,&Gliner,J.A.(1991).Executionofrapidbimanualtasksinpatientswithunilateralcerebralhemispheredamage.Unpublishedmaster'sthesis,
ColoradoStateUniversity,FortCollins.
Lan,W.Y.,&Repman,J.(1995).Theeffectsofsociallearningcontextandmodelingonpersistenceanddynamisminacademicactivities.JournalofExperimental
Education,64,5367.
Lancy,D.F.(1993).Qualitativeresearchineducation.NewYork:Longman.
Levin,J.,Marascuilo,L.,&Hubert,L.(1978).N=1nonparametricrandomizationtests.InT.Kratochwill(Ed.),Singlesubjectresearch:Strategiesfor
evaluatingchange(pp.167196).NewYork:AcademicPress.
Likert,R.(1932).Atechniqueforthemeasurementofattitudes.ArchivesofPsychology.No.140.
Lincoln,Y.S.,&Guba,E.G.(1985).Naturalisticinquiry.NewburyPark,CA:Sage.
Lipsey,M.W.(1990).Designsensitivity:Statisticalpowerforexperimentalresearch.NewburyPark,CA:Sage.
Lipsey,M.W.(1992).Juveniledelinquencytreatment:Ametaanalyticinquiryintothevariabilityofeffects.InT.D.Cook,H.Cooper,D.S.Cordray,H.Hartmann,
L.V.Hedges,R.J.Light,T.A.Louis,&F.Mosteller,(Eds.),Metaanalysisforexplanation:Acasebook.NewYork:RussellSageFoundation.
Lipsey,M.W.,&Wilson,D.B.(1993).Theefficacyofpsychological,educational,andbehavioraltreatment:Confirmationfrommetaanalysis.American
Psychologist,48,11811209.
Locke,L.F.,Silverman,S.J.,&Spirduso,W.W.(1998).Readingandunderstandingresearch.ThousandOaks,CA:Sage.
Loftus,G.R.,&Loftus,E.F.(1982).Essenceofstatistics.Monterey,CA:Brooks/Cole.
Lord,F.M.,&Novick,M.R.(1968).Statisticaltheoriesofmentaltestscores.Reading,MA:AddisonWesleyPublishingCo.
Louth,R.,McAllister,C.,&McAllister,H.A.(1992).Theeffectsofcollaborativewritingtechniquesonfreshmanwritingandattitudes.JournalofExperimental
Education,61,215224.
Luft,J.A.,&Pizzini,E.L.(1998).Thedemonstrationclassroominservice:Changesintheclassroom.ScienceEducation,82,147163.
MacPhee,D.,Kreutzer,J.C.,&Fritz,J.J.(1994).Infusingadiversityperspectiveintohumandevelopmentcourses.ChildDevelopment,65,699715.
McCleary,R.,&Welsh,W.(1992).Philosophicalandstatisticalfoundationsoftimeseriesexperiments.InT.Kratochwill&J.Levin(Eds.),Singlecaseresearch
designandanalysis(pp.4192).Hillsdale,NJ:LawrenceErlbaumAssociates.
McDougall,D.,&Granby,C.(1996).Howexpectationofquestioningmethodaffectsundergraduates'preparationforclass.TheJournalofExperimental
Education,65,4354.
McNemar,Q.(1960).Atrandom:Senseandnonsense.AmericanPsychologist,15,295300.
MentalMeasurementsYearbooks.Lincoln,NE:BurosInstituteofMentalMeasurements,UniversityofNebraska.
Meltzoff,J.(1997).Criticalthinkingaboutresearch:Psychologyandrelatedfields.Washington,DC:AmericanPsychologicalAssociation.
Micceri,T.(1989).Theunicorn,thenormalcurve,andotherimprobablecreatures.PsychologicalBulletin,105(1),156166.
Milgram,S.(1974).Obediencetoauthority:Anexperimentalview.NewYork:Harper&Row.
Moisan,M.O.(1990).Effectsofanadaptiveskiprogramontheselfperceptionandperformanceofchildrenwithphysicaldisabilities.Unpublishedmaster's
thesis,ColoradoStateUniversity,FortCollins.
Morgan,G.A.,&Griego,O.V.(1998).EasyuseandinterpretationofSPSSforWindows:Answeringresearchquestionswithstatistics.Mahwah,NJ:
LawrenceErlbaumAssociates.
Morgan,G.A.,Harmon,R.J.,&MaslinCole,C.A.(1990).Masterymotivation:Definitionandmeasurement.EarlyEducationandDevelopment,1,318339.
Morgan,G.A.,MaslinCole,C.A.,Harmon,R.J.,BuschRossnagel,N.A.,Jennings,K.D.,HauserCram,P.,&Brockman,L.(1993).Parentandteacher
perceptionsofyoungchildren'smasterymotivation:Assessmentandreviewofresearch.InD.Messer(Ed.),Masterymotivationinearlychildhood:Development,
measurementandsocialprocesses(pp.109131).London:Routledge.

Page446

Morgan,G.A.,&Ricciuti,H.N.(1969).Infants'responsestostrangersduringthefirstyear.InB.M.Foss(Ed.),Determinantsofinfantbehavior(pp.272353).
London:Methuen.
Murphy,J.B.,&Gliner,J.A.(1988).Visualandmotorsequencinginnormalanddyspraxicchildren.OccupationalTherapyJournalofResearch,8,89103.
NationalCommissionfortheProtectionofHumanSubjectsofBiomedicalandBehavioralResearch.(1978).TheBelmontReport:Ethicalprinciplesand
guidelinesfortheprotectionofhumansubjectsofresearch(DHEWPublicationNo.(OS)780012).Washington,DC:GovernmentPrintingOffice.
Nelson,D.L.,&Peterson,C.Q.(1989).Enhancingtherapeuticexercisethroughpurposefulactivity:Atheoreticalanalysis.TopicsinGeriatricRehabilitation,4,
1222.
Nunnally,J.C.(1978).Psychometrictheory.NewYork:McGrawHill.
Osgood,C.E.,Suci,G.J.,&Tannenbaum,P.H.(1957).Themeasurementofmeaning.Urbana,IL:UniversityofIllinoisPress.
Ottenbacher,K.(1986).Evaluatingclinicalchange.Baltimore,MD:Williams&Wilkins.
Parsonson,B.,&Baer,D.(1992).Thevisualanalysisofdata,andcurrentresearchintothestimulicontrollingit.InT.Kratochwill&J.Levin(Eds.),Singlecase
researchdesignandanalysis(pp.1540).Hillsdale,NJ:LawrenceErlbaumAssociates.
Patton,M.Q.(1990).Qualitativeevaluationandresearchmethods(2nded.).NewberryPark,CA:Sage.
Pedhazur,E.J.,&Schmelkin,L.P.(1991).Measurement,design,andanalysis:Anintegratedapproach.Hillsdale,NJ:LawrenceErlbaumAssociates.
Phillips,D.C.(1992).Thesocialscientist'sbestiary.Oxford,UK:PergamonPress.
Porter,A.C.(1997).Comparativeexperimentsineducationalresearch.InJaeger,R.M.(Ed),Complementarymethodsforresearchineducation(2nded.,pp.
524544).Washington,DC:AmericanEducationalResearchAssociation.
Pross,C.(1992).Nazidoctors,Germanmedicine,andhistoricaltruth.InG.J.Annas&M.A.Grodin(Eds.),TheNazidoctorsandtheNuremburgCode(pp.
3252).NewYork:OxfordUniversityPress.
Puderbaugh,J.K.&Fisher,A.G.(1992).Assessmentofmotorandprocessskillsinnormalyoungchildrenandchildrenwithdyspraxia.OccupationalTherapy
JournalofResearch,12,195216.
Redding,R.,Morgan,G.A.,&Harmon,R.J.(1988).Masterymotivationininfantsandtoddlers:Isitgreatestwhentasksaremoderatelychallenging?Infant
BehaviorandDevelopment,11,423434.
Reichardt,C.S.(1979).Thestatisticalanalysisofdatafromnonequivalentgroupdesigns.InT.D.Cook&D.T.Campbell(Eds.),Quasiexperimentation:Design
andanalysisissuesforfieldsettings(pp.147205).Boston:HoughtonMifflin.
Rennie,D.,Yank,V.,&Emanuel,L.(1997).Whenauthorshipfails:Aproposaltomakecontributorsaccountable.JournaloftheAmericanMedicalAssociation,
278,579585.
Rheingold,H.L.,&Eckerman,C.O.(1973).Fearofthestranger.InH.W.Reese(Ed.),Advancesinchilddevelopmentandbehavior(pp.185222).New
York:AcademicPress.
Richardson,A.G.,&Fergus,E.E.(1993).Learningstyleandabilitygroupinginthehighschoolsystem:SomeCaribbeanfindings.EducationalResearch,35,69
74.
Robinson,D.H.,Katayama,A.D.,Dubois,N.F.,&Devaney,T.(1998).Interactiveeffectsofgraphicorganizersanddelayedreviewonconceptapplication.
JournalofExperimentalEducation,67,1731.
Robinson,D.H.,&LevinJ.R.(1997).Reflectionsonstatisticalandsubstantivesignificance,withasliceofreplication.EducationalResearcher,26,2126.
Robnett,R.,&Gliner,J.A.(1995).QualOT:Aqualityoflifeassessmenttool.OccupationalTherapyJournalofResearch,15,198214.
Roethlisberger,F.S.,&Dickson,W.J.(1939).Managementandtheworker.Cambridge,MA:HarvardUniversityPress.
Rosenthal,R.(1979).The''filedrawerproblem"andtolerancefornullresults.PsychologicalBulletin,86,638641.
Rosenthal,R.(1985).Writingmetaanalyticreviews.PsychologicalBulletin,118,169312.
Rosenthal,R.(1994).Parametricmeasuresofeffectsize.InH.Cooper&L.Hedges(Eds.),Thehandbookofresearchsynthesis(pp.231244).NewYork:
RussellSageFoundation.

Page447

Rozeboom,W.W.(1960).Thefallacyofthenullhypothesissignificancetest.PsychologicalBulletin,57,416428.
Rumrill,P.D.,&Garnette,M.R.(1997).Careeradjustmentviareasonableaccommodations:Theeffectsofanemployeeempowermentinterventionforpeoplewith
disabilities.Work,9,5764.
Salant,P.,&Dillman,D.A.(1994).Howtoconductyourownsurvey.NewYork:Wiley.
Salsburg,D.S.(1985).Thereligionofstatisticsaspracticedinmedicaljournals.TheAmericanStatistician,39,220223.
Schalock,R.,Keith,K.,Hoffman,K.,&Karan,O.(1989).Qualityoflife:Itsmeasurementanduse.MentalRetardation,27,2531.
Scheffe,H.(1953).Amethodforjudgingallcontrastsintheanalysisofvariance.Biometrika,40,87104.
Schmitt,N.(1996a).Statisticalsignificancetestingandcumulativeknowledgeinpsychology:Implicationsfortrainingofresearchers.PsychologicalMethods,1,
115129.
Schmitt,N.(1996b).Usesandabusesofcoefficientalpha.PsychologicalAssessment,8,350353.
Shavelson,R.J.(1981).Statisticalreasoningforthebehavioralsciences.Boston:Allyn&Bacon.
Shavelson,R.J.(1988).Statisticalreasoningforthebehavioralsciences(2nded.).Boston:AllynandBacon.
Shuster,E.(1997).Fiftyyearslater:ThesignificanceoftheNurembergcode.TheNewEnglandJournalofMedicine,337,14361440.
Sieber,J.E.(1992).Planningethicallyresponsibleresearch:Aguideforstudentsandinternalreviewboards.NewburyPark,CA:Sage.
Siegel,S.,&Castellan,N.J.(1988).Nonparametricstatisticsforthebehavioralsciences(2nded.).NewYork:McGrawHill.
Silverstein,B.,Kilgore,K.M.,Fisher,W.P.,Harley,J.P.,&Harvey,R.F.(1991).Applyingpsychometriccriteriatofunctionalassessmentinmedicalrehabilitation:
I.Exploringunidimensionality.ArchivesofPhysicalMedicineandRehabilitation,73,507518.
Smith,M.L.(1981).Naturalisticresearch.PersonnelandGuidanceJournal,59,585589.
Smith,M.L.,Gabriel,R.,Schott,J.,&Padia,W.L.(1976).EvaluationeffectsofOutwardBound.InG.V.Glass(Ed.),Evaluationstudiesreviewannual(Vol.1,
pp.400421).BeverlyHills,CA:Sage.
Spector,P.E.(1992).Summatedratingscaleconstruction:Anintroduction.NewburyPark,CA:Sage.
Spitz,R.A.(1965).Thefirstyearoflife.NewYork:InternationalUniversitiesPress.
Stevens,J.(1986).Appliedmultivariatestatisticsforthesocialsciences.Hillsdale,NJ:LawrenceErlbaumAssociates.
Stevens,S.S.(1951).Mathematics,measurement,andpsychophysics.InS.S.Stevens(Ed.)Handbookofexperimentalpsychology.NewYork:Wiley.
Stone,W.L.,Lemanek,K.L.,Fishel,P.T.,Fernandez,M.C.,&Altemeier,W.A.(1990).Playandimitationskillsinthediagnosisofautisminyoungchildren.
Pediatrics,86,267272.
Stuart,A.(1984).Theideaofsampling(3rded.).NewYork:Macmillan.
Sudman,S.,&Bradburn,N.(1989).Askingquestions:Apracticalguidetoquestionnairedesign.SanFrancisco:JosseyBass.
Suen,H.K.(1990).Principlesoftesttheories.Hillsdale,NJ:LawrenceErlbaumAssociates.
Tabachnick,B.G.,&Fidell,L.S.(1989).Usingmultivariatestatistics(2nded.).NewYork:HarperColling.
Tarr,S.J.,&Bishop,P.(1992).Effectsofinstructionalapproachesonmotorperformanceofboyswithdevelopmentaldisabilities.ClinicalKinesiology,45,1822.
Tate,S.(1997).Peppermintoil:Atreatmentforpostoperativenausea.JournalofAdvancedNursing,26,543549.
Testcritiques(annually).Austin,TX:ProEd.
Tests(3rded.).(1991).Austin,TX:ProEd.
Testsinprint(4thed.).(1994).Lincoln,NE:BurosInstituteforMentalMeasurements.
Thompson,B.(1995).Stepwiseregressionandstepwisediscriminantanalysisneednotapplyhere:Aguidelineseditorial.EducationalandPsychological
Measurement,55,525534.

Page448

Thompson,B.(1996).AERAeditorialpoliciesregardingstatisticalsignificancetesting:Threesuggestedreforms.EducationalResearcher,25,2630.
Thorndike,R.L.,&Hagen,E.(1992).Measurementandevaluationinpsychologyandeducation(5thed.).NewYork:JohnWileyandSons.
Tuckman,B.W.(1992).Theeffectofstudentplanningandselfcompetenceonselfmotivatedperformance.JournalofExperimentalEducation,60,119127.
Tuckman,B.W.(1996).Therelativeeffectivenessofincentivemotivationandprescribedlearningstrategyinimprovingcollegestudents'courseperformance.
JournalofExperimentalEducation,64,197210.
Tukey,J.W.(1953).Theproblemofmultiplecomparisons.Unpublishedmanuscript,PrincetonUniversity,Princeton,NJ.
Tukey,J.W.(1977).Exploratorydataanalysis.Reading,MA:AddisonWesley.
Tun,C.G.,Tun,P.A.,&Wingfield,A.(1997).Cognitivefunctionfollowinglongtermspinalcordinjury.RehabilitationPsychology,42,163182.
Webster,P.E.,Plante,A.S.,&Couvillion,L.M.(1997).Phonologicimpairmentandprereading:Updateonalongitudinalstudy.JournalofLearningDisabilities,
30,365375.
Webster'sseventhnewcollegiatedictionary(1972).Springfield,MA:C&GMerriam.
Weinfurt,K.P.(1997).Multivariateanalysisofvariance.InL.G.Grimm&P.R.Yarnold(Eds.),Readingandunderstandingmultivariatestatistics.Washington,
D.C.:AmericanPsychologicalAssociation.
White,O.(1974).The"splitmiddle":A"quickie"methodoftrendestimation(3rdrevision).Unpublishedmanuscript,UniversityofWashington,Experimental
EducationUnit,ChildDevelopmentandMentalRetardationCenter,Seattle,WA.
Wilkinson,L.,&TheTaskForceonStatisticalInference.(1999).Statisticalmethodsinpsychologyjournals:Guidelinesandexplanations.AmericanPsychologist.
54,594604.
Winer,B.J.(1962).Statisticalprinciplesinexperimentaldesign.NewYork:McGrawHill.
Woodward,J.,&Goodstein,D.(1996).Conduct,misconductandstructureofscience.AmericanScientist,84,479490.
Young,E.D.(1999).Researchinvolvinghumansubjects:Historicalreview,ethicaltheory,andStanfordguidelines.Unpublishedmanuscript,Stanford
UniversitySchoolofMedicine,Stanford,CA.

Page449

AUTHORINDEX
A
Ager,C.,114,441
Altemeier,W.A.,293,299,447
Altman,E.,355,441
Anastasi,A.,312,321,324,332,441
Ary,D.,18,441
Atler,K.E.,252254,372,441
B
Baer,D.,118,119,446
Bakan,D.,176,441
Bakshi,R.,5,441
Bambara,L.,114,441
Barrington,B.L.,10,444
Bartko,J.J.,318,441
Beatty,W.W.,85,441
Bedi,J.F.,54,443
Bergin,D.A.,266,441
Berkowitz,L.,337,441
Bernspang,B.,323,441
Bhambhani,Y.,5,441
Bishop,P.,247,248,447
Booth,C.L.,228,442
Bounds,W.G.Jr.,100,225,444
Bradburn,N.,332,447
Bradley,D.R.,233,441
Bradley,T.D.,233,441
Brighton,C.,6,70,278,444
Brockman,L.,324,445
Bryze,K.A.,322,442
Bunnell,D.E.,54,443
Burleigh,S.A.,256,442
BuschRossnagel,N.A.,324,445
C
Camilli,G.,233,442
Campbell,D.T.,20,8284,86,87,96,97,100,158,368,373,374,375,377,389,392,442
Campbell,J.P.,312,324,443
Caracelli,V.J.,26,27,442
Carlos,J.,245,442
Carpenter,W.T.,318,441
Castellan,N.J.,227,247,447
Chambers,T.,279,442
Chiara,T.,245,442
Childs,K.,113,443
Clarke,S.,113,443
Cohen,J.,176,178,374,381,428,433,442
Cook,T.D.,82,83,86,87,96,97,100,368,373375,377,380,389,392,442
Cooper,H.,379,380,442
Cooper,H.M.,427,442
Cordray,D.S.,380,442
Cormier,W.H.,100,225,444
Cortina,J.,177,442
Cosby,P.C.,97,183,442
Couvillion,L.M.,239,241,242,250,448
Creagh,U.,246,442
Creswell,J.W.,27,442
Cromack,T.R.,322,442

Page450

Cronbach,L.J.,310,313,314,322,323,324,442
Crowe,T.W.,228,442
Cutcomb,S.D.,233,441
Czerniecki,J.G.,228,442
D
Daniel,L.G.,313,316,442
Davis,J.A.,22,233,442
Deitz,J.C.,228,442
Devaney,T.,264266,446
Dickerson,A.E.,7,442
Dickson,W.J.,19,446
Dillman,D.A.,332,339,340,442,447
DiPasqualeLehnerz,P.,65,66,71,442
Dittmar,C.M.,372,442
Drew,C.,20,23,24,46,442
Dubois,N.F.,264,265,266,446
Dunlap,G.,113,443
Dunlap,W.P.,177,442
Dunn,I.,103,444
Dunn,O.J.,294,443
Dunnett,C.W.,225,443
Durr,R.,224,225,443
E
Eckerman,C.O.,160,446
Edgington,E.,120,443
Emanuel,L.,358,359,446
F
Farber,R.S.,256,442
Fergus,E.E.,266,446
Fernandez,M.C.,293,299,447
Fidell,L.S.,304,447
Fine,M.A.,359,443
Fink,A.,428,443
Fishel,P.T.,293,299,447
Fisher,A.G.,6,7,185,320,322,323,441,442,443,446
Fisher,R.A.,225,443
Fisher,W.P.,323,444,447
FosterJohnson,L.,113,443
Fowler,F.J.,Jr.,150,155,332,443
Fritz,J.J.,157,445
Frone,M.R.,257,258,443
G
Gabriel,R.,26,447
Gange,J.J.,85,441
Garnette,M.R.,238,447
Gerst,J.,19,443
Ghiselli,E.E.,312,324,443
Gilfoyle,E.M.,98,99,278,443
Gillard,M.,256,442
Glass,G.V.,20,379,443
Gliner,J.A.,26,27,54,68,71,98,99,231,232,233,242,243,244,250,252,253,254,255,278,288,301,302,372,395,441,442,443,445,446
Goodstein,D.,361,362,448
Granby,C.,215222,445
Greene,J.C.,26,27,442
Griego,O.V.,185,229,296,297,299,300,443,445
Griffin,M.M.,85,443
Grimm,L.G.,292,300,302,443
Guba,E.G.,1622,2427,29,82,445
Guglielmino,L.,224,225,443
Guglielmino,P.,224,225,443
H
Haber,E.,231,232,233,443
Hagen,E.,332,335,448
Hanpachern,C.,185,443
Harley,J.P.,323,444,447
Harmon,R.J.,6,70,324,374,445,446
Harter,S.,71,96,443
Hartmann,H.,380,442
Harvey,R.F,323,444,447
HauserCram,P.,324,445
Hedges,L.V.,379,380,442
Heinsman,D.T.,380,444
Heller,J.,32,444
Hendricks,B.,10,444
Hernon,P.,355,441
Hoffman,K.,242,447
Holm,M.B.,254,444
Hopkins,K.D.,233,442
Horvath,S.M.,54,443
Howe,K.R.,27,444
Hsieh,C.,238,242,444
Hubert,L.,120,445
Huck,S.W.,100,225,282,285,444
Humphreys,L.,33,348,349,444
Hunter,J.E.,379,444
I
Iverson,A.M.,85,444
Iverson,G.L.,85,444
J
Jacobs,L.C.,18,441
Janelle,S.,228,444
Jennings,K.D.,324,445

Page451

Jones,J.H.,32,444
Jongbloed,L.,6,70,278,444
K
Karan,O.,242,447
Katayama,A.D.,264,265,266,446
Kazdin,A.,110,118,119,444
Keith,K.,242,447
Keppel,G.,208,288,369,371,444
Kerlinger,F.,5,23,44,45,46,50,52,55,219,268,444
Kern,L.,113,443
Kilgore,K.M.,323,444,447
King,G.A.,238,444
Kircher,M.A.,5,444
Kirchner,G.L.,254,444
Kirk,R.E.,50,183,225,444
Komesaroff,P.A.,16,444
Kraemer,H.C.,128,157,158,444
Krathwohl,D.R.,55,444
Kratochwill,T.,110,444
Kreutzer,J.C.,157,445
Kuhn,T.S.,16,444
Kurdek,L.A.,359,443
L
Laconte,M.A.,103,444
Lafferty,A.,372,445
Lan,W.Y.,278,280,281,282,445
Lancy,D.F.,27,445
Lees,A.,246,442
Lemanek,K.L.,293,299,447
Levin,J.,110,120,444,445,446
LevinJ.R.,176,446
Light,R.J.,380,442
Likert,R.,337,445
Lincoln,Y.S.,1622,2427,29,82,445
Lipsey,M.W.,20,370,371,380,389,445
Liu,Y.,7,443
Locke,L.F.,46,445
Loftus,E.F.,171,174,369,445
Loftus,G.R.,171,174,369,445
Lord,F.M.,323,445
Louis,T.A.,380,442
Louth,R.,278,282,283,285,286,288,445
Luft,J.A.,238,445
Lukin,L.E.,85,444
Lutterman,K.,337,441
M
MacPhee,D.,157,445
Madill,H.,5,441
Marascuilo,L.,120,445
Martin,D.,245,442
Marvel,M.K.,10,444
MaslinCole,C.A.,6,324,374,445
McAllister,C.,278,282,283,285,286,288,445
McAllister,H.A.,278,282,283,285,286,288,445
McCleary,R.,119,445
McDougall,D.,215222,445
McGaw,B.,20,379,443
McGrath,S.G.,233,441
McLean,R.A.,282,285,444
McNemar,Q.,428,445
Meltzoff,J.,356,445
Micceri,T.,428,445
Milgram,S.,32,348,349,445
Miller,R.,245,442
Moisan,M.O.,96,445
Morgan,G.A.,6,70,160,185,229,296,297,299,300,324,374,443,445,446
Mosteller,F.,380,442
Murphy,J.B.,68,71,255,446
N
Nadeau,S.,245,442
Nelson,D.L.,5,238,242,444,446
Novick,M.R.,323,445
Nunnally,J.C.,313,446
O
Osgood,C.E.,338,446
Ottenbacher,K.,5,7,110,119,318,446
P
Padia,W.L.,26,447
Pan,A.W.,7,443
Parsonson,B.,118,119,446
Patton,M.Q.,26,27,446
Pedhazur,E.J.,266,446
Peterson,C.Q.,5,238,242,444,446
Phillips,D.C.,1719,446
Pizzini,E.L.,238,445
Plante,A.S.,239,241,242,250,448
Porter,A.C.,66,69,82,446

Page452

Pross,C.,32,446
Puderbaugh,J.K.,7,446
R
Razavieh,A.,18,441
Redding,R.,70,446
Redekop,W.,238,444
Reichardt,C.S.,282,446
Reilly,T.,246,442
Rennie,D.,358,359,446
Repman,J.,278,280,281,282,445
Rheingold,H.L.,160,446
Ricciuti,H.N.,160,446
Richardson,A.G.,266,446
Risebrough,N.,238,444
Robinson,D.H.,176,264,265,266,446
Robnett,R.,301,302,446
Roethlisberger,F.S.,19,446
Rosenthal,R.,177,378,381,446
Rozeboom,W.W.,176,447
Rumrill,P.D.,238,447
S
Salant,P.,332,339,340,447
Salsburg,D.S.,428,447
Sample,P.,27,242,243,244,250,288,395,443
Schalock,R.,242,447
Scheffe,H.,225,447
Schmelkin,L.P.,266,446
Schmidt,F.L.,379,444
Schmitt,N.,176,316,379,447
Schott,J.,26,447
Schultz,I.,238,444
Shadish,W.R.,380,444
Shavelson,R.J.,168,318,372,447
Shaw,D.,103,444
Shuster,E.,32,447
Sieber,J.E.,34,35,37,38,39,349,352,354,447
Siegel,S.,227,247,447
Silverman,S.J.,46,445
Silverstein,B.,323,444,447
Smith,D.A.,238,242,444
Smith,M.L.,4,5,10,20,26,379,443,447
Specht,J.A.,238,444
Spector,P.E.,316,338,447
Spirduso,W.W.,46,445
Spitz,R.A.,160,447
Stacey,S.,6,70,278,444
Stanley,J.C.,20,84,158,377,389,442
Steiner,M.,323,444
Stevens,J.,293,447
Stevens,S.S.,128,447
Stone,W.L.,293,299,447
Stuart,A.,149,447
Suci,G.J.,338,446
Sudman,S.,332,447
Suen,H.K.,315,447
T
Tabacknick,B.G.,304,447
Tannenbaum,P.H.,338,446
Tarr,S.J.,247,248,447
Tate,S.,229,447
Thiemann,S.,157,158,444
Thompson,B.,176,298,447
Thorndike,R.L.,332,335,448
Tuckman,B.W.,84,99,100,222,224227,266,267,270273,448
Tukey,J.W.,136,225,356,448
Tun,C.G.,220,221,448
Tun,P.A.,220,221,448
V
Velozo,C.A.,7,443
Venzon,M.A.,323,444
W
WarrLeeper,G.,238,444
Webster,P.E.,239,241,242,250,448
Weinfurt,K.P.,293,448
Weise,J.,231233,443
Welsh,W.,119,445
White,O.,119,448
Wilkinson,L.,178,207,427,429434,448
Wilson,D.B.,380,445
Winer,B.J.,187,448
Wingfield,A.,220,221,448
Witta,E.L.,313,316,442
Woodward,J.,361,362,448
Y
Yank,V.,358,359,446
Yarnold,P.R.,292,300,302,443
Young,E.D.,32,448
Z
Zedeck,S.,208,312,324,443,444

Page453

SUBJECTINDEX
A
Abscissa,134
Absolutevalues,255
Abstractofanarticle,428
Accessiblepopulation,146148,168,seealsoPopulationandNonprobabilitysample
Accidentalsample,seeNonprobabilitysamples
Achievementtests,334335
Activeindependentvariable,4849,184,seealsoExperimentalapproaches
Actualsample,146161
responserate,147,156,338,347
Alpha,171
Alternateformsreliability,seeParallelformsreliability
Alternativehypothesis,166,167,170,171,173,369
Analysisofcovariance,85,286288
adjustedposttestmeans,287
generaluse,288
linearityassumption,286
parallelslopesassumption,286
posthoctest,287
singlefactorANCOVA,286288
sourcetable,287
Analysisofvariance(ANOVA),seeSinglefactorANOVA,TwofactorANOVA,MixedANOVA
Animalresearch,350
Anonymous,37,352
Antecedentvariable,seeIndependentvariable
Apparatus,seeInstrumentation
Appliedresearch,7
Aptitudetest,335336
group,336
StanfordBinet,336
Wechsler,336
Association
chisquaretest,231
Associationalinferentialstatistics,76,199208,251261,294303
Associationalresearchapproach,6264,7173,8687
basicdesigns,251261
complexdesigns,291305
Associationalresearchhypothesisorquestion,7478,198208,438439
examples,77,438439
Assumptions,seeStatisticalassumptions
Attitudescales,337338
connotativemeaning,338
semanticdifferential,338
summatedorLikert,337338
Attributeindependentvariable,4950,53,6264,6872,184,252
Authenticassessment,335
Authorship,seePublication
Axioms,1723,seealsoConstructivistparadigmandPositivistparadigm

Page454

B
Basicassociationaldesignsandstatistics,203206,251261
Cramer'sV,205206,256257
linearregression,257259
Pearsonproductmomentcorrelation(r),203208,252255
phicoefficient,205206,256257
Spearmanrankordercorrelation(rho),205206,255256
Basicassociationalquestion,seeAssociationalquestion
Basicdifferencequestion,seeDifferencequestion
Basicdifferencestatistic,seeDifferenceinferentialstatisticsandSinglefactordesigns
Basicdescriptivequestion,seeDescriptivequestion
Behavioralobservation,seeObservation
Benefitsofresearch,seeHumanresearchparticipants
Beta(typeIIerror),171
Betaweights,295
Betterquasiexperimentaldesigns,97102
Betweengroupsdesign,182,184,187190,200205
describingthedesign,190
diagrammingthedesign,187188
numberandtypeofindependentvariables,184
singlefactordesigns,214235
Betweengroupsindependentvariable,184,187,264,278,280,284
Betweengroupsvariance,223224
Bias,431,seealsoNonprobabilitysample,Datacollection,andInternalvalidity
Blindreview,seePeerreview
Boxandwhiskerplot,136,137
C
Calculatedvalue,174
Canonicalcorrelation,207
Carryovereffects,183,239,314
Casestudy,110
Cases,145146
Categoricalvariable,52,132,230231
Causalcomparative,seeComparativeapproach
Causeandeffect,18,50,61,6669,8283,88,207,356,387388,393,434
seealsoIndependentvariable
causalityaxiom,2122,82
Changeovertimeindependentvariable,186,187
Chisquaretestforindependence,175,203204,230233
marginaltotals,
minimalexpectedfrequenciesassumption,233
Yatescorrection,233
Citation,seePublication
Classicaltesttheory,312
Clustersampling,147148,152153
CochranQtest,247248
Coefficientofequivalence,314
Coefficientofstability,313
Comparativeresearchapproach,6264,6873,8586,215
factorialdesigns,266
mixedfactorialdesigns,278
multivariateanalysisofvariance,293
withinsubjects,238
Comparisongroup,97106
Complexassociationalanalyses(statistics),205208,294305
discriminantAnalysis,205207,293,298300
factoranalysis,207,300303
logisticregressionanalysis,205207,300
multipleregressionanalysis,205207,294298
Complexassociationalquestion,7678,438439
example,439
Complexdifferencequestionsandanalyses,7678,292294,437438
example,438
Compositevariable,seeFactoranalysis
Concurrentevidenceforcriterionvalidity,321322
Confidenceinterval,176177,221222
standarderrorofmeasurement,316317
Confidential,37,351
Conflictofinterest,347,360361
Confounding,375,394
Consentform,seeInstitutionalreviewboard
Construct,48,332324,338,374375,393394

Page455

Constructvalidity,322324,393394
convergentevidenceforconstructvalidity,323324
discriminantevidenceforconstructvalidity,323324
examples,323324
factorialevidenceforconstructvalidity,301,323324
Constructivistparadigm,seealsoQualitative,viii,4,810,1526
blendedapproaches,2627
causalityaxiom,2122,27,82
collectingandrecordingdata,9,25
dataanalysis,9,25
generalizationaxiom,2021,27
humaninstrument,25
knowertoknownaxiom,1920,27
planninglogistics,25
realityaxiom,1819,27
trustworthiness,26
valuesaxiom,2223,27
Consumerofresearch,seeResearchandEvaluatingresearch
Contentvalidity,320
Continuousvariable,82,132,252,seealsoRatioscale
Controlgroup,66,70,seealsoComparisongroup
Controlofextraneousvariables,seeInternalvalidity
Conveniencesample,seeNonprobabilitysampling
Convergentevidenceforconstructvalidity,323
Correlationalapproach,seeAssociationalapproach
Correlationcoefficients,72,253,312313
Correlationmatrix,254255,295,301
diagonal,255
Cramer'sV,205206,256257
Credible,seeQualitative
Criterion,321
Criterionreferencedtests,334
Criterionrelatedvalidity,321322
concurrentevidenceforcriterionvalidity,321322
predictiveevidenceofcriterionvalidity,321
Criterionvariable,257,293297,seealsoDependentvariable
categorical,298
Criticalvalue,174
Cronbach'salpha,301,314,316
formula,316
D
d(effectsizeindex),177178
Data,355356
fabricationandfalsification,355356
integrityof,355356
Datacollection,330341,351356
Debriefing,349
Deception,348350
Decisiontrees,202203,215,226,228,240,245
Deductivereasoning,24
Degreesoffreedom(df),218,240,269
Dependentvariable,51,200,252
Descriptiveresearchapproach,6264,7274,330332
Descriptiveresearchquestions,7478,437
examples,77,437
Descriptivestatistics,136146
Designandanalysissectionofanarticle,431
Designclassification,seegeneraldesignclassification
Designterminology,95
Dichotomousvariable,131133,201207,298300,304
Differencehypothesisorquestions,7478,198206,437438
examples,77,437438
Differenceinferentialstatistics,76,199206
factorial(complex),205206
MANOVA,205206
singlefactor(basic),202204
Differencescores,285
Directionalhypothesis,seealsoAlternativehypothesis,167
negativedirectionalhypothesis,167,171
positivedirectionalhypothesis,167,171
Directobservation,seeObservation
Directionofeffect,175
Disciplinedinquiry,seeResearch
Discretevariable,seealsocategoricalvariable,132
Discriminantanalysis,293,298300
conditions,300
discriminantfunctioncoefficients,299
discriminantfunctionequation,299

Page456

example,299300
Discriminantevidenceforconstructvalidity,323
Discussionsection,395,433434
Disordinalinteraction,seeinteractioneffect
Dropouts,seeExperimentalmortality
Dummyvariable,seeDichotomousvariable
Dunnett'stest,225
E
Ecologicalexternalvalidity,seeExternalvalidity
Effectsize,20,176,177178,370,378380,381
deffectsizeindex,177178
interpretation,178
metaanalysis,378380
r(Pearsonproductmomentcorrelationcoefficient),177178,253254
Elements,145146
Equalityofvariances,seeHomogeneity
Equivalenceofgroupsonparticipantcharacteristics,seeInternalvalidity
Error,312,317
Errorvariance,217,239,265,371
Ethicalissues,3139,346362,seealsoInstitutionalreviewboard
Belmontreport,3338
historyof,3233
principlesandpolicies,3339
Evaluatingresearch,viii,57,82,89,158162,386398
example,395396
Experimentalapproaches,6268,94106,330332,349350
Experimentalmortalitythreat,8687,347348
Externalvalidity,65,82,156162,376,377,391397
ecological,159160,333,335,391397
population,158162,346,391397
randomselection,169
singlesubjectdesigns,120
ExtraneousVariable,52
F
Fstatistic(...ratio),175,223224,243,270,280281,293
Facevalidity,320
Factor,300
Factoranalysis,300303
conditions,303
constructvalidity,323
example,301303
factorloadings,301
stoppingrules,302
FactorialANOVA,seeTwofactorANOVA
Factorialdesign,51,205206,265273
analysis,269273
Factorialevidenceforconstructvalidity,301,323
Fieldresearch,710,160
Fisherexacttest,120,204
Focusgroup,330332,341,351
Frequencydistribution,134135,136,137
Frequencypolygons,133,136137
Friedmantwowayanalysisofvariancebyranks,246
G
Gainscoreanalysis,285286
Generaldesignclassifications,ix,181195,200201,387
Generallinearmodel(GLM),207208
Generalizationaxiom,2021
Generalize,seeExternalvalidity
Ghostauthor,seePublication
Graphingdata,267
Guestauthor,seePublication
H
Hawthorneeffect,1920,88,393
Hierarchicalmultipleregression,297
Historythreat,88
Homogeneityofvariance(assumptionof...),202,219
HotellingT2293
Humanresearchparticipants,3339,352354
Belmontreport,seeEthicalissues
ethicalissues,3339,346354
risksandbenefits,3739,354
voluntaryinformedconsent,3237
Hypothesis,2324,45,5355,7477,436439
Hypothesistesting,166176
alternativehypothesis,166
nullhypothesis,166

Page457

I
Independence(assumptionof...),200,202,219
Independentsamples,204,seealsoBetweengroups
Independentvariable,4851,199200,252
active,4849
attribute,4950,53
causalinferences,50
factor,51
levelsof,5051,6971,200
numberof,51,200
Individualdifferencesapproach,6264,6872,330332,350
Inferentialprocess,168170
Inferentialstatistics,ix,76,165180
Informedconsent,seeVoluntaryinformedconsent
Institutionalreviewboard(IRB),33,347354
consentform,354
howtheywork,352353
researchprotocol,353354
Instrumentvalidity,seeMeasurementvalidity
Instrumentation,330341,351,430
threat,8990
Intelligencetest,seeAptitudetest
Interactioneffect,264269,282284,437438
disordinal,268
factorialdesigns,264269
mixedfactorialdesigns,282,284
ordinal,268
Interitemreliability,314316
Internalconsistencyreliability,314316
Internalvalidity,65,8290,158,161162,376,377,387397
randomassignment,169
singlesubjectdesign,120
Interquartilerange,139
Interrater(interobserver)reliability,317318
Intervalscaleofmeasurement,130
equalintervals,131
intraclasscorrelationcoefficients,318
Intervention,seeTreatment,andActiveindependentvariable
Interview,330332,338341
facetoface,340341
indepth,341
telephone,340341
Intraclasscorrelationcoefficient(ICC),318
Introductionofanarticle,428429
Inverserelationship,seeNegativerelationship
K
Kappastatistic,318
KruskalWallis(onewayANOVA),204,228230
KuderRichardson20(KR20),314315
formula,315
L
Laboratoryresearch,710,160
Leastsignificantdifference(LSD)posthoctest,225
Leastsquaresfit,296
Levelsandscalesofmeasurement,129133,201207
Levelsofavariable,5051,6971
Likertscales,316,337338,340
Linearcombination,295
Linearcompositedependentvariable,292293
Linearregression,seeCorrelation,257259
calculation,257259
singlefactorANCOVA,286
Literature,seeResearchliterature
Literaturereview,4647,428429
Logicalpositivistparadigm,seePositivistparadigm
Logisticregressionanalysis,300
conditions,300
Longitudinalstudies,8788,186,388
M
Maineffect,264269,284
factorialdesigns,264269
mixeddesigns,284
Manipulatedvariable,seeActiveindependentvariable
MannWhitneyUTest,120,203204,227228
example,228
posthoccomparison,230
MANOVA,seeMultivariateanalysisofvariance

Page458

Maskedreview,seePeerreview
Matching,85,105,185186,239,288
Maturationthreat,88
McNemartest,204,247248
Mean(Alsocalledarithmeticaverage),138139
calculation,138
Meansquare(MS),224,243,269
Measuredvariable,seeattributeindependentvariable
Measurement,128142,387
Measurementreliability,310319,334335,338,351,355,376,388389,430
example
interraterreliability,317318
reliabilityassessment,313319
summary,319
testscores,311312
Measurementreliabilityandstatistics,82,368376,389397
appropriatenessofinterpretation,373374
appropriatenessofpower,368373
appropriatenessofstatisticaltechniques,373
reliabilityofmeasures,368
Measurementscales,seeLevelsandscalesofmeasurement
Measurementvalidity,82,319324,332,334338,351,388389,393,430
contentvalidity,320
constructvalidity,322324
criterionrelatedvalidity,321322
facevalidity,320
Measurementvalidityandgeneralizabilityoftheconstructs,82,319,374375,391397,430
examples,374375
Measuresofcentraltendency,138139
mean,138139
median,138
mode,138
Measuresofvariability,139
interquartilerange,139
standarddeviation,139
Median,138
Metaanalysis,20,378381
effectsize,378380
issueswithuse,380381
performingametaanalysis,381
Methodsection,346354,429431
Misconduct,seeScientificmisconduct,Plagerism,andEthicalissues
MixedANOVA,205206,279282
errorterms,281
Fvalues,280281
interactioneffect,282
PretestPosttestdesign,282285
simplemaineffectsanalysis,282
sourcetable,280281
Mixedfactorialdesigns,183184,189,190,201,264,277290
analysisofcovarianceapproach,286288
describingthedesign,190
diagrammingthedesign,189
gainscoreanalysis,285286
mixedANOVAanalysis,279282
numberandtypeofindependentvariables,184
PretestPosttestdesign,282288
truemixeddesigns,279282
Mode,138
Moderatestrengthquasiexperimentaldesigns,9899
Monotonicrelationship,256
Multiplebaselinesinglesubjectdesigns,113116
acrossbehaviors,115
acrosssettings,116
acrosssubjects,114
flexibility,116
randomassignment,116
Multipleregressionanalysis,72,294298
betaweights,295
conditions,298
hierarchicalmultipleregression,297
partialregressioncoefficients,295
simultaneousmultipleregression,295297
stepwiseregression,297298
Multiplettests,222
Multivariate,7678,292,seealsoComplexassociational
MultivariateAnalysisofVariance(MANOVA),205206,291,292294
conditions,294
followupprocedures,293
HotellingT2,293
Pillai'strace,293
Roy'slargestroot,293
Wilk'slambda,293

Page459

MultivariateF,293
N
Naturalistparadigm,seeConstructivistparadigm
Negativerelationship,253,312
NewmanKeulsposthoctest,225
Nominalscaleorvariable,52,129130,132,201206,230,244,340
Nondirectionalhypothesis,167
Nonequivalentdependentvariablesdesign,97
moderatestrengthquasiexperimentaldesigns,9899
strongquasiexperimentaldesigns,99100
weakquasiexperimentaldesigns,98
Nonexperimental,seeIndividualdifferencesapproaches
Nonlinear,300
Nonparametricstatistics,169,202207,227234,244248
CochranQtest,247248
Cramer'sV,256257
Friedmantwowayanalysisofvariancebyranks,246
KruskalWallis(onewayANOVA),204,228230
MannWhitneyUtest,227228
McNemartest,247248
phicoefficient,256257
Spearmanrankordercorrelation(rho),255256
Wilcoxonsignedranksmatchedpairstest,245246
Nonprobabilitysampling,149,152157,161,339
convenience,155157
purposive,154155
quota,154
snowball,155156
whycommon,156157
NormalCurve,134136
areas,135
frequencydistribution,134135
kurtosis,135
probabilitydistribution,135
skewed,135
Normallevelofmeasurement,seeNormallydistributedvariable
Normality,assumptionof...,201206,219
Normallydistributedvariable,132,201206
Normreferencedtests,334
''NoTreatment"effect,70
Nullhypothesis,166,167,170,171,369
Nullhypothesissignificancetesting(NHST),176178,356
O
Objectivemeasures,9,334338
Objectivity,9,2223
Observation,810,177,330333
breadth,333
detail,383
observerasparticipantsetting,333
Observedscore,311312,317
Onegroupposttestonlydesign,9596
Onegrouppretest/posttestdesign,9697
OnewayANOVA,seeSinglefactorANOVA
Openendedquestions,339341
Operationaldefinition,48
Operationalize,seeOperationaldefinition
Ordereffect,238
Orderedvariable,52
Ordinalinteraction,seeInteractioneffect
Ordinalscaleorlevelofmeasurement,130,132,201205,244,255256
Orthogonal,300302
Outcomevariable,seeDependentvariable
Outliers,139
P
pvalue,172
Pairedsamplesdesign,seeWithinsubjectsdesign
Paperpenciltests,117
Paradigm,1617
qualitativeparadigm,17
quantitativeparadigm,17
Parallelformsreliability,314
Parametricstatistics,201205,215227,240244,252259
linearregression,257259
Pearsonproductmomentcorrelation(r),252255
singlefactorANOVA,222227
singlefactorANOVAwithrepeatedmeasures,240244

Page460

ttestofindependentsamplesorgroups,215222
ttestforcorrelatedorpairedsamples,240242
Partialregressioncoefficients,295
Participants,145146,429430,seealsoHumanresearchparticipants
Pathanalysis,207
Pearsonproductmomentcorrelation(r),252255
effectsize,177178,253254
r2,259
statisticalsignificance,253
testretestreliability,253
Peerreview,360,386
Percentageagreementmethods,317318
Performanceassessment,335
Phicoefficient,205206,256
Pillai'strace,293
Pilotdata,95,348,353
pilottestingvs.pilotstudy,353
Placebo,350,seealso"Notreatment"effectandControlgroup
Plagerism,355,357
Poorquasiexperimentaldesign,9597,193
onegroupposttestonlydesign,9596
onegrouppretest/posttestdesign,9697
posttestonlydesignwithnonequivalentgroups,97
Population,55,145161
accessible,146148,155161
theoreticalortarget,146148,159161
Populationexternalvalidity,seeExternalvalidity
Populationparameter,166
Positiverelationship,253,312
Positivistparadigm,4,89,1526
blendedapproaches,2627
causalityaxiom,2122,27
generalizationaxiom,2021,27
knowertoknownaxiom,1920,27
realityaxiom,1819,27
valuesaxiom,2223,27
Posthoctests,225
Dunnett'stest,225
followingsinglefactorANOVA,225227
leastsignificantdifference(LSD)posthoctest,225
NewmanKeulsposthoctest,225
Scheffeposthoctest,225
TukeyHSDhonestlysignificantdifference(HSD)posthoctest,225,244
Posttestonlycontrolgroupdesign,103,314
Posttestonlydesignwithnonequivalentgroups,97
Posttestscores,286
Power(...ofastatistic),158,220,354,368373,389,430
appropriatenessofpower,368373
conceptofpower,370373
evaluatingpower,373
increasingpower,370373
Practicalsignificanceorimportance,157
Predictiveevidenceofcriterionvalidity,321
Predictorvariable,257,293297,seealsoIndependentvariable
categorical,300
PretestPosttestcontrolgroupdesign,103104,311
PretestPosttestdesign,282288
analysisofcovarianceapproach,286288
gainscoreanalysis,285286
mixedANOVAanalysis,283285
recommendations,288
PretestPosttestnonequivalentcomparisongroupdesigns,97100,192,seealsoQuasiexperimental
moderatestrengthquasiexperimentaldesigns,9899
strongquasiexperimentaldesigns,99100
weakquasiexperimentaldesigns,98
Pretestscores,286
Privacy3637
Probability,166176
Probabilitydistribution,135
Probabilitysampling,149153,161
clusterrandom,152153
simplerandom,149150,153
stratifiedrandom,151153
systematicrandom,150153
Proceduresection,430431
Projectivetechniques,337
Publication,357362
authorship,358359
multiple,357
plagerism,357
Purposivesampling,seeNonprobabilitysampling

Page461

Q
Qualitativedataanalysis,9
Qualitativedataanddatacollection,9
Qualitativeresearch,seealsoconstructivist,viii,810,74,154156,330332,341,348,350,355
ethnography,19
Quantitativedataanalysis,9,193308
Quantitativeresearch,810,17,330333,348
Quasiexperimentalapproach,6268,84,95102,106,193,214
factorialdesigns,266
mixedfactorialdesigns,278279
withinsubjects,238
Questionnaire,330332,336340
directlyadministered,339
mailed,339
typesofitems,339340
Questionnaireitem,337340
Quotasampling,seeNonprobabilitysampling
R
r(Pearsonproductmomentcorrelationcoefficient),175,177178,295,312,315
r2,259,295
R,295296
R2,295297
Randomassignmentofparticipantstogroups,6367,161162,394,430
Randomassignmentoftreatmentstogroups,84,99100
Randomnumbertable,64,149150
Randomorderofconditionsortreatmentstoeachparticipant,192
Randomselectionorsampling,65,149153,161162,394,seealsoProbabilitysampling
Randomizationtests,120
Randomizedexperimentaldesignwithmatching,105
Randomizedexperimentaldesigns,103105,193
measurementreliability,311
posttestonlycontrolgroupdesign,103
PretestPosttestcontrolgroupdesign,103104
randomizedexperimentaldesignwithmatching,105
Solomonfourgroupdesign,104105
Randomizedexperimentalresearchapproach,6267,103106,192,214,238,266,278
example,6667
factorialdesigns,266
mixedfactorialdesigns,278
withinsubjects,238
Rank,133,201,204205,227
Rate,133,301,336340
Ratingscale,301
Ratioscale,130
Realityaxiom,1819
Referencesectionofanarticle,434
Relatedsamplesdesign,186,seealsoWithinsubjectsdesign
Reliability,seemeasurementreliability
Reliabilityassessment,313319
Cronbach'salpha,314,316
interraterreliability,317318
KuderRichardson(KR)20,314
parallelformsreliability,314
splithalfreliability,314315
standarderrorofmeasurement,316317
testretestreliability,313314
Reliabilitycoefficient,312313
Repeatedmeasuresdesigns,seeWithinsubjectsdesigns
Repeatedmeasuresindependentvariable,seeWithinsubjectsindependentvariable
Repeatedtestingthreat,8889
Representativesampling,seeSampling
Research,410
articleorpaper,427434
atrocities,32
consumersof,viii,57
definitionof,45
disciplinedinquiry,4
dimensionsanddichotomies,710
laboratoryvs.field,710
qualitativevs.quantitative,810
selfreportvs.observation,810
theoreticalvs.applied,7
purposesof,57
developtheory,56
developtools,67
increaseknowledge,57
increaseownunderstanding,7
practicalapplication,6

Page462

Researchapproach,6274,198199,387
associational,6264,7173
combinationsof,74
comparative,6264,6873
descriptive,6264,7274
individualdifferences,62,6872
quasiexperimental,6268
randomizedexperimental,6267
Researchliterature,46
Researchproblem,4448,5455,435436
broadvs.narrow,47
characteristics,4648
examples,4445,5455,436
notpersonalproblem,46
notsocialproblem,46
sourcesof,46
wellresearchedvs.unknown,4748
widespreadvs.limitedinterest
Researchprocessstepsin,346360
Researchprotocal,seeInstitutionalreviewboard
Researchquestion,viii,23,5355,7478,198199,387388,436439
associational,7478,438439
basicvs.complex,7478,437439
descriptive,7478,437
difference,7478,437438
Researchreliability,376
Researchreplication,378381
metaanalysis,378381
significancetesting,378
Researchvalidity,8290,158162,368377,389397
externalvalidity,158162
internalvalidity,158162
measurementreliabilityandstatistics,368376
measurementvalidityandgeneralizabilityoftheconstructs,319,374375
reliabilityoffindings,377
summary,375376
validitycategories,377
Responserate,147148,seealsoActualsample
Restrictedrangeofscores,321322
Results,355360,395
dissemination,357360
interpretation,356,392
sectionofanarticle,355356,432433
Resume,seePublication
Reversalsinglesubjectdesigns,111113
baselinephase,111
example,111
stabilityofbaseline,112
treatmentphase,111
treatmentwithdrawal,112
Risk,seeHumanresearchparticipants
Roy'slargestroot,293
S
Sample,144162,seealsoSelectedsampleandActualsample
Samplesize,372
Samplestatistic,166
Sampling,144162,346,394,429430
advantages,145
keyconcepts,145148
nonprobability,149,152157
populationexternalvalidity,158162
probability,149153,394
representative,145152
samplesize,104
stepsin,148149
typesof,147156
Samplingdesign,147148,seealsoSampling
Samplingframe,seeAccessiblepopulation
Scalesofmeasurement,seeMeasurementscales
Scheffeposthoctest,225
Scientificmethod,2324
Scientificmisconduct,355356,361362
Screening,155
Selectedsample,146161
Selectionofstatistics,ch.9,198208,389392
basicassociational,203206
basicdifference,202204
complexassociational,205208
complexdifference,205206
descriptive,133,136
inferential,198208
Selectionthreat,87
Selfassignmentorselection,84,98
Selfreportmeasures,810,160,330332
Semanticdifferentialscale,338
Serialdependency,118,119
Sharedvariance,295
Significancelevel(alphalevel),171,174
Significancetesting,378
Simplemaineffects,272273,282,284

Page463

Simplerandomsampling,seeRandomselectionandProbabilitysampling
Simultaneousmultipleregression,295297
equation,296
SinglefactorANOVA,203204,215,222227
interpretationofFvalue
meansquarebetweengroups,224
meansquarewithinsubjects,224
posthoctest,225227
sourcetable,223
sumsofsquares(SS)(ch.14ff),224
SinglefactorANOVAwithrepeatedmeasures,204,240244
sumsofsquares(SS),243
Singlefactorbetweengroupsdesigns,190,191,202204,214235
chisquaretestforindependence,203204,230233
KruskalWallis(onewayANOVA)(ch.13,14),204,228230
nonparametricstatistics,203204,227234
MannWhitneyUtest,203204,227228
SinglefactorANOVA,203204,222227
ttestofindependentsamplesorgroups,203,215222
Singlefactordesigns,71,201204,214235,237250,251261
betweengroupsdesigns,214235
continuoussinglefactordesigns,251261
withinsubjectsdesigns,237250
Singlefactorwithinsubjectsdesigns,204,237250
CochranQtest,204,247248
degreesoffreedom(df),240
Friedmantwowayanalysisofvariancebyranks,204,246
McNemartest,204,247248
meansquares,243
ordereffect,238
posthoctest,244
singlefactorANOVAwithrepeatedmeasures,204,240244
sumsofsquares,243
ttestforcorrelatedorpairedsamples,204,240242
Wilcoxonsignedranksmatchedpairstest,204,245246
Singlesubjectdesign,68,109123
conceptualizationof,110
evaluationoftheresults,117120
level,118
meanlevel,118
methodsofmeasurement,117
measurementperiods,117
multiplebaselinedesigns,113116
reversaldesigns,111113
slope,119
statisticalanalysis,119120
trend,118
trendlines,119
variability,119
visualanalysis,118120
Snowballsampling,seeNonprobabilitysampling
Socialdesirability,331332
Solomonfourgroupdesign,104105,191,193
SourceTable(sourceofvariancetable),223,243,269,280281
mixedANOVA,280281
singlefactorANOVA,223
singlefactorwithinsubjectsANOVA,243
twofactorANOVA,269
SpearmanBrownformula,315
Spearmanrankordercorrelation(rho),205206,255256,seealsoCorrelationandPearsonproductmomentcorrelation
Specificexperimentaldesigns,93108,388
Splithalfreliability,314315
SpearmanBrownformula,315
Standarddeviation,139140,217,371372
calculation,139140
deviationscoreformula,139
rawscoreformula,140
Standarderrorofmeasurement,316317
equation,316
Standarderrorofthedifferencebetweenmeans,217
Standardnormalcurve,136
Standardizedinstruments,330338
achievementtests,334335
aptitudetests,335336
attitudescales,337338
MentalMeasurementsYearbook,332334,336
objectivescoring,334338
performanceandauthenticassessments,335

Page464

personalityinventories,336337
TestCritiques,332,334,336
tests,334336
Statisticalassumptions,201208
Statisticalconclusionvalidity,seemeasurementreliabilityandstatistics
StatisticalPackagefortheSocialSciences(SPSS),174
Statisticalregressionthreat,86
Statisticallysignificant,157158,170,174
Stepwiseregression,297298
Strata,147,seealsoSampling
Stratifiedrandomsampling,147,151153
Stratifiedsamplingwithdifferentialprobabilitiesofselection,151153
Strongpositiverelationship(ch.16)
Strongquasiexperimentaldesigns,99100
Structuredequationmodeling(SEM),207
Subjects,seeParticipants
Summatedratingscale,336338,340
Sumsofsquares(SS),224,243,269
Surveypopulation,seeAccessiblepopulation
Surveyresearch,338339,seealsoQuestionnaireandInterview
Systematicrandomsampling,150153
Systematicvariation,312
T
ttest(...ofindependentsamplesorgroups),175,177,202204,215222
calculationwithequalsamplesizes,217
calculationwithunequalsamplesizes,217
conceptualview,216
degreesoffreedom(df),218
degreesoffreedomformula,219
homogeneityofvarianceassumption,219
independenceassumption,219
multiplettests,222
normalityassumption,219
understandingthetvalue,218
ttestforcorrelatedorpairedsamples,204,240242
calculation,241242
Testing,314,331338,seealsoStandardizedinstruments
Testretestreliability,313314
criteria,313314
Testscores,311312,334336
Theoreticalortargetpopulation,168,311
Theory,5,7,25,4647,5355
developmentof,5
Thirdvariable,288
Threatstointernalvalidity,8390,96
Timeseriesdesign,6768,100102
multiplegrouptimeseriesdesigns,102
continuoustreatment,102
temporarytreatment,102
singlegrouptimeseriesdesigns,101102
continuoustreatment,101102
temporarytreatment,101
singlesubjectdesign,109123
Titleofanarticle,385,427428
Treatment,seeActiveindependentvariable
Treatmentvariance,216,239,371
Trueexperimentaldesigns,seeRandomizedexperimentaldesigns
Truemixeddesigns,279282
mixedANOVAanalysis,279282
Truescore,312,317
Tukeyhonestlysignificantdifference(HSD)posthoctest,225,244,287
Twobytwofactorialdesign,seeFactorialdesigns
Twofactordesigns,seeFactorialdesign
Twofactordesignwithrepeatedmeasuresonthesecondfactor,seeMixedfactorialdesigns
Twofactor(twoway)ANOVA,205206,269273
interactioneffect,272273
interpretation,271273
maineffect,272273
simplemaineffects,272273
sourcetable,270
Twotailedtest,174
TypeIerror,170173,174,223,294,369,373
TypeIIerror,170173,369
U
Univariate,76,292
Unorderedvariable,seeNominalvariable
Unrepresentativesample,seeNonprobabilitysample
Unsystematicvariation,312
V
Validity,seeMeasurementvalidityandResearchvalidity

Page465

Values,seeLevelsofavariable
Variable,4853,387
definitionof48
dependent,51,387
extraneous,52
independent,4851,387
levelsof,5051
orderedvs.nominal,52
otherconsiderations,5253
setsofvariables,53
Variability,139140
Voluntaryinformedconsent,3237,349
assent,35
comprehension,35
debriefing,36
information,35
voluntariness,3536
W
Weakquasiexperimentaldesigns,98
Wilcoxonsignedranksmatchedpairstest,204,245246
Wilk'slambda,293
Withinsubjectsdesigns,183186,190,200205
attributeindependentvariable,185186
describingthedesign,190
diagrammingthedesign,187190
examplesofwithinsubjectsdesign,185186
matchingofparticipants,185186
numberandtypeofindependentvariables,184186
relatedsamplesdesign,186
Withinsubjectsindependentvariable,184,187,264,278,280,284
Withinsubjectsrandomizedexperimentaldesign,192
Withinsubjectsvariance,224
Y
Yatescorrection,233
Z
zscore,136,317
Zeroordercorrelations,255