See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/306068730

Cross-Battery Assessment? XBA PSW? A case of mistaken identity: A

commentary on Kranzler and colleagues' “Classification agreement analysis of
Cross-Battery Assessment in the ident...

Article · July 2016

DOI: 10.1080/21683603.2016.1192852

CITATIONS READS

4 261

2 authors:

Dawn P. Flanagan W. Joel Schneider

St. John's University Temple University
68 PUBLICATIONS   1,691 CITATIONS    31 PUBLICATIONS   730 CITATIONS   

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Dawn P. Flanagan on 21 May 2018.

The user has requested enhancement of the downloaded file.

INTERNATIONAL JOURNAL OF SCHOOL & EDUCATIONAL PSYCHOLOGY
2016, VOL. 4, NO. 3, 137–145
http://dx.doi.org/10.1080/21683603.2016.1192852
POINT-COUNTERPOINT: RESPONSE
Cross-Battery Assessment? XBA PSW? A case of mistaken identity: A commentary
on Kranzler and colleagues’ “Classification agreement analysis of Cross-Battery
Assessment in the identification of specific learning disorders in children and
youth”
Dawn P. Flanagana and W. Joel Schneiderb
a
Department of Psychology, St. John’s University, Queens, New York, USA; bDepartment of Psychology, Illinois State University, Normal, Illinois, USA
When education works, it works to the benefit of all.
It creates productive, innovative citizens eager to
contribute to a well-functioning democracy. In contrast,
educational failure has lifelong consequences, with
some individuals experiencing decades of preventable
hardship. Like Kranzler, Floyd, Benson, Zabowski, and
Thibodaux (this issue), we have dedicated our lives to
finding better ways of identifying children who are falling
behind in school and need additional help. With finite
resources, every diagnostic error is a misallocation of time,
effort, and money with predictably damaging conse-
quences for individuals, families, and society as a whole.
Science thrives when all ideas are subject to scrutiny,
including (and especially) those that are longstanding and
influential. Hard-hitting criticism of current practices (or
even gentle, thoughtful critiques) may be difficult for
advocates of current practices to hear, but free and open
exchange of opinion is essential for progress. Flanagan and
colleagues have presented a set of systematic procedures for
identifying specific learning disabilities (SLDs; Flanagan,
Ortiz, & Alfonso, 2013). Because the procedures have been
widely adopted by school psychologists, they should
be carefully scrutinized and rigorously evaluated. To the
degree that the procedures are suboptimal, they should be
adjusted, reformed, or set aside for something better. This
does not necessarily mean that they were missteps. Some
previously popular but now abandoned ideas were essential
stepping stones on the path to progress. Indeed, the
procedures that Flanagan and colleagues currently advocate
have undergone many revisions since they were first
proposed, often in response to critical feedback. Revisions
that take into account valid criticism are quite likely needed
to optimize SLD identification.
CONTACT Dawn P. Flanagan flanagad@stjohns.edu School Psychology Program, Department of Psychology, St. John’s University, 8000 Utopia Parkway,
Queens, NY 11439, USA.
Note from the Editor: This is the second of a three article series.
q 2016 International School Psychology Association
Kranzler et al. (this issue) present evidence that they
believe casts doubt on the validity of Flanagan and
colleagues’ procedures. Because of our great respect for
these scholars’ previous work, we were extremely puzzled
by the conceptual errors in their arguments. We found
their evidence to be neither persuasive nor, to speak
plainly, at all relevant to the question of the accuracy of
SLD identification. Unfortunately, although their critique
is nominally directed at Flanagan and colleagues’ methods,
in truth it is a critique of a set of procedures that are
endorsed by no one, most emphatically not by Flanagan
and colleagues. As will be shown, the differences between
what Kranzler et al. (this issue, p. 125) call “XBA PSW” and
what Flanagan and colleagues advocate are stark, and in
many important ways are diametrically opposed.
Aspirations and conflicts of interest
It should be disclosed that one of us (DPF) is the primary
architect of the ideas and procedures under scrutiny,
and, despite earnest affirmations of her commitment to
scientific neutrality, has a financial stake in these matters in
the form of book sales, software sales, and paid workshops.
The second author (WJS) is an independent scholar with no
personal, professional, or financial interest in the outcome
of this commentary. WJS has declared no allegiance to the
procedures advocated by Flanagan and colleagues and has a
history of suggesting alternative procedures of his own (e.g.,
Schneider, 2013), a practice he intends to continue. We join
together here because both of us simply want the field to
advance. We believe it is better to be accurate in the long run,
than to have only the appearance of being accurate right
now. However, if Flanagan and colleagues’ methods need to
138 D. P. FLANAGAN AND W. J. SCHNEIDER
change, the motivating arguments for change must be valid.
For reasons we articulate below, we do not believe that
the evidence presented by Kranzler et al. provides a sound
rationale for change at this time.
Preliminary clarification of terms: XBA
PSW
Before addressing the heart of the matter before us, a
little housecleaning is in order: We need to disentangle
Cross-Battery Assessment (XBA) from SLD identifi-
cation. Until recently, Flanagan and colleagues referred
to their method of SLD identification as simply “an
operational definition of SLD” or “a CHC-based
operational definition of SLD.” Their operational
definition was first presented nearly 15 years ago
(Flanagan, Ortiz, Alfonso, & Mascolo, 2002) and has
been revised and updated several times over the past
decade to insure that it was consistent with contemporary
theory and research (e.g., Flanagan, Ortiz, Alfonso, &
Mascolo, 2006; Flanagan, Ortiz, & Alfonso, 2007, 2013;
Ortiz, Flanagan, & Alfonso, 2015). However, because
their operational definition was often called by others
“XBA,” rather than being conceived of as a method that
was separate from yet compatible with XBA, Flanagan
and colleagues (2013) gave it a name—the Dual
Discrepancy/Consistency operational definition of SLD
(or the DD/C method for short).
Flanagan and colleagues developed a software
program to assist with the quantitative analyses necessary
to determine whether a particular pattern of strengths
and weaknesses in cognitive and academic performance
supports SLD based on DD/C criteria. The program that
Kranzler et al. referred to in their study (although did not
use in the manner it was intended to be used) is called,
“Cross-Battery Pattern of Strengths and Weaknesses
Analyzer” (PSW-A v1.0; Flanagan et al., 2013), hence the
use of the term “XBA PSW” in their study. The revised
version of the PSW-A program is included in the Cross-
Battery Assessment Software System (X-BASS; Ortiz et al.,
2015), wherein the output page for the PSW analysis
is clearly labeled: Dual-Discrepancy/Consistency Model:
Summary of PSW analyses for SLD. One does not need
to use cross-batteries or use the XBA approach to apply
DD/C criteria. However, since most single batteries
do not measure the breadth of cognitive and academic
constructs considered necessary to inform SLD identifi-
cation decisions, crossing batteries is typically necessary;
accordingly, there is a close connection between XBA and
the DD/C method.
XBA is a method for combining tests from different
batteries and predates DD/C by several years (Flanagan
& McGrew, 1997; Flanagan & Ortiz, 2001). The XBA
approach is grounded mainly in Cattell-Horn-Carroll
(CHC) theory and research (McGrew, 2005; 2009;
Schneider & McGrew, 2012). Unlike other “flexible
battery” practices, rigorous procedures and methods
accompany XBA to insure that any assessment that
expands beyond the confines of a single battery is both
psychometrically and theoretically defensible. To assist
in XBA and in interpretation of cross-battery data,
X-BASS was developed (Ortiz et al., 2015). X-BASS is
an integration and substantial revision of the software
programs that accompanied the second and third
editions of Essentials of Cross-Battery Assessment
(Flanagan et al., 2007, 2013).
Although XBA can be used in the context of SLD
identification, it has many other applications. Given that
DD/C and XBA have been described in the same works
and that they are often used together, it is nevertheless
easy to see how they can be conflated. However, XBA
and DD/C are independent and conceptually distinct
procedures. If somehow well-grounded critiques caused
one of them to fall, the other would still be left standing,
fully intact.
Given that the use of the term DD/C is fairly recent,
it is understandable that Kranzler et al. referred to DD/C
as “XBA PSW.” Nevertheless, it is a misleading term.
Kranzler et al.’s study has no bearing on XBA per se:
Multiple batteries were not crossed, nor were multiple
scores combined. That is, the authors used a single
battery in their study, the WJ III. Single-battery
assessment is not Cross-Battery Assessment. Still, it can
be conceded that XBA, more broadly conceived, is a
procedure for combining scores, even if scores come from
the same battery. However, this is not what Kranzler et al.
did. Even though multiple measures of relevant
constructs from the WJ III were available to the authors,
they chose to use single measures (subtests) instead. XBA,
narrowly or broadly conceived, makes no appearance
whatsoever in Kranzler et al.’s study. Thus, it should
be clear that whatever the results of Kranzler et al.’s study,
they do not have any bearing on the validity of the XBA
procedures for combining scores in a theoretically and
psychometrically defensible manner.
Limitations of Kranzler et al.’s methods and flaws
in their reasoning
Our primary concern about the methods used by
Kranzler et al. is that the DD/C operational definition
of SLD (which Kranzler et al. refer to as “XBA PSW”) was
not applied accurately. While some of these inaccuracies
are probably unavoidable, the most consequential
inaccuracies are “unforced errors” on the part of Kranzler
et al. We will start with comparatively minor concerns
and proceed to more serious errors.
 INTERNATIONAL JOURNAL OF SCHOOL & EDUCATIONAL PSYCHOLOGY
Regrettable but probably necessary
methodological limitations
How accurate is DD/C when there is no background
information available?
It would be impossible for Kranzler et al. to evaluate the
DD/C method in its entirety using only data from the
WJ III standardization sample. The DD/C method is
intended to be used in conjunction with multiple data
sources, including prereferral intervention information,
parent and teacher reports, classroom observations,
results of an evaluation of exclusionary factors, and so
forth. These additional data sources are necessary to
assist practitioners in interpreting the results of the DD/C
analysis. In fact, when all components of the DD/C
analysis are supported, meaning that the pattern suggests
SLD, the program output (PSW-A and X-BASS) states,
“This pattern of results does not automatically confirm
the presence of SLD. This pattern must be considered
within the context of the entire case history of the
individual. In addition, other data gathered through
multiple methods need to be considered.” Similarly, when
one or more components of the DD/C analysis are not
supported, meaning that the pattern does not suggest
SLD, the program output states, “This pattern of results
does not automatically rule out the presence of SLD.”
Thus, any study on the classification agreement of
the DD/C operational definition that relies solely on a
quantitative analysis without consideration of other data
sources will yield many false positives and false negatives.
It is quite likely that some, if not many, of the children
in the standardization sample with otherwise SLD-like
scores would fail to meet criteria for SLD if more
information were available. In Kranzler et al.’s analysis,
such children are counted as diagnostic errors, errors that
would not have been made by practitioners following
DD/C procedures accurately. To be fair, several times
Kranzler et al. acknowledge that such cases exist.
However, their analyses do not, nor are their conclusions
softened by an acknowledgement that their analyses
are a worst-case scenario with respect to exclusionary
criteria. That is, because exclusionary factors were not
considered in their study, not a single member of the WJ
III standardization sample met exclusionary criteria and
thus every so-called “error” counted against DD/C.
How accurate would DD/C be if it were implemented
by robots?
When a diagnostic procedure is proposed, it is legitimate
to ask how accurate it would be if it were unflinchingly
followed to the letter, whether it made sense to do so or
not. Such a thought experiment might seem silly, but it
139
helps us know how robust a procedure is with relation to
the complexities and vicissitudes of human experience.
Kranzler et al.’s study essentially answers the following
question: If practitioner judgment were replaced with
inflexibly programmed robots who examined the scores
from children plucked at random from the general
population and did not consider whether the scores
looked unusual, did not consider behavioral observations,
educational history, or exclusionary factors, did not
conduct follow-up testing, did not listen to parent and
teacher reports, and did not pay attention to clinical
details, how accurate would the scores be in and of
themselves in identifying who has an SLD? Despite the
silly way we have framed the question, it is actually an
important and fair question to ask. If we can identify the
situations in which our hypothetical robots fail to arrive
at the correct diagnosis, the procedures can be amended
and improved. However, whenever asking such ques-
tions, researchers should be clear that the answers have
limited scope in how we evaluate the overall validity of
the method.
None of the PSW models currently in use are intended
to be applied in a robotic fashion. Also, any quantitative
data (such as those derived from a PSW analysis) that are
not considered within the context of exclusionary factors
is inconsistent with statutory and regulatory require-
ments. That said, requiring that scholars evaluate the
accuracy of DD/C only when it is performed exactly right
to the last detail is an unrealistic standard. The balanced
approach is to allow for such departures from standard
practice but to clearly label them as such, tempering one’s
conclusions as needed. We believe that Kranzler et al.’s
negative conclusions about DD/C should have been
described clearly as limited in scope vis-à-vis the
necessary limitations of their study’s design.
Methodological shortcomings that are both
regrettable and unnecessary
Single measures make for poor measurement
Dismantling a flawed version of an opposing view is
largely unproductive, and possibly destructive if it
misleads people to dismiss a perfectly valid stronger
version of the view. To conduct a thoroughly convincing
takedown of an argument, it is much more persuasive
to start by restating the best possible version of the
argument and then prove that it is incorrect. To provide a
convincing demonstration that DD/C is error-prone, it
is necessary to show that DD/C, as recommended by
Flanagan and colleagues, is error-prone. Showing that
an inferior version of DD/C is error-prone, especially one
explicitly discouraged by Flanagan and colleagues, simply
140 D. P. FLANAGAN AND W. J. SCHNEIDER
proves that Flanagan and colleagues were right in
discouraging its use.
Kranzler et al. were not forced by necessity to use single
measures instead of multiple measures, as recommended
by Flanagan and colleagues. The WJ III standardization
sample they used includes literally thousands of cases
with reasonably complete data. Kranzler et al. could easily
have used the multiple-measure broad ability scores
from the WJ III (Gf, Gc, Gv, Ga, Glr, Gsm, and Gs).
Indeed, the WJ III has enough supplementary scores that
with a few lines of extra programming, they could have
simulated the decision process of practitioners who need
to conduct follow-up testing when subtest scores in the
same domain are widely discrepant, meaning that the
broad ability score is not cohesive. However, Kranzler
et al. chose to use single measures so that their analyses
would supposedly be more informative for real-world
practices in which saving time is an important
consideration. Unfortunately, this choice makes their
study largely uninformative for the evaluation of DD/C.
Flanagan and colleagues recommend using multiple
measures of a construct for a good reason: single
measures are generally not sufficiently reliable for DD/C,
nor do they have adequate content validity. That is, they
do not cover the broad range of abilities implied by the
broad constructs in CHC theory. Again, even if Kranzler
et al. had succeeded in showing that a single-measure
version of DD/C is not sufficiently accurate, this would
merely have demonstrated that Flanagan and colleagues
were correct in discouraging using single measures in
DD/C. However, this is not where things stand. Although
Kranzler et al. claimed they have shown that a single-
measure version of DD/C is often inaccurate, their
analyses answer an entirely different question, one that
has little bearing on the accuracy of DD/C.
Continuous variables, needlessly dichotomized
Cognitive and academic abilities are continuous vari-
ables. When continuous variables are used to make
dichotomous decisions, as is often the case in SLD
identification, then cut scores must reluctantly be used.
The inescapable downside to using continuous scores
to make dichotomous decisions is that there is little
justice for those who are very close to the wrong side of
the cut score, and the whole procedure is likely to seem
arbitrary and unfair to them.
In the beginning of their paper, Kranzler et al. stated
that its aim is to evaluate what they believe is one of
DD/C’s assumptions: Specific cognitive deficits cause
academic deficits, even in the absence of general cognitive
deficits. This question is not about the accuracy of
DD/C but about the validity of what Kranzler et al.
believe is one of its assumptions. To evaluate this
assumption, dichotomizing continuous data is not
needed. Furthermore, doing so underestimates the effects
of cognitive abilities on academic abilities, thus needlessly
stacking the deck against the viability of the so-called
assumption of DD/C.
Unforced errors and fatal flaws
Here we come to the heart of our critique of Kranzler
et al.’s study. The DD/C method must be evaluated in
terms of the claims that it actually makes. However, the
methods used by Kranzler et al. make DD/C responsible
for predictions it does not and should not make. Not only
do Kranzler et al. start with false premises, the reasoning
they apply to their results is not valid, which leads them
to unsound conclusions about the accuracy of DD/C.
Kranzler et al. misinterpret what Flanagan and
colleagues mean by the term causal
First Kranzler et al. correctly state that the DD/C approach
to SLD identification “is based on the postulate that deficits
in cognitive abilities in the presence of otherwise average
general intelligence are causally related to academic
achievement weaknesses” (p. 127). Then, also correctly,
they state that, “If the model is properly specified, the
purported relations between specific cognitive abilities and
specific areas of academic achievement should hold”
(p. 127). However, their reasoning takes a wrong turn here:
“If specific broad cognitive abilities are causally related to
academic deficits, as Flanagan et al. (2013) presume, then
any sample of individuals with significant cognitive
deficits will, according to their theory, necessarily have
corresponding weaknesses in related academic areas”
(p. 133). In the context of the entire paper, they appear to
take the term causal to mean deterministic, meaning that if
we know the causal inputs, the outcomes can be predicted
perfectly. No one, most certainly not Flanagan and
colleagues, has made such a claim about the causal
relationships between cognitive and academic abilities.
We suspect that Kranzler et al. would deny that this was
their intention, yet this is precisely what their analyses
(both statistical and conceptual) imply.
Kranzler et al.’s analyses demonstrate how well
dichotomized cognitive scores (deficit/no deficit) can
predict dichotomized academic scores (deficit/no deficit).
Kranzler et al. imply that DD/C should be faulted for the
fact that cognitive abilities imperfectly predict academic
abilities. That is, whenever there is a mismatch, it is a
diagnostic error on the part of DD/C. In such an analysis,
a high positive and negative predictive value could only
have been achieved simultaneously if cognitive abilities
 INTERNATIONAL JOURNAL OF SCHOOL & EDUCATIONAL PSYCHOLOGY
Specific Cognitive Deficits
Absent Present
Absent True Negative False Positive
Academic Deficits
Present False Negative True Positive
Figure 1. Diagnostic hits and misses according to Kranzler et al.’s
version of DD/C.
Note. Children with general cognitive deficits are excluded from
this classification.
explained a much, much higher percentage of variance
in academic abilities than they actually do. Yes, cognitive
abilities are causally related to academic abilities, but
the causal relationship is of moderate size, and only
probabilistic, not deterministic. No one, not least
Flanagan and colleagues, has claimed otherwise.
Are we being unfair to Kranzler et al.’s conceptual
analysis? We think not. Implicit in Kranzler et al.’s
classification of diagnostic errors (see Figure 1) are the
following false assumptions about DD/C:
False assumption #1. Among children of at least average
intelligence, everyone with certain specific cognitive deficits
should have corresponding specific academic deficits.
If children with these specific cognitive deficits are
observed to have no academic deficits, then DD/C has
failed. Such children are counted as “false positives”
(Kranzler et al., this issue, p. 132).
False assumption #2. Among children of at least average
intelligence, everyone with certain specific academic deficits
should have corresponding specific cognitive deficits. If
children with these specific academic deficits are observed
to have no specific cognitive deficits, then DD/C has failed.
Such children are counted as “false negatives.”
Both of these assumptions are extreme and unfair to
DD/C.1 To repeat, both claims are explicitly and
emphatically denied by Flanagan and colleagues in their
formulation of DD/C. While some evidence is available to
support causality for certain cognitive abilities and certain
academic outcomes, such as the seminal work of Wagner
and Torgesen (1987), the bulk of the research is
correlational, a fact that has been reported and explained
by Flanagan and colleagues in all of their writings on SLD
identification (e.g., Flanagan, Ortiz, Alfonso, & Mascolo,
2002, 2006; Flanagan et al., 2007, 2013). Whatever causality
1. We assume implicitly that every argument was advanced by Kranzler
et al. in good faith. When we say that an argument is unfair to DD/C, the
unfairness resides in the implications of Kranzler et al.’s arguments, not in
their intentions.
141
claims may have been made by Flanagan and colleagues are
probabilistic in nature. That is, cognitive deficits raise the
risk of academic deficits; they do not guarantee them.
Why not? If a particular specific cognitive deficit causes
a particular academic deficit, should it not always have the
same result? No. The great conceptual leap that Pearson
(1930, p. 2) credits Galton with is the idea that partial
causation can be quantified: “Galton turning over two
different problems in his mind reached the conception of
correlation: A is not the sole cause of B, but it contributes
to the production of B; there may be other, many or few,
causes, some of which we do not know and may never
know.” From here, it is not hard to imagine that there
are many modifiers of the effect of cognitive abilities on
academic abilities. That is, a specific cognitive deficit may
severely undercut academic achievement in some
circumstances but not at all in others. For example,
phonological processing deficits may not result in reading
decoding deficits if the child has an exceptionally skilled
first-grade reading teacher or if the child is especially
motivated to learn to read. Just because an effect is not
always present does not make it any less causal.
Is this a case of special pleading? Hardly. We know,
for example, that exposure to certain viruses cause the
common cold. Yet, not every exposure to the virus results
in illness. So it is with cognitive deficits and academic
problems. It is possible that from the vantage point of
complete omniscience we would see that every step in
the causal sequence from cognitive deficits to academic
problems is fully deterministic. However, from the
position of incomplete understanding, probabilistic
models are our best and perhaps only option.
Disastrous equivocation: Kranzler et al.’s “false posi-
tives” in the prediction of academic deficits are not false
positives in DD/C’s diagnostic accuracy. Kranzler et al.
call children who have specific cognitive deficits but
no academic deficits “false positives,” provided that
they have at least average intelligence (p. 132). If, for the
sake of argument, we conceded that DD/C does in fact
imply that cognitive deficits are necessarily associated
with academic deficits, there would be nothing wrong
with the use of the term false positive in this context.
However, in the context of diagnosis, the term false
positive refers to when a diagnostic procedure incorrectly
identifies a person with a condition that is not actually
present. Later in the paper, the high false positive rate
in the prediction of cognitive deficits is used to imply
that DD/C has low positive predictive power in SLD
identification: “[O]ur PPV results suggest that clinicians
will spend a great deal of time conducting assessments
that have a very low probability of accurately identifying
true SLD” (p. 134).
142 D. P. FLANAGAN AND W. J. SCHNEIDER
Elsewhere, Kranzler et al. state that, “No PPV value for
any CHC broad cognitive ability exceeded 40% [with
a few minor exceptions]. These findings mean that a
diagnosis of SLD using the XBA PSW method has at best
a chance of about 50% that the individual truly has SLD
when the assessment outcome is positive” (p. 131).
Huh? No, this means that when a child plucked at
random from the WJ III standardization sample has at
least one cognitive deficit on one of the seven standard
subtests, there is at best a 50% chance that the person also
has an academic deficit on one of several WJ III academic
tests. This fact has very little to do with whether a person
really has SLD. The assessment outcome is not “positive”
for SLD in such a case. Here Kranzler et al. have (no
doubt unintentionally) committed the logical fallacy of
equivocation and are led to an incorrect conclusion.
The logical fallacy of equivocation occurs when an
argument hinges on an unacknowledged and improper
shift in the meaning of a word or phrase. For example,
“Limburger cheese smells funny. Funny things make
people laugh. Therefore, the smell of Limburger cheese
makes people laugh.”
Laid bare, the logic of Kranzler et al.’s equivocation
goes like this:
. DD/C says that people with specific cognitive
deficits should always have academic deficits. There
are a lot of false positives in DD/C’s prediction of
academic deficits from cognitive deficits. Thus, DD/
C is wrong.
. False positives occur when a procedure falsely
identifies people with a condition.
. Therefore, DD/C falsely identifies a lot of people
with SLD who do not in fact have SLD.
That is, they use the term false positive from one
context (the prediction of hypothesized academic
deficits), to apply to false positives in another context
(the accurate identification of SLD). The truth is that
in DD/C one cannot be identified as having SLD if no
academic weaknesses or deficits are present. According
to Flanagan and colleagues, “If weaknesses or deficits in
the child’s academic achievement profile are not
identified, then the issue of SLD may be moot because
such weaknesses are a necessary component of the
definition” (Flanagan et al., 2012). In fact, this is one
of the distinctive features of DD/C. In some circum-
stances, other procedures (e.g., Fiorello et al., 2012)
allow for people to be diagnosed with SLD even if the
academic ability is well above average. In DD/C,
although specific cognitive abilities may cause relative
underperformance, such cases are not termed disabil-
ities unless the abilities are actually below average,
differ significantly from overall ability, and the
difference between expected and predicted performance
in the scores that represent the abilities is unusual in
the general population. Kranzler et al. did not apply the
DD/C criteria correctly.2
It is illuminating to consider who might be among
Kranzler et al.’s “false positives.” Presumably, there are
many students with cognitive processing deficits who
never fall behind academically because dedicated parents,
skillful teachers, and thoughtful school psychologists
worked long and hard to prevent problems that would
otherwise have occurred. To think that the existence of
such students points to a failing of DD/C is simply not fair.
Another group of students that Kranzler et al.’s
analyses classify as “false positives” would be students
who do have clear cognitive deficits but their effect causes
only mild academic underperformance. For example,
a child with high overall ability but low working
memory ability may struggle a bit with multistep math
calculations but still manage to perform in the average
range. The fact that DD/C does not give such a child
the SLD label is a feature, not a bug. A child with mild
academic problems might need a little extra help, but
assigning the child SLD status and marshalling special
education resources to provide this help is like sending
an F-22 Raptor to do routine crop dusting. Implemented
properly, DD/C conserves precious resources for students
most in need of them.
Kranzler et al.’s “false negatives” are not false negatives
in SLD identification. The same equivocation on the
term false positive is also present with respect to the term
false negative, which cause Kranzler et al. to misapply their
negative predictive value results to the question of DD/C’s
accuracy. The fact that this reasoning error worked to DD/
C’s favor this time is irrelevant: These results have little to
no bearing on the question of DD/C’s accuracy.
DD/C allows for people who score low on academic
achievement tests for reasons other than cognitive
processing problems. Kranzler et al. appear to agree,
stating, “It is important to note that a significant academic
2. In DD/C, the expected score in both the cognitive and academic areas of
identified weakness is predicted from an overall ability score, called
the “Intact Ability Estimate (IA-e)” in the PSW-A program (note that the
program allows for a score such as the broad cognitive ability score from the
WJ III to be used in place of the IA-e). The criterion of a difference between
predicted and actual performance was not used in the Kranzler et al. study
(or any other studies that claim to have evaluated the diagnostic accuracy of
DD/C), but is nevertheless an important one in DD/C because it is used
to identify “domain-specific cognitive weaknesses” and “academic
underachievement.” Instead, Kranzler et al. (this issue) used scores less
than 90 to represent cognitive and academic weaknesses, many of which
would likely not have been considered actual weaknesses in the PSW
analysis based on DD/C criteria. Moreover, they incorrectly stated, “In the
XBA PSW method, the difference between the broad cognitive ability and
achievement scores is not tested statistically” (p. 129). It most certainly is.
 INTERNATIONAL JOURNAL OF SCHOOL & EDUCATIONAL PSYCHOLOGY
weakness is not ipso facto evidence of SLD, as deficits
in achievement may result from a number of non-
cognitive factors (e.g., low motivation or poor instruc-
tion)” (p. 130). Therefore, it is perplexing that all cases of
academic weaknesses without cognitive weaknesses are
misleadingly called “false negatives,” as if practitioners
using DD/C should assign such children the SLD label, but
fail to. DD/C explicitly excludes such cases, and for good
reason.
As acknowledged by Kranzler et al., DD/C procedures
include an extensive list of exclusionary criteria for
SLD. There are many reasons that children underperform
academically for reasons that have nothing to do with
SLD. For example, a student of average intelligence who
misses two years of school because of severe and chronic
medical problems would likely fall behind academically.
This child would be called a “false negative” by Kranzler
et al.’s analyses, suggesting that the student truly has
an SLD and the DD/C method missed that student. One
need not be an adherent of DD/C to see why this is a bad
idea. Indeed, we are quite confident that Kranzler et al.
would not identify such a child as having an SLD. Thus,
it is unfair to hold DD/C responsible for an “error” that is
actually the right call to make. We could generate many,
many more examples like this but do not wish to belabor
the obvious: These are not diagnostic errors.
Kranzler et al. versus Stuebing et al.:
An instructive contrast
Like Kranzler et al., Stuebing et al. (2012) also conclude
that DD/C is likely to identify children with SLD who
do not have it. However, their reasoning does not suffer
from the same flaws as that of Kranzler et al. and thus
constitutes a call to action. The study by Stuebing et al. is
not the final word on the matter, but it should be credited
for getting the conversation started about estimating
the accuracy of SLD identification procedures. Although
Stuebing et al. fail to implement DD/C properly in several
important ways (e.g., specific criteria are not evaluated,
the procedure is followed robotically, no follow-up
testing is conducted, and it only considers the accuracy
of DD/C under the conditions of low population base
rates instead of the more realistic base rates in referred
populations), they do frame their questions properly and
the overall logic of their approach is sound:
1. Create simulated latent and observed scores
according to patterns hypothesized to exist by
the theory underlying the PSW method.
2. Identify cases that truly meet the PSW criteria for
SLD.
3. Simulate the decision processes recommended by
the PSW method.
143
4. Evaluate the accuracy of those decisions by
comparing the results obtained from observed
scores to the true status defined by latent
scores.
This approach has at least the theoretical possibility
of estimating DD/C’s diagnostic accuracy. Future studies
along these lines may prove very informative as to how
DD/C should be amended to increase its diagnostic
accuracy. In contrast, the methods of Kranzler et al. could
be replicated a thousand times and we would be no
nearer to reforming SLD identification. Why?
Unlike the study by Stuebing et al. in which it was possible
to know which simulated cases truly met criteria for SLD, the
study by Kranzler et al. used observed scores from the WJ
III standardization sample. There is no way to know which
observed scores are accurate and which are not. This is where
the whole enterprise should have stopped. With real cases,
there is no error-free standard to which one can appeal to
decide if a diagnostic error has occurred. At best, with real
scores one can evaluate how closely two diagnostic methods
agree. If one procedure is strongly suspected to be highly
accurate, it can act as the “Gold Standard” method,
substituting for the “true diagnosis.”
For example, suppose that each child not meeting DD/C
exclusionary criteria is given 10 tests of each relevant
cognitive and academic ability and then the scores were
used to come to a diagnostic decision about SLD. If such a
procedure were feasible, it is likely that it would be quite
accurate. Suppose that we select two measures of each ability
at random for each child to simulate the accuracy of DD/C
as it is more typically practiced. If the results of the 2-test
version of DD/C are highly consistent with the results of the
10-test version of DD/C, this would suggest that the 2-test
version is a fairly accurate method. If not, the 2-test
procedure recommended by DD/C may need to be updated
to a higher number of tests. It is even possible to refine one’s
recommendations such that highly correlated broad abilities
(e.g., verbal comprehension and knowledge) need fewer
measures, whereas less correlated broad abilities (e.g., long-
term memory) require many more measurements before
scores converge on a trustworthy estimate.
If we could rewind time and rewrite Kranzler
et al.’s paper
There is an important point to be found in Kranzler et al.’s
results that, if properly framed, deserves attention in the
field. Too often, researchers and practitioners learn about
small-to-moderate correlational effects and fail to think
through what exactly they mean. It is absolutely true, for
example, that phonological processing deficits are associ-
ated with reading decoding problems (Wagner & Torgesen,
144 D. P. FLANAGAN AND W. J. SCHNEIDER
1987). However, the effects are not deterministic and are
not nearly as large as many people suppose.
For example, imagine that a 6-year-old child has a WJ
IV Comprehension-Knowledge (Gc) cluster of 100, a
Fluid Reasoning (Gf) cluster of 100, and an Auditory
Processing (Ga) cluster of 64. Most of us would consider
this to be the start of what is likely to be a very SLD-ish
cognitive profile. A score of 64 on the Ga cluster is very
low, and given how many studies have investigated Ga’s
relationship with reading decoding skills, we would not
be surprised to learn that the child scored an 80 on the
WJ IV Basic Reading Skills cluster (BRS). Indeed, if there
was any surprise, it would likely be that the BRS score
wasn’t lower than 80, perhaps closer to 64.
Using the correlation matrices from the WJ IV
Technical Manual and a little matrix algebra (available
upon request), it is possible to create a multiple
regression equation to estimate the typical BRS score
among children with known scores on Gc, Gf, and Ga:
Predicated BRS ¼ 19:91 þ 0:180Gc þ 0:346Gf
þ 0:275Ga
Amazingly enough, if we plug Gc ¼ 100, Gf ¼ 100,
and Ga ¼ 64 into the equation, the predicted BRS score
would be not 64, not 80, but 90. How can that possibly be?
A score of 90 is in the average range! More than half of
children with this cognitive profile (Gc ¼ 100, Gf ¼ 100,
Ga ¼ 64) are expected to score in the average range or
better on BRS? Yes. In fact, a child with this cognitive
profile who scores 80 on BRS is about 10 points below
expectations. Assuming that the scores are multivariate
normal, only about 1 in 5 children with Gc ¼ 100,
Gf ¼ 100, and Ga ¼ 64 would score 80 or lower on BRS.
Does this mean that Flanagan and colleagues are wrong
about specific cognitive deficits having negative effects on
academic performance? Of course not! In this thought
experiment, if the child’s Ga score were also 100, the
predicted BRS would be 100 as well, which is 10 points
higher than if the Ga score were 64. Under the assumption
of multivariate normality, only about 4% of children with
scores of 100 on all three cognitive scores would score 80 or
lower on BRS. Putting this information together, a child
with a severe cognitive deficit in Ga (standard score ¼ 64)
but otherwise average cognitive abilities in other areas (i.e.,
Gc ¼ 100, Gf ¼ 100) is nearly 5 times as likely to score 80
or lower on BRS than a child without any cognitive deficits
(i.e., Gc, Gf, and Ga ¼ 100). This is an illustration of the
general principle that even severe specific cognitive deficits
do not guarantee the presence of academic deficits, but they
do raise the risk that academic deficits will be observed,
sometimes by a lot.
The happy truth is that many, if not most, people with
specific cognitive weaknesses and deficits find various
ways around them. The children who do not, the ones
most likely to be referred for SLD evaluations, may very
well be the exceptions to this pattern. This, or something
like it, in our opinion, should have been the main point
of Kranzler et al.’s paper: Not that DD/C is an inaccurate
method of diagnosing SLD, but that we should not
overestimate the effects of cognitive deficits on academic
performance.
Simply because we see cognitive and academic deficits
together in the same child does not mean that our work
as psychologists is done. Cognitive abilities are indeed
extremely important causal determinants of academic
abilities. However, there is a host of other factors that
can in aggregate be much more important than cognitive
abilities in influencing academic outcomes, though their
effects may be small individually. Psychologists need to
give cognitive abilities their proper consideration, but
must also weave together all the evidential threads into
a coherent narrative of the child’s academic difficulties.
Only then can psychologists be in the position to give
truly helpful advice to parents and teachers trying to help
children who have fallen behind.
About the authors
Dr. Dawn P. Flanagan is Professor of Psychology at St. John's
University in Queens, NY. She is also Clinical Professor at Yale Child
Study Center, Yale University School of Medicine in New Haven, CT.
She is a widely published author in the areas of cognitive assessment,
specific learning disabilities, and professional issues in school
psychology. She also serves as an expert witness, learning disabilities
consultant, and test/measurement consultant and trainer for
organizations both nationally and internationally.
W. Joel Schneider completed a doctorate in clinical psychology at
Texas A&M University and is a Professor of Psychology at Illinois State
University, where he has the good fortune to be able to follow his
scholarly interests wherever they lead. He develops new psychometric
methods and develops clinician-friendly computer programs to help
clinicians make better inferences about individuals. He writes essays,
makes videos, and sometimes sings about psychology, psychometrics,
and software at his blog, AssessingPsyche.
References
Fiorello, C. A., Hale, J. B., & Wycoff, K. L. (2012). Cognitive
hypothesis testing: Linking test results to the real world.
In D. P. Flanagan & P. L. Harrison (Eds.), Contemporary
intellectual assessment: Theories, tests, and issues
(pp. 484 –496). New York, NY: Guilford Press.
Flanagan, D. P., Alfonso, V. C., Mascolo, J. T., & Sotelo-
Synega, M. (2012). Use of ability tests in the identification
of specific learning disabilities within the context of an
operational definition. In D. P. Flanagan & P. L. Harrison
(Eds.), Contemporary intellectual assessment: Theories, tests,
and issues (3rd ed., pp. 643 – 669). New York, NY: Guilford
Press.
 INTERNATIONAL JOURNAL OF SCHOOL & EDUCATIONAL PSYCHOLOGY
Flanagan, D. P., & McGrew, K. S. (1997). A cross-battery
approach lo assessing and interpreting cognitive abilities:
Narrowing the gap between practice and cognitive science.
In D. P. Flanagan, J. L. Genshaft, & P. L. Harrison (Eds.),
Contemporary intellectual assessment: Theories, tests, and
issues (pp. 314– 325). New York, NY: Guilford Press.
Flanagan, D. P., & Ortiz, S. O. (2001). Essentials of cross-battery
assessment. New York, NY: Wiley.
Flanagan, D. P., Ortiz, S. O., & Alfonso, V. C. (2007). Essentials
of cross-battery assessment (2nd ed.). New York, NY: Wiley.
Flanagan, D. P., Ortiz, S. O., & Alfonso, V. C. (2013). Essentials
of cross-battery assessment (3rd ed.). Hoboken, NJ: Wiley.
Flanagan, D. P., Ortiz, S. O., Alfonso, V. C., & Mascolo,
J. (2002). The Achievement Test Desk Reference (ATDR):
Comprehensive assessment and learning disabilities. Boston,
MA: Allyn & Bacon.
Flanagan, D. P., Ortiz, S. O., Alfonso, V. C., & Mascolo, J. T.
(2006). Achievement test desk reference: A guide to
learning disability identification (2nd ed.). New York, NY:
Wiley.
Kranzler, J. H., Floyd, R. G., Benson, N., Zaboski, B., &
Thibodaux, L. (this issue). Classification agreement analysis
of Cross-Battery Assessment in the identification of specific
learning disorders in children and youth. International
Journal of School & Educational Psychology, 4(3), 124 – 136.
McGrew, K. S. (2005). The Cattell-Horn-Carroll theory of
cognitive abilities: Past, present, and future. In D. P. Flanagan
& P. L. Harrison (Eds.), Contemporary intellectual assess-
View publication stats
145
ment: Theories, tests, and issues (2nd ed., pp. 136 – 182). New
York, NY: Guilford Press.
McGrew, K. S. (2009). Editorial: CHC theory and the human
cognitive abilities project: Standing on the shoulders of the giants
of psychometric intelligence research. Intelligence, 37, 1–10.
Ortiz, S. O., Flanagan, D. P., & Alfonso, V. C. (2015). Cross-
battery assessment software system X-BASS (Version 1.2)
[software]. Hoboken, NJ: Wiley. Retrieved from www.wiley.
com/go/XBASS
Pearson, K. (1930). The life letters and labours of Francis
Galton: Vol. 3. Researches of middle life. Cambridge, England:
Cambridge University Press.
Schneider, W. J. (2013). What if we took our models seriously?
Estimating latent scores for individuals. Journal of Psychoe-
ducational Assessment, 31, 186– 201.
Schneider, J. W., & McGrew, K. S. (2012). The Cattell-Horn-
Carroll model of intelligence. In D. P. Flanagan &
P. L. Harrison (Eds.), Contemporary intellectual assessment:
Theories, tests, and issues (3rd ed., pp. 99 –144). New York,
NY: Guilford Press.
Stuebing, K. K., Fletcher, J. M., Branum-Martin, L., & Francis,
D. J. (2012). Evaluation of the technical adequacy of three
methods for identifying specific learning disabilities based
on cognitive discrepancies. School Psychology Review, 41,
3 – 22.
Wagner, R. K., & Torgesen, J. K. (1987). The nature of
phonological processing and its causal role in the
acquisition of reading skills. Psychological Bulletin,
101(2), 192 – 212.