PRACTICE VARIATION

IN THE NICU
WHAT DOES IT MEAN?
WHY DOES IT MATTER?

Joseph Schulman MD, MS

Director, NICU Quality Measurement and Improvement
California Children’s Services
California Department of Health Care Services
Joseph.Schulman@dhcs.ca.gov
 HOW DO YOU EVALUATE YOUR
NICU’S PERFORMANCE?
Whatever you track, it’s not enough to confirm your
value lies in the “desirable” part of the distribution…
“Which rate is right?”
• To make sense of where our NICU’s outcome measure lies
on the overall distribution, we need to understand risk
adjustment and other statistical modeling methods
o Beyond our scope today; important complementary knowledge, we
should examine this another time (further reading list on last slide)
• Our scope for today: To help you reflect on where your
NICU’s process measure lies on the overall distribution
 WHICH RATE IS RIGHT?

What explains observed practice variation?

• Does it reflect clearly articulated aims a NICU is trying
to achieve?
• Is it evidence-based?
• Does it reflect incentives other than the patients’ best
interests?
• Is it undisciplined?
• Are we confident we know what we need to know to
reflect on a particular practice?
 OUR AIMS = WHY WE DO
WHAT WE DO

Does your NICU have explicitly

articulated aims (that you can
recite now if asked)?

Yes No
 TWO MINDSETS OF
WORKING IN THE NICU

Do, Do, Do Ready, Aim, Fire

Exactly Why Do You Do What You Do?
NICU AIMS
What are Your…

For Infants Whose Care Needs Can Not Be Met in a

Newborn Nursery or at Home…
Promote as much as possible the physiological function
and developmental potential of a normal newborn –
using only the resources and exposures demonstrated
to be beneficial by current professional knowledge
• Even when this is not possible (hopeless prognosis), always to
minimize suffering for infant and family
 NICU AIMS

Manage a Family’s Experience of Their Infant’s Special

Care Requirements
• Promote realistic expectations
• Provide timely explanations of care and outcomes
that make sense and allay needless worry
• Appreciate how the family determines the care team
are doing a good job, and how NICU staff criteria for
doing a good job may differ the family’s criteria
 NICU AIMS
Implement work processes that speed the flow of
component tasks and minimize activity without value (no
Muda)

• Leverage Human
o Resources: teamwork
o Cognitive ability with IT
§ Data mastery; prevent error; learn from aggregate experience

• Promote Continual Learning

• Appreciate how our work and results compare with others
o Understand how to emulate or better those achieving the highest
likelihood of desired outcomes
§ Gain ever-greater mastery of our work’s fine structure
 NICU AIMS

Support Family Physicians, Obstetricians, Pediatricians

• Provide a seamless and efficient transition between the
care patients receive before they reach us, and after they
leave us
o Provide timely feedback and summary information that
informs care and performance evaluation and improvement

Establish and Meet Educational Goals for Professional

Trainees in our NICU
 NICU AIMS

Ensure Staff Feel They Do Good

• For those we serve: promote a sense of trust, involvement,
meaning, caring, compassion, and when possible, joy

• For ourselves: promote trust, recognition, personal

growth, and professional advancement
Alice came to a fork
in the road.
“Which road do I
take?” she asked.
“Where do you
want to go?”
responded the
Cheshire cat.
“I don’t know,”
Alice answered.
“Then,” said the cat,
“it doesn’t matter.”
 LET’S PAUSE FOR A TIME-OUT

Now that we’ve thought about aims,

What’s the ultimate aim of improving quality?

NOW, WE’RE READY TO LOOK AT A
COUPLE OF EXAMPLES OF
PRACTICE VARIATION
 JAMA Pediatr. 2018;172(1):17-23
doi:10.1001/jamapediatrics.2017.3913
 WE MEASURED GA-STRATIFIED INBORN NICU
ADMISSION RATES AND EXAMINED ASSOCIATIONS

With
• High illness acuity
• Designated NICU level of care
• Overall inborn NICU admission rate

We hypothesized inborn admission rates

would correlate positively with % high
illness acuity admissions
 VARIABLE DEFINITIONS

• Inborn admission rate: % all live births at a hospital who were

admitted to the NICU

• CPQCC high illness acuity admission rate: % of all inborn

admissions for a particular GA stratum that meet CPQCC criteria
for high acuity of illness
o All infants BW < 1500 g = high illness acuity; for >1500 g / > 34 weeks:
1. Death
2. Intubated- or nonintubated-assisted ventilation for 4 hours or more
3. Early bacterial sepsis
4. Major surgery requiring anesthesia,
5. Acute transport out of the NICU
6. Suspected encephalopathy or suspected perinatal asphyxia
7. Active therapeutic hypothermia
 WHAT DID WE FIND?
LOTS OF VARIATION: SIMILAR ACROSS ALL GA

• Inborn admission rates for

neonates born at GA ≥ 34
weeks varied 34-fold : 1.1%
- 37.7%
• Distribution of values was
similar to overall inborn
admission rates of all GAs

• Among the 8 extreme high

outliers for 34 weeks or
more, only 4 represented
regional NICUs
 WHAT DID WE FIND?
LOTS OF VARIATION: ACROSS ALL LEVEL NICUS

• % of admissions ≥ 34 weeks
that met high illness acuity
criteria varied 40- fold:
2.4% - 95.0%

• No significant differences
across levels of care
 WHAT DID WE FIND?
ADMIT RATE AND C ASE MIX: NOT WHAT WE
EXPECTED

Inverse correlation between inborn admission rate

and % high-acuity admissions (Spearman ρ =
−0.3034, P < .001)

• Largely driven by the community-level NICUs and

intermediate-level NICUs
Inborn admission rates for specific
GA strata correlated strongly
with overall inborn admission
rates for all other GA strata
(including GA<34 weeks but excluding
those in the stratum of interest):
• 34 to 36 weeks, ρ = 0.61, P < .001
• 37 to 38 weeks, ρ = 0.78, P < .001
• ≥ 39 weeks, ρ = 0.68, P < .001
 WHO WOULD HAVE THOUGHT…

1. Neonates ≥ 34 weeks account for 79.2% of all inborn NICU admissions

o Only 11.9% of these have high illness acuity
o Therefore, proportion not with high illness acuity:
§ 88.0% of inborn NICU admissions at > 34 weeks’ gestation
§ 70.0% of all inborn NICU admissions
2. 34-fold (!) variation in inborn NICU admission rates
3. Admission rates seem driven by local practice rather than illness
acuity
o 40-fold (!) variation in proportion of high acuity admissions
o In hospitals with relatively higher admission rates, % of these
admissions with high illness acuity tends to be relatively low
o In hospitals with relatively lower admission rates, % of these admissions
with high illness acuity tends to be relatively high
o A NICU’s inborn admission rate for a GA substratum predicts its inborn
admission rate for all other GA strata
 HOW C AN WE EXPLAIN THE FINDINGS?

ØHigh illness acuity criteria do not identify all newborns with high
medical needs, including
• Some congenital cardiac anomalies
• Dysmorphic conditions
• Seizures
• Some neonatal abstinence syndrome cases

• NICUs often serve functions other than provision of critical or

intensive care
• 4% - 8% of newborns experience difficulties of perinatal transition that may last only
hours but require evaluation and observation not available in the well-baby area of a
hospital
 HOW C AN WE EXPLAIN THE FINDINGS?

Supply-sensitive care:
• Reflects available service capacity and payment systems
that incentivize service provision
• Availability of NICU beds becomes a determinant of
NICU care

An additional empty NICU bed the day before an infant’s birth

increases the probability that the infant will be admitted to the
NICU, particularly for infants of higher birthweight

Freedman S. Capacity and utilization in health care: the effect of empty beds on neonatal intensive care admission.
Am Econ J Econ Policy. 2016;8(2):154-185.
PEDIATRICS Volume 142, number 3, September 2018:e20180115
 AUR (%) AUR (%)
0 10 20 30 40 50 60 70 80 90 100 0 10 20 30 40 50 60 70 80 90 100

All
All

No A.S.
No A.S.

2015
2013

A.S.
A.S.

AUR (%)
AUR (%)

0 10 20 30 40 50 60 70 80 90 100 0 10 20 30 40 50 60 70 80 90 100
All
All

No A.S.
No A.S.

2016
2014

A.S.
A.S.
 AUR DECLINED, VARIATION NARROWED

• By 2016 overall AUR declined by 21.9%

• Antibiotic days declined by 42,960

• 2016 AUR among NICUs in known externally organized AS

efforts declined to 19.9% (28.7% decrease)
o 22.8% (16.2% decrease) among NICUs not participating (19.9% versus
22.8% = 12.7% relative difference between NICU groups)
 2013 2014

6
5

5
4

4
kdensity (%)

kdensity (%)
3

3
2

2
1

1
0

0
0 10 20 30 40 50 60 70 80 90 100 0 10 20 30 40 50 60 70 80 90 100
AUR (%) AUR (%)

Regional NICUs Community NICUs Regional NICUs Community NICUs

Intermediate NICUs Non-CCS NICUs Intermediate NICUs Non-CCS NICUs

2015 2016
6

6
5

5
4

4
kdensity (%)
kdensity (%)
3

3
2

2
1

1
0

0 10 20 30 40 50 60 70 80 90 100 0 10 20 30 40 50 60 70 80 90 100
AUR (%) AUR (%)

Regional NICUs Community NICUs Regional NICUs Community NICUs

Intermediate NICUs Non-CCS NICUs Intermediate NICUs Non-CCS NICUs

 2016 AUR CORRELATIONS

• AUR correlated neither with proven infection nor NEC

o As in 2013

• Only among regional NICUs did AUR correlate with

surgical case volume (rho = 0.53, P = 0.01), NICU mortality
rate (rho = 0.57, P = 0.004), and AvLOS (rho = 0.62, P =
0.002)

• AUR correlated with inborn admission rate across all

NICUs (rho = 0.23, P = 0.009) and community NICUs (rho
= 0.3, P = 0.006)
 REGIONAL NICUS AND AUR CORRELATES

Positive correlations between AUR and surgical case volume,

mortality rate, and AvLOS:
• Might AUR represent a proxy measure for surgical case volume and illness
severity?
o Surgical volume and NICU mortality rate strongly correlated (rho = 0.74, P < 0.001)

• Regional AUR correlations do not scale linearly

o Largely driven by only 3 of 23 NICUs – those with the highest AUR values, between
30% and 57%

ØWhile surgical case volume and illness severity might sometimes

drive AUR, relatively high AUR also may independently predict
mortality and AvLOS
 Regional NICUs, all AUR values (rho = 0.75, P < 0.001) Regional NICUs with AUR < 30% (rho = 0.68, P = 0.001)
600

600
500

500
Number of Surgical Cases

Number of Surgical Cases

400

400
300

300
200

200
100

100
0

0
0 1 2 3 4 5 6 0 1 2 3 4 5 6
NICU Mortality Rate (%) NICU Mortality Rate (%)

Regional NICUs (rho = 0.53 , P = 0.01) Regional NICUs (rho = 0.57, P = 0.004)
60

60
50

50
40

40
AUR (%)

AUR (%)
30

30
20

20
10

10
0

0 100 200 300 400 500 600 0 1 2 3 4 5 6

Number of Surgical Cases NICU Mortality Rate (%)
 I N T E R M E D I AT E N I C U AU R R A I S E S PA RT I C U L A R C O N C E R N

2013 2014

• 2016 Quartile 1 cutpoint =

6
17.5% for intermediate,

5
14.4% for all NICUs

4
kdensity (%)

kdensity (%)
3

3
2

2
• If decision rules changed over

1
time so that fewer neonates were

0
admitted to “rule out sepsis,” then 0 10 20 30 40 50
AUR (%)
60 70 80 90 100 0 10 20 30 40 50
AUR (%)
60 70 80 90 100

patient days could decrease Regional NICUs Community NICUs Regional NICUs Community NICUs

disproportionately to antibiotic Intermediate NICUs Non-CCS NICUs Intermediate NICUs Non-CCS NICUs

days 2015 2016

6

6
o Ironically, AUR could rise
5

5
4

4
kdensity (%)
kdensity (%)
3

3
• Findings don’t support this
2

2
explanation
1

1
0

0
0 10 20 30 40 50 60 70 80 90 100 0 10 20 30 40 50 60 70 80 90 100
AUR (%) AUR (%)

• Therefore, antibiotic overuse is Regional NICUs Community NICUs Regional NICUs Community NICUs

clearest at intermediate NICUs Intermediate NICUs Non-CCS NICUs Intermediate NICUs Non-CCS NICUs

with both relatively high AUR

and inborn admission rate
 WHAT DO THESE FINDINGS MEAN?

• Continued opportunity to decrease antibiotic use, and need

for more complete case-mix characterization
• For example, blood culture-negative meningitis, pneumonia,
osteomyelitis

• NICUs with relatively high AURs generally serve populations

with similar measured clinical correlates as NICUs at the
same level of care with lower AURs
• Excepting perhaps, highest surgical volume NICUs
 The US is
Not the
Only Nation
With
Practice
Variation
(It Merely
Seems to
Require
Looking)
“In summary, there is unwarranted variation in all the areas examined. .. there seems to be
general variation across the whole range, except for admissions of infants born before 34
weeks, where there is little variation.
Both the extent of variation and the potential overuse of health services give grounds to
question whether sick neonates have equitable provision of health services and whether this
variation affects quality and patient safety.”

https://helseatlas.no/sites/default/files/norwegian-neonatal-healthcare.pdf
PRACTICE VARIATION IN
THE NICU
NOW THAT YOU’VE SEEN SOME
EXAMPLES,

WHY DOES IT MATTER?

“When different physicians are
recommending different things for
essentially the same patients,
it is impossible to claim that
they are all doing the right thing.”

Eddy DM. Evidence-based medicine: a unified approach.

Health Affairs 2005;24: 9–17.
WHAT’S THE ULTIMATE AIM OF QI?

Eliminate site of care

as an outcome determinant

Just as travelers using a major airline need not

worry that carrier choice influences safe arrival at
their destination, in the future, where a patient
receives care should not influence outcome.
• Does your NICU aim to use a rigorous
evidentiary base to guide practice?
Do you consider a potentially better practice a hypothesis to be
tested?
• Does your NICU aim to use resources
efficiently?
Exactly how have you designed your processes to do so? Can you
confirm you achieve your aim?
 COMMENTS, QUESTIONS?

Further reading:

Schulman, J: Evaluating the Processes of Neonatal Intensive Care. 2004, BMJ

Books, London. (distributed in US by John Wiley & Sons)

Schulman, J., Spiegelhalter, D. J., Parry, G.: How to Interpret Your Dot: Decoding the
Message of Clinical Performance Indicators. (State of the Art Article) Journal of
Perinatology; 2008; 28:588-596.

Schulman, J., Saiman, L., Metrics for NICU Antibiotic Use: Which Rate is Right?
(editorial) Journal of Perinatology; 2011; 31:511-513.