Applying Principal Strata Concepts to

Discontinuity Designs

Bernie Black
Advanced Causal Inference Workshop
August 2013
1

Consider Lee and McCrary (2009) RD crime study

But no change in arrest rate

Unobservables in McCrary crime study?

Charging decision is discretionary:
Are juveniles charged as adults similar to juveniles charged
as juveniles?
Are they similar to adults charged as adults?

Helps to condition on RD running variable r (here, age),

x (this charge, number and nature of prior charges and
convictions, prior incarceration)
Unobservables are likely still important

So for whom do they measure an effect?

4

Principal Strata: Different criminal types

Imagine 3 types:
Kinda nice = charged as juvenile, whether juvenile or barely adult
Not so nice = charged as adult if barely adult; as juvenile if juvenile
Really nasty = charged as adult, whether juvenile or adult

Assume: Criminals know their own type on average

Who is affected by turning 18?
Only the not so nice group

Compare to IV with 2-sided noncompliance

Treatment = charged as adult
Mapping:

kinda nice never takers

not so nice compliers
really nasty always takers
assume no defiers
Assignment to treatment (age > 18) = instrument for treatment
(charged as adult)

Wald estimator for the compliers:

ITT
E[yi|zi=1] E[yi|zi=0]
E[yi|wi=1] E[yi|wi=0]=

E[wi|zi=1] E[wi|zi=0]
w
6

Noncompliance: Randomized Expt vs RD

Some differences in the meaning of principal
strata between RExpt and RD:
RExpt: Design choice affects who is complier vs.
never taker
Stronger encouragement more compliers

Example: Diet study:

Advice
Advice + recipes
Advice + recipes + weekly meetings

Different designs different complier groups

different LATEs
7

More complex for RD

Researcher often cant vary level of
encouragement
But can vary bandwidth around threshold r0

Who is a complier = f(r0, )

So, vary bandwidth vary complier group
No longer a clean basis for choosing bandwidth
Even in simple case where y is linear in r near r0

Iliev (2010) study of Sarbanes-Oxley Act

Start with an already excellent paper
Show how use of principal strata could make it better
And change some of the results
Grateful to Peter Iliev for sharing his data and code
And to my coauthor, Vladimir Atanasov, for our joint work
on this example
Our dataset (based on Ilievs) and code are posted

Overview
Driver of higher auditing cost: SOX 404
Requires auditors to attest to adequacy of internal
controls
great for auditors
worth the cost for firms?
Audit costs jump in 2004 when 404 kicks in
Research question: How much?
Without more, this is interrupted time series design
Assumes no other shock in 2004

10

Discontinuity for small firms

Small firms lobbied SEC to relax SOX for them
SEC mostly refuses, but:
If free float < $75M in each of 2002, 2003, 2004: compliance
deferred to 2007 (later, to 2010, then indefinitely)
Free float = market value of shares not held by insiders

A discontinuity!
Firms just above and just below threshold are likely similar
Firms above $75M are treated with SOX 404
Firms below $75M are exempt, for a while

First look at what Iliev did

Much to like . . .
Then at what else he might have done
11

Limit sample to firms near $75M

Bandwidth: free float [$50M, $100M] in 2004
Assesses robustness to broader [$40M, $110M] and
narrower [$60M, $90M] bandwidths
Ill come back to bandwidth choice
And to when one selects the sample

Initially, treat as sharp RD, as if:

All firms above threshold comply
No firms below threshold voluntarily comply
If not truly sharp, this is intent to treat design

12

Can firms manipulate their free float?

Some can, some cant
Rule adopted 2003, uses free float in 2002
If over $75M in 2002, youre stuck
Short of merger or delisting

If under $75M in 2002, can try to stay below in 2003, 2004

Grow business more slowly
Limit free float
Repurchase publicly held shares
Insiders buy shares in market
Recruit more insiders

13

Do firms (still in the sample) avoid threshold?

Yes! (Iliev Fig. 1)
Density of firms is smooth around threshold in 2002, 2003
In 2004, 2005: fewer firms above but near threshold

14

Do some firms exit the sample altogether?

Kamar, Karaca-Mandic and Talley (2009): smaller
firms are more likely to go private post-SOX
DiD analysis (US targets vs. foreign targets, after vs. before SOX)
Not a great control group

Iliev: Mild evidence: 14/188 (7.4%) of firms required

to file as of 2003 exit sample by 2004
vs. 8/149 (5.4%) of firms not required as of 2003 (not signif.)
If difference is real, he cant address it: no data on exiters.

15

How about covariate balance?

Treated and controls similar in 2003
Similar on most covariates in 2004
Differences are interesting. Treated firms have:
lower discretionary accruals
more likely to have slightly negative earnings

These are things that SOX 404 plausibly affects:

Less earnings manipulation.
Why?: The accountants are watching . . .

16

Placebo thresholds
Iliev reports: no effect with other thresholds:
$125M
$150M

I might wish that he had also tried:

$50M
$100M
Sample chosen based on 2002 float, not 2004

But Iliev is asking these questions!

Vary bandwidth, check for threshold manipulation, assess sample exit,
assess covariate balance, test placebo thresholds.
Many papers dont . . .
This is a state-of-the-art (or better) paper in finance or accounting.
17

Implications of SOX avoidance

Who are the SOX-avoiders?
Presumably firms with higher compliance cost

Iliev argues: estimate, that ignores avoidance,

is likely biased downward versus ATE
Partly testable, well see how to use principal
strata concepts to test it.
Likely bias is small

Downward bias is tolerable here, but bias

could go the other way in another study.
18

Nave estimate of SOX audit fees

Visually apparent jump in mean fees for 404-reporters in 2004

19

How else might you plot this?

Show individual points and line with jump
not just averages

How about three figures: lines plus data points:

audit fee in 2002 vs [free float in 2002]
should be smooth at $75M

audit fee in 2003 vs [max free float in 2002 or 2003]

should be smooth at $75M

audit fee in 2004 vs [max free float in 2002 or 2003 or 2004]

should jump at $75M

20

Float vs. audit fees in 2002

Predicted line:
ln(fees) = 11.52 + 0.013*float (t = 1.75) - 0.24*large (t = 1.12).

21

Max float over (2002-2003) vs 2003 fees

Predicted line:
ln(fees) = 12.40 + 0.001*maxfloat (t = 0.15) + 0.07*large (t = 0.40)
Looks good so far . . .
22

Max float over (2002-2004) vs 2004 fees

Predicted line:
ln(fees) = 11.47 + 0.018*maxfloat (t = 2.78) + 0.27*large
(t = 1.47; = 0.184)
A jump, but not significant . . .
23

Max float (2002-2004) vs 2004 fees: means

Difference in means is significant:

ln(fees) = 12.62 + 0.705*large (t =7.49)
Now highly significant . . .
24

Ilievs RD analysis (x-sectional data, in 2004)

Note: Careful control
for firm size (including cubic
in public float)
Coeff. w. controls: 0.744
e0.744 = 2.10
What else might Iliev
have done?

Dependent Variable
MR in fiscal 2004
Log sales 2004
Log assets 2004
Log market equity 2003
Leverage 2004
Receivables /total assets 2004

Big auditor 2004

Business segments 2004
Geographic segments 2004
Cubic in float, industry FE

ln(2004 audit fees)

0.866*** 0.744***
[7.57]
[7.39]
0.031
[1.09]
0.235***
[3.35]
0.050
[0.51]
0.612***
[2.62]
0.086
[0.35]
0.370***
[3.94]
0.040
[1.45]
0.070***
[2.91]
Yes
Yes
25

Export tables and figures to Word, Excel

26

Better research design: Combined DiD/RD

Iliev runs cross-sectional regression in 2004
Throws away valuable data from prior & later years
Can instead combine DiD and RD

RD ensures similar treated and controls

addresses the core challenge for DiD

DiD provides before and after

less worry about remaining differences between treated
and controls
can use DiD technology to:
assess parallel trends in pre-period
see if effect grows or shrinks in post-period
cumulate data during post-period (better signal/noise)
27

How about the threshold manipulation?

Ilievs strategy: [2002 float > $75M] = instrument for
(2002 or 2003 or 2004 float > $75M)
Cant manipulate 2002 float (rule wasnt written yet)

Does this work?

Exogenous = yes
Must believe only through: conditioned on controls:
[> $75M float in 2002] audit fees in 2004 only through [SOX
404 compliance]
Need flexible control for firm size [which he has]

Why use IV: grower-avoiders [firms that would have grown to >
$75M float, but for SOX, might have higher SOX compliance costs
than grower-accepters [firms that grow from small to large]

underestimate treatment effect

28

Iliev 2SLS results

Coefficients higher than OLS.
As expected.
But:
What else might he have done?
What LATE is he estimating?

Dependent Variable
instrumented MR in fiscal 2004

Log sales 2004

Log assets 2004
Log market equity 2003
Leverage 2004
Receivables /total assets 2004
Big auditor 2004
Business segments 2004
Geographic segments 2004
Cubic in public float
Industry FE

ln(Audit fees in 2004)

1.171***
0.983***
[4.95]
[3.65]
= 0.237
= 0.269
0.034
[1.09]
0.218***
[2.79]
-0.052
[-0.38]
0.647***
[2.75]
0.129
[0.55]
0.373***
[3.86]
0.047*
[1.71]
0.069***
[2.77]
No
No
Yes
Yes

29

Does IV help?
Will argue: Not much.
Can do better using principal strata
Including using (stronger) DiD/RD design
First differences: audit fees2004 audit fees2003
Can also use the strata to understand Ilievs estimates
LATE using his IV overestimates ATE

30

First develop plausible strata

Let L = large (float > $75M); S = small (float < $75M)
Limit sample to firms near $75M threshold
Divide firms based on growth without SOX (z = 0) into:
growers (L in 2004, whether L or S in 2002)
stable (L in 2004 if L in 2002; S in 2004 if S in 2002)
shrinkers (if L in 2002; will be S by 2004; if S in 2002, could drop out of
sample by 2004)

Also divide firms based on response to SOX

SOX-avoiders = who will change behavior to avoid SOX 404, if they can.
Growers who will not grow to avoid SOX

SOX-accepters = firms who will not change behavior to avoid SOX

31

Use these strata to compare similar firms

What sample decomposition is possible?
Lets try it and see

32

Potential strata and observed outcomes

VL = > $100M; L = $75-100M; S = $50-75M; VS = < $50M
Which outcomes are observed in 2004?

Empty column,
dropped below

Outcomes y(z,w)
Principal stratum

y(z=0,w=0)

y(z=0,w=1)

y(z=1,w=0)

y(z=1,w=1)

grower-accepter

VS2002S2004

S2002L2004

L2002(L or VL)2004

grower-avoider

(VS2002S2004;
S2002 S2004)

L2002(L or VL)2004

stable-accepter

S2002S2004

L2002L2004

stable-avoider

S2002S2004

L2002L2004

shrinker-accepter

S2002(S or VS)2004

L2002S2004

shrinker-avoider

S2002(S or VS)2004

L2002S2004
33

Strata imputation 1
Use only through exclusion restriction for shrinkers.
Identify members of these strata together (not separately): they shrank
Estimate SOX 404 cost for these two strata (DiD, control for firm size).
Sample: Based on free float in 2002 (if use 2004, lose SVS shrinkers)
Outcomes y(z,w)
Principal strata

y(z=0,w=0)

y(z=0,w=1)

y(z=1,w=1)

grower-accepter

VS2002S2004

S2002L2004

L2002(L or VL)2004

grower-avoider

(VS2002S2004;
S2002 S2004)

L2002(L or VL)2004

stable-accepter

S2002S2004

L2002L2004

stable-avoider

S2002S2004

L2002L2004

shrinker-accepter

S2002(S or VS)2004

L2002S2004

shrinker-avoider

S2002(S or VS)2004

L2002S2004

34

Strata imputation 2
Identify other four strata together. Again use exclusion restriction.
Estimate SOX 404 cost for these strata (DiD, control for firm size).
Sample: Based on free float in 2002 plus VS2002S2004)
Imperfect: (z=0,w = 0) sample has more grower-avoiders than (z=1,w=1)
But might be okay with DiD design
Outcomes y(z,w)
Principal strata

y(z=0,w=0)

y(z=0,w=1)

y(z=1,w=1)

grower-accepter

VS2002 S2004

S2002L2004

L2002(L or VL)2004

grower-avoider

(VS2002S2004;
S2002 S2004)

L2002(L or VL)2004

stable-accepter

S2002 S2004

L2002L2004

stable-avoider

S2002 S2004

L2002L2004

shrinker-accepter

S2002(S or VS)2004

L2002S2004

shrinker-avoider

S2002(S or VS)2004

L2002S2004

35

Compute ATT
Two estimates for particular strata (or mixes)
ATT= Weighted sum across groups
Note we never used IV
Did use only through restriction, for each group

36

Imperfect control group

No clean way to separate growers-avoiders from stable firms
no ideal control group
But can hope:
FE addresses most unobserved differences (if not subject to SOX 404)
Remaining unobserved differences dont (much) affect audit-fees-ifcontrol (if not subject to SOX 404).
(observed) after-minus-before (audit fees) for controls is good proxy
for (unobserved) after-minus-before (audit fees) for treated, if not
treated
DiD design much stronger than comparison in levels

37

DiD specification w. two treated groups

Different estimates different for strata bundles we observe?
Two treated groups: a and b:
Usual DiD with firm FE and one treatment group:

yit fi gt ( DiD * wit ) it

With two treated groups, one control group, this becomes:

yit fi gt ( DiD,a * wit ,a ) ( DiD,b * wit ,b ) it

Are the two treatment effects different?

Let wit = 1 for both groups, wit,b = 1 only for group b. Then:

yit fi gt ( DiD,a * wit ) ( DiD,b-a * wit ,b ) it

If wit,b is small, have more confidence that weakness in one
comparison isnt too troublesome.

38

What local effect is Iliev estimating with IV?

Some possibilities [lets take a vote]: LATE for
which strata?
grower-accepters
grower-avoiders
stable-accepters
stable-avoiders
shrinker-accepters
shrinker-avoiders
Something else?
39

Lets examine the Wald estimate

ITT
E[yi|zi=1] E[yi|zi=0]
LATE =

E[wi|zi=1] E[wi|zi=0]
w
E[wi|zi=1] = 1.00 (all L2002 firms comply; they are forced takers)

E[wi|zi=0] = (grower-accepters/all S2002 firms) = fga

E[yi|zi=1] = audit fees for forced takers =ASOXft
E[yi|zi=0] = (fga*ASOXga) + (1-fga)*(Anon-gano-SOX ))

LATE =

SOX
no SOX
ASOX

[
f
*
A

(1f
)
*
A
ft
ga
ga
ga
non ga ]

(1 f ga )
40

Unpack Wald Estimate

Let (A)SOXga = ASOXga Ano-SOXga. Same for other strata:
LATE =

SOX
SOX
SOX
no SOX
{( f ga * Aga
) (1 f ga )* Anon
(1- f ga )* Anon
ga } [ f ga * Aga
ga ]

(1 f ga )

SOX
no SOX
Anon
ga Anon ga

Meaning:
This setting: no defiers or never takers.
IV estimate removes always-takers (grower-accepters)
LATE for instrument-compliers: compliance changed by
instrument
Theyll comply with SOX-404 only if float > $75M in 2002

But once drop grower-accepters (lower comply cost?)

Estimate now biased up instead of down
41

Is This the Estimate we Want?

Probably not. At least not by itself.

Likely want ATE; not this particular LATEnon-ga

If yes, can get without IV (see above)

Blowup problem with all IV estimates
Imprecise estimates
Heavy stress on exclusion restriction

42

DiD: Intent to Treat Analysis

Sample: float [$50M, 100M] in both 2002 and 2004

ITT = 0.27 (t = 3.38; = 0.080)

43

DiD Excluding Future Growers (Always Takers)

5 other strata = 0.681 (t = 6.36; = 0.107)

Also exclude future shrinkers:4 other strata = 0.634 (t = 5.73; = 0.111)
Compare Wald = ITT/ w = 0.270/(1 - .709) = 0.925
44

Take Future Growers off the Chart

Can now see difference between treated, controls

45

Broader message before you use IV

A good instrument isnt enough
Need to understand exactly what LATE a particular
IV is estimating
Analyze in terms of principal strata
Similar to Randomized Expt with non-compliance
But the strata may be different

There can be non-IV alternatives

46

References

Iliev, Peter (2010), The Effect of SOX Section 404: Costs, Earnings Quality, and Stock Prices, 65 Journal of
Finance 1163-1196.
Kamar, Ehud, Pinar Karca-Mandic, and Eric Talley (2009), Going-Private Decisions and the Sarbanes-Oxley
Act of 2002: A Cross-Country Analysis, 25 Journal of Law, Economics and Organization 107-133.
Lee, David, and Justin McCrary (2009), The Deterrence Effect of Prison: Dynamic Theory and Evidence,
working paper

47