HOME

RADIO

Art & Design Film & Music

PEOPLE
Sc ienc e

ABOUT US
Soc iety Words

World

Fac ebook Google+ T witter T umblr

The maths of the paper disproving

conspiracy theories don't add up
By Martin Robbins

646

384

R e s e a r c h
c l a i m i n g
t o
h a v e
c r
m y s t e r i o u s
p l o t s
i s
f l a w e d
There are several ways to tell when youve written a paper thats not quite up to scratch.
Lingering doubts perhaps, or stinging comments from the reviewers. A definite clue is when you
turn on Thought for the Day and hear it cited as evidence for the resurrection of Christ.

Listen:

You need to install Flash Player to

play this content.

Download Flash Player now

"Conspiracy theories abound about what happened to Jesus of

Nazareth." Anne Atkins - 28/01/16
Thought for the Day
Thought for the Day
Thought for the Day

The paper in question, On the Viability of Conspiratorial Beliefs, was published in open-access
peer-review journal PLOS One a few days ago. It starts out with quite a good premise. There are
a lot of widely held beliefs that, if true, would require an extraordinary number of people to be
hiding something. If you think that man made climate change is a hoax, for example, then at a
minimum tens of thousands of climate researchers and other scientists must be in on it in some
way. If you think that NASA faked the moon landings then you have to deal with the fact that over
400,000 NASA employees would have been in some way connected to it.
Given those kinds of numbers, it seems almost inevitable that leaks would occur. Can we
predict how long a conspiracy of a given size will last? Thats the question asked by Dr David
Grimes - the researcher, skeptic and writer who published the paper. If we can prove
mathematically that a conspiracy involving 400,000 people cant last more than a few months or
years, then we can easily dismiss a number of popular beliefs.
Its a nice idea. Unfortunately the answer is a resounding "no", and the resulting paper ends up
being a sort of case study in how not to do statistics. Inevitably media outlets loved it, and so now
news feeds are full of headlines like: Most conspiracy theories are mathematically impossible,
The maths equation threatening to disprove conspiracy theories, Maths study shows

conspiracies prone to unraveling and so on and on.

Do you want to know a secret?

Lets imagine we want to predict how long a conspiracy will last. One approach would be to
collect a number of examples of existing theories of different sizes and durations, and plot them
on a chart. It might look something like the following graph, though ideally with a lot more data
points. If were really lucky, we might even see a pattern.

Grimes takes a different approach, except he has a grand total of, well, three examples to
extrapolate from. Its not exactly a data goldmine.

The examples are three real life conspiracies that were eventually exposed the NSAs PRISM
programme, the Tuskegee syphilis experiment, and a scandal at the FBI about the accuracy of
their forensics tests. In each case the number of people involved is complete guesswork. For the
NSA Grimes uses a figure of 30,000, which assumes the entire workforce from director down to
janitor were involved in one project. In reality it could have been hundreds or even just tens of
people. The data is basically nonsense you could put the points almost anywhere to make
whatever pattern you liked.
So there arent enough examples, and the few listed arent very reliable. Theyre also the wrong
kind. Were missing any data on conspiracies that have remained secret for obvious reasons.
Two of the three predate the Internet era, and youd expect a revolutionary global
communications system to have some impact on communications. There arent any small
conspiracies of say 10 or 20 people. All of the examples are big, institutional or community
affairs. All three are based in the United States, two in law enforcement or security services
where secrecy is part of the job description and the cost of breaking it is extreme.
From these examples, Grimes calculates that the likelihood of a typical person blabbing about
their secret conspiracy in a given year is roughly five ten thousandths of a percent. If this were
true, there wouldnt be a word for gossip in the dictionary. Wed walk around in stony silence all
day tapping our noses at people. The standard response to How are you? would be Never you
mind. Its the sort of number you accept without thinking when youve forgotten that the numbers
youre playing with relate to actual human beings.

Do you promise not to tell?

We could get deeper into the basic premises of the paper. How do you even define a conspiracy,
and how is it different to a secret? What does it mean for a conspiracy to leak, and can people
even agree on whether a leak has occurred? After all, climate skeptics would point to the
University of East Anglia e-mail leak as an example of a leak occurring and large numbers of
people becoming aware of the truth.
For that matter, why should dying prevent someone being the source of a leak (history is littered
with examples of lost papers found in attics), and why would his or her probability of leaking
remain constant over time, when some conspiracies by their nature are less important to conceal
years after the fact?
As it happens though, all thats irrelevant because theres a much bigger issue. As far as I can
tell the maths underpinning his model is wrong. Several statisticians who I spoke to or who
checked the paper independently agree.
The root of the problem lies in this graph. Forget the jargon for a moment. All this shows is the
cumulative probability over time of a conspiracy being discovered.

cumulative probability over time of a conspiracy being discovered.

The assumption is that the average person has a 0.0005% chance of leaking in any given year.
The blue line shows how the probability grows year after year if 5000 people are involved, all
those individual chances adding up with time. This is exactly the same type of curve youd get if
you plotted the odds of rolling a six from a growing number of dice your odds get closer and
closer to 1 but never quite reach it.
The other lines show what happens if you assume that people die off, using two different rates.
The orange line assumes half of the people involved die every 10 years, so after a decade there
are 2500 people left, 1250 after two decades and so on. The pink line uses a more sophisticated
equation for mortality.

Got that? If so, youre doing better than the author. Heres the problem: if the probability of
exposure is accumulating over time then why does it start going down? You cant make a
conspiracy secret again! The increase each year should fall toward 0 as the population dies off,
leaving the plots to trail off as flat lines.
In fact they dont account for a changing population at all, as I discovered when I wrote a simple
computer simulation to test his model. Thanks to a very basic calculus error, the lines actually
show what the cumulative probability would have been if the population had started at the current

level and stayed constant. So if we take the orange line at 10 years, instead of telling us the
cumulative probability if the population had started at 5000 and fallen to 2500 over time, it gives
us the value if the population had started at 2500 and stayed constant. Since this error occurs in
equation 1, which the rest of Grimess equations are derived from well, you get the idea.
In truth it doesnt affect the conclusions too much, because the timelines for the conspiracies
Grimes looked at were so short that mortality didnt really kick in. The conclusion still makes
sense if you have a lot of people, hiding a conspiracy becomes incredibly unlikely. But we
knew this already. Even if we corrected his mistake, the model Grimes presents would just be a
standard probability curve with some improbable assumptions plugged into it. It doesnt get us
any closer to a prediction for the longevity of any real world conspiracy. As equations go, its up
there with the infamous Blue Monday a plausible sounding story for the press that lacks any
scientific rigour.
It would be easy to blame Grimes for all this, but the bigger failure here is in PLOS ONEs peer
review process. Its easy to screw up calculus. Whats less excusable is that expert reviewers
looked at this paper ahead of publication, and none of them spotted an elementary mistake that
myself and others saw almost immediately. Numerous other helpers are cited in the
acknowledgements, but none of them seem to have glanced at the math or challenged some
really odd assumptions. Grimes made a mistake we all do but he was also severely let down
by his peers and colleagues.
And its frustrating. Its frustrating because skeptics and rationalists ended up retweeting news
stories that were just as bogus as the bad science they claim to stand against. Its frustrating
because a paper that lashes out against the idea that scientists might be engaged in covering up
bad research turns out to be an example of bad research that slipped through peer review. Its
frustrating because the model was so ill thought that Anne Atkins on Radio 4's Thought for the
Day was able to use it as evidence for the resurrection of Christ after all, if so many people
bore witness to it then the maths "proves" that a conspiracy to conceal the truth could not have
lasted two millennia.
Which leaves perhaps the biggest question of all: was this really just a bad paper, or was there
some deeper purpose behind it? Is Doctor Grimes engaged in some kind of charade, running
interference on behalf of a master or masters unknown? Is he still the real Grimes, or has he
been replaced by a foppish-haired lizard impersonator? The truth is out there
(Many thanks to Adam Jacobs of statsguy.co.uk for double-checking my late-night maths)
Science

mat hemat ics

conspiracy t heories

Science

646

Jan 29, 2016

384

M A

I N

I N

Martin Robbins is a writer and talker at the messy border of science and culture. He is a
columnist at VICE, and blogs for The Guardian and the New Statesman.

R E L A T E D

P O S T

NOW READ

Anti-Semitism is in the very marrow of conspiracy theory

[X]

Corbyn, Trump and Farage: The Rise of the Paranoid Style

Blue Mondays not real, but the happiness industry can still get you down

Little Atoms magazine Issue 1 on sale now

14 comments
22 people listening

Sign in

+ Follow

Share

Post comment as...

Newest | Oldest | Top Comments

ChadWeber

Feb 8, 2016

Grimes is a space lizard who is trying to make his own work seem shoddy so that people will
suspect him of being a space lizard because Grimes knows that everyone else will laugh at the
people who believe in space lizards. That's the only logical explanation. And I'm pretty sure the
author of this article is also a space lizard, who mentions the possibility of Grimes being a lizard
in jest in an attempt to make the entire premise of space lizards seem laughable.
Like

nmrqip

Reply

Feb 6, 2016

I think I have worked out the underlying physical model implied by his equations. Of course this
isn't what he actually did: he worked out an incorrect mathematical description of a slightly more
sensible physical model. But it is possible to back out the implicit physical model for which his
maths is correct.

Grimes assumes that a conspiracy fails on a single defection. This is a hopelessly naive model
(note that the mafia and drug cartels continue to exist despite multiple defections), but it is at
least well defined. He then correctly notes that if a conspiracy member dies before defecting then
the number of potential defectors is reduced, and so is the probability of new defections.
However his mathematical description implies that if a conspirator dies after defecting then the
effects of his previous defection are mysteriously undone, and if he as the only defector at that
stage then the conspiracy leaps back into secrecy again. This is why his failure probabilities fall
to zero once everyone has died and all possible defections have been reversed.

The more I think about this paper the worse it gets.

Like

Conspiracy_X

Reply

Feb 2, 2016

This was naive! Ever heard of compartmentalization? Just an example, or two: NASA admits
multiple times Moon landings couldn't have happened. We admittedly haven't figured out yet how
to get man through the Van Allen Radiation Belt. Technology isn't there yet:
https://youtu.be/51DED8dcNkA ------- Or, how about the admission that all of the Middle-Eastern
wars were planned before 9/11 and before Bush Jr. even stepped undeserved foot into office:

wars were planned before 9/11 and before Bush Jr. even stepped undeserved foot into office:
https://youtu.be/9RC1Mepk_Sw ------- How about the MLK Jr. family winning a lawsuit against
the US government in 1999 for being directly behind his assassination:
http://www.washingtonsblog.com/2015/01/martin-luther-king-assassinated-us-govt-kingfamily-civil-trial-verdict.html ------- Or, maybe you'd like to hear dozens of Politicians and the
like admitting to an agenda called the New World Order: https://youtu.be/uIH4jUIK5zs ------- But,
just as important are the psychologists explaining why it's so hard for morons like yourself to
accept reality: https://youtu.be/f462ya0DC0g

NASA Admits They Can't Send Humans Through The Van

Allen Radiation Belts

Like

TylerAndrewMaire

Reply

Feb 5, 2016

@Conspiracy_X Nah man, you can get through it if you keep wearing your tinfoil hat
Like

greybuscat

Reply

Feb 1, 2016

I wonder why he only picked three. The Tonkin Gulf incident is a great example of a real-life
conspiracy. In the US, it's the probably THE best conspiracy one could choose.
Like

cbauch

Reply

Jan 31, 2016

I was the handling editor and the point about the choice of population size being overlooked in
peer review is wrong.In fact, this issue was flagged in the review process, but in most cases there
are no data to decide on what the relevant subpopulation size should be.As the handling editor, I
agreed it was better to use a known data point and to discuss the implications of it being too large
in the papers Discussion section, rather than guess what the data should be, thus providing only
the illusion of accuracy.In other cases, the population size the author used might have been too
small, as in the case of vaccines where academic researchers conducting the clinical trials would
also have to have been in on the conspiracy.A univariate sensitivity analysis could have been
conducted, but again, we have no data to decide on suitable bounds for the sensitivity analysis.
As it stands, someone who actually reads the entire paper will be able note the limitation of using
the entire workforce size and understand how that might influence results.As I doubt NASA has

the entire workforce size and understand how that might influence results.As I doubt NASA has
given serious thought to how many employees would have to have been involved in a Moon
Landing hoax 40 years ago, we need to leave the problem of subpopulation size to future work.For
now, the paper has the nontrivial value of illustrating an interesting application of failure theory.

Questioning the validity of the study because evangelical Christians use it as evidence for the
resurrection of Christ is also wrong, for the same reason that we cant discard the Big Bang
Theory just because the Catholic Church likes it, or that we cant discard legitimate research
exploring why some individuals (as it so happens, 10% of individuals of European descent) are
resistant to HIV because white supremacists cite this as evidence of white superiority. Maybe
Robbins was being facetious, but its hard to tell for sure.

The point about changing population size is valid and the author should have used a different type
of statistical model called a non-homogeneous Poisson model, but this should not change the
conclusions and in fact should make them stronger, so the current analysis is actually
conservative (although I have recommended the author be given a chance to respond through an
erratum).The homogeneous Poisson model is in fact a simplifying approximation in the Gompertz
case for the first 30 years or so.Thank you to Jonathan Jones and colleagues for pointing out the
issue of changing population size on the PLOS ONE website in a professional and balanced
manner.All scientists make mistakes at one point or another, and this is one way in which
science corrects its mistakes and moves forward. Its good to see the process working and to see
non-scientists taking interest in it, although the confusion and distortion of the papers objectives,
merits and peer review process are unfortunate.

Chris Bauch
Like

MartinRobbins

Reply

Feb 1, 2016

@cbauch This is a truly surprising response. For the sake of brevity I'm going to focus
on just one statement, which I think highlights just how poorly the handling editor of the
paper appears to understand the work being handled. You said: " this should not
change the conclusions and in fact should make them stronger, so the current analysis
is actually conservative."

As I said over at the comments on PLOS, "If a major mathematical error in a model
fails to change the conclusion then one has to ask whether the methodology properly
evaluated the model. In this case, we can see that it did not." The only reason the
conclusions stand is because the model was not tested on any examples with less
than tens of thousands of participants, so that none of the conspiracies investigated
would have collapsed in a long enough time period for mortality to be a factor.

To quote myself at PLOS again, "This is equivalent to Newton suggesting a law of

gravity that all objects fall to the ground within two seconds, but only ever testing it up
to heights of one metre." The idea that somehow the conclusions are strengthened by
this failure to test them on a viable set of data is beyond parody. On what possible
basis are they strengthened?! If it's simply that they conform more closely to the
constant population model then of course they do - if you select examples where
mortality isn't an issue then it will naturally lead to the constant population model being
a better fit!

Then let's take population. We have three examples where the populations are out by
potentially a factor of 100 or more, and even then one candidate value for the
probability per person per year is 50 times larger than the other two. You can't just take
the smallest value and say that you're being conservative when the value of 'p' chosen
could be 100s of times out! Can you not see that this would have a dramatic impact on

could be 100s of times out! Can you not see that this would have a dramatic impact on
the viability of any conspiracy theory?

Yes, this is hand-waved a little in the conclusion, but I'm lost as to what exactly the
paper achieves when 2 of the 3 models are incorrect, the data used to test them is too
small and seriously flawed, and the third model is simply as standard Poisson
distribution. You refer to "the confusion and distortion of the papers objectives."

Can you please explain what you believe those objectives to be, since they don't
apparently involve either developing a novel model of conspiracy theory viability or
testing such a model?

Like

BarryJWoods

Reply

Feb 1, 2016

@MartinRobbins @cbauch
I imagine you will be shown the red flag soon..

Nature: Research Integrity: Don't let transparency damage science

http://www.nature.com/news/research-integrity-don-t-let-transparencydamage-science-1.19219
Like

Twundit

Reply

Jan 31, 2016

Well.. this was an interesting experiment to see how long it took for people to notice and come out
with the obvious error. All in line with the subject matter.
Like

anarcissie

Reply

Jan 31, 2016

@Twundit -- Many people -- dozens or hundreds -- may have noticed the errors (that's
a plural), but not had time or inclination to write to the media which passed it along.
There are at least two major social problems here. One is the very low quality of
reporting on science; it is evidently done in many cases by people who know very little
about any science. The other is the deterioration of supposedly real science into
pseudoscience at the source, as in the present case. Scientists are not supposed to
just make stuff up. I am still not sure the theory is not some kind of joke.
Like

anarcissie

Reply

Jan 30, 2016

When I first saw this story, I thought it was a hoax or a satire of some sort. Regardless of the
mathematics used, the author seems to be pulling most of the numbers out of his nether regions.
You don't need to be any sort of mathematician or sociologist to see that.
Like

nmrqip

Reply

Jan 30, 2016

Martin Robbins is entirely correct in what he says here. The two of us detected the flaw
independently, by very much the same reasoning (cumulative failure curves just can't turn down!).

I posted a comment on the PLOS article page

at http://www.plosone.org/annotation/listThread.action?root=88142 first thing on Friday
morning explaining the error. The original author, David Grimes, has replied, implicitly accepting
that I have correctly identified an error, but attempting to play down its significance. In reality the
error is fundamental: the incorrect formula is the only novel element of the paper, and everything
that remains is essentially trivial. The paper should probably be withdrawn, but at the very least a
formal correction to the published article should be made.
Like

BarryJWoods

Reply

Jan 29, 2016

oops - 19% thought gov covered up existence of aliens..

Like

BarryJWoods

Reply

Jan 29, 2016

Slightly off topic, but a survey of 137 2nd year pyschology undergraduates, showed 19% of them
'believed in the moon landings were faked, and 54% thought Dianas death not an accident, and
14% that governments covering up the existence of aliens. That is what the data says, didn't
make it into the paper though..

Dead and Alive- Wood et al - Michael wood quickly sent out the datasets, so if curious just ask.
http://www.winchester.ac.uk/academicdepartments/psychology/staff/Pages/DrMichaelWood.
aspx

funny thing, he used stats to come up with the headline title, and not a single person actually
agreed with that hypothesis, in the dataset.

my point, doing tickbox surveys in psychology, not very useful!

2nd point, why survey a group of 2nd yr, 20.3 av yr ld, 83% female psychology undergrads, and
think it has any relevance with respect to the public.
Like

Reply

R E C E N T

P O S T S
W h y
d e t e

d i d
n t i o

A
n

i
c

r
e

S i
t r

m a
e ?

Read more
T h
m e

e
s

s
s

e
a

v e
e

r y
f o

j
r

u n i o r
J e r e m

o
n

Read more
P
S

o
e

l a n
m i t

d
i c

h
p

o
a

u l
s t

Read more
I

Read more
B
U

o
n

u
i

t
t

r
e

o
d

B o
N a t

u
i

t
o

r o
n s

Read more
Tweets by @little_atoms

C O N T R I B U T O R S
Maruxa Ruiz del rbol

Adam Rutherford

Ade Adepitan

Ade Adepitan

Alex Masters

Alex Mik

Alison Rooper

Andrew Copson

Andrew Mueller

Anthony Nolan

Bill Thompson
1 of 8

next