A truly dreadful study into the effects of prayer

In the middle of 2009, a small group of religious scholars and doctors lead by Candice Brown, Associate Professor of Religious Studies at Indiana University, travelled to Mozambique to find out if charismatic healing really works. So they went to some events laid on by Evangelical missions, and ran some tests on the recipients.

Unfortunately, what resulted was one of the shoddiest studies I’ve ever read!

Basically, the set up was this. The preacher asks if there’s anyone who has hearing or sight problems. The researchers measure their hearing and vision, then the patient gets a bit of laying on of hands, then the researchers re-run the hearing and vision test.

Well, the good news is that every single one of the 20-odd recipients was healed. The bad news is that they broke just about every rule in the book for conducting medical trials. Really, it’s a case study in how not to do these things.

First off, there was no control group. That means there’s nothing to compare it with. So there’s no way of knowing whether it was the healing, or something else that did the magic.

They measured hearing using an audiometer, which fires a a ‘beep’ at different volumes and frequencies into the ear. Sight was measured using a sight chart. Now, these are fine for use in diagnosis, but it’s useless for these sorts of studies because it’s entirely subjective.

And that, of course, means that it’s heavily susceptible to the placebo effect (and remember, there was no placebo control group). We know that the placebo effect is stronger the higher the level of drama and expectation you generate (think: “laying on of hands”!). And we know that remote prayer doesn’t work – probably because there’s no opportunity to harness the placebo effect. 

It gets worse than that. They were measuring hearing in a noisy environment (a big no-no). They were also measuring hearing on people who have no experience of such a hearing test. It’s well known that people get better at these with practice.

Well, all that is bad enough to disregard this study. But if that was all that was wrong with it, I’d just shrug my shoulders and say “meh”.

But the most egregious flaw in their study is something rather more interesting, and that’s something called the demand effect. That’s where subjects consciously or unconsciously respond in a way that they think is going to please the experimenter.

Cast your mind back to the set-up. They asked people to come forward if they were hard-of hearing. Then they tell them that they are going to test them, to see if they really are deaf. They fire a ‘beep’ into there ear and ask “Did you hear that”?

Now, what is the subject going to do in a situation like this? He’s already told the preacher that he’s deaf, and he doesn’t want to look a fool. He’s in a noisy environment, so it’s already quite hard to hear the beeps. He hears something faint… was it a beep? Nah, probably not. He’s deaf, after all, and so can’t hear very well!

Then the preacher comes along and does all the showmanship. They run the test again. Can they hear the beep this time? Well, he doesn’t want to let the preacher down, so of course he tries his damnedest to hear the faintest of beeps.

And that simple trick explains why both hearing and sight appears to have dramatically improved among these poor, superstitious villagers. Both they and the researchers have conned themselves.

You might wonder why they didn’t try doing the study in controlled conditions back home in the USA. Well, that’s because prayer doesn’t heal people in such environments. Seriously, that’s what they say!

The remarkable thing is that all these flaws are actually acknowledged in the paper (all they don’t seem to have got their heads round just what a headache the ‘Demand Effect’ is for this study).

They know about the flaws, but they choose to pretend they’re not important. Just check out the press release, which doesn’t mention a single one.

It’s a useless study, then. Now, a lot of people set out to do useless studies, but mostly they don’t get funded or published. So what happened here?

Well, it won’t surprise you to learn that the Templeton Foundation paid for this pitiful charade of science. They got their money’s worth, though, in terms of gullible press headlines.

But why on earth did it get published? The Southern Medical Journal is not exactly a world-class publication, but it does normally publish your more mundane mix of research and medical bits and pieces. They know the difference between a good study and a bad one.

Oh, and prayer. They publish a lot of stuff on prayer – 137 articles in the last 5 years alone.

If this study had been on anything other than prayer, it would never have been published. We can only guess at the reasons why this one got through.

Just to finish off, here’s a great (and short) video on the placebo effect. Perhaps we should sit Candice down and make her watch it before she tries her hand at any more medical research.

Brown CG, Mory SC, Williams R, & McClymond MJ (2010). Study of the therapeutic effects of proximal intercessory prayer (STEPP) on auditory and visual impairments in rural Mozambique. Southern medical journal, 103 (9), 864-9 PMID: 20686441

This article by Tom Rees was first published on Epiphenom. It is licensed under Creative Commons.