A little more than a month ago, a paper about atheists and distrust garnered some attention and outrage in nonreligious circles, not necessarily because of what the study said but how it was interpreted—bloggers and irresponsible reporters claimed this study demonstrated that atheists were less trusted than rapists. I suspected almost immediately that this claim would be echoed without any real understanding of it, like the nonsense notion that atheists are the most reviled, most hated, or least trusted minority in the United States. I don’t know what it is about statistical research in the social sciences that makes otherwise intelligent people repeat sensational claims again and again, but it’s extremely frustrating.
But on to the actual study. The Friendly Atheist writes:
Somehow, we’re less trusted than even rapists. That’s disheartening, but it really says more about how religious people think than anything about atheists.
And he’s partway right: if we were less trusted than rapists that would be really disheartening and controversial, but thankfully, as I wrote last month, it’s not at all true.
But since I’m still working my way through the finer nuances of blogging and writing more generally, I’ve learned that I can occasionally suck terribly at explaining things. Reading through some comments on the article, it looks like people aren’t quite following what I meant to say. There are also a lot of common objections I’ve read, particularly with how I addressed the issue of a ceiling effect. So I think with those in mind, the study deserves another pass.
To just summarize my last post, there were basically two problems with the press this paper was getting. First, there was a problem with how people were interpreting the graph that’s been circulated (from those who claim that atheists are more distrusted than rapists), and second, there’s a problem with the measure the authors of the study used (not necessarily for the purposes of the paper, but for the purposes of supporting the claim that atheists and rapists are similarly or comparably mistrusted).
For the first claim, if you take a look at the graph, it reports the number of participants in the study who performed what’s known as a conjunction fallacy. The claim that participants made this fallacy more for atheists than rapists (and thus trust atheists less than rapists), is not at all supported by the data. Looking at how huge the error bars on the graph are, it’s almost guaranteed that any difference between the two is simply statistical noise due to a small sample size. (The error bars are the thin lines going up and down from each bar, which represent the range where the real value of the statistic lies. Big error bars mean that you can’t tell exactly where the actual value is.)
As for the content and methods of the study, the participants were asked which was more likely: that a terrible person was a teacher or a teacher and x, where x was either a Christian, Muslim, rapist, or atheist.
A moment of reflection or knowledge of basic probability should make it clear that it’s always more likely for the terrible person to be a teacher rather than a teacher and something else. But we have a known bias called the representativeness heuristic, which leads us to make errors in judgment like this. We don’t see terrible behavior as representative of a teacher, but we do see it as representative of a rapist (or perhaps an atheist), so we see that option as more likely.
But this doesn’t support the conclusion that the two are similarly mistrusted, because as I argued, it doesn’t rule out the possibility of a ceiling effect. In short, a ceiling effect in this instance might go as follows: people just aren’t that likely to make the conjunction fallacy. No matter how representative a distrusted group may be for immoral behavior, it could simply be the case that most people just won’t make the error (that is, the measure hit a ceiling in sensitivity). If x was anything that was stereotypically associated with distrust, be it an atheist, rapist, burglar, Neo-Nazi, or whatever, it could be that only about half of the participants might fall for the trick. It would be similar to reading the results of a really easy test. You’d see a difference in scores only for the less bright kids, but past a certain point all the marginally intelligent kids (as well as the brilliant kids) look about the same, because of the ceiling (a perfect score). So it’d be a mistake to say two kids are just as smart because they have the same score on a test, because the test might not be very good at measuring what it’s supposed to. A situation like this might be going on here with distrust in this study (and I think it is, considering how crazy the claim people are making from the results).
I’ve seen some comments dismissing my post by suggesting that I posit a ceiling effect without any evidence, but that’s precisely backwards because science is in the business of ruling out alternative explanations for the claims you make. If the data could be accounted for by two different stories—either that atheists are more distrusted than rapists (a shocking and outlandish claim, thus the huge press surrounding it) or that the measure the study used wasn’t very good—then to say “this study found that rapists are more trusted than atheists” isn’t at all justifiable because it does nothing to rule out the alternative.
There are some broader theoretical points worth making too, I think. Implicit measures such as these often don’t report explicit attitudes or predict some behaviors very well, but rather just report whatever stereotype is most readily accessible. For example, implicit measures of African Americans tend to show racist attitudes, even from other African Americans. The effect goes away, however, if participants are primed with high-achieving examples of African Americans, such as President Obama. Often simply the presence and availability of a stereotype can strongly influence such measures, so it’s always important to be careful about reading too much into the results of studies with implicit measures such as these.
Essentially, it’s just an absurdly strong conclusion that’s based on a bizarre implicit measure whose data don’t support a claim the authors of the paper weren’t even trying to make. A lot more work needs to be done to support a claim as counterintuitive as “rapists are more trusted than atheists (or comparably trusted, or whatever),” and until good reasons to actually believe a claim like this surface, I’m more than confident dismissing it.
So hopefully this time I made the point a bit more succinct and clear. If I’m missing something or still failing on any explanations, do let me know and I’ll edit or try again as necessary.