follow CCP

Recent blog entries
popular papers

What Is the "Science of Science Communication"?

Climate-Science Communication and the Measurement Problem

Ideology, Motivated Cognition, and Cognitive Reflection: An Experimental Study

'Ideology' or 'Situation Sense'? An Experimental Investigation of Motivated Reasoning and Professional Judgment

A Risky Science Communication Environment for Vaccines

Motivated Numeracy and Enlightened Self-Government

Ideology, Motivated Cognition, and Cognitive Reflection: An Experimental Study

Making Climate Science Communication Evidence-based—All the Way Down 

Neutral Principles, Motivated Cognition, and Some Problems for Constitutional Law 

Cultural Cognition of Scientific Consensus

The Tragedy of the Risk-Perception Commons: Science Literacy and Climate Change

"They Saw a Protest": Cognitive Illiberalism and the Speech-Conduct Distinction 

Geoengineering and the Science Communication Environment: a Cross-Cultural Experiment

Fixing the Communications Failure

Why We Are Poles Apart on Climate Change

The Cognitively Illiberal State 

Who Fears the HPV Vaccine, Who Doesn't, and Why? An Experimental Study

Cultural Cognition of the Risks and Benefits of Nanotechnology

Whose Eyes Are You Going to Believe? An Empirical Examination of Scott v. Harris

Cultural Cognition and Public Policy

Culture, Cognition, and Consent: Who Perceives What, and Why, in "Acquaintance Rape" Cases

Culture and Identity-Protective Cognition: Explaining the White Male Effect

Fear of Democracy: A Cultural Evaluation of Sunstein on Risk

Cultural Cognition as a Conception of the Cultural Theory of Risk

« So we know we can't defeat entropy; but what about overplotting???? | Main | The science of science documentary filmmaking: the missing audience hypothesis »

Cognitive reflection and "belief in" evolution: critically engaging the evidence

1.   Two hypotheses on "disbelief in evolution"

Why do 45% or so of Americans consistently say they don’t “believe” humans evolved from an earlier species?

How come about only one-third of them say they accept a conception of evolution—science’s conception—that features  mechanisms of natural selection, random mutation, and genetic variance  (the modern synthesis) as opposed to an alternative religious one that asserts a “supreme being guided the evolution of living things for the purpose of creating humans and other life in the form it exists today?”

These questions get asked constantly. Makes sense: they’re complicated,  and also extremely consequential for the status of science in a liberal democratic society.

One popular answer attributes “disbelief in” evolution to a defecit in critical reasoning that interferes with people’s ability to recognize or accept scientific evidence.  I’ve referred to this in other contexts as the “public irrationality thesis” (PIT) (Kahan in press).

Actually, I think PIT, while a plausible enough conjecture, is itself contrary to weight of the scientific evidence on who believes what and why about human evolution.  

It’s well established that there is no meaningful correlation between what a person says he or she “believes” about evolution and having the rudimentary understanding of natural selection, random mutation, and genetic variance necessary to pass a high school biology exam (Bishop & Anderson 1990; Shtulman 2006).

Click on it! Item repsonse profiles rock!There is a correlation between “belief” in evolution and possession of the kinds of substantive knowledge and reasoning skills essential to science comprehension generally.  

But what the correlation is depends on religiosity: a relatively nonreligious person is more likely to say he or she “believes in” evolution, but a relatively religious person less likely to do so, as their science comprehension capacity goes up (Kahan 2015).

That’s what “belief in” evolution of the sort measured in a survey item signifies: who one is, not what one knows. 

Americans don’t disagree about evolution because they have different understandings of or commitments to science.  They disagree because they subscribe to competing cultural worldviews that invest positions on evolution with identity-expressive significance. 

As with the climate change debate, the contours and depth of the divide on evolution are a testament not to defects in human rationality but to the adroit use of it by individuals to conform their “beliefs” to the ones that signal their allegiance to groups engaged in a (demeaning, illiberal, and unnecessary) form of cultural status compeitition.  

Call this the “expressive rationality thesis” (ERT). It's what I believe—on the basis of my understanding of the best currently available evidence (Kahan 2015).

2. New evidence for PIT?

But if one gets how science works, then one knows that all one’s positions—all of one’s “beliefs”—about empirical issues are provisional.  If I encounter evidence contrary to the view I just stated, I’ll revise my beliefs on that accordingly (I’ve done it before; it doesn’t hurt!).

So I happily sat down last weekend to read Gervais., W, “Override the controversy: Analytic thinking predicts endorsement of religion,” Cognition, 142, 312-321 (2015).

Gervais is a super smart psychologist at the University of Kentucky. He's done a number of interesting and important studies that I think are really cool, including one  that shows that people engage in biased information processing to gratify their animus against atheists (Garvais, Shariff & Norenzayan 2011), and another that reports a negative association between critical reasoning and religiosity (Gervaise & Norenzayan 2012). 

In this latest study, Gervais correlated the scores of two samples of Univ. of Kentucky undergrads on the Cognitive Reflection Test (CRT) and their beliefs on evolution.

As discussed in 327 previous posts, the CRT  is regarded as the premiere measure of the capacity and disposition to use conscious, effortful, “System 2” information processing as opposed to unconscious, heuristic “System 1” processing, the sort that tends to be at the root of various cognitive miscues, from confirmation bias to the gambler's fallacy, from base rate neglect to covariance non-detection (Frederick 2005).

Gervais hypothesized that disbelief in evolution is associated with overreliance on “intuitive” or heuristic “System 1” forms of information processing as opposed to conscious or “analytic” “System 2” forms.

 “[M]any scientific concepts are difficult for people to grasp intuitively while supernatural concepts may come more easily,” he explains.

From a young age, children view things in the world as existing for a reason; they view objects as serving functions. This promiscuous teleology persists into adulthood, even among those with advanced scientific training. Further, functionally specialized features of animals (such as a zebra’s stripes or a kangaroo’s tail) are viewed as inherently characteristics of an animal’s ‘‘kind,’’ perhaps implying a deeper and more temporally stable essence of the animal. If objects in the world, including living things, are intuitively imbued with function and purpose, it seems a small step to viewing them as intentionally designed by some external agent. . ..

Given that children and adults alike share the intuition that objects in the world, including living things, serve functions and exist for purposes, they may infer intentional agency behind intuited purpose.

Finding a negative correlation between CRT and belief in evolution, he treats the results of his study as supporting the hypothesis that “analytic thinking consistently predicts endorsement of evolution.” 

Because the influence of CRT persists after the inclusion of religiosity covariates, Gervais concludes that the “cultural” influence of religiosity, while not irrelevant, is “less robust” an explanation for “disbelief in” evolution than overreliance on heuristic reasoning.

In sum, Gervais is offering up what he regards as strong evidence for PIT.

3. Weighing Gervais’s evidence

So what do I think now?

I think Gervais's data are really cool and add to the stock of evidence that it makes sense to assess in connection with competing conjectures on the source of variance in belief in evolution.

But in fact, I don’t think the study results furnish any support for PIT! On the contrary, on close examination I think they more strongly support the alternative expressive rationality thesis (ERT).

a. Just look at the data. To begin, the correlation that Gervais reports between CRT scores and disbelief in evolution  actually belies his conclusion.

Sure, the correlation is “statistically significant.” But that just tells us we wouldn’t expect to find an effect as big as or bigger than that if the true correlation were zero.  The question we are interested in is whether the effect is as big as PIT implies it should be.

The answer is no way!

People familiar with logistic regression would probably have an inkling of this when Gervais reports that the “odds ratio” coefficient for CRT is a mere 1.3. An odds ratio of “1” means that there is no effect—and 1.3 isn’t much different from 1.

But researcher shouldn’t presuppose readers have “inklings,” much less leave them with nothing more to go on.  They should graphically display the data in a way that makes their practical effect amenable to reasoned assessment by any reflective person.

The simplest way to do that is to look at the raw data here.

Admirably, Gervais posted his data to his website.  Here’s a scatterplot that helps convey what the “OR = 1.3” finding means as a practical matter:

These scatter plots relate CRT to endorsement of the modern synthesis position as opposed to either “new earth creationism” or a “divine agency” conception of evolution in which a “supreme being guided the evolution of living things for the purpose of creating humans and other life in the form it exists today."

I think that’s the right comparison if we are trying to assess Gervais's conjecture that overreliance on System 1 reasoning accounts for the stubbornness of “the intuition that objects in the world, including living things, serve functions and exist for purposes" reflecting "intentional agency." But the picture is pretty much the same when we look at how CRT relates to endorsement of the proposition that “God created human beings pretty much in the present form at one no part in guided the present time within the last 10,000 years or so."

Sure, there’s a modest uptick in belief in evolution as CRT increases.

But even those extremely reflective "3's"--a decided majority of whom attribute the natural history of human beings to divine agency-- don't exactly look like a sample of Richard Dawkinses to me!

Gervais states that these “results suggest that it does not take a great deal of analytic thinking to overcome creationist intuitions.”

But in fact they show that, at least for the overwhelming majority of University of Kentucky undergrads, it would take an amount that far exceeds the maximum value on the CRT scale!

This just isn't the picture one would expect to see if resistance to science's account of evolution was a consequence of overreliance on heuristic or System 1 reasoning.

b. Test the alternative hypothesis!  Even more important, the data do look like what you’d expect if the expressive rationality thesis (ERT) explained “belief”/“disbelief” in evolution. 

ERT posits that individuals will use their reason to fit their beliefs to the ones that predominate in their cultural group (Kahan 2013).  As explained, existing evidence is consistent with that: it shows that individuals who have a cultural style that features modest Mmm raw data! Always demand a helping of it when served statistically processed datareligiosity become more likely, but those with one that features strong religiosity less likely, to profess belief in evolution.

The way to test for such an effect is not to put religion into a multivariate model as a “control” as Gervais did,  but to examine whether there is an interaction between religiosity and CRT such that the effect of the latter depends on the level of the former.

Here’s what what that interaction looks like in a regression model of "belief in evolution" for a general population sample, in which religiosity is measured with a composite scale reflecting self-reported church attendance, frequency of prayer, and importance of religion in one’s life (α = 0.80):

If we look, we can find the same interaction in Gervais’s data. 

This figure graphically displays output of a regression model that uses Study 1’s 7-point “belief in God” scale.


The modest impact of CRT in the sample as a whole is driven entirely by its effect on relatively less religious subjects.

yummy! raw data for regression model above!

Study 2 has a “belief in God” measure, too, scaled 1-100.  One-hundred point measures are a very bad idea; they aren’t going to measure variance any better than a 10-point (or probably even 7-point) one, but are going to have tons of noise in them.

The study also had a 7-point church attendance measure, so I combined these two into a scale.

Here’s what the raw data look like when we examine how CRT relates to acceptance of the modern synthesis position on evolution in Study 2:

Once more, it's plain to see that CRT isn't having any effect on subjects above average in religiosity.  The interaction is there in the regression model, too, but because of the wobbly religiosity measure and smaller sample the model is underpowered (b = -.36, p = 0.07, for "theistic evoluition vs "creationism"; b = -.39, p = 0.19, for "naturalistic" vs. "creationism"). (Actually, if one just uses the 100-point "belief in God" measure, there it is, "statistically significant"--for those who view p < 0.05 as having talismanic significance!)

Contrary to what Gervais concludes from his analyses, then, the evidence doesn’t in fact show a “consistent pattern whereby individuals who are more prone and/or able to engage in analytic thinking” use that capacity to “override” the “intuition objects in the world, including living things, serve functions and exist for purposes” reflecting “intentional agency” in their creation.

We see that “pattern” consistently only in non-religious individuals.

That’s what ERT predicts: as individuals become more cognitively proficient, they become even more successful at forming and persisting in beliefs that express their identity.

I think Gervais missed this because he didn’t structure his analyeses to assess the relative support of his data for the most important rival hypothesis to his own.

In fairness, Gervais does advert to some analyses in his footnotes that might have led him to believe he could ruled out this view. E.g., he didn’t find an interaction between the predictors, he reports, when he regressed belief in evolution on CRT and a “religious upbringing” variable in study 1.  But that's hardly surprising: that variable was dichotomous and answered affirmatively by 75% of the subjects; it doesn’t have enough variance, and hence enough statistical power, to detect a meaningful interaction.

In study 2, Gervais administered a nonstandard collection of variables he calls “CREDS,” or “credibility enhancing displays.”  Unfortunately, the item wording wasn't specified in the paper, but Gervaise describes them as measuring variance in “believing” in and “acting” on “supernatural beliefs.”  

Gervais reports that the CREDS had only a modest correlation with disbelief in evolution, and also didn’t interact with religiosity when included as predictors of CRT.  I really don’t know what to say about that, except that the discrepancy in the performance of the CRED items, on the one hand, and the Belief in God and church attendance ones, on the other, make me skeptical about what the former is measuring.

I think Gervais should have displayed a bit more skepticism too before he concluded that his data supported PIT.

5.  Limits of Yucky NHT

One last point, this one on methods.

The problem I have with Gervais’s paper is that it relies on an analytical strategy that doesn't test the weight of the evidence in his data in relation to hypotheses of consequence.  He tells us that he has found a “significant” correlation—but doesn’t show us that the effect observed supports the inference that his hypothesis depends on or rules out a contrary inference supportive of an alternative hypothesis .

These problems are intrinsic to so-called “null hypothesis testing.” Because the “null” is not usually a plausible hypothesis, and because “rejecting" it is often perfectly consistent with multiple competing hypotheses that are plausible, a testing strategy that aims only to “reject the null” will rarely give us any reason to revise our prior assessments of how the world works.

Good studies pit opposing hypotheses against each other in designs where the result, whatever it is, is highly likely to give us more reason than we had before for crediting one over the other.

Gervais is a very good psychologist, whose previous studies definitely reflect this strategy. This one, in my view, wasn’t as well designed—or at least as well analyzed—as his previous ones.

Or maybe I'm missing something, and he or someone else will helpfully tell me what that is!

But no matter what, given the balance of the evidence, I remain as convinced that Gervais is a superb scholar as I am that PIT doesn't explain conflicts over evolution, climate change, and other culturally contested science issues in the U.S.


Bishop, B.A. & Anderson, C.W. Student conceptions of natural selection and its role in evolution. Journal of Research in Science Teaching 27, 415-427 (1990).

Frederick, S. (2005). Cognitive Reflection and Decision Making. Journal of Economic Perspectives, 19(4), 25-42.

Gervais, W. M. (2015). Override the controversy: Analytic thinking predicts endorsement of evolution. Cognition, 142, 312-321. doi:

Gervais, W. M., & Norenzayan, A. (2012). Analytic Thinking Promotes Religious Disbelief. Science, 336(6080), 493-496. doi: 10.1126/science.1215647

Gervais, W. M., Shariff, A. F., & Norenzayan, A. (2011). Do you believe in atheists? Distrust is central to anti-atheist prejudice. Journal of Personality and Social Psychology, 101(6), 1189.

Kahan, D.M. Climate-Science Communication and the Measurement Problem. Advances in Political Psychology 36, 1-43 (2015).

Kahan, D.M. Ideology, Motivated Reasoning, and Cognitive Reflection. Judgment and Decision Making 8, 407-424 (2013).

Kahan, D.M. What is the science of science communication?” J. Sci. Comm. (in press).

Shtulman, A. Qualitative differences between naïve and scientific theories of evolution. Cognitive Psychology 52, 170-194 (2006).


PrintView Printer Friendly Version

EmailEmail Article to Friend

Reader Comments (13)

A rather classic blogospheric discussion, IMO. You say that he makes an argument that he says that isn't his argument.

I wonder if, to some degree, such convos aren't a function of the electronic medium, where instead of an interactive discussion that likely involves listening and responding in real time, we have a context where people have a serial string of discrete and non-interactive communication acts with a much enhanced ability to filter everything their interlocutor says (in a way that reminds me of the distinction between hearing and listening) so as to confirm biases.

I wonder if there's something useful here w/r/t climate change, as much of the "discussion" between those in disagreement takes place in an electronic medium... or at least much of the discussion that is visible to the public.

If the two of you sat down over a beer to discuss the implications of your findings, respectively, while disagreement wouldn't likely evaporate, I wonder if the disagreement would be of a different nature - such that the conclusions wouldn't be that you're talking to people who aren't even involved in the exchange, or talking in circles?

August 25, 2015 | Unregistered CommenterJoshua


Well, someone is relying on others simply not reading for themselves.

But for sure not typical scholarly exchange, for whatever reason.

August 27, 2015 | Registered CommenterDan Kahan

"Consistently" should always specify "across what categories".

Gervais' understanding of "consistently" seems to be "consistently across choices of covariates included in the model and the analysis strategies he thought up." When he wrote that "analytic thinking consistently predicts endorsement of evolution," the "consistently" there is in opposition to the factors expounded in his previous sentence. Gervais's writing did not specify this clearly.

In contrast, Dan, your understanding of "consistently" seems to be "consistently across subpopulations of society" - something it's true Gervais didn't think to include in his analysis at all. He's already admitted that he didn't think of this strategy - I considered that a concession from him.

More importantly, where does this all leave us? I think one lesson is that the default hypothesis is not always the null hypothesis. In this case, the default hypothesis is to expect that because the theory of evolution is a scientific theory, people of greater analytical skill should outperform people of worse analytical skill. (More colloquially, a chipmunk doesn't understand evolution.) This hypothesis is supported by the data representing Kentucky's non-religious population. The really interesting finding from Gervais' data is that the inability to use analytical reasoning to accept evolutionary theory is particular to Kentucky's religious undergraduates.

The association of evolution nonacceptance with religiosity in Kentucky's religious environment has a clear mechanism. Young-Earth creationist theology depends on it and demands it. It's no surprise that the kind of religiosity you see in Kentucky is a kind that is associated with the inhibition of analytic thinking about evolution, and it's sad to see this expectation confirmed.

All this suggests that it is necessary for subject educators to confront mindsets and identities toxic to the application of reason to their subject, religious or not. That is, there indeed are situations that call for educators to make students choose between what they are willing to think and learn about and who they are willing to be. The teaching of evolution to students who have espoused antiquated theologies is one such situation.

August 27, 2015 | Unregistered Commenterdypoon


But remember: there is zero (as in zilch) correlation between response a person gives to a ridiculous "belief in" evolutuion survey question and what that individual understands about evolutiionary theory or comprehends about science generally.

Remember too that it's no harder to teach a kid who says "I don't believe in" evolution the modern synthesis than it is to teach a kid who says he or she does "believe" -- unless their science teacher doesn't get this & creates the impression in the mind of the former student that the point of the class is to get him or her to say "I believe in evolution."

All this is true not only in Ky, but through out the good old USA!

The same USA where no official, high or petty, can prescribe what shall be orthodox in politics, nationalism, religion, or other matters of opinion or force citizens to confess by word or act their faith therein." West Va. Bd v. Barnette, 319 U.S. 624 (1943).

August 27, 2015 | Registered CommenterDan Kahan

I understand all those facts. Just to be sure, I'm going to repeat them back at you in my own words.

1) Measures of "analytical capability" are not associated with individual acceptance of evolution as a social being among Americans. It is important not to confuse what a person understands with what they themselves believe. (Question: Are such measures similarly dissociated for other counterintuitive but socially uncontroversial scientific theories, such as special relativity (SR)? I would not expect so. My impression of the SR kooks is that they don't actually understand what they're railing against, and couldn't write a Lorentz transformation to save their life.)

2) Educators who try to transmit their values and judgments encounter (the expected) resistance when their students do not share those values. Again, it is important not to confuse what a person understands with what they themselves believe.

3) Everyone is entitled to live by their own opinions and values in the USA. Valuing this freedom makes this nation great.

I don't see the argument you're trying to make. Could you please explain your point?

My point is simply that certain mindsets selectively preclude critical thinking in certain areas of study. I believe that in the general case, an educator charged to expand a student's capability in such an area has a duty to help that student address the personal conflicts of interest associated with growing in that area, which means you have to help them confront the choices between who they are and what they know whereever they pop up. Pretending those conflicts don't exist does people disservice when they do.

In this specific case, I see evidence that a religious mindset is the problematic one. Greater analytical capability in non-religious U-Kentucky undergrads is associated with greater acceptance of evolution, but not in religious U-Kentucky undergrads. I believe that this particular issue being religious doesn't absolve the educator of their basic duty to make the conflict clear. That we're American means that we don't expect or encourage educators to encourage students to denounce their faith when it is in conflict.

Do you really think there's something fundamentally unAmerican about this stance?

August 27, 2015 | Unregistered Commenterdypoon


It's pretty clear that one *doesn't* see the same cultural bias (in the sense that that term is used in standardized testing) for other science literacy "factual knowledge" items that as one sees for Big Bang & Evolution. That's why NSF has stopped using latter in doing across-time & across-country comparisons.

You ask what my argument is ... I thought after 1-3 you had it!

Actually, though, why wouldn't I understand this

In this specific case, I see evidence that a religious mindset is the problematic one. Greater analytical capability in non-religious U-Kentucky undergrads is associated with greater acceptance of evolution, but not in religious U-Kentucky undergrads ...

to mean that you don't accept (1) & (2) in your own restatement? "Acceptance" is not a measure of comprehension of either evolutionary science or science generally; teachers & others who don't get that interfere with the acquisition of scientific knowledge of evolution & other things among certain subcommunities? There's also no conflict between saying "I don't believe in evolution" & *using* evolutionary science to do the things that can be done properly only with it -- so teachers or others who mistake getting people to say "I believe" for equipping people *to do* things w/ what science knows are acting contrary to their own purposes.

*If* someone says that they accept all of this -- that they recognize the verbal affirmation "I believe in evolution" is nothing but a statement of identity, w/ zero connection to imparting knowledge that people can use --but still say they think it is the job of the state to get kids (ones who are learning evolutionary science, & very appropriately, being evaluated in consequential ways for their knowledge!) to mouth those words, then I'd say for sure they are Illiberal. I think that's a lot worse thing to be called than "anti-Amerikan"

My guess is that you don't "accept all of this" -- neither of your two comments reflects such "acceptance" (say it! say it! say you "accept"/"believe in" everything I wrote in last 2 paragraphs!). If you did "accept all of this," then it is hard for me to believe that you'd be disagreeing w/ me.

August 27, 2015 | Registered CommenterDan Kahan

I think the problem is that the competing hypotheses that you've set up aren't really competing. They could both be true.

First, let me translate what I take to be the competing hypotheses:

1) Gord's version of Will Gervais: Analytic thinking is the cognitive process (or set of processes) by which people typically come to reject creationism in favor of "evolution" (scare quotes because beliefs that laypersons hold about evolution should not be confused with what scientists believe about evolution).

2) Gord's version of Dan Kahan: Analytic thinking is used to protect identity related beliefs (i.e., via motivated reasoning).

These two hypotheses happen to both be encapsulated by my very own dual-process model. [how convenient!] Essentially, our cognitive architecture is such that reasoning must (at least initially) be focused on whatever first response (intuition) comes to mind. However, once analytic thinking is cued (e.g., by a conflict between intuitions), the reasoner may spend their time either a) overriding or b) rationalizing (i.e., motivated reasoning) the initial intuitive response. That is, analytic thinking is responsible for overriding and bolstering/rationalizing our intuitions.

Now, on to Will's paper. It is quite obvious that people's evolution/creation beliefs are largely cultural. It would be very, very surprising if this were not true, given what we know about belief formation. Moreover, it would be very, very surprising if evolution-related beliefs were not suspect to the same motivated reasoning processes as other types of beliefs (identity protective and otherwise). It isn't easy to override even unimportant beliefs, let alone ones that have some personal relevance. As a consequence, the primary role of analytic thinking may very well be to rationalize what we already believe. Indeed, Mercier and Sperber have argued very clearly that reasoning evolved for argumentation.

The interesting result from Will's paper is that evolution may be one of those key exceptions where analytic thinking is at least sometimes used to override a belief. I think this is important even if this is true only insofar as evolution-related-beliefs are associated with religious beliefs. Importantly, the finding that analytic thinking is at least sometimes used to override creationist beliefs does not mean that analytic thinking is not also sometimes used to rationalize evolutionary beliefs. Indeed, within the same hour one might start as a creationist, think about it analytically, become an evolutionist, and then rationalize their evolution belief... and very little of the content of that thought would probably come close to what an evolutionary biologist understands evolution to be!

August 27, 2015 | Unregistered CommenterGord Pennycook

The second part you requoted of mine is just a restatement of your own conclusion from Gervais' data. You concluded that the response of evolution acceptance to analytical capability is driven entirely by the non-religious subpopulation. There is no response in the religious subpopulation. I'm pretty sure we agree on these facts.

You must therefore have a problem with the first half - my claim that the problematic anomaly reflected by the discrepancy is morally attributable entirely to the unanalytical behavior of the religious population regarding evolution (and apparently the Big Bang as well, in contrast to SR.) Clearly we find ourselves currently on opposite sides of a fence, so let's see if I -do- agree with everything you wrote in those two paragraphs.

Upon examination, Dan, you're absolutely right that I do not agree with everything in the 2 paragraphs.

I agree that acceptance isn't a measure of competence because it's confounded by identity. For example, in the case of climate change mitigation and adaptation, the identity is rooted in the weighting of risk and action. The acceptance of human-caused climate change is unnecessary to recognize the utility of adapting to ongoing climate change. Hence, the civic case for climate change adaptation is inherently much stronger than that for mitigation. This last is something you've pointed out in previous posts as well.

However, I take particular issue with the idea that people who accept and who don't accept a theory use it similarly in making judgments and evaluating risks. When Pakistani Drs. use evolution situationally, but don't believe in it generally, I don't trust that they'll be able to extend their understanding of evolution to evaluate other potentially civically important risks, such as the antibiotic resistance risks of antibiotic overuse, or the food security risks of pesticide resistance in weeds following the widespread use of pesticides. Nor do I trust, even if they do understand, that they will be capable of advocating for civically responsible action against their own cultural impulses, if they cannot even personally accept the theory.

Said another way, it's important not to confuse a person's understanding with a person's belief, and there are situations when a person's belief does compromise their effectiveness. I don't trust a situational believer with a nevertheless complete understanding in their capacity as a citizen.

So I think end up in a slightly different place, following your same line of argument. While I agree that the government doesn't have an agenda that survives strict scrutiny, a democratically made decision to require the teaching of evolution as scientific fact survives rational basis. This form of argument generalizes: we do not trust fellow citizens arbitrarily, but only with shared and affirmed values as statements of identity. To that end, we require that all residents, many of whom will become citizens, be educated. We draw a line against requiring any affirmation of pure opinion.

Does my disagreement make more sense to you now?

For full disclosure, I am a biologist who considers evolution a fact and a process. I believe that it is unnecessary to believe in evolution, just as it is unnecessary to believe in gravity. Evolution is a process that happens whether we like it or not, whether we think of it or not, and humans have continuously dealt with and utilized its effects.

While individuals are entitled to their opinions in liberal societies, we are not entitled to arbitrary facts. The distinction between a fact and an opinion is itself a social reality, and liberal societies depend on a functioning marketplace of truth to adaptively redefine that distinction. While I may think of evolution as not even a theory, others in the public discourse consider it and attempt to discredit it as such. Neither of these opinions controls the civic discourse; presumably, we all have agendas in the remaining room.

Mine, in terms of evolution, is simply that its acceptance move beyond public controversy, for I believe it need remain there no longer, just as the concept of building American society on slavery has been definitively rejected. As long as educators respect the right of individuals to make up their minds about what they believe, they have done no wrong, and if this leads to the near-extinction of certain forms of Biblical literalism, without governmental establishment of religion, or prohibition on free exercise, the state has done no wrong. After all, as I understand the American social contract, there is no inherent right for any group to exist.

So if you think I am illiberal for attempting to advance a personal ideological agenda in the public discourse, I would counter that I am simply participating in the system as intended, and it seems to me that you have unjustifiably chosen to privilege the current state of discourse in your moral framework of liberalism. Does this also make sense?

To extend the point, amd possibly clarify: if the key issue of controversy in evolution were moral (as I believe it is for the urgency of climate change mitigation) instead of factual, I think I would be much closer to agreeing with you, if not in complete agreement. I cannot trust a person incapable of incorporating known facts into their worldview, but I will gladly share a ballot box with a person with differences of moral opinion.

August 28, 2015 | Unregistered Commenterdypoon



I'll reply "presently"-- unfortunately someone is trying to make me do actual work...

August 28, 2015 | Registered CommenterDan Kahan


There's more to say, precisely b/c you raise so many interesting points.

But let's start with this: why would you say Gervais's data support *his* hypothesis?

Forget my *rival* hypothesis (& whether it really is one; I know you mean to challenge that).

As I tried to help people *see* by showing the raw data, the relationship between CRT & belief in evolution is trivial. It doesn't support the inferences that Will purports to draw.

To avoid yet another round of the "I didn't say that!" game, let's not even paraphrase. Here is Will's hypothesis:

In sum, many scientific concepts are difficult for people to grasp intuitively while supernatural concepts may come more easily.... [But] dual-process approaches to reasoning suggest that people can often analytically inhibit or override their intuitions. Thus, I tested the primary hypothesis that performance on an analytic thinking task would predict greater endorsement of evolution.

Here is his characterization of his study findings:

.. [c]onsistent with dual process approaches to cognition in general, and supernatural cognition in particular[,] an analytic cognitive style predicts increased endorsement of evolution. Reliably developing intuitions may give creationist views an early cognitive advantage. This early advantage also is likely bolstered by early enculturation advantages for creationist, rather than evolutionary, concepts in many cultural contexts. However, individuals who are better able to analytically control their thoughts are more likely to eventually endorse evolution’s role in the diversity of life and the origin of our species.

I really do mean to say that I think the conclusion Will reaches is not consistent with his own data.

The *only* basis for Will's saying that the data do support his hypothesis is that he finds a "significant" correlation between CRT scores and acceptance of evolution.

But as you know, "significant" correlations don't furnish any information about the inferential weight of the evidence in relation to competing hypotheses. A correlation significantly different from zero is *consistent* with an infinite number of hypotheses about the "true effect size," including the "null hypothesis" itself! See, e.g., Rozeboom, W.W. The fallacy of the null-hypothesis significance test. Psychological bulletin 57, 416 (1960); Cohen, J. The Earth is Round (p < .05). Am Psychol 49, 997 - 1003 (1994).

The question is *how much more consistent" is the observed effect with one hypothesis that another.

Consider three hypotheses:

H0: There’s no difference in mean CRT scores of evolution believers & disbelievers.
H1: Evolution believers mean CRT scores are at least 0.5 greater than disbelievers.
H2: Evolution believers mean CRT scores are at least 1.0 greater than disbelievers.

What is your intuition? Which one of these hypotheses is most supported by Will’s data? And how strongly in relation to the others?

Again, I know you know that it will not do any good to remind ourselves that Will reported a positive correlation between CRT scores and believing in evolution that was different from 0 at p < 0.05. That result is in fact consistent with all three hypotheses. The question is how much more consistent it is with one versus another.

To put it that way is to suggest how one figures out the weight of the evidence in the data: by computing the likelihood ratio associated with observing the data under one hypothesis vs. each of the others.

I’m sure you understand what I’m talking about, but those who want an accessible overview, I recommend:

Goodman, S.N. Toward evidence-based medical statistics. 2: The Bayes factor. Annals of internal medicine 130, 1005-1013 (1999).

Goodman, S.N. Introduction to Bayesian methods I: measuring the strength of evidence. Clin Trials 2, 282 - 290 (2005).

I calculated these likelihood ratios, a very basic variant of the “Bayes Factors,” in relation to H0 vs. H2 & H0 vs. H1. The analysis reveals that Gervais’s data is orders of magnitude more consistent with H0—the hypothesis that there is no difference in the mean CRT scores of evolution “believers” & “disbelievers”—than with the rival hypotheses.

Specifically, Gervais’s data is 4x10^8 more consistent with the hypothesis that “believers” & “disbelievers” don’t differ in their CRT scores than that the former score even 1 point higher, and 4x10^4 more consistent with the hypothesis that believers score even half a point higher on average!

Why would this surprise anybody? There is ample research already that shows that what people say in response to the public-opinion polling questions on evolution featured in Will’s study doesn’t measure anything related to people’s facility with or comprehension of science.

I know you know that bad psychology studies, bad economic studies, and bad political science studies routinely treat being able to “reject the null” at “p < 0.05” as license for making super strong claims that aren’t genuinely supported by the reported data.

Will’s not a bad psychologist, though, nor are you. I think he should address the question of the weight of the data in relation to his hypotheses and plausible rivals rather than just report that he has succeeded in “rejecting the null at p<0.05,” an accomplishment that is consistent with many alternatives to his own hypothesis.

So as I said, here’s a start. If you want to discuss whether there’s something I’m getting wrong here, let me know.

Once we settle up with that, I’ll address the question whether my position is “consistent” with Will’s or, if not, whether there is anything in his data that helps us to assess which is more likely to be true.

August 29, 2015 | Registered CommenterDan Kahan

You make an excellent point. I don't have a problem with you saying (essentially): "this is the effect size required for me to care". That is certainly something that we should all consider. It's more difficult than taking a more black-and-white effect/no-effect stance, but keeps us from chasing unimportant or inconsequential findings.

I should say, though, that I (and presumably Will & the fine folks over at Cognition) find any consistent association between CRT and evolution-belief to be interesting. And I'm unapologetic about it. As you know, even a relatively small effect (assuming it's replicable) may be of some consequence. For example, in your NCC paper, the correlation between science literacy/numeracy and concern about climate change risks was only .08 for egalitarian communitarians and -.12 for hierarchical individualists. [I bring this up not as a weak attempt at a "gotcha", but because I distinctly remember thinking about these very issues when I first read your paper. "Not a huge effect, but very cool and quite important" is what I probably mumbled to myself.]

I'm washing over a lot of your content. Regardless, I think it would be an amicable compromise to say that the key difference between yourself and Will (and, I guess, me... although I don't want to give the impression that I have a strong opinion on the matter) is that reflective reasoning would have to be *more* influential for such a finding to modify anything you believe about how and why people do and don't believe in evolution. Perhaps that's the right stance, but I'm of the opinion that results such as this need to be published for people to make that sort of decision for themselves.

Also, since I'm here, 0.5-1 point increase on the CRT may seem small... but it really isn't. It's only a 3 item measure, so that represents a 16.7-33.3% increase in performance. That seems pretty big to me. In Frederick's original paper, the difference between Harvard students (1.43) and an set of online samples (1.10) was less than 0.5. The highest mean score that Frederick reports is MIT, and it's only 2.18/3. As you've blogged about in the past, the CRT is a really difficult measure (as one would expect if it truly taps into our innate cognitive miserliness). A whole point difference between evolutionists and creationists would be huge!

To reiterate, I don't have a problem with you requiring a larger effect/more evidence to care. It's just that I also don't have a problem with someone getting excited about and/or publishing a small effect... so long as they aren't speaking above the data, which Will isn't.

August 29, 2015 | Unregistered CommenterGord Pennycook


Excellent! We are making progress.

Again, will respond "presently"!

August 30, 2015 | Registered CommenterDan Kahan


Okay, this is great! The issue of how to assess the weight of the evidence that study data furnishes in relation to study hypotheses is a super interesting & hard question. You are pushing me to be clearer about why I don’t think the weight of the evidence in Will’s study supports his hypothesis as much as it does a genuine rival to it—and you are making me think more clearly about this.


I do want to preface my answer with one important point: I 100% don’t think Will’s paper "shouldn’t have been published” or that it wasn’t “good” etc.

On the contrary, as this exchange illustrates, I think it’s a *good* study precisely b/c it helps to sharpen scholarly engagement with a difficult and important question.

I just happen to think Will’s wrong. And I want to explain now why I disagree w/ your initial comment that the inference Will draws from his data and the one I draw from it can happily co-exist (although I do really like your conception of dual process reasoning; it seems right to me).

But if you think I’m blowing it here, tell me! I will be as grateful to you for telling me why as I was to you for posting your last comment.


You agree that when one is testing rival hypotheses, one should indicate the effect sizes they predict. Then we can assess the data to see how much more consistent they are with one hypothesis than the other.

What sort of effect size does Will’s theory—that disbelief in evolution will yield as people become more proficient in cognitive reflection--predict?

There are a number of ways we could try to figure this out. One very conventional way to do it is to see how much better we do in predicting whehther people believe in evolution once we know there CRT scores.

If we have nothing else to go on, we should predict that everyone in Will’s sample disbelieves in evolution. That’s becasues in fact only 31% (in Study 1; the % was even lower in Study 2!) overall indicated acceptance of evolution uninfluenced by divine agency. Accordingly, predicting “everyone disbelieves” will give us a 621% accuracy rate. For every person we predict “believes in” evolution, there is a 62% chance we’ll be wrong.

If “an analytic cognitive style predicts increased endorsement of evolution”—Will’s interpretation of his data—then we should be able to do better than 62% accuracy once we know people’s CRT scores, right? It’s a modest prediction, but let’s say that knowing subjects’ CRT scores will boost our predictive accuracy to 70%.

Sound fair?

Well, by that test, Will’s hypothesis—his theory—is less supported than the only alternative he addresses-- the “null/nill” hypothesis.

That can be confirmed by looking at the “count R2” of the model—the % of subjects it classifies correctly—which is 62%.

It can be confirmed too by looking at the“adjusted count R2” for the model. The adjusted count R2 is the proportion of cases we’d misclassify if we predicted everyone was a “nonbeliever,” the modal outcome. The adjusted count R2 is 0.00%.

In sum, even when we one uses a model that treats CRT as a predictor of “belief in” evolution unguided by human agency, the “statistically significant” CRT predictor matters so little that that the model throws up its hands and just predicts “everyone disbelieves!”

So in fact, I would say Will is “speaking above the data here,” to use your phrase, when he says things like, "analytic thinking consistently predicts endorsement of evolution,” “individuals who are better able to analytically control their thoughts are more likely to eventually endorse evolution’s role in the diversity of life and the origin of our species,” his “results suggest that it does not take a great deal of analytic thinking to overcome creationist intuitions” etc.

Another & in my view better way to think about “practical effect” here is go back to the Bayesian strategy I used in my last comment: what inference is more supported by the data -- that evolution "believers" are "more reflective" than evolution "nonbelievers" or that there's no meaningful difference?

How big a difference in CRT should we expect to see between non-believers and believers in order to believe that there is a meaningful difference between them?
You suggest that a 0.50 difference in the mean CRT for believers & nonbelievers would be really big—more than big enough to “count” as practically significant.

But how should we assess that? I don't think just dividing the number "3" or "4"-- the number of categories on the CRT -- by some amount & using intuition is really the way to do it. Instead, let's use what we know about the psychometric properties of the CRT!

Remember, the CRT has *no* discriminatory power—cannot measure differences in cognitive reflection—for 60% of the population. That’s about the percentage people in a general population sample who get 0 answers correct!

In Will’s study 1 Univ. of Ky sample, 66% got 0.

If we use < Item Response Theory to assess the measurement precision of CRT, we find that the test doesn’t give us any real power to discriminate in individuals’ levels of cognitive reflection until their scores get to +1 SD or above—a mean of about 1.4 answers correct.

In fact, you and I have in fact talked about this problem before. Your position—one that might well be right; I know additional smart people who agree with yout-- is that there’s probably just no meaningful differences in reflection to measure in the majority of the population.

In Will’s sample, the mean CRTs for nonbelievers & believers in evolution was 0.55 and 0.74, respectively. . . .

This reported difference in the means for people who score that far below +1 SD falls far below the power of CRT to measure differences in cognitive reflection!

Even a 1/2 point difference wouldn’t do it here.

Will didn’t specify either before or after reporting his results what a meaningful effect size would be. But if he had *predicted* 0.19*-- an amount not nearly high enough to put “believers” in the range in which CRT can actually measure differences in cogntive reflection – I thnk it would have been fair to say that that’s not a big enough effect to support his theory!

You state that the effect size in a study my collaborators & did – one that showed that higher proficience in science comprehension magnifies cultural polarization on climate change – might be questioned on similar grounds.

Very fair to raise this point! I wish I had anticipated it & addressed it squarely when we wrote the paper.

I’m sure you agree the size that counts as practically meaningful can’t be defined independently of what the competing hypotheses are.

As that paper explained, the “public irrationality thesis” hypothesized that polarization would decrease by some meaningful degree as science comprehension increased. Accordingly, a finding that it in fact increased---even if by a small amount—counted as a very strong evidence against that theory!

ERT—the “expressive rationality thesis”—predicted that polarization would increase. Necessarily, the evidence was much more consistent with that hypothesis then!

How much more? Well, let’s do what I did with Will’s data and calculate some likelihood ratios.

You cited some correlation coefficents in your comment, but I suspect you’d agree that such coefficients don’t have a lot of meaning in the abstract.

How about we use “standard deviations” in the outcome variable—perceived risk of global warming—as our effect size currency?


H_PIT: Polarization in risk perception will decrease by 0.5 standard deviations as science comprehension increases.
H_ERT: Polarization in risk perception will increase by 0.5 standard deviations as science comprehension increases.

If we view those as reasonable predictions for the rival theories(PIT & ERT), it turns out the evidence was approximately 8,000x more consistent with H_ERT than H_PIT! See for yourself!

*Now* let’s go back to Will’s data, so I can explain to you why I think my “hypothesis” about his data and his are not compatible.

Again, Will surmises that the reason so high a % of public opinion poll rspts say they don’t “believe in” evolution is that “many scientific concepts are difficult for people to grasp intuitively while supernatural concepts may come more easily.” But because “dual-process approaches to reasoning suggest that people can often analytically inhibit or override their intuitions,” he hypothesized that “performance on an analytic thinking task would predict greater endorsement of evolution.”

That theory doesn’t predict only *certain* kinds of people will become more likely to believe in evolution if their CRT scores increases. If only certain kinds did, then we’d have reason to think that the theory—that rejection of evolution is a consequence of how hard it is for those who cannot “inhibit their intutions” to “grasp” such “difficult scientific concepts” as natural selection—isn’t right.

Indeed, the point is actually much stronger than this!

As you note & he points out, Will recognizes that “culture,” of the sort that manifests itself in religiosity, counts too. People who are nonreligious are disposed to say they “believe in” evolution; people who are religious are predisposed to say they do “not believe” (although, for the billionth time, none of this correlates w/ knowledge of anything having to do with how evolution works or with science generally. . .).

Well, if Will’s *theory* is right, then those those who are most disposed by religiosity to “disbeleive” in evolution ought to be the ones whose are *most influenced* to accept evolution as their CRT scores improve: their beliefs are much more susceptible of changing in the predicted direction than are nonreligious people, who already tend to say they “believe in” evolution!

Accordingly, we ought to see less division among religious and nonreligious types as CRT increases under Will’s theory.

Sound okay?

I actually would not expect that to happen, though, b/c I have a *different theory* of why people say they do or do not “believe in” evolution. That heory is the “expressive rationality thesis” (ERT)

For the vast majority of the population “beliefs” in evolution don’t convey any meaningful view about the “scientific evidence” of the natural history of humans. Instead they are *attitudes* that express their identity. Because people generally can be expected to use all of their cognitive resources to form attitudes that express their identities, their *cultural* predispositions on evolution will be reinforced as CRT goes up.

So consider:

H_Gervais: Polarization in evolution belief will decrease 20 pct points as CRT increases from 0 to 3.
H_ERT: Polarization in evolution belief will increase by 20 pct points as CRT increases from 0 to 3.

Again, if compute the likelihod ratios associated with these hypotheses, we find that the data is much more consistent with H_ERT and H_Gervais.

Indeed, it is over 100x more consistent with the former than the latter.

Again, though, that doesn't make Will's paper bad or even mean that anyone, on the basis of what I've said, has to reject his theory & adopt mine etc.

As I've said form the start, & as you recognized implicitly in your proposed "amicable compromise" (to me, this is amicable, even w/o compromise on anyone's part!), what one "believes" if one is engaging the world empirically is always just one's best understanding of the weight of the currently available evidence. There are other pieces of evidence out there; and more might be forthcoming.

But we won't make any progress in this enterprise if we don't figure out how to *weigh* that evidence. I know we all agree that just being able to "reject the null at p< 0.05" doesn't help us to do that at all!

Got a better way that the two I proposed here? If so, tell me what I'm missing!

August 30, 2015 | Registered CommenterDan Kahan

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>