follow CCP

Recent blog entries
popular papers

What Is the "Science of Science Communication"?

Climate-Science Communication and the Measurement Problem

Ideology, Motivated Cognition, and Cognitive Reflection: An Experimental Study

'Ideology' or 'Situation Sense'? An Experimental Investigation of Motivated Reasoning and Professional Judgment

A Risky Science Communication Environment for Vaccines

Motivated Numeracy and Enlightened Self-Government

Ideology, Motivated Cognition, and Cognitive Reflection: An Experimental Study

Making Climate Science Communication Evidence-based—All the Way Down 

Neutral Principles, Motivated Cognition, and Some Problems for Constitutional Law 

Cultural Cognition of Scientific Consensus

The Tragedy of the Risk-Perception Commons: Science Literacy and Climate Change

"They Saw a Protest": Cognitive Illiberalism and the Speech-Conduct Distinction 

Geoengineering and the Science Communication Environment: a Cross-Cultural Experiment

Fixing the Communications Failure

Why We Are Poles Apart on Climate Change

The Cognitively Illiberal State 

Who Fears the HPV Vaccine, Who Doesn't, and Why? An Experimental Study

Cultural Cognition of the Risks and Benefits of Nanotechnology

Whose Eyes Are You Going to Believe? An Empirical Examination of Scott v. Harris

Cultural Cognition and Public Policy

Culture, Cognition, and Consent: Who Perceives What, and Why, in "Acquaintance Rape" Cases

Culture and Identity-Protective Cognition: Explaining the White Male Effect

Fear of Democracy: A Cultural Evaluation of Sunstein on Risk

Cultural Cognition as a Conception of the Cultural Theory of Risk

« I ♥ Item Response Theory -- and you can too! | Main | Cross-cultural cultural cognition road trip: Australia »

Don't even think of going there: the "whose is bigger" question on climate science literacy

A curious correspondent posed these questions to me relating to scores on the "ordinary climate science intelligence" assessment:

My question is about the last figure in your posting here on your OCSI instrument and results.

The last figure is a historgram of the No. correct (on your OCSI instrument?) personal beliefs about warming causes (human, natural, no warming).

I have several questions:

1. INTERPRETATION of final figure. Am I interpreting your result correctly by concluding that it shows that you found that those believing in no warming had more correct than those who believed in natural causes of warming, who, in turn, scored higher than those who believed in human caused warming?

I am just asking about the absolute differences, not their statistical significance.

2. SAMPLE. How big was it and who were they? (undergrads, Mechanical Turk, something else, national representative...).

3. STATS. Were the differences in that final figure significant? And, regardless of significance, can you send along the effect sizes?

My responses:

You can get more information on the OCSI scale here: "Climate Science Communication and the Measurement Problem," Advances in Pol. Psych. (forthcoming).  But on your queries:

1. Interpretation. The last figure is a bar chart w/ number of correct for rspts who answered standard "belief in" climate change items that asked "[f]rom what you’ve read and heard, is there solid evidence that the average temperature on earth has been getting warmer over the past few decades" [yes/no]; and (if yes), "Do you believe that the earth is getting warmer (a) mostly because of human activity such as burning fossil fuels or (b) mostly because of natural patterns in the earth’s environment?"

You are eyeballing the differences in mean scores for the 3 groups-- "no warming," "naturally caused warming" and "human warming."  

But my interpretation would be that everyone did about the same.  Among all respondents -- regardless of the answer they gave to "believe in" global warming items -- there was a strong tendency  to attribute to climate scientists pretty much any conclusion that *sounded* consistent with  global warming being serious environmental risk.  Only respondents who were high in science comprehension generally avoided that mistake -- that is, identified accurately which "high risk" conclusions climate scientists have endorsed & which ones not.  Those rspts successfully did that regardless of how they answered the "believe in" question.  

That's why I think the responses members of the public give to surveys that ask whether they "believe in" human-caused global warming are eliciting an expression of an outlook or attitude that is wholly unrelated to what anyone knows or doesn't know about climate science or science generally.  Social scientists (myself included) and pollsters haven’t really understood in the past what items like this are actually measuring:  not what you know, but who you are.

2. Sample. US general population sample.  Stratified for national representativeness.  Recruited for on-line study by the firm YouGov, which uses sampling strategies shown to generate election result estimates at least as reliable as those generated by the major polling firms that still use random-digit dial (I'm basing this on Nate Silver's rankings).  In my view, only YG & GfK use on-line sampling techniques that are valid for studying the effect of individual differences -- cognitive & political -- on risk perceptions.  Mturk is definitely not valid for this form of research.

3. Stats. The diff between "no warming" & "human-caused warming" rspts was significant statistically -- but not practically. N = 2000 so even small differences will be statistically significant.  The difference in the mean scores of those 2 groups of rspts was a whopping 1/3 of 1 SD.  Whether respts were in "no warming," "human cauased warming" or "natural warming" classes explained about 1% of the variance in the the OCSI scores:

I reported "number of correct" in the figure b/c I figured that would be easier for readers to grasp but I scored results of the climate science literacy test with an IRT model and standardized the scores (so mean = 0, of course).  In regression output, belief in "human warming" is the reference group--so their score is actually the constant. 

The constant & the regression coefficients are thus the fractions of a standard deviation below or above average the different groups' performances were!

You can easily compute the means: human warmers = -0.12; natural warmers is 0.07; and no warmers 0.14.

It would be just as embarrassing --just as childish -- for "skeptics" to seize on these results as evidence that skeptics "know more" climate science as it would be for "believers" to keep insisting that a knowledge disparity explains the conflict over climate change in US society.

So don't go there, pls...

But if you have thoughts, reactions, comments, suggestions, disagreements, etc. -- particularly based on analyses as they appear in draft paper -- please do share them w/ me.

PrintView Printer Friendly Version

EmailEmail Article to Friend

Reader Comments (7)


First, I want to say that I consider your even-handed and thoughtful approach to these
issues is a huge breath of fresh air, compared to some of the politicized social science out there.

Second, though, I have to disagree about how to interpret your results. I am going to "go there."
1/3SD is a Cohen's d of .33, which corresponds to an r of about .16. The average effects in social
psychology are only about r=.20, so this is right there.

Furthermore, your effect size is as large or larger than many other effects in social psychology
that have been characterized by our colleagues as powerful, dramatic, or "default," such as stereotype threat,
self-fulfilling prophecies, the role of stereotypes in person perception, and the gender
difference in math SAT scores.

To be sure, it is true that one cannot conclude anything too general.
One cannot, on the basis of your results, declare:
"Skeptics are smarter than believers"
"Skeptics know more science than do believers"
"Skeptics are less biased about science than are believers."

But skeptics score significantly higher (r=.16) than believers on your OCSI
test of scientific knowledge about environmental issues.
That seems to me to mean that ... skeptics score higher than believers.

The importance of this difference is a matter of subjective professional judgment.

But the difference seems to me to be there in your data. It seems to me justified for
someone to look at your data and conclude "Skeptics scored higher on environmental
scientific knowledge than believers." Put differently, it seems reasonable to "go there"
based on your data -- long as you go to there reflected in your data
(they did better on your OCSI) and not somewhere else ("this vindicates conservatives writ large").



August 12, 2014 | Unregistered CommenterLee Jussim

Dan Kahan,

If your data demonstrates that "skeptics scored higher on the OCSI than AGW believers", why on earth are you not willing to say that? (go there)??? Why are you NOT willing to admit what your own research demonstrates?

"It would be just as embarrassing --just as childish -- for "skeptics" to seize on these results as evidence that skeptics "know more" climate science as it would be for "believers" to keep insisting that a knowledge disparity explains the conflict over climate change in US society."

Would it be as embarrassing and childish as ignoring/denying the obvious, or make false comparisons about it?

Your research SHOWS that skeptics who took the OCSI assessment DID KNOW MORE basic climate science than the "believers" did. Period. So how would it be embarrassing or childish to actually point that out? It might be embarrassing for YOU, but it wouldn't be embarrassing for them at all.

But it WOULD be embarrassing for believers to keep insisting on a knowledge disparity explains the conflict because your research shows that shows that they score on the lower end of any disparity there might be.
That WOULD be embarrassing for them

August 22, 2014 | Unregistered CommenterSaudadia


I have given you data. Go wherever you like with it.

to me it is obvious that anyone who gets into a debate over "who knows more" based on this data will make a fool of him- or herself. The median "skeptic" thinks that green house gas emissions reduce photosynthesis; we are supposed to think that person knows anything about climate change? Be my guest: go there.

There's no correlation between what people know & what they believe. Variance in knowledge is not the explanation for conflict over climate change.

August 22, 2014 | Unregistered Commenterdmk38


Please forgive any perceived snarkiness on my part. I can be logical to the point of bluntness sometimes and blunt can come across poorly online.

"The median "skeptic" thinks that green house gas emissions reduce photosynthesis; we are supposed to think that person knows anything about climate change"

I'm an intense information seeker as well. How was the question phrased? Which green house gas was being talked about? Was the amount of a specific green house gas discussed? I ask because for example, too little CO2, as well as too much CO2, CAN and DOES restrict photosynthesis in plants.

And how does the fact that someone might not understand plant photosynthesis equate to "knows notihng about climate change?" What did median "believers" get wrong in order to categorize them as median? Based on their median status, we are supposed to think that person knows nothing about climate change. Correct?

"There's no correlation between what people know & what they believe. Variance in knowledge is not the explanation for conflict over climate change."

I'm not saying that it is. But your study has proven something that will frustrate "believers" and validate "skeptics" to some degree. People are people.

But how have you proven that what the people taking the test "know" does not affect or relate to what they believe? Did you ask them? For example, let's say that I KNOW how CO2 reacts to long wave radiation in scientific terms and everything. And BECAUSE I know how CO2 reacts to long wave radiation, I don't believe that it can be the sole or even major driver of atmospheric temperatures, much less ocean temperatures. In this case, what I KNOW absolutely correlates to what I believe.

August 22, 2014 | Unregistered CommenterSaudadia


No worries!

The wording of the items & the results for general population sample to whom it was administered (including how those results varied in relation to political outlooks, scores on a general science comprehension assessment, and "beliefs in" AGA) are reported in The Measurement Problem paper. The section that describes derivation of the items & the strategy used to try to avoid confounding of the scale w/ measures of ideology & affective orientations toward "belief" & "skepticism" appeares in this post.

The *mean* believer, *mean* nonbeliever, & *mean* "naturalist" (whatever we might call him or her) all do no better on the test than would a person flipping a coin. I suspect the "significance" of the differences in mean score are about what you'd expect if you flipped a coin 2000 times. In any case, the "smartest" group is still no "smarter" than a coin -- hence my "don't go there," which is a warnign designed to save people from the pain of having this pointed out!

The scores of those highest in science comprehension did *much* better than a coin! But there scores also did not correlate w/ "believing in" climate change to any meaningful degree. You'll be able to see that in the paper.

August 22, 2014 | Registered CommenterDan Kahan

"The smartest group is still no smarter than a coin"

"The scores of those highest in science comprehension did much better than a coin."

Either you just contradicted yourself, or I'm not grasping some kind of qualifier you are using to differentiate here.

In my mind, the "group with the highest science comprehension" would by default be the "smartest group" no matter what combination of believers, nonbelievers, naturalists etc might be in that group.

"The *mean* believer, *mean* nonbeliever, & *mean* "naturalist" (whatever we might call him or her) all do no better on the test than would a person flipping a coin."

I understand the comparison to an extent, but it also troubles me. The result of a coin flip does not involve a decision or a cognitive choice being made. It's simply random choice. But in a true false test, while any or every answer COULD be the result of a random choice made by the testee, without any evidence to support that behavior, you should view as most likely being based on actual choice/cognitive decisions. But why would you design the test so that it's average/mean results were at least more determinate than a coin flip would be? That certainly limits the extent to which the results can be logically applied doesn't it?

Again, you seem to have a problem with the idea that people can be extremely well versed in scientific knowledge....things that are true or false based on scientific laws and principles, AND still not believe in the AGW theory for perfectly reasonable, rational reasons that have less to NOTHING to do with their political affiliation or ideology! That is amazing to me! How can someone who studies human behaviors as a profession not allow for that possibility to be one of the options?

The only thing I can think of is that your OWN biases, BELIEFS are preventing you from seeing or accepting that very real possibility. Your posts are filled with insinuations of cognitive dissonance and frustration that could logically be resolved if you allowed such an option to even exist.

Do you KNOW for yourself based upon your scientific knowledge in all areas that are encompassed in the field of climate science that the AGW theory is true? Have you done a thorough examination of ALL of the science from all publishing scientists and determined for yourself that they think the theory is true? Or do you just BELIEVE that the science regarding climate change/global warming is settled based upon your own (or others) opinions, ideologies, affiliations,conclusions?

Your own ideologies and beliefs and cultural cognitions have to be as completely removed from your research as possible or you'll never get to the real truth.

August 22, 2014 | Unregistered CommenterSaudadia


1. The contradiction:

The *mean* believer, *mean* nonbeliever, & *mean* "naturalist" (whatever we might call him or her) all do no better on the test than would a person flipping a coin. I suspect the "significance" of the differences in mean score are about what you'd expect if you flipped a coin 2000 times. In any case, the "smartest" group is still no "smarter" than a coin...

The groups being compared here are the 3 I just mentioned -- believers, nonbelievers, & naturalists. None does any better on avg than a coin toss.

The scores of those highest in science comprehension did *much* better than a coin!

"Highest in science comprehension" here refers to subjects generally -- and contrasts with those who didn't score as highly. High scoring individuals can be "believers," "nonbelivers," or "naturalists"-- those groups aren't part of this comparison.

Does that help? There's also the paper, where things are spelled out in detail (and w/ more care).

2. Why design a *valid* test where average score is 50%? Why not? If it is much higher than that then variance will be compressed at the top & it will be difficult to sort out who knows more etc. If much less, then it will for the same reason be difficult to sort out people who are middling from people who are low.

It's actually takes effort & technical skill to create a *valid* test where mean is 50%. Since guessing on true-false results in 50%, the mean can be 50% on a valid test only if people who don't know answers are *more likely than not* to think the wrong answers are right. I assume you can see why.

I keep saying *valid* b/c the validity of the test is something that has to be established indpendently of what fraction of the rspts get the right answer.

3. I don't have any problem w/ anything really. And no, of course, I don't know based on my own understanding of the science involved whether the prevailing scientific views on AGW are correct. What I find puzzling is that people who display knowledge that implies a certain conclusion -- that human beings are causing global warming -- deny that conclusion. And that what predicts that is cultural identity. But if you can see in the evidence some alternative interpretation that shows I'm being dense, good! I'd rather have the data admit of some interesting insight than that the insight be the one I think it supports.

August 23, 2014 | Registered CommenterDan Kahan

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>