Unconfounding knowledge from cultural identity--as big a challenge for measuring the climate-science literacy of middle schoolers as grown ups
A friend (of the best sort—one who has “got your back” to protect you from entropy’s diabolical plan to deprive you of the benefits of advances in collective knowledge) sent me a very interesting new study:
Stevenson, K. T., Peterson, M. N., Bondell, H. D., Moore, S. E., & Carrier, S. J., Overcoming skepticism with education: interacting influences of worldview and climate change knowledge on perceived climate change risk among adolescents, Climatic Change, 126(3-4), 293-304 (2014).
I very much like the SPBMC paper.
One cool thing about it is that it tests the influence of cultural predispositions on the global-warming beliefs of middle schoolers. It’s not the only study that has adapted the cultural cognition worldview measures to students, but it’s one of only a few and the only one I know of that is applying the measures to kids this young.
Consistent with research involving adult subjects, SPBMC find that cultural outlooks—in particular “individualism”—predicts skepticism about climate change.
SPBMC decided not to use (or at least not to report results involving) the hierarchy-egalitarianism worldview measure (maybe they figured some of the items weren’t suited for minors; I could understand that).
Instead they used a “social dominance” one and found that it didn’t predict anything relating to climate change attitudes—also interesting.
But of course the most important & interesting thing is what SPBMC have to say about the relationship between climate-literacy & acceptance/belief in human-caused global warming, & the influence of cultural individualism on the same.
I found this part of the paper extremely valuable & informative. I have a strong feeling that they have mined only a portion of the rich deposits of knowledge that their data contain.
Nevertheless, I found myself unconvinced (at least at this point) that the results they reported had the significance that they attached to them.
SPBMC present two principal findings. One is that acceptance of human-caused climate change in their student sample was associated with higher climate-science literacy.
The other is that climate-science literacy had a bigger impact on kids who were relatively individualistic. That is, as those kids display higher levels of climate science literacy, the change in the probability that they will believe in human-caused climate change increases even more than it does in kids who are relatively “communitarian” as their science-literacy levels increase.
SPBMC infer from these findings that “[c]climate literacy efforts designed for adolescents may represent a critical strategy to overcoming climate change related challenges, given stable or declining concern among adults that is driven in part by entrenched worldviews.”
For adults, worldviews are well entrenched and exert considerable influence over climate change risk perception. During the teenage years, however, worldviews are still forming, and this plasticity may explain why climate change knowledge overcomes skepticism among individualist adolescents . . . .
I myself strongly agree with SPBMC that climate-science education can make a big contribution to overcoming cultural polarization on climate change—although for reasons that I think differ from those of SPBMC. But put that aside for a second.
The problem, in my view, is that the measure of climate-science literacy that SPBMC constructed fails to address what existing research teaches us is the biggest challenge in measuring public understanding of climate science.
That challenge is how to unconfound or disentangle genuine knowledge from the positions people take by virtue of their cultural identity. An assessment instrument must overcome this challenge in order to be a valid measure of climate-science literacy.
In general, people’s perceptions of risk reflect affective appraisals—positive or negative—of the putative risk source (nuclear power, guns, vaccines, etc.).
For most people most of the time, these feelings don’t reflect their comprehension of scientific data or the like. On the contrary, how people feel is more likely to shape their assessments of all manner of information, which they can be expected to conform to their pro- or con-attitude toward the putative risk source.
In this circumstance, survey items that elicit people’s understandings of the risks and benfits associated with some activity or state of affairs are best understood as simply indicators of the unobserved or latent affective orientation that people have toward that activity or state of affairs. That attitude is all they are genuinely measuring (Loewenstein et al. 2001; Slovic et al. 2004).
This is a huge issue for measuring climate-science literacy.
Sadly, propositions of fact on climate change—like whether it is happening & whether humans are causing it—have become entangled in antagonistic cultural meanings, transforming them into badges of membership & loyalty to affinity groups of immense significance in people’s everyday lives.
Study respondents can thus be expected to answer questions relating to climate change in a manner that reflects the pro- or con- affective stance that corresponds to their cultural identities.
If they are the sort of persons who are culturally predisposed to believe in human-caused global warming (or “accept” it; let’s be sure to avoid the confused & confusing idea that there’s an important distinction between “believing” something & “accepting” or “knowing” it), they will affirm pretty much any proposition that to them sounds like the sort of thing one who “believes in” climate change would say.
As a result, they’ll incorrectly agree that human-caused global warming will increase the incidence of skin cancer, that industrial sulfur pollutions are causing climate change, that water vapor traps more heat than any other greenhouse gas etc.
Their “acceptance” of human-caused global warming, in other words, doesn’t reflect knowledge of the basic mechanisms that drive climate change or of the scientific evidence for how they work.
Study after study after study after study has demonstrated this (Bostrom et al 1994 Reynolds et al. 2010; Tobler, Visschers & Siegrist 2012; Guy et al. 2014).
To be valid, then, a climate-science literacy scale must successfully distinguish between respondents whose correct answers reflect only their identity-based affective orientation toward global warming from those whose correct answers show genuine climate-science comprehension.
The only way to design such a scale is to include a sufficiently large number of appropriately weighted items for which the incorrect answers are likely to seem correct to someone who is culturally predisposed to believe in climate change but who lacks understanding of the scientific basis for that position.
Now here’s the most interesting thing: if one includes a mix of items that successfully distinguishes those who “accept” human-caused climate change based on their predispositions from those who genuinely get the mechanisms of climate change, then one will discover that those who don’t “accept” or “believe in” human-caused climate change know just as much about the mechanisms of climate change as those who say they do accept it.
For sure, most “skeptics” are painfully ignorant about climate change science.
But that’s true for most “believers” too!
Only a very small portion of the general public—consisting of individuals who score very high on a general science comprehension test—can consistently distinguish propositions that most expert climate scientists accept from propositions that sound like ones such experts might accept but that in fact are wholly out of keeping with the basic mechanisms and dynamics of global warming.
Yet even among these very climate-science literate members of the public, there is no consensus on whether global warming is occurring. Just like their climate-science illiterate counterparts, their “beliefs” about human-caused global warming are predicted by their cultural identities (Kahan in press).
In sum, “acceptance” or “belief in” human caused global warming is not a valid indicator in them either. It is an indicator of who one is, culturally speaking—nothing more and nothing less.
Judging from the results they reported in their paper, at least, SPBMC did not construct a climate-science literacy measure geared to avoiding the “identity-knowledge” confound.
In fact, they actually selected from a larger battery of items (Tobler, Visschers & Siegrist 2012) a subset skewed toward ones that a test-taker who is culturally predisposed to “believe in” human-caused global warming could be expected to answer correctly regardless of how much or little that person actually knows about the mechanisms of climate change (e.g., “For the next few decades, the majority of climate scientists expect a warmer climate to increase the melting of polar ice, which will lead to an overall rise of the sea level ”; “... an increase in extreme events, such as droughts, floods, and storms”; “... a cooling down of the climate”; “The decade from 2000 to 2009 was warmer than any other decade since 1850.”).
SPBMC left out of their battery items that Tobler et al. (2012) and other studies have found believers in climate change are highly likely to get wrong (e.g., “For the next few decades, the majority of climate scientists expect an increasing amount of CO2 risks will cause more UV radiation and therefore a larger risk for skin cancer”; “Water vapor is a greenhouse gas”; “In a nuclear power plant, CO2 is emitted during the electricity production”; “On short-haul flights (e.g., within Europe) the average CO2 emission per person and kilometer is lower than on long-haul flights (e.g., Europe to America).”)
By my count, only 3 of the 17-19 items SPBMC identify as ones included in their scale (there is a discrepancy in the number that they report using in the text and number that appear in the on-line supplementary information, where the item-wording appears) are ones that existing studies have shown were likely to elicit wrong answers from low climate-science comprehending respondents who are nonetheless culturally predisposed to believe in climate change (“the ozone hole is the main cause of the greenhouse effect [true-false]”; “For the next few decades, the majority of climate scientists expect a precipitation increase in every region worldwide”; “Carbon dioxide (CO2) is harmful to plants”).
If one constructs a “climate science literacy” scale like this, it is bound to correlate with “acceptance” of global warming because the scale will itself be measuring the same cultural predisposition that inclines people to accept human-caused global warming.
Indeed, included in the SPBMC scale were true-false items that measured acceptance of human-caused climate change:
- The increase of greenhouse gasses is mainly caused by human activities.
- With a high probability, the increase of carbon dioxide (CO2) is the main cause of climate change.
- Climate change is mainly caused by natural variations (such as changes in solar radiation and volcanic eruptions).
Obviously, if one is testing the hypothesis that acceptance/belief in human-caused global warming is caused by understanding of climate science, then the former must be defined independently of the latter.
Because SPBMC put "acceptance" items in their climate literacy scale, their finding that global-warming acceptance is associated with climate-science literacy is circular.
The same problem, in my view, characterizes SPBMC’s finding on the relative impact of climate-science literacy on students who are relatively individualistic.
Again, SPBMC’s climate-science measure is itself measuring acceptance of human-caused climate change.
So for them to say (based on a correlational model) that “climate science literacy” has a bigger impact on individualists' willingness to “accept climate change” than it does on communitarians’ is equivalent (mathematically/logically) to saying: “Reducing climate-skepticism in cultural individualists who don't believe in climate change would have a bigger impact on their willingness to accept human-caused climate change than would reducing the skepticism of cultural communitarians who already believe in climate change. . . .”
Can’t argue with that—but only because it’s essentially a tautology.
The practical question has always been why individualists are so strongly predisposed to skepticism (and communitarians to belief—same thing).
There is already evidence that the cultural individualists who score highest on a valid climate-science literacy scale are not more likely than low-scoring cultural individualists to say they accept/believe in human-caused global warming.
Because it is unclear that SPBMC constructed a scale that measures knowledge & not just a pro-belief affective orientation—indeed, because their climate-science comprehension scale includes acceptance of human-caused climate change—it doesn’t support any inference that greater climate-science comprehension would have such an effect in culturally individualistic middle schoolers.
As I mentioned, I do believe that improving climate-science education would make a very big contribution to dissipating political polarization on global warming.
The reason isn’t that understanding climate-science in itself can be expected to induce people to say they “believe in” climate change. Again, what people say about what they believe in climate change isn’t a measure of what they know; it is a measure of why they are.
But precisely for that reason, learning to teach kids climate science will require teachers to learn how to dispel from the classroom the toxic affiliation between climate change positions and identities that now divides adults in the political realm. When teachers learn how to do that—as I’m confident they will—then we can apply those lessons more broadly to the political domain so that there too we can use what we know rather than fight over whose side the state is going to take in a mean, illiberal status competition.
Indeed, that SPBMC performed a study like this in an educational context fills me with deep admiration. This is the sort of research we desperately need more of, in my view.
And notwithstanding the critique I’m offering, I’m convinced there is a lot that can be learned from this paper.
In particular, I really really hope SPBMC will report more of their data—including the psychometric properties of their climate-science literacy scale and summary data on how scores actually are distributed in their sample.
They’d certainly be welcome to do so in this blog!
Still, as a scholar grappling with the central psychometric issues involved in measuring climate science literacy, I just don’t think the particular results SPBMC have reported support the conclusions that they purport to draw.
I’m sure they’d agree with me, too, that scholars investigating these issues are obliged to speak up when they see a study that they think hasn’t fully addressed them. If scholars don't do this out of some misplaced sense of politeness (or any other sensibility, for that matter, that constrains open and candid scholarly exchange), then science communicators and educators who are relying on empirical work to make informed judgments will end up making serious and costly errors.
It should also go without saying that it is a mistake to think peer review happens only before a paper is published. If anything, that’s precisely when meaningful peer review begins.
Bostrom, A., Morgan, M. G., Fischhoff, B., & Read, D. (1994). What Do People Know About Global Climate Change? 1. Mental Models. Risk Analysis, 14(6), 959-970. doi: 10.1111/j.1539-6924.1994.tb00065.x
Guy, S., Kashima, Y., Walker, I., & O'Neill, S. (2014). Investigating the effects of knowledge and ideology on climate change beliefs. European Journal of Social Psychology, 44(5), 421-429.
Loewenstein, G.F., Weber, E.U., Hsee, C.K. & Welch, N. Risk as Feelings. Psychological Bulletin 127, 267-287 (2001).
Reynolds, T. W., Bostrom, A., Read, D., & Morgan, M. G. (2010). Now What Do People Know About Global Climate Change? Survey Studies of Educated Laypeople. Risk Analysis, 30(10), 1520-1538. doi: 10.1111/j.1539-6924.2010.01448.x
Slovic, P., Finucane, M.L., Peters, E. & MacGregor, D.G. Risk as Analysis and Risk as Feelings: Some Thoughts About Affect, Reason, Risk, and Rationality. Risk Analysis 24, 311-322 (2004).
Stevenson, K. T., Peterson, M. N., Bondell, H. D., Moore, S. E., & Carrier, S. J. (2014). Overcoming skepticism with education: interacting influences of worldview and climate change knowledge on perceived climate change risk among adolescents. Climatic Change, 126(3-4), 293-304.
Tobler, C., Visschers, V. H. M., & Siegrist, M. (2012). Addressing climate change: Determinants of consumers' willingness to act and to support policy measures. Journal of Environmental Psychology, 32(3), 197-207. doi: http://dx.doi.org/10.1016/j.jenvp.2012.02.001
B/c this post is linked in an insightful Wonkblog writeup on the SPBMC paper, & b/c "S" has now written a very informative response to my post in the comments section, it occurs to me that I should be clear about my own attitude toward the study.
I think I would bet against the claim that SPBMC are making.
But I'd be hedged: if I lost the bet, the gain in knowledge would more than compensate me for whatever amount I lost on the bet!
My current best understanding is that for grownup Americans belief in or acceptance of global warming has nothing to do with comprehension of either science generally or climate science in particular. It is simply an expression of cultural identity.
I wouldn't be surprised to learn that in middle schoolers there is no or only a very weak relationship between belief in human-caused global warming & cultural outlooks. It's plausible to me that these sorts of connections get worked out in people's minds at a later age.
But SPBMC report that cultural individualism is correlated with "disbelieving" human-caused global warming. Also very believable (& sad).
Yet they also report that this effect abates as the level of climate-science knowledge in middle schoolers increases. That's the part I find surprising. It's not what one sees in adults.
& it's puzzling, at least to me: if middle school kids are already "mature" enough to start to figure out what sorts of "positions" on global warming "go with" their cultural identities -- are already socialized enough to start to form stances that protect their status within groups important to their well being-- then I'd expect their positions to be unaffected by their science comprehension, b/c the sort of symbolic stance-taking involved in climate change, evolution & other culturally contested "science" issues in fact has nothing to do w/ what people (grownups at least) know about science.
But what SPBMC report finding could well be true! At which point, I'd certainly like to know what the best explanation is for such an important result.
But before accepting an explanation for it, I'd like to be sure (or be assured) that it is true that the impact of cultural individualism on middle schoolers' acceptance of human-caused climate change abates as their climate science comprehension increases.
To be assured, I'd need more information about the performance of the scale & the relationship of its items to the study subjects' cultural outlooks. I want to see enough to be sure the scale is valid & doesn't involve the identity-knowledge confound or knowledge-acceptance circularity problems I advert to.
SPBMC have whetted my appetite; I am hungry for even more insight more from their study!