"How confident should we be ..."
A thoughtful journalist asks in relation to our Nature Climate Change study:
It would be really helpful to get your reflection on the research. In particular, I'm interested in the polarising effect you were able to identify. From the figure (Fig.2) this appears to be quite subtle, albeit in the opposite direction to that which was predicted by the SCT thesis. It would be great if you could identify to what extent/how confident we can be to say that increasing numeracy and literacy polarises risk perception about climate change, and what can explain this polarisation.
This was such a thoughtful way of putting the question, I felt impelled -- only in part by OCD; one shouldn't ask a good question if one wants an imprecise, casual response -- to give a reasonably precise & detailed answer:
1. All study results are provisional. That's in the nature of science. Valid studies give you more evidence than you otherwise would have had to believe something. They never "settle" the issue; one continues to revise one's assessment of what to treat as true and how likely it is not to be as more valid studies, more valid evidence, accumulates. Forever & ever (Popper 1962).
So it is never sensible (it is a misundersanding of the nature of empirical proof) to say, "this study proves this" or "this study doesn't necessailry prove that" etc. Instead it is very sensible to ask, as you have, "how confident should we be" in a particular conclusion given the evidence presented in a particular study.
2. As you know, our study investigated two hypotheses: the science comprehension thesis (SCT), which attributes public conflict over climate change to deficits in science comprehension; and the cultural cognition thesis (CCT), which asserts that conflict over climate change is a consequence of the unconscious tendency of individuals to fit the beliefs about risk to positions that dominate in their group, and which in its strongest form would say that this tendency will be reinforced or magnified by grater science comprehension, which can be used to promote such fitting.
3. The study furnishes relatively strong evidence that SCT is incorrect. SCT would predict that cultural polarization abates as science comprehension increases. Even if we had found that the impact of science comprehension on cultural polarization was nil, the study would supply the basis for a high degree of confidence that public conflict over climate change is not a consequence of low science comprehension.
4. The study is consistent with CCT and furnishes modest evidence that CCT in its strongest form is correct. That position would predict that cultural polarization will be greater among individuals with the greatest science comprehension. The results fit that hypothesis-- on both climate change & nuclear power risks; the latter helps to furnish more reason to think that the effect is genuine one for climate change.
But I'd say only modest evidence mainly because of the design of the study. It's observational --correlational -- only. Observed correlations that fit a hypothesis supply supporting evidence in proportion to which they rule out other explanations. Maybe something else is going on that causes both increased science comprehension & increased polarization in certain people. The only way to tell is through (well designed) experiments. We are conducting some now.
5. You note the effect size of the interaction is modest. Maybe; it's hard to know how to characterize such things in the abstract (and realize, too, that polarization is so great even for low-comprehending respondents that it would be hard for it to grow much for high-comprehending respondents!).
The size of the interaction effect we observed is probably about what you would expect for an observational study, and if the source of the effect is CCT, it should be easy to produce much more dramatic effects through properly designed experiments (Cohen, Cohen, Aiken & West 2003, pp. 297-98). So rather than try to extract more information from the effect size about how confident or not to be in the strong CCT position, it makes sense to do experiments. Again that's what we are now doing.
6. By itself, then, the study furnishes only modest reason to be confident in CCT (in its strongest form) relative to other possibilities (one has to be able to identify such possibilities, of course, in order to have any reason to doubt CCT; I can think of possibilities, certainly). I myself am more than modestly confident -- but only because this study is not the only thing I count as evidence that (strong) CCT is correct.
7. An aside: Nothing in our study suggests that making people more science literate or numerate causes polarization. If CCT is correct, there is something about climate change (and certain other issues) that makes people try to maximize the fit between their beliefs and positions that predominate within their groups, which themselves are impelled into opposing stances on certain facts. That thing is the cause in the practical, normative sense. We should find it and get rid of it.
references:
Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences (3rd ed.). Mahwah, N.J.: L. Erlbaum Associates.
Popper, K. R. (1962). Conjectures and refutations; the growth of scientific knowledge. New York,: Basic Books.
Reader Comments