Can you spot which "study" result supports the "gateway belief model" and which doesn't? Not if you use a misspecified structural equation model . . .
As promised “yesterday”: a statistical simulation of the defect in the path analysis that van der Linden, Leiserowitz, Feinberg & Maibach (2015) present to support their “gateway belief model.”
VLFM report finding that a consensus message “increased” experiment subjects’ “key beliefs about climate change” and “in turn” their “support for public action” to mitigate it. In support of this claim, they present this structural equation model analysis of their study results:
As explained in my paper reanalyzing their results, VLFM’s data don’t support their claims. They nowhere compare the responses of subjects “treated” with a consensus message and those furnished only a “placebo” news story on a Star Wars cartoon series. In fact, there was no statistically or practically significant difference in the “before and after” responses of these two groups of subjects’ expressions of belief in climate change or support for global warming mitigation.
The VLFM structural equation model obscures this result. The model is misspecified (or less technically, really messed up) because it contains no variables for examining the impact of the experimental treatment—exposure to a consensus message—on any study outcome variable besides subjects’ estimates of the percentage of climate scientists who adhere to the consensus position on human-caused global warming.
To illustrate how this misspecification masked the failure of the VLFM data to support their announced conclusions, I simulated two studies designed in the same way as VLFM’s. They generated these SEMs:
As can be seen, all the path parameters in the SEMs are positive and significant—just as was true in the VLFM path analysis. That was the basis of VLFM’s announced conclusion that “all [their] stated hypotheses were confirmed.”
But by design, only one of the simulated study results supports the VLFM hypotheses. The other does not; the consensus message changes the subjects’ estimates of the percentage of scientists who subscribe to the consensus position on human-caused climate change, but doesn’t significantly affect (statistically or practically) their beliefs in climate change or support for mitigation--the same thing that happened in the actual VLFM study.
The path analysis presented in the VLFM paper can’t tell which is which.
Can you? If you want to try, you can download the simulated data sets here.
To get the right answer, one has to examine whether the experimental treatment affected the study outcome variable (“mitigation”) and the posited mediators (“belief” and “gwrisk”) (Muller, Judd & Yzerbyt 2005). That’s what VLFM’s path analysis neglects to do. It’s the defect in VLFM that my re-analysis remedies.
For details, check out the “appendix” added to the VLFM data reanalysis.
Have fun—and think critically when you read empirical studies.
Muller, D., Judd, C.M. & Yzerbyt, V.Y. When moderation is mediated and mediation is moderated. Journal of personality and social psychology 89, 852 (2005).
van der Linden SL, Leiserowitz A.A., Feinberg G.D., Maibach E.W. The Scientific Consensus on Climate Change as a Gateway Belief: Experimental Evidence. PLoS ONE (2015), 10(2): e0118489.doi:10.1371/journal.pone.0118489.