I still hate NHT!
Wednesday, January 4, 2017 at 5:17AM
Dan Kahan

The 14 billion regular readers of this blog know that I really despise "null hypothesis tesing."  There are lots of reasons but one of the principal ones is that it short circuits practical inference. Sure, a hypothesis might imply/entail rejection of the null; but rejection of the null still might not support the hypothesis -- either because the effect is smaller than one would expect if the hypothesis were true or, even more importantly, because numerous other alternative hypotheses might also entail rejection of the null.

That problem supplied the motivation for the latest CCP paper on the relationship between pathogen-disgust sensitivity and the perceived risks of vaccines and GM foods: the correlations between the disgust scale and those two putative risk sources were no different from the correlation between the scale and myriad other risks that have nothing to do with disgust. 

 Here's an excerpt that expressly connects the research findings to this defect in NHT.

3. Study

3.1. Inference strategy

This paper rests on a simple theoretical premise: that rejection of a “null hypothesis” with respect to the correlation between pathogen disgust sensitivity, on the one hand, and GM-food and vaccine risk perceptions, on the other, is not sufficient to support the conclusion that disgust sensitivity meaningfully explains these risk perceptions (Rozeboom 1960; Ziliak & McCloskey 2008).  Like all valid latent variable instruments, any scale used to measure pathogen disgust sensitivity will be imperfect. Such a scale should be highly correlated with, and thus reliably measure, a particular form of disgust sensitivity. But such a scale can still be expected to correlate weakly or even modestly with additional negative affective dispositions (Chapman & Anderson 2013).  As a result, there can be modest yet practically meaningless correlations between the pathogen disgust sensitivity scale and all manner of risk perceptions that excite negative affective reactions unrelated to disgust.

A comparative analysis is thus appropriate.  If disgust genuinely explains perceived risks of vaccines and GM foods, the degree of the correlation between such concerns and a valid measure of pathogen disgust should be comparable to the relatively large correlation between PDS and attitudes already understood to be grounded in disgust. By the same token, one can infer that PD is not a particularly important source of variance in GM-food and vaccine risk perceptions if the correlation between PDS and these putative risk sources is comparable to correlations between pathogen disgust sensitivity and risk sources that do not plausibly excite disgust.

This was the inference strategy that informed design of this study.

* * *

5. Discussion and Conclusion

In assessing risk perceptions, simple correlations can be misleading.  Bare null-hypothesis testing doesn’t in itself support inferences without benchmarks to help interpret the uniqueness and magnitude of observed “significant” correlations.

This paper supplied benchmarks for appraising the relationship between pathogen disgust sensitivity and perceptions of vaccine and GM food risks.  With respect to both, the correlations with an established disgust-sensitivity scale were no greater than the correlations of myriad risks that were unrelated to disgust, such as the danger of a crash of a commercial airliner or the catastrophic malfunctioning of an elevator in a high-rise building.

In addition, the analyses revealed at least some reason to doubt the discriminant validity of one of the disgust measure that is being used in the study of childhood-vaccine and of GM-food risk perceptions. The conventional PDS scale, it turns out, is even better for predicting who will worry about carjacking and mass shootings than it is for predicting who will worry about the hazards of consuming food additives or being exposed to noxious wastes, not to mention who will be afraid of vaccines and GM foods.

Obviously, this is only one study of many examining the sources of variance in these risk perceptions. A thoughtful reader ought to weigh all of them in forming an opinion, which itself should be open to revision as new evidence arises. We submit, however, that the weight of the evidence presented here ought to be placed on the side of the balance suggesting that disgust is not a meaningful influence on GM-food and vaccine risk perceptions at the general population level.


Chapman, H.A. & Anderson, A.K. Things rank and gross in nature: A review and synthesis of moral disgust. Psychological Bulletin 139, 300 (2013).

Kahan, D.M. & Hilgard, J.. The impact of pathogen-disgust sensitivity on vaccine and GM food risk perceptions: Some evidence for skepticism, APPC/CCP Science of Science Communication working paper No. 6 (2016).

Rozeboom, W.W. The fallacy of the null-hypothesis significance test. Psychological bulletin 57, 416-428 (1960).

Ziliak, S.T. & McCloskey, D.N. The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives (University of Michigan Press, 2008).




Article originally appeared on cultural cognition project (http://www.culturalcognition.net/).
See website for complete article licensing information.