follow CCP

Recent blog entries
popular papers

What Is the "Science of Science Communication"?

Climate-Science Communication and the Measurement Problem

Ideology, Motivated Cognition, and Cognitive Reflection: An Experimental Study

'Ideology' or 'Situation Sense'? An Experimental Investigation of Motivated Reasoning and Professional Judgment

A Risky Science Communication Environment for Vaccines

Motivated Numeracy and Enlightened Self-Government

Ideology, Motivated Cognition, and Cognitive Reflection: An Experimental Study

Making Climate Science Communication Evidence-based—All the Way Down 

Neutral Principles, Motivated Cognition, and Some Problems for Constitutional Law 

Cultural Cognition of Scientific Consensus

The Tragedy of the Risk-Perception Commons: Science Literacy and Climate Change

"They Saw a Protest": Cognitive Illiberalism and the Speech-Conduct Distinction 

Geoengineering and the Science Communication Environment: a Cross-Cultural Experiment

Fixing the Communications Failure

Why We Are Poles Apart on Climate Change

The Cognitively Illiberal State 

Who Fears the HPV Vaccine, Who Doesn't, and Why? An Experimental Study

Cultural Cognition of the Risks and Benefits of Nanotechnology

Whose Eyes Are You Going to Believe? An Empirical Examination of Scott v. Harris

Cultural Cognition and Public Policy

Culture, Cognition, and Consent: Who Perceives What, and Why, in "Acquaintance Rape" Cases

Culture and Identity-Protective Cognition: Explaining the White Male Effect

Fear of Democracy: A Cultural Evaluation of Sunstein on Risk

Cultural Cognition as a Conception of the Cultural Theory of Risk


Can you spot which "study" result supports the "gateway belief model" and which doesn't? Not if you use a misspecified structural equation model . . .

As promised “yesterday”: a statistical simulation of the defect in the path analysis that van der Linden, Leiserowitz, Feinberg & Maibach (2015) present to support their “gateway belief model.”

VLFM report finding that a consensus message “increased” experiment subjects’ “key beliefs about climate change” and “in turn” their “support for public action” to mitigate it. In support of this claim, they present this structural equation model analysis of their study results:


As explained in my paper reanalyzing their results, VLFM’s data don’t support their claims. They nowhere compare the responses of subjects “treated” with a consensus message and those furnished only a “placebo” news story on a Star Wars cartoon series.  In fact, there was no statistically or practically significant difference in the “before and after” responses of these two groups of subjects’ expressions of belief in climate change or support for global warming mitigation.

The VLFM structural equation model obscures this result.  The model is misspecified (or less technically, really messed up) because it contains no variables for examining the impact of the experimental treatment—exposure to a consensus message—on any study outcome variable besides subjects’ estimates of the percentage of climate scientists who adhere to the consensus position on human-caused global warming.

To illustrate how this misspecification masked the failure of the VLFM data to support their announced conclusions, I simulated two studies designed in the same way as VLFM’s. They generated these SEMs:

As can be seen, all the path parameters in the SEMs are positive and significant—just as was true in the VLFM path analysis.  That was the basis of VLFM’s announced conclusion that “all [their] stated hypotheses were confirmed.”

"Study No. 1" -- is this the one that supports the "gateway model"?But by design, only one of the simulated study results supports the VLFM hypotheses.  The other does not; the consensus message changes the subjects’ estimates of the percentage of scientists who subscribe to the consensus position on human-caused climate change, but doesn’t significantly affect (statistically or practically) their beliefs in climate change or support for mitigation--the same thing that happened in the actual VLFM study.

The path analysis presented in the VLFM paper can’t tell which is which.

Can you?  If you want to try, you can download the simulated data sets here.

To get the right answer, one has to examine whether the experimental treatment affected the study outcome variable (“mitigation”) and the posited mediators (“belief” and  “gwrisk”) (Muller, Judd & Yzerbyt 2005). That’s what VLFM’s path analysis neglects to do.  It’s the defect in VLFM that my re-analysis remedies.

Or is it "Study No. 2"? Download the data & see; it's not hard to figure out if you don't use a misspecified SEMFor details, check out the “appendix” added to the VLFM data reanalysis

Have fun—and think critically when you read empirical studies.


Muller, D., Judd, C.M. & Yzerbyt, V.Y. When moderation is mediated and mediation is moderated. Journal of personality and social psychology 89, 852 (2005).

van der Linden SL, Leiserowitz A.A., Feinberg G.D., Maibach E.W. The Scientific Consensus on Climate Change as a Gateway Belief: Experimental Evidence. PLoS ONE (2015), 10(2): e0118489.doi:10.1371/journal.pone.0118489.



Serious problems with "the strongest evidence to date" on consensus messaging ... 

So . . .

van der Linden, Leiserowitz, Feinberg & Maibach (2015) posted the data from their study purporting to show that subjects exposed to a scientific-consensus message “increased” their “key beliefs about climate change” and “in turn” their “support for public action” to mitigate it.

Christening this dynamic the "gateway belief" model, VLFM touted their results as  “the strongest evidence to date” that “consensus messaging”— social-marketing campaigns that communicate scientific consensus on human-caused global warming—“is consequential.”

At the time they published the paper, I was critical because of the opacity of the paper’s discussion of its methods and the sparseness of the reporting of its results, which in any case seemed underwhelming—not nearly strong enough to support the strength of the inferences the authors were drawing.

But it turns out the paper has problems much more fundamental than that.

I reanalyzed the data, which VLFM posted in March, a little over a year after publication,  in conformity with the “open data” policy of PLOS ONE, the journal in which the article appeared.

As I describe in my reanalysis, VLFM fail to report key study data necessary to evaluate their study hypotheses and announced conclusions. 

Their experiment involved measuring the "before-and-after" responses of subjects who received a “consensus message”—one that advised them that “97% of climate scientists have concluded that human-caused climate change is happening”—and those who read only “distractor” news stories on things like a forthcoming animated Star Wars cartoon series. 

In such a design, one compares the “before-after” response of the “treated” group to the “control,” to determine if the "treatment"—here the consensus message—had an effect that differed significantly from the control placebo. Indeed, VLFM explicitly state that their analyses “compared” the response of the consensus-message and control-group subjects

But it turns out that the only comparison VLFM made was between the groups' respective estimates of the percentage of climate-change scientists who subscribe to the consensus position. Subjects who read a statement that "97% of climate scientists have concluded that climate-change is happening" increased theirs more than did subjects who viewed only a distractor news story.

But remarkably VLFM nowhere report comparisons of the two groups' post-message responses to items measuring any of the beliefs and attitudes for which they conclude perceived scientific consensus as a critical "gateway" .

Readers including myself, initially, thought that such comparisons were being reported in a table of “differences” in “Pre-” and “Post-test Means” included in the article.

These aren't experimental effects after all...

But when I analyzed the VLFM data, I realized that, with the exception of the difference in "estimated scientific consensus," all the "pre-" and "post-test" means in the table had combined the responses of consensus-message and control-group subjects.

There was no comparison of the pre- and post-message responses of the two group of subjects; no analysis of whether their responses differed--the key information necessary to assess the impact of being exposed to a consensus message.

Part of what made this even harder to discern is that VLFM presented a complicated “path diagram” that can be read to imply that exposure to a consensus message initiated a "cascade" (their words) of differences in before-and-after responses, ultimately leading to “increased support for public action”—their announced conclusion.

The misspecified "gateway belief" SEM...

But this model also doesn't compare the responses of consensus-message and control-group subjects on any study measure except the one soliciting their estimates of the "percentage of scientists [who] have concluded that human-caused climate change is happening."

That variable is the only one connected by an arrow to the "treatment"--exposure to a consensus message.

As I explain in the paper, none of the other paths in the model distinguishes between the responses of subjects “treated” with a consensus message and those who got the "placebo" distractor news story. Accordingly, the "significant" coefficients in the path diagram reflect nothing more than correlations between variables one would expect to be highly correlated given the coherence of people’s beliefs and attitudes on climate change generally.

In the paper, I report the data necessary to genuinely compare the responses of the consensus-message and control-group subjects.

It turns out that, subjects exposed to a consensus message didn’t change their “belief in climate change” or their “support for public action to mitigate it” to an extent that significantly differed, statistically or practically, from the extent to which control subjects changed theirs.

Indeed, the modal and median effects of being exposed to the consensus message on the 101-point scales used by VLFM to measure "belief in climate change" and "support for action" to mitigate it were both zero--i.e., no difference in "after" or "before" responses to these  study measures. 

No one could have discerned that from the paper either, because VLFM didn't furnish any information on what the raw data looked like. In fact, both the consensus-message and placebo news-story subjects' '"before-message" responses were highly skewed in the direction of belief in climate change and support for action, suggesting something was seriously amiss with the sample, the measures, or both--all the more reason to give little weight to the the study results.

But if we do take the results at face value, the VLFM data turn out to be highly inconsistent with their announced conclusion that "belief in the scientific consensus functions as an initial ‘gateway’ to changes in key beliefs about climate change, which in turn, influence support for public action.”

The authors “experimentally manipulated” the expressed estimates of the percentage of scientists who subscribe to the consensus position on climate change. 

Yet the subjects whose perceptions of scientific consensus were increased in this way did not change their level of "belief" in climate change, or their support for public action to mitigate it, to an extent that differed significantly, in practical or statistical terms, from subjects who read a "placebo" story about a Star Wars cartoon series.

That information, critical to weighing the strength of the evidence in the data, was simply not reported.

VLFM have since conducted an N = 6000 "replication."  As I point out in the paper, "increasing sample" to "generate more statistically significant results" is recognized to be a bad research practice born of a bad convention--namely, null-hypothesis testing; when researchers resort to massive samples to invest minute effect sizes with statistical significance, "P values are not and should not be used to define moderators and mediators of treatment" (Kraemer, Wilson, & Fairburn 2002, p, 881). Bayes Factors or comparable statisics that measure the inferential weight of the data in relation to competing study hypotheses should be used instead (Kim & Je 2015; Raftery 1995). Reviewers will hopefully appreciate that. 

But needless to say, doing another study to try to address lack of statistical power doesn't justify claiming to have found significant results in data in which they don't exist. VLFM claim that their data show that being exposed to a consensus message generated a “a significant increase” in “key beliefs about climate change” and in "support for public action" when “experimental consensus-message interventions were collapsed into a single ‘treatment’ category and subsequently compared to [a] ‘control’ group” (VLFM p. 4).  The data -- which anyone can now inspect-- say otherwise.

Hopefully reviewers will pay more attention too to  how a misspecified SEM model can conceal the absence of an experimental effect in a study design like the one reflected here (and in other "gateway belief" papers, it turns out...). 

As any textbook will tell you, “it is the random assignment of the independent variable that validates the causal inferences such that X causes Y, not the simple drawing of an arrow going from X towards Y in the path diagram” (Wu & Zumbo 2007, p. 373).  In order to infer that an experimental treatment affects an outcome variable, “there must be an overall treatment effect on the outcome variable”; likewise. in order to infer that an experimental treatment affects an outcome variable through its effect on a “mediator” variable, “there must be a treatment effect on the mediator” (Muller, Judd & Yzerbyt 2005, p. 853). Typically, such effects are modeled with predictors that reflect the “main effect of treatment, main effect of M [the mediator], [and] the interactive effect of M and treatment” on the outcome variable (Kraemer, Wilson, & Fairburn 2002, p, 878).

Because the VLFM structural equation model lacks such variables, there is nothing in it that measures the impact of being “treated” with a consensus message on any of the study’s key climate change belief and attitude measures. The model is thus misspecified, pure and simple.

To illustrate this point and underscore the reporting defects in this aspect of VLFMI'll post "tomorrow" the results of a fun statistical simulation that helps to show how the misspecified VLFM model-- despite its fusillade of triple-asterisk-tipped arrows--is simply not capable of distinguishing the results a failed experiment from one that actually does support something like the “gateway model” they proposed.

BTW, I initiatlly brought all of these points to the attention of the PLOS One editorial office.  On their advice, I  posted a linke to my analyses in the comment section, after first soliciting a response from VLFM.

A lot of people are critical of PLOS ONE

I think they are being unduly critical, frankly.

The mission of the journal--to create an outlet for all valid work-- is a valuable and admirable one.

Does PLOS ONE publish bad studies? Sure. But all journals do! If they want to make a convincing case, the PLOS ONE critics should present some genuine evidence on the relative incidence of invalid studies in PLOS ONE and other journals.  I at least have no idea what such evidence would show.

But in any case, everyone knows that bad studies get published all the time-- including in the "premier" journals. 

What happens next-- after a study that isn't good is published --actually matters a lot more. 

In this regard, PLOS ONE is doing more than most social science journals, premier ones included, to assure the quality of the stock of knowledge that reserchers draw on. 

The journal's "open data" policy and its online fora for scholarly criticsm and discussion supply scholars with extremely valuable resources for figuring out that a bad study is bad and for helping other scholars see that too.

If what's "bad" about a study is that the inferences its data support are just much weaker than the author or authors claim, other scholars will know to give the article less weight.

If the study suffers from some a serious flaw (like unreported material data or demonstrably incorrect forms of analysis), then the study is much more likely to get corrected or retracted than it would be if it managed to worm its way into a "premier" journal that lacked an open-data policy and a forum for online comments and criticism.

Peer review doesn't end when a paper is published.  If anything, that's when it starts. PLOS ONE gets that. 

I do have the impression that in the social sciences, at least, a lot of authors think they can dump low quality studies on PLOS ONE.  But that's a reason to be mad at them, not the journal, which if treated appropriately by scholars can for sure help enlarge what we know about how the world works.

So don't complain about PLOS ONE. Use the procedures it has set up for post-publication peer review to make authors think twice before denigrating the journal's mission by polluting its pages with bull shit studies.


Kraemer, H.C., Wilson, G.T., Fairburn, C.G. & Agras, W.S. Mediators and moderators of treatment ef-fects in randomized clinical trials. Archives of general psychiatry 59, 877-883 (2002).

Muller, D., Judd, C.M. & Yzerbyt, V.Y. When moderation is mediated and mediation is moderated. Journal of personality and social psychology 89, 852 (2005).

van der Linden SL, Leiserowitz A.A., Feinberg G.D., Maibach E.W. The Scientific Consensus on Climate Change as a Gateway Belief: Experimental Evidence. PLoS ONE (2015), 10(2): e0118489. doi:10.1371/journal.pone.0118489.

Wu, A.D. & Zumbo, B.D. Understanding and Using Mediators and Moderators. Social Indicators Re-search 87, 367-392 (2007).






Job opening for social-science editor at Nature! 

Are you a trained social scientist now driving a cab (or uber-registered vehicle)?

Or a gainfully employed US social scientist looking for an exit strategy in case Donald Trump is elected president?

Well, here you go! A job at Nature!

The job itself would be lots of fun, I'm sure, but think of all the cool things you could learn from office scuttlebutt as the journal issues get put together every week!



WSMD? JA! "Confidence intervals" for "Political Polarization Literacy" test

Winning entry. Verified by "Virgin Mary on your French toast, you say?"™ miracle certifiersFormer Freud expert & current stats legend  Andrew Gelman and Josh " 'Hot Hand Fallay' Fallacy" Miller have announced publicly that they scored perfect 14.75's (higher, actually) on the CCP/APPC "Political Polarization Literacy" test.  

They have now demanded that they be awared the "Gelman Cup." That request actually made their "political polarization literacy" scores a bit more credibile, since obviously they are too busy measuring public opinion to stay current with CCP contests and their respective prizes (I've sent them an authentic "Worrship the Industrial Strength Risk Perception Measure!" Virgin Mary Frenchtoast slice" for their performances).

But speaking of CCP games ... you guessed it: Time for another installment of

 the insanely popular CCP series, "Wanna see more data? Just ask!," the game in which commentators compete for world-wide recognition and fame by proposing amazingly clever hypotheses that can be tested by re-analyzing data collected in one or another CCP study. For "WSMD?, JA!" rules and conditions (including the mandatory release from defamation claims), click here.

Statistical Modeling, Causal Inference, and Social Science commentator@Rahul wondered what it would look like if the plots in the "Political Polarization Literacy" test figure had confidence intervals.

Here's the answer: 

(If you forgot what the policy items are, you can either go back to the original post or just click here.)

Actually, I'm not sure CIs add interesting information here.

Once one knows that the N = 1200 & the sample is representative, it's pretty easy to know what the CIs will look like (around 0.04 at pr = 0.50; smaller as one approaches pr = 0 & pr = 1.0).

The intersting information here is in the covariances of positions and left_right. The CIs don't make that any clearer; if anything, they make that a bit harder to see!  So I'd say for the purposes of the game, the lowess plots, sans CIs, were all the "statistics" & "modeling" needed for us to start learning something (about WEKS) from the data.

But that's my view. Others might disagree.

Who knows-- they might even disagree with me that "spike plots" rather than, say, colored confidence bands are a prettier way to denote 0.95 CI zones if one thinks there is something to be gained by fitting a model to data like these!


In awe of the Industrial Strength Risk Perception Measure. . . .

This is a little postscript on yesterday’s post on the CCP/APPC "Political Polarization Literacy" test.

A smart friend asked me whether responses to the items in the “policy preferences” battery from yesterday might be different if the adjective “government” were not modifying “policies” in the introduction to the battery.

I think, frankly, that 99% of the people doing public opinion research would find this question to be a real snoozer but in fact it it’s one that ought to keep them up all night (assuming they are the sort who don’t stay up all night as a matter of course; if they are, then up all day) w/ anxiety.

It goes to the issue of what items like these are really measuring—and how one could know what they are measuring.  If one doesn’t have a well-founded understanding of what responses to survey items are measuring—if anything—then the whole exercise is a recipe for mass confusion or even calculated misdirection. I’m not a history buff but I’m pretty sure the dark ages were ushered in by inattention to the basic dimension of survey item validity; or maybe we still are in the dark ages in public opinion research as a result of this (Bishop 2005)?

In effect, my colleague/friend/collaborator/fellow-perplexed-conversant was wondering if there was something about the word “government” that was coloring responses to all the items, or maybe a good many of them, in way that could confound the inferences we could draw from particular ones of them . . . .

I could think of a number of fairly reasonable interpretive sorts of arguments to try to address this question, all of which, it seems to me, suggest that that’s not likely so.

But the best thing to do is to try to find some other way of measuring what I think the policy items are measuring, one that doesn’t contain the word “government,” and see if there is agreement between responses to the two sets of items. If so, that supplies more reason to think, yeah, the policy items are measuring what I thought; either that or there is just a really weird correspondence between the responses to the items—and that’s less a likely possibility in my view.

What do I think the “policy” items are measuring?  I think the policy items are measuring, in a noisy fashion (any single item is noisy)  pro- or con- latent or unobserved attitudes toward particular issues that themselves are expressions of another latent attitude, measured (nosily but less so  because there are two “indicators” or indirect measures of it) by the aggregation of the “partisan self-identification” and “liberal-conservative” ideology items that “Left_right” comprise.

That’s what I think risk perception measures are too—observable indicators of a latent pro- or con-affective attitude, one that often is itself associated with some more remote measure of identity of the sort that could be measured variously with either cultural worldview items, religiosity, partisan political identity, and the like (see generally Peters & Slovic 1996; Peters Burraston & Mertz 2004; Kahan 2009).

The best single indicator I can think of for latent affective attitudes is . . . the Industrial Strength Risk Perception Measure!

As the 14 billion readers of this blog know, ISRPMs consist in 0-7 or 0-10 rankings of the “risk” posed by a putative risk source. I’m convinced it works best when each increment in the Likert scale has a descriptive label, which favors 0-7 (hard to come up w/ 10 meaningful labels).

As I’ve written about before, the ISRPM has a nice track record.  Basically, so long as the putative risks source is something people have a genuine attitude about (e.g., climate change, but not GM foods), it will correlate pretty strongly with pretty much anything more specific you ask (is climate change happening? are humans causing it? are wesa gonna die?) relating to that risk.  So that makes the ISRPM a really economical way to collect data, which can then be appropriately probed for sources of variance that can help explain who believes what & why about the putative risk source.

It also makes it a nice potential validator of particular items that one might think are measuring the same latent attitude.  If those items are measuring what you think, they ought to display the same covariance patterns  that corresponding ISRPMs do in relation to whatever latent identity one posits explains variance in the relevant ISRPM.

With me? Good!

Now the nice thing here is that the ISRPM measure, as I use it, doesn’t explicitly refer to “government." The intro goes like this ...

and then you have individual "risk sources," which, when I do a study at least, I always randomize the order of & put on separate "screens" or "pages" so as to minimize comparative effects:

Obviously, certain items on an ISRPM  battery will nevertheless imply government regulation of some sort.

But that’s true for the “policy item” batteries, the validity of which was being interrogated (appropriately!) by my curious friend.

So, my thinking went, if the ISRPM items had the same covariance pattern as the policy items in respect to “Left_right,” the latent identity attitude formed by aggregation of a 7-point political identity item and a 5-point liberal conservative measure, that would be a pretty good reason to think (a) the two are measuring the same “latent” attitude and (b) what they are measuring is not an artifact of the word “government” in the policy items—even if attitudes about government might be lurking in the background (I don’t think that in itself poses a validity problem; attitudes toward government might be integral to the sorts of relationships between identity and “risk perceptions” and related “policy attitudes” variance in which we are trying to explain).

So. . .

I found 5 “pairs” of policy-preference items an corresponding ISRPMs.

The policy-preferences weren’t all on yesterday’s list. But that’s because only some of those had paired ISRPMs.  Moreover, some ISRPMs had had corresponding policy items not on yesterday’s list.  But I just picked the paired ones on the theory that covariances among “paired items” would give us information about the performance of items on the policy list generally, and in particular whether the word “government” matters.

Here are the corresponding pairs:

I converted the responses to z-scores, so that they would be on the same scale. I also reverse coded certain of the risk items, so that they would have the same valence (more risk -> support policy regulation; less risk -> oppose).

Here are the corresponding covariances of the responses to the items—policy & ISRPM—in relation to Left_right, the political outlook scale


Spooky huh?!  It’s harder to imagine a tighter fit!

Note that the items were administered to two separate samples

That’s important! Otherwise, I’d attribute this level of agreement to a survey artifact: basically, I’d assume that respondents were conforming their answer to whichever item (ISRPM or policy) that came second so that it more or less cohere with the one they gave to the first.

But that’s not so; these are response from two separate groups of subjects, so the parallel covariances gives us really good reason to believe that the “policy” items are measuring the same thing as the ISRPMs—and that the world “government” as it appears in the former isn’t of particular consequence.

If, appropriately, you want to see the underlying correlation matrix in table form, click here (remember, the paired items were administered to two separate samples so we have no information about their correlation with each other--only their respective correlations with left_right.)

So two concluding thoughts:

1. The question "what the hell is this measuring??," and being able to answer it confidently, are vital to the project of doing good opinion research.  It is just ridiculous to assume that survey items think they are measuring what you think; you have to validate them.  Otherwise, the whole enterprise becomes a font of comic misunderstanding.

2. We should all be friggin’ worshiping ISRPM! 

I keep saying that it has this wonderful quality, as a single-item measure, to get at latent pro-/con- attitudes toward risk; that responses to it are highly likely to correlate with more concrete questions we can ask about risk perceptions, and even with behavior in many cases.  There’s additional good research to support this.

But to get such a vivid confirmation of its miraculous powers in a particular case! Praise God!

It’s like seeing Virgin Mary on one’s French Toast!


Bishop, G.F. The illusion of public opinion : fact and artifact in American public opinion polls (Rowman & Littlefield, Lanham, MD, 2005).

Kahan, D.M. Nanotechnology and society: The evolution of risk perceptions. Nat Nano 4, 705-706 (2009).

Peters, E. & Slovic, P. The Role of Affect and Worldviews as Orienting Dispositions in the Perception and Acceptance of Nuclear Power. J Appl Soc Psychol 26, 1427-1453 (1996).

Peters, E.M., Burraston, B. & Mertz, C.K. An Emotion-Based Model of Risk Perception and Stigma Susceptibility: Cognitive Appraisals of Emotion, Affective Reactivity, Worldviews, and Risk Perceptions in the Generation of Technological Stigma. Risk Analysis 24, 1349-1367 (2004).


Hey everybody -- take the cool CCP/APPC "Political Polarization Literacy" test!

Because we, unlike certain other sites that I won’t deign to identify, actually listen to our 14 billion regular readers, CCP Blog is adding yet another member to its stable of wildly popular games (which, of course, include MAPKIA!, WSMD? JA!, & HFC! CYPHIMU?): the CCP/APPC “Political Polarization Literacy” Test! 

Official game motto: “Because the one thing we all ought to agree on is what we disagree about!”

Ready . . . set . . . open your test booklet and begin!

Match the policies on this list . . .



to the plotted lines in this figure:

Take your time, no rush.

Keep going.  

Scroll down when the proctor declares that the exam is over.

If you finish early, feel free to click on the random ass pictures that I've inserted to prevent you from inadverently spotting the answers before completing the test!

On their way to Canada, possibly?

Time’s up!

Okay, here’s what I’m going to do.  First, I’m going to start by showing you the “answer key,” which consists of the original figure with labels.

Second, I’m going to tell you how to score your answers.

To do that, I’ll display separate figures for (a) policies that are strongly polarizing; (b) policies that are weakly polarizing; (c) policies that reflect bipartisan ambivalence; and (d) policies that reflect bipartisan support. In connection with each of these figures, I’ll supply scoring instructions.

So . . .

“Answer key”


Point-score figures

1. Strongly polarizing


Award yourself (or your child or pet, if you are scoring his or her test) 1 point for each policy that appears in this set and that you (or said child or pet) matched with any of these five plotted lines regardless of which of the lines you actually matched it with. 

Got that? No? Okay, well, e.g., if you matched “stricter carbon emission standards to reduce global warming” with the “magenta” colored line you get 1 point; but you also get 1 point if you matched it with red, blue, midblue, or cyan-colored lines. Same for every other friggin’ policy in this set—okay?

Good. Now give yourself (et al.) a 3-point bonus if you matched “Gay marriage (allowing couples of the same sex to marry each other)” with any of the plotted lines depicted in this figure.

2. Weakly polarizing


Award yourself (et al.) 1.5 points for each policy in this set that you (enough of this) matched with either of the two plotted lines in this figure.

3. Bipartisan ambivalence


Award yourself 0.75 points if you got this one.

4. Bipartisan support

Award yourself 3 points if you matched “Approval of an amendment to the U.S. constitution that would allow Congress and state legislatures to prohibit corporations from contributing money to candidates for elected office” with either of the plotted lines in this figure.

Subtract 5 points if you failed to match “Requiring children who are not exempt for medical reasons to be vaccinated against measles, mumps, and rubella” with one of the two lines in this figure.

Subtract 5 points if you matched “Gay marriage (allowing couples of the same sex to marry each other)” with either of the two lines in this figure.


17: you are a cheater and are banned from this site until Pete Rose is inducted into the Baseball Hall of Fame, Hell Freezes Over, or Donald Trump is elected President, whichever happens first.

14.75: you are either a “political polarization genius,” pr = 0.25, or a liar, pr = 0.75.

10-14.74: Damn! You are one of the 14 billion regular readers of this blog!

5-10: Meh.

0-4: Not awful.

-10: You win! Obviously you have better things to do with your time than waste them viewing the sad spectacle of unreason that our democracy has become!  (But what the hell are you doing on this site?)

Now, some explanation on the scoring.

It was done by a Hal9001 series super computer,  which designed the “game” (obviously if you missed anything, that is due to human error).

But note that the Hal9001 put a lot emphasis on two policies in particular:

“Requiring children who are not exempt for medical reasons to be vaccinated against measles, mumps, and rubella”

“Gay marriage (allowing couples of the same sex to marry each other)”

The reason, I’m told by Hal9001, is that getting these ones wrong is a sign that your child or pet (obviously, we aren’t talking about you here!) is over-relying on heuristic, Political Polarization System 1 reasoning.  As a result, your child or pet is succumbing to the extremely common “what everyone knows syndrome,” or WEKS, a bias that consists in, well, treating “what everyone knows” as evidence. 

Or more specifically, treating as evidence the views of the biased sample (in a measurement sense, not necessarily a cognitive or moral one) of people who one happens to be exposed to disproportionately as a result of the natural, understandable tendency to self-select into “discourse communities” populated w/ people who basically have relatively uniform outlooks and motivations and experiences.

For sure WEKS biases people’s opinions on public opinion vaccines.

There is overwhelming empirical evidence of public support for universal immunization across all political, cultural, religious, etc. groups. Yet commentators, treating each other’s views as evidence,  keep insisting that either one group or another (“the conservative don’t-tread-on-me crowd that distrusts all government recommendations,” “limousine liberals,” blah blah) is hostile to vaccines or even  more patently false a “growing distrust of vaccinations” among  “a large and growing number” of “otherwise mainstream parents.”  And lots of people assume, gee, if “everyone knows that” it must be true!

Same on gay marriage.

In the sources that people on the “left” consult, “everyone knows that” there has been an been “an astounding transformation of public opinion.”  They constantly call for "replicating the success of marriage equality" on climate change, e.g.

Actually, the “transformation” on gay marriage was primarily just a bulging of public support among people on the “left.” Support among people who identify as “liberal” grew from 56% to 79%, and among those who identify as Democrat from 43% to 66%, in the period from 2000 to 2015; among self-identified “conservatives” and Republicans, the uptick was much more modest--from 18% to 30% and 21% to 32% respectively.

That’s a shift, sure. But 79%:30%/66%:32% is . . . political polarization.

The “how to replicate gay marriage” on climate change meme rests on a faulty WEKS premise or set of them. 

One is that Gay marriage isn’t as divisive than climate change. It is.  Or if it isn't, it's only because  there is still a higher probability that a “liberal Democrat” and a “conservative Republican” will agree that gay marriage shouldn’t be legalized than they will agree that the U.S. should or shouldn’t adopt “stricter carbon emission standards to reduce global warming.” 

Maybe climate change advocates should "replicate the success" of gun control advocates and affirmative action proponents, too?

Another faulty premise has to do with the instrument of legal change in gay marraiage.

Legalization of gay marriage occurred primarily by judicial action, not legislation: of the 37 states where gay marriage was already legally recognized before the U.S. Supreme Court decided Obergefell v. Hodges, 26 were in that category as a result of judicial decisions invalidating apparently popularly supported legal provisions (like California’s 2008 popular referendum “Prop. 8”) disallowing it.

Those judicial decisions, in my view, were 100% correct: the right to pursue one’s own conception of happiness in this way shouldn’t depend on popular concurrence.

But I don't think it's a good idea to propogate a false narrative about what really happened here or about what today’s reality is.   False narratives, underwritten by WEKS, lead people to make mistakes in their practical decisionmaking.

Indeed, WEKS-- the disposition of people to confuse the views of people who share one's outlooks, motivations, experience as “evidence” of how the world works—on public opinion and other topics too is one of the reasons we have polarization on facts that admit of being assessed with valid empirical evidence.

One of the reasons in other words that we are playing this stupid game.


Raw data: the best safeguard against empirical bull shit!

The last two posts were so shockingly well received that it seemed appropriate to follow them up w/ one that combined their best features: (a) a super smart guest blogger; (b) a bruising, smash-mouthed attack against those who are driving civility from our political discourse by casting their partisan adversaries as morons; and (c) some kick-ass graphics that are for sure even better than "meh" on the Gelman scale!  

The post helps drive home one of the foundational principles of critical engagement with empirics: if you don't want to be the victim of bull-shit, don't believe any statistical model before you've been shown the raw data!

Oh-- and I have to admit: This is actually a re-post from So if 5 or 6 billion of you want to terminate your subscription to this blog & switch over to that one after seeing this, well I won''t blame you --  now I really think Gelman was being kind when he described my Figure as merely "not wonderful...."

When studies studying bullshit are themselves bullshit...

We have a problem wth PLoS publishing bullshit studies.

Look, I really appreciate some aspects of PLoS. I like that they require people to share data. I like that they will publish null results. However, I really hope that someday the people who peer-review papers for them step up their game. 

This evening, I read a paper that purports to show a relationship between seeing bullshit statements as profound and support for Ted Cruz. 

The paper begins with an interesting premise: does people's bullshit receptivity--that is, their perception that vacuous statements contain profundity--predict their support for various political candidates? This is a particularly interesting question. I think we can all agree that politicians are basically bullshit artists

Specifically, though, the authors are not examining people's abilities to recognize when they are being lied to; they define bullshit statements as 

communicative expressions that appear to be sound and have deep meaning on first reading, but actually lack plausibility and truth from the perspective of natural science.

OK, they haven't lost me yet. 

The authors then reference some recent literature that has describes conservative ideology as what amounts to cognitive bias (at the least) and mental defect (at the worst).

I identify as liberal. However, I think that this is the worst kind of motivated reasoning on the part of liberal psychologists.  Some of this work has been challenged (see Kahan take on some of these issues). But let's ignore this for right now and pretend, that the research they are citing here is not flawed.

The authors have the following hypotheses:

  • Conservativism will predict judging bullshit statements as profound. (I can tell you right off that if this were mostly a conservative issue, websites like spirit science would not exist)
  • The more individuals have favorable views of Republican candidates, the more they will see profoundness in bullshit statements. (So here, basically using support for various candidates as another measure of conservativism).
  • Conservativism should not be significantly related to seeing profoundness in mundane statements. 
These hypotheses were clearly laid out, which I appreciate. I also appreciate that the authors followed the guidelines and posted their data. Makes post-publication peer-review possible.

Here is one of my first criticisms of the method of this paper. The authors chose to collect a sample of 196 participants off of Amazon's Mechanical Turk. Now, I understand why, MTurk is a really reasonably priced way of getting participants who are not psych 101 undergraduates. However, there are biases with MTurk samples. Mainly that they are disproportionately male, liberal, and educated. Particularly when researchers are interested in examining questions related to ideology, MTurk is not your best bet. But, let's take a look at the breakdown of their sample based on ideology, just to check--especially since we know that they want to make inferences about conservatives in particular.
I don't have the exact wording of their measure of conservativism, but they describe it as asking participants to place themselves on a 7-point Likert scale where 1 is liberal and 7 is conservative. The above table shows the frequency of participants at each level. You can see that the sample is disproportionately liberal/liberal leaning. In fact, if you total the participants in columns 5, 6, and 7 (the more conservative columns), you have a whopping total of 46 participants choosing a somewhat conservative designation versus the 108 participants in columns 1, 2, and 3.

Thus, it is unfair--in my opinion--to think that you can really make inferences about conservatives in general from this data. Many studies in political science and communications use nationally-representative data with over 1500 participants. At Annenberg Public Policy Center we get uncomfortable sometimes making inferences from our pre/post panel data (participants who we contact two times) because we end up with only around 600. I'm not saying that it is impossible to make inferences from less than 200 participants, but that the authors should be very hesitant, particularly when they have a very skewed sample.

I'm going to skip past analyzing the items that they use for their bullshit and mundane statements. It would be worth doing a more comprehensive item analysis on the bullshit receptivity scale--at least going beyond reporting Cronbach's alpha. But, that can be done another time.

The favorability ratings of the candidates are another demonstration of how the sample is skewed. The sample demonstrates the highest support for Bernie Sanders and the lowest support for Trump. 
This figure shows the density of participants rating four of the candidates (the authors also include Martin O'Malley and Marco Rubio) on the 5 point scale. You can see that a huge proportion of participants gave Trump and Cruz the lowest rating. In contrast, People tended to like Clinton and Sanders more. 

Moving onto their results.

The main claim that the authors make is that:
Favorable ratings of Ted Cruz, Marco Rubio, and Donald Trump were positively related to judging bullshit statements as profound, with the strongest correlation for Ted Cruz. No significant relations were observed for the three democratic candidates.
Now, the authors recognized that their samples were skewed, so they conducted non-parametric correlations. But, I'm not sure why the authors didn't simply graph the raw data and look at it.

Below, I graph the raw data with the bullshit receptivity scores on the x-axis and the support scores for each candidate on the y-axis. The colored line is the locally-weighted regression line and the black dashed line treats the model as linear. I put Ted Cruz first, since he's the one that the authors report the "strongest" finding for. 
So, does this look like a linear relationship to you? As the linear model goes up (which I imagine is what drives the significant correlation), the loess line goes down. I imagine that this is a result fo the outliers who score high on BSR. Note that of the ones scoring highest on BSR, the largest clump actually shows low support for Cruz, with one outlier rating him at 4. Indeed, the majority of the sample sits at 1 for supporting Ted cruz, and the samples' BSR scores are grouped in the middle.

You can see similar weirdness for the Trump and Rubio Ratings. The Trump line is almost completely flat--and if we were ever to think that support for a candidate predicted bullshit receptivity, it would be support for Trump---but I digress.... Note, too, how low support is.  Rubio, on the other hand, shows a light trend upwards when looking at the linear model (the black dashed line), but most people are really just hovering around the middle. Like with Cruz, the people with the highest bullshit receptivity (scores of around 5) rate Rubio low (1 or 2).

So, even if I don't agree that your significant correlations are meaningful for saying that support for conservatives is predicted by bullshit receptivity (or vice versa), you might still argue that there is a difference between support for liberals and support for conservatives. So, let's look at the democratic candidates.
I put Hillary Clinton on top, because I want to point out that her graph doesn't look much different from that of Cruz or Rubio. It still has a trend upwards, but not just for the linear model. In fact, where Cruz had one person that gave high ratings for him and had the highest level of bullshit receptivity, Hillary has three.  In fact, let's take a look at Clinton's loess Line and Cruz's loess line mapped on the same figure....
Kind of looks to me like there is a stronger argument for Clinton supporters having more bullshit receptivity. Now, I do support Hillary Clinton, so don't get me wrong. My point is only to demonstrate that significant p values for one type of correlation do not mean that the finding is real.

Here are the figures for Sanders and O'Malley. Again, pretty straight lines. But for the highest level of bullshit receptivity, Bernie indeed has a cluster of supporters--people rating him at the highest level. I do not think that this supports the opposite conclusion to what these authors found--that bullshit receptivity predicts democratic candidate support--but I don't think that the authors should have made the conclusions that they did. These results really just appear to be noise to me.
In conclusion...

The authors *do* list the limitations of their study. They state that their research is correlational and that their sample was not nationally representative. But they still make the claim that conservatism is related to seeing profoundness in bullshit statements.. Oh, which reminds me, we should have looked at that too...
That's a graphical representation of conservatism predicting bullshit receptivity. 

What concerns me, here, is two-fold.

First, despite what p values may or may not be below a .05 threshold, there is no reason to think that this data actually demonstrates that conservatives are more likely to see profundity in bullshit statements than liberals--But the media will love it.

Moreover, there is no reason to believe that such bullshit receptivity predicts support for conservative candidates--but the media will love it. This is exactly the type of fodder picked up  because it suggests that conservatism is a mental defect of some sort. It is exciting for liberals to be able to dismiss conservative worldviews as mental illness or as some sort of mental defect. However, rarely do I think these studies actually show what they purport to. Much like this one.

Second, it is this type of research that makes conservatives skeptical of social science. Given that these studies set out to prove hypotheses that conservatives are mentally defective, it is not surprising that conservatives become skeptical of social science or dismiss academia as a bunch of leftists. Check out this article on The Week about the problem of liberal bias in social science

If we actually have really solid evidence that conservatives are wrong on something, that is totally great and fine to publish. For instance, we can demonstrate a really clear liberal versus conservative bias in belief of climate change. But we have to stop trying to force data to fit the view that conservatives are bad. I'm not saying that this study should be retracted,  but it is indicative of a much larger problem with the trustworthiness of our research.

Bounded rationality, unbounded out-group hate

  By popular demand & for a change of pace ... a guest post from someone who actually knows what the hell he or she is talking about!

Bias, Dislike, and Bias

Daniel Stone

Read this! Or you are just a jerk, like all the other members of your stupid political party!Thanks Dan K for giving me the chance to post here.  Apologies - or warning at least - the content, tone etc might be different from what's typical of this blog.  Rather than fake a Kahan-style piece,[1] I thought it best to just do my thing.  Though there might be some DK similarity or maybe even influence.  (I too appreciate the exclamation pt!)

Like Dan, and likely most/all readers of this blog, I am puzzled by persistent disagreement on facts.  It also puzzles me that this disagreement often leads to hard feelings.  We get mad at - and often end up disliking - each other when we disagree.  Actually this is likely a big part of the explanation for persistent disagreement; we can't talk about things like climate change and learn from each other as much as we could/should - we know this causes trouble so we just avoid the topics. We don’t talk about politics at dinner etc.  Or when we do talk we get mad quickly and don’t listen/learn.  So understanding this type of anger is crucial for understanding communication.


It's well known, and academically verified, that this is indeed what's happened in party politics in the US in recent decades - opposing partisans actually dislike each other more than ever.  The standard jargon for this now is 'affective polarization'.  Actually looks like this is the type of polarization where the real action is since it’s much less clear to what extent we’ve polarized re policy/ideology preferences- though it is clear that politician behavior has diverged - R's and D's in Congress vote along opposing party lines more and more over time.  For anyone who doubts this, take a look at the powerful graphic in inset to the left, stolen from this recent article.

So—why do we hate each other so much? 

Full disclosure, I'm an outsider to this topic.  I'm an economist by training, affiliation, methods.  Any clarification/feedback on what I say here is very

The fingerprint(s) of polarization in Congress....


Anyway my take from the outside is the poli-sci papers on this topic focus on two things, "social distance" and new media.  Social distance is the social-psych idea that we innately dislike those we feel more "distance" from (which can be literal or figurative).  Group loyalty, tribalism etc.  Maybe distance between partisans has grown as partisan identities have strengthened and/or because of gridlock in DC and/or real/perceived growth in the ideological gap between parties.  New media includes all sorts of things, social media, blogs, cable news, political advertising, etc.  The idea here is we're exposed to much more anti-out party info than before and natural this would sink in to some extent.

There's a related but distinct and certainly important line of work in moral psychology on this topic – if you’re reading this there’s a very good chance you’re familiar with Jonathan Haidt's book The Righteous Mind in particular.  He doesn't use the term social distance but talks about a similar (equivalent?) concept—differences between members of the parties in political-moral values and the evolutionary explanation for why these differences lead to inter-group hostility.

So—this is a well-studied topic that we know a lot about.  Still, we have a ways to go toward actually solving the problem.  So there’s probably more to be said about it.

Here’s my angle: the social distance/Haidtian and even media effects literatures seem to take it as self-evident that distance causes dislike.  And the mechanism for this causal relationship is often treated as black box.  And so, while it’s often assumed that this dislike is “wrong” and this assumption seems quite reasonable—common sense, age-old wisdom etc tell us that massive groups of people can’t all be so bad and so something is seriously off when massive groups of people hate each other—this assumption of wrongness is both theoretically unclear and empirically far from proven.

Citizens of the the Liberal Republic of Science-- unite against partyism!But in reality when we dislike others, even if just because they’re different, we usually think (perhaps unconsciously) they’re actually “bad” in specific ways.  In politics, D’s and R’s who dislike each other do so (perhaps ironically) because they think the other side is too partisan—i.e. too willing to put their own interests over the nation’s as a whole.  Politicians are always accusing each other of “playing politics” over doing what’s right.  (I don’t know of data showing this but if anyone knows good reference(s) please please let me know.)

That is, dislike is not just “affective” (feeling) but is “cognitive” (thinking) in this sense.  And cognitive processes can of course be biased.  So my claim is that this is at least part of the sense in which out-party hate is wrong—it’s objectively biased.  We think the people in the other party are worse guys than they really are (by our own standards).  In particular, more self-serving, less socially minded. 

This seems like a non-far-fetched claim to me, maybe even pretty obviously true when you hear it.  If not, that’s ok too, that makes the claim more interesting.  Either way, this is not something these literatures (political science, psychology, communications) seem to talk about.  There is certainly a big literature on cognitive bias and political behavior, but on things like extremism, not dislike.

Here come the semi-shameless[2] plugs.  This post has already gotten longer than most I’m willing to read myself so I’ll make this quick.

In one recent paper, I show that ‘unrelated’ cognitive bias can lead to (unbounded!) cognitive (Bayesian!) dislike even without any type of skewed media or asymmetric information. 

In another, I show that people who overestimate what they know in general (on things like the population of California)--and thus are more likely to be overconfident in their knowledge in general, both due to, and driving, various more specific cognitive biases--also tend to dislike the out-party more (vs in-party), controlling carefully for one’s own ideology, partisanship and a bunch of other things.

Feedback on either paper is certainly welcome, they are both far from published.

So—I’ve noted that cognitive bias very plausibly causes dislike, and I’ve tried to provide some formal theory and data to back this claim up and clarify the folk wisdom that if we understood each other better, we wouldn’t hate each other so much.  And dislike causes (exacerbates) bias (in knowledge, about things like climate change, getting back to the main subject of this blog).  Why else does thinking of dislike in terms of bias matter?  Two points.

1) This likely can help us to understand polarization in its various forms better.  The cognitive bias literature is large and powerful, including a growing literature on interventions (nudges etc).  Applying this literature could yield a lot of progress. 

2) Thinking of out-party dislike (a.k.a. partyism) as biased could help to stigmatize and as a result reduce this type of behavior (as has been the case for other 'isms').  If people get the message that saying “I hate Republicans” is unsophisticated (or worse) and thus uncool, they’re going to be less likely to say it. 

For a decentralized phenomenon like affective polarization, changing social norms may ultimately be our best hope. 


[1] Ed.: Okay, time to come clean. What he's alluding to is that I've been using M Turk workers to ghost write my blog posts for last 6 mos. No one having caught on, I’ve now decided that it is okay after all to use M Turk workers in studies of politically motivated reasoning.

[2] Ed.: Yup, for sure he is not trying to imitate me. What’s this “semi-” crap?


Hey, everyone! Try your hand at graphic reporting and see if you can win the Gelman Cup!


Former Freud expert & current stats legend  Andrew Gelman posted a blog (one he likely wrote in the late 1990s; he stockpiles his dispatches, so probably by the time he sees mine he'll have completely forgotten this whole thing, & even if he does respond I’ll be close to 35 yrs. old  by then & will be interested in other things like drinking and playing darts) in which he said he liked one of my graphics!

Actually, he said mine was “not wonderful”—but that it kicked the ass of one that really sucked!


Alright, alright.

Celebration over.

Time to get back the never-ending project of self-improvement that I’ve dedicated my life too.

The question is, How can I climb to that next rung—“enh,” the one right above “not wonderful”?

I’m going to show you a couple of graphics. They aren’t the same ones Gelman showed but they are using the same strategy to report more interesting data.  Because the data are more interesting (not substantively, but from a graphic-reporting point of view), they’ll supply us with even more motivation to generate a graphic-reporting performance worthy of an “enh”—or possibly even a “meh,” if we can get really inspired here.

I say we because I want some help.  I’ve actually posted the data & am inviting all of you—including former Freud expert & current stats legend Gelman (who also is a bully of WTF study producers , whose only recourse is to puff themselves up to look really big, like a scared cat would)—to show me what you’d do differently with the data.

Geez, we’ll make it into a contest, even!  The “Gelman Graphic Reporting Challenge Cup,” we’ll call it, which means the winner will get—a cup, which I will endeavor get Gelman himself to sign, unless of course he wins, in which case I’ll sign it & award it to him!

Okay, then. The data, collected from a large nationally representative sample, shows the relationship between religiosity, left-right political outlooks, and climate change.  

It turns out that religiosity and left-right outlooks actually interact. That is, the impact of one on the likelihood someone will report “believing in” human-caused climate change depends on the value of the other.

Wanna see?? Look!!

That’s  a scatter plot with left_right, the continuous measure of political outlooks, on the x-axis, and “belief in human-caused climate change” on the right.

Belief in climate change is actually a binary variable—0 for “disbelief” and 1 for “belief.”

But in order to avoid having the observations completely clumped up on one another, I’ve “jittered” them—that is, added a tiny bit of random noise to the 0’s and 1’s (and a bit too for the left_right scores) to space the observations out and make them more visible.

Plus I’ve color-coded them based on religiosity!  I’ve selected orange for people who score above the mean on the religiosity scale and light blue for those who score below the mean. That way you can see how religiosity matters at the same time that you can see that political outlook matters in determining whether someone believes in climate change.

Or at least you can sort of see that. It’s still a bit blurry, right?

So I’ve added the locally weighted regression lines to add a little resolution.  Locally weighted regression is a nonmodel way to model the data. Rather than assuming the data fit some distributional form (linear, sigmoidal, whatever) and then determining the “best fitting” parameters consistent with that form, the locally weighted regression basically slices the x-axis predictor  into zillions of tiny bits, with individual regressions being fit over those tiny little intervals and then stitched together.

It’s the functional equivalent of getting a running tally of the proportion of observations at many many many contiguous points along left_right (and hence my selection of the label “proportion agreeing” on the y-axis, although “probability of agreeing” would be okay too; the lowess regression can be conceptualized as estimating that). 

What the lowess lines help us “see” is that in fact the impact of political outlooks is a bit more intense for subjects who are “low” in religiosity. The slope for their S-shaped curve is a bit steeper, so that those at the “top,” on the far left, are more likely to believe in human-caused climate change. Those at the “bottom,” on the right, seem comparably skeptical.

The difference in those S-shaped curves is what we can model with a logistic regression (one that assumes that the probability of “agreeing” will be S-shaped in relation to the x-axis predictor).  To account for the possible difference in the slopes of the curve, the model should include a cross-product interaction term in it that indicates how differences in religiosity affect the impact of differences in political outlooks in “believing” in human-caused climate change.

Okay, it's important to report this. But if someone gives you *nothing* more than a regression output when reporting their data ... well, make them wish they had competed for & won a Gelman Cup...I’ve fit such a model, the parameters of which are in the table in the inset.

That  regression actually corroborates, as it were, what we “saw” in the raw data: the parameter estimates for both religiosity and political outlooks “matter” (they have values that are practically and statistically significant), and so does the parameter estimate for the cross-product interaction term.

But the output doesn’t in itself doesn’t show us what the estimated relationships  look like. Indeed, precisely because it doesn’t, we might get embarrassingly carried away if we started crowing about the “statistically significant” interaction term and strutting around as if we had really figured out something important. Actually, insisting that modelers show their raw data is the most important way to deter that sort of obnoxious behavior but graphic reporting of modeling definitely helps too.

So let’s graph the regression output:


Here I’m using the model to predict how likely a person who is relatively “high” in religiosity—1 SD above the population mean—and a person who is relatively “low”—1 SD below the mean—to agree that human-caused climate change is occurring.  To represent the model’s measurement precision, I’m using solid bars—25 of them evenly placed—along the x-axis.

Well, that’s a model of the raw data.

What good is it? Well, for one thing it allows us to be confident that we weren’t just seeing things.  It looked like there was  a little interaction between religiosity and political outlooks. Now that we see that the model basically agrees with us—the parameter that reflects the expectation of an interaction is actually getting some traction when the model is fit to the data—we can feel more confident that’s what the data really are saying (I think this is the right attitude, too, when one hypothesized the observed effect as well as when one is doing exploratory analysis).  The model disciplines the inference, I’d say, that we drew from just looking at the data.

Also, with a model, we can refine, extend,  and appraise  the inferences we draw from the data. 

You might say to me, e.g., “hey, can you tell me  how much more likely a nonreligious liberal Democrat to accept human-caused climate change than a religious one?”

I’d say, well, about “12%, ± 6, based on my model.”  I’d add, “But realize that even the average religious liberal Democrat is awfully likely to believe in human-caused climate change—73%, ± 5%, according to the model.”

“So there is an interaction between religiosity & political outlooks, but it's nothing to get excited about--the way somone trained only to look at  the 'significance' of regression model coefficients might -- huh?” you’d say.

“Well, that’s my impression as well. But others might disagree with us. They can draw their own conclusions about how important all of this is, if they look at the data and use the model to make sense of it .”

Or whatever!


What’s Gelman’s reservation? How come my graphic rates only “not awful” instead of “enh” or “meh”?

He says “I think all those little bars are misleading in that they make it look like it’s data that are being plotted, not merely a fitted model . . . .”

Hm. Well, I did say that the graphic was a fitted model, and that the bars were 0.95 CIs.

The 0.95 CIs *could* mislead people --if they were being generated by a model that didn't fairly convey what the actual data look like. But that's why one starts by looking at, and enabling others to see, what the raw data “look like.”

But hey--I don’t want to quibble; I just want to get better!

So does anyone have a better idea about how to report the data?

If so, speak up. Or really, much much better, show us what you think is better.

I’ve posted the data.  The relevant variables are “left_right,” the continuous political outlook scale; “religiosity,” the continuous religiosity scale; and “AGW,” belief in climate human-caused-climate change =1 and disbelief = 0. I’ve also included “relig_category,” which splits the subjects at the mean on religiosity (0 = below the mean, 1 = above; see note below if you were using "relig" variable).  Oh, and here's my Stata .do file, in case you want to see how I generated the analyses reported here.

So ... either link to your graphics in the comments thread for this post or send them to me by email.  Either way, I’ll post them for all to see & discuss.

And remember, the winner—the person who graphically reports the data in a way that exceeds “not wonderful” by the greatest increment-- will get the Gelman Cup! 


Another “Scraredy-cat risk disposition”™ scale "booster shot": Childhood vaccine risk perceptions

You saw this coming I bet.

I would have presented this info in "yesterday's" post but I'm mindful of the groundswell of anxiety over the number of anti-BS inoculations that are being packed into a single data-based booster shot, so I thought I'd space these ones out.

"Yesterday," of course, I introduced the new CCP/Annenberg Public Policy Center “Scaredy-cat risk disposition”™ measure.  I used it to help remind people that the constant din about "public conflict" over GM food risks--and in particular that GM food risks are politically polarizing-- is in fact just bull shit.  

The usual course of treatment to immunize people against such bull shit is just to show that it's bull shit.  That goes something  like this:


The  “Scraredy-cat risk disposition”™  scale tries to stimulate people’s bull shit immune systems by a different strategy. 

Rather than showing that there isn’t a correlation between GM food risks and any cultural disposition of consequence (political orientation is just one way to get at the group-based affinities that inform people’s identities; religiosity, cultural worldviews, etc.,  are others—they all show the same thing w/r/t GM food risk perceptions), the  “Scraredy-cat risk disposition”™ scale shows that there is a correlation between it and how afraid people (i.e, the 75%-plus part of the population that has no idea what they are being asked about when someone says, “are GM foods safe to eat, in your opinion?”) say they are of GM foods and how afraid they are of all sorts of random ass things (sorry for technical jargon) including,

  • Mass shootings in public places

  • Armed carjacking (theft of occupied vehicle by person brandishing weapon)

  • Accidents occurring in the workplace

  • Flying on a commercial airliner

  • Elevator crashes in high-rise buildings

  • drowning of children in swimming pools

A scale comprising these ISRPM items actually coheres!

But what a high score on it measures, in my view, is a not a real-world disposition but a survey-artifact one that reflects a tendency (not a particularly strong one but one that really is there) to say “ooooo, I’m really afraid of that” in relation to anything a researcher asks about.

The “Scraredy-cat risk disposition”™  scale “explains” GM food risk perceptions the same way, then, that it explains everything,

which is to say that it doesn’t explain anything real at all.

So here’s a nice Bull Shit test.

If variation in public risk perceptions are explained just as well or better by scores on the “Scraredy-cat risk disposition”™  scale than by identity-defining outlooks & other real-world characteristics known to be meaningfully related to variance in public perceptions of risk, then we should doubt that there really is any meaningful real-world variance to explain. 

Whatever variance is being picked up by these legitimate measures is no more meaningful than the variance picked up by a randm-ass noise detector. 

Necessarily, then whatever shred of variance they pick up, even if "statistically significant" (something that is in fact of no inferential consequence!) cannot bear the weight of sweeping claims about who— “dogmatic right wing authoritarians,” “spoiled limousine liberals,” “whole foodies,” “the right,” “people who are easily disgusted” (stay tuned. . .), “space aliens posing as humans”—etc. that commentators trot out to explain a conflict that exists only in “commentary” and not “real world” space.

Well, guess what? The “Scraredy-cat risk disposition”™  scale “explains” childhood vaccine risk perceptions as well as or better than the various dispositions people say “explain” "public conflict" over that risk too.

Indeed, it "explains" vaccine-risk perceptions as well (which is to say very modestly) as it explains global warming risk percepitons and GM food risk perceptions--and any other goddam thing you throw at it.

See how this bull-shit immunity booster shot works?

The next time some know it all says, "The rising tide of anti-vax sentiment is being driven by ... [fill in bull shit blank]," you say, "well actually, the people responsible for this epidemic of mass hysteria are the ones who are worried about falling down elevator shafts, being the victim of a carjacking [how 1980s!], getting flattened by the detached horizontal stabilizer of a crashing commercial airliner, being mowed down in a mass shooting, getting their tie caught in the office shredder, etc-- you know those guys!  Data prove it!"

It's both true & absurd.  Because the claim that there is meaningful public division over vaccine risks is truly absurd: people who are concerned about vaccines are outliers in every single meaningful cutlural group in the U.S.

Click to see "falling" US vaccination rates...Remember, we have had 90%-plus vaccinate rates on all childhood immunizations for well over a decade.

Publication of the stupid Wakefield article had a measurable impact on vaccine behavior in the UK and maybe elsewhere (hard to say, b/c on the continent in Europe vaccine rates have not been as high historically anyway), but not the US!  That’s great news!

In addition, valid opinion studies find that the vast majority of Americans of all cultural outllooks (religious, political, cultural, professional-sports team allegiance, you name it) think childhood vaccines are the greatest invention since . . . sliced GM bread!  (Actually, wheat farmers, as I understand it, don’t use GMOs b/c if they did they couldn’t export grain to Europe, where there is genuine public conflict over GM foods).

Yes, we do have pockets of vaccine-hesitancy and yes they are a public health problem.

But general-population surveys and experiments are useless for that—and indeed a wast of money and attention.  They aren't examining the right people (parents of kids in the age range for universal vaccination).  And they aren't using measures that genuine predict the behavior of interest.

We should be developing (and supporting researchers doing the developing of) behaviorally validated methods for screening potentially vaccine  hesitant parents and coming up with risk-counseling profiles speciifically fitted to them.

And for sure we should be denouncing bull shit claims—ones typically tinged with group recrimination—about who is causing the “public health crisis” associated with “falling vaccine rates” & the imminent “collapse of herd immunity,” conditions that simply don’t exist. 

Those claims are harmful because they inject "pollution" into the science communication environment including  confusion about what other “ordinary people like me” think, and also potential associations between positions that genuinely divide people—like belief in evolution and positions on climate change—and views on vaccines. If those take hold, then yes, we really will have a fucking crisis on our hands.

If you are emitting this sort of pollution, please just stop already!

And the rest of you, line up for a  “Scraredy-cat risk disposition”™  scale booster shot against this bull shit. 

It won’t hurt, I promise!  And it will not only protect you from being misinformed but will benefit all the rest of us too by helping to make our political discourse less hospitable to thoughtless, reckless claims that can in fact disrupt the normal processes by which free, reasoning citizens of diverse cultural outlooks converge on the best available evidence.

On the way out, you can pick up one of these fashionable “I’ve been immunized by  the ‘Scraredy-cat risk disposition’™  scale against evidence-free bullshit risk perception just-so stories” buttons and wear it with pride!


Scientists discover source of public controversy on GM food risks: bitter cultural division between scaredy cats and everyone else!

Okay. Time for a “no, GM food risks are not politically polarizing—or indeed a source of any meaningful division among members of the public” booster shot.

Yes, it has been administered 5000 times already, but apparently, it has to be administered about once every 90 days to be effective.

Actually, I’ve monkeyed a bit with the formula of the shot to try to make it more powerful (hopefully it won’t induce autism or microcephaly but in the interest of risk-perception science we must take some risks).

We are all familiar (right? please say “yes” . . .) with this:

It’s just plain indisputable that GM food risks do not divide members of the U.S. general public along political linies. If you can’t see the difference between these two graphs, get your eyes or your ability to accept evidence medically evaluated.

But that’s the old version of the booster shot!

The new & improved one uses what I’m calling the “scaredy-cat risk disposition” scale!

That scale combines Industrial Strength Risk Perception Measure (ISRPM) 0-7 responses to an eclectic -- or in technical terms "random ass" -- set of putative risk sources. Namely:

MASSHOOT. Mass shootings in public places

CARJACK. Armed carjacking (theft of occupied vehicle by person brandishing weapon)

ACCIDENTS. Accidents occurring in the workplace

AIRTRAVEL. Flying on a commercial airliner

ELEVATOR. Elevator crashes in high-rise buildings

KIDPOOL. Accidental drowning of children in swimming pools

Together, these risk perceptions form a reliable, one-dimensional scale (α = 0.80) that is distinct from fear of environmental risks or of deviancy risks (marijuana legalization, prostitution legalization, pornography distribution, and sex ed in schools).

Scaredy-cat is normally distributed, interestingly.  But unsurprisingly, it isn’t meaningfully correlated with right-left political predispositions.

So what is the relationship between scaredy-cat risk dispositions & GM food risk perceptions? Well, here you go:

Got it?  Political outlooks, as we know, don’t explain GM food risks, but variance in the sort of random-ass risk concerns measured by the Scaredy-cat scale do, at least to a modest extent.

We all are famaliar with this fundamental "us vs. them" division in American life.  

On the one hand, we have those people who who walk around filled with terror of falling down elevator shafts, having their vehicles carjacked, getting their arms severed by a workplace “lathe,” and having their kids fall into a neighbor’s uncovered swimming pool and drowning.  Oh—and being killed by a crashing airplane either b/c they are a passenger on it or b/c they are  the unlucky s.o.b. who gets nailed by a piece of broken-off wing when it  comes hurtling to the ground.

On the other, there are those who stubbornly deny that any of these  is anything to worry about.

Bascially, this has been the fundamenal divide in American political life since the founding: Anti-federalist vs. Federaliststs, slaveholders vs. abolitionists, isolationists vs. internationalists, tastes great vs. less filling.

Well, those same two groups are the ones driving all the political agitation over GM foods too!

... Either that or GM food risk perceptions are just meaningless noise. Those who score high on the Scaredy-cat scale are the people who, without knowing what GM foods are (remember 75% of people polled give the ridiculous answer that they haven’t ever eaten any!), are likely to say they are more worried about them in the same way they are likely to say they are worrid about any other random-ass thing you toss into a risk-perception survey.

If the latter interpretation is right, then the idea that the conflict between the scaredy-cats and the unscaredy-cats is of any political consequence for the political battle over GM foods is obviously absurd.  

If that were a politically consequential division in public opinion, Congress would not only be debating preempting state GM food labels but also debating banning air travel, requiring swimming pool fences (make the Mexicans pay for those too!), regulations for mandatory trampolines at the bottom of elevator shafts, etc.

People don’t have opinions on GM foods. They eat them.

The political conflict over GM foods is being driven purely by interest group activity unrelated to public opinion.

Got it?

Good.  See you in 90 days.

Oh, in case you are wondering, no, the division between scaredy-cats and unscaredy-cats is not the source of cultural conflict in the US over climate change risks.

You see, there really is public division on global warming. 

GM foods are on the evidence-free political commentary radar screen but not the public risk-perception one.

That's exactly what the “scaredy-cat risk disposition” scale helps to illustrate.



New "strongest evidence yet" on consensus messaging!

Yanking me from the jaws of entropy just before they snapped permanently shut on my understanding of the continuing empirical investigation of "consensus messaging," a friend directed my attention to a couple of cool recent studies I’d missed.

For the 2 members of this blog's list of 14 billion regular subscribers who don't know," consensus messaging” refers to a social-marketing device that involves telling people over & over & over that “97% of scientists” accept human-caused global warming.  The proponents of this "strategy" believe that it's the public's unawareness of the existence of such consensus that accounts for persistent political polarization on this issue.

The first new study that critically examines this position is Cook, J. & Lewandowsky, S., Rational Irrationality: Modeling Climate Change Belief Polarization Using Bayesian Networks, Topics in Cognitive Science 8, 160-179 (2016).

Lewandowsky was one of the authors of an important early study (Lewandowsky, S., Gignac, G.E. & Vaughan, S, The pivotal role of perceived scientific consensus in acceptance of science, Nature Climate Change 3, 399-404 (2012)), that found that advising people that a “97% consensus” message increased their level of acceptance of human-caused climate change.

It was a very decent study, but relied on a convenience sample of Australians, the most skeptical members of which were already convinced that human activity was responsible for global warming.

Cook & Lewandowsky use representative samples of Australians and Americans.  Because climate change is a culturally polarizing issue, their focus, appropriately, was on how consensus messaging affects individuals of opposing cultural predispositions toward global warming.

Take a look at C&L's data. Nice graphic reporting!They report (p. 172) that “while consensus information partially neutralized worldview [effects] in Australia, in replication of Lewandowsky, Gignac, et al. (2013), it had a polarizing effect in the United States.”

“Consensus information,” they show, “activated further distrust of scientists among Americans with high free-market support” (p. 172). 

There was a similar “worldview backfire effect” (p. 161) on the belief that global warming is happening and caused by humans among Americans with strong conservative (free-market) values,” although not among Australians (pp. 173-75).

Veeeery interesting.

The other study is Deryugina, T. & Shurchkov, O, The Effect of Information Provision on Public Consensus about Climate Change. PLOS ONE 11, e0151469 (2016).

D&S did two really cool things.

First, they did an experiment to assess how a large (N = 1300) sample of subjects responded to a “consensus” message.” 

They found that exposure to such a message increased subjects’ estimate of the percentage of scientists who accept human-caused global warming.

However, they also found that  [the vast majority of] subjects did not view the information as credible. [see follow up below]

  “Almost two-thirds (65%) of the treated group did not think the information from the scientist survey was accurately representing the views of all scientists who were knowledgeable about climate change,” they report.

This finding matches one from a  CCP/Annenberg Public Policy Center experiment, results of which I featured a while back, that shows that the willingness of individuals to believe "97% consensus" messages is highly correlated with their existing beliefs about climate change.

In addition, D&S find that relative to a control group, the message-exposed subjects did not increase their level of support for climate mitigation policies.  

Innovatively, D&S measured this effect not only attitudinally, but behaviorally: subjects in the study were able to indicate whether they were willing to donate whatever money they were eligible to win in a lottery to an environmental group dedicated to “prevent[ing] the onset of climate change through promoting energy efficiency.”

In this regard, D&S report “we find no evidence that providing information about the scientific consensus affects policy preferences or raises willingness to pay to combat climate change” (p. 7).

Subjects exposed to the study’s consensus message were not significantly more likely—in a statistical or practical sense—to revise their support for mitigation policies, as measured by either the attitudinal or behavioral measures feature in the D&S design.

This is consistent with a model where people look to climate scientists for objective scientific information but not public policy recommendations, which also require economic (i.e. cost-benefit) and ethical considerations,” D&S report (p. 7).

Second, D&S did a follow-up survey, in this part of the study, they re-surveyed subjects who received a consensus message to the consensus message six-months after the initial message exposure.

Still no impact on the willingness of message exposed subjects to support mitigation policies (indeed, all the results were negative, Tbl. 7,albeit “ns”).

In addition, whereas immediately after message exposure, subjects had reported higher responses on 0-100 measures of their perceptions of the likelihood of temperature increases by 2050, D&S report that they “no longer f[ound] a significant effect of information”—at least for the most part. 

Actually, there was significant increase in responses to items soliciting belief that temperatures would increase by more than 2.5 degrees Celsius by that time -- and that they would decrease by that amount.

D&S state they are “unable to make definitive conclusions about the long-run persistence of informational effects” (p. 12).  But to the extent that there weren’t any “immediate” ones on support for mitigation policies, I’d say that the absence of any in the six-month follow up as well rules out the possibility that the effect of the message just sort of percolates in subjects' psyches, blossoming at some point down the road into full-blown support for aggressive policy actions on climate change.

In my view, none of this implies that nothing can be done to promote support for collective action on climate change. Only that one has to do something other-- something much more meaningful-- than march around incanting "97% of scienitists!"

But the point is, these are really nice studies, with commendably clear and complete reporting of their results. The scholars who carried them out offer their own interpretations of their data-- as they should-- but demonstrate genuine commitment to making it possible for readers to see their data and draw their own inferences. (One can download the D&S data, too, since they followed PLOS ONE policy to make them available upon publication.)

Do these studies supply what is now the “strongest evidence to date” on the impact of consensus-messaging? 

Sure, I’d say so-- although in fact I think there's nothing in the previous "strongest evidence to date" that would have made these findings at all unexpected.

What do you think?


First things first: the science of normal (nonpathological) science communication

From something I'm working on...

The priority of the science of normal science science communication 

The source of nearly every science-communication misadventure can be traced to a single mistake: the confusion of the processes that make science valid  for the ones that vouch for the validity of it.  As Popper (1960) noted, it is naïve, to view the “truth as manifest” even after it has been ascertained by science. The scientific knowledge that individuals rely on in the course of their everyday lives is far too voluminous, far too specialized for any—including a scientist—to comprehend or verify for herself.  So how do people manage to pull it off?  What are social cues they rely on to distinguish the currency of scientific knowledge from the myriad counterfeit alternatives to it? What processes generate those cues? What are the cognitive faculties that determine how proficiently individuals are able to recognize and interpret them? Most importantly of all, how do the answers to these questions vary--as they must in modern democratic societies--across communities of culturally diverse citizens, whose members are immersed in a plurality of parallel systems suited for enabling them to identify who knows what about what? These questions not only admit of scientific inquiry; they demand it.  Unless we understand how ordinary members of the public ordinarily do manage to converge on the best available evidence, we will never fully understand why they occasionally do not, and what can be done to combat these noxious sources of ignorance.



Popper, K.R. On the Sources of Knowledge and of Ignorance. in Conjectures and Refutations 3-40 (Oxford University Press London, 1960).



"Now I'm here ... now I'm there ...": If you look, our dualistic identity-expressive/science-knowledge-acquiring selves go through only one slit

From correspondence with a thoughtful person: on the connection between the "toggling" of identity-expressive and science-knowledge-revealing/acquiring information processing & the "science communication measurement problem."

So tell me what you think of this:


I think it is a variant of [what Lewandowsky & Kirsner (2000) call] partitioning.

When the "according to climate scientists ..." prefix is present, the subjects access "knowledge of science"; when it is not, they access "identity-enabling knowledge" -- or some such.  

Why do I think that?

Well, as you know,  it's not easy to do, but it is possible to disentangle what people know from who they are on climate change with a carefully constructed climate-science literacy test.

Of course, most people aren't very good climate-science literacy test takers ("they can tell us what they know -- just not very well!"). The only people who are particularly good are those highest in science comprehension.

Yet consider this!

"WTF!," right?

I had figured the "person" who might help us the most to understand this sort of thing was the high science-comprehension "liberal/Democrat."

She was summoned, you see, because some people thought that the reason the high science-comprehension "conservative/republican"  "knows" climate change will cause flooding when the prefix is present yet "knows" it won't otherwise is that  he simply "disagrees" with climate scientists; b/c he knows they are corrupt, dishonest, stupid commies" & the like.

I don't think he'd say that, actually. But I've never been able to find him to ask...

So I "dialed" the high-science comprehension "liberal/democrat."

When you answer " 'false' " to " 'according to climate scientists,  nuclear generation contributes to global warming,'" I asked her, "are you thinking, 'But I know better--those corrupt, stupid, dishonest commies'  or the like?"

"Don't be ridiculous!," she said. "Of course climate scientists are right about that-- nuclear power doesn't emit CO2 or any other greenhouse gas. "  "Only an idiot," she added, "would see climate scientists as corrupt, stupid, dishonest etc."  A+!

So I asked her why, then, when we remove the prefix, she does say that nuclear power causes  global warming.

She replied: "Huh? What are you talking about?"

"Look," I said, "it's right here in the data: the 'liberal democrats' high enough in science comprehension to know that nuclear power doesn't cause global warming 'according to climate scientists' are the people most likely to answer 'true' to the statement 'nuclear power generation contributes to global warming' when one removes the 'according to climate scientists' prefix. "

"Weird," she replied.  "Who the hell are those people? For sure that's not me!"

Here's the point: if you look, the high-science comprehension "liberal/democrat" goes through only one slit. 

If you say, "according to climate scientists," you see only her very proficient science-knowledge acquirer self.

But now take the prefix away and "dial her up" again, and you see someone else--or maybe just someone's other self.

"That's a bogus question," she insists. "Nuclear power definitely causes global warming; just think a bit harder-- all the cement . . . .  Hey, you are a shill for the nuclear industry, aren't you!"



... She has been forced to be her (very proficient) identity-protective self.

And so are we all by the deformed political discourse of climate change ...

"Here I stand . . . "


Lewandowsky, S. & Kirsner, K. Knowledge partitioning: Context-dependent use of expertise. Memory & Cognition 28, 295-305 (2000).


Weekend update: Priceless



Three pretty darn interesting things about identity-protective reasoning (lecture summary, slides)

Got back to New Haven CT Wed. for first time since Jan. to give a lecture to cognitive science program undergraduates.  Since the lecture was on the science communication undergraduates.

The lecture (slides here) was on the Science of Science Communication. I figured the best way to explain what it was was just to do it.  So I decided to present data on three cool things:

1. MS2R (aka, "motivated system 2 reasoning").

Contrary to what many decision science expositors assume, identity-protective cognition is not attributatle to overreliance on heuristic, "System 1" reasoning.  On the contary, studies using a variety of measures and using both observational and experimental methods support the conclusion that the effortful, conscious reasoning associated with "System 2" processing magnify the disposition to selectively credit and dismssis evidence in patterns that conform one's assessment of contested societal risks into alignment with those of other with whom shares important group ties.

Why? Because it's rational to process information this way: the stake ordinary indidivudals have in forming beliefs that convincingly evince their group commitments is bigger than the stake they have in forming "correct" understandings of facts on risks that nothing they personally do--as consumers, voters, tireless advocates in blog post comment sections etc--will materially affect.

If you want to fix that--and you should; when everyone processes information this way, citizens in a diverse democratic society are less likely to converge on valid scientific evidence essential to their common welfare--then you have to eliminate the antagonistic social meanings that turn positions on disputed issues of fact into badges of group membership and loyalty.

2. The science communication measurement problem

There are several

One is, What does belief in "human caused climate change" measure?

Look at this great latent-variable measure of ideology: just add belief in climate change, belief nuclear power causes global warming, belief global warming causes flooding to liberal-conserative ideology & party identification!The answer is, Not what you know but who you are.

A second is, How can we measure what people know about climate change independently of who they are?

The answer is, By unconfounding identity and knowledge via appropriately constructed "climate science literacy" measures, like OCSI_1.0 and _2.0.

The final one is, How can we unconfound identity and knowledge from what politics meaures when culturally diverse citizens address the issue of climate change?

The answer is ... you tell me, and I'll measure.

3. Identity-protective reasoning and professional judgment

Is the legal reasoning of judges affected by identity-protective cogniton?

Not according to an experimental study by the Cultural Cogniton Project, which found that judges who were as culturally divided as members of the public on the risks posed by climate change, the dangers of legalizing marijuana, etc., nevertheless converged on the answers to statutory intepretation problems that generated intense motivated-reasoning effects among members of the public.

Lawyers also seemed largely immune to identity-protective reasoning in the experiment, while law students seemed to be affected by an intermediate degree.

The result was consistent with the hypothesis that professional judgment--habits of mind that enable and motivate recogniton of considerations relevant to making expert determinations--largely displaces identity-protective cognition when specialists are making in-domain determinations.

Combined with other studies showing how readily members of the public will display identity-protective reasoninng when assessing culturally contested facts, the study suggests that judges are likely more "neutral" than citizens perceive.

But precisely because citizens lack the professional habits of mind that make the neutrality of such decisons apparent to them, the law will have a "neutrality communication problem" akin to the "science communication problem" that scientists have in communicating valid science to private citizens who lack the professional judgment to reccognize the same.



Hey--want your own "OSI_2.0" dataset to play with? Here you go!

I've uploaded the dataset, along with codebook, for the data featured in Kahan, D.M. "Ordinary Science Intelligence": A Science Comprehension Measure for Study of Risk and Science Communication, with Notes on Evolution and Climate Change. J. Risk Res.  (in press). Enjoy!





Weekend update: modeling the impact of the "according to climate scientists prefix" on identity-expressive vs. science-knowledge revealing responses to climate science literacy items

I did some analyses to help address issues that arose in an interesting discussion with @dypoon about how to interpret the locally weighted regression outputs featured in "yesterday's" post. 

Basically, the question is what to make of the respondents at the very highest levels of Ordinary Science Intelligence

When the prefix "according to climate scientists" is appended to the items, those individuals are the most likely to get the "correct" response, regardless of their political outlooks. That's clear enough.

It's also bright & clear that when the prefix is removed, subjects at all levels of OSI are more disposed to select the identity-expressive answer, whether right or wrong. 

What's more those highest in OSI seem even more disposed to select the identity-expressive "wrong" answer than those modest in that ability.  Insfar as they are the ones most capable of getting the right answer when the prefix is appended, they necessarily evince the strongest tendency to substitute the incorrect identity-expressive for the correct, science-knowledge-evincing response when the prefix is removed.

But are those who are at the very tippy top of the OSI hierarchy resisting the impulse (or the consciously perceived opportunity) to respond in an identity-protective manner--by selecting the incorrect but ideologically congenial answer-- when the prefix is removed?  Is that what the little little upward curls mean at the far right end of the dashed line for right-leaning subjects in "flooding" and for left-leaning ones in "nuclear"?

@Dypoon seems to think so; I don't.  He/she sensed signal; I caught the distinct scent of noise.

Well, one way to try to sort this out is by modeling the data.

The locally weighted regression just tells us the mean probabilities of "correct" answers at tiny little increments of OSI. A logistic regression model can show us how the precision of the estimated means--the information we need to try to ferret out signal from noise-- is affected by the number of observations, which necessarily get smaller as one approaches the upper end of the Ordinary Science Intelligence scale.

Here are a couple of ways to graphically display the models (nuclear & flooding). 

This one plots the predicted probability of correctly answering the items with and without the prefix for subjects with the specified political orientations as their OSI scores increase: 


This one illustrates, again in relation to OSI, how much more likely someone is to select the incorrect, identity-expressive response for the no-prefix version than he or she is to select the incorrect response for the prefix version:

The graphic shows us just how much the confounding of identity and knowledge in a survey item can distort measurement of how likely an individual is to know climate-science propositions that run contrary to his or her ideological predisposition on global warming.

I think the results are ... interesting.

What do you think?

To avoid discussion forking (the second leading cause of microcephaly in the Neterhlands Antilles), I'm closing off comments here.  Say your piece in the thread for "yesterday's" post.


Toggling the switch between cognitive engagement with "America's two climate changes"--not so hard in *the lab*

So I had a blast last night talking about “America’s 2 climate changes” at the 14 Annual “Climate Predication Applications Workshop,” hosted by NOAA’s National Weather Service Climate Services Branch, in Burlington Vermont (slides here).

It’s really great when after a 45-minute talk (delivered in a record-breaking 75 mins) a science-communication professional stands up & crystallizes your remarks in a 15-second summary that makes even you form a clearer view of what you are trying to say! Thanks, David Herring!

In sum, the “2 climate changes” thesis is that there are two ways in which people engage information about climate change in America: to express who they are as members of groups for whom opposing positions on the issue are badges of membership in one or another competing cultural group; and to make sense of scientific information that is relevant to doing things of practical importantance—from being a successful farmer to protecting their communities from threats to vital natural resources to exploiting distinctive commercial opportunities—that are affected by how climate is changing as a result of the influence of humans on the environment.

I went through various sorts of evidence—including what Kentucky Farmer has to say about “believing in climate change” when he is in his living room versus when he is on his tractor.

Also the inspired leadership in Southeast Florida, which has managed to ban conversation of the “climate change” that puts the question “who are you, whose side are you on?” in order to enable conversation of the “climate change” which asks “what do we know, what should we do?”

But I also featured some experimental data that helped to show how one can elicit one or the other climate change in ordinary study respondents.

The data came from the study (mentioned a few times in previous entries) that CCP and the Annenberg Public Policy Center conducted to refine the Ordinary Climate Science Intelligence assessment (“OSI_1.0”).  

OSI_1.0 used a trick from the study of public comprehension of evolutionary science to “unconfound” the measurement of “knowledge” and “identity.” 

It’s well established that there is no correlation between the answer survey respondents give to questions about their belief in (acceptance of) human evolution and what they understand about science in general or evolutionary science in particular. No matter how much or little individuals understand about science’s account of the natural history of human beings, those who have a cultural identity that features religiosity answer “false” to the statement “human beings evolved from an earlier species of animals,” and those who have a cultural identity that doesn’t  say “true.”  

But things change when one adds the  prefix “according to the theory of evolution” to the standard true-false survey item:

At that point, religious individuals who manifest their identity-expressive disbelief in evolution by answering “false” can now reveal they are in fact familiar with science’s account of the natural history of human beings (even if they, like the vast majority of those who answer “true” with or without the prefix, couldn’t pass a high school biology exam that tested their comprehension of the modern synthesis).

What people say they “believe” about climate change (at least if they are members of the general public in the US) is likewise an expression of who they are, not what they know.

That is, responses to recognizable climate-change survey items—“is it happening,” “are humans causing it,” “are we all going to die,” “what’s the risk on a scale of 0-10,” etc.— are all simply indicators of a latent cultural disposition. The disposition is easily enough measured with right-left political orientation measures, but cultural worldviews are even better and no doubt plenty of other things (even religiosity) work too.

There isn’t any general correlation—positive or negative—between how much people know either about science in general or about climate-science in particular and their “belief” in human-caused climate change.

Click me ... or Donald Trump will become President!But there is an interaction between their capacity for making sense of science and their cultural predispositions.  The greater a person’s proficiency in one or another science-related reasoning capacity (cognitive reflection, numeracy, etc.) the stronger the relationship between their cultural identity (“who they are”) and what they say they “believe” etc. about human-caused climate change.

Why? Presumably because people can be expected to avail themselves of all their mental acuity to form beliefs that reliably convey their membership in and commitment to the communities they depend on most for psychic and material support.

But if one wants to “unconfounded” identity-expressive from knowledge-evincing responses on climate change, one can use the same trick that one uses to accomplish this objective in measuring comprehension of evolutionary science.   OSI_1.0 added the clause “climate scientists believe” to its batery of true-false items on the causes and consequences of human-caused climate change. And lo and behold, individuals of opposing political orientations—and hence opposing “beliefs” about human-caused climate change—turned out to have essentially the equivalent understandings of what “climate science” knows.

Click me ... and Bernie will become President!In general, their understandings turned out to be abysmal: the vast majority of subjects—regardless of their political outlooks or beliefs on climate change—indicated that “climate scientists believe” that  human CO2 emissions stifle photosynthesis, that global warming will cause skin cancer, etc. 

Only individuals at the very highest levels of science comprehension (as measured by the Ordinary Science Intelligence assessment) consistently distinguished genuine from bogus assertions about the causes and consequences of climate change. Their responses were likewise free of the polarization--even though they are the people in whom there is the greatest political division on “belief in” human-caused climate change.


But in collecting data for OSI_2.0, we decided to measure exactly how much of an impact it makes in response to use the identity-knowledge “scientists believe” unconfounding device.

The impact is huge!

Here are a couple of examples of just how much a difference it makes:

Subjects of opposing political outlooks—and hence opposing “beliefs” about human-caused climate change--don't disagree about whether “human-caused global warming will result in flooding of many coastal regions” or whether “nuclear power generation contributes to global warming” when those true-false statements are introduced with the prefix “according to climate scientists” (obviously, the "nuclear" item is a lot harder--that is, people on average, regardless of political outlook, are about as likely to get it wrong as right; "flooding" is a piece of cake).

But when the prefix is removed, subjects of opposing outlooks answer the questions in an (incorrect) manner that evinces their identity-expressive views.  

That prefix is all it takes to toggle the switch between an “identity-expressive” and a “science-knowledge-evincing” orientation toward the items.

All it takes to show that for ordinary members of the public there are two climate changes: one on which their beliefs express “who they are” as members of opposing cultural groups; and another on which their beliefs reflect “what they know” as people who use their reason to acquire their (imperfect in many cases) comprehension of what science knows about the impact of human behavior on climate change.

Now what’s really cool about this pairing is the opposing identity-knowledge "valencess" of the items. The one on flooding shows how the “according to climate scientists" prefix unconfounds climate-science knowledge from a mistaken identity-expressive “belief” characteristic of a climate-skeptical cultural style.  The item on nuclear power, in contrast, uncounfounds  climate-science knowledge from a mistaken identity-expressive “belief” characteristic of a climate-concerned  style.

I like this because it answers the objection—one some people reasonably raised—that adding the “scientists believe” clause to OSI_1.0 items didn't truly elicit climate-science knowledge in right-leaning subjects.  The right-leaning subjects, the argument went, were attributing to climate scientists views that right-leaning subjects themselves think are contrary to scientific evidence but that they think climate scientists espouse becasuse climate scientists are so deceitful, misinformed etc.

I can certainly see why people might offer this explanation.

But it seems odd to me to think that right-leaning subjects would in that case make the same mistakes about climate scientists' positions (e.g., that global warming will cause skin cancer, and stifle photosynthesis) that left-leaning ones would; and even more strange that only right-leaning subjects of low to modest science comprehension would impute to climate scientists these comically misguided overstatements of risk, insofar as high science-comprehending, right-leaning subjects are the most climate skeptical & thus presumably most distrustful of "climate scientists."

Well, these data are even harder to square with this alternative account of why OSI_1.0 avoided eliciting politically polarized responses.

One could still say "well, conservatives just think climate scientsts are full of shit," of course, in response to the effect of removing the prefix for the “flooding” item.

But on the “nuclear power causes climate change” item, left-leaning subjects were the ones whose responses shifted strongly in the identity-expressive direction when the “according to climate scientists prefix” was removed.  Surely we aren’t supposed to think that left-leaning, climate-concerned subjects find climate scientists untrustworthy, corrupt etc. , too! 

The more plausible inference is that the “according to science prefix” does exactly what it is supposed to: unconfound climate-science knowledge and cultural identity, for everyone.

Thus, if one is culturally predisposed to give climate-skeptical answers to express identity, the prefix stifles incorrect "climate science comprehension" responses that evince climate skepticism—e.g., that climate change will cause flooding.

If one is culturally predisposed to give climate-concerned responses, in contrast, then the prefix stifles what would be the identity-expressive inclination to express incorrect beliefs about the contribution of human activities to climate change—e.g., that nuclear power is warming the planet.

The prefix turns everyone from who he or she is when processing information for identity protection into the person he or she is when induced to reveal whatever "science knowledge" he or she has acquired.

This inference is reinforced by considering how these responses interact with science comprehension. 

As can be seen, for the "prefix" versions of the items, individuals of both left- and right-leaning orientations are progressively more likely to give correct "climate science comprehension" answers as their OSI scores increase.  This makes a big difference on the “nuclear power” item, because it’s a lot harder than the “flooding” one.

Nevertheless, when the “prefix” is removed, those who are high in science comprehension (right-leaning or left-) are the most likely to get the wrong answer when the wrong answer is identity-expressive! 

That’s exactly what one would expect if the prefix were functioning to suppress an identity-expressive response, since those high in OSI are the most likely to form identity-expressive beliefs as a result of motivated reasoning.

Suppressing such a response, of course, is what the “according to scientists” clause is supposed to do as an identity/science-knowledge unconfounding device.

This result is exactly the opposite of what one would expect to see, though, under the alternative, “just measuring conservative distrust of/disagreement with climate scientists” explanation of the effect of the prefix: the subjects who such an explanation implies ought to be most likely to attribute an absurdly mistaken "climate concerned" position to climate scientists--the right-leaning subjects highest in science comprehension--were in fact the least likely to do so.

But it was definitely very informative to look more closely at this issue.

Indeed, how readily one can modify the nature of the information processing that subjects are engaging in—how easily one can switch off identity-expression and turn on knowledge-revealing—is pretty damn amazing.

Of course, this was done in the lab.  The million dollar question is how to do it in the political world so that we can rid our society once and for all of illiberal, degrading, welfare-annihilating consequences of the first climate change. . . .


"America's two climate changes ..." & how science communicators should/shouldn't address them ... today in Burlington, VT

More "tomorrow," but a preview ... you tell me what it means!



Page 1 ... 2 3 4 5 6 ... 39 Next 20 Entries »