Incentives and politically motivated reasoning: we can learn something but only if we don't fall into the " 'external validity' trap"
From revision to "The Politically Motivated Reasoning Paradigm" paper. Been meaning to address the interesting new studies on how incentives affect this form of information processing. Here's my (provisional as always) take. It owes a lot to helpful exchanges w/ Gaurav Sood, who likely disagrees with everything I say; maybe I can entice/provoke him into doing a guest post! But in any case, his curiosity & disposition to acknowledge complexity equip him both to teach & learn from others regardless of how divergent his & their "priors."
6. Monetary incentives
Experiments that reflect the PMRP design are “no stake” studies: that is, subjects answer however they “feel” like answering; the cost of a “wrong” answer and the reward for a “correct “one are both zero. In an important development, several researchers have recently reported that offering monetary incentives can reduce or eliminate polarization in the answers that subjects of diverse political outlooks give to questions of partisan import (Khanna & Sood 2016; Prior, Sood & Gaurav 2015; Bullock, Gerber, Hill & Huber 2015).
The quality of these studies is uneven. The strongest, Khanna & Sood (2016), uses the PMRP design. K&S show that offering incentives reduces the tendency of high numeracy subjects to supply politically biased answers in interpreting covariance data in a gun-control experiment, a result reported in Kahan et al. (2013) and described in Section 4.
PSG and BGHH, in contrast, examine subject responses to factual quiz questions (e.g., “. . . has the level of inflation [under President Bush] increased, stayed the same, or decreased?”;“how old is John McCain?,” (Bullock et al. 2015, pp. 532-33)). Because this design does not involve information processing, it doesn’t show how incentives affect the signature feature of politically motivated reasoning: the opportunistic adjustment of the weight assigned to new evidence conditional on its political congeniality.
Both K&S and BGHH, moreover, use M Turk worker samples. Manifestly unsuited for the study of politically motivated reasoning generally (see Section 3.3), M Turk samples are even less appropriate for studies on the impact of incentives on this form of information processing. M Turk workers are distinguished from members of the general population by their willingness to perform various forms of internet labor for pennies per hour. They are also known to engage in deliberate misrepresentation of their identities and other characteristics to increase their on-line earnings (Chandler & Shapiro 2016). Thus, how readily they will alter their reported beliefs in anticipation of earning monetary rewards for guessing what researchers regard as “correct” answers furnishes an unreliable basis for inferring how members of the general public form beliefs outside the lab, with incentives or without them.
But assuming, as seems perfectly plausible, that studies of ordinary members of the public corroborate the compelling result reported in K&S, a genuinely interesting, and genuinely complex, question will be put: what inference should be drawn from the power of monetary incentives to counteract politically motivated reasoning?
BGHH assert that such a finding would call into doubt the external validity of politically motivated reasoning research. Attributing the polarized responses observed in “no stake” studies to the “expressive utility that [study respondents] gain from offering partisan-friendly survey responses,” BGHH conclude that the “apparent gulf in factual beliefs between members of different parties may be more illusory than real” (Bullock et al., pp. 520, 523).
One could argue, though, that BGHH have things exactly upside down. In the real world, ordinary members of the public don’t get monetary rewards for forming “correct” beliefs about politically contested factual issues. In their capacity, as voters, consumers, or participants in public discussion, they don’t earn even the paltry expected-value equivalent of the lottery prizes that BGHHG offered their M Turk worker subjects for getting the “right answer” to quiz questions. Right or wrong, an ordinary person’s beliefs are irrelevant in these real-world contexts, because any action she takes based on her beliefs will be too inconsequential to have any impact on policymaking.
The only material stake most ordinary people have in the content of their beliefs about policy-relevant facts is the contribution that holding them makes to the experience of being a particular sort of person. The deterrent effect of concealed-carry laws on violent crime, the contribution of human activity to global warming, the impact of minimum wage laws on unemployment—all of these are positons infused with social meanings. The beliefs a person forms about these “facts” reliably dispose her to act in ways that others will perceive to signify her identity-defining group commitments (Kahan in press_a). Failing to attend to information in a manner that generates such beliefs can have a very severe impact on her wellbeing—not because the beliefs she’d form otherwise would be factually wrong but because they would convey the wrong message about who she is and whose side she is on. The interest she has in cultivating beliefs that reliably summon an identity-expressive affective stance on such issues is what makes politically motivated reasoning rational.
No-stake PMRP designs seek to faithfully model this real-world behavior by furnishing subjects with cues that excite this affective orientation and related style of information processing. If one is trying to model the real-world behavior of ordinary people in their capacity as citizens, so-called “incentive compatible designs”—ones that offer monetary “incentives” for “correct” answers”—are externally invalid because they create a reason to form “correct” beliefs that is alien to subjects’ experience in the real-world domains of interest.
On this account, expressive beliefs are what are “real” in the psychology of democratic citizens (Kahan in press_a). The answers they give in response to monetary incentives are what should be regarded as “artifactual,” “illusory” (Bullock et al., pp. 520, 523) if we are trying to draw reliable inferences about their behavior in the political world.
It would be a gross mistake, however, to conclude that studies that add monetary incentives to PMRP designs (e.g., Khanna & Sood 2016) furnish no insight into the dynamics of human decisionmaking. People are not merely democratic citizens, not only members of particular affinity groups, but also many other things, including economic actors who try to make money, professionals who exercise domain-specific expert judgments, and parents who care about the health of their children. The style of identity-expressive information processing that protects their standing as members of important affinity groups might well be completely inimical to their interests in these domains, where being wrong about consequential facts would frustrate their goals.
Understanding how individuals negotiate this tension in the opposing “stakes” they have in forming accurate beliefs and identity-expressive ones is itself a project of considerable importance for decision science. The theory of “cognitive dualism” posits that rational decisionmaking comprises a capacity to employ multiple, domain-specific styles of information processing suited to the domain-specific goals that individuals have in using information (Kahan 2015b). Thus, a doctor who is a devout Muslim might process information on evolution in an identity-expressive manner “at home”—where “disbelieving” in it enables him to be a competent member of his cultural group—but in a truth-seeking manner “at work”—where accepting evolutionary science enables him to be a competent oncologist (Hameed & Everhart 2013). Or a farmer who is a “conservative” might engage in an affective style of information processing that evinces “climate skepticism” when doing so certifies his commitment to a cultural group identified with “disbelief” in climate change, but then turn around and, join the other members of that same cultural group in processing such information in a truth-seeking way that credits climate science insights essential to being a successful farmer (Rejesus et al. 2013).
If monetary incentives do meaningfully reverse identity-protective forms of information processing in studies that reflect the PMRP design, then a plausible inference would be that offering rewards for “correct answers” is a sufficient intervention to summon the truth-seeking information-processing style that (at least some) subjects use outside of domains that feature identity-expressive goals. In effect, the incentives transform subjects from identity-protectors to knowledge revealers (Kahan 2015a), and activate the corresponding shift in information-processing styles appropriate to those roles.
Whether this would be the best understanding of such results, and what the practical implications of such a conclusion would be, are also matters that merit further, sustained emirical inquiry. Such a program, however, is unlikely to advance knowledge much until scholars abandon the pretense that monetary incentives are the “gold standard” of experimental validity in decision science as opposed to simply another methodological device that can be used to test hypotheses about the interaction of diverse, domain-specific forms of information processing.
Chandler, J. & Shapiro, D. Conducting Clinical Research Using Crowdsourced Convenience Samples. Annual Review of Clinical Psychology (2016), advance on-line publication at http://www.annualreviews.org/doi/abs/10.1146/annurev-clinpsy-021815-093623.
Everhart, D. & Hameed, S. Muslims and evolution: a study of Pakistani physicians in the United States. Evo Edu Outreach 6, 1-8 (2013).
The Politically Motivated Reasoning Paradigm. Emerging Trends in Social & Behavioral Sciences, (in press).
Khanna, Kabir & Sood, Gaurav. Motivated Learning or Motivated Responding? Using Incentives to Distinguish Between Two Processes (working), available at http://www.gsood.com/research/papers/partisanlearning.pdf.
Prior, M., Sood, G. & Khanna, K. You Cannot be Serious: The Impact of Accuracy Incentives on Partisan Bias in Reports of Economic Perceptions. Quarterly Journal of Political Science 10, 489-518 (2015).
Rejesus, R.M., Mutuc-Hensley, M., Mitchell, P.D., Coble, K.H. & Knight, T.O. US agricultural producer perceptions of climate change. Journal of agricultural and applied economics 45, 701-718 (2013).