follow CCP

Recent blog entries
popular papers

Science Curiosity and Political Information Processing

What Is the "Science of Science Communication"?

Climate-Science Communication and the Measurement Problem

Ideology, Motivated Cognition, and Cognitive Reflection: An Experimental Study

'Ideology' or 'Situation Sense'? An Experimental Investigation of Motivated Reasoning and Professional Judgment

A Risky Science Communication Environment for Vaccines

Motivated Numeracy and Enlightened Self-Government

Making Climate Science Communication Evidence-based—All the Way Down 

Neutral Principles, Motivated Cognition, and Some Problems for Constitutional Law 

Cultural Cognition of Scientific Consensus
 

The Tragedy of the Risk-Perception Commons: Science Literacy and Climate Change

"They Saw a Protest": Cognitive Illiberalism and the Speech-Conduct Distinction 

Geoengineering and the Science Communication Environment: a Cross-Cultural Experiment

Fixing the Communications Failure

Why We Are Poles Apart on Climate Change

The Cognitively Illiberal State 

Who Fears the HPV Vaccine, Who Doesn't, and Why? An Experimental Study

Cultural Cognition of the Risks and Benefits of Nanotechnology

Whose Eyes Are You Going to Believe? An Empirical Examination of Scott v. Harris

Cultural Cognition and Public Policy

Culture, Cognition, and Consent: Who Perceives What, and Why, in "Acquaintance Rape" Cases

Culture and Identity-Protective Cognition: Explaining the White Male Effect

Fear of Democracy: A Cultural Evaluation of Sunstein on Risk

Cultural Cognition as a Conception of the Cultural Theory of Risk

« More on public "trust of scientists": *You* tell *me* what it means! | Main | "Energy future 2030" talk (slides, video) »
Tuesday
May202014

The "generalizability problem" -- a fragment

From something I'm working on (one of many things distracting me from this blog; I've experienced a curious inversion recently in proscrastination diversions....)

One of the major challenges confronting the science of science communication is generalizability.  This problem is obvious when researchers engage in  lab experiments. By quieting the cacophony of uncontrollable real-world influences, such experiments enable the researcher to isolate and manipulate mechanisms of interest, and thus draw confident inferences about their significance, or lack thereof. But how, then, can one know whether the effects observed in these artificially tranquil conditions will hold up in the chaotic real-life environment from which the researcher sought refuge in the lab? 

It would be a mistake, though, to think that this difficulty reflects some fatal defect in laboratory methods.  And not just because such methods do indeed play an indispensable role in the formation of communication strategies that can subsequently be tested outside the lab. For any empirical testing that occurs in the field must also confront the question of generalizability: how is one to know that what worked in one distinctively messy real-world setting will work in another distinctively messy one?

The generalizability problem is central to the motivation for our proposal.  Disturbingly, a large fraction of researchers offering counsel to conservation advocates and policymakers simply ignore this issue altogether. 

But just as bad, a large fraction of the remainder try to address it in the wrong way.  They believe that the goal of empirical research is to identify a fixed set of universally effective “techniques” or “best practices” that can, with the benefit maybe of cartoon-illustrated instruction manuals, be confidently and more-or-less thoughtlessly applied by communicator "consumers." 

But in fact, the only technique of the science of science communication that generalizes—the sole valid “best practice” it has to offer—is its method. Successful lab experiments and field studies alike do enlarge understandings of how the world works. But how the insights they generate can be brought successfully to bear on any new problem will always be a question that those promoting science-informed conservation policymaking will have to answer for themselves.  The only way they can reliably do so, moreover, is by using empirical methods to adapt what the science of science communication knows to the distinctive circumstances at hand.  

Perfecting knowledge of how to use empirical methods in the everyday practice of conservation-science communication—so that the generalizability issue will always be confronted and confronted effectively—is the whole point of the proposed ....


PrintView Printer Friendly Version

EmailEmail Article to Friend

Reader Comments (10)

When I do experiments, I worry about generalizibility constantly, especially in the early design phase of a study. I try to list all the variables related to the experiment. (I have found that many others only list the variables that they are comfortable with, those that fit within their expertise. I try to list all the variables.) Then I make quantitative approximations (using matrix or tensor mathematics as needed) of the effect of each variable on the generalized problem. I try to come up with an experiment that probes the most quantitatively important variables and yields defensible results. This design process is hard and humbling.
I don't know how others approach the generalizibility problem. I hope that my approach is useful to others and that others can help me to improve my process.

May 20, 2014 | Unregistered CommenterEric Fairfield

There is a science of science communication, but there is also an art of science communication or more accurately, there is an art (and science) of communication. Subjecting art to empirical study is obviously quite difficult.

In my experience in education, as a general rule, theoreticians underestimate the importance of practice (and developing an artist's skills), and practitioners underestimate the importance of theory - and both groups over-generalize about what works (motivated reasoning).

May 20, 2014 | Unregistered CommenterJoshua

Joshua,
Agreed.
My friends and I live between the theoreticians and the experimentalists and, so, have to communicate successfully with both sides. Sometimes it is easy; sometimes not.

May 20, 2014 | Unregistered CommenterEric Fairfield

This, to me, is where the difference between general "strategies" and more specific "tactics" really becomes key.

May 20, 2014 | Unregistered CommenterJen

@Joshua:

I think art vs. science is contrived dichotomory -- or else an unclear way to make a point that deserves to be made clearly. That point is that professional expertise involves professional judgment -- the dominant element of which is the proper assimilation of particulars to prototypes acquired via experience, refined & calibrated through reflection, & then accessed through a largely automatic & unconscious faculty of recognition. No amount of empirics will get anyone anywhere -- or even be useful -- unless it is guided by & can be turned into a guide by professional judgment. But professional judgment can definitely be better when professionals have at their disposal & direction empirical methods for acquiring evidende that enables them to assess & reflect on the validity of their prototypes

May 20, 2014 | Registered CommenterDan Kahan

There's a concept of Action Research in education (well, it's also a more general term but I am speaking mainly from the experience of having been a teacher for several years before moving into another field). In many education programs, there is coursework on "Action Research."

In other words, in addition to coursework on pedagogy and psychology, many programs include courses that aim to teach young teachers how to perform their own in-class research in a just-in-time and as-they-need-it process that answers questions unique to their own classrooms and curriculum, etc. Rather than "here are the methods we've found best for teaching 11th graders about electromagnetism" it offers "here are the strategies you can employ to first identify the key questions you have when you are teaching 11th graders electromagnetism, and then b) design some kind of experimental ways to get answers to those questions.. and then c) reflect on the outcomes you observe and improve your practice with that reflection."

Dan, I often hear all the various comments you make here and elsewhere about wanting science communicators to do their own research, and I'm starting to wonder just why the concept of "Action Research" as has been embraced in many education programs isn't more common across fields...

May 20, 2014 | Unregistered CommenterJen

Dan -

--> "I think art vs. science is contrived dichotomory -- or else an unclear way to make a point that deserves to be made clearly. "

Yes. Agreed. I don't think of them as dichotomous - I should have been more careful. I was more speaking to how practitioners and theoreticians fall into a trap of failing to understand how much theory and practice can inform each other, in fact need to inform each other, and the relationship between art and science is more of a tangent. With both dyads (art vs. science and theory vs. practice), IMO, neither opposing entity really exists in some way that is mutually exclusive to the other, but I do think that it can be useful to conceptualize the relationships in a spectrum.

I have to mull over the concept of "professional judgement" as the criterion for assessment. Something sticks for me there. Thinking of fine arts, if an artist refines his/her technique as the result of experimentation and experience, that would amount to professional judgement, but there is also a kind of intuitive input that comes from a different form of reasoning that it hard for me to categorize with professional judgement. I think that the same applies to teaching at some level. Not sure about effective communicating.

May 20, 2014 | Unregistered CommenterJoshua

Generalizibility 2
There is a problem with writing out how to generalize things. You have to read your own words and see if you actually do what you said you do. It turns out that I don't quite follow my own recommendations.
In our research on brain function, cell by cell and synapse by synapse, the overall rule is that you are not allowed to bring ad hoc approaches into the system So, I can't drop the knowledge of Bessel functions into the workings of vision sensing cells with a very strong biological reason to do so. A second piece of building the model is to collect more and more pieces of how biology remembers and learns. These pieces will be used later on. A third piece of building the model is to try to create a model into which the collected pieces can be inserted easily at some later time without needing to rebuild the entire model and without distorting the biological function of the new piece.
Given those caveats, the current model has incorporated about 30 pieces of biology (out of 100). I think that the 30 pieces are the most important but cannot prove it. The model has emergent behavior that closely mimics behaviors seen in humans and other animals as they grow and learn. This is very encouraging but does not prove that, once we add in the other pieces, our model will generalize successfully to the more complex situation of a real person.
The strongest statement that we can make now is that the model appears to be generalizable, the preliminary results are very exciting, and the results may or may not generalize as the model approaches the complexity of real life. We believe that most results will generalize, but proof of this belief will take a while.
Thank you for getting me to think about the problem of generalizibility a little more clearly.

May 20, 2014 | Unregistered CommenterEric Fairfield

Generalizibility 2
There is a problem with writing out how to generalize things. You have to read your own words and see if you actually do what you said you do. It turns out that I don't quite follow my own recommendations.
In our research on brain function, cell by cell and synapse by synapse, the overall rule is that you are not allowed to bring ad hoc approaches into the system So, I can't drop the knowledge of Bessel functions into the workings of vision sensing cells with a very strong biological reason to do so. A second piece of building the model is to collect more and more pieces of how biology remembers and learns. These pieces will be used later on. A third piece of building the model is to try to create a model into which the collected pieces can be inserted easily at some later time without needing to rebuild the entire model and without distorting the biological function of the new piece.
Given those caveats, the current model has incorporated about 30 pieces of biology (out of 100). I think that the 30 pieces are the most important but cannot prove it. The model has emergent behavior that closely mimics behaviors seen in humans and other animals as they grow and learn. This is very encouraging but does not prove that, once we add in the other pieces, our model will generalize successfully to the more complex situation of a real person.
The strongest statement that we can make now is that the model appears to be generalizable, the preliminary results are very exciting, and the results may or may not generalize as the model approaches the complexity of real life. We believe that most results will generalize, but proof of this belief will take a while.
Thank you for getting me to think about the problem of generalizibility a little more clearly.

May 20, 2014 | Unregistered CommenterEric Fairfield

Jen -

One aspect of teaching that I have always found interesting is how different students can have such diametrically different reactions to the same teacher.. And another aspect that is interesting is how different teachers have a wide range of success utilizing similar methodologies.

So two implications: different methodologies work better or worse contingent on the individual student. And different methodologies work better or worse for different teachers.

The first implication would be supported by "Action Research" because it would enable the teacher to understand more about which methodologies work best for a particular student. The more methodologies a teacher knows how to implement, the better they are able to individualize instruction. The second would also be supported by "Action Research" because it would help teachers to gain a better understanding of which methodology works better for them individually.


One aspect of the success of public education in Finland is that it seems to recognize the importance of teachers developing their own "professional judgement" - as Dan speaks to (e.g., by encouraging teachers to design their own tests as opposed to using standardized tests). And:

--> "In South Korea—much like Japan and Singapore—only about 35 percent of teachers’ working time is spent teaching pupils. Teachers work in a shared office space during out-of-class time since the students
stay in a fixed classroom while the teachers rotate to teach them different subjects. The shared office
space facilitates sharing of instructional resources and ideas among teachers, which is especially helpful
for new teachers."

https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CC4QFjAA&url=https%3A%2F%2Fedpolicy.stanford.edu%2Fsites%2Fdefault%2Ffiles%2Fpublications%2Fhow-high-achieving-countries-develop-great-teachers.pdf&ei=YH97U-2LDdCGyASbrYHICw&usg=AFQjCNEXuVfrfIO9oRcZu2_qCTtIAYC6qA&sig2=_c1wLV91YGDGnMGMLZ4roA&bvm=bv.67229260,d.aWw&cad=rja

May 20, 2014 | Unregistered CommenterJoshua

PostPost a New Comment

Enter your information below to add a new comment.

My response is on my own website »
Author Email (optional):
Author URL (optional):
Post:
 
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>