In the last couple of posts (one on evolution believers' & nonbelievers' engagement with an evolution-science documentary, and another on measuring "science curiosity") I've summarized some of the findings from Study No. 1 of the Annenberg/CCP ESFI--"Evidence-based Science Filmmaking Initiative."
Those findings are described in more detail in a study Report, which also spells out the motivation for the study and its relation to ESFI overall.
Indeed, the Report is an unusual document--or at least an unusual sort of document to share.
It isn't styled as announcing to the world the "corroboration of" or "refutation" of some specified set of hypotheses. It is in fact an internal report prepared for consumption of the the investigators in an ongoing research project, one that is in fact at a very preliminary stage!
Why release something like that? Well, in part because even at this point in the investigation, we do think there are things to report that will be of interest to other scholars and reflective people generally, many of whom can be counted on to supply us w/ feedback that will itself make what we do next even more useful.
But in addition, one of the aims of the project, in addition to generating evidence relevant to questions of interest to professional science filmmakers, is to model the process of using evidence-based methods to answer those very questions.
As explained in the ESFI "main page," the project is itself meant to supply evidence relevant to the hypothesis that the methods distinctive of the science of science communication can make a positive contribution to the craft of science filmmaking by furnishing those engaged in it with the information relevant to the exercise of their professional judgment.
Of course, those engaged in ESFI, including its professional science communication members, believe (with varying levels of confidence!), that in fact the science of science communication can make such a contribution; but of course, too, others, including other professional science filmmakers, are likely to disagree with this conjecture.
I wouldn't say "no point arguing about it" just b/c reasonable, and informed, people can disagree.
But I would say that these are exactly the conditions in which the argument will proceed in a more satisfactory way with additional information of the sort that can be generated by science's signature methods of disciplined observation, reliable measurement, and valid inference.
Hence ESFI: Let's do it -- and see what a collaboration between professional science filmmakers and allied communicators, on the 1 hand, and & "scientists of science communication" on the other, produces. Then, on the basis of that evidence, those who are involved in science filmmaking can use their own reason to judge for themselves what that evidence signifies, and update accordingly their assessments of the utility of integrating the science of science communication into the craft of science filmmaking (not to mention related forms of science communication, like science journalism).
Precisely b/c the Report is an internal research document that takes stock of early findings in a multi-stage project, it furnishes a glimpse of the project in action. It thus gives those who might consider using such methods a chance to form a more concrete picture of what these practices look like, and a chance to use their own experience-informed imaginations to assess what they might do if they could add evidence-based methods to their professional tool kits.
But of course this is only the start-- only the first Report, both of results and of the experience of doing evidence-based filmmaking.
A. Overview and summary conclusions
This report summarizes the preliminary conclusions of Study No. 1 in the Annenberg/CCP “Evidence-based Science Filmmaking Initiative.” The goal of the initiative is to promote the integration of the emerging science of science communication into the craft of science filmmaking. Study No. 1 involved an exploratory investigation of viewer engagement with an excerpt from Your Inner Fish, a documentary on human evolution.
The study had two objectives.
One was to gather evidence relevant to an issue of debate among science filmmakers: what explains the perceived demographic homogeneity of the audience for high-quality documentaries featured on NOVA, Nature, and similar PBS shows? Is the answer the distribution of tastes for learning about scientific discovery in the general population, or instead some feature of those shows collateral to their science content that makes them uncongenial to individuals who subscribe to certain cultural styles?
The other study objective was to model how evidence-based methods could be used by science filmmakers. Hard questions—ones for which the number of plausible answers exceeds the number of correct ones—are endemic to the activity of producing science films. By testing competing conjectures on an issue of consequence to their craft, Study No. 1 illustrates how documentary producers might use empirical methods to enlarge the stock of information pertinent to the exercise of their professional judgment in answering such questions.
Principal conclusions of Study No. 1 include:
1. By combining appropriately subtle self-report items with behavioral and performance-based ones, it is possible to construct a valid scale for measuring individuals’ general motivation to consume information about scientific discovery for personal satisfaction. Desirable properties of the “Science Curiosity Scale” (SCS) include its high degree of measurement precision, its appropriate relationship with science comprehension and other pertinent covariates, and (most importantly) its power to predict meaningful differences in objective manifestations of science curiosity.
2. By similar means, one can construct a satisfactory scale for measuring viewer engagement with material such as that featured in the YIF clip. Such a scale was again formed by combining self-report and objective measures, including duration of viewing time and requested access to the remainder of the documentary. Designated the “Engagement Index” (EI), the scale had the expected relationships with education and general science comprehension. The strongest predictor of EI was the study subjects’ SCS scores.
3. Engagement with the clip did not vary to a meaningful degree among subjects who had comparable SCS scores but opposing “beliefs” about human evolution. Evolution “believers” and “nonbelievers” with high SCS scores formed comparably positive reactions to the YIF clip. The show didn’t “convert” the latter. But like “believers” with high SCS scores, high-scoring “nonbelievers” were very likely to accept the validity of the science featured in the clip. This finding is consistent with research suggesting that professions of “disbelief” in evolution are an indicator of cultural identity that poses no barrier to engagement with scientific information on evolution, so long as that information itself avoids mistaking exacting professions of “belief” for communicating knowledge.
4. Engagement with the show did vary across culturally identifiable groups. The members of one cultural group, whose members are in fact distinguished in part by their pro-technology attitudes, appeared to display less engagement the clip than was predicted by their SCS scores. This finding furnishes at least some support for the conjecture that some fraction of the potential audience for science documentary programing is discouraged from viewing it by uncongenial cultural meanings collateral to the science content of such programming.
5. But additional, more fine-grained analysis of the data is necessary. In particular, the science-communication-professional members of the research team must formulate concrete, alternative hypotheses about the identity of culturally identifiable groups who might well be responding negatively to collateral cultural meanings in the clip. Those hypotheses can in turn be used by the science-of-science-communication team members to develop more fine-tuned cultural profiles that can be used to probe such conjectures.
6. Depending on the results of these additional analyses, next steps would include experimental testing that seeks to modify collateral meanings or cues in a manner that eliminates any disparity in engagement among individuals of diverse cultural identities who share a high level of curiosity about science.