Weblog Menu

Admin

Friday, January 12, 2007

Methods:  Survey Experiments and Resilience

Some meaty methodological advice on experimentation in surveys is found in the Winter 2007 issue of Political Analysis (the journal of the Methodology Section of the American Political Science Association). In “The Logic of the Survey Experiment Reexamined,” authors Gaines, Kuklinski and Quirk (2006, pages 1-20) offer an overview of the power of survey experiments to identify causal relationships, along with a bundle of caveats scholars should keep in mind when designing, conducting and interpreting survey experiments. (The article is available to the Calvin community through the publisher’s web site).

For example, Gaines et al review research showing that treatment effects are often very transitory, rendering suspect the applicability of experimental findings to real-world politics where “treatments” (such as campaign speeches, television ads and peer interactions) are experienced long before the outcome of interest (such as vote choice). The authors suggest that researchers consider measuring the impact of repeated treatments over time in panel study formats to discover whether treatment effects are transitory, durable, cumulative, and so forth.

The resilience-research theme reminds me of conversations with several colleagues, especially Calvin sociologist Kurt Ver Beek. Though he is not engaged in survey experiments, his studies of the effects of short-term missions lead him to grapple with the problem of resilient predispositions. Like “saplings” being trained by a gardener (a metaphor Kurt uses), participants in short-term missions seem to be only temporarily shaped by their experiences; after returning home, they rebound into their old straight-up position and remain unbent by their exposure to poverty and need. There are similar issues raised by the effects of educational efforts, anti-racism programs, elements of worship and other cultural “treatments” whose aim is to alter long-term behavior.

What does it take to transform a life or a community, to make a quantum leap from one stable state to another? It seems to me that social research on religion has something to offer many disciplines:  we need to link research on individual conversion and corporate revival to a general theory of change. This is not to say that social science can capture and explain the work of the Spirit; on the contrary, research is often best used to debunk deterministic theories and demonstrate the mere plausibility of the ineffable and transcendent.

Back, then, to the original topic of survey experimentation. One purpose of such experiments can be to document whether small changes in wording or question ordering dispose people to express more or less support for an idea or proposal. While the effect may be very transitory, such findings can open the door for practical communication strategies that capitalize on a momentary increase in openness to gain commitments to participate or listen to further efforts at education, persuasion and so on. From evangelism to epidemiology, it may take a miracle to achieve radical change. But it usually takes a disciplined strategy of communication and action to bring people to the place where miracles happen. Think of the Apostle Paul - “and how can they hear without someone preaching to them?” (Romans 10:14)

CSR has had some good experiences with random assignment of survey respondents to “treatment conditions” through our Inquisite web survey system. For example, we have randomized the order of respondents’ exposure to questions about gender in two studies in recent years. Findings from one internal Calvin study were remarkable, but cannot be published due to a lack of Internal Review Board approval; the other study is in a pilot phase and results are forthcoming. We are also working on a study in which respondents will be randomly assigned a proposed price for a service and asked how likely they are to pay that price; Gainey et al refer to this “random variation” approach in their first footnote. The result will provide much more informative pricing information than other question formats, most of which would allow respondents to wildly underestimate the cost of the proposed service.

If you’re doing social research, your project might benefit from some experimental strategies. Talk to CSR--we’ll keep the caveats in mind as we work with you!

<< Back to main