Some meaty methodological advice on experimentation in surveys is found in the Winter 2007 issue of Political Analysis (the journal of the Methodology Section of the American Political Science Association). In “The Logic of the Survey Experiment Reexamined,” authors Gaines, Kuklinski and Quirk (2006, pages 1-20) offer an overview of the power of survey experiments to identify causal relationships, along with a bundle of caveats scholars should keep in mind when designing, conducting and interpreting survey experiments. (The article is available to the Calvin community through the publisher’s web site).
For example, Gaines et al review research showing that treatment effects are often very transitory, rendering suspect the applicability of experimental findings to real-world politics where “treatments” (such as campaign speeches, television ads and peer interactions) are experienced long before the outcome of interest (such as vote choice). The authors suggest that researchers consider measuring the impact of repeated treatments over time in panel study formats to discover whether treatment effects are transitory, durable, cumulative, and so forth.
The resilience-research theme reminds me of conversations with several colleagues, especially Calvin sociologist Kurt Ver Beek. Though he is not engaged in survey experiments, his studies of the effects of short-term missions lead him to grapple with the problem of resilient predispositions. Like “saplings” being trained by a gardener (a metaphor Kurt uses), participants in short-term missions seem to be only temporarily shaped by their experiences; after returning home, they rebound into their old straight-up position and remain unbent by their exposure to poverty and need. There are similar issues raised by the effects of educational efforts, anti-racism programs, elements of worship and other cultural “treatments” whose aim is to alter long-term behavior.
What does it take to transform a life or a community, to make a quantum leap from one stable state to another? It seems to me that social research on religion has something to offer many disciplines: we need to link research on individual conversion and corporate revival to a general theory of change. This is not to say that social science can capture and explain the work of the Spirit; on the contrary, research is often best used to debunk deterministic theories and demonstrate the mere plausibility of the ineffable and transcendent.
Back, then, to the original topic of survey experimentation. One purpose of such experiments can be to document whether small changes in wording or question ordering dispose people to express more or less support for an idea or proposal. While the effect may be very transitory, such findings can open the door for practical communication strategies that capitalize on a momentary increase in openness to gain commitments to participate or listen to further efforts at education, persuasion and so on. From evangelism to epidemiology, it may take a miracle to achieve radical change. But it usually takes a disciplined strategy of communication and action to bring people to the place where miracles happen. Think of the Apostle Paul - “and how can they hear without someone preaching to them?” (Romans 10:14)
CSR has had some good experiences with random assignment of survey respondents to “treatment conditions” through our Inquisite web survey system. For example, we have randomized the order of respondents’ exposure to questions about gender in two studies in recent years. Findings from one internal Calvin study were remarkable, but cannot be published due to a lack of Internal Review Board approval; the other study is in a pilot phase and results are forthcoming. We are also working on a study in which respondents will be randomly assigned a proposed price for a service and asked how likely they are to pay that price; Gainey et al refer to this “random variation” approach in their first footnote. The result will provide much more informative pricing information than other question formats, most of which would allow respondents to wildly underestimate the cost of the proposed service.
If you’re doing social research, your project might benefit from some experimental strategies. Talk to CSR—we’ll keep the caveats in mind as we work with you!
What has distinguished growing American congregations from their stagnant and dwindling cousins? Some tentative answers are found in a new report from Faith Communities Today: a growing, youthful demographic setting, a multiethnic constituency, a “vital,” contemporary worship style, and a purposeful organizational disposition to grow and change. Drums and “joyful” worship often went with growth; worship described as “reverent,” unfortunately, did not often accompany numeric growth in weekly attendance (see pages 9 and 10 of the report).
Whether these recent trends are worthy of emulation is a theological and social matter the current report does not address directly. But scholars and laypeople of all stripes may find evidence to inform their perspectives. The report, covering many faiths and denominations, is based on nationwide data collected in 2005 by the Calvin College Center for Social Research.
About the data
The Faith Communities Today (FACT) research team has issued its first extended report on the 2005 FACT survey data. Authored by Kirk Hadaway of the Episcopal Church, the FACTs on Growth report notes the leading correlates (not necessarily causes!) of reported numeric growth in average attendance from 2000 to 2005.
The Center for Social Research participated in survey instrument design, implemented the design in print and on the web, built an online database of congregational contacts, collected survey data, and prepared the data for analysis, including collating and merging U.S. Census and Department of Education statistics into the data.
The complexity of joy, excitement, and reverence
Nestled in the report is an interesting nugget that should help nuance congregations’ responses to the study’s findings. What makes a congregation grow may depend heavily on how well its growth strategy fits its constituency; the correlates do not represent a prescription for growth, let alone a “one-size-fits-all” prescription. For example, author Hadaway notes on page 9 that congregations reporting “joyful” worship were significantly more likely to grow (see figure snapshot above), while the adjective “exciting” correlated with growth only for evangelical denominations and not for mainline congregations. “[E]xciting worship may seem to foreign or perhaps too evangelical” to mainline congregations, Hadaway writes. This kind of nuance might also explain the apparently negative association of “reverent” worship with growth found on page 10; though less reverent congregations were about twice as likely to grow as congregations reporting very reverent worship, thirty-one percent of very reverent congregations did grow. The culture of reverence may be a minority culture today, but it can still be associated with growth in many congregations.
Multivariate model identifies internal conflict as number-one growth killer
Hadaway reports that reverence remained a strong negative correlate of growth in multivariate models (see page 16), even after “controlling” for other factors such as location, faith tradition and initial size. But the number one growth-killer was the presence of internal conflict. He notes that numerous denomination-wide conflicts over sexuality issues probably portend continued decline in mainline congregations. The conflict issue raises a potential research question for future investigation: is the price of conflict great enough that conflicted denominations should consider policies and programs aimed at amicable separation, as well as the usual reconciliation efforts?
|What:||Social Science Division Symposium|
|When:||Tuesday, November 14 @ 3:30PM|
|Where:||DeVos Communication Center room 170|
|Who:||All social science faculty, with other faculty and students welcome.|
|Speaker:||Neil Carlson, Assistant Director of the Center for Social Research|
|Topic:||Following the money: mapping flows of campaign contributions.|
This presentation wll build on skills and a database developed through the CSISS SPACE workshop at UCSB in August, updated with recent FEC data for the 2006 campaign.
Campaign contributions are both a worthy substantive topic and an ideal context for demonstrating more general spatial analysis strategies. Free data from the Federal Election Commission can readily be “geocoded” to a number of levels of analysis. The data has relevance for all social science disciplines and offers an opportunity for classroom integration of spatial analysis in politics (campaign strategy), economics (incentive structures), sociology (wealth and policy influence) and psychology (the public donor personality). The presentation will be brief (15-20 minutes), leaving ample time for questions and discussion of the topic and spatial analysis in social science generally.
Discover magazine led me to a great find, a new satellite imagery study of urban sprawl in the US by a University of Toronto team led by economist Matthew Turner. The study surprisingly finds less sprawl than expected overall, but major differences among metropolitan areas. Miami is compact, Pittsburgh sprawls. Inter-city differences are explained by differences in “ground water availability, temperate climate, rugged terrain, decentralized employment, early public transport infrastructure, uncertainty about metropolitan growth, and unincorporated land in the urban fringe.” See the working paper or get a copy of the published version from the Quarterly Journal of Economics on the IDEAS site (the download did not work for me, but the citation and abstract are complete).
The Calvin Institute of Christian Worship (CICW) has received a grant from the Lilly Endowment to better understand the transformative nature of worship. A team of nine Calvin College faculty and Center for Social Research (CSR) staff are working with John Witvliet and the Worship Institute staff to study issues of “Worship, Worldview and Way of Life.”
Five of these faculty members are working together and partnering with ten local congregations in Grand Rapids to do a series of studies. The ten churches represent both urban and suburban locations as well as a variety of denominations. These studies cover worship’s relationship to everything from conceptions of community to issues of race. In keeping with the Worship Institute’s practical goals, the aim is to help congregations become more reflective about how worship can empower congregants to be transformative agents in society. CSR staff and student research assistants are providing research support, especially Gwen Einfeld, who is organizing tracking of recordings and transcripts. Read on for details of these studies and biographical sketches of the faculty conducting them.
God and Nature: Depictions of the Asian Tsunami and Hurricane Katrina in Worship
Dr. Janel Curry is conducting a project in the study group’s ten partner Grand Rapids churches. As available, transcripts of the three worship services following both the Asian tsunami (January 2, 9, and 16, 2005) and Hurricane Katrina (September 4, 11, and 18, 2005) from each church will be analyzed to see how relationships among God, nature, and humans are depicted. She will also compare sermons of the same time period from similar congregations posted on the web.
The Framing of Community and Faith: A Rhetorical Analysis of Sermons
Dr. Kathi Groenendyk is studying twelve Sunday morning worship services, spanning one year, from each of the ten partner churches in urban and suburban Grand Rapids. In the first of a series of studies, she will examine the themes, language, and rhetorical appeals to individuals and communities made within the sermons.
Understanding Racialization in Congregations
A multi-racial research team, composed of Dr. Gail Heffner, Director of Community Engagement and Dr. Denise Isom, Assistant Professor of Education, will be exploring the role of religious institutions in bridging the racial divide. Questions will be: How do the participants in these congregations understand the notion of ‘race’ and what meaning does it have in their lives and in the life of their congregation? What do their worship practices reveal about the racial ideology and identity of the congregation? The focus of this research will be to understand the lived-experiences of race for people of color and for whites in these congregations.
The Meaning Making World of African American Children
Dr. Isom is also doing a study of African American children, to give voice to how they see the world and themselves. Using several months of observational analysis, as well as interviews and focus groups, Dr. Isom is looking to capture how Black children define maleness, femaleness, “Blackness”, Christianity, and the way those ideas and identities intersect and interact. She serves as a volunteer in one of the church youth programs as well as attending the church on Sundays. That engagement will allow her to get to know the children and observe the way they live out what they think. The final stages of the study will include interviews and focus groups with the children.
Community and the Congregation
The work of Dr. Mark Mulder will examine how worship practices might affect or influence a congregation’s conception of the immediate community. The research will focus on questions of the connection between worship practices and connections to the local neighborhood. He will examine historical documents, observe churches in action, and interview members of the various congregations.
In gratitude for their participation in the studies, the Calvin Institute of Christian Worship will offer the participating congregations a variety of worship-related materials and/or conference registration. The research results will also be shared with the participating congregations. These studies have been cleared by the College’s Institutional Review Board to ensure that researchers maintain high levels of confidentiality, in turn maintaining a high level of trust with congregations.