Posted on July 14, 2011
My bishop sent a letter to Indiana clergy calling for participation in the Call to Action process. Here is a direct quote from the letter:
This denominational focus on Vital Congregations means that every United Methodist congregation in the world will be setting goals around these five factors (which have been proven through the Call to Action study to be those areas which best measure the vitality of a local congregation):
Disciples worship – average worship attendance
Disciples make new disciples – professions of faith
Disciples grow – number of small groups for faith development
Disciples engage in missions – number of people engaged in mission
Disciples give to mission – amount of money given to mission
It all sounds good, yes?
But there are a couple of problems.
First, the Call to Action study does not prove anything. It provides some statistical evidence for some conclusions. How much evidence it gives we cannot know because the actual data and statistics that would have to be reported to make that judgement have not been shared.
Second, the five “factors” listed above come from two different places in the CTA report. Here is a brief summary of the process.
The report first collected a set of measurements that it said were “proxies” for congregational vitality. These included things such as worship attendance, giving by members, and total membership. These were not causes of vitality, but the way we measure it. The were identified in large part because they were numbers the denomination already collects.
The consultants then engaged in a statistical process known as exploratory factor analysis to group these measures of vitality into a groups or “factors.” Using these groups, they sorted UM congregations into high, medium, and low vitality based on arbitrary dividing lines.
Then, they used another statistical tool called multiple regression to figure out what data that we have about the congregations most account for some of them being high vitality and others being low or middle. From this regression, the consultants indicated four “drivers,” including things such as number of small groups and the mix of worship styles at a church.
Aside from my generic concern that the statistical tools used in this analysis do not prove anything about the causes of vitality, we also continue to act as if the tools are magic black boxes that spit out truth. What statistical tests actually do, when reported in ways that allow this, is provide evidence that can be used to engage in further thinking, research, and action. We treat the report like holy writ either because we do not understand it or want to appeal to the authority of numbers.
But here is the big thing: The conference is misreading the report even as it comes up with its five factors.
One of the factors (number of small groups) is a “key driver” of vitality according to the consultants. But not really, since the factor described in the letter is “number of small groups for faith development.” The CTA does not distinguish between different types of small groups, it just counts them.
Three of the factors (worship attendance, professions of faith, and missions giving) are not “drivers” of vitality but are used in the CTA report to determine which congregations are vital. They do not cause vitality; they measure it. And they are not “proven” to measure vitality, they are chose as the best proxies we have based on surveys and other conversations.
Finally, one factor (number of people engaged in mission) I cannot find in either measures or drivers of congregational vitality in the Towers Watson appendix to the CTA report.
So, we are using the Call to Action as the rhetorical justification for the new processes, and yet of the five factors listed in the letter, only one is even a “driver” of congregational vitality according to the Call to Action report. This list of five looks more to me like a set of measures from the North Alabama dashboard that has gotten so much attention recently.
Here is where I come down on all this. I would rather just be told to do something than be “persuaded” into doing something with sloppy arguments. I’d rather we not misuse statistics. I’d rather we talk about the Bible and Wesleyan tradition. But absent these, just give me orders. Please, don’t ask me assent to arguments I know to be flawed and full of holes.