Measurement Malaise?

Event Summary    

Bill Schambra’s introduction invited the panel to consider the following questions: “Can we in fact really measure what we’re doing in the world of philanthropy? Can any of the scales, graphs, charts, yardsticks and calipers that we use to generate numbers give us a more reliable sense of how we’re doing than would, say, an educated hunch or wise reflection? Or as Peter Dobkin Hall suggests, is this devotion to measurement ultimately a kind of cultural artifact, a part of the ethos of the world of philanthropy, more akin to ceremony than to science?”

 
The panel focused on Peter Dobkin Hall’s essay, “A Solution is a Product in Search of a Problem: A History of Foundations and Evaluation Research” (available at: http://ksghome.harvard.edu/~phall/), and panelists included Hall, Gary Walker, director of Public/Private Ventures, Andrew Mott, director of the Community Learning Project, and Rick Cohen, executive director of the National Committee for Responsive Philanthropy.     

Panel Presentations   

Hall summarized his essay, which argued that evaluation threatens the promotion of pluralism in foundations. “That notion that was at the heart of the Bob Jones University v. United States decision back in 1983. The core of that decision was that you didn’t get the tax exemption unless you were fulfilling whatever the relevant public policy in that area was, and it was a very disturbing decision. In fact, it was so disturbing that Justice Brennan wrote a dissent in which he said…the reason why the law in our country treats charity as it does is not because it wants to turn charity into an arm of government but because it wants to foster pluralism, and that’s why our charities law is generally not concerned with the substance of what is charitable.” He added that nonprofits are in a unique position to “We shouldn’t confuse [the professionalization of a philanthropy] with a notion that we’re going to get totally objective or even useful conclusions from using evaluation research in organizations whose purposes are fungible and which have very little accountability. They have no owners, they have no clients.”

 
Gary Walker agreed with Professor Hall that one of the major problems in nonprofits was their lack of accountability and the lack of media attention that might otherwise create accountability. Contributing to this problem was the fact that board members “who have made millions of dollars by… being very practical... And they walk into that foundation boardroom, and they love to set all that aside” and think in grandiose terms about changing the world. He argued that evaluations, though very imperfect, are at least a means to assess a foundation’s work somewhat objectively in the face of a great deal of unaccountability. Evaluation also allows a foundation to get a rough sense of what is working and what needs to be improved. “All those evaluations showing that things don’t work as well as everybody dreamed they would, you’re getting an increasing number of foundations, particularly a lot of new ones, with money coming out of the high tech sector saying, ‘A-ha! The problem here is we should be focusing on quality implementation…’”

Walker also warned against the lure of scientific methodology that leads foundations to hire professional evaluators who may not be familiar with the programs they are evaluating, or the nature of the problems those programs seek to address. He suggested that experts in methodology should be paired with someone familiar with the context in order to perform a more thorough evaluation.

Andy Mott also added that evaluation often leads to an often inappropriate emphasis on quantitative results where such results are difficult to obtain or meaningless, as in the low-income communities where he worked. He followed up on Professor Hall’s point about the political nature of evaluation with a short discussion of the problem of censorship in nonprofit evaluation. Despite this, Mott supported evaluation as an indispensable tool in some contexts, though he felt that, in cases where the evaluation is of an organization receiving funding directly, the evaluation should be used “as an opportunity to strengthen organizations, to strengthen the grantees.”

He recommended that non-professional evaluators who simply have experience of the organization’s context and know the nonprofit world be used rather than those with academic training in the field. The emphasis of evaluations should be placed on creating a strong framework for effective self-assessment within the organization. Mott distinguished between such evaluations of organizations receiving direct funding, and evaluations of government programs. In the case of government evaluations, he argued that the conventional large, longitudinal evaluations were unnecessary where simpler kinds of evaluation involving citizen monitoring would suffice. He gave several detailed examples, which can be found in the original transcript.

Rick Cohen discussed the usefulness of evaluation as a way of monitoring the work of foundations which themselves are loathe to evaluate their own programs. The political motives of foundations that do evaluate, he added, are often more along the lines of fending off criticism than “generating information.” In government programs, by contrast, because of the Freedom of Information Act, evaluations will eventually become public and have a greater ability to influence the work of government agencies. Since this is not true of private foundations and there is little external pressure to change, evaluation results frequently fail to measure or recommend substantial changes. More often than not, evaluation serves as legitimating function for foundations.

Professor Hall summarized the speakers’ comments as all leading to the conclusion that evaluations served a largely therapeutic purpose, and it shouldn’t be advertised “as anything but an exercise in organizational psychology. Schambra said that this was because, fundamentally, evaluations cannot work unless a scientific control group could be established to test the success of any program. However, Gary Walker disagreed that the scientific method was always appropriate to the situation and interviews and other simpler measures could suffice, though in some cases a control group could be telling. Mott suggested that because of the industry’s recent turn towards professionalization, simple questions tended to get asked in unnecessarily complex ways, which resulted in money wasted on overly complicated evaluations where simple ones would have sufficed.

At this point, the discussion was opened to the audience. Dorothy Weiss, of OMB Watch, reported that she had seen evaluations used to change foundation policy about grantmaking, and that they have helped small family and regional foundations appraise their effectiveness. However, she rejected the notion that all philanthropy could be measured, a sentiment she felt came from business models of organization. Professor Hall agreed, saying that “We’re still using rational choice models where they’re completely inappropriate and it’s really too bad.” Gary Walker replied that, “Organizations, like people, are a combination of irrational and rational behavior. But if you have a tool that you think might promote a little more of the latter, since the former does dominate most of the time, you use it.”

More discussion of the effectiveness of foundations followed, and can be found in the full transcript.    

For Further Information  

To request further information on this event, the transcript, or the Bradley Center, please contact Hudson Institute at (202) 974-2424 or e-mail Kristen at kmcintyre@hudson.org.