Outcomes measurement for nonprofits: Who does the analysis?

I invite you to participate in this survey, bearing in mind that it is for recreational purposes, and has no scientific value:

There are many reasons that this survey is of dubious value, for example:

  • No pilot testing has been done to ensure that the choices offered are both exhaustive and mutually exclusive.

The list could go on, but I’ll leave it at that.  Although most of my training is in qualitative social research, I have taken undergraduate and graduate level courses on quantitative research, and the points I made about what’s wrong with my survey are what I could pull out of memory without consulting a standard text on statistics.

In other words, when it comes to quantitative analysis, I know just enough to be dangerous.

Meanwhile, I worry about nonprofit organizations that are under pressure to collect, analyze, and report data on the outcomes of their programs.  There are a lot of fantastic executive directors, program managers, and database administrators out there – but it’s very rare for a nonprofit professional who falls into any of those three categories to also have solid skills in quantitative analysis and social research methods.  Nevertheless, I know of plenty of nonprofit organizations where programmatic outcomes measurement is done by an executive director, program manager, or database administrator whose skill set is very different from what the task demands.  In many cases, even if they come up with a report, the nonprofit staff members may not even be aware that what have done is presented a lot of data, without actually showing that there is any causal relationship between the organization’s activities and the social good that they are in business to deliver.

Let’s not be too hasty in deprecating the efforts of these nonprofit professionals.  They are under a lot of pressure, especially from grantmaking foundations, to report on programmatic outcomes.  In many cases, they do the best they can to respond, even if they have neither the internal capacity to meet the task nor the money to hire a professional evaluator.

By the way, I was delighted to attend gathering this fall, in which I heard a highly-regarded philanthropic professional ask a room full of foundation officers, “are you requiring $50,000 worth of outcomes measurement for a $10,000 grant?” It’s not the only question we need to ask, but it’s an extremely cogent one!

I’d love to see nonprofit professionals, philanthropists, and experts in quantitative analysis work together to address this challenge.

We should also be learning lessons from the online tools that have already been developed to match skilled individuals with nonprofit professionals who need help and advice from experts.  Examples of such tools include the “Research Matchmaker,” and NPO Connect.

We can do better.  It’s going to take time, effort, money, creativity, and collaboration – but we can do better.

Tagged: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

3 thoughts on “Outcomes measurement for nonprofits: Who does the analysis?

  1. Mary Ann Scheirer 11/27/2012 at 11:40 am Reply

    I agree with your points on this blog, Deborah! I’m an independent evaluation consultant and have worked on short-term training with many non-profits to help them collect evaluative data to help manage their projects. It’s an uphill struggle to get them to be motivated to collect even short term outcome data, and to have the skills for it!

    The one point of your blog that I question is that non-profits with short term grant funding should be trying to show causal relationships. I focus instead on short term outcomes that they might influence with their program activities, especially good attendance and participation of clients, responsive services, and near-term behavior changes. See the recent Forum in June 2012 issue of Am. J. of Evaluation, detailing different types of evaluation as appropriate for different stages of a programmatic life cycle.

    By the way, as I see you are focused on technology for the non-profit sector, are you in touch with Idealware? See Idealware.org for lots more information!

    Best wishes, Mary Ann

  2. Ann K. Emery 11/27/2012 at 2:46 pm Reply

    Deborah,

    Great insights! Innovation Network recently conducted research on nonprofit evaluation practices and capacity — The State of Evaluation project. We surveyed nonprofits across the US and included questions similar to yours. You can download the 2012 and 2010 reports via http://www.stateofevaluation.org.

    Please let me know what you think of the State of Evaluation findings. I’m interested in hearing your reactions to the research.

    Thanks, Ann

  3. Cathy Burack 12/04/2012 at 11:10 am Reply

    I’m at the Center for Youth and Communities at the Heller School, Brandeis University, and we have served as the external evaluators on hundreds of funded projects at schools, higher ed institutions and non-profits. In addition, we are committed to increasing the capacity of organizations to evaluate their own programs and often work with groups in that regard. We always talk about evaluation as a management tool that can both “prove and improve” programs. Our work is also informed by our social justice mission, and we see evaluation as a way that those without a voice can have one (e.g. “that program isn’t doing what it’s supposed to”). No matter what size they are, evaluations require resources, thus it’s important that organizations be able to articulate why they are evaluating something. If it’s for internal decision making, marketing, etc. then the threshold for evidence (e.g. an exit survey of clients) is not nearly as high as it would be to get a Dept of Ed grant (e.g. requires a comparison group). You are so right about the pressures on organizations to demonstrate impact and be accountable so increasingly the amount of resources that institutions must commit to creating systems of evidence is growing. What we know is that most organizations are awash in current data that they under utilize, and that even when extra data are gathered being able to analyze it and report it is a huge barrier. Perhaps one way to think about this, especially for those of us who serve as evaluation partners, is to figure out cost effective ways that organizations can get help up front in thinking through the evaluation for what question, and evaluation planning and then at a later point in time for doing analysis and reporting. One example on the reporting end was a system we set up with Abt Associates for the Learn and Serve grantees of the Corporation for National and Community Service. It was web based and grantees entered their data, and were able to pull reports – the numbers were crunched by the system. Alas, the entire program was cut, but I think it serves as a good example of what can be created to help organizations.
    Thanks Deborah, you always get us thinking and acting!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: