Tag Archives: theory of change

How grant makers and nonprofit grant recipients can do great things together with data and evaluation

This is not actually a photo from the dialogue series. We refrained from taking photos, because we wanted to foster an atmosphere of candor and comfort as grantors and grantees engaged in conversation about a difficult topic. However, it is a favorite photo from another recent Tech Networks of Boston event.

 

Oh, my!  It took Tech Networks of Networks almost two years to organize and implement a series of candid dialogues about data and evaluation for grantors and nonprofit grantees, and now it’s complete.  The process was a collaboration in itself, with TSNE MissionWorks, and Essential Partners serving as co-hosts. An advisory group and planning group gave crucial input about the strategy and tactics for this event.

What you see here are a few notes that reflect my individual experience. In this article, I am not speaking on behalf of any organization or individual.

As far as I can ascertain, this series was the first in which grant makers and nonprofit grant recipients came together in equal numbers and met as peers for reflective structured dialogue. World class facilitation and guidance was provided by Essential Partners, with the revered Dave Joseph serving as facilitator-in-chief.

Here’s how I’d characterize the three sessions:

  • June 2017:  Let’s get oriented. What is the heart of the matter for grantors and grantees?
  • September 2017:  You know, we really need to address the imbalance of power in the grantor/grantee relationship.
  • January 2018:  Ok, can we agree on some best practices how to address this as grantors and grantees? Why, yes. We can.

The plan is to make the recommendations that came out of the final dialogue publicly available online, to provide a starting point for a regional or even national conversation about data and evaluation.

Meanwhile, I’d like to offer my own recommendations.  Mine are based on what I learned during the dialogue series, and also on untold numbers of public and private conversations on the topic.

 

_____________________________________________________________________________

 

My Recommendations

 

Funders can help by: 

  • Understanding that nonprofits perceive funders as having not just money but also much more power.
  • Asking nonprofits to define their goals, their desired outcomes, and their quantitative measures of success – rather than telling them what these should be.
  • Factoring in the nonprofit organization’s size, capacity, and budget – making sure that the demand for data and evaluation is commensurate.
  • Understanding the real cost in dollars to grantees who provide the data reporting and evaluation that you request.  These dollar amounts might be for staff time, technology, training, an external consultant, or even for office supplies.
  • Providing financial support for any data or evaluation that the funder needs –  especially if the nonprofit does not have an internal need for that data or evaluation.    Items to support might include staff time, technology, training, or retaining an external consultant with the necessary skill set.
  • Putting an emphasis on listening.

 

Nonprofits can help by: 

  • Engaging in a quantitative analysis of their operations and capacity, and sharing this information with funders.
  • Understanding that grant makers are motivated to see nonprofit grant recipients succeed.
  • Understanding that grant makers are often under pressure from donors and their boards to deliver a portfolio of outcomes.
  • Integrating the use of data and evaluation into most areas of operation – this means building skills in data and evaluation across the entire organization.
  • Gathering with other nonprofits that have similar desired outcomes and comparing notes on failures and best practices.
  • Fostering a data-friendly, continuous learning culture within nonprofit organizations.

 

Both groups can help by: 

  • Engaging in self-scrutiny about how factors such as race and class affect how data is collected, categorized, analyzed, and reported.
  • Talking frankly about how power dynamics affect their relationships.
  • Engaging in ongoing dialogue that is facilitated by a third party who is experienced in creating a safe space.
  • Talking about and planning the evaluation process well before the grant begins.
  • Creating clear definitions of key terms pertaining to data and evaluation.
  • Making “I don’t know” an acceptable response to a question.
  • Measuring what you really value, rather than simply valuing what you can easily measure.
  • Working toward useful standards of measurement.  Not all programs and outcomes are identical, but very few are entirely sui generis.
  • Sharing responsibility for building the relationship.
  • Speaking with each other on a regular basis.
  • Studying (and implementing) community-based participatory research methods.

 

_____________________________________________________________________________

 

And now, because I can insert a poll here, I’m going to.

 

 

 

_____________________________________________________________________________

 

And now, because I can insert a contact form here, I’m going to.  Please feel free to let me know if you’re interested in being part of a regional or national conversation about how grantors and grantees can move forward and work constructively with data and evaluation.

 

 

_____________________________________________________________________________

 

Creative Commons License
Some rights reserved. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

 

_____________________________________________________________________________

My ever expanding theory of change for nonprofit data and evaluation

Every nonprofit needs a theory of change for its technology. . .and for its evaluation process

if then

I’ve spent a lot of my professional life (thus far) thinking about the missions of nonprofit organizations, and about information/communication technologies for nonprofits.

In the past few years, it’s become fashionable to talk about the importance of a “theory of change” for nonprofits.  This is merely a way of underlining the importance of making an explicit statement about the causal relationship between what a nonprofit organization does and the impact that it has promised to deliver.  I applaud this!  It’s crucial to say, “if we take all of the following resources, and do all of the following actions, then we will get all of the following results.”  An organization that lacks the capacity to marshal those resources and take those actions needs to reconsider, because it is on track to fail. If its capacity is not aligned with its commitment, it should acquire the resources or change its commitment to results.  Of course, it some cases, it will merely need to revise its theory of change.  In any case, it will have to work backward from its mission, and understand how each component contributes to achieving it.

This kind of thinking has lead to a lot of conversations (and a lot of anxiety) in the nonprofit sector about performance measurement, outcomes management, evaluation, and impact assessment.

I’d love to have some of this conversation focus on the information/communication technologies that nonprofit organizations are using.  In other word, it’s time to be explicit about a theory of change that explains in detail how every component of the technology an organization uses contributes (directly or indirectly) to its ability to deliver a specific kind of social, cultural, or environmental impact.

Likewise, I’d love to have the conversation address the ways in which the efforts of a nonprofit organization’s performance measurement, outcomes management, evaluation, or impact assessment team contributes (directly or indirectly) to its ability to deliver the kind of impact that it promised its stakeholders.

 

 

“We count our successes in lives”

Brent James

Brent James is one of my new heroes.  He’s a physician, a researcher, and the chief quality officer of Intermountain Healthcare’s Institute for Health Care Delivery Research.

We had a very inspiring telephone conversation this afternoon, about whether the lessons learned from evidence-based medicine could be applied to nonprofits that are seeking to manage their outcomes.  We also swapped some stories and jokes about the ongoing struggle to document a causal relationship between what a health care organization (or a social service agency, or an arts group, or an environmental coalition, for that matter) does and what the organization’s stated aims are.  In fact, documenting that an organization is doing more good than harm, and less harm than doing nothing at all, continues to be a perplexing problem.  The truth may be less than obvious – in fact, it may be completely counter-intuitive.

In this phone conversation, we also waded into deep epistemological waters, reflecting on how we know we have succeeded, and also on the disturbing gap between efficacy and effectiveness.

It’s not merely a philosophical challenge, but a political one, to understand where the power lies to define success and to set the standards of proof.

I doubt that this is what William James (no relation to Brent, as far as I know) had in mind when he referred to success as “the bitch-goddess,” but there’s no doubt that defining, measuring, and reporting on one’s programmatic success is a bitch for any nonprofit professional with intellectual and professional integrity.  It’s both difficult and urgent.

What particularly struck me during my conversation with Brent was his remark about Intermountain Healthcare:

“We count our successes in lives.”

On the surface, that approach to counting successes seems simple and dramatic.  The lives of patients are on the line.  They either live or die, with the help of Intermountain Healthcare.  But it’s really a very intricate question, once we start asking whether Intermountain’s contribution is a positive one, enabling the patients to live the lives and die the deaths that are congruent with their wishes and values.

These questions are very poignant for me, and not just because I’m cancer patient myself, and not just because yesterday I attended the funeral of a revered colleague and friend who died very unexpectedly.  These questions hit me where I live professionally as well, because earlier this week, I met with the staff of a fantastic nonprofit that is striving to do programmatic outcomes measurement, and is faced with questions about how to define success in a way that can be empirically confirmed or disconfirmed.  Their mission states that they will help their clients excel in a specific industry and in their personal lives.  They have a coherent theory of change, and virtually all of their criteria of professional and personal success are quantifiable.  Their goals are bold but not vague. (This is a dream organization for anyone interested in outcomes management, not to mention that the staff members are smart and charming.)  However, it’s not entirely clear yet whether the goals that add up to success for each client are determined solely by the staff or by the client or some combination thereof.  I see it as a huge issue, not just on an operational level, but on a philosophical one; it’s the difference between self-determination and paternalism.  I applaud this organization’s staff for their willingness to explore the question.

When Brent talked about counting successes in terms of lives, I thought about this nonprofit organization, which defines its mission in terms of professional and personal success for its clients.  The staff members of that organization, like so many nonprofit professionals, are ultimately counting their successes in lives, though perhaps not as obviously as health care providers do.  Surgeons receive high pay and prestige for keeping cancer patients alive and well – for the most part, they fully deserve it.  But let’s also count the successes of the organization that helps a substantial number of people win jobs that offer a living wage and health insurance, along with other benefits such as G.E.D.s, citizenship, proficiency in English, home ownership, paid vacations, and college educations for the workers’ children. Nonprofit professionals who can deliver that are also my heroes, right up there with Brent James.  While we’re holding them to high standards of proof of success, I hope that we can find a way to offer them the high pay and prestige that we already grant to the medical profession.

%d bloggers like this: