How grant makers and nonprofit grant recipients can do great things together with data and evaluation

This is not actually a photo from the dialogue series. We refrained from taking photos, because we wanted to foster an atmosphere of candor and comfort as grantors and grantees engaged in conversation about a difficult topic. However, it is a favorite photo from another recent Tech Networks of Boston event.

 

Oh, my!  It took Tech Networks of Networks almost two years to organize and implement a series of candid dialogues about data and evaluation for grantors and nonprofit grantees, and now it’s complete.  The process was a collaboration in itself, with TSNE MissionWorks, and Essential Partners serving as co-hosts. An advisory group and planning group gave crucial input about the strategy and tactics for this event.

What you see here are a few notes that reflect my individual experience. In this article, I am not speaking on behalf of any organization or individual.

As far as I can ascertain, this series was the first in which grant makers and nonprofit grant recipients came together in equal numbers and met as peers for reflective structured dialogue. World class facilitation and guidance was provided by Essential Partners, with the revered Dave Joseph serving as facilitator-in-chief.

Here’s how I’d characterize the three sessions:

• June 2017:  Let’s get oriented. What is the heart of the matter for grantors and grantees?
• September 2017:  You know, we really need to address the imbalance of power in the grantor/grantee relationship.
• January 2018:  Ok, can we agree on some best practices how to address this as grantors and grantees? Why, yes. We can.

The plan is to make the recommendations that came out of the final dialogue publicly available online, to provide a starting point for a regional or even national conversation about data and evaluation.

Meanwhile, I’d like to offer my own recommendations.  Mine are based on what I learned during the dialogue series, and also on untold numbers of public and private conversations on the topic.

 

_____________________________________________________________________________

 

My Recommendations

 

Funders can help by: 

• Understanding that nonprofits perceive funders as having not just money but also much more power.
• Asking nonprofits to define their goals, their desired outcomes, and their quantitative measures of success – rather than telling them what these should be.
• Factoring in the nonprofit organization’s size, capacity, and budget – making sure that the demand for data and evaluation is commensurate.
• Understanding the real cost in dollars to grantees who provide the data reporting and evaluation that you request.  These dollar amounts might be for staff time, technology, training, an external consultant, or even for office supplies.
• Providing financial support for any data or evaluation that the funder needs –  especially if the nonprofit does not have an internal need for that data or evaluation.    Items to support might include staff time, technology, training, or retaining an external consultant with the necessary skill set.
• Putting an emphasis on listening.

 

Nonprofits can help by: 

• Engaging in a quantitative analysis of their operations and capacity, and sharing this information with funders.
• Understanding that grant makers are motivated to see nonprofit grant recipients succeed.
• Understanding that grant makers are often under pressure from donors and their boards to deliver a portfolio of outcomes.
• Integrating the use of data and evaluation into most areas of operation – this means building skills in data and evaluation across the entire organization.
• Gathering with other nonprofits that have similar desired outcomes and comparing notes on failures and best practices.
• Fostering a data-friendly, continuous learning culture within nonprofit organizations.

 

Both groups can help by: 

• Engaging in self-scrutiny about how factors such as race and class affect how data is collected, categorized, analyzed, and reported.
• Talking frankly about how power dynamics affect their relationships.
• Engaging in ongoing dialogue that is facilitated by a third party who is experienced in creating a safe space.
• Talking about and planning the evaluation process well before the grant begins.
• Creating clear definitions of key terms pertaining to data and evaluation.
• Making “I don’t know” an acceptable response to a question.
• Measuring what you really value, rather than simply valuing what you can easily measure.
• Working toward useful standards of measurement.  Not all programs and outcomes are identical, but very few are entirely sui generis.
• Sharing responsibility for building the relationship.
• Speaking with each other on a regular basis.
• Studying (and implementing) community-based participatory research methods.

 

_____________________________________________________________________________

 

And now, because I can insert a poll here, I’m going to.

 

 

 

_____________________________________________________________________________

 

And now, because I can insert a contact form here, I’m going to.  Please feel free to let me know if you’re interested in being part of a regional or national conversation about how grantors and grantees can move forward and work constructively with data and evaluation.

 

Name(required)

Email(required)

Website

Comment(required)

Submit

Δ

 

_____________________________________________________________________________

 

Some rights reserved. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

 

_____________________________________________________________________________

My ever expanding theory of change for nonprofit data and evaluation

Drowning in data, drafting a data checklist, and asking “WHY?”

Two poster boys of nonprofit data sanity: Bob Penna (l) and Steve Pratt (r).

Now that TNB Labs is up and running, we’re receiving a lot of requests from nonprofit organizations who are perplexed about how to manage the data that they have, before they plunge any further into data analytics or think about acquiring a new data analysis tool.  This has given me a lot of opportunities to reflect on how difficult it can be for people whose expertise lies elsewhere to orient themselves to data governance.

Steve Pratt‘s blog article “Drowning in Data?” has been a huge inspiration.  In it, he explains the importance of data inventories, and offers to send the Root Cause template to anyone who requests it.  I highly recommend that you send an email to info@rootcause.org, and ask for a copy.

At the same time, as I went over Steve’s template, I had a nagging feeling that we needed something even more elementary.  Remembering my friend Bob Penna‘s exhortation of a few months before, about asking “who, when, where, what, how, and why,” I quickly drafted a data checklist that focused on those basic questions.  When I sent it to Bob, he very quickly returned it with some excellent enhancements; the most brilliant one was to start the checklist with the question “WHY?”  As he very sensibly pointed out, if you can’t come up with a good reason why you are collecting, analyzing, reporting, and archiving information, you might as well stop there.  In the absence of a persuasive answer to the question “why?” there’s no need to ask “who, when, where, what, and how;” in fact there’s no reason to collect it at all.

With that wisdom in mind, I have tweaked the draft of the data checklist, and herewith present it to you for feedback. This version is the result of a Penna/Finn collaboration:

You can view it by clicking on this link.

Before you take a look at it, I recommend reading “Drowning in Data?”  After you’ve perused the spreadsheet, I recommend reading Bob Penna’s book, “The Nonprofit Outcomes Toolbox.”

 

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

Every nonprofit needs a theory of change for its technology. . .and for its evaluation process

I’ve spent a lot of my professional life (thus far) thinking about the missions of nonprofit organizations, and about information/communication technologies for nonprofits.

In the past few years, it’s become fashionable to talk about the importance of a “theory of change” for nonprofits.  This is merely a way of underlining the importance of making an explicit statement about the causal relationship between what a nonprofit organization does and the impact that it has promised to deliver.  I applaud this!  It’s crucial to say, “if we take all of the following resources, and do all of the following actions, then we will get all of the following results.”  An organization that lacks the capacity to marshal those resources and take those actions needs to reconsider, because it is on track to fail. If its capacity is not aligned with its commitment, it should acquire the resources or change its commitment to results.  Of course, it some cases, it will merely need to revise its theory of change.  In any case, it will have to work backward from its mission, and understand how each component contributes to achieving it.

This kind of thinking has lead to a lot of conversations (and a lot of anxiety) in the nonprofit sector about performance measurement, outcomes management, evaluation, and impact assessment.

I’d love to have some of this conversation focus on the information/communication technologies that nonprofit organizations are using.  In other word, it’s time to be explicit about a theory of change that explains in detail how every component of the technology an organization uses contributes (directly or indirectly) to its ability to deliver a specific kind of social, cultural, or environmental impact.

Likewise, I’d love to have the conversation address the ways in which the efforts of a nonprofit organization’s performance measurement, outcomes management, evaluation, or impact assessment team contributes (directly or indirectly) to its ability to deliver the kind of impact that it promised its stakeholders.

 

 

TNB Labs has launched, and its first priority is data services for nonprofits!

 

Greg Palmer and I are very pleased to announce the launch of TNB Labs, a company dedicated to providing nonprofit organizations with high quality data services and program management.  TNB Labs works with organizations to audit and assess data management methodology, develop and implement data standards, and provide structural oversight through data governance to organizations of all sizes.

Through our partnership with Tech Networks of Boston, we have had a unique opportunity to listen to hundreds of stakeholders at TNB’s Roundtables uncover a need to elevate the role of data in nonprofit organizations.  TNB Labs is focused on improving the capacity of nonprofit organizations at every stage of the outcome management process.

The immediate priorities of TNB Labs are:

1) Providing Master Data Management (MDM) services to nonprofit organizations in support of their missions, focusing on data governance, data quality, data modeling, data visualizations, and program evaluation.

2) Providing workforce program management for Desktop Support Technicians (DST), Data Support Analysts (DSA), and Data Analytics/Data Evaluation entry level professionals.

3) Managing the TNB Roundtable series, which is now jointly owned by Tech Networks of Boston and TNB Labs.

TNB Labs is led by Greg Palmer (chief executive officer), and Deborah Elizabeth Finn (chief strategic officer).  The other co-founders are Bob Master (former CEO of Commonwealth Care Alliance) and Susan Labandibar (founder of Tech Networks of Boston).

TNB Labs is here to solve your problems.  Please contact us with any questions and comments you have about TNB Labs, or to learn more about data management or program management services that might be helpful to your organization.

Best regards from Deborah and Greg

Greg Palmer
gpalmer@tnblabs.org
508.861.4535

Deborah Elizabeth Finn
definn@tnblabs.org
617.504.8188

TNB Labs, LLC
PO Box 2073
Framingham, MA 01703
www.tnblabs.org

 

 

It’s not just a half-day outcomes management training for nonprofit executives – it’s an occasion for rejoicing!

For more than two years, I have been worrying aloud about the lack of training for nonprofit professionals who want to lead their organizations in implementing outcomes management and data visualization.  Today I’m rejoicing, because Tech Networks of Boston opened registration for a free (and sales-pitch-free) half-day outcomes management training for nonprofit executives.

It’s happening in April because some wonderful allies have stepped up – such as TNB’s co-hosts, the Mel King Institute and the College of Public and Community Service at the University of Massachusetts, and the wonderful Kathryn Engelhardt-Cronk of Community TechKnowledge, who will serve as our trainer.

This isn’t the full series of three day-long trainings on outcomes management and outcomes data visualization that I had originally envisioned, and that I still hope we can organize.  If we are able to do that, the other trainers will be the equally wonderful Beth Kanter and Georges Grinstein.  Right now, I’m looking at plans for Kathryn’s half-day outcomes management training as a miracle in itself, but also as the thin edge of the wedge.  (If you prefer more up to date jargon, you can call it a “proof of concept.”)

Of course, my thinking has become even more grandiose since I originally came up with the idea of a three-day outcomes/data viz training series.  Now I’m thinking in terms of a “Massachusetts Institute of Nonprofit Technology,” in which the first initiative would be a degree program in nonprofit data analysis.

Let’s take this training opportunity, which will be brief in comparison to the more elaborate programs that I’ve envisioned, and build on it!

 

 

How much fun is the Nonprofit Technology Conference? This much fun. (Plus some thoughts about shifting from tactical to strategic support of nonprofit organizations.)

Photo by Peggy Duvette of Netsuite.Org.

The good folks of Netsuite.Org had a great idea for their exhibit area at the Nonprofit Technology Conference this year.  They asked attendees to describe their technology visions in three words.  I chose “shared” “data,” and “outcomes.” and an artist quickly drew up a visual to express this.  (Unfortunately, I did not note down her name; I hope I can find it in order to give her proper credit for her work.)  The photo shown above was taken by Peggy Duvette, and as you can see, I was delighted to see this concept, which is part of Tech Networks of Boston’s strategic thinking, become part of the patchwork quilt of ideas that were being expressed.

Here’s a close-up of the TNB concept:

I took this photo at the Netsuite.Org booth, at the 2015 Nonprofit Technology Conference. Alas, I did not note down the name of the artist who did this drawing.

At TNB, we are thinking more and more about collaborative technology management – not just in terms of how we work with our nonprofit clients, but also about how clusters of NTAPs and nonprofits can work together toward a shared long term goal.   We have great relationships (and in many cases, shared nonprofit clients) with some great local nonprofit technology assistance providers, such as Annkissam* and 501Partners.  The three NTAPs are already collaborating on a series of sales-pitch-free evenings in which local nonprofit professionals are offered pro bono tech consultations.

However, the potential exists to do so much more, especially considering how many clients we share.

Wouldn’t it be great if the three NTAPs could offer their shared clients the following:

1) Seamless integration of TNB, AK, and 501P’s services.

2) Shared best practices for clusters of nonprofits with similar programs, operations, or missions.

3) Coordinated outcomes measurement and management for nonprofits that have overlapping constituencies.

The joy of #15NTC is in realizing that although we are just three NTAPs in one region, we are part of a wider movement.  In fact, if you were to look at the entire collection of artist’s renderings that were done at the Netsuite.Org exhibit area, you’d see that many nonprofit organizations are on the cusp of dreaming this dream.  Most of in the nonprofit sector understand that for lasting positive change in the world, one program at a single nonprofit organization is not enough.  The future is in sharing and coordinating our work.  What if nonprofit technology assistance providers started with that challenge, rather than the challenge of keeping a network server from crashing?  The emphasis would shift from the tactical support of nonprofits to the strategic support of their missions.  And by “missions,” I don’t mean vague statements; I mean specific (and even quantifiable) positive changes that nonprofit profits have committed themselves to delivering to their stakeholders.

Because mission achievement is why we all get up in the morning to do our jobs.

And because building a nonprofit technology movement that supports mission achievement is the best possible reason for participating in the Nonprofit Technology Conference.

 

* I also serve Annkissam directly as a consultant.

 

 

 

“Accidental Evaluator” is the new “Accidental Techie.” I’m just saying.

Laura Beals, who is director of evaluation at Jewish Family and Children’s Services of Boston, published a great article on the NTEN blog earlier this month, called “Are You an ‘Accidental Evaluator?’ “

I think that this is a great question to ask, because many nonprofit professionals currently managing program evaluation within small nonprofits are indeed coming to the task with less preparation than they would like.  Perhaps they are program directors, or grant writers, or chief financial officers, or database administrators.  And now the pressure is on them to come up with numbers that show that their organizations are actually creating the positive change in the world that the organization has promised to deliver.

In fact, many of today’s accidental evaluators at nonprofits are in the same position that accidental techies were ten or fifteen years ago.

I respectfully disagree with those of my esteemed colleagues who want to help nonprofit professionals by reassuring them that they don’t have to meet the standards of academic peer reviewed journals when they use data to tell their stories.  While it’s true that the level of rigor required for nonprofit programmatic evaluation is much less strict, it’s not enough to point this out and encourage nonprofit professionals to relax.  Those nonprofit professionals are running organizations with a special legal status that make them answerable to the public and responsible for contributing to the common good.  This is a serious ethical obligation.

From my point of view, those of us who understand the importance of evaluation in the nonprofit sector should be working to deliver appropriate forms of professional development to “accidental evaluators,” just as NTEN has labored mightily to deliver professional development to “accidental techies.”

In fact, NTEN itself is in a very good position to assist “accidental evaluators,” because many technology topics are intimately tied up with nonprofit evaluation, such as database development, data integration, and data visualization. Indeed, if you look at some the companion articles on the NTEN blog, you’ll see that this effort is underway:

• How Small Organizations Can Use Affordable ICT-tools to Measure Impact
• On Transparency, Data, and Trust
• Measuring What Matters: Outcome-based Evaluation in Arts and Social Justice Projects

I’m pleased to say that here in Boston we’re actively addressing this.  For example, Laura and her wonderful JF&CS colleague Noah Schectman recently led a meeting of local nonprofit professionals who are seeking to improve their skills in bridging between evaluation and technology.  A pivotal moment at this session came when the executive director of a tiny nonprofit raised her hand and asked Noah, “Will you be my best friend?”  Noah’s face lit up, and he told her that he would.  That’s the kind of reassurance that we should be offering nonprofit professionals who feel overwhelmed; we should be telling them that support and training are on the way.

 

Visualizing the role of data for mission-based organizations – Round II

I am much obliged to all the good folks who have posted suggestions and feedback about my first attempt to create an image that would represent my thinking on the role of data in mission-based organizations.  Likewise, those who emailed me their thoughts deserve thanks!
I’ve created a revised version that incorporates some of the feedback.  Before you take a look at it, please bear in mind that:

1. I am not a graphic designer.
2. I am not attempting to create a graphic that illustrates everyone’s ideas about the role of data in a mission-based organization.  I am merely trying to illustrate my ideas.

Visualizing the role of data for mission-based organizations – Round II

Item #2 on the list notwithstanding, I am enjoying very much the opportunity to learn more about what others in the field think about (and visualize) when they ponder the role of data in our sector.  Once again, I invite you to post your reflections, suggestions, and questions in the comments section here on this blog.

Visualizing the role of data for mission-based organizations

First of all, I want to say that I distrust the term “data-driven organization,” even though some very highly respected colleagues use it in their writings and their conversations.

I firmly believe that if your organization is driven by data, you’re stopping too soon.

It’s important to roll that data (which is raw material) into information (which has been sorted and analyzed), to roll that information into knowledge (which has been enhanced by understanding of context), and to roll that knowledge into wisdom (which been enhanced by experience and intuition). From there you can proceed to good decisions, and ultimately to mission success; moreover, at this point you of course now have more data.  From there, it’s an opportunity for continuous improvement and possibly even further innovation.

My challenge right now is to come up with a clear image to convey this.  The one that springs naturally to my mind is linear, but trusted advisors seem to favor a more cyclical illustration.

Please take a look at these two logos (which were created by yours truly), and tell me which one gets my message across most effectively:

Linear logo

Cyclical logo

And please feel free to post comments here (or send me email) to elaborate on your thoughts about this!

“Forgive and remember”

File this under “lessons about failure that the nonprofit sector can learn from medical sociology.”

Forgive and Remember: Managing Medical Failure, by Charles Bosk, is a classic of medical sociology, an analysis of how coping with failure is built into the training that surgeons receive in hospital rounds, mobidity and mortality conferences, and other settings.

Please note that I am not claiming that surgeons themselves have a lot to teach us about dealing with failure, because my experience is that while their sub-culture does have rituals and protocols that they enact privately, they still have a way to go in transparency and accountability to others.

This has been my experience in three instances of major surgery:

1. Surgeon did not follow instructions given by the specialist physician managing my condition.  Acknowledgement: Partial.  Apology: No.
2. Surgeon did not inform me that the tumor to be removed might be malignant and require addition surgery until I was under anesthesia. Acknowledgement: Yes, after I complained. Apology: Yes, after I complained.
3. Surgeon did not respect my request regarding administration of anesthesia:  Acknowledgement: Yes, after I complained. Apology: No.

Not that I am bitter.

But let’s face it:  most of us are highly invested in showing the world that we are skillful, trustworthy, and deserving of whatever prestige is ascribed to us.  As a patient, I naturally blamed the surgeons, not only for their errors in judgement, but for the instances in which they failed to acknowledge or apologize for their mistakes.  As a fellow human being, I completely empathize with their reluctance.  I imagine that that reluctance is more acute among professionals who have to cut people open.  Their work is obscenely invasive but often lifesaving,  and therefore must maintain an impeccably trustworthy reputation.

That’s why Brent James is one of my heroes, along with surgeons and physicians like him who are putting it on the line for evidence-based practice.  There will be no accountability, transparency, or improvement in health care unless successes and failures are accurately documented.  Those results must then be carefully analyzed, made available to the public in appropriate ways, and used to improve their efforts.

As with medicine, so with other mission-based organizations.  We need to track outcomes, acknowledge failures, and then do better.  If it takes a pink feather boa and an amusing ritual for nonprofits to get there, I’m all for it, though I’m not expecting surgeons to adopt the feather boa.

As for the slogan, “forgive and remember,” I think of it as both a spiritual and a practical precept.  We not only need to forgive ourselves and others when we have failed – we also need to bear the lessons of failure in mind.  Both individuals and organizations not only need to keep learning, but to take appropriate action to protect those who are at risk.

I used to work in violence prevention, and for me, one of the most heart-rending aspects of it was the well-documented difficulty in stopping offenders from repeatedly battering their loved ones.  In some cases, they simply didn’t see their behavior as abusive, or their loved ones didn’t see any alternative to accepting abuse.

As I reflect on that today, it drives home very painfully the lesson that we cannot always change others, or even control a specific behavior of theirs.  The old cliche that they “have to really want to change” is true, and it’s also true that not everyone who wants to change can do so. This is the really difficult side of facing failure for nonprofit organizations – in some cases, there may be no alternative to severing ties with individuals or organizations, if the organization is going to face its failure and move on.  It’s going to take more than a pink feather boa, a “joyful funeral,” or a FailFaire to get past that.  When the well-being of vulnerable people is on the line, there are cases where forgiving and remembering is crucial, but it isn’t enough.

---

Bonus item:

Q:  How many psychotherapists does it take to change a lightbulb?

A:  Only one, but the light bulb really has to want to change.

Adventures in failure (and ritual studies): The “joyful funeral”

Beth Kanter and I are not twins who were separated at birth, but we have some things in common.

• Distant past:  We both graduated from Bennington College. (She was a couple of years ahead of me.)
• More recent past: We both joined the circuit rider movement and became nonprofit technology professionals. (She was a couple of years ahead of me here as well.)
• Present: We both have been thinking a lot lately about the power of acknowledging failure. (She’s written much more on this topic than I have.)

So perhaps if we were twins, I wouldn’t be the evil twin (I hope) but the lagging twin.  She’s succeeding at writing in a very engaging and helpful way about failure, and I am definitely benefiting from that.  Thinking about how to acknowledge failure flows very naturally from my current absorption in outcomes management for nonprofit organizations.

Beth recently published a blog article on “Six Ways Nonprofits Learn from Affordable Losses or Little Bets to Improve Impact” that appealed to me greatly, mostly because some the practices described have a ritual component.

I was especially excited when I saw that Beth had included the “Joyful Funeral” custom that was created by Moms Rising.  I had heard rumors of this ritual in nonprofit management circles, but couldn’t remember the details.  Fortunately, Beth’s article includes a cool video, in which she interviews Ashley Boyd about what it really entails.

Now, unlike Beth, I studied sociology of religion as a graduate student, and have a longstanding interest in ritual studies.  Regardless of one’s religious beliefs and affiliation – or lack thereof – it’s easy to see that ritual often has great power in assisting human communities that are confronted by change or loss.

Let’s look at the characteristic stages of a “rite of passage:”

• Detachment or withdrawal from the status quo
• Transition
• Reincorporation into the social group

Likewise, consider a purification ritual, in which the transition in question is from an “unclean” to a “clean” state.

I propose that we think of a joyful funeral as a combination of passage and purification.  The individual or organization has an opportunity to mark the change (which may also be a loss) from a viable initiative to a failure, to acknowledge shortcomings, to mourn, to be supported by the community, and to achieve closure, and to begin the next stage of life.

Many people are left cold by any kind of ritual, and others are put off by the links between elaborate ritual and religious institutions from which they are alienated.  For that reason, I would never argue that a joyful funeral (or any of the other celebrations of failure that Beth describes) should be attempted by everyone.  But for many of us, a ritual can be a comfort, especially if it doesn’t demand that we buy into a dogma or denomination.  A ritual can also be goofy and fun.

I like the idea of building laughter without humiliation into a ritual acknowledgement of failure. It’s less scary and less punitive than a solemn occasion, and better for strengthening ties among the team and making it fun to learn from mistakes.  For this reason, I recommend the “DoSomething PinkBoa FailFest” to beginners in the art of failing and moving on.

---

Bonus item:  a joke for people who take ritual a little too seriously.

Q:  What’s the difference between a liturgist and a terrorist?

A:  You can negotiate with a terrorist.

---

 

Let’s revisit the concept of failure-friendliness

Eight years ago, I wrote a blog article about failure-friendliness in nonprofit technology. It was very much inspired by my friend and colleague, Dan Scharfman. Since Dan died this week, and this is also a week when I have been thinking hard about the obstacles that nonprofit organizations face in tracking their outcomes, it seems appropriate to reprise the article here and now. Having coped with the need for failure-friendliness in nonprofit technology for years, I see that my understanding is still superficial when it comes to the difficulties that nonprofits have in acknowledging programmatic failure. I invite your thoughts on how we can be more transparent about and more open to learning from failure. Meanwhile, special thanks go out to Beth Kanter, for her outstanding blog articles on this topic.

Wed 26 Jan 2005 05:41 PM EST

The term “failure-friendly organization” was first introduced to me by a colleague I revere – Dan Scharfman of Baird Associates.

My first impression was that he was an unlikely champion of failure, since Massachusetts is well-supplied with nonprofit organizations that consider the technology services that he has provided to them very successful indeed.

However, many of us in the nonprofit sector have seen the following things happen with major implementations or upgrades:

• The technology doesn’t work, or doesn’t work nearly as well as it should;
• The intended users won’t have anything to do with the technology;
• Major changes in technology in the outside world quickly render the organization’s choices obsolete;
• Programmatic priorities change, and the technology is all but irrelevant;
• The organization has not factored in the shocking cost of customizing, tweaking, maintaining, and upgrading the technology.

Although techies vary greatly in their attitudes about projects that don’t work out, we also tend to make tacit assumptions that everyone concerned understands that we are not engaged in an exact science but in an evolving process.

Techies also tend to regard failure as pretty interesting – as a good source of information about what ought to be fixed when Version 97.53.01 of the software is released.  We also enjoy working on cool tools, even if such tools don’t actually deliver the outcomes desired by those who are underwriting the project.  This form of process orientation can be less than endearing to decision-makers in nonprofit organizations.

Oddly enough, nonprofit workers tend to be very good at process orientation when they are on familiar ground.

Sometimes this process orientation is a grim necessity, with governmental agencies strictly mandating, auditing, and enforcing protocols that nonprofits must follow in order to maintain their tax-exempt status, accreditation, or contracts for services.  These are headaches that would impel just about any organization or individual to worry a great deal about operating according to plan and documenting the process, rather than ensuring a specific outcome. This of course is a very “functional” (or “instrumental“) form of process orientation.

A more “expressive” form of process orientation is also frequently seen in nonprofit organizations – manifesting as a desire to be flexible and responsive to changing situations, or as a desire to arrive at decisions through consensus.  However, it can be difficult to extend that attitude to technology, which tends to be difficult for non-specialists to comprehend, time-consuming, and expensive.

Another challenge is that organizations and individuals (including yours truly) can be reluctant to cut their losses, and say, “This isn’t working.  Let’s stop, figure out why, and decide on some next steps.”  Of course, in some settings, the decoded version of this message is “Let’s find someone to blame and punish…maybe YOU.”

Yikes!

Is there any solution in sight?  I only wish I had something certain and simple to offer.  Here are a few ideas, although none of them come with guarantees of success:

• Techies need to understand the nonprofit organizational cultures in which they are operating.  Progress toward this goal is possible if the techies listen, ask questions, and listen some more.  These conversations should start early in the planning phase.
• Nonprofit workers need to understand how technology innovations and implementations happen in real life, and have a reasonable idea of what factors can lead to unexpected outcomes in technology projects.  Progress is possible if – yes, you guessed it – the nonprofit workers listen, ask questions, and listen some more.
• Everyone needs to cooperate in creating incentives for spotting, discussing, and correcting errors rather than evading their detection.  I freely admit that I always find it easier to do these things when the mistake was made by someone else, but am always striving to do better.

I wish I could remember who it was that first said to me, “This is not about one person against another. This is about our team against the problem.”  Anyone who can say that is a saint, a boddhisatva, a tzadik, or an unusually effective manager.

“We count our successes in lives”

Brent James is one of my new heroes.  He’s a physician, a researcher, and the chief quality officer of Intermountain Healthcare’s Institute for Health Care Delivery Research.

We had a very inspiring telephone conversation this afternoon, about whether the lessons learned from evidence-based medicine could be applied to nonprofits that are seeking to manage their outcomes.  We also swapped some stories and jokes about the ongoing struggle to document a causal relationship between what a health care organization (or a social service agency, or an arts group, or an environmental coalition, for that matter) does and what the organization’s stated aims are.  In fact, documenting that an organization is doing more good than harm, and less harm than doing nothing at all, continues to be a perplexing problem.  The truth may be less than obvious – in fact, it may be completely counter-intuitive.

In this phone conversation, we also waded into deep epistemological waters, reflecting on how we know we have succeeded, and also on the disturbing gap between efficacy and effectiveness.

It’s not merely a philosophical challenge, but a political one, to understand where the power lies to define success and to set the standards of proof.

I doubt that this is what William James (no relation to Brent, as far as I know) had in mind when he referred to success as “the bitch-goddess,” but there’s no doubt that defining, measuring, and reporting on one’s programmatic success is a bitch for any nonprofit professional with intellectual and professional integrity.  It’s both difficult and urgent.

What particularly struck me during my conversation with Brent was his remark about Intermountain Healthcare:

“We count our successes in lives.”

On the surface, that approach to counting successes seems simple and dramatic.  The lives of patients are on the line.  They either live or die, with the help of Intermountain Healthcare.  But it’s really a very intricate question, once we start asking whether Intermountain’s contribution is a positive one, enabling the patients to live the lives and die the deaths that are congruent with their wishes and values.

These questions are very poignant for me, and not just because I’m cancer patient myself, and not just because yesterday I attended the funeral of a revered colleague and friend who died very unexpectedly.  These questions hit me where I live professionally as well, because earlier this week, I met with the staff of a fantastic nonprofit that is striving to do programmatic outcomes measurement, and is faced with questions about how to define success in a way that can be empirically confirmed or disconfirmed.  Their mission states that they will help their clients excel in a specific industry and in their personal lives.  They have a coherent theory of change, and virtually all of their criteria of professional and personal success are quantifiable.  Their goals are bold but not vague. (This is a dream organization for anyone interested in outcomes management, not to mention that the staff members are smart and charming.)  However, it’s not entirely clear yet whether the goals that add up to success for each client are determined solely by the staff or by the client or some combination thereof.  I see it as a huge issue, not just on an operational level, but on a philosophical one; it’s the difference between self-determination and paternalism.  I applaud this organization’s staff for their willingness to explore the question.

When Brent talked about counting successes in terms of lives, I thought about this nonprofit organization, which defines its mission in terms of professional and personal success for its clients.  The staff members of that organization, like so many nonprofit professionals, are ultimately counting their successes in lives, though perhaps not as obviously as health care providers do.  Surgeons receive high pay and prestige for keeping cancer patients alive and well – for the most part, they fully deserve it.  But let’s also count the successes of the organization that helps a substantial number of people win jobs that offer a living wage and health insurance, along with other benefits such as G.E.D.s, citizenship, proficiency in English, home ownership, paid vacations, and college educations for the workers’ children. Nonprofit professionals who can deliver that are also my heroes, right up there with Brent James.  While we’re holding them to high standards of proof of success, I hope that we can find a way to offer them the high pay and prestige that we already grant to the medical profession.

Harsh truths

Perhaps as a counter-balance to my previous blog post, I’ve been reflecting on an article by David Wong that appeared this month in Cracked.Com: “Six Harsh Truths That Will Make You A Better Person.”

While I don’t agree with all of Wong’s assertions, there are a few that I think we should take to heart in the nonprofit sector.  I present them here as bullet points:

• “The World Only Cares About What It Can Get from You”

Wong points out two corollaries to this:  “…society is full of people who need things….”  and “(e)ither you will go about the task of seeing to those needs by learning a unique set of skills, or the world will reject you, no matter how kind, giving and polite you are.”

He’s right, or at least he should be.

(We all know of high-profile nonprofit professionals whose main achievement is making people feel warm inside.  They don’t contribute to any lasting and positive change, but they get plenty of photo opportunities.  If this were an entirely fair universe, the world would be rejecting them, and instead it applauds them.  However, they are the exceptions.)

Most of us are plebs in the nonprofit sector, and we are obliged to solve problems in order to justify our continued employment.  That’s the way it should be.  At the end of the day, if our organization’s mission is to save whales, then we should be judged on how our efforts add up to success in saving whales.  That might involve sticking to methods that have stood the test of time, or to finding ways to save more whales with fewer resources, or to finding ways to ensure that once we save them they stay healthy for the long run.   We have to keep our eye on results, and not give free passes to people who are merely impassioned without being effective.

By this I don’t mean that personalities and processes should be completely discounted.  It’s important to be considerate, ethical, and fair.  It’s also important to value the processes enough to learn from both failures and successes.  But having a pure heart is not enough, either as a process or an outcome, when your self-proclaimed mission is to save the whales.

• “What You Produce Does Not Have to Make Money, But It Does Have to Benefit People”

‘Nuff said, right?  This the nonprofit sector, right? It’s not just that people ask “what’s in it for me?” when they donate, it’s that we’re living in an WIIFM universe. If what’s in it for us, as nonprofit professionals, is the dubious glamor of altruism, I suppose that’s alright, but we still have an obligation to make our work more than a vanity project.  Again, if it were a fair world, then effective projects would get more love than vanity projects.  But even though it’s unfair, there’s still a principle involved that obligates us to be as effective as possible in benefiting the world, even when that’s less glamorous.

• “Everything Inside You Will Fight Improvement”

Thank you, Mr. Wong.  This is true indeed – not just for individuals, but for nonprofit organizations.  This one of the most compelling reasons why outcomes management is such an uphill battle in our sector.  It’s not just that we don’t like being judged by our results; it’s that we don’t like having to change.

I’m not using the first person plural (e.g., “we”) here in a abstract, vague, editorial way.  I’m using it, because I’m talking about flaws in which I fully partake.  When Wong says, “(t)he human mind is a miracle, and you will never see it spring more beautifully into action than when it is fighting against evidence that it needs to change,” he’s certainly speaking to my condition.

Here is his list of the powerful defense mechanisms of which I am (and perhaps you are) capable of fielding when we are challenged or criticized:

              “Intentionally Interpreting Any Criticism as an Insult”

              “Focusing on the Messenger to Avoid Hearing the Message”

              “Focusing on the Tone to Avoid Hearing the Content”

              “Revising (My) Own History”

             “Pretending That Any Self-Improvement Would Somehow Be Selling Out (My) True Self”

Of course, my personal favorite is the last mentioned. When I start wallowing in it, I do my best to remember that there are more important (and perhaps more valid) ethical principles at stake than expressing what I take to be my essential nature.

In fact, it’s our obligation to submit to public scrutiny and criticism when we work in the nonprofit sector.  (At least, it is in the U.S.; I can’t say much about other countries.)  As George McCully points out in his book Philanthropy Reconsidered, we are engaged in private initiatives for the public good, and the public has a right to evidence that we deserve the trust that is vested in us.  They deserve to know that we are striving to serve them with both processes and results that are valid, and that we are quickly learning from processes and results that do not yield strong positive benefits over the long run.

Outputs, outcomes – what’s the difference for nonprofits?

These days, I deal with a lot of people in the nonprofit/philanthropic sector for whom the topic of programmatic outcomes is fraught with anxiety.

It’s always unnerving when I discover that they equate outcomes with what they do (i.e., outputs), rather than with the change that occurs as a result of what they do (i.e., outcomes).  In other words, there a lot of nonprofit and philanthropic professionals who are not only anxious about programmatic outcomes, but who are confused about the exact nature of what they are tracking.

Fortunately, help is at hand.  A while back, while I was working on a nonprofit management information project with Third Sector New England, we shot a short video, in which Deborah Linnell (then with TSNE, now with the van Beuren Charitable Foundation) explained about outputs, outcomes, and logic models.

This video is not only brief but clear.  I recommend it to anyone who is working in a mission-based organization, and urge you to replay it on a regular basis, lest confusion set in again.  It’s worth the time, effort, and bandwidth to stay clear on this important distinction.

How grantmakers and how nonprofits use information about outcomes

I’m sitting here, reflecting on the Innovation Network’s “State of Evaluation 2012” report.

I encourage you to download it and read it for yourself; start with pages 14 and 15. These two pages display infographics that summarize what funders (also known as “grantors,” or if you’re Bob Penna, as “investors”) and nonprofits (also known as “grantees”) are reporting about why they do evaluation and what they are evaluating.

Regardless of whether you call it evaluation, impact assessment, outcomes management, performance measurement, or research – it’s really, really difficult to ascertain whether a mission-based organization is delivering the specific, positive, and sustainable change that it promises to its stakeholders. Many organizations do an excellent job at tracking outputs, but falter when it comes to managing outcomes. That’s in part because proving a causal relationship between what the nonprofit does and the specific goals that it promises to achieve is very costly in time, effort, expertise, and money.

But assuming that a mission-based organization is doing a rigorous evaluation, we still need to ask:  what is done with the findings, once the analysis is complete?

What the aforementioned infographics from the “State of Evalution 2012”  tell me is that both grantors and grantees typically say that the most important thing they can do with their outcome findings is to report them to their respective boards of directors.  Considering the depth of the moral and legal responsibility that is vested in board members, this is a pretty decent priority.  But it’s unclear to me what those boards actually do with the information.  Do they use it to guide the policies and operations of their respective organizations?  If so, does anything change for the better?

If you have an answer to the question of how boards use this information that is based on firsthand experience, then please feel to post a comment here.

What I learned about outcomes management from Robert Penna

Yesterday, along with a number of colleagues and friends from Community TechKnowledge, I had the privilege of attending a training by Robert Penna, the author of The Nonprofit Outcomes Toolbox.

As you probably  know, I’ve been on a tear about outcomes measurement for a few months now; the current level of obsession began when I attended NTEN’s Nonprofit Data Summit in Boston in September.  I thought that the presenters at the NTEN summit did a great job addressing some difficult issues – such as how to overcome internal resistance to collecting organizational data, and how to reframe Excel spreadsheets moldering away in file servers as archival data.  However, I worked myself into a tizzy, worrying about the lack, in that day’s presentations, of any reference to the history and literature of quantitative analysis and social research.  I could not see how nonprofit professionals would be able to find the time and resources to get up to speed on those topics.

Thanks to Bob Penna, I feel a lot better now.  In yesterday’s training, he showed me and the CTK team just how far you can go by stripping away what is superfluous and focusing on what it really takes to use the best outcomes tools for job.  Never mind about graduate level statistics! Managing outcomes may be very, very difficult because it requires major changes in organizational culture – let’s not kid ourselves about that.  However, it’s not going to take years out of each nonprofit professional’s life to develop the skill set.

Here are some other insights and highlights of the day:

• Mia Erichson, CTK’s brilliant new marketing manager, pointed out that at least one of the outcomes tools that Bob showed us could be easily mapped to a “marketing funnel” model.  This opens possibilities for aligning a nonprofits programmatic strategy with its marcomm strategy.
• The way to go is prospective outcomes tracking, with real time updates allowing for course correction.  Purely retrospective outcomes assessment is not going to cut it.
• There are several very strong outcomes tools, but they should be treated as we treated a software suite that comprises applications that are gems and applications that are junk.  We need to use the best of breed to meet each need.
• If we want to live in Bob Penna’s universe, we’re going to have to change our vocabulary.  It’s not “outcomes measurement – it’s “outcomes management.” The terms “funder” and “grantmaker” are out – “investor” is in.

Even with these lessons learned, it’s not a Utopia out there waiting for nonprofits that become adept at outcomes management.  Not only is it difficult to shift to an organizational culture that fosters it, but we have to face continuing questions about how exactly the funders (oops! I should have said “investors”) use the data that they demand from nonprofit organizations.  (“Data” is of course a broad term, with connotations well beyond outcomes management.  But it’s somewhat fashionable these days for them to take an interest in data about programmatic outcomes.)

We should be asking ourselves, first of all, whether the sole or primary motivation for outcomes management in nonprofits should be the demands of investors.  Secondly, we should be revisiting the Gilbert Center’s report, “Does Evidence Matter to Grantmakers? Data, Logic, and the Lack thereof in the Largest U.S. Foundations.” We need to know this. Thirdly, we should be going in search of other motivations for introducing outcomes management.  I realize that most nonprofits go forward with it when they reach a point of pain (translation:  they won’t get money if they don’t report outcomes). 

During a break in Bob’s training, some of my CTK colleagues were discussing the likelihood that many nonprofit executives simply hate the concept of outcomes management.  Who wants to spend resources on it, if it subtracts from resources available for programmatic activities?  Who wants to risk finding out (or to risk having external stakeholders find out) that an organization’s programs are approximately as effective as doing nothing at all?  Very few – thus the need to find new motivations, such as the power to review progress and make corrections as we go.  I jokingly told my CTK colleagues, “the truth will make you free, but first it will make you miserable.”  Perhaps that’s more than a joke.

What if we had a pro bono training on outcomes measurement for nonprofit professionals in Massachusetts?

As you can probably guess, I spend a lot of time these days worrying about outcomes measurement for nonprofits; I also devote time to discussing this topic with experts and with nonprofit professionals.

As I talk to some of the most impressive mavens in this field I  sometimes ask, “would you travel to Massachusetts at your own expense, to give a free day-long training on outcomes measurement to nonprofit professionals here?

Nothing ventured, nothing gained – am I right?  (Or as my dear sister once put it, I am The Mouth That Knows No Fear.)

A few really stellar experts actually agreed to do it, if a training event could be arranged to suit their schedules and other reasonable needs.  Of course, I am stunned, overwhelmed with gratitude.  Never underestimate the kindness of mavens!

So now I turn to my nonprofit colleagues in Massachusetts, with another unscientific survey.  I want to get a sense of who would be interested in day-long free training.  This survey is for them.  If you’re not a nonprofit professional based in Massachusetts, please do the honorable thing, and refrain from participating in this survey.

Outcomes measurement for nonprofits: Who does the analysis?

I invite you to participate in this survey, bearing in mind that it is for recreational purposes, and has no scientific value:

There are many reasons that this survey is of dubious value, for example:

• I’m using a convenience sample of respondents, so there’s no way to generalize about the findings.

• As the survey’s principal (and only) investigator, I’m hopelessly biased in favor of certain responses and against others; no effort has been made to create a neutral instrument.

• No pilot testing has been done to ensure that the choices offered are both exhaustive and mutually exclusive.

The list could go on, but I’ll leave it at that.  Although most of my training is in qualitative social research, I have taken undergraduate and graduate level courses on quantitative research, and the points I made about what’s wrong with my survey are what I could pull out of memory without consulting a standard text on statistics.

In other words, when it comes to quantitative analysis, I know just enough to be dangerous.

Meanwhile, I worry about nonprofit organizations that are under pressure to collect, analyze, and report data on the outcomes of their programs.  There are a lot of fantastic executive directors, program managers, and database administrators out there – but it’s very rare for a nonprofit professional who falls into any of those three categories to also have solid skills in quantitative analysis and social research methods.  Nevertheless, I know of plenty of nonprofit organizations where programmatic outcomes measurement is done by an executive director, program manager, or database administrator whose skill set is very different from what the task demands.  In many cases, even if they come up with a report, the nonprofit staff members may not even be aware that what have done is presented a lot of data, without actually showing that there is any causal relationship between the organization’s activities and the social good that they are in business to deliver.

Let’s not be too hasty in deprecating the efforts of these nonprofit professionals.  They are under a lot of pressure, especially from grantmaking foundations, to report on programmatic outcomes.  In many cases, they do the best they can to respond, even if they have neither the internal capacity to meet the task nor the money to hire a professional evaluator.

By the way, I was delighted to attend gathering this fall, in which I heard a highly-regarded philanthropic professional ask a room full of foundation officers, “are you requiring $50,000 worth of outcomes measurement for a $10,000 grant?” It’s not the only question we need to ask, but it’s an extremely cogent one!

I’d love to see nonprofit professionals, philanthropists, and experts in quantitative analysis work together to address this challenge.

We should also be learning lessons from the online tools that have already been developed to match skilled individuals with nonprofit professionals who need help and advice from experts.  Examples of such tools include the “Research Matchmaker,” and NPO Connect.

We can do better.  It’s going to take time, effort, money, creativity, and collaboration – but we can do better.