Evaluation Case Studies
Introduction
Evaluation is no easy task. Grantees often resist it. Funders are not always sure what to ask for.
To help EITC Funders Network members learn about the range of evaluations the field is undertaking, the EITC Funders Network has prepared five case studies—examining efforts in Harford, Michigan, Minnesota, San Antonio and Seattle.
What do we mean by “Evaluation”? Evaluation is a broad term that means a lot of things to different people. For this effort, the EITC Funders Network has taken a somewhat broad look at evaluation. Here we include:
information that funders and program providers collect to help them understand the impact of the investment, the effectiveness of the program or strategy, and/or the impact on target communities.
What information do the case studies contain? These case studies focus on several key questions:
Evaluating EITC-Related Programs: Five Case Studies |
- Why undertake evaluation?
- What was the evaluation methodology?
- What were the success factors?
- What were the challenges?
- What has changed as a result of the evaluation work?
While the cases study examples provide some information on the outcomes of the evaluation, that is not the primary purpose. Rather, the narrative focuses on the process of evaluation. We hope that this will enable the reader to look beyond the specifics of the program and its evaluation and identify core project components and learnings that can be applied to their own work.
How were the case study sites selected? The EITC Funders Network asked you—members of the Network—as well as field partners to recommend non-profit organizations that had in the past or are currently undertaking evaluation. From the recommended sites, EITC Funders Network staff selected the following five case study examples based on geographic representation and a mix of kinds of evaluation approaches. Click on the name to read the case study:
- Hartford – an organization hosting a centralized EITC outreach and free-tax preparation effort evaluates service delivery annually and over time.
- Michigan – a statewide EITC outreach and free-tax preparation coalition evaluates the dispersed efforts of coalition members.
- Minnesota – an organization undertaking EITC outreach and free-tax preparation gathers information about program operations and effectiveness and has plans for impact evaluation.
- San Antonio – a public/private partnership examines the impact of EITC benefits on the economy of San Antonio.
- Seattle – an organization undertaking EITC outreach and free-tax preparation gathers information regarding program participation and ability to reach target audiences.
Do you have advice for funders? Yes! Each case study contains some advice for funders interested in supporting evaluation based on the experience of the case study site. In addition, the EITC Funders Network provides some summary thoughts in our conclusion section.
Where do I find evaluation tools? The EITC Funders Network “Evaluation Tools” webpage contains sample evaluation tools used by EITC Funders Network members and their grantee partners.
Conclusion
In late 2010, the EITC Funders Network undertook an effort to raise awareness of evaluation strategies being implemented by charitable foundations and field representatives interested in understanding the impact and importance of their EITC-related work. For this effort, we collected a sampling of tools currently being used by funders and the field to evaluate their work and prepared five case studies that took a closer look at evaluation being undertaken across the nation.
The EITC Funders Network envisioned this effort to begin the conversation about evaluation—not to end it. This project was neither meant to provide a compendium of evaluation strategies nor provide a road map to the ideal evaluation.
However, even at this early stage of the conversation, there are a number of findings that emerged during the work that are worth sharing:
- Evaluation Can (and Should!) be Undertaken—because of the nature of most EITC-related programming—it centers around a concrete interaction that happens in a time-specific way—basic evaluation is fairly easy to do and does not necessarily require a lot of extra work by you or your grantees. Even slightly more advanced forms of evaluation—tracking impact and people over time—can be implemented straightforwardly if grantee partners (and/or their evaluators) set up mechanisms in advance of the tax filing season and have the resources in place for multi-year work.
- Don’t Recreate the Wheel—there are quite a number of tools, examples, and other resources for you if you are interested in evaluating your EITC-related work. Some of them are gathered on our website (www.eitcfunders.org). These can be a starting point in your quest to learn about the impact of your EITC-related programming. Further, before asking your grantee partner to start a new evaluation process, determine if they are already taking on evaluatory activities that you can build on.
- Build it into Grants—because evaluation can be so valuable to programs, the field and funders, it is important that evaluation be built into program grants. While some funders and program providers may feel that funding evaluation takes scarce dollars away from vital direct services, it is critical to ensure quality of services and for making the case for valued programs over the longer term.
- Funder Initiated, but Grantee Informed—while many program partners may be interested to evaluate their programs, most take it on at the request of a funder. Field representatives report that the best structure for evaluations are ones that are informed, supported and encouraged by the funder, but are ultimately designed and implemented in ways that are directly useful to the program provider.
- Compare to the Field—there are literally thousands of EITC outreach and free-tax preparation efforts being undertaken nationwide. Many of these programs are collecting similar, basic participant data as part of their evaluation and are sharing them via the EITC Platform portal (www.eitcplatform.org). Encourage your grantees to participate in this field-wide data collection effort and gauge their progress against their colleagues across the nation.
- Look at Short- and Long-Term Impact—it has been very easy for most of the evaluation in the EITC field to look at short-term outcomes—how many low-income families got their taxes completed for free? How much funding came back into the community during this tax season through EITC refunds? How did a specific outreach strategy impact program participation? However, taking on the larger and longer term questions have been harder. While funders are interested in the longer-term impact of the EITC and its ability to transform lines, so, too, are grantee partners! Funders should engage in a conversation with their grantees about how best to collect both the short-term impact as well as some of the longer-term impact information.
Click here to return to the Evaluation Home Page
Acknowledgements The EITC Funders Network would like to thank the members of the Network who provided information on their evaluation strategies and tools. We would like to thank Steve Holt, of Holt Solutions and Jacky Lynn Coleman of the National Community Tax Coalition who provided insights on evaluation in the EITC and free-tax preparation field. And, we would like to thank the staff of organizations profiled in the case studies, including Donna Taglianetti of Co-Opportunity, Bob Gillett and Christine Robison of Michigan Poverty Law Program, Tracy Fischman of AccountAbility Minnesota, Richard Keith and John Scarfo of Department of Community Initiatives, San Antonio, Anna Weaver of United Way of San Antonio, and Courtney Noble of United Way of King County. |