Impact evaluation assesses the changes that can be attributed to a particular intervention, such as a project, program or policy, both the intended ones, as well as ideally the unintended ones.
[3] Impact evaluation helps people answer key questions for evidence-based policy making: what works, what doesn't, where, why and for how much?
There are a range of accepted approaches to determining an appropriate comparison group for counterfactual analysis, using either prospective (ex ante) or retrospective (ex post) evaluation design.
Randomization and isolation from interventions might not be practicable in the realm of social policy and may be ethically difficult to defend,[8][9] although there may be opportunities to use natural experiments.
[14] A well conducted RCT will yield a credible estimate regarding the average treatment effect within one specific population or unit of assignment.
[15] Natural experiments are used because these methods relax the inherent tension uncontrolled field and controlled laboratory data collection approaches.
Quasi-experimental methods include matching, differencing, instrumental variables and the pipeline approach; they are usually carried out by multivariate regression analysis.
The assumption is that as they have been selected to receive the intervention in the future they are similar to the treatment group, and therefore comparable in terms of outcome variables of interest.
When there is an absence of the assumption of equivalence, the difference in outcome between the groups that would have occurred regardless creates a form of bias in the estimate of program effects.
[17] Selection bias can occur through natural or deliberate processes that cause a loss of outcome data for members of the intervention and control groups that have already been formed.
For example, when a community's birth rate is declining, a program to reduce fertility may appear effective because of bias stemming from that downward trend (Rossi et al., 2004, p273).
Impact evaluation needs to accommodate the fact that natural maturational and developmental processes can produce considerable change independently of the program.
According to an interview with Jim Rough, former representative of the American Evaluation Association, in the magazine D+C Development and Cooperation, this method does not work for complex, multilayer matters.
ITT therefore provides a lower-bound estimate of impact, but is arguably of greater policy relevance than TOT in the analysis of voluntary programs.
[17] Common definitions of 'impact' used in evaluation generally refer to the totality of longer-term consequences associated with an intervention on quality-of-life outcomes.
For example, UNICEF defines impact as "The longer term results of a program – technical, economic, socio-cultural, institutional, environmental or other – whether intended or unintended.
While hundreds of thousands of documents call for them, rarely do donors have the funding flexibility - or interest - to return to see how sustained, and durable our interventions remained after project close out, after resources were withdrawn.
In addition, interventions are often active rather than passive, requiring a greater rather than lesser degree of participation among beneficiaries and therefore behavior change as a pre-requisite for effectiveness.
A theory-based approach enables policy-makers to understand the reasons for differing levels of program participation (referred to as 'compliance' or 'adherence') and the processes determining behavior change.
Theory-Based approaches use both quantitative and qualitative data collection, and the latter can be particularly useful in understanding the reasons for compliance and therefore whether and how the intervention may be replicated in other settings.
Methods of qualitative data collection include focus groups, in-depth interviews, participatory rural appraisal (PRA) and field visits, as well as reading of anthropological and political literature.
[36][37] CCT programs have since been implemented by a number of governments in Latin America and elsewhere, and a report released by the World Bank in February 2009 examines the impact of CCTs across twenty countries.
3ie has launched an online database of impact evaluations covering studies conducted in low- and middle income countries.
Other organisations publishing Impact Evaluations include Innovations for Poverty Action, the World Bank's DIME Initiative and NONIE.
The IEG of the World Bank has systematically assessed and summarized the experience of ten impact evaluation of development programs in various sectors carried out over the past 20 years.
3ie seeks to improve the lives of poor people in low- and middle-income countries by providing, and summarizing, evidence of what works, when, why and for how much.
COSA is developing and applying an independent measurement tool to analyze the distinct social, environmental and economic impacts of agricultural practices, and in particular those associated with the implementation of specific sustainability programs (Organic, Fairtrade etc.).
The focus of the initiative is to establish global indicators and measurement tools which farmers, policy-makers, and industry can use to understand and improve their sustainability with different crops or agricultural sectors.
COSA aims to facilitate this by enabling them to accurately calculate the relative costs and benefits of becoming involved in any given sustainability initiative.
Systematic reviews aim to bridge the research-policy divide by assessing the range of existing evidence on a particular topic, and presenting the information in an accessible format.