Program evaluation
In the public, private, and voluntary sector, stakeholders might be required to assess—under law or charter—or want to know whether the programs they are funding, implementing, voting for, receiving or opposing are producing the promised effect.
Needs assessment involves research and regular consultation with community stakeholders and with the people that will benefit from the project before the program can be developed and implemented.
Rossi, Lipsey & Freeman (2004) gave an example that: a person identifying some battered children may be enough evidence to persuade one that child abuse exists.
Creating an actual logic model is particularly important because it helps clarify for all stakeholders: the definition of the problem, the overarching goals, and the capacity and outputs of the program.
This approach, known as integrative propositional analysis (IPA), is based on research streams finding that theories were more likely to work as expected when they had better structure (in addition meaning and data).
Then, the researcher examines the number of concepts and causal relationships between them (circles and arrows on the diagram) to measure the breadth and depth of understanding reflected in the theory's structure.
[18] According to Mouton (2009) measuring the impact of a program means demonstrating or estimating the accumulated differentiated proximate and emergent effect, some of which might be unintended and therefore unforeseen.
[29] This approach has responded to the continued greater need for evaluation processes that are more rapid and economical under difficult circumstances of budget, time constraints and limited availability of data.
[36][37] Culture is defined by Ebbutt (1998, p. 416) as a "constellation of both written and unwritten expectations, values, norms, rules, laws, artifacts, rituals and behaviors that permeate a society and influence how people behave socially".
The first, and probably most common, is the positivist approach, in which evaluation can only occur where there are "objective", observable and measurable aspects of a program, requiring predominantly quantitative evidence.
[40] A detailed example of the positivist approach is a study conducted by the Public Policy Institute of California report titled "Evaluating Academic Programs in California's Community Colleges", in which the evaluators examine measurable activities (i.e. enrollment data) and conduct quantitive assessments like factor analysis.
[41] The second paradigm identified by Potter (2006) is that of interpretive approaches, where it is argued that it is essential that the evaluator develops an understanding of the perspective, experiences and expectations of all stakeholders.
The evaluator's contact with the program is often over an extended period of time and, although there is no standardized method, observation, interviews and focus groups are commonly used.
[42] Potter (2006) also identifies critical-emancipatory approaches to program evaluation, which are largely based on action research for the purposes of social transformation.
Because of its critical focus on societal power structures and its emphasis on participation and empowerment, Potter argues this type of evaluation can be particularly useful in developing countries.
The role of the evaluator during this task is to facilitate interactive discussion amongst members in an attempt to establish some baseline of shared meaning and understanding pertaining to the key activities.
Donna Mertens, primary researcher in this field, states that the transformative paradigm, "focuses primarily on viewpoints of marginalized groups and interrogating systemic power structures through mixed methods to further social justice and human rights".
[48] Many health and evaluation leaders are careful to point out that cultural competence cannot be determined by a simple checklist, but rather it is an attribute that develops over time.
Important skills include: ability to build rapport across difference, gain the trust of the community members, and self-reflect and recognize one's own biases.
The transformative paradigm's axiological assumption rests on four primary principles:[47] Differences in perspectives on what is real are determined by diverse values and life experiences.
In turn these values and life experiences are often associated with differences in access to privilege, based on such characteristics as disability, gender, sexual identity, religion, race/ethnicity, national origins, political party, income level, age, language, and immigration or refugee status.
[50] Methodological decisions are aimed at determining the approach that will best facilitate use of the process and findings to enhance social justice; identify the systemic forces that support the status quo and those that will allow change to happen; and acknowledge the need for a critical and reflexive relationship between the evaluator and the stakeholders.
Daniel Solorzano describes the role of CRT as providing a framework to investigate and make visible those systemic aspects of society that allow the discriminatory and oppressive status quo of racism to continue.
Because of the sensitivity of issues surrounding LGBTQ status, evaluators need to be aware of safe ways to protect such individuals’ identities and ensure that discriminatory practices are brought to light in order to bring about a more just society.
[47] Given the Federal budget deficit, the Obama Administration moved to apply an "evidence-based approach" to government spending, including rigorous methods of program evaluation.
An inter-agency group delivers the goal of increasing transparency and accountability by creating effective evaluation networks and drawing on best practices.
The framework is as follows: In January 2019, the Foundations for Evidence-Based Policymaking Act introduced new requirements for federal agencies, such as naming a Chief Evaluation Officer.
These four aspects of CIPP evaluation assist a decision-maker to answer four basic questions: This involves collecting and analysing needs assessment data to determine goals, priorities and objectives.
[55] This involves the steps and resources needed to meet the new goals and objectives and might include identifying successful external programs and materials as well as gathering information.
By continuously monitoring the program, decision-makers learn such things as how well it is following the plans and guidelines, conflicts arising, staff support and morale, strengths and weaknesses of materials, delivery and budgeting problems.
