Authors:Tineke A. Abma, Merel Visse, Anders Hanberger, Helen Simons, Jennifer C. Greene Pages: 131 - 146 Abstract: Evaluation, Volume 26, Issue 2, Page 131-146, April 2020. Recently, several authors have called for a critical assessment of the normative dimensions of evaluation practice. This article responds to this call by demonstrating how evaluation practice can be enriched through deliberate engagement with care ethics. Care ethics has a relational and practice view of morality and places caring relationships and responsibilities at the forefront of our being in the world. We will demonstrate how care ethics, in particular Joan Tronto’s moral-political theory of democratic caring, can help evaluators to reshape our way of working by placing caring and relationality at the centre of our evaluative work. Care ethics as a normative orientation for evaluation stretches beyond professional codes of conduct, and rule- or principled-based behaviour. It is part of everything we do or not do, how we interact with others, and what kinds of relationships we forge in our practice. This is illustrated with two examples: a democratic evaluation of a programme for refugee children in Sweden; and a responsive evaluation of a programme for neighbours of people with an intellectual disability in The Netherlands. Both examples show that a caring ethos offers a promising pathway to address the larger political, public issues of our times through the interrogation of un-caring practices. We conclude a caring ethos can help evaluators to strengthen a caring society that builds on people’s deeply felt need to care, to relate, and to connect within and across communities. Citation: Evaluation PubDate: 2020-04-23T07:35:38Z DOI: 10.1177/1356389019893402 Issue No:Vol. 26, No. 2 (2020)
Authors:Rosaria Lumino, Dora Gambardella Pages: 147 - 165 Abstract: Evaluation, Volume 26, Issue 2, Page 147-165, April 2020. Although the relation between accountability and learning is a key issue in the field of evaluation, the existing conceptualizations include significant gaps and shortcoming. This article seeks to bridge some of the identified gaps, overcoming the dichotomy between learning and accountability and the overemphasis on accountability, and offering a more nuanced account of various forms of positive and negative learning. We propose a theory-informed reconceptualization of both accountability and learning in order to develop an interpretative framework that recognizes their plurality and brings them together through evaluation. To make visible the analytical potential of our reconceptualization, we present the Italian higher education evaluation system as a source of illustrative examples.Our analysis confirms that accountability and learning are inextricably interconnected and that multiple forms of learning can be envisaged by taking into account the mutual relationship among organizational arrangements, evaluative practices and cultural codes. We argue that such kind of analysis allows to remove the taken-for-granted aspects of evaluation practices, challenging the often-implicit assumptions about their virtues as well as their weaknesses in order to see what evaluation actually does in a particular situation. Citation: Evaluation PubDate: 2020-04-23T07:35:40Z DOI: 10.1177/1356389019901304 Issue No:Vol. 26, No. 2 (2020)
Authors:Veronica Gaffey Pages: 166 - 176 Abstract: Evaluation, Volume 26, Issue 2, Page 166-176, April 2020. The European Commission has strengthened its requirements for impact assessment and evaluation over the years. In 2002, it introduced a requirement for impact assessments for new policy proposals and regulations. Such impact assessments had to define the need for European Union action and analyse a variety of options for action. In 2006, it established an Impact Assessment Board, made up of senior managers from across Commission directorates-general. The Board members worked part-time on the Board and did not sit in judgement on proposals from their own directorate-general. Citation: Evaluation PubDate: 2020-04-23T07:35:38Z DOI: 10.1177/1356389019897703 Issue No:Vol. 26, No. 2 (2020)
Authors:Astrid Brousselle, Jim McDavid Pages: 190 - 204 Abstract: Evaluation, Volume 26, Issue 2, Page 190-204, April 2020. In the last century, human-led activities have drastically altered natural systems. The environmental impacts of human activity are so deleterious to living species and our biosphere that geologists have named this new geological era the Anthropocene, from anthropos, human being. Responses to the Anthropocene era call for drastic changes in all domains of activity. As evaluators, we claim to work for social betterment. We have a responsibility to adapt our approaches and practices to respond to this environmental challenge. The aim of this article is to raise awareness on the need to develop new approaches for evaluators in the Anthropocene. We first describe what this state of urgency represents for humans, the international commitments to take action, the solutions that exist, and what responding to this environmental challenge means for our profession. Citation: Evaluation PubDate: 2020-04-23T07:35:35Z DOI: 10.1177/1356389020911060 Issue No:Vol. 26, No. 2 (2020)
Authors:Barbara Schmidt-Abbey, Martin Reynolds, Ray Ison Pages: 205 - 226 Abstract: Evaluation, Volume 26, Issue 2, Page 205-226, April 2020. The need for, and possibilities of, a second-order shift in evaluation practice are explored. Second-order evaluation practice enables an evaluator to improve practice as a skilled practitioner, acknowledging her embeddedness within an evaluand. The article explores evaluation practice as experienced by professional evaluators, using ideas from developmental evaluation coupled with systemic evaluation in the tradition of systems thinking in practice. Systemic evaluation aims to capture systemic sensibilities – the bigger picture – of complex turbulent situations of change underpinning evaluands. Attributes of second-order practice with systemic evaluation are understood as being aligned with both systemic and systematic modes of evaluation praxis. Personal experiences are provided where this juxtaposing praxis has been found wanting. By example, a systems thinking in practice framework is explored as heuristic support for making a second-order practice shift. The article concludes with a discussion of some implications for developments in professionalising evaluation practice and research. Citation: Evaluation PubDate: 2020-04-23T07:35:43Z DOI: 10.1177/1356389020909120 Issue No:Vol. 26, No. 2 (2020)
Authors:Kerstin Junge, Joe Cullen, Giorgia Iacopini Pages: 227 - 245 Abstract: Evaluation, Volume 26, Issue 2, Page 227-245, April 2020. This article explores the usefulness of Contribution Analysis to understand and assess the outcomes and impacts of emergent, large-scale and transformational change processes. As these take place over long periods of time and at national if not supra-national scale, they amplify some of the key challenges complex interventions pose to evaluators, most notably that of the emergence of activities and outcomes. While Contribution Analysis is discussed in the literature as useful for the evaluation of complex change processes, it is not usually recommended for experimental and emergent interventions. This article draws on the authors’ experience of using Contribution Analysis in one such case and explores the relative merits of the method for large-scale/transformational change interventions generally. Citation: Evaluation PubDate: 2020-04-23T07:35:36Z DOI: 10.1177/1356389020912270 Issue No:Vol. 26, No. 2 (2020)
Authors:Ghislain Arbour Abstract: Evaluation, Ahead of Print. The term “evaluation framework” and other associated terms are common in the practice and the discipline of program evaluation. These terms correspond to a variety of meanings across contexts and organizations and, thus, tend to lack overall consistency. In response, this article provides a model to analyze frameworks for program evaluation organized around four dimensions. The model states that a framework for evaluation is an intellectual framework made of concepts and/or theories (first dimension: types of ideas) about an object related to evaluation (second dimension: object), where the said concepts and theories can be positive and/or normative (third dimension: analytical perspective). These three dimensions provide the means to describe, explain, or judge an evaluation-related matter. A fourth and optional dimension, the institutional character of a framework, allows an evaluation framework to become a form of regulation for behaviors related to program evaluation (fourth dimension: institutional dimension). Citation: Evaluation PubDate: 2020-06-11T06:00:57Z DOI: 10.1177/1356389020920890
Authors:Sophia Rodriguez, Jeremy Acree Abstract: Evaluation, Ahead of Print. In this article, the authors theorize the practice of evaluation as linked to truth-telling and organizing future societies. Drawing on Foucauldian notions of biopolitical governmentality, the authors examine the origins of the field of evaluation, theorize it as a truth-telling practice that aims to control populations and futures, and consider the implications of this for a current evaluation project with transnational newcomer migrant youth in the United States. The authors raise the following questions about evaluation as a social practice: Who/what knowledge is produced in the process' What mechanisms/technologies are deployed to reason, compare, and quantify migrant youth experiences, and at what cost' What are the ethical imperatives underlying this truth-telling process' The article offers a productive critique of current evaluation practices, providing theoretical and methodological implications of this analysis, arguing to expose the politics of governance embedded in evaluation. Citation: Evaluation PubDate: 2020-05-07T05:19:44Z DOI: 10.1177/1356389020914314
Authors:Jone Martínez-Palacios, Igor Ahedo Abstract: Evaluation, Ahead of Print. This article explains why and how it is possible to give greater democratic meaning to an evaluative process by bringing into dialogue the Gender + perspective and a critical approach to deliberative theory within its design. Furthermore, it identifies the methodological challenges of using this intersectional approach to create more inclusive evaluative procedures. Taking into account the experience of the evaluation of Law 4/2005 for Equality between Women and Men in the Basque Country (Spain, 2015–2016), the text explains that the enclave deliberation praxis proposed by critical deliberative theory helps to resolve the challenges that emerge from the implementation of the Gender + perspective in the evaluation of public policies. These challenges include the incorporation of the empowerment perspective into the evaluation, and guaranteeing dialogue between expert knowledge on gender and other more ‘intuitive’ types of knowledge. Citation: Evaluation PubDate: 2020-04-17T10:11:22Z DOI: 10.1177/1356389020912779
Authors:Eran Raveh, Yuval Ofek, Ron Bekkerman, Hertzel Cohen Abstract: Evaluation, Ahead of Print. Evaluators worldwide are dealing with a growing amount of unstructured electronic data, predominantly in textual format. Currently, evaluators analyze textual Big Data primarily using traditional content analysis methods based on keyword search, a practice that is limited to iterating over predefined concepts. But what if evaluators cannot define the necessary keywords for their analysis' Often we should examine trends in the way certain organizations have been operating, while our raw data are gigabytes of documents generated by that organization over decades. The problem is that in many cases we do not know what exactly we need to look for. In such cases, traditional analytical machinery would not provide an adequate solution within reasonable time—instead, heavy-lifting Big Data Science should be applied. We propose an automated, quantitative, user-friendly methodology based on text mining, machine learning, and data visualization, which assists researchers and evaluation practitioners to reveal trends, trajectories, and interrelations between bits and pieces of textual information in order to support evaluation. Our system automatically extracts a large amount of descriptive terminology for a particular domain in a given language, finds semantic connections between documents based on the extracted terminology, visualizes the entire document repository as a graph of semantic connections, and leads the user to the areas on that graph where most interesting trends can be observed. This article exemplifies the new method on 1700 performance reports, showing that the method can be used successfully, supplying evaluators with highly important information which cannot be revealed using other methods. Such exploratory exercise is vital as a preliminary exploratory phase for evaluations involving unstructured Big Data, after which a range of evaluation methods can be applied. We argue that our system can be successfully implemented on any domain evaluated. Citation: Evaluation PubDate: 2020-03-04T07:22:38Z DOI: 10.1177/1356389020905322
Authors:Barbara Befani Abstract: Evaluation, Ahead of Print. This article discusses the integration of a diagnostic lens in qualitative or mixed methods evaluations, arguing that this will improve quality: in particular, it will improve transparency, credibility and reliability of evaluation findings.We start by unpacking the notion of evaluation quality and pointing out the typical weaknesses of qualitative methods. We then introduce the basic notions of diagnostic approaches and how they relate to theory-based evaluation, process tracing, and Bayesian updating, arguing for the merits of taking a formal Bayesian approach, founded on the confusion matrix, which amongst else reduces confirmation bias and conservatism. This article draws parallels between the process tracing tests and elements of the confusion matrix. Citation: Evaluation PubDate: 2020-02-26T06:26:34Z DOI: 10.1177/1356389019898223
Authors:Moses C. Simuyemba, Obrian Ndlovu, Felicitas Moyo, Eddie Kashinka, Abson Chompola, Aaron Sinyangwe, Felix Masiye Abstract: Evaluation, Ahead of Print. The Full Country Evaluations were Gavi-funded real-time evaluations of immunisation programmes in Bangladesh, Mozambique, Uganda and Zambia, from 2013 to 2016. The evaluations focused on providing evidence for improvement of immunisation delivery in these countries and spanned all phases of Gavi support. The process evaluation approach of the evaluations utilised mixed methods to track progress against defined theories-of-change and related milestones during the various stages of implementation of the Gavi support streams. This article highlights complexities of this type of real-time evaluation and shares lessons learnt on conducting such evaluation from the Zambian experience. Real-time process evaluation is a complex evaluation methodology that requires sensitivity to the context of the evaluation, catering for various information needs of stakeholders, and establishment of mutually beneficial relationships between programme implementers and evaluators. When used appropriately, it can be an effective means of informing programme decisions and aiding programme improvement for both donors and local implementers. Citation: Evaluation PubDate: 2020-02-03T11:49:25Z DOI: 10.1177/1356389019901314
Authors:Guillaume Fontaine Abstract: Evaluation, Ahead of Print. The literature on policy design provides insightful resources to improve policy evaluation. We compare how this literature addresses causation, evaluation, instrumentation, and intervention, based on different methodologies. Then, we explain how the realist approach can contribute to theory-based evaluation with multi-methods. We go through the process leading from the adoption of policy aims to the production of a policy outcome by a causal mechanism linking agenda setting to policy formulation, cross-sectorial coordination, political interplays, and this policy’s outcome. This hypothesis is then tested against a typology of policy instruments classified by the State resources. These instruments are hereby treated as expected empirical observations that provide an empirical basis to assess the theoretical process following a Bayesian logic. We eventually present a five-step protocol aimed at making realist policy evaluation easier, which includes process theorization and operationalization, empirical tests design, case selection and justification, congruence analysis, and deep within-case study. Citation: Evaluation PubDate: 2020-02-03T11:46:58Z DOI: 10.1177/1356389020902496
Hybrid journal (It can contain Open Access articles)
[1089 journals]
