Authors:Pete Barbrook-Johnson, Amy Proctor, Sara Giorgi, Jeremy Phillipson Pages: 315 - 332 Abstract: Evaluation, Volume 26, Issue 3, Page 315-332, July 2020. There is a well-documented interest in how insights from the study of complexity can be applied to policy evaluation. However, important questions remain as to how complexity is understood and used by policy evaluators. We present findings from semi-structured interviews with 30 UK policy evaluators working in food, energy, water and environment policy domains. We explore how they understand, use and approach complexity, and consider the implications for evaluation research and practice. Findings reveal understandings of complexity arising from contextual factors, scale-related issues and perceptions of unpredictability. The evidence indicates terminological and analogical use of complexity and its concepts by policy evaluators, but limited evidence of its literal use. Priorities for the future include framing complexity more pragmatically and as an opportunity not a cost. Communicating this up the policy hierarchy is the key to progressing complexity-appropriate evaluation – this can be enabled by strengthening links between policy evaluation and academic communities. Citation: Evaluation PubDate: 2020-07-18T10:33:06Z DOI: 10.1177/1356389020930053 Issue No:Vol. 26, No. 3 (2020)
Authors:Kim Dunphy, John Smithies, Surajen Uppal, Holly Schauble, Amy Stevenson Abstract: Evaluation, Ahead of Print. The cultural sector (arts, heritage and library agencies) receives increasing demands to articulate and evaluate outcomes of its work. These demands are challenging because of pervasive conceptions that the intangible nature of cultural activities makes them inherently unmeasurable, while their ‘intrinsic’ properties render them essentially valuable. This article addresses this challenge in positing a schema of measurable cultural outcomes of cultural engagement developed through literature analysis and an iterative Delphi-style stakeholder consultation. The outcomes are as follows: creativity stimulated; aesthetic enrichment experienced; knowledge, ideas or insight gained; diversity of cultural expression appreciated; and sense of belonging to cultural heritage deepened. Citation: Evaluation PubDate: 2020-09-28T10:59:50Z DOI: 10.1177/1356389020952460
Authors:Ghislain Arbour Abstract: Evaluation, Ahead of Print. The term “evaluation framework” and other associated terms are common in the practice and the discipline of program evaluation. These terms correspond to a variety of meanings across contexts and organizations and, thus, tend to lack overall consistency. In response, this article provides a model to analyze frameworks for program evaluation organized around four dimensions. The model states that a framework for evaluation is an intellectual framework made of concepts and/or theories (first dimension: types of ideas) about an object related to evaluation (second dimension: object), where the said concepts and theories can be positive and/or normative (third dimension: analytical perspective). These three dimensions provide the means to describe, explain, or judge an evaluation-related matter. A fourth and optional dimension, the institutional character of a framework, allows an evaluation framework to become a form of regulation for behaviors related to program evaluation (fourth dimension: institutional dimension). Citation: Evaluation PubDate: 2020-06-11T06:00:57Z DOI: 10.1177/1356389020920890
Authors:Sophia Rodriguez, Jeremy Acree Abstract: Evaluation, Ahead of Print. In this article, the authors theorize the practice of evaluation as linked to truth-telling and organizing future societies. Drawing on Foucauldian notions of biopolitical governmentality, the authors examine the origins of the field of evaluation, theorize it as a truth-telling practice that aims to control populations and futures, and consider the implications of this for a current evaluation project with transnational newcomer migrant youth in the United States. The authors raise the following questions about evaluation as a social practice: Who/what knowledge is produced in the process' What mechanisms/technologies are deployed to reason, compare, and quantify migrant youth experiences, and at what cost' What are the ethical imperatives underlying this truth-telling process' The article offers a productive critique of current evaluation practices, providing theoretical and methodological implications of this analysis, arguing to expose the politics of governance embedded in evaluation. Citation: Evaluation PubDate: 2020-05-07T05:19:44Z DOI: 10.1177/1356389020914314
Authors:Jone Martínez-Palacios, Igor Ahedo Abstract: Evaluation, Ahead of Print. This article explains why and how it is possible to give greater democratic meaning to an evaluative process by bringing into dialogue the Gender + perspective and a critical approach to deliberative theory within its design. Furthermore, it identifies the methodological challenges of using this intersectional approach to create more inclusive evaluative procedures. Taking into account the experience of the evaluation of Law 4/2005 for Equality between Women and Men in the Basque Country (Spain, 2015–2016), the text explains that the enclave deliberation praxis proposed by critical deliberative theory helps to resolve the challenges that emerge from the implementation of the Gender + perspective in the evaluation of public policies. These challenges include the incorporation of the empowerment perspective into the evaluation, and guaranteeing dialogue between expert knowledge on gender and other more ‘intuitive’ types of knowledge. Citation: Evaluation PubDate: 2020-04-17T10:11:22Z DOI: 10.1177/1356389020912779
Authors:Eran Raveh, Yuval Ofek, Ron Bekkerman, Hertzel Cohen Abstract: Evaluation, Ahead of Print. Evaluators worldwide are dealing with a growing amount of unstructured electronic data, predominantly in textual format. Currently, evaluators analyze textual Big Data primarily using traditional content analysis methods based on keyword search, a practice that is limited to iterating over predefined concepts. But what if evaluators cannot define the necessary keywords for their analysis' Often we should examine trends in the way certain organizations have been operating, while our raw data are gigabytes of documents generated by that organization over decades. The problem is that in many cases we do not know what exactly we need to look for. In such cases, traditional analytical machinery would not provide an adequate solution within reasonable time—instead, heavy-lifting Big Data Science should be applied. We propose an automated, quantitative, user-friendly methodology based on text mining, machine learning, and data visualization, which assists researchers and evaluation practitioners to reveal trends, trajectories, and interrelations between bits and pieces of textual information in order to support evaluation. Our system automatically extracts a large amount of descriptive terminology for a particular domain in a given language, finds semantic connections between documents based on the extracted terminology, visualizes the entire document repository as a graph of semantic connections, and leads the user to the areas on that graph where most interesting trends can be observed. This article exemplifies the new method on 1700 performance reports, showing that the method can be used successfully, supplying evaluators with highly important information which cannot be revealed using other methods. Such exploratory exercise is vital as a preliminary exploratory phase for evaluations involving unstructured Big Data, after which a range of evaluation methods can be applied. We argue that our system can be successfully implemented on any domain evaluated. Citation: Evaluation PubDate: 2020-03-04T07:22:38Z DOI: 10.1177/1356389020905322
Authors:Guillaume Fontaine First page: 296 Abstract: Evaluation, Ahead of Print. The literature on policy design provides insightful resources to improve policy evaluation. We compare how this literature addresses causation, evaluation, instrumentation, and intervention, based on different methodologies. Then, we explain how the realist approach can contribute to theory-based evaluation with multi-methods. We go through the process leading from the adoption of policy aims to the production of a policy outcome by a causal mechanism linking agenda setting to policy formulation, cross-sectorial coordination, political interplays, and this policy’s outcome. This hypothesis is then tested against a typology of policy instruments classified by the State resources. These instruments are hereby treated as expected empirical observations that provide an empirical basis to assess the theoretical process following a Bayesian logic. We eventually present a five-step protocol aimed at making realist policy evaluation easier, which includes process theorization and operationalization, empirical tests design, case selection and justification, congruence analysis, and deep within-case study. Citation: Evaluation PubDate: 2020-02-03T11:46:58Z DOI: 10.1177/1356389020902496
Authors:Barbara Befani First page: 333 Abstract: Evaluation, Ahead of Print. This article discusses the integration of a diagnostic lens in qualitative or mixed methods evaluations, arguing that this will improve quality: in particular, it will improve transparency, credibility and reliability of evaluation findings.We start by unpacking the notion of evaluation quality and pointing out the typical weaknesses of qualitative methods. We then introduce the basic notions of diagnostic approaches and how they relate to theory-based evaluation, process tracing, and Bayesian updating, arguing for the merits of taking a formal Bayesian approach, founded on the confusion matrix, which amongst else reduces confirmation bias and conservatism. This article draws parallels between the process tracing tests and elements of the confusion matrix. Citation: Evaluation PubDate: 2020-02-26T06:26:34Z DOI: 10.1177/1356389019898223
Authors:Moses C. Simuyemba, Obrian Ndlovu, Felicitas Moyo, Eddie Kashinka, Abson Chompola, Aaron Sinyangwe, Felix Masiye First page: 367 Abstract: Evaluation, Ahead of Print. The Full Country Evaluations were Gavi-funded real-time evaluations of immunisation programmes in Bangladesh, Mozambique, Uganda and Zambia, from 2013 to 2016. The evaluations focused on providing evidence for improvement of immunisation delivery in these countries and spanned all phases of Gavi support. The process evaluation approach of the evaluations utilised mixed methods to track progress against defined theories-of-change and related milestones during the various stages of implementation of the Gavi support streams. This article highlights complexities of this type of real-time evaluation and shares lessons learnt on conducting such evaluation from the Zambian experience. Real-time process evaluation is a complex evaluation methodology that requires sensitivity to the context of the evaluation, catering for various information needs of stakeholders, and establishment of mutually beneficial relationships between programme implementers and evaluators. When used appropriately, it can be an effective means of informing programme decisions and aiding programme improvement for both donors and local implementers. Citation: Evaluation PubDate: 2020-02-03T11:49:25Z DOI: 10.1177/1356389019901314
Authors:Niklas Andreas Andersen Pages: 257 - 274 Abstract: Evaluation, Volume 26, Issue 3, Page 257-274, July 2020. In recent years, evaluation research has taken an increasing interest in the proliferation of evaluation systems. One of the main assumptions being that such evaluation systems can enhance the use of evaluations and their findings. The current article analyses this assumption by empirically studying the workings of a specific evaluation system within the Danish Ministry of Employment. By applying the concept of constitutive effects, the article analyses how the evaluation system affects processes of policymaking within the central government. It is shown how the institutionalisation of an evidence-based evaluation system, at the ministerial level, structures the process of policymaking around a specific understanding of policy content, timeframes and expertise. Based on these findings, it is argued that evaluation systems can both enhance the instrumental use of evaluations and significantly alter policymaking processes. Citation: Evaluation PubDate: 2019-09-19T11:24:22Z DOI: 10.1177/1356389019876661 Issue No:Vol. 26, No. 3 (2019)
Authors:Shushu Chen, Ian Henry Pages: 275 - 295 Abstract: Evaluation, Volume 26, Issue 3, Page 275-295, July 2020. Olympic impact/legacy analysis as a research area has become a frequently discussed topic in the sociology of sport literature in general and in event management in particular since the 2000s. Although Olympic impact/legacy analysis has increased in volume, many studies still face methodological challenges (e.g. failure to adopt rigorous theoretical frameworks or overlooking additionality). The aim of this article is to unpack the logic of stakeholders in a critical evaluation of two London 2012 legacy programmes, critically reviewing the outcomes sought, and the actions adopted to achieve those outcomes, with reference to the concepts of programme theory, action theory and process tracing. We seek to redress the shortcomings of previous Olympic legacy claims which have failed to link theoretically informed accounts of the nature of Olympic-led/Olympic-inspired outcomes to explanations of the specific causal mechanisms that account for the generation of such outcomes. Citation: Evaluation PubDate: 2019-03-18T11:01:09Z DOI: 10.1177/1356389019836675 Issue No:Vol. 26, No. 3 (2019)
Authors:Lesley M. Harper, Michelle Maden, Rumona Dickson Pages: 350 - 366 Abstract: Evaluation, Volume 26, Issue 3, Page 350-366, July 2020. This article describes The Evidence of Impact Model that was recently developed to support the analysis of short- to medium-term impact arising from Participatory Action Research and Developmental Evaluation approaches that were used within the Evidence for Change initiative. Evidence for Change was a team capacity building initiative that supported rapid, bottom-up, evidence informed change in health and social care environments. To explain the need for The Evidence of Impact Model and outline how it was developed, a variety of similar models are explored and the challenges of capturing impact arising from Evidence for Change are discussed. The resulting new model based on the existing concept of micro, meso and macro levels of impact is outlined in detail with supporting qualitative evidence. The authors conclude that The Evidence of Impact Model provides an effective lens for exploring impact at a variety of levels within an emerging and complex environment. Citation: Evaluation PubDate: 2019-06-04T08:27:31Z DOI: 10.1177/1356389019850844 Issue No:Vol. 26, No. 3 (2019)
Authors:Melanie Handley, Frances Bunn, Jennifer Lynch, Claire Goodman Pages: 380 - 393 Abstract: Evaluation, Volume 26, Issue 3, Page 380-393, July 2020. This article outlines how a realist evaluation of dementia care in hospitals used non-participant observation to support the refinement and testing of mechanisms likely to lead to the use of person-centred care. We found that comments and explanations of their actions from hospital staff during observation periods provided insights into the reasoning that generated their actions for care in real time. This informed subsequent data collection and analysis. Two worked examples of mechanisms first identified during non-participant observation demonstrate (1) how they were uncovered, and (2) how this informed research activities for theory refinement. Early, iterative engagement with the analytic process, primarily involving reflection and debate with the research team, maximised the potential of observation data to support surfacing underlying mechanisms, linking them to specific contexts and outcomes. Citation: Evaluation PubDate: 2019-08-09T06:05:26Z DOI: 10.1177/1356389019869036 Issue No:Vol. 26, No. 3 (2019)
Hybrid journal (It can contain Open Access articles)
[1093 journals]
