JournalTOCs

Evaluation
Journal Prestige (SJR): 0.83

Citation Impact (citeScore): 2
Number of Followers: 20

Hybrid journal * Containing 1 Open Access article(s) in this issue *
ISSN (Print) 1356-3890 - ISSN (Online) 1461-7153
Published by Sage Publications

[1088 journals]

Editorial
- Authors: Elliot Stern
  Pages: 3 - 5
  Abstract: Evaluation, Volume 26, Issue 1, Page 3-5, January 2020.
  
  Citation: Evaluation
  PubDate: 2020-01-01T08:00:00Z
  DOI: 10.1177/1356389019900443
  Issue No: Vol. 26, No. 1 (2020)
Implementer learning in developmental evaluation: A cultural historical
activity theory and expansive learning analysis
- Authors: Cheryl K. Baldwin
  Pages: 27 - 48
  Abstract: Evaluation, Volume 26, Issue 1, Page 27-48, January 2020.
  Cultural historical activity and expansive learning theories were employed in a retrospective case study to investigate implementer learning and how knowledge is co-created in the early stage of an inter-agency partnership supported with developmental evaluation. Data collection methods were semi-structured qualitative interviews, evaluator field notes of implementer discursive interactions, evaluator log, and ethnographic program observations. Activity system analyses revealed a primary contraction between organizational independence and interdependence, which inhibited program implementation. Learning as negotiated work with conflicts manifesting from this contradiction evolved implementer perspectives and created a new model of interdependent implementer leadership practice. The mediating role of the developmental evaluator involved both reality testing and serving as a neutral mediating presence that supported implementers importing their resources for problem-solving as expansive learning.
  Citation: Evaluation
  PubDate: 2020-01-01T08:00:00Z
  DOI: 10.1177/1356389019895033
  Issue No: Vol. 26, No. 1 (2020)
From disenchantment to renewal
- Authors: Robert Picciotto
  Pages: 49 - 60
  Abstract: Evaluation, Volume 26, Issue 1, Page 49-60, January 2020.
  Evaluation as a free-standing discipline arose out of the ashes of World War II, a time of optimism, when government turned to the academy to guide public policy. The evaluation pioneers shared a bracing vision: a search for truth in the public interest. Seventy years later the glitter has faded, and disenchantment has taken hold. Evaluation, a quintessential public good, has become a market good, and eminent evaluation thinkers are asking the same questions about evaluation that they have been routinely asking of others—with sobering results. Yet, countervailing currents and turbulent streams lie just below the surface. Once a tipping point is reached, a new wave of evaluation diffusion will begin to curl. What might it look like'
  Citation: Evaluation
  PubDate: 2020-01-01T08:00:00Z
  DOI: 10.1177/1356389019897696
  Issue No: Vol. 26, No. 1 (2020)
News from the Community
- Pages: 119 - 121
  Abstract: Evaluation, Volume 26, Issue 1, Page 119-121, January 2020.
  
  Citation: Evaluation
  PubDate: 2020-01-01T08:00:00Z
  DOI: 10.1177/1356389019899500
  Issue No: Vol. 26, No. 1 (2020)
French language abstracts
- Pages: 122 - 124
  Abstract: Evaluation, Volume 26, Issue 1, Page 122-124, January 2020.
  
  Citation: Evaluation
  PubDate: 2020-01-01T08:00:00Z
  DOI: 10.1177/1356389019898550
  Issue No: Vol. 26, No. 1 (2020)
Applying Big Data visualization to detect trends in 30 years of
performance reports
- Authors: Eran Raveh, Yuval Ofek, Ron Bekkerman, Hertzel Cohen
  Abstract: Evaluation, Ahead of Print.
  Evaluators worldwide are dealing with a growing amount of unstructured electronic data, predominantly in textual format. Currently, evaluators analyze textual Big Data primarily using traditional content analysis methods based on keyword search, a practice that is limited to iterating over predefined concepts. But what if evaluators cannot define the necessary keywords for their analysis' Often we should examine trends in the way certain organizations have been operating, while our raw data are gigabytes of documents generated by that organization over decades. The problem is that in many cases we do not know what exactly we need to look for. In such cases, traditional analytical machinery would not provide an adequate solution within reasonable time—instead, heavy-lifting Big Data Science should be applied. We propose an automated, quantitative, user-friendly methodology based on text mining, machine learning, and data visualization, which assists researchers and evaluation practitioners to reveal trends, trajectories, and interrelations between bits and pieces of textual information in order to support evaluation. Our system automatically extracts a large amount of descriptive terminology for a particular domain in a given language, finds semantic connections between documents based on the extracted terminology, visualizes the entire document repository as a graph of semantic connections, and leads the user to the areas on that graph where most interesting trends can be observed. This article exemplifies the new method on 1700 performance reports, showing that the method can be used successfully, supplying evaluators with highly important information which cannot be revealed using other methods. Such exploratory exercise is vital as a preliminary exploratory phase for evaluations involving unstructured Big Data, after which a range of evaluation methods can be applied. We argue that our system can be successfully implemented on any domain evaluated.
  Citation: Evaluation
  PubDate: 2020-03-04T07:22:38Z
  DOI: 10.1177/1356389020905322
Quality of quality: A diagnostic approach to qualitative evaluation
- Authors: Barbara Befani
  Abstract: Evaluation, Ahead of Print.
  This article discusses the integration of a diagnostic lens in qualitative or mixed methods evaluations, arguing that this will improve quality: in particular, it will improve transparency, credibility and reliability of evaluation findings.We start by unpacking the notion of evaluation quality and pointing out the typical weaknesses of qualitative methods. We then introduce the basic notions of diagnostic approaches and how they relate to theory-based evaluation, process tracing, and Bayesian updating, arguing for the merits of taking a formal Bayesian approach, founded on the confusion matrix, which amongst else reduces confirmation bias and conservatism. This article draws parallels between the process tracing tests and elements of the confusion matrix.
  Citation: Evaluation
  PubDate: 2020-02-26T06:26:34Z
  DOI: 10.1177/1356389019898223
Real-time evaluation pros and cons: Lessons from the Gavi Full Country
Evaluation in Zambia
- Authors: Moses C. Simuyemba, Obrian Ndlovu, Felicitas Moyo, Eddie Kashinka, Abson Chompola, Aaron Sinyangwe, Felix Masiye
  Abstract: Evaluation, Ahead of Print.
  The Full Country Evaluations were Gavi-funded real-time evaluations of immunisation programmes in Bangladesh, Mozambique, Uganda and Zambia, from 2013 to 2016. The evaluations focused on providing evidence for improvement of immunisation delivery in these countries and spanned all phases of Gavi support. The process evaluation approach of the evaluations utilised mixed methods to track progress against defined theories-of-change and related milestones during the various stages of implementation of the Gavi support streams. This article highlights complexities of this type of real-time evaluation and shares lessons learnt on conducting such evaluation from the Zambian experience. Real-time process evaluation is a complex evaluation methodology that requires sensitivity to the context of the evaluation, catering for various information needs of stakeholders, and establishment of mutually beneficial relationships between programme implementers and evaluators. When used appropriately, it can be an effective means of informing programme decisions and aiding programme improvement for both donors and local implementers.
  Citation: Evaluation
  PubDate: 2020-02-03T11:49:25Z
  DOI: 10.1177/1356389019901314
The contribution of policy design to realist evaluation
- Authors: Guillaume Fontaine
  Abstract: Evaluation, Ahead of Print.
  The literature on policy design provides insightful resources to improve policy evaluation. We compare how this literature addresses causation, evaluation, instrumentation, and intervention, based on different methodologies. Then, we explain how the realist approach can contribute to theory-based evaluation with multi-methods. We go through the process leading from the adoption of policy aims to the production of a policy outcome by a causal mechanism linking agenda setting to policy formulation, cross-sectorial coordination, political interplays, and this policy’s outcome. This hypothesis is then tested against a typology of policy instruments classified by the State resources. These instruments are hereby treated as expected empirical observations that provide an empirical basis to assess the theoretical process following a Bayesian logic. We eventually present a five-step protocol aimed at making realist policy evaluation easier, which includes process theorization and operationalization, empirical tests design, case selection and justification, congruence analysis, and deep within-case study.
  Citation: Evaluation
  PubDate: 2020-02-03T11:46:58Z
  DOI: 10.1177/1356389020902496
Polycentric inspection: A catalyst for sparking positive interactions in
educational networks
- Authors: Martin Brown, Gerry McNamara, Joe O’Hara, Paddy Shevlin
  First page: 76
  Abstract: Evaluation, Ahead of Print.
  The core theory of polycentric inspection is that when schools reach a certain quality threshold, they can further improve best through a coordinated, collaborative effort between clusters of schools and external agencies such as, for example, social services and training providers. The suggested role of an external agency with the respect and resources of the inspectorate is to provide stimulus and support to make the network effective. Using a bounded case study method, this research seeks to assess the potential of polycentric inspection as a tool for improving school effectiveness and outcomes. Evidence from this study suggests that this mode of evaluation has had a significant impact on improving schools, supporting teachers’ practice and, arguably, increasing student examination outcomes in the network examined. In consequence, it is suggested that these findings have wider implications for the changing conception of school evaluation and how improvement can be achieved in education.
  Citation: Evaluation
  PubDate: 2020-01-01T08:00:00Z
  DOI: 10.1177/1356389019895031
The constitutive effects of evaluation systems: Lessons from the
policymaking process of Danish Active Labour Market Policies
- Authors: Niklas Andreas Andersen
  Abstract: Evaluation, Ahead of Print.
  In recent years, evaluation research has taken an increasing interest in the proliferation of evaluation systems. One of the main assumptions being that such evaluation systems can enhance the use of evaluations and their findings. The current article analyses this assumption by empirically studying the workings of a specific evaluation system within the Danish Ministry of Employment. By applying the concept of constitutive effects, the article analyses how the evaluation system affects processes of policymaking within the central government. It is shown how the institutionalisation of an evidence-based evaluation system, at the ministerial level, structures the process of policymaking around a specific understanding of policy content, timeframes and expertise. Based on these findings, it is argued that evaluation systems can both enhance the instrumental use of evaluations and significantly alter policymaking processes.
  Citation: Evaluation
  PubDate: 2019-09-19T11:24:22Z
  DOI: 10.1177/1356389019876661
Using non-participant observation to uncover mechanisms: Insights from a
realist evaluation
- Authors: Melanie Handley, Frances Bunn, Jennifer Lynch, Claire Goodman
  Abstract: Evaluation, Ahead of Print.
  This article outlines how a realist evaluation of dementia care in hospitals used non-participant observation to support the refinement and testing of mechanisms likely to lead to the use of person-centred care. We found that comments and explanations of their actions from hospital staff during observation periods provided insights into the reasoning that generated their actions for care in real time. This informed subsequent data collection and analysis. Two worked examples of mechanisms first identified during non-participant observation demonstrate (1) how they were uncovered, and (2) how this informed research activities for theory refinement. Early, iterative engagement with the analytic process, primarily involving reflection and debate with the research team, maximised the potential of observation data to support surfacing underlying mechanisms, linking them to specific contexts and outcomes.
  Citation: Evaluation
  PubDate: 2019-08-09T06:05:26Z
  DOI: 10.1177/1356389019869036
Across five levels: The evidence of impact model
Open Access Article
- Authors: Lesley M. Harper, Michelle Maden, Rumona Dickson
  Abstract: Evaluation, Ahead of Print.
  
  Citation: Evaluation
  PubDate: 2019-06-04T08:27:31Z
  DOI: 10.1177/1356389019850844
Assessing Olympic legacy claims: Evaluating explanations of causal
mechanisms and policy outcomes
- Authors: Shushu Chen, Ian Henry
  Abstract: Evaluation, Ahead of Print.
  
  Citation: Evaluation
  PubDate: 2019-03-18T11:01:09Z
  DOI: 10.1177/1356389019836675
A worked example of initial theory-building: PARTNERS2 collaborative care
for people who have experienced psychosis in England
- Authors: Ruth Gwernan-Jones, Nicky Britten, Jon Allard, Elina Baker, Laura Gill, Helen Lloyd, Tim Rawcliffe, Ruth Sayers, Humera Plappert, John Gibson, Michael Clark, Maximillian Birchwood, Vanessa Pinfold, Siobhan Reilly, Linda Gask, Richard Byng
  First page: 6
  Abstract: Evaluation, Ahead of Print.
  
  Citation: Evaluation
  PubDate: 2019-05-27T03:49:41Z
  DOI: 10.1177/1356389019850199
Evaluating unintended consequences: New insights into solving practical,
ethical and political challenges of evaluation
- Authors: Kathryn Oliver, Theo Lorenc, Jane Tinkler
  First page: 61
  Abstract: Evaluation, Ahead of Print.
  
  Citation: Evaluation
  PubDate: 2019-06-10T08:16:30Z
  DOI: 10.1177/1356389019850847
Theoretically integrative evaluation practice: A step-by-step overview of
an eclectic evaluation design process
- Authors: Kelsey L. Deane, Hilary Dutton, Pat Bullen
  First page: 98
  Abstract: Evaluation, Ahead of Print.
  
  Citation: Evaluation
  PubDate: 2019-05-28T06:17:21Z
  DOI: 10.1177/1356389019850201