Authors:W. E. De Leng; K. M. Stegers-Jager; A. Husbands; J. S. Dowell; M. Ph. Born; A. P. N. Themmen Pages: 243 - 265 Abstract: Situational Judgment Tests (SJTs) are increasingly used for medical school selection. Scoring an SJT is more complicated than scoring a knowledge test, because there are no objectively correct answers. The scoring method of an SJT may influence the construct and concurrent validity and the adverse impact with respect to non-traditional students. Previous research has compared only a small number of scoring methods and has not studied the effect of scoring method on internal consistency reliability. This study compared 28 different scoring methods for a rating SJT on internal consistency reliability, adverse impact and correlation with personality. The scoring methods varied on four aspects: the way of controlling for systematic error, and the type of reference group, distance and central tendency statistic. All scoring methods were applied to a previously validated integrity-based SJT, administered to 931 medical school applicants. Internal consistency reliability varied between .33 and .73, which is likely explained by the dependence of coefficient alpha on the total score variance. All scoring methods led to significantly higher scores for the ethnic majority than for the non-Western minorities, with effect sizes ranging from 0.48 to 0.66. Eighteen scoring methods showed a significant small positive correlation with agreeableness. Four scoring methods showed a significant small positive correlation with conscientiousness. The way of controlling for systematic error was the most influential scoring method aspect. These results suggest that the increased use of SJTs for selection into medical school must be accompanied by a thorough examination of the scoring method to be used. PubDate: 2017-05-01 DOI: 10.1007/s10459-016-9720-7 Issue No:Vol. 22, No. 2 (2017)
Authors:Maureen E. Kelly; Siun O’Flynn Pages: 267 - 286 Abstract: Aptitude tests are widely used in selection. However, despite certain advantages their use remains controversial. This paper aims to critically appraise five sources of evidence for the construct validity of the Health Professions Admission Test (HPAT)-Ireland, an aptitude test used for selecting undergraduate medical students. The objectives are to identify gaps in the evidence, draw comparisons with other aptitude tests and outline future research directions. Our appraisal of the literature found that stakeholder feedback indicates that there is reasonable evidence for test content validity for two of the three sections of HPAT-Ireland. By contrast the Non-Verbal Reasoning section is widely criticised as having limited relevance to medical school performance and future clinical practice. In terms of concurrent validity there is a significant small to medium, negative correlation with school exit examinations, but not consistently so across all studies (r = −0.18, −0.28, 0.017). Likewise predictive validity studies vary, from negative to moderate strength correlations with examination performance during early years at medical school. Five studies indicate that HPAT-Ireland is supported in principle by the majority of stakeholders. While one consequence of its introduction is that successful applicants are now coming from more diverse academic backgrounds, there is no evidence that the socio-economic background of medical school entrants has been altered significantly. Negative perceptions of unfairness relating to gender, coaching and socio-economics remain. The evidence to date suggests that while there are slight gender differences, initially favouring males, these vary year on year. In conclusion, the attitudes towards, and performance of, HPAT-Ireland is not unlike that of other aptitude tests widely used internationally. The main justifications for its introduction have been achieved, in that Ireland no longer relies exclusively on a single measure of academic record for selection to medical school. However a number of areas require further research and exploration. PubDate: 2017-05-01 DOI: 10.1007/s10459-016-9728-z Issue No:Vol. 22, No. 2 (2017)
Authors:Louise C. Urlings-Strop; Axel P. N. Themmen; Karen M. Stegers-Jager Pages: 287 - 298 Abstract: Several medical schools include candidates’ extracurricular activities in their selection procedure, with promising results regarding their predictive value for achievement during the clinical years of medical school. This study aims to reveal whether the better achievement in clinical training of students selected on the basis of their extracurricular activities could be explained by persistent participation in extracurricular activities during medical school (msECAs). Lottery-admitted and selected student admission groups were compared on their participation in three types of msECAs: (1) research master, (2) important board positions or (3) additional degree programme. Logistic regression was used to measure the effect of admission group on participation in any msECA, adjusted for pre-university GPA. Two-way ANCOVA was used to examine the inter-relationships between admission group, participation in msECAs and clerkship grade, with pre-university GPA as covariate. Significantly more selected students compared to lottery-admitted students participated in any msECA. Participation in msECAs was associated with a higher pre-university GPA for lottery-admitted students only, whereas participation in msECAs was associated with higher clerkship grades for selected students only. These results suggest that persistent participation in extracurricular activities of selected students favours better clinical achievement, supporting the inclusion of ECAs in the selection procedure. More insight in the rationale behind participation in extracurricular activities during medical school may explain differences found between lottery-admitted and selected students. PubDate: 2017-05-01 DOI: 10.1007/s10459-016-9729-y Issue No:Vol. 22, No. 2 (2017)
Authors:Kelly L. Dore; Harold I. Reiter; Sharyn Kreuger; Geoffrey R. Norman Pages: 327 - 336 Abstract: Typically, only a minority of applicants to health professional training are invited to interview. However, pre-interview measures of cognitive skills predict for national licensure scores (Gauer et al. in Med Educ Online 21 2016) and subsequently licensure scores predict for performance in practice (Tamblyn et al. in JAMA 288(23): 3019–3026, 2002; Tamblyn et al. in JAMA 298(9):993–1001, 2007). Assessment of personal and professional characteristics, with the same psychometric rigour of measures of cognitive abilities, are needed upstream in the selection to health profession training programs. To fill that need, Computer-based Assessment for Sampling Personal characteristics (CASPer)—an on-line, video-based screening test—was created. In this paper, we examine the correlation between CASPer and Canadian national licensure examination outcomes in 109 doctors who took CASPer at the time of selection to medical school. Specifically, CASPer scores were correlated against performance on cognitive and ‘non-cognitive’ subsections of both the Medical Council of Canada Qualifying Examination (MCCQE) Parts I (end of medical school) and Part II (18 months into specialty training). Unlike most national licensure exams, MCCQE has specific subcomponents examining personal/professional qualities, providing a unique opportunity for comparison. The results demonstrated moderate predictive validity of CASPer to national licensure outcomes of personal/professional characteristics three to six years after admission to medical school. These types of disattenuated correlations (r = 0.3–0.5) are not otherwise predicted by traditional screening measures. These data support the ability of a computer-based strategy to screen applicants in a feasible, reliable test, which has now demonstrated predictive validity, lending evidence of its validation for medical school applicant selection. PubDate: 2017-05-01 DOI: 10.1007/s10459-016-9739-9 Issue No:Vol. 22, No. 2 (2017)
Authors:Eamonn Ferguson; Filip Lievens Pages: 387 - 399 Abstract: This paper has two objectives: (1) presenting recent advances in personality theory whereby personality traits are conceptualized within a framework that focuses on the dynamic interactions of behaviour, biology, context, and states, and (2) discussing the implications of these developments for measurement and medical selection. We start by presenting evidence that traits are no longer regarded as stable deterministic predictors of behaviour. Instead, traits are found to change across generations, the life span, and in response to environmental contingencies. Thus, there is an urgent need to explore how traits change as function of medical education. Second, drawing on recent theory and research (behavioural reaction norms and the density distribution model) we highlight evidence to show how the expression of trait relevant behaviour is dependent on context, and is distributed with an average (typical behaviour or personality) and a variance (plasticity or adaptability), with traditional personality measure associated with typical responding. Third, we demystify that some traits are better than others showing that so-called “good” traits have a dark-side. Fourth, we show how these developments impact on how personality might be assessed, thereby presenting recent evidence on the use of contextualized personality measures, situational judgment tests, other reports, and implicit measures. Throughout the paper, we outline the key implications of these developments for medical selection practices. PubDate: 2017-05-01 DOI: 10.1007/s10459-016-9751-0 Issue No:Vol. 22, No. 2 (2017)
Authors:Fran Cousans; Fiona Patterson; Helena Edwards; Kim Walker; John C. McLachlan; David Good Pages: 401 - 413 Abstract: Although there is extensive evidence confirming the predictive validity of situational judgement tests (SJTs) in medical education, there remains a shortage of evidence for their predictive validity for performance of postgraduate trainees in their first role in clinical practice. Moreover, to date few researchers have empirically examined the complementary roles of academic and non-academic selection methods in predicting in-role performance. This is an important area of enquiry as despite it being common practice to use both types of methods within a selection system, there is currently no evidence that this approach translates into increased predictive validity of the selection system as a whole, over that achieved by the use of a single selection method. In this preliminary study, the majority of the range of scores achieved by successful applicants to the UK Foundation Programme provided a unique opportunity to address both of these areas of enquiry. Sampling targeted high (>80th percentile) and low (<20th percentile) scorers on the SJT. Supervisors rated 391 trainees’ in-role performance, and incidence of remedial action was collected. SJT and academic performance scores correlated with supervisor ratings (r = .31 and .28, respectively). The relationship was stronger between the SJT and in-role performance for the low scoring group (r = .33, high scoring group r = .11), and between academic performance and in-role performance for the high scoring group (r = .29, low scoring group r = .11). Trainees with low SJT scores were almost five times more likely to receive remedial action. Results indicate that an SJT for entry into trainee physicians’ first role in clinical practice has good predictive validity of supervisor-rated performance and incidence of remedial action. In addition, an SJT and a measure of academic performance appeared to be complementary to each other. These initial findings suggest that SJTs may be more predictive at the lower end of a scoring distribution, and academic attainment more predictive at the higher end. PubDate: 2017-05-01 DOI: 10.1007/s10459-017-9755-4 Issue No:Vol. 22, No. 2 (2017)
Authors:Nienke R. Schripsema; Anke M. van Trigt; Susanna M. Lucieer; Anouk Wouters; Gerda Croiset; Axel P. N. Themmen; Jan C. C. Borleffs; Janke Cohen-Schotanus Pages: 463 - 476 Abstract: Many different medical school selection processes are used worldwide. In this paper, we examine the effect of (1) participation, and (2) selection in a voluntary selection process on study performance. We included data from two cohorts of medical students admitted to Erasmus MC, Rotterdam and VUmc, Amsterdam, The Netherlands and compared them to previously published data from Groningen medical school, The Netherlands. All included students were admitted based on either (1) a top pre-university grade point average, or (2) a voluntary selection process, or (3) weighted lottery. We distinguished between lottery-admitted students who had participated in the voluntary selection process and had been rejected, and lottery-admitted students who had not participated. Knowledge test scores, study progress, and professionalism scores were examined using ANCOVA modelling, logistic regression, and Bonferroni post hoc multiple-comparison tests, controlling for gender and cohort. For written test grades, results showed a participation effect at Groningen medical school and Erasmus MC (p < 0.001), and a selection effect at VUmc (p < 0.05). For obtained course credits, results showed a participation effect at all universities (p < 0.01) and a selection effect at Groningen medical school (p < 0.005). At Groningen medical school, a participation effect seemed apparent in on time first-year completion (p < 0.05). Earlier reported selection and participation effects in professionalism scores at Groningen medical school were not apparent at VUmc. Top pre-university students performed well on all outcome measures. For both the participation effect and the selection effect, results differed between universities. Institutional differences in curricula and in the design of the selection process seem to mediate relations between the different admissions processes and performance. Further research is needed for a deeper understanding of the influence of institutional differences on selection outcomes. PubDate: 2017-05-01 DOI: 10.1007/s10459-017-9762-5 Issue No:Vol. 22, No. 2 (2017)
Authors:S. Nicholson; J. A. Cleland Pages: 477 - 490 Abstract: In the UK widening access (WA) activities and policies aim to increase the representation from lower socio-economic groups into Higher Education. Whilst linked to a political rhetoric of inclusive education such initiatives have however failed to significantly increase the number of such students entering medicine. This is compounded by a discourse that portrays WA applicants and students as lacking the essential skills or attributes to be successful in medical education. Much of the research in this area to date has been weak and it is critical to better understand how WA applicants and students negotiate medical admissions and education to inform change. To address this gap we amalgamated a larger dataset from three qualitative studies of student experiences of WA to medicine (48 participants in total). Inductively analysing the findings using social capital as a theoretical lens we created and clustered codes into categories, informed by the concepts of “weak ties” and “bridging and linking capital”, terms used by previous workers in this field, to better understand student journeys in medical education. Successful applicants from lower socio-economic groups recognise and mobilise weak ties to create linking capital. However once in medical school these students seem less aware of the need for, or how to create, capital effectively. We argue WA activities should support increasing the social capital of under-represented applicants and students, and future selection policy needs to take into account the varying social capital of students, so as to not overtly disadvantage some social groups. PubDate: 2017-05-01 DOI: 10.1007/s10459-016-9735-0 Issue No:Vol. 22, No. 2 (2017)
Authors:Barbara Griffin; Erik Porfeli; Wendy Hu Pages: 491 - 504 Abstract: A frequently cited rationale for increasing the participation of students from low socioeconomic status (SES) backgrounds is that it will create a workforce who will choose to work in low SES and medically underserviced communities. Two theoretical arguments, one that supports and one that contradicts this assumption, are proposed to explain the practice location intentions of medical students which we examine in a longitudinal analysis. SES background and future intentions of 351 applicants to an undergraduate medical degree were assessed at Time 1, with intentions re-assessed one year later for 96% of those who were enrolled as medical students. Students from very low (and very high) SES backgrounds indicated lower intention to practice in low SES or medically underserviced areas than those from mid-range SES backgrounds. Males and students from non-English speaking backgrounds indicated less desire to work in low SES areas, perhaps explained by high aspirational motivation. SES accounted for a relatively small amount of variance in practice intentions. Alternate predictors of practice location, including individual values and training effects, and their implications for selection practice, are discussed. PubDate: 2017-05-01 DOI: 10.1007/s10459-016-9726-1 Issue No:Vol. 22, No. 2 (2017)
Authors:Neil Stringer; Michael Chan; Yaw Bimpeh; Philip Chan Pages: 505 - 519 Abstract: This study investigates the effects of socioeconomic status and schooling on the academic attainment of a cohort of students at a single medical school (N = 240). Partial least squares structural equation modelling was used to explore how students’ summative assessment scores over 4 years of medical school were affected by: attainment in secondary school examinations (GCSEs and A-levels); the Income Deprivation Affecting Children Index (IDACI) rank associated with students’ home postcodes; the performance of students’ A-level institutions, measured as the percentage of A-level students achieving 3 A-levels at AAB or higher in two or more facilitating subjects. The effects were consistent across time; the final linear regression model used students’ cumulative scores (the basis of the medical school’s UK Foundation Programme submission) as the dependent variable. The final model fit was quite poor (R2 = .184, n = 178). IDACI Rank was non-significant and excluded from the final model. Both GCSE (.340, p < .001) and A-level (.204, p < .005) scores were associated with increasing Cumulative Score; School Performance was associated with decreasing Cumulative Score (−.159, p < .05). This study confirmed the predictive validity of prior academic attainment and found the same inverse relationship between schooling and medical course performance as previous studies. The study found no evidence that socioeconomic background affects course performance; however, students admitted to medicine from poorly performing schools achieve higher academic attainment throughout the course than students admitted from better-performing schools with the same grades. Schooling could be taken into account for admissions purposes. PubDate: 2017-05-01 DOI: 10.1007/s10459-016-9714-5 Issue No:Vol. 22, No. 2 (2017)
Authors:Nienke R. Schripsema; Anke M. van Trigt; Jan C. C. Borleffs; Janke Cohen-Schotanus Pages: 521 - 532 Abstract: Situational Judgement Tests (SJTs) are increasingly implemented in medical school admissions. In this paper, we investigate the effects of vocational interests, previous academic experience, gender and age on SJT performance. The SJT was part of the selection process for the Bachelor’s degree programme in Medicine at University of Groningen, the Netherlands. All applicants for the academic year 2015–2016 were included and had to choose between learning communities Global Health (n = 126), Sustainable Care (n = 149), Intramural Care (n = 225), or Molecular Medicine (n = 116). This choice was used as a proxy for vocational interest. In addition, all graduate-entry applicants for academic year 2015–2016 (n = 213) were included to examine the effect of previous academic experience on performance. We used MANCOVA analyses with Bonferroni post hoc multiple comparisons tests for applicant performance on a six-scenario SJT. The MANCOVA analyses showed that for all scenarios, the independent variables were significantly related to performance (Pillai’s Trace: 0.02–0.47, p < .01). Vocational interest was related to performance on three scenarios (p < .01). Graduate-entry applicants outperformed all other groups on three scenarios (p < .01) and at least one other group on the other three scenarios (p < .01). Female applicants outperformed male applicants on three scenarios (p < .01) and age was positively related to performance on two scenarios (p < .05). A good fit between applicants’ vocational interests and SJT scenario was related to better performance, as was previous academic experience. Gender and age were related to performance on SJT scenarios in different settings. Especially the first effect might be helpful in selecting appropriate candidates for areas of health care in which more professionals are needed. PubDate: 2017-05-01 DOI: 10.1007/s10459-016-9747-9 Issue No:Vol. 22, No. 2 (2017)
Authors:Marieke de Visser; Cornelia Fluit; Janke Cohen-Schotanus; Roland Laan Abstract: In medical school selection, non-cognitive performance in particular correlates with performance in clinical practice. It is arguable, therefore, that selection should focus on non-cognitive aspects despite the predictive value of prior cognitive performance for early medical school performance. The aim of this study at Radboud University Medical Center, the Netherlands, is to determine the effects of admitting students through an autonomous non-cognitive procedure on early medical school performance. We compared their performance to the performance of students selected through an autonomous cognitive selection procedure, enrolling in the Bachelor’s curriculum simultaneously. 574 students (2013 and 2014 cohorts), admitted through non-cognitive selection (based on portfolio, CASPer and MMI, n = 135) or cognitive selection (curriculum sample selection, n = 439) were included in the study. We compared dropout rates, course credits and grades, using logistic and linear regression. The dropout rate was the highest in the non-cognitive selection group (p < 0.001). Students admitted through non-cognitive selection more often obtained the highest grade for the nursing attachment (p = 0.02) and had a higher mean grade for the practical clinical course in year 3 (p = .04). No differences in course grades were found. The results indicate that students perform best on the elements of the curriculum that are represented most strongly in the selection procedure they had participated in. We recommend the use of curriculum sample procedures, resembling the early medical school curriculum,—whether it has a more cognitive or a more non-cognitive focus—, to select the students who are likely to be successful in the subsequent curriculum. PubDate: 2017-06-10 DOI: 10.1007/s10459-017-9782-1
Authors:Simon C. Kitto; Rachel E. Grant; Jennifer Peller; Carol-Anne Moulton; Steven Gallinger Abstract: In 2007 the Cancer Care Ontario Hepatobiliary-Pancreatic (HPB) Community of Practice was formed during the wake of provincial regionalization of HPB services in Ontario, Canada. Despite being conceptualized within the literature as an educational intervention, communities of practice (CoP) are increasingly being adopted in healthcare as quality improvement initiatives. A qualitative case study approach using in-depth interviews and document analysis was employed to gain insight into the perceptions and attitudes of the HPB surgeons in the CoP. This study demonstrates how an engineered formal or idealized structure of a CoP was created in tension with the natural CoPs that HPB surgeons identified with during and after their training. This tension contributed to the inactive and/or marginal participation by some of the surgeons in the CoP. The findings of this study represent a cautionary tale for such future engineering attempts in two distinct ways: (1) a CoP in surgery cannot simply be created by regulatory agencies, rather they need to be supported in a way to evolve naturally, and (2) when the concept of CoPs is co-opted by governing bodies, it does not necessarily capture the power and potential of situated learning. To ensure CoP sustainability and effectiveness, we suggest that both core and peripheral members need to be more directly involved at the inception of the COP in terms of design, organization, implementation and ongoing management. PubDate: 2017-06-09 DOI: 10.1007/s10459-017-9776-z
Authors:Viktoria C. T. Joynes Abstract: This paper is concerned with exploring the relationship between perceptions of professional identities, interprofessional education (IPE) and collaborative practice. It seeks to introduce the concept of interprofessional responsibility as both a shift in the way in which to conceptualise the professional identity of Health and Social Care (H&SC) staff and as a new set of practices that help to inform the way in which students are prepared for collaborative working. The presented research, undertaken as part of a Ph.D. study, is based upon semi-structured interviews (n = 33) with H&SC staff who were recruited from both the United Kingdom (UK) Health Service and UK universities. Drawing upon thematic analysis of the data, the results of the research identified that previous conceptualisations of professional identity aligned to a whole profession do not relate to the way in which professionals perceive their identities. Senior professionals claimed to be more comfortable with their own professional identity, and with working across professional boundaries, than junior colleagues. Academic staff also identified that much IPE currently taught in universities serves the purpose of box-ticking rather than being delivered in meaningful way. It is proposed that the findings have implications for the way in which IPE is currently taught, and that adoption of the proposed concept of ‘interprofessional responsibility’ may help address some of the concerns these findings raise. PubDate: 2017-05-17 DOI: 10.1007/s10459-017-9778-x
Authors:Steven E. Gay; Sally A. Santen; Rajesh S. Mangrulkar; Thomas H. Sisson; Paula T. Ross; Nikki L. Bibler Zaidi Abstract: Medical school admissions interviews are used to assess applicants’ nonacademic characteristics as advocated by the Association of American Medical Colleges’ Advancing Holistic Review Initiative. The objective of this study is to determine whether academic metrics continue to significantly influence interviewers’ scores in holistic processes by blinding interviewers to applicants’ undergraduate grade point averages (uGPA) and Medical College Admission Test (MCAT). This study examines academic and demographic predictors of interview scores for two applicant cohorts at the University of Michigan Medical School. In 2012, interviewers were provided applicants’ uGPA and MCAT scores; in 2013, these academic metrics were withheld from interviewers’ files. Hierarchical regression analysis was conducted to examine the influence of academic and demographic variables on overall cohort interview scores. When interviewers were provided uGPA and MCAT scores, academic metrics explained more variation in interview scores (7.9%) than when interviewers were blinded to these metrics (4.1%). Further analysis showed a statistically significant interaction between cohort and uGPA, indicating that the association between uGPA and interview scores was significantly stronger for the 2012 unblinded cohort compared to the 2013 blinded cohort (β = .573, P < .05). By contrast, MCAT scores had no interactive effects on interviewer scores. While MCAT scores accounted for some variation in interview scores for both cohorts, only access to uGPA significantly influenced interviewers’ scores when looking at interaction effects. Withholding academic metrics from interviewers’ files may promote assessment of nonacademic characteristics independently from academic metrics. PubDate: 2017-05-13 DOI: 10.1007/s10459-017-9779-9
Hybrid journal (It can contain Open Access articles)
[2345 journals]
