Measuring up in medicine: Chris McManus and Katherine Woolf analyse the impact of selection methods in medical education

Posted by Biome on 14th November 2013 - 0 Comments


Selection is a key process at all stages of a medical career: selection into medical school, selection into postgraduate training, and selection into specialist posts. However, selection is only justified if it is valid, those chosen going on to perform better than those who were not chosen. Meaningful selection also needs to consider performance right through a medical career, not just in the first few years. The best method to assess the outcomes of selection is through cohort studies, in which large groups of medical students or junior doctors are followed longitudinally through their careers. A great strength and success of UK social science has been its studies of very large cohorts of individuals, such as the birth cohort studies of 1946, 1958, 1970 and the Millennium.  A review in the journal Science a few years ago described how, “Longitudinal surveys, which collect information about the same persons over many years, have given the social sciences their Hubble telescope … allow[ing] the observing researcher to look back in time and record the antecedents of current events and transitions” (Science 2006, 312: 1900).

Hubble space telescope in orbit. Image source: Flickr, NASA Goddard Photo and Video

In the trilogy of articles we recently published in BMC Medicine, we emphasise the value of cohort studies to medical education researchers by describing data from six different cohort studies. The earliest was begun in the 1970s by the late Dr Peter Fleming, the next three were started by Chris McManus and the late Professor Peter Richards in 1980, 1985 and 1990, a newer one was initiated in 2004 by Katherine Woolf, Chris McManus and Jane Dacre, and the most recent is based on data collected from 2006 onwards as a part of the routine use of the UKCAT test in student selection in UK medical schools.

In recent follow-ups the cohort data can be linked into other large databases, such as that of performance at postgraduate examinations, and can appear on the General Medical Council (GMC) specialist and general practice registers – all of which are indicators of career outcomes. Data linkage is generally straightforward because UK doctors are identifiable through their GMC registration number.

Medical education, we argue, is currently entering ‘a golden age’ when, for the first time and because of linkage, large studies can assess which measures are important in selection, how undergraduate medical schools differ in the quality and type of doctors they produce, how training in different postgraduate deaneries and in different hospitals and trusts results in doctors of greater or lesser competence, and how all such factors impact on life-long learning, and hence on the care and medical expertise provided by doctors within the UK’s National Health Service.

The three articles use the cohort studies to develop a model of how doctors acquire ‘medical capital’  – the mass of knowledge, theory, craftsmanship and expertise that underpins the scientific practice of medicine. Acquisition of new capital at each stage of training is built upon capital acquired at an earlier stage, as is recognised in various ‘spiral’ models of curricula. Earlier successes allow later successes, but also problems at earlier stages ripple through into problems at later stages.

The first article assesses how academic attainment at any one stage is correlated with previous attainment, earlier knowledge supporting, buttressing, and underpinning later knowledge acquisition in a process we call the ‘academic backbone’. The important practical implication for medical school selectors is that academic attainment at secondary school is likely to be an important predictor of attainment at medical school and beyond, a result we find in all of our six cohort studies. In the second article which describes the large UKCAT-12 study, and despite the correlations with school-level attainment only being assessed into the first year of the course, the results of the other five cohort studies suggest it is reasonable to expect the predictive power to continue into further years of the course, as the study progresses.

Path analyses of 'the academic backbone' indicating the relationships between performance measures obtained prior to medical school (blue), at medical school (green) and at postgraduate level (red), in the UK. Image source: McManus et al, BMC Medicine, 2013, 11:242

The third article looks in depth at how A-levels, GCSEs and academic aptitude tests predict undergraduate and postgraduate performance, assessing their ‘construct-level predictive validity’. The major conceptual problem for any analysis of selection is that although a wide range of abilities may be present in applicants, the range of abilities in those selected  must necessarily be narrower because selectors choose those more likely to have the highest abilities. As a result the correlations of selection measures and outcomes are lower in entrants than they would have been had candidates been allowed to enter from the entire range of abilities (as, say, had selection been at random). Assessing the effectiveness of selection needs estimates of the predictor-outcome correlations across the entire ability range of applicants, not of entrants, and we have used the recently developed techniques of Hunter, Schmidt and Le (J App Psych. 2006, 91: 594-612) to calculate those correlations, corrected for the unreliability of measures and for range restriction.

The construct-level predictive validities suggest perhaps two thirds of the true variance in medical school and postgraduate academic attainment is accountable in terms of pre-medical school academic attainment. That is important; not only does it justify using academic attainment as the mainspring of medical school selection, but it also reminds us forcefully that the remaining one third of the variance is not accounted for, and that something else must explain it. It is not measurement error, and neither is it correlates of academic attainment, such as, say, personality or motivation; it is something separate. Identifying and understanding that missing variance has to be a key challenge for those working in medical student selection. Given our earlier usage of the Hubble Space Telescope as a metaphor for the importance of cohort studies in understanding social processes, we also refer to the missing variance as ‘dark’ variance, echoing the search for ‘dark matter’ and ‘dark energy’ which so dominates modern astrophysics.

 

More about the author(s)

Chris McManus, Professor of Psychology and Medical Education at University College London, UK.

Chris McManus is currently Professor of Psychology and Medical Education at University College London, UK. McManus read psychology at the University of Cambridge, UK, where he also completed his PhD in human leterality. He completed his medical degree at the University of Birmingham, UK and went on to practice in neurosurgery and general medicine. McManus went on to lecture in psychology at St Mary’s Hospital Medical School, Bedford College, University College London, which led to his apoointment as Professor of Psychology at Imperial College School of Medicine at St Mary’s and University College London. In addition to his current position as University College London, he also serves as educational adviser to the MRCP UK (Membership of the Royal College of Physicians of the United Kingdom) and is a member of the PLAB Review Body for the UK’s General Medical Council.

Katherine Woolf, lecturer and honorary research associate at University College London Medical School, UK.

Katherine Woolf is a lecturer and honorary research associate at University College London Medical School, UK. She studied Psychology at Goldsmiths College, London, UK, and obtained her PhD in medical education at University College London, UK. Woolf is a fellow of the UK’s Higher Education Academy and a member of the British Psychological Society and the Association for the Study of Medical Education. She also serves as an Associate Editor for BMC Medical Education. Woolf’s research interests focus on analysing what affects the performance of medical students and doctors, probing what improves and what hinders performance with the goal of ultimately improving patient care. She takes a social psychological and multivariate statistical approach, investigating the effects of stereotyping on performance, and the influence of social networks on learning and attainment.