Selection is a key process at all stages of a medical career: selection into medical school, selection into postgraduate training, and selection into specialist posts. However, selection is only justified if it is valid, those chosen going on to perform better than those who were not chosen. Meaningful selection also needs to consider performance right through a medical career, not just in the first few years. The best method to assess the outcomes of selection is through cohort studies, in which large groups of medical students or junior doctors are followed longitudinally through their careers. A great strength and success of UK social science has been its studies of very large cohorts of individuals, such as the birth cohort studies of 1946, 1958, 1970 and the Millennium. A review in the journal Science a few years ago described how, “Longitudinal surveys, which collect information about the same persons over many years, have given the social sciences their Hubble telescope … allow[ing] the observing researcher to look back in time and record the antecedents of current events and transitions” (Science 2006, 312: 1900).
In the trilogy of articles we recently published in BMC Medicine, we emphasise the value of cohort studies to medical education researchers by describing data from six different cohort studies. The earliest was begun in the 1970s by the late Dr Peter Fleming, the next three were started by Chris McManus and the late Professor Peter Richards in 1980, 1985 and 1990, a newer one was initiated in 2004 by Katherine Woolf, Chris McManus and Jane Dacre, and the most recent is based on data collected from 2006 onwards as a part of the routine use of the UKCAT test in student selection in UK medical schools.
In recent follow-ups the cohort data can be linked into other large databases, such as that of performance at postgraduate examinations, and can appear on the General Medical Council (GMC) specialist and general practice registers – all of which are indicators of career outcomes. Data linkage is generally straightforward because UK doctors are identifiable through their GMC registration number.
Medical education, we argue, is currently entering ‘a golden age’ when, for the first time and because of linkage, large studies can assess which measures are important in selection, how undergraduate medical schools differ in the quality and type of doctors they produce, how training in different postgraduate deaneries and in different hospitals and trusts results in doctors of greater or lesser competence, and how all such factors impact on life-long learning, and hence on the care and medical expertise provided by doctors within the UK’s National Health Service.
The three articles use the cohort studies to develop a model of how doctors acquire ‘medical capital’ – the mass of knowledge, theory, craftsmanship and expertise that underpins the scientific practice of medicine. Acquisition of new capital at each stage of training is built upon capital acquired at an earlier stage, as is recognised in various ‘spiral’ models of curricula. Earlier successes allow later successes, but also problems at earlier stages ripple through into problems at later stages.
The first article assesses how academic attainment at any one stage is correlated with previous attainment, earlier knowledge supporting, buttressing, and underpinning later knowledge acquisition in a process we call the ‘academic backbone’. The important practical implication for medical school selectors is that academic attainment at secondary school is likely to be an important predictor of attainment at medical school and beyond, a result we find in all of our six cohort studies. In the second article which describes the large UKCAT-12 study, and despite the correlations with school-level attainment only being assessed into the first year of the course, the results of the other five cohort studies suggest it is reasonable to expect the predictive power to continue into further years of the course, as the study progresses.
The third article looks in depth at how A-levels, GCSEs and academic aptitude tests predict undergraduate and postgraduate performance, assessing their ‘construct-level predictive validity’. The major conceptual problem for any analysis of selection is that although a wide range of abilities may be present in applicants, the range of abilities in those selected must necessarily be narrower because selectors choose those more likely to have the highest abilities. As a result the correlations of selection measures and outcomes are lower in entrants than they would have been had candidates been allowed to enter from the entire range of abilities (as, say, had selection been at random). Assessing the effectiveness of selection needs estimates of the predictor-outcome correlations across the entire ability range of applicants, not of entrants, and we have used the recently developed techniques of Hunter, Schmidt and Le (J App Psych. 2006, 91: 594-612) to calculate those correlations, corrected for the unreliability of measures and for range restriction.
The construct-level predictive validities suggest perhaps two thirds of the true variance in medical school and postgraduate academic attainment is accountable in terms of pre-medical school academic attainment. That is important; not only does it justify using academic attainment as the mainspring of medical school selection, but it also reminds us forcefully that the remaining one third of the variance is not accounted for, and that something else must explain it. It is not measurement error, and neither is it correlates of academic attainment, such as, say, personality or motivation; it is something separate. Identifying and understanding that missing variance has to be a key challenge for those working in medical student selection. Given our earlier usage of the Hubble Space Telescope as a metaphor for the importance of cohort studies in understanding social processes, we also refer to the missing variance as ‘dark’ variance, echoing the search for ‘dark matter’ and ‘dark energy’ which so dominates modern astrophysics.
The Academic Backbone: longitudinal continuities in educational achievement from secondary school and medical school to MRCP(UK) and the specialist register in UK medical students and doctors
BMC Medicine 2013, 11:242
Go to article >>
The UKCAT-12 study: educational attainment, aptitude test performance, demographic and socio-economic contextual factors as predictors of first year outcome in a cross-sectional collaborative study of 12 UK medical schools
BMC Medicine 2013, 11:244
Go to article >>