Email updates

Keep up to date with the latest news and content from BMC Medical Education and BioMed Central.

Open Access Research article

The validity of a behavioural multiple-mini-interview within an assessment centre for selection into specialty training

Chris Roberts1*, Tyler Clark1, Annette Burgess1, Michael Frommer1, Marcia Grant2 and Karyn Mossman1

Author Affiliations

1 Sydney Medical School, University of Sydney, Sydney, NSW 2006, Australia

2 General Practice Education and Training, 10 Rudd Street, GPO Box 2914, Canberra, ACT 2601, Australia

For all author emails, please log on.

BMC Medical Education 2014, 14:169  doi:10.1186/1472-6920-14-169

Published: 13 August 2014

Abstract

Background

Entry into specialty training was determined by a National Assessment Centre (NAC) approach using a combination of a behavioural Multiple-Mini-Interview (MMI) and a written Situational Judgement Test (SJT). We wanted to know if interviewers could make reliable and valid decisions about the non-cognitive characteristics of candidates with the purpose of selecting them into general practice specialty training using the MMI. Second, we explored the concurrent validity of the MMI with the SJT.

Methods

A variance components analysis estimated the reliability and sources of measurement error. Further modelling estimated the optimal configurations for future MMI iterations. We calculated the relationship of the MMI with the SJT.

Results

Data were available from 1382 candidates, 254 interviewers, six MMI questions, five alternate forms of a 50-item SJT, and 11 assessment centres. For a single MMI question and one assessor, 28% of the variance between scores was due to candidate-to-candidate variation. Interviewer subjectivity, in particular the varying views that interviewer had for particular candidates accounted for 40% of the variance in scores. The generalisability co-efficient for a six question MMI was 0.7; to achieve 0.8 would require ten questions. A disattenuated correlation with the SJT (r = 0.35), and in particular a raw score correlation with the subdomain related to clinical knowledge (r = 0.25) demonstrated evidence for construct and concurrent validity. Less than two per cent of candidates would have failed the MMI.

Conclusion

The MMI is a moderately reliable method of assessment in the context of a National Assessment Centre approach. The largest source of error relates to aspects of interviewer subjectivity, suggesting enhanced interviewer training would be beneficial. MMIs need to be sufficiently long for precise comparison for ranking purposes. In order to justify long term sustainable use of the MMI in a postgraduate assessment centre approach, more theoretical work is required to understand how written and performance based test of non-cognitive attributes can be combined, in a way that achieves acceptable generalizability, and has validity.

Keywords:
Selection; Validity; Postgraduate training; Assessment; Multiple-mini-interview; Situational judgment test; Assessment centre; General practice