Email updates

Keep up to date with the latest news and content from BMC Medical Research Methodology and BioMed Central.

Open Access Research article

A randomised trial and economic evaluation of the effect of response mode on response rate, response bias, and item non-response in a survey of doctors

Anthony Scott1, Sung-Hee Jeon1, Catherine M Joyce2*, John S Humphreys3, Guyonne Kalb1, Julia Witt4 and Anne Leahy1

Author Affiliations

1 Melbourne Institute of Applied Economic and Social Research, Faculty of Business and Economics The University of Melbourne Level 7, Alan Gilbert Building, 161 Barry Street, Carlton, VIC 3053, Australia

2 Department of Epidemiology and Preventive Medicine School of Public Health and Preventive Medicine Monash University, Alfred Hospital Melbourne 3004, Australia

3 Monash University School of Rural Health PO Box 666 Bendigo Central VIC 3552, Australia

4 Department of Economics 501 Fletcher Argue Building University of Manitoba, Winnipeg, MB R3T 5V5 Canada

For all author emails, please log on.

BMC Medical Research Methodology 2011, 11:126  doi:10.1186/1471-2288-11-126

Published: 5 September 2011

Abstract

Background

Surveys of doctors are an important data collection method in health services research. Ways to improve response rates, minimise survey response bias and item non-response, within a given budget, have not previously been addressed in the same study. The aim of this paper is to compare the effects and costs of three different modes of survey administration in a national survey of doctors.

Methods

A stratified random sample of 4.9% (2,702/54,160) of doctors undertaking clinical practice was drawn from a national directory of all doctors in Australia. Stratification was by four doctor types: general practitioners, specialists, specialists-in-training, and hospital non-specialists, and by six rural/remote categories. A three-arm parallel trial design with equal randomisation across arms was used. Doctors were randomly allocated to: online questionnaire (902); simultaneous mixed mode (a paper questionnaire and login details sent together) (900); or, sequential mixed mode (online followed by a paper questionnaire with the reminder) (900). Analysis was by intention to treat, as within each primary mode, doctors could choose either paper or online. Primary outcome measures were response rate, survey response bias, item non-response, and cost.

Results

The online mode had a response rate 12.95%, followed by the simultaneous mixed mode with 19.7%, and the sequential mixed mode with 20.7%. After adjusting for observed differences between the groups, the online mode had a 7 percentage point lower response rate compared to the simultaneous mixed mode, and a 7.7 percentage point lower response rate compared to sequential mixed mode. The difference in response rate between the sequential and simultaneous modes was not statistically significant. Both mixed modes showed evidence of response bias, whilst the characteristics of online respondents were similar to the population. However, the online mode had a higher rate of item non-response compared to both mixed modes. The total cost of the online survey was 38% lower than simultaneous mixed mode and 22% lower than sequential mixed mode. The cost of the sequential mixed mode was 14% lower than simultaneous mixed mode. Compared to the online mode, the sequential mixed mode was the most cost-effective, although exhibiting some evidence of response bias.

Conclusions

Decisions on which survey mode to use depend on response rates, response bias, item non-response and costs. The sequential mixed mode appears to be the most cost-effective mode of survey administration for surveys of the population of doctors, if one is prepared to accept a degree of response bias. Online surveys are not yet suitable to be used exclusively for surveys of the doctor population.