A survey was carried out in the Grampian region of Scotland with a random sample of 10,000 adults registered with a General Practitioner in Grampian. The study complied with new legislation requiring a two-stage approach to identify and recruit participants, and examined the implications of this for response rates, non-response bias and speed of response.
A two-stage survey was carried out consistent with new confidentiality guidelines. Individuals were contacted by post and asked by the Director of Public Health to consent to receive a postal or electronic questionnaire about communicating their views to the NHS. Those who consented were then sent questionnaires. Response rates at both stages were measured.
25% of people returned signed consent forms and were invited to complete questionnaires. Respondents at the consent stage were more likely to be female (odds ratio (OR) response rate of women compared to men = 1.5, 95% CI 1.4, 1.7), less likely to live in deprived postal areas (OR = 0.59, 95% CI 0.45, 0.78) and more likely to be older (OR for people born in 1930–39 compared to people born in 1970–79 = 2.82, 95% CI 2.36, 3.37). 80% of people who were invited to complete questionnaires returned them. Response rates were higher among older age groups. The overall response rate to the survey was 20%, relative to the original number approached for consent (1951/10000).
The requirement of a separate, prior consent stage may significantly reduce overall survey response rates and necessitate the use of substantially larger initial samples for population surveys. It may also exacerbate non-response bias with respect to demographic variables.
Accurate information on population characteristics and healthcare needs is an essential pre-requisite for informed healthcare decision making. There are a number of ways of eliciting this population-level information; one of the most common being self-completion questionnaire surveys. Self-completion surveys have been used with significant success in a number of fields; for example, to elicit information on lifestyle, occupation and health. A substantial review of published surveys is provided by McColl et al. . Self-completion questionnaires have been used widely in epidemiological and community health studies and they are also potentially important to ascertain public opinion relating to the development of health services .
It is well known that accurate estimates of population information can be ascertained through the use of a sample of the population, rather than having to census the entire population . However, to ensure that the information received from a sample is representative of the underlying population, the sampling strategy must be chosen carefully to ensure that the underlying population structure (age, gender, postal area) is accurately represented . Of the various sampling frames available for population-based surveys, those containing both individual names and associated demographic data are particularly valuable because they make it possible to use personalised mailings and stratified sampling (sampling that accounts for the underlying structure of the population) to compare respondent and non-respondent profiles. The UK electoral roll is one possible sampling frame but this holds very little demographic information and has long been acknowledged to be a biased sampling frame because it excludes foreign nationals and those choosing not to register. A better sampling frame is Scotland's Community Health Index (CHI) (equivalent to Primary Care Trust (PCT) patient registration databases in England and Wales), which contains details of all people registered with a general practitioner, together with data such as age, gender and postal area. These sampling frames have been used extensively in the past with success, achieving response rates of 70–80% [5,6]. However, the introduction of a number of legislative reforms in recent years [7-9], has resulted in severe limitations being imposed on access to these registers . For example, due to a legal challenge to the release of personal details from the electoral roll, access is now only allowed to an edited register, which excludes people withholding consent to be listed, thus introducing an additional bias in the demographic profile .
In particular, the strengthening of legislation and guidance to protect the confidentiality of individual data has restricted the ways in which previously recorded information about individuals can be used for secondary purposes [7-9]. In the health field, "Caldicott Guardians" (senior health professionals appointed to be responsible for ensuring that patients' rights to confidentiality are respected) are now required to obtain consent from individuals before releasing their demographic information to researchers . In Scotland, Directors of Public Health act as Caldicott Guardians for the CHI. Within our region (NHS Grampian), confidentiality protocols now require that potential survey participants identified from the Community Health Index are sent a letter from the Director of Public Health asking them for their consent for a researcher to contact them directly. This is an "opt in" process. Non-respondents to this initial approach are assumed to have withheld consent and further contact with them (even to issue a reminder letter) is not permitted.
The implications of these new requirements for the conduct and findings of surveys are as yet unclear. In this paper, we examine response rates, non-response bias and response times for initial requests for consent and subsequent self-completion questionnaires in the context of a population-based survey carried out in Grampian according to the new protocols.
The study context
The survey was conducted as part of a study funded by the Scottish Executive Health Department to compare postal and electronic versions of a questionnaire. At present, there are no population-based registers of e-mail addresses in the UK. We therefore included, within the newly mandated initial request for consent to receive a questionnaire, a request that respondents let us know their e-mail address (if they had one).
The questionnaire focussed on ways in which service users and the public could feedback/communicate their views to the NHS. The electronic version was presented on LetsHearU software  and hosted on a website provided by British Telecom, which recorded individual responses. The questionnaire findings and details of the comparison between electronic and paper versions are being reported separately.
The study was carried out in the area covered by NHS Grampian, which accounts for approximately 10% of the Scottish population. The CHI for Grampian is maintained to a high standard, being continually updated with information such as GP registrations, birth and death information, transfers out of the area and name/address changes. It is also synchronised with General Practitioner and NHS Central Registry databases to maintain currency and uniformity.
The Grampian CHI uses four bands of deprivation category – none, low, medium and high, corresponding to Jarman scores of less than 30, 30 to 39.9, 40–49.9 and over 50, respectively, correlated with residential postcode. 4% of the Grampian population is designated as resident in deprived postal areas.
In October 2002, a computer-generated random sample of 10,000 individuals (sampling rate 2.3%, N = 434,690), aged 18 and over, was extracted from the Grampian CHI. Each individual was allocated a unique, anonymised study identifier. Demographic information, including deprivation levels based on Jarman scores, was stored on a secure, confidential database set up on behalf of NHS Grampian.
The lead author, who is also the Data Manager for the Faculty of Medicine and Medical Sciences, University of Aberdeen, has an honorary position with NHS Grampian and liaises directly with the Director of Public Health for the administration of all research studies involving CHI. Under this arrangement, no patient-identifiable information is passed to researchers without prior written consent from potential subjects. For this particular study, only anonymised data was provided to other members of the research team. The questionnaire mailings were administered by the lead author from a confidential database.
The confidential database was used to generate letters from the Grampian Director of Public Health, inviting people to agree to receive either a postal or electronic version of a questionnaire about communicating their views on NHS issues for a research study by a consortium of Grampian Local Health Council, Health Services Research Unit, University of Aberdeen and NHS Scotland. An information leaflet and a consent form, including a section for the provision of an e-mail address, were enclosed with the invitation letter, together with a pre-paid reply envelope to be returned to the Director of Public Health.
Non-respondents at the consent stage are referred to in this manuscript as "Group A". No reminder letters were sent, in accordance with local confidentiality guidelines. People who gave consent but no e-mail address, are referred to as "Group B" and were sent a postal questionnaire with a pre-paid reply envelope to be returned to Grampian Local Health Council. Postal questionnaires were despatched on receipt of subject consent. The questionnaires were fairly short, consisting of 4 pages (16 main questions, most of which had multiple choice response options and some which included boxes for free text comments). People who consented and provided an e-mail address were allocated alternately to "Group C" (sent a postal questionnaire) or "Group D" (sent an e-mail to enable them to access the electronic questionnaire). The e-mail message contained a hyperlink to the electronic questionnaire, an access code and a unique study identifier that served as a username to allow on-line completion of the questionnaire. For the postal questionnaire groups, if no response was received within 3 weeks, reminder letters were sent out with a duplicate questionnaire and pre-paid envelope. Similarly, reminder e-mails were sent to non-respondents in the electronic questionnaire group.
The confidential study database was used to record mailing and receipt dates for consent forms, to administer the mailing of questionnaires and reminders, and to record mailing and receipt dates for these. Electronic questionnaire data was recorded on the web server together with a date and time stamp. These were downloaded weekly from the web server to update the database.
Response rates to the consent form were calculated after a period of 10 weeks had elapsed following the initial mailing, and to the subsequent postal and electronic questionnaires after 3 weeks (first questionnaire) and a further 7 weeks (reminder questionnaire) had elapsed. In the light of recent experience with similar research studies, 10 weeks for each stage of the study was considered sufficient for the vast majority of people to respond to the consent form and subsequent questionnaire, respectively. However, 4 responses to the consent form (0.04%) were received several weeks later and were excluded from the survey. Individual response times were calculated as the time elapsed in days between the mailing of the consent form or questionnaire and receipt of a reply.
Median response times and interquartile ranges (IQR) were calculated for return of consent forms and questionnaires.
Response rates by age, gender and postal area deprivation status were analysed for each stage of the study. Odds ratios (OR) were calculated, together with 95% confidence intervals (CI) using male, 1970–79 and no deprivation as reference groups. A 5% significance level was used throughout for statistical comparisons and the Chi-squared (χ2) test used to compare categorical data.
The random sample of 10,000 people reflected the demographic characteristics of the Grampian population (Figure 1). As for the sampling frame, 4% of the random sample were resident in deprived postal areas.
Figure 1. Percentage distribution by year of birth and gender for the random sample of 10,000 people compared with Grampian CHI population.
Of 10,000 invitation letters sent out, a total of 2449 (25%) were returned completed with consent within a ten-week period. A further five people returned uncompleted forms explicitly refusing consent. 27 letters were returned as undeliverable and 6 with an explanation that the intended recipient had died. A further 4 consent forms were returned after the initial 10 week period. These were not included in the study.
Non-response bias at the consent stage
The characteristics of non-response bias at the consent form stage are presented as odds ratios in Figure 2. Respondents at the consent form stage were more likely than non-respondents to be older, female and not living in areas of deprivation. The lowest response rates were among the oldest and youngest age groups. Overall, more females (29%) responded than males (21%) (OR = 1.5, 95% CI 1.4, 1.7), but in the older age groups (year of birth range 1910–1939), a higher proportion of males responded. A lower response rate (16%) was found for people living in postal areas of deprivation compared to 25% for people living outside areas of deprivation (OR = 0.59, 95% CI 0.45, 0.78). In general, the highest response rate to the consent form was found among females and individuals born in 1930–1939 (OR = 2.82, 95% CI 2.36, 3.37).
Figure 2. Odds ratios with 95% CI for response rates to consent form by gender, age and residential deprivation.
Analysis of response times for consent stage
50% of those who responded at the consent stage did so within 7 days and 97% within 4 weeks. There was no difference in response time by gender; both sexes had a median response time of 7 days (IQR 6, 11 days). However, there was a non-statistically significant trend for younger respondents taking slightly longer to reply than older respondents (IQR 7, 9 days and 8, 13 days, respectively).
1688 people returned a consent form without providing an e-mail address and were sent a postal questionnaire (group B). 761 people who returned a consent form and provided an e-mail address were alternately allocated to be sent a postal questionnaire (N = 381, group C) or sent access details for an electronic questionnaire (N = 380, group D) (Figure 3). All postal questionnaires and e-mails for accessing electronic questionnaires were sent out within 7 days of receiving a signed consent form.
Figure 3. Diagramatic summary of stages in study.
1452/2069 completed postal questionnaires were returned and 201/380 electronic questionnaires were completed within 3 weeks. Reminders were sent to non-respondents at this stage. A further 255 completed postal questionnaires and 43 completed electronic questionnaires were subsequently received, adding 12% to the overall questionnaire response rate. The final response rate for people who originally gave consent was 80% (1951/2449) – equivalent to 20% of the original sample.
Non-response bias for questionnaire
Characteristics of non-response bias to the questionnaire are presented as odds ratios in Figure 4. In contrast to the consent form stage, there was no significant difference between respondent and non-respondent groups in terms of their gender and postal area deprivation status distributions. However, as with the consent form stage, response rates at the questionnaire stage differed by age group, with higher response rates among older age groups.
Figure 4. Odds ratios with 95% CI for response rates to questionnaires by gender, age and residential deprivation.
Figure 5 provides odds ratios for an explicit comparison of respondents and non-respondents for the consent form and questionnaire combined. However, the odds ratios are influenced by the low response rate observed at the consent stage and consequently Figures 2 and 5 are very similar.
Figure 5. Odds ratios with 95% CI for overall response rates to questionnaires by gender, age and residential deprivation.
Analysis of response times for questionnaire stage
Figure 6 illustrates the responses times for the first mailing of questionnaires. Some people completed electronic questionnaires on the same day that the e-mails were despatched from the study office, giving a response time of zero days. For all questionnaire groups, 50% of respondents at the questionnaire stage had replied within 8 days and 97% within 4 weeks (IQR 7, 15 days), very similar to the response times obtained for the consent stage. Similar response time curves were obtained when reminder questionnaires were sent out.
Figure 6. Elapsed time between sending and receiving back initial postal and electronic questionnaires.
A high response rate is essential to ensure the representativeness and generalisability of the results of a sample survey . In addition, comparable response rates from all the different sub-groups within a sample is desirable . The overall response rate for this study was low (20%), primarily due to the low response rate at the consent stage (25%).
Differences between the demographic profiles of respondents and non-respondents in this study were more marked at the consent stage than at the questionnaire stage. Respondents at the consent stage were more likely to be older, female and not living in postal areas of deprivation.
For both consent and questionnaire stages, the majority of replies were received within a 4-week time interval. The use of reminders, which was only possible at the questionnaire stage, did appear to increase the overall response rate, which is consistent with previous studies .
Low response rates, such as that seen in this study, are not a regular feature of surveys conducted among the population under study. Previous surveys that have used the Grampian CHI as a sampling frame and sent questionnaires about health service topics directly (without a consent stage) have achieved response rates of 70–80% [5,6]. The questionnaire stage of this study achieved a response rate at the upper end of this range – although it was based on a self-selected subset of the original population sample.
Possible reasons for low response rates at the consent stage
Our local confidentiality protocol prevented us from sending follow-up reminders at the consent stage and from making any further contact with non-respondents about this study. We therefore could not systematically investigate whether non-respondents actively did not want to participate in this particular survey or whether there were other reasons for their non-response. Forgetfulness or inertia are obvious possibilities, but we can suggest other possible reasons for the low response rates obtained at the consent stage.
People might be less likely to return an initial consent form to receive a questionnaire than they would be to return a questionnaire sent to them directly because the sense of contributing might be less immediate when completing a consent form rather than a questionnaire, and by returning a consent form people are in some sense committing themselves to fill in an as-yet-unseen questionnaire at a later date.
Our particular consent form might have been off-putting to some because it asked for an e-mail address. Although provision of an e-mail address was voluntary, people who did not have one might have felt that the questionnaire was less relevant to them, and people who were concerned about the volume of unsolicited e-mail they received might have been put off from replying. If considerations relating to the request for an e-mail address were particularly influential on consent rates, they might account in part for the demographic differences between respondents and non-respondents.
We also received some unsolicited feedback from a few respondents to the effect that it was wasteful for us to send them two separate mailings (for consent and then for the questionnaire), and it is possible that some people declined to consent because they were concerned about this.
A number of other studies have also examined reasons for non-consent to surveys. For example, a multi-site health services research survey that explored participation rates at different locations in the United States, compared locations that used prior consent to locations in which direct contact was made with potential participants. It also found that requirements for prior written consent were associated with significantly lower participation rates (27% compared to 58%) .
Woolf et al.  carried out a study to compare the characteristics of patients who gave prior consent to complete an in-office survey, refused or did not answer. They found that older patients and those in poorer health were more likely to grant consent and concluded that studies restricted to patients who give consent may misrepresent outcomes for the general population.
It is also interesting to note that Cartwright et al. showed that lower response rates were obtained when a question asking whether respondents would be willing to participate in future research was included in a questionnaire . This is a similar scenario to our situation of requesting consent prior to subsequent survey.
While requesting prior consent to be sent a questionnaire seems to have a detrimental effect on overall responses rates, pre-notification that a questionnaire will be sent seems to increase response rates [1,14]. In effect, pre-notification seems to serve to increase the number of contacts with potential respondents, which has also been shown to increase response rates [1,14], without asking them to do anything extra. The requirement of prior consent, however, means that pre-notification or other additional contacts are precluded until consent has been obtained.
Among the other factors known to contribute to poor response rates are respondents' lack of interest in the questionnaire topic  and poor questionnaire design . The people who were invited to participate in this study were given a brief description of the survey topic (How do you want to let the NHS know your views?) but did not see the questionnaire at the consent stage, so the latter could not have directly influenced response rates at this stage. The 80% response rate among those who were sent questionnaires suggests these factors were not problematic in this study.
Response rates at the questionnaire stage
Reviews by Edwards et al.  and McColl et al.  have shown that pre-notification of a forth-coming questionnaire leads to improved response rates, as does increasing number of contacts with potential respondents. In our study, those people who gave consent to receive a questionnaire had effectively been pre-notified by the time they received it. 80% of those people who consented to receive a questionnaire completed and returned one. This is at the upper end of the range of response rates that have been obtained previously for surveys that used the same population sampling frame but had no prior consent stage. Various factors, including some of those mentioned above as possible reasons for non-response at the consent stage, might have moderated the usually positive effect of pre-notification, on response rates at the questionnaire stage. We cannot tell what the response rate would have been for this study if people had been sent the questionnaire directly.
The requirement for Caldicott Guardians to obtain consent from individuals before allowing researchers to contact them adds another level of complexity to the management of health-related studies. As our investigation has highlighted, the additional contact may increase the bias that is present in survey findings by significantly reducing overall response rates  and by reducing response rates disproportionately for some demographic groups. In addition, the consent stage requires that an additional contact be made with potential participants, incurring additional resources for mailing and administration, and extending the duration of the study. All of these add to the costs of carrying out surveys.
Concerns have already been expressed about the implications of requirements for prior patient consent in non-clinical health services research [19,20]. Our study data raises substantial concerns about the implications of prior consent and "opt-in" designs for survey response rates and bias. Respect for individual privacy and confidentiality is important, but, as yet, it is not clear that receipt of a letter asking people to opt in to a questionnaire survey is any less intrusive that direct receipt of a questionnaire (both of which can, in any event, be ignored by the recipient).
If the requirement for prior consent remains, future postal and electronic surveys are likely to require larger population samples to compensate for the associated detrimental impact on overall response rates. They may also need to over-sample at the initial stage those demographic groups that are particularly prone to be among non-respondents.
The full study was initiated by KW. All authors contributed to the development of the study design and protocol. VA coordinated the distribution of consent forms, questionnaires and reminders and designed the database to monitor response rates and record questionnaire responses. VA analysed the response characteristics and led the drafting of the paper supported by VE. All authors read and approved the final manuscript.
This project was funded by the Scottish Executive Health Department and run by a consortium of Grampian Local Health Council, the Health Services Research Unit and the Faculty of Medicine and Medical Sciences at the University of Aberdeen, NHS Education (North Scotland) and British Telecom Computing Partners.
Thanks are due to Marion Campbell for helpful comments on drafts of this paper and to Newland Andrews of BT Computing Partners for providing regular weekly downloads of respondent data for the electronic questionnaires.
We are also grateful to the BMC Health Services Research reviewers: Phil Edwards, Elaine McColl and Marie Pirotta for their helpful suggestions.
McColl E, Jacoby A, Thomas L, Soutter J, Bamford C, Steen N, Thomas R, Harvey E, Garratt A, Bond J: Design and use of questionnaires: A review of best practice applicable to surveys of health service staff and patients.
Parliamentary Act: [http://www.legislation.hmso.gov.uk/acts/acts1998/19980029.htm] webcite
Scottish Executive: [http://www.show.scot.nhs.uk/sehd/publications/ppcr/ppcr.pdf] webcite
Evaluation and Program Planning 2003, 26:91-7. Publisher Full Text
The pre-publication history for this paper can be accessed here: