Sampling and coverage issues of telephone surveys used for collecting health information in Australia: results from a face-to-face survey from 1999 to 2008

Abstract

Background

To examine the trend of "mobile only" households, and households that have a mobile phone or landline telephone listed in the telephone directory, and to describe these groups by various socio-demographic and health indicators.

Method

Representative face-to-face population health surveys of South Australians, aged 15 years and over, were conducted in 1999, 2004, 2006, 2007 and 2008 (n = 14285, response rates = 51.9% to 70.6%). Self-reported information on mobile phone ownership and usage (1999 to 2008) and listings in White Pages telephone directory (2006 to 2008), and landline telephone connection and listings in the White Pages (1999 to 2008), was provided by participants. Additional information was collected on self-reported health conditions and health-related risk behaviours.

Results

Mobile only households have been steadily increasing from 1.4% in 1999 to 8.7% in 2008. In terms of sampling frame for telephone surveys, 68.7% of South Australian households in 2008 had at least a mobile phone or landline telephone listed in the White Pages (73.8% in 2006; 71.5% in 2007). The proportion of mobile only households was highest among young people, unemployed, people who were separated, divorced or never married, low income households, low SES areas, rural areas, current smokers, current asthma or people in the normal weight range. The proportion with landlines or mobiles telephone numbers listed in the White Pages telephone directory was highest among older people, married or in a defacto relationship or widowed, low SES areas, rural areas, people classified as overweight, or those diagnosed with arthritis or osteoporosis.

Conclusion

The rate of mobile only households has been increasing in Australia and is following worldwide trends, but has not reached the high levels seen internationally (12% to 52%). In general, the impact of mobile telephones on current sampling frames (exclusion or non-listing of mobile only households or not listed in the White Pages directory) may have a low impact on health estimates obtained using telephone surveys. However, researchers need to be aware that mobile only households are distinctly different to households with a landline connection, and the increase in the number of mobile-only households is not uniform across all groups in the community. Listing in the White Pages directory continues to decrease and only a small proportion of mobile only households are listed. Researchers need to be aware of these telephone sampling issues when considering telephone surveys.

Background

With the rapid changes to the telecommunications industry, it is unknown whether telephone surveys can continue to be used to reliably collect representative information regarding health status and health risk behaviours. Telephone surveys, traditionally using landline telephones, have been used to collect and monitor health-related information over the last 30 years [1–3] and have been used to determine the prevalence of chronic conditions, health-related risk behaviours, and assess knowledge, attitudes, and opinions on health issues. Telephone surveys using Computer Assisted Telephone Interviewing (CATI) technology have been seen as a cost-effective and timely method of collecting health information [4–9] and have provided greater standardisation of administration through closer supervision of interviewers compared to traditional face-to-face methods.

In Australia, for the last decade, the coverage of households with a telephone connected (landline) and the adequacy of the sampling frame(s) have been a concern for those involved in epidemiologically-sound telephone surveys. The proportion of people who do not have a landline telephone connected in the household is not uniformly distributed in the population[10, 11] and there is difficulty in obtaining a complete sampling frame[12]. With the exception of the Northern Territory, the landline telephone coverage in Australian households has been historically very high (96.8% in South Australia in 2002[10], 94.4% in Australia in 1991[13]). There are two main population sampling methods used in Australia: Australian Electronic White Pages (EWP) directory and Random Digit Dialling (RDD). EWP consists of all landline telephone numbers, names and address details for a household or business. All these telephone numbers are centrally located and routinely listed in the EWP regardless of the telecommunication carrier. Households can opt, at a cost, to not have their telephone number have listed in the EWP (also known as silent numbers). Mobile numbers are not routinely listed in the EWP, so owners can choose, at a cost, to have their mobile number listed in the EWP. This exclusion of unlisted numbers from the sampling frame can have an effect on the estimates for the population in telephone surveys and remains an important concern for researchers [11, 12]. RDD methods in Australia are based on the prefixes of the telephone numbers in the EWP to generate a sampling frame in order to include the silent numbers. This method is referred to as list-assisted RDD (LA-RDD). However, these RDD sampling methods do not include mobile numbers in their sampling frame. The differences and similarities between these two methods, in the Australian context, have been described elsewhere [10–12, 14].

Since the early 2000 s, international trends have seen households change to new telecommunications technologies, whereby individuals in the household are solely contactable by mobile phones or other means such as Voice over Internet Protocol (VoIP), not by the traditional landline telephone. This has negatively impacted on telephone surveys and sampling methodologies. International studies have shown a dramatic increase in mobile only households (those not having a landline telephone connected to the household). In 2006, 11.8% of households in the United States [15], 52% in Finland [16] and 17% in France [16] were mobile only. In 2002, 13.1% of households in Italy were mobile only households [17].

As a result of these rapid changes both in Australia and worldwide, determining an adequate sampling frame to include these non-traditional telephone numbers and to demarcate geographic locations, is becoming increasingly important. This paper presents how two factors impact household telephone surveys in Australia; the presence of mobile only individuals and the lack of full enumeration of telephone numbers in a telephone directory. Mobile only households are not covered in the RDD sampling frame and unlisted telephone numbers (landlines and mobiles) are not covered in the EWP telephone directory. Both samplings frames exclude people with no mobiles or landline telephones. The aim of this study is to examine the trend of mobile only households and households that have a mobile telephone or landline telephone listing in the EWP, and to describe these groups by various socio-demographics and health indicators to determine the potential bias due to non-coverage in the sample in the Australian context.

Methods

Survey design and sample section

Questions regarding landline telephone, mobile, and internet connections were included in the 1999, 2004, 2006, 2007 and 2008 South Australian Health Omnibus Surveys (HOS)[18, 19]. HOS is a multi-stage, systematic, clustered area sample of households conducted face-to-face annually in spring based on the Australian Bureau of Statistics (ABS) collector districts (CDs). The HOS samples included households randomly selected from CDs, from the metropolitan Adelaide area and country towns with a population of 1,000 people or more. Within each CD, a random starting point was selected and from this point 10 households were then selected in a given direction with a fixed skip interval. Hotels, motels, hospitals, hostels and other institutions were excluded from the sample. Approach letters were sent to selected households informing them of the survey. One person aged 15 years or over, who was last to have a birthday, was randomly selected from each household for interview. The interviews were conducted in people's homes by trained interviewers. Up to six call back visits were made to the chosen households to interview the selected person. There was no replacement for non-respondents. Response rates (and sample size) from the surveys were 70.6% (n = 3012) in 1999, 68.4% (n = 2982) in 2004, 54.9% (n = 2969) in 2006, 51.9% (n = 2507) in 2007 and 53.6% (n = 2824) in 2008. Each individual data set was weighted by five year age groups, sex, area (metropolitan Adelaide and SA country) and household size to the most recent ABS Census or Estimated Residential Population for South Australia to provide population estimates.

Data items

Questions about landline telephone connection and listing in the EWP (alphabetic directory of non-silent phone numbers belonging to residential households and businesses which include surname and address details) were included in the 1999, 2004, 2006, 2007 and 2008 surveys. These surveys also included questions on mobile phone ownership and usage in the household. Additional questions on mobile phone listing in the EWP, and future landline and mobile phone ownership plans were included in 2006, 2007 and 2008.

Identical demographic variables included in each survey were age, sex, area of residence, country of birth, education level, marital status, gross annual household income, and work status. The Index of Relative Social Disadvantage (IRSD) developed by the ABS was also calculated to identify the geographical areas that were relatively disadvantaged[20]. The IRSD is a composite measure based on selected Census variables such as income, educational attainment and employment status. The IRSD scores were grouped into quintiles for analysis where the highest quintile comprises postcodes with the highest IRSD scores (most advantaged areas).

Chronic conditions included medically confirmed diabetes, current asthma, arthritis and osteoporosis. Self-reported health risk factor data included smoking status and body mass index (BMI) which was derived from self-reported weight and height and recoded into four categories (underweight, normal weight, overweight and obese) [21]. Mobile only households were defined as households with no landline telephone connected to the house and if mobile phones were currently being used by members of the household.

The questionnaire and methodology for these surveys were approved by the Human Research Ethics Committee of the South Australian Department of Health (SA Health).

Data analyses

Data analysis was conducted using SPSS for Windows Version 15.0. The conventional p value of 0.05 was used as the criterion for statistical significance. To compare prevalence over time (from 1999 to 2008), χ² test for trend was used for mobile only households, households with landline telephone only connections and households with both a landline telephone connection and mobile. A comparison between 2006, 2007 and 2008 using χ² tests for trend was undertaken to assess for change in various socio-demographic and health-related variables between mobile only households and households with a landline or a mobile telephone number listed in the EWP.

Separate univariate analyses using χ² tests were undertaken for 2006, 2007 and 2008, to assess mobile only households, and households with a landline or a mobile telephone number listed in the EWP on a range of socio-demographic and health-related variables.

Results

Table 1 shows the trends of mobile only and landline households from 1999, 2004, 2006 to 2008. The proportion of mobile only households has been steadily increasing over the eight years (χ² _trend = 177.01, p < 0.001). There was a statistically significant decline in the proportion of households with landline telephone only connections (χ² _trend = 1693.6, p < 0.001) from 44.6% in 1999 to 6.9% in 2008, and a statistically significant increase in households with both a landline telephone connection and mobile (χ² _trend = 1188.6, p < 0.001) from 52.7% in 1999 to 83.8% in 2008.

Table 1 Telephone (landline) and mobile status of household, and household has landline and/or mobile telephone listed in the Australian Electronic White Pages by year

Respondents were asked if their mobile or landline telephones were listed in the EWP. Of mobile only households, 6.9% had their mobile number listed in the EWP in 2008 (8.0% in 2006; 3.4% in 2007). Of households with both a mobile and landline telephone connected, 7.4% had their mobile number listed in 2008 (7.3% in both 2006 and 2007). Examination of households with a landline telephone connection revealed that 77.0% of these households in 2006, 76.3% in 2007 and 74.1% in 2008, had their landline telephone numbers listed in the EWP (χ² _trend = 18.6, p < 0.001). Hence, 68.7% of South Australian households in 2008 had at least a mobile phone and/or landline telephone listed in the EWP (73.8% in 2006, 71.5% in 2007) (Table 1).

When the proportion of mobile only household respondents was compared on selected demographics and health indicators over the three years (2006 to 2008) (Table 2), increased trends were significant for a wide range of variables. When households with a mobile or landline telephone number listed in the EWP (Table 3) were examined over the three years (2006 to 2008), decreasing trends were apparent for a range of variables.

Table 2 Proportion of people living in mobile only households by selected demographic, health conditions, and health related risk factors from 2006 to 2008

Table 3 Proportion of people living in households where mobile phone or landline telephone is listed in the White Pages by selected demographic, health conditions, and health related risk factors from 2006 to 2008

When examined by selected demographics for 2006, 2007 and 2008 (Table 2), respondents who lived in a mobile only household were statistically significantly more likely to be in the younger age groups, separated, divorced or never married, unemployed, born in Australia, have at least obtained secondary schooling, living in rural areas of South Australia, from low income households or from low SES areas of South Australia, and statistically significantly less likely to be in the older age groups, widowed, married or in a defacto relationship, born in UK/Ireland, living in metropolitan Adelaide, from high income households ($80,000 or more per annum), retired, to have a bachelor degree or higher and from high SES areas of South Australia. In terms of health conditions and health related risk factors, respondents who live in mobile only households were statistically significantly more likely to be current smokers, classified as having normal BMI or diagnosed with current asthma, and were statistically significantly less likely to be classified as overweight.

Respondents from households with a mobile or landline telephone number listed in the EWP (Table 3) were statistically significantly more likely to be in the older age groups, married or in a defacto relationship or widowed, retired, living in rural areas and from high SES areas of South Australia, and statistically significantly less likely to be in the younger age groups, never married, separated or divorced, a student, unemployed or not working because of an injury, living in metropolitan Adelaide or from low SES areas of South Australia. They were also statistically significantly more likely to be classified as being overweight, to have arthritis, or statistically significantly less likely to be current smokers and have normal BMI.

Further questions were asked to determine the likelihood of people with a landline telephone switching to being a mobile only household. Overall, 6.9% in 2006, 5.9% in 2007, and 8.1% in 2008 indicated that they were 'very likely' while 11.0%, 10.8%, and 12.1% respectively indicated they were 'somewhat likely' to discontinue their landline connection.

Discussion

This study has shown, using large representative surveys, the proportion of mobile only households has been increasing in Australia and is following international trends. The prevalence of mobile only households in South Australia among people aged 15 years and over (8.7% in 2008), is not as high as other international studies: 11.8% in the United States[15]; 52% in Finland[16] and 17% in France in 2006[16] and, 13.1% in Italy in 2002[17]. However, the pattern of increasing prevalence remains the same and there are also changes among a range of demographic, health status and health risk behaviours groups. The prevalence of households with neither a mobile phone nor landline telephone has remained low and is likely to have a minimal effect on surveys using mobile phone or landline telephones. However, the mobile only prevalence may increase in South Australia over the next few years since 8% of survey respondents indicated they were very likely to become a mobile only household.

From this study, using LA-RDD methodology to generate a sampling frame to include unlisted landline telephone numbers excludes mobile only households as well as households with no landline telephone connection which is 9% of the population. This could be considered small [22] and one could argue that excluding this group would have minimal impact on health estimates. However, the results presented in this study indicate that mobile only households have different demographic characteristics to households with landline and/or mobiles. These demographic differences are similar to US studies [15, 23] with a higher proportion of males, younger people, people who are unemployed, separated, divorced or never married, people living in rural areas of South Australia, and low SES households (low income households and reside in the most disadvantaged areas) living in mobile only households. From this study, in terms of health indicators, people classified as overweight, having current asthma and current smokers would also be under-represented in these surveys.

There are some data quality and collection issues that need to be taken into account when including mobile telephones into the sample frame. One is the location or the situation of the respondent at the time of the interview: respondents may choose not to answer a call to save battery life; answering a call which may incur a cost to both the respondent and the researcher (if the respondent is overseas the fee may be much higher depending on distance from Australia and contractual agreement with individual telecommunication providers); and safety and legal issues, eg the respondent may be driving and using their mobile (texting and talking) at the same time which is illegal in Australia [16]. A study conducted in the US [24] found that those respondents who participated in the survey using a mobile phone, 56% were at home while undertaking the survey, 14% were driving and 13% were at work. The remaining respondents were at other locations such as in public areas, in another person's home, in a car but not driving or on holidays. Another issue found in this US study [24] was the higher proportion of calls to mobile phones resulting in ineligible respondents due to age (people excluded if less than 18 years of age), a lower response rate than calls to landline telephones and a higher refusal rate.

Furthermore, the selection of the respondent differs between mobile and landline telephones. The mobile telephone is usually individually owned and accessed by that one individual most of the time, compared to landline telephones that belong to a household which may be accessed by one or more people. Hence, consideration needs to be given when sampling strategies in terms of randomly selecting a single person to interview versus a number of eligible people in a household [16].

This study has highlighted the need to acquire a representative sampling frame and sampling methodology for household telephone (landline) surveys that minimises selection bias and is efficient in terms of administration and cost. With landline telephone numbers, the majority of the telephone numbers are listed in the EWP and the prefix of the telephone numbers are geographically based. Mobile telephones are the opposite; they are rarely listed (7.3% of mobile telephone users found in this study) and the number structure does not provide any details of geographical location, hence making it difficult to generate a sampling frame similar to current cost effective RDD methods. The large proportional difference in the EWP directory listing between landline and mobile telephone numbers would be mainly due to the options provided to the owners: owners of landline telephone need to pay to have their telephone numbers not listed in the EWP, and owners of mobile telephone need to pay to have their mobile telephone numbers in the EWP. Hence EWP samples are likely to continue to have a small proportion (6.9% in 2008) of mobile only households in the sampling frame. According to these results, if the option is to sample from the EWP, approximately 30% of the population will be excluded, particularly young people, those who have never married, those who reside in rural areas, people on lower income levels, the unemployed and students. In terms of health indicators, people in the normal weight range and current smokers could be under-represented.

Another emerging technology that has not been examined in this study is VoIP (Voice over Internet Protocol). In Australia, the impact of VoIP on sampling frames is not known. VoIP is seen as a cost effective system that utilises broadband data lines. Similar to mobile phones, the structure of VoIP telephone numbers (or also known as virtual number) are not geographically based and owners have the option of listing their VoIP telephone number in the EWP directory. More research is required on the uptake of VoIP including usage and impact on sampling frames.

The results of this study are potentially biased due to survey non-response. The response rates from these surveys (51.9% to 70.6%) could be considered moderately acceptable for a population survey of this kind. With increasingly inaccessible buildings (eg locked gates), busy lifestyles, and security and privacy concerns, an ongoing impact on response rates is expected, following patterns and trends interstate and overseas [25]. The unweighted age distribution had a higher proportion of older people and a lower proportion of younger people. This indicates the proportion of mobile only households could be under-estimated, and listings in the EWP over-estimated. Another limitation is the self-reported nature of this study. People might not want to divulge that they have a landline or mobile phone that is listed in the EWP because they want to avoid telephone calls from telemarketers or researchers [22] resulting in an under-estimation of telephone listings.

What does this mean for telephone (landline) surveys? Researchers need to be aware of the rapid changes in the telecommunication industry that potentially have an impact on collecting representative and reliable data on health-related issues using household telephone (landline) surveys. Studies like this are important because of the increasing need to monitor public health issues in a timely manner in an environment with limited and sometimes conflicting resources. Within these limits, there is a need to determine valid and reliable methods to verify the health estimates used for policy, planning of resources, and evaluation of health promotion interventions. Further research is needed in the area of mobile telephones such as how often the mobile is turned on, whether telephone calls are made more on the mobile or landline, and the likelihood of completing a health survey on a mobile telephone. Further Australian research is also required in terms of different weighting or post-survey adjustment strategies (eg raking) [26], improved sampling strategies [27] and the advantages and disadvantages of mixed mode surveying [28] (such as telephone, face-to-face, mail or internet), in order to improve the coverage of the sampling frame and minimise bias.

Conclusion

Coverage of households with a telephone connected (landline) and the adequacy of the sampling frame(s) have been a concern for those involved in epidemiologically-sound telephone surveys. The rate of mobile only households in South Australia has been increasing and is following worldwide trends but has not reached the high levels seen internationally (12% to 52%). Presently, the impact of mobile telephones on current sampling frames (exclusion of mobile only households or non-listings in the White Pages directory) may be small in relation to the health estimates obtained using telephone surveys. However, researchers need to be aware that mobile only households have distinctly different characteristics compared to households with a landline connection and the increase in the number of mobile-only households is not uniform across all groups in the community. Listing in the White Pages directory is continuing to decrease and only a small proportion of mobile only households are listed. Researchers need to be aware of these telephone sampling issues when considering telephone surveys.