The validity of self-reported cancer screening history and the role of social disadvantage in Ontario, Canada

Abstract

Background

Self-report may not be an accurate method of determining cervical, breast and colorectal cancer screening rates due to recall, acquiescence and social desirability biases, particularly for certain sociodemographic groups. Therefore, the aims of this study were to determine the validity of self-report of cancer screening in Ontario, Canada, both for people in the general population and for socially disadvantaged groups based on immigrant status, ethnicity, education, income, language ability, self-rated health, employment status, age category (for cervical cancer screening), and gender (for fecal occult blood testing).

Methods

We linked multiple data sources for this study, including the Canadian Community Health Survey and provincial-level health databases. Using administrative data as our gold standard, we calculated validity measures for self-report (i.e. sensitivity, specificity, positive and negative likelihood ratios, positive and negative predictive values), calculated report-to-record ratios, and conducted a multivariable regression analysis to determine which characteristics were independently associated with over-reporting of screening.

Results

Specificity was less than 70% overall and for all subgroups for cervical and breast cancer screening, and sensitivity was lower than 80% overall and for all subgroups for fecal occult blood testing FOBT. Report-to-record ratios were persistently significantly greater than 1 across all cancer screening types, highest for the FOBT group: 1.246 [1.189-1.306]. Regression analyses showed no consistent patterns, but sociodemographic characteristics were associated with over-reporting for each screening type.

Conclusions

We have found that in Ontario, as in other jurisdictions, there is a pervasive tendency for people to over-report their cancer screening histories. Sociodemographic status also appears to influence over-reporting. Public health practitioners and policymakers need to be aware of the limitations of self-report and adjust their methods and interpretations accordingly.

Background

Screening for cervical, breast and colorectal cancer (CRC) using the Papanicolaou (Pap) test, mammography, and the fecal occult blood test (FOBT) respectively are commonly accepted practices in primary care in Canada. Because of the effectiveness of the Pap test and its widespread use, Canada currently has one of the world’s lowest annual incidence and mortality rates for invasive cervical cancer [1-3]. Mammography is associated with significant reductions in the relative risk of death from breast cancer and has long been recommended by the Canadian Task Force on Preventive Health Care [4]. Colorectal screening rates in Canada are currently low, but it has been estimated that if 70% of the eligible population participated in screening, mortality due to CRC could drop by 13-15% [5].

Despite the known benefits of screening, the literature suggests that people who are members of certain sociodemographic groups are subject to cancer screening inequities, particularly ethnoracial minorities, immigrants, people with low levels of education, people with complex medical conditions, and those of low income [6-12]. The benefits of screening combined with the apparent inequalities in screening based on social disadvantage demonstrate the need for valid methods of determining and monitoring screening rates. It is therefore concerning to consider that self-report, a commonly used method for determining screening history, may have validity issues and that people from certain sociodemographic groups might be more likely to inaccurately report cancer screening than their peers [13-16]. Under-estimating screening prevalence or over-estimating screening inequalities could lead to wasted resources, and conversely, over-estimating screening prevalence or under-estimating screening inequalities could lead to missed opportunities for improving screening. Self-report is potentially vulnerable to acquiescence bias (the tendency to give positive responses to questions when in doubt) and social desirability bias (the tendency to respond in a manner that respondents believe will be viewed favourably), and both types of bias may be more common among socially disadvantaged groups [14,16,17].

Literature from the US suggests that these biases in self-report might exist differentially, and that Hispanics and African-Americans may be more likely to over-report screening [14,15]. However, little is known about the validity of self-report of cancer screening in Canadian populations or among particular Canadian sociodemographic groups. Therefore, the objectives of this study were: i) to determine the validity of self-report of cervical, breast and CRC screening in Ontario for people in the general population, and ii) to determine the validity of self-report for socially disadvantaged groups based on: immigrant status, ethnicity, education, income, language ability, self-reported health, employment status, age category (for cervical cancer screening), and gender (for CRC screening).

Methods

Data sources

Multiple data sources were used for this study. The Canadian Community Health Survey (CCHS) is a national, biennial, cross-sectional survey conducted by Statistics Canada that collects information related to health status and health care utilization, including questions on cervical, breast, and CRC screening, as well as demographic characteristics such as visible minority status, immigrant status, language ability, education, age and gender. The CCHS data to which we had access contain responses from approximately 30,000 Ontarians (approximately 60% of Ontario respondents) per cycle who agreed to data linkage with administrative health data and whose data were able to be linked. We used data from four cycles of CCHS data (2000–2001, 2003, 2005, 2007). The Ontario Health Insurance Plan (OHIP) Claims’ Database contains claims for physician and hospital services as part of the provincial universal health insurance plan and includes approximately 95% of physician claims in the province [18]. The Canadian Institute of Health Information Discharge Abstract Database consists of fee codes and diagnostic codes claimed by Ontario physicians for hospital stays and procedures. Cytobase is Ontario’s electronic Pap test registry. Finally, the Ontario Breast Screening Program data record the date of mammogram for all women who participate in the provincial screening program. Using a unique encoded identifying number that allows individuals to be tracked through multiple databases, we linked respondents from the four cycles of CCHS with the aforementioned health databases. This study was approved by the Research Ethics Board of Sunnybrook Health Sciences Centre.

Inclusion/exclusion criteria

We created three groups in this study. Survey respondents were included if they had answered the questions about cancer screening and were eligible for cervical cancer screening (cervix group), breast cancer screening (breast group), or fecal occult blood testing (FOBT group) respectively. Cervical cancer screening eligibility consisted of being 24 to 69 years of age on the date that they completed the CCHS, being continuously eligible for health insurance coverage for the three years prior to the CCHS interview, no history of hysterectomy, and no previous diagnosis of cervical cancer. Similarly, breast cancer screening eligibility consisted of being 52 to 69 years of age on the date that they completed the CCHS, being continuously eligible for Ontario’s universal health insurance coverage for the two years prior to the CCHS interview, and no previous diagnosis of breast cancer. Women were excluded if they reported on the CCHS that they had a mammogram for diagnostic (as opposed to screening) purposes. Eligibility for screening with FOBT was defined as being 52 to 74 years of age on the date of the CCHS, being continuously eligible for health insurance coverage for the two years prior to the CCHS interview, no history of barium enema or sigmoidoscopy in the preceding five years, no history of colonoscopy in the preceding ten years, and no previous diagnosis of CRC. People were excluded if they reported on the CCHS that they had an FOBT as follow-up of a problem or follow-up of CRC treatment. Ages and time periods for each cancer screening type were determined by provincial guidelines [19-21].

Validity measures

We calculated validity measures i.e. sensitivity, specificity, positive and negative likelihood ratios, positive and negative predictive values, and simple kappa statistics for each cancer screening type. A kappa of 0.6 to 0.79 indicates substantial agreement, whereas a kappa of 0.4 to 0.59 indicates only moderate agreement [22]. Specifically, we assessed if those eligible self-reported or had a record of a Pap test within the three years prior to the CCHS, a mammogram within two years, and FOBT within the two years prior to the CCHS. A record of a test in administrative data was viewed as the gold standard against which self-report was validated. Only the latter three cycles of the CCHS asked respondents about history of FOBT.

We then calculated these validity measures for various demographic subgroups, specifically by self-perceived health (excellent, very good, good, fair, poor), household income ($30,000 or less, more than $30,000), highest education obtained (less than high school, high school or higher), mother tongue (English, non-English), immigrant status (Canadian-born, foreign-born), recent immigrant status among the foreign-born (immigrated 5 years or less from the date of interview vs. immigrated more than 5 years prior), employment status (currently employed vs. not), racial group (White, non-White), age category for the cervix group (24–52 years vs. 53–69 years), and gender for the FOBT group.

We calculated report-to-record ratios (the ratio of the proportion of women who reported a screening test during the relevant time period to the proportion of women who had a record of a screening test during that time period) and their confidence intervals for each screening group and for each subgroup within each screening group [23].

Finally, we conducted a multivariable regression analysis. We limited each screening group to those members who had no screening test during the appropriate time frame in administrative records, and then determined the adjusted relative risk of over-reporting (i.e. falsely reporting a screening test during that time frame), using the variables described above, with the exception of recency of immigration.

Results

Table 1 describes the demographics of the three screening groups. All three groups were predominantly White, Canadian-born, and at least high school graduates. The majority of women (68.2%) in the cervix group were in the younger age category. In the FOBT group, 53.3% of men were employed whereas 41.5% of women were employed.

Table 1 Demographic characteristics of the 39027 women in the cervical cancer screening group, 15877 women in the breast cancer screening group, and 14297 people in the FOBT screening group

The measures of validity of self-report for the three groups are reported in Table 2. Sensitivity was overall quite high for the cervix (96.5% [95% confidence interval 96.3-96.7%]) and breast (96.6% [96.3-97.0%]) groups, but was noticeably lower for the FOBT group (77.4% [75.7-79.0%]). Specificity was lowest for the cervix group (49.5% [48.6-50.5%]) and highest for the FOBT group (89.8% [89.2-90.3%]). The FOBT group had the lowest overall positive predictive value (62.1% [60.4-63.8%] and highest overall negative predictive value (94.8% [94.4-95.2%]. The highest kappa statistic value was for women in the breast group who reported excellent self-perceived health (0.677) and the lowest was for women in the cervix group who were foreign-born (0.505).

Table 2 Measures of validity of self-report for cervical cancer screening, breast cancer screening, and fecal occult blood testing, using administrative data as the gold standard

Figures 1, 2 and 3 describe the report-to-record ratios for each screening group and demographic subgroup. Ratios were persistently significantly greater than 1, suggesting pervasive over-reporting, except for more recent immigrants where confidence intervals crossed unity for each screening type. For the cervix group (Figure 1), women who were older, less educated, and of fair or poor self-perceived health had report-to-record ratios that were higher than the overall estimate, and women who were younger, foreign-born, recent immigrants, non-White, and non-English speaking had ratios lower than the overall estimate. For the breast group (Figure 2), women of poor self-perceived health had a report-to-record ratio higher than the range of the overall estimate. For the FOBT group (Figure 3), no group had a report-to-record ratio outside of the range of the overall estimate. The FOBT group (Figure 3) had the highest overall report-to-record ratio (1.246 [1.189-1.306]) of the three groups.

Figure 1

Report-to-record ratios for cervical cancer screening.

Figure 2

Report-to-record ratios for breast cancer screening.

Figure 3

Report-to-record ratios for fecal occult blood testing.

In multivariable regression analyses, women who were younger, White, higher educated, English native speakers, and of higher income were less likely to over-report cervical cancer screening. Higher income women were less likely to over-report breast cancer screening. Men were less likely to over-report FOBT use, whereas employed people were more likely to over-report FOBT use (adjusted relative risk 1.18 [95% confidence interval 1.03-1.36] (Table 3)).

Table 3 Adjusted relative risks [with 95% confidence intervals] of over-reporting, defined as reporting a Pap test in the past 3 years (cervix group), a mammogram in the past 2 years (breast group), and a fecal occult blood test in the past 2 years (FOBT group), but with no record of same in administrative data

Discussion

In this study of the validity of self-report of up-to-date cancer screening among Ontarians, using administrative health data as our gold standard, we found that there is cause for concern when using self-report as the sole method of quantifying cancer screening rates. Specificity was less than 70% overall and for all subgroups for cervical and breast cancer screening and sensitivity was lower than 80% overall and for all subgroups for screening with FOBT. Report-to-record ratios were pervasively significantly greater than 1, indicating that over-reporting of cancer screening is common in Ontario.

Self-report of FOBT stood out among the three cancer screening types for having the highest report-to-record ratios, the lowest sensitivity and positive predictive values, and the highest specificity and negative predictive values. Taken together, these results suggest that false positives and true negatives are both quite high for FOBT use, with true negatives predominating. The high false positive rate implies that estimates of FOBT utilization based on self-report could be particularly erroneous and should be avoided where possible, and our finding of high true negatives is in line with the known low prevalence of CRC screening by FOBT in Ontario [24].

There was no clear pattern suggesting that Ontarians with social disadvantage were consistently more likely to over-report than their more advantaged peers. For example, in regression analyses, women who were White, higher educated, English native speakers and of higher income were less likely to over-report cervical cancer screening in Ontario than their respective counterparts, but this did not hold true for the other screening types. The higher sample size in the cervix group may have driven the higher number of significant results for cervical cancer screening than for the other two forms of screening. However, we have demonstrated that sociodemographic characteristics do appear to play a role in Ontario populations and should be considered when examining cancer screening gaps based on self-reported data.

Report-to-record ratios indicated that people of poorer self-rated health tended to be more likely to over-report up-to-date screening than those of better self-rated health for both cervical and breast cancer screening. The reason for this finding is unclear, but it has been reported elsewhere in the literature [25]. It may reflect more frequent contact with the health care system for people of poorer self-rated health, and therefore a more difficult time differentiating which medical tests and procedures one has received. Distinct event are easier for patients to remember, and screening may be less likely to be a distinct event for patients of worse health status [26].

Our findings have implications for public health researchers, practitioners and policymakers who are interested in cancer screening in Ontario. It is apparent that self-report of cancer screening is not the most accurate way to determine cervical, breast and colorectal cancer screening rates in Ontario, and will over-estimate screening rates. Where possible, the use of more objective data should be encouraged. Where not possible, the pervasive over-reporting that we demonstrated should be remembered and considered. The results we have presented could be used in the Ontario or Canadian setting as correction factors to determine more accurate screening rates [27]. Also, our findings suggest that there are opportunities to increase patients’ basic knowledge about screening, and their understanding of which screening tests they have and have not received. Many women mistake any pelvic examination for a Pap test [28-30], and it is feasible that patients may similarly mistake digital rectal exams for FOBT screening [31]. Better provider-patient communication and patient education may improve the accuracy of self-report, and may be of particular importance for disadvantaged populations [14,28].

Few Canadian studies have examined the validity of self-report of cancer screening for cervical, breast, and colorectal cancer. In their 2009 study, Wang et al. demonstrated that rates in Ontario administrative data were consistently lower for breast and cervical cancer screening and for influenza vaccination than rates in self-reported CCHS data [32]. Khoja et al. compared self-report of any form of CRC screening to medical records from physicians’ offices and found a kappa of 0.66. They reported that it was common for study participants to believe a test had occurred more recently than records showed [33]. Fehringer et al. compared cervical cancer screening rates in Cytobase, Ontario’s Pap test registry, with self-reported CCHS rates across 37 public health units in the province and found that Cytobase rates were consistently lower, concluding this was likely due to over-reporting [34]. Walker et al. found that women at high familial risk of breast cancer accurately reported mammogram use within the previous 12 months [35]; however, Larouche et al. assessed validity of self-reported mammography among women who had undergone breast cancer-specific genetic testing and concluded that self-report overestimates mammography use in this population [36].

This study has several limitations. First, administrative data are not a perfect gold standard [13,32]. For example, Pap tests performed and interpreted solely within hospitals in Ontario may not be captured by either administrative data or by Cytobase. However, the number of screening Pap tests to which this would apply is expected to be quite small. As well, we used a variety of databases (OHIP, OBSP, Cytobase) to find objective records of screening. Second, the CCHS is voluntary. Therefore, CCHS respondents, particularly those who agreed to have their survey answers linked to administrative health data, may systematically differ from the general population. However, they are likely to be representative of any sample who would agree to self-report cancer screening. Third, the CCHS has some missing data, particularly around income levels. Although these missing data affect the power of our results, they are unlikely to affect the veracity of our findings. Finally, we dichotomized many of the characteristics that we examined, such as race, recent immigrant status, and education, which may not have provided enough detail to uncover differences. For example, ethnicity has been reported as a factor affecting self-report in the US literature, with African-Americans and Hispanics highlighted [14,15]. Although we did examine racial group in this study, we only looked at White vs. non-White. However, sample size often would not allow for more detailed categorization.

Conclusions

In this work, we have found that in Ontario as in other jurisdictions, there is a pervasive tendency for people to over-report their cancer screening histories. Sociodemographic status also appears to influence the likelihood of over-reporting screening. Public health practitioners and policymakers need to be aware of the limitations of self-report and adjust their methods and interpretations accordingly.