Skip to main content
Download PDF

Missingness mechanisms and generalizability of patient reported outcome measures in colorectal cancer survivors – assessing the reasonableness of the “missing completely at random” assumption

BMC Medical Research Methodology volume 24, Article number: 104 (2024) Cite this article

Abstract

Background

Patient-Reported Outcome Measures (PROM) provide important information, however, missing PROM data threaten the interpretability and generalizability of findings by introducing potential bias. This study aims to provide insight into missingness mechanisms and inform future researchers on generalizability and possible methodological solutions to overcome missing PROM data problems during data collection and statistical analyses.

Methods

We identified 10,236 colorectal cancer survivors (CRCs) above 18y, diagnosed between 2014 and 2018 through the Danish Clinical Registries. We invited a random 20% (2,097) to participate in a national survey in May 2023. We distributed reminder e-mails at day 10 and day 20, and compared Initial Responders (response day 0–9), Subsequent Responders (response day 10–28) and Non-responders (no response after 28 days) in demographic and cancer-related characteristics and PROM-scores using linear regression.

Results

Of the 2,097 CRCs, 1,188 responded (57%). Of these, 142 (7%) were excluded leaving 1,955 eligible CRCs. 628 (32%) were categorized as initial responders, 418 (21%) as subsequent responders, and 909 (47%) as non-responders. Differences in demographic and cancer-related characteristics between the three groups were minor and PROM-scores only marginally differed between initial and subsequent responders.

Conclusion

In this study of long-term colorectal cancer survivors, we showed that initial responders, subsequent responders, and non-responders exhibit comparable demographic and cancer-related characteristics. Among respondents, Patient-Reported Outcome Measures were also similar, indicating generalizability. Assuming Patient-Reported Outcome Measures of subsequent responders represent answers by the non-responders (would they be available), it may be reasonable to judge the missingness mechanism as Missing Completely At Random.

Peer Review reports

Background

Due to advancements in medical science, increasing numbers of people are living with and after colorectal cancer worldwide [1]. The value of new treatments is no longer isolated to how long you survive, but also how well you survive [2, 3]. Quality of life measures are necessary to include as primary or at least secondary outcomes in interventional research [4,5,6].

Patient Reported Outcomes (PROs) [7, 8] have gained attention in healthcare as a way to gather information directly from patients about their health conditions. Patient-Reported Outcome Measures (PROMs) are specific tools used to assess PROs, often through self-completed questionnaires [8]. PROMs provide valuable insights into the patient’s perspective on treatment-related issues, functional abilities, and quality of life [9]. However, the nature of PROMs introduces certain challenges related to data collection [10], as they rely on patient willingness and ability to respond, hence the response can be affected by various factors such as illness severity and study design [11].

Missing data is a frequent problem in studies using PROMs [12, 13], and is often adressed by conducting complete-case analyses and ignoring the missing data [14]. However, this method may lead to biased results, reduced statistical power to detect differences between treatments, limited generalizability and misleading conclusions [13, 15,16,17,18,19].

In studies with missing data (> 5%) possible statistical solutions depend on the missingness mechanisms [16]. There are three missingness mechanisms: missing at random (MAR), missing not at random (MNAR) and missing completely at random (MCAR) [15, 16, 20]. If the missingness mechanism and hence the probability of response depend on the observed data then data is said to be MAR. In case of MAR, missing data may be addressed using i.e. multiple imputation to predict the missing values based on observed data. If the probability of response depends on both observed data and missing data then data is said to be MNAR, and may be adressed by conducting sensitivity analyses with best-and-worst case scenarios. If the probability of response is not associated with either the observed or missing values, data is said to be MCAR - and responders are considered representative of non-responders and complete-case analyses may not cause bias, but only enlarged standard errors due to the reduced statistical power.

In a study using PROM scores as an outcome then MCAR means the PROM value may not differ between responders and non-responders. Likewise, responders and non-responders may not differ with regard to other data observed. The MCAR assumption is hard to prove given that the PROMs are actually missing, however using a dataset collected with repeating reminders of the patients to respond may shed light upon the nature of missing PROMs. Comparing responders, who respond following the first invitation (hereafter denoted “Initial responders”) to the responders who were actually non-responders until 2nd or 3rd invitation but replyed subsequently (hereafter denoted “Subsequent responders”) creates an opportunity to evaluate the PROMs and characterize the patients who were non-responders at first, and to assess if the MCAR assumption may be reasonable. Comparing registry data between responders and non-responders provides insights into the generalizability of the respondents.

Acknowledging the issues with PROMs, including handling of missing data and related statistical approaches, we aimed to investigate the generalizability of respondents in a random sample of 20% of our registry-based national cohort of > 10,000 long-term colorectal cancer survivors (CRC) before continuing with a large cross-sectional study [21] on prevalence of symptoms indicative of psycho-oncological late effects and quality of life.

We assessed missingness mechanisms of PROM-data among initial responders and subsequent responders. We assumed, that PROM-data from subsequent responders may be indicative of PROM-data from non-responders, had the subsequent responder not been prompted with several reminders, and hence give insight into the missing data of the non-responders. Furthermore, we addressed the generalizability of all responders compared to non-responders by comparing clinical and demographic characteristics extracted from national registries. The study will provide insight into the missingness mechanisms in the setting of a nationwide cross-sectional study and add to the literature on issues with generalizability and possible methodological solutions to overcome missing data problems during statistical analyses. By addressing this issue associated with PROMs, researchers can better understand how to collect, interpret and utilize PROM data in future studies.

Method

This study complies with STROBE guidelines [22] for reporting observational studies in epidemiology. The data presented in this study is drawn from a population-based cross sectional study of “Late Effects After Colorectal Cancer” conducted among Danish long-term (4–10 years post initial treatment) CRCs, which is also being used for recruitment to a randomized controlled trial (RCT) of an online psychological intervention, see published protocol [21].

Study population

We identified eligible CRCs through the Danish Colorectal Cancer Group (DCCG) [23] database hosted by The Danish Clinical Registries (RKKP) with no need to enter confidential electronic patient records. We invited all Danish CRCs above age 18, able to read and understand Danish, who have completed curative-intent cancer treatment with surgery and/or radiation and/or adjuvant chemotherapy between March 2014 (when the national Danish Colorectal Cancer Screening Program was launched) and December 2018, to participate.

Data collection and definition of responders

Invited participants were a random 20% sample of the total cohort. We invited participants to answer an electronic questionnaire to screen for late effects. We distributed surveys through REDCap using the Danish national-wide secure, personal electronic mail box (e-Boks) between May 8th 2023 and May 12th 2023. In case of no answer on Day 10, the survey was automatically re-distributed. In case of no answer of Day 20, the survey was again re-distributed. Initial responders were defined as responders completing the questionnaire in the first 10 days following the initial invitation. Subsequent responders were defined as responders completing the questionnaire following either first or second reminder (10–28 days after initial invitation). Non-responders were defined as responders not completing the questionnaire within 4 weeks after last reminder. No incentives were offered.

The e-Boks system offers the option to set up automatic SMS/text message notifications whenever an email is received. Since this feature is personalized, we do not have specific data on its usage. No other phone calls or SMS/text messages were utilized to remind participants to respond.

Patient reported outcome measures (PROMs)

The selected PROMs aim to capture the diversity in psycho-oncological late effects experienced by colorectal cancer survivors. Participants completed an 83-item survey (see Table 1) comprising six questionnaires validated in Danish:

  1. 1.

    The Fear of Cancer Recurrence Inventory– Short Form (FCRI-SF) [24, 25] measures severity of fear of cancer recurrence on a 9-item subscale with response categories on a 5-point Likert like scale ranging from 0 to 4. Scale scores range from 0 to 36 with higher scores indicating greater severity.

  2. 2.

    The Fear of Cancer Recurrence – 1 revised (FCR-1r) [26] measures fear of cancer recurrence on a single item ranging from 0 to 10. Higher score indicates greater severity.

  3. 3.

    Symptom Checklist-90-R (SCL) [27] subscales [28] measuring anxiety (SCL-anx) in 4 items, depression (SCL-dep) in 6 items and emotional distress (SCL-distress) in 8 items on a 5-point Likert like scale rating from 0 to 4 resulting in simple sum scores ranging from 0 to 16, 24 and 32 respectively.

  4. 4.

    The Whiteley-6 [29, 30] measures health anxiety on a 5-point Likert like scale with a simple sum score ranging from 0 to 24.

  5. 5.

    The EQ-5D-5 L and EQ VAS scale [31] measures health state in five dimensions: mobility, self-care, usual activities, pain/discomfort and anxiety/depression on 5 levels resulting in a 5-digit number that describes the patient’s health state. The VAS targets health state today and ranges from 0 to 100. EQ-5D-5 L was analysed as sumscore index value and standardized according to the US EQ-5D-5L value set as recommended by EuroQoL [32].

  6. 6.

    The BDS Checklist [33] measures physical symptom load on a 5-point Likert like scale with a simple sum score rating from 0 to 100. Higher scores indicates greater symptom load. The 25-item questionnaire covers symptoms related to cardio/pulmonary function, bowel function, musculoskeletal function and general function. The BDS Checklist’s conceptualization for diagnosing bodily distress syndrome (BDS) was not applied since this was a cross-sectional survey with no participant contact. A high score on physical parameters can then represent comorbidity, late effects or functional disorders.

The BDS Checklist was supplemented by nine items covering well known late effects after colorectal cancer not included in the checklist. Three items relate to bowel function (Involuntary passage of air and/or loose stools, difficulty emptying the bowels during toilet visits and urgent bowel movement), five items relate to urinary function (Involuntary bladder leakage, bleeding from the bladder, frequent urination, difficulty emptying the bladder and bladder pain) and one item addresses impaired sexual function. Response categories on all additional items were on a 5-point Likert like scale similar the BDS Checklist, and the scale score range for the individual organ systems is shown in Table 1.

Quality of life was rated using a VAS scale (QoL VAS) inspired by the EuroQoL Health VAS scale, as the word “health” was changed into “Quality of life”. Additionally, the survivors were asked to report demographic and cancer-related outcomes supplementing the data drawn from registries.

In case of missing single items within the above mentioned questionnaires, the item was replaced with a 0 indicating a conservative approach, assuming that the symptom in question was not present.

Data cleaning

Returned questionnaires were excluded in case of cancer recurrence or if “dementia”, “terminal/too somatic ill” or “dead” were communicated by caregivers to the primary investigator by telephone or email. In certain instances, survivors indicated “no memory of cancer” or “do not want to participate”, leading to their exclusion. Similarly, responses lacking consent to participate were excluded. Cases that missed the reversed wording of item 5 on the FCRI-SF and presented with a response pattern of “0” were excluded, as this was interpreted as inattention or lack of motivation [34], see Fig. 1. Only cases with complete demographic data were analyzed, hence cases with missing items were also excluded.

Statistics

We compared demographic and cancer-related group characteristics (i.e., initial vs. subsequent vs. non-responders) using balance diagnostics and reported as standardized differences accompanied by descriptive statistics. An absolute standardized difference of > 0.1 was interpreted as imbalance between groups [35]. Cancer-related characteristics were compared using Chi-squared test and reported as absolute numbers, proportions and p-values. PROM-scores were analyzed using generalized linear regression models and reported as β-coefficients, 95% CI and p-values for subsequent responders using initial responders as the reference. Assumptions of normality, linearity and homoscedasticity were checked and found to be reasonable.

Analyses were performed on responders with complete PROMs and all non-responders (in total N = 1,955). Statistical analyses were performed in Stata 17 (Stata Corp., College Station, TX, USA).

Results

A total of 10,236 participants were diagnosed with colorectal cancer and treated with curative intend in the period March 2014 to December 2018. Of this cohort, 20% (2,097) were randomly selected for inclusion in this cross-sectional study and emailed a questionnaire between May 8th and May 12th 2023.

Fig. 1
figure 1

Participant flow chart

Full size image

Of the 2,097 eligible CRCs, 1,188 responded (57%), see Fig. 1. Of these, 142 (7%) were excluded due to comorbidity (n = 7), unwillingness to participate/no history of cancer (n = 16), recurrence (n = 41) or low quality/incomplete data (n = 78) resulting in 1,955 eligible CRCs.

628 respondents (32%) were categorized as initial responders and 418 respondents (21%) as subsequent responders. There were 909 (47%) non-responders.

Response pattern

37% responded on Day 0 and 6% on Day 1. Response rates Day 2–9 were < 4%. A similar pattern was seen after 1st reminder on Day 10 (11%) and 2nd reminder at Day 20 (7%) see Fig. 2.

Fig. 2
figure 2

Responses divided on days. Day 0 equals the distribution day. Reminders are send on day 10 and day 20 (vertical lines)

Full size image

Differences in demographic characteristics

We observed minor differences in demographic characteristics. Non-responders tend to be marginally older than initial and subsequent responders (median age 75 vs. 73 and 71, standardized difference 0.18 and 0.16, respectively), and consequently a larger proportion of the subsequent responders were employed (27% vs. 23%, standardized difference 0.18), see Table 2. The initial and subsequent responders were similar in terms of education, sex, marital status, language, ethnicity and children.

Differences in cancer-related characteristics

We observed no major differences in cancer-related characteristics between initial-responders, subsequent-responders and non-responders. A marginally higher proportion of both initial and subsequent responders, who were eligible for enrollment in the national colorectal cancer screening program (aged 50–75 at the time of diagnosis) had their cancer detected through screening as opposed to non-responders (45% and 43% compared to 39%). However, no disparity was observed in terms of T-category. N-category differed between groups mainly due to an uneven distribution of missing data, see Table 3.

Differences in PROMs

Concerning the physical and psychological parameters, only the single-item PROMs (the FCR-1r and sexual function) differed between initial responders and subsequent responders, as subsequent responders reported slightly higher FCR-1r (β 0.45, 95% CI(0.13–0.78)) and lower sexual dysfunction (β -0.24, 95% CI(-0.41 – -0.08)). We observed no difference in quality of life, health state, symptoms of anxiety, health anxiety, depression, or physical symptoms in general, see Table 4. As psychological parameters were measured through PROMs, we do not have data on non-responders.

Discussion

This study analysed generalizability and missingness mechanisms of PROMs in a random sample of long-term colorectal cancer survivors. Based on the present data we report a high grade of comparability between initial responders, subsequent responders and non-responders concerning demographic and cancer-related characteristics. In addition, initial and subsequent responders showed similar outcomes on PROMs, suggesting that the mechanisms of missing data among non-responders in this population can be assumed to be missing completely at random.

In research, high response rates are desirable to enhance validity and generalizability, and several strategies to enhance PRO response rates [10, 11, 13, 36,37,38,39,40] and deal with missing data have been developed [16, 20, 41]. However, potential bias does not depend on the response rate, but on the degree of similarity between respondents and non-respondents [42]. Achieving high response rates increases the likelihood of similarity, but often requires significant resources for reminders (either postal mail, email or phone call) and may also be burdensome for the participants.

If reminders are to be distributed, the data collecting process can be accelerated, as the response pattern of this population reveals a rapid decline in response rates similar to previous research [39], dropping below 1% within a week.

The findings in this study align with previous research demonstrating limited evidence of non-response bias. In a population-based survey of childhood, adolescent and young adult Norwegian long-term cancer survivors (the NOR-CAYACS study) [43] and in children participating in the Swiss Childhood Cancer survivor study [44] the authors found no evidence of relevant non-response bias in any of the investigated outcomes when comparing observed prevalence in respondents to expected prevalence in a constructed total population. Both studies were performed with postal surveys, and a long period between initial invitation and reminders (up to five months), which do not represent the electronic era of today.

In contrast to our findings, observational data from the Dutch PROFILES study, which included a heterogeneous cancer population, found respondents to be healthier than the population of interest, despite achieving a high response rate of 69% [45]. Similarly, in a longitudinal examination of an American breast cancer population using PROMs non-responders tended to be older and more frequently identified as non-English speaking, of Hispanic ethnicity, or of Black race compared to responders. Consequently, the authors concluded that the PROM results may not accurately represent the experiences of the entire American breast cancer population [46]. Downing et al. also found nonwhite ethnicity and those living in the most socioeconomically deprived areas less likely to participate in a population-level study, and concluded that the results presented may underestimate the true impact of colorectal cancer on health related quality of life [47]. The Danish population of CRCs is relatively homogeneous, and the current study was not designed to specifically investigate the influence of language, socioeconomic status or ethnicity.

While the findings of this study cannot be extrapolated to clinical cancer surveillance in general, understanding the characteristics of patients who do not respond to PROMs is essential especially given the current trend in survivorship care, which emphasizes a patient-centered approach, encouraging patients to self-report symptoms of recurrence, side effects, or late effects as they arise [48]. Based on this study, no explicit demographic or cancer-related characterization of this group can be made.

Strengths and limitations

A significant strength of this study lies in the comprehensive dataset available from the DCCG database, which includes valuable background variables and clinical information for all participants. With a database completeness exceeding 95%, we were provided with a unique opportunity to compare responders and non-responders. By recruiting study participants from this registry, we ensured the assembly of a complete and unbiased sample, allowing for a thorough examination of generalizability.

Nonetheless, we acknowledge certain limitations. Given the cross-sectional study design, PROM-data from non-responders are unavailable, and hence PROM-responses from subsequent responders, who were non-responders until distribution of reminders, may be the closest available. The assumption that subsequent responders may be indicative of non-responders cannot be tested. Multiple PRO scores from different assessment time points would add information regarding the nature of missingness of subsequent time points. Nevertheless, this will still not add information on never-responders.

Whenever missing data is an issue, study dependent missingness mechanisms should be explored and handled accordingly taking the specific research question, outcomes, exposures, covariates etc. into account. Therefore, more general advice on specific imputation strategies cannot be made based on this study. However, this study may help to inform the decision on how to handle missing data in similar populations or study designs.

Another limitation pertains to the methods of data registration, as data on for example lifestyle factors, comorbidity, and performance are recorded by the primary surgeon within 30 days after surgery, potentially introducing recall bias. Another limitation of this study is that the three survey invitations do not have individualized links. Consequently, a person may have responded to the initial invitation on day 11 without a reminder. The generalizability to other cancer diagnoses cannot be determined based on our findings.

Conclusion

In this study of long-term colorectal cancer survivors, we showed that initial responders, subsequent responders, and non-responders exhibit comparable demographic and cancer-related characteristics. Among respondents, PROMs were also similar, indicating generalizability.

Assuming PROMs of subsequent responders represents PROMs of the non-responders (would they be available), as suggested by our analysis, it may be reasonable to judge the missingness mechanism as Missing Completely At Random. Hence, imputation methods may be an option to enhance statistical power.

Table 1 Data sources
Full size table
Table 2 Demographic characteristics of the initial responders, subsequent responders and non-responders (N = 1,955) compared using balance diagnostics. Standardized differences > 0.1 indicate imbalance between groups
Full size table
Table 3 Cancer-related characteristics of initial responders, subsequent-responders and non-responders. P-values corresponds to chi-squared test
Full size table
Table 4 Generalized linear regression models comparing PROM-scores in initial responders (reference group) to subsequent responders (N = 1,046)
Full size table

Data availability

The data that support the findings of this study are available from the corresponding author upon reasonable request. Study data were collected and managed using REDCap electronic data capture tools hosted at Open Patient data Explorative Network (OPEN), Region of Southern Denmark.

Abbreviations

CRC:

Colorectal Cancer

FCR:

Fear of Cancer recurrence

MAR:

missing at random

MCAR:

missing completely at random

MNAR:

missing not at random

PROM:

Patient Reported Outcome Measure

References

  1. The global. Regional, and national burden of colorectal cancer and its attributable risk factors in 195 countries and territories, 1990–2017: a systematic analysis for the global burden of Disease Study 2017, the lancet. Gastroenterol Hepatol. 2019;4:913–33.

    Google Scholar 

  2. Kluetz PG, Slagle A, Papadopoulos EJ, Johnson LL, Donoghue M, Kwitkowski VE, Chen WH, Sridhara R, Farrell AT, Keegan P, Kim G, Pazdur R. Focusing on Core patient-reported outcomes in Cancer clinical trials: symptomatic adverse events, physical function, and Disease-related symptoms, clinical cancer research: an official journal of the American Association for Cancer Research, 22 (2016) 1553–8.

  3. Maspero M, Hull T. Patient-reported outcomes in colorectal surgery. Clin Colon Rectal Surg. 2023;36:240–51.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Mercieca-Bebber R, King MT, Calvert MJ, Stockler MR, Friedlander M. The importance of patient-reported outcomes in clinical trials and strategies for future optimization. Patient Relat Outcome Measures. 2018;9:353–67.

    Article  Google Scholar 

  5. Giesinger JM, Efficace F, Aaronson N, Calvert M, Kyte D, Cottone F, Cella D, Gamper EM. Past and current practice of patient-reported outcome measurement in Randomized Cancer clinical trials: a systematic review. Value Health: J Int Soc Pharmacoeconomics Outcomes Res. 2021;24:585–91.

    Article  Google Scholar 

  6. Au H-J, Ringash J, Brundage M, Palmer M, Richardson H, Meyer RM. Added value of health-related quality of life measurement in cancer clinical trials: the experience of the NCIC CTG. Expert Rev PharmacoEcon Outcomes Res. 2010;10:119–28.

    Article  PubMed  Google Scholar 

  7. Porter ME, Larsson S, Lee TH. Standardizing patient outcomes measurement. N Engl J Med. 2016;374:504–6.

    Article  CAS  PubMed  Google Scholar 

  8. Higgins TJ, Chandler JPT, Cumpston J, Li M, Page T, Welch MJ. VA, Cochrane Handbook for Systematic Reviews of Interventions version 6.3updated February (2022). Chapter 18: Patient-reported outcomes, Cochrane 2022, www.training.cochrane.org/handbook., (2022).

  9. I.S.f.Q.o.L, Research. Dictionary of quality of life and health outcomes measurement, Isoqol, Nancy Mayo, Milwaukee (WI), USA, (2015).

  10. Mercieca-Bebber R, Palmer MJ, Brundage M, Calvert M, Stockler MR, King MT. Design, implementation and reporting strategies to reduce the instance and impact of missing patient-reported outcome (PRO) data: a systematic review. BMJ open. 2016;6:e010938.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Palmer MJ, Mercieca-Bebber R, King M, Calvert M, Richardson H, Brundage M. A systematic review and development of a classification framework for factors associated with missing patient-reported outcome data. Clin Trials. 2018;15:95–106.

    Article  PubMed  Google Scholar 

  12. Galea S, Tracy M. Participation rates in epidemiologic studies. Ann Epidemiol. 2007;17:643–53.

    Article  PubMed  Google Scholar 

  13. Little RJ, D’Agostino R, Cohen ML, Dickersin K, Emerson SS, Farrar JT, Frangakis C, Hogan JW, Molenberghs G, Murphy SA, Neaton JD, Rotnitzky A, Scharfstein D, Shih WJ, Siegel JP, Stern H. The prevention and treatment of missing data in clinical trials. N Engl J Med. 2012;367:1355–60.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  14. Bell ML, Fiero M, Horton NJ, Hsu CH. Handling missing data in RCTs; a review of the top medical journals. BMC Med Res Methodol. 2014;14:118.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Sterne JA, White IR, Carlin JB, Spratt M, Royston P, Kenward MG, Wood AM, Carpenter JR. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ. 2009;338:b2393.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Jakobsen JC, Gluud C, Wetterslev J, Winkel P. When and how should multiple imputation be used for handling missing data in randomised clinical trials - a practical guide with flowcharts. BMC Med Res Methodol. 2017;17:162.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Fairclough DL, Peterson HF, Cella D, Bonomi P. Comparison of several model-based methods for analysing incomplete quality of life data in cancer clinical trials. Stat Med. 1998;17:781–96.

    Article  CAS  PubMed  Google Scholar 

  18. Fairclough DL, Peterson HF, Chang V. Why are missing quality of life data a problem in clinical trials of cancer therapy? Stat Med. 1998;17:667–77.

    Article  CAS  PubMed  Google Scholar 

  19. Rosett HA, Locke SC, Wolf SP, Herring KW, Samsa GP, Troy JD, LeBlanc TW. An analysis of missing items in real-world electronic patient reported outcomes data: implications for clinical care. Supportive care cancer: Official J Multinational Association Supportive Care Cancer. 2020;28:5099–107.

    Article  Google Scholar 

  20. Dziura JD, Post LA, Zhao Q, Fu Z, Peduzzi P. Strategies for dealing with missing data in clinical trials: from design to analysis. Yale J Biol Med. 2013;86:343–58.

    PubMed  PubMed Central  Google Scholar 

  21. Lyhne JD, Smith AB, Frostholm L, Fink P, Jensen LH. Study protocol: a randomized controlled trial comparing the efficacy of therapist guided internet-delivered cognitive therapy (TG-iConquerFear) with augmented treatment as usual in reducing fear of cancer recurrence in Danish colorectal cancer survivors. BMC Cancer. 2020;20:223.

    Article  PubMed  PubMed Central  Google Scholar 

  22. von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP. The strengthening the reporting of Observational studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Epidemiol (Cambridge Mass). 2007;18:800–4.

    Article  Google Scholar 

  23. Ingeholm P, Gögenur I, Iversen LH. Dan Colorectal Cancer Group Database Clin Epidemiol. 2016;8:465–8.

    Google Scholar 

  24. Simard S, Savard J. Fear of Cancer Recurrence Inventory: development and initial validation of a multidimensional measure of fear of cancer recurrence. Supportive care cancer: Official J Multinational Association Supportive Care Cancer. 2009;17:241–51.

    Article  Google Scholar 

  25. Hovdenak Jakobsen I, Jeppesen MM, Simard S, Thaysen HV, Laurberg S, Juul T. Initial validation of the Danish version of the fear of Cancer Recurrence Inventory (FCRI) in colorectal cancer patients. J cancer Survivorship: Res Pract. 2018;12:723–32.

    Article  Google Scholar 

  26. Smith A, Gao M, Tran M, Ftanou M, Jegathees S, Wu V, Jefford M, Lynch F, Dhillon HM, Shaw J, McDowell L, White A, Halloran C, Wiesenfeld D. Bamgboje-Ayodele, evaluation of the validity and screening performance of a revised single-item fear of cancer recurrence screening measure (FCR-1r). Psycho-oncology. 2023;32:961–71.

    Article  PubMed  Google Scholar 

  27. Olsen LR, Mortensen EL, Bech P. The SCL-90 and SCL-90R versions validated by item response models in a Danish community sample. Acta Psychiatrica Scandinavica. 2004;110:225–9.

    Article  CAS  PubMed  Google Scholar 

  28. Christensen K.S., Fink P., Toft T., Frostholm L., Ornbøl E., Olesen F. A brief case-finding questionnaire for common mental disorders: the CMDQ. Fam Pract. 2005;22:448–57.

  29. Veddegjærde KE, Sivertsen B, Wilhelmsen I, Skogen JC. Confirmatory factor analysis and item response theory analysis of the Whiteley Index. Results from a large population based study in Norway. The Hordaland Health Study (HUSK), Journal of psychosomatic research, 77 (2014) 213–218.

  30. Carstensen TBW, Ørnbøl E, Fink P, Pedersen MM, Jørgensen T, Dantoft TM, Benros ME, Frostholm L. Detection of illness worry in the general population: a specific item on illness rumination improves the Whiteley Index. J Psychosom Res. 2020;138:110245.

  31. Rabin R, de Charro F. EQ-5D: a measure of health status from the EuroQol Group. Ann Med. 2001;33:337–43.

  32. Pickard AS, Law EH, Jiang R, Pullenayegum E, Shaw JW, Xie F, Oppe M, Boye KS, Chapman RH, Gong CL, Balch A, Busschbach JJV. United States Valuation of EQ-5D-5L Health States using an International Protocol, Value in health: the journal of the International Society for Pharmacoeconomics and Outcomes Research, 22 (2019) 931–41.

  33. Budtz-Lilly A, Fink P, Ørnbøl E, Vestergaard M, Moth G, Christensen KS, Rosendal M. A new questionnaire to identify bodily distress in primary care: the ‘BDS checklist’. J Psychosom Res. 2015;78:536–45.

    Article  PubMed  Google Scholar 

  34. van Sonderen E, Sanderman R, Coyne JC. Ineffectiveness of reverse wording of questionnaire items: let’s learn from cows in the rain. PLoS ONE. 2013;8:e68967.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Austin PC. Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity-score matched samples. Stat Med. 2009;28:3083–107.

    Article  PubMed  PubMed Central  Google Scholar 

  36. Chowdhry AK, Gondi V, Pugh SL. Missing Data in Clinical studies. Int J Radiat Oncol Biol Phys. 2021;110:1267–71.

    Article  PubMed  Google Scholar 

  37. Aiyegbusi OL, Roydhouse J, Rivera SC, Kamudoni P, Schache P, Wilson R, Stephens R, Calvert M. Key considerations to reduce or address respondent burden in patient-reported outcome (PRO) data collection. Nat Commun. 2022;13:6026.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  38. Papa N, Bensley JG, Hall K, Evans M, Millar JL. Quantifying the effect email reminders have on patient reported outcome measure returns in a large prostate cancer registry. J Patient Rep Outcomes. 2022;6:19.

    Article  PubMed  PubMed Central  Google Scholar 

  39. Nielsen LK, King M, Möller S, Jarden M, Andersen CL, Frederiksen H, Gregersen H, Klostergaard A, Steffensen MS, Pedersen PT, Hinge M, Frederiksen M, Jensen BA, Helleberg C, Mylin AK, Abildgaard N. Strategies to improve patient-reported outcome completion rates in longitudinal studies. Qual life Research: Int J Qual life Aspects Treat care Rehabilitation. 2020;29:335–46.

    Article  Google Scholar 

  40. Bulkley JE, O’Keeffe-Rosetti M, Wendel CS, Davis JV, Danforth KN, Harrison TN, Kwan ML, Munneke J, Brooks N, Grant M, Leo MC, Banegas M, Weinmann S, McMullen CK. The effect of multiple recruitment contacts on response rates and patterns of missing data in a survey of bladder cancer survivors 6 months after cystectomy, quality of life research: an international journal of quality of life aspects of treatment. care Rehabilitation. 2020;29:879–89.

    Google Scholar 

  41. Hamel JF, Sebille V, Le Neel T, Kubis G, Boyer FC, Hardouin JB. What are the appropriate methods for analyzing patient-reported outcomes in randomized trials when data are missing? Stat Methods Med Res. 2017;26:2897–908.

    Article  CAS  PubMed  Google Scholar 

  42. Groves RM. Nonresponse rates and nonresponse bias in household surveys. Int J Public Opin Q. 2006;70:646–75.

    Article  Google Scholar 

  43. Lie HC, Rueegg CS, Fosså SD, Loge JH, Ruud E, Kiserud CE. Limited evidence of non-response bias despite modest response rate in a nationwide survey of long-term cancer survivors-results from the NOR-CAYACS study. J cancer Survivorship: Res Pract. 2019;13:353–63.

    Article  Google Scholar 

  44. Rueegg CS, Gianinazzi ME, Michel G, Zwahlen M, von der Weid NX, Kuehni CE. No evidence of response bias in a population-based childhood cancer survivor questionnaire survey - results from the Swiss Childhood Cancer Survivor Study. PLoS ONE. 2017;12:e0176442.

    Article  PubMed  PubMed Central  Google Scholar 

  45. de Rooij BH, Ezendam NPM, Mols F, Vissers PAJ, Thong MSY, Vlooswijk CCP, Oerlemans S, Husson O, Horevoorts NJE. Poll-Franse, Cancer survivors not participating in observational patient-reported outcome studies have a lower survival compared to participants: the population-based PROFILES registry. Qual life Research: Int J Qual life Aspects Treat care Rehabilitation. 2018;27:3313–24. van de.

    Article  Google Scholar 

  46. Srour MK, Tadros AB, Sevilimedu V, Nelson JA, Cracchiolo JR, McCready TM, Silva N, Moo TA, Morrow M. Who are we missing: does Engagement in patient-reported outcome measures for breast Cancer Vary by Age, Race, or Disease Stage? Ann Surg Oncol. 2022;29:7964–73.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Downing A, Morris EJ, Richards M, Corner J, Wright P, Sebag-Montefiore D, Finan P, Kind P, Wood C, Lawton S, Feltbower R, Wagland R, Vernon S, Thomas J, Glaser AW. Health-related quality of life after colorectal cancer in England: a patient-reported outcomes study of individuals 12 to 36 months after diagnosis. J Clin Oncology: Official J Am Soc Clin Oncol. 2015;33:616–24.

    Article  Google Scholar 

  48. Hovdenak I, Thaysen HV, Bernstein IT, Christensen P, Hauberg A, Iversen LH, Johansen C, Larsen SL, Laurberg S, Madsen AH, Madsen MR, Rasmussen HV, Thorlacius-Ussing O, Juul T. Quality of life and symptom burden after rectal cancer surgery: a randomised controlled trial comparing patient-led versus standard follow-up. J cancer Survivorship: Res Pract, (2023).

Download references

Acknowledgements

Thanks to Harald Hammershøi and OPEN, Odense Patient data Explorative Network, Odense University Hospital, Odense, Denmark: www.sdu.dk/ki/open, for assistance with REDCap and to all the colorectal cancer survivors who participated in the study.

Funding

JDL was supported by the Danish Cancer Society, Grant# R280 A16561, Einar Willumsens memorial grant, and Dagmar Marshalls Foundation. ABS was funded by the New South Wales (NSW) Government through a Cancer Institute NSW Career Development Fellowship (#2021/DCF1138). The funding bodies played no role in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript.

Open access funding provided by University of Southern Denmark

Author information

Authors and Affiliations

  1. Department of Oncology, University Hospital of Southern Denmark, Beriderbakken 4, Vejle, 7100, Denmark

    Johanne Dam Lyhne, Lars Henrik Jensen, Torben Frøstrup Hansen & Signe Timm

  2. The Daffodil Centre, The University of Sydney, A Joint Venture with Cancer Council NSW, Sydney, NSW, Australia

    Allan ‘Ben’ Smith

  3. Research Clinic for Functional Disorders and Psychosomatics, Aarhus University Hospital, Aarhus, Denmark

    Lisbeth Frostholm

  4. Department of Clinical Medicine, Aarhus University, Aarhus, Denmark

    Lisbeth Frostholm

Authors
  1. Johanne Dam Lyhne
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Allan ‘Ben’ Smith
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lars Henrik Jensen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Torben Frøstrup Hansen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Lisbeth Frostholm
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Signe Timm
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

JDL and ST designed the study. Data collection was performed by JDL. Statistical analysis was performed by ST. Results was interpreted in collaboration between JDL, ABS, LHJ, THS, LF and ST, and all authors participated in the discussion. All authors have revised previous versions of the manuscript critically and approved the final version for publication.

Corresponding author

Correspondence to Johanne Dam Lyhne.

Ethics declarations

Ethics approval

This study was performed according to the principles of the Declaration of Helsinki. The project was approved on the 20th of August 2019 by The Regional Committees on Health Research Ethics for Southern Denmark (Project-ID: S-20190061).

Consent to participate

Informed consent was obtained from all individual participants included in the study.

Consent for publication

Not Applicable.

Competing interests

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lyhne, J.D., Smith, A.‘., Jensen, L.H. et al. Missingness mechanisms and generalizability of patient reported outcome measures in colorectal cancer survivors – assessing the reasonableness of the “missing completely at random” assumption. BMC Med Res Methodol 24, 104 (2024). https://doi.org/10.1186/s12874-024-02236-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12874-024-02236-z

Keywords

BMC Medical Research Methodology

ISSN: 1471-2288

Contact us