Open Access Highly Accessed Research article

A critical analysis of test-retest reliability in instrument validation studies of cancer patients under palliative care: a systematic review

Carlos Eduardo Paiva123*, Eliane Marçon Barroso2, Estela Cristina Carneseca23, Cristiano de Pádua Souza1, Felipe Thomé dos Santos1, Rossana Verónica Mendoza López3 and Sakamoto Bianca Ribeiro Paiva23

Author Affiliations

1 Department of Clinical Oncology, Barretos Cancer Hospital, Barretos, São Paulo CEP 14784-400, Brazil

2 Palliative Care and Quality of Life Research Group, Post-Graduate Program, Barretos Cancer Hospital, Barretos, São Paulo, Brazil

3 Researcher Support Center, Learning and Research Institute, Barretos Cancer Hospital, Barretos, São Paulo, Brazil

For all author emails, please log on.

BMC Medical Research Methodology 2014, 14:8  doi:10.1186/1471-2288-14-8

Published: 21 January 2014



Patient-reported outcome validation needs to achieve validity and reliability standards. Among reliability analysis parameters, test-retest reliability is an important psychometric property. Retested patients must be in a clinically stable condition. This is particularly problematic in palliative care (PC) settings because advanced cancer patients are prone to a faster rate of clinical deterioration. The aim of this study was to evaluate the methods by which multi-symptom and health-related qualities of life (HRQoL) based on patient-reported outcomes (PROs) have been validated in oncological PC settings with regards to test-retest reliability.


A systematic search of PubMed (1966 to June 2013), EMBASE (1980 to June 2013), PsychInfo (1806 to June 2013), CINAHL (1980 to June 2013), and SCIELO (1998 to June 2013), and specific PRO databases was performed. Studies were included if they described a set of validation studies. Studies were included if they described a set of validation studies for an instrument developed to measure multi-symptom or multidimensional HRQoL in advanced cancer patients under PC. The COSMIN checklist was used to rate the methodological quality of the study designs.


We identified 89 validation studies from 746 potentially relevant articles. From those 89 articles, 31 measured test-retest reliability and were included in this review. Upon critical analysis of the overall quality of the criteria used to determine the test-retest reliability, 6 (19.4%), 17 (54.8%), and 8 (25.8%) of these articles were rated as good, fair, or poor, respectively, and no article was classified as excellent. Multi-symptom instruments were retested over a shortened interval when compared to the HRQoL instruments (median values 24 hours and 168 hours, respectively; p = 0.001). Validation studies that included objective confirmation of clinical stability in their design yielded better results for the test-retest analysis with regard to both pain and global HRQoL scores (p < 0.05). The quality of the statistical analysis and its description were of great concern.


Test-retest reliability has been infrequently and poorly evaluated. The confirmation of clinical stability was an important factor in our analysis, and we suggest that special attention be focused on clinical stability when designing a PRO validation study that includes advanced cancer patients under PC.

Validation studies; Test-retest reliability; Systematic review; Cancer; Palliative care