Email updates

Keep up to date with the latest news and content from BMC Medical Research Methodology and BioMed Central.

Open Access Highly Accessed Research article

Systematic reviews need to consider applicability to disadvantaged populations: inter-rater agreement for a health equity plausibility algorithm

Vivian Welch12*, Kevin Brand3, Elizabeth Kristjansson4, Janet Smylie56, George Wells789 and Peter Tugwell101112

Author Affiliations

1 Clinical Epidemiology Unit, Ottawa Hospital Research Institute, Ottawa Hospital, Ottawa, ON, Canada

2 Centre for Global Health, Institute of Population Health, University of Ottawa, 1 Stewart Street, Ottawa, ON, K1N6N5, Canada

3 Telfer School of Management, University of Ottawa, Ottawa, ON, Canada

4 School of Psychology, Centre for Global Health, Institute of Population Health, University of Ottawa, 1 Stewart Street, Ottawa, ON, K1N6N5, Canada

5 Centre for Research on Inner City Health, Keenan Research Centre of the Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Toronto, ON, Canada

6 Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada

7 Department of Epidemiology and Community Medicine and Department of Medicine, University of Ottawa, Ottawa, ON, Canada

8 University of Ottawa Heart Institute, Ottawa, ON, Canada

9 Clinical Epidemiology Unit, Ottawa Hospital Research Institute, Ottawa Hospital, Ottawa, ON, Canada

10 Department of Epidemiology and Community Medicine and Department of Medicine, University of Ottawa, Ottawa, ON, Canada

11 Clinical Epidemiology Unit, Ottawa Hospital Research Institute Ottawa Hospital, Ottawa, ON, Canada

12 Centre for Global Health, Institute of Population Health, University of Ottawa, 1 Stewart Street, Ottawa, ON, K1N6N5, Canada

For all author emails, please log on.

BMC Medical Research Methodology 2012, 12:187  doi:10.1186/1471-2288-12-187

Published: 19 December 2012

Abstract

Background

Systematic reviews have been challenged to consider effects on disadvantaged groups. A priori specification of subgroup analyses is recommended to increase the credibility of these analyses. This study aimed to develop and assess inter-rater agreement for an algorithm for systematic review authors to predict whether differences in effect measures are likely for disadvantaged populations relative to advantaged populations (only relative effect measures were addressed).

Methods

A health equity plausibility algorithm was developed using clinimetric methods with three items based on literature review, key informant interviews and methodology studies. The three items dealt with the plausibility of differences in relative effects across sex or socioeconomic status (SES) due to: 1) patient characteristics; 2) intervention delivery (i.e., implementation); and 3) comparators. Thirty-five respondents (consisting of clinicians, methodologists and research users) assessed the likelihood of differences across sex and SES for ten systematic reviews with these questions. We assessed inter-rater reliability using Fleiss multi-rater kappa.

Results

The proportion agreement was 66% for patient characteristics (95% confidence interval: 61%-71%), 67% for intervention delivery (95% confidence interval: 62% to 72%) and 55% for the comparator (95% confidence interval: 50% to 60%). Inter-rater kappa, assessed with Fleiss kappa, ranged from 0 to 0.199, representing very low agreement beyond chance.

Conclusions

Users of systematic reviews rated that important differences in relative effects across sex and socioeconomic status were plausible for a range of individual and population-level interventions. However, there was very low inter-rater agreement for these assessments. There is an unmet need for discussion of plausibility of differential effects in systematic reviews. Increased consideration of external validity and applicability to different populations and settings is warranted in systematic reviews to meet this need.

Keywords:
Systematic reviews; Applicability; Health equity; Sex and gender; Socioeconomic status