How confidence intervals become confusion intervals
1 Faculty of Pharmaceutical Sciences, University of British Columbia, Vancouver BC, Canada
2 Alberta Research Centre for Health Evidence, University of Alberta, Edmonton Alberta, Canada
3 Evidence-Based Medicine, Department of Family Medicine, University of Alberta, Room 1706 College Plaza, 8215 - 112 Street NW, Edmonton AB, Canada
BMC Medical Research Methodology 2013, 13:134 doi:10.1186/1471-2288-13-134Published: 31 October 2013
Controversies are common in medicine. Some arise when the conclusions of research publications directly contradict each other, creating uncertainty for frontline clinicians.
In this paper, we review how researchers can look at very similar data yet have completely different conclusions based purely on an over-reliance of statistical significance and an unclear understanding of confidence intervals. The dogmatic adherence to statistical significant thresholds can lead authors to write dichotomized absolute conclusions while ignoring the broader interpretations of very consistent findings. We describe three examples of controversy around the potential benefit of a medication, a comparison between new medications, and a medication with a potential harm. The examples include the highest levels of evidence, both meta-analyses and randomized controlled trials. We will show how in each case the confidence intervals and point estimates were very similar. The only identifiable differences to account for the contrasting conclusions arise from the serendipitous finding of confidence intervals that either marginally cross or just fail to cross the line of statistical significance.
These opposing conclusions are false disagreements that create unnecessary clinical uncertainty. We provide helpful recommendations in approaching conflicting conclusions when they are associated with remarkably similar results.