Publication bias, chance, and heterogeneity: how researchers interpret the funnel plot

Norma C Terrin^† , Christopher H Schmid and Joseph Lau

Division of Clinical Care Research, New England Medical Center, Boston, USA

author email† Presenting author

9th International Cochrane Colloquium
Lyon, France, 9-13 October 2001

Cochrane 2001, 1:op006

Received:	19 July 2001
Published:	26 August 2001

Background

The existence of publication bias in the medical literature is well documented: studies which achieve statistical significance are more likely to be published than those which do not. The funnel plot has been recommended as a tool for detecting potential publication bias, and is frequently used in medical meta-analyses. The standard interpretation is to infer publication bias if the plot is asymmetric, although it is known that factors other than publication bias can influence the shape of the plot.

Objectives

1) To show that both chance and heterogeneity play a large role in the degree of asymmetry in funnel plots for meta-analyses of typical size. 2) To determine whether readers of funnel plots can distinguish between asymmetry due to publication bias and asymmetry due to chance or heterogeneity.

Methods

A questionnaire was designed with 22 funnel plots of simulated and actual published meta-analyses. The questionnaire began with an explanation of funnel plots taken from a standard reference. The plots had varying degrees of symmetry, as measured by the number of points in the plot to the left of the most precise study. The plots from simulated data included meta-analyses with and without publication bias, and with and without heterogeneity. All had sample sizes typical of actual meta-analyses: 10 studies with 50 to 500 patients each. Odds ratio (log scale) was on the horizontal axis with inverse standard error on the vertical axis. Respondents were asked to say whether they believed there was evidence of publication bias ("yes", "maybe", or "no") for each of the 22 plots. The participants were 20 research fellows and students in a meta-analysis course. We also analyzed large numbers of simulated meta-analyses generated from the same models used in the questionnaire.

Results for objective 1

Among 1000 simulated homogeneous meta-analyses without bias, 16% had 8-9 studies to the left of the most precise study, 28% had 6-7, and 56% had 0-5. With bias, the corresponding percents were 35%, 32%, and 33%, respectively. Under one simulation model for heterogeneity, over 80% of meta-analyses had 8-9 studies to the left, whether or not there was publication bias.

Results for objective 2

When there were 4 studies to the left of the most precise study, 22% of the answers were "yes", 26% "maybe" and 56% "no". With 7 studies to the left, the percents were 44%, 38%, and 18%, and with 9 studies to the left, the percents were 54%, 29%, and 17%, respectively. The responses had no relation to the existence of publication bias, when conditioned on degree of symmetry.

Conclusions

Researchers tend to read asymmetric funnel plots as evidence of publication bias, even though meta-analyses without publication bias frequently have asymmetric plots and meta-analysis with publication bias frequently have symmetric plots, simply due to chance. Use of funnel plots is even more unreliable when there is heterogeneity.