To determine characteristics that should be abstracted, we reviewed the literature as to how RCTs should be reported 1516 and evaluated 17, how OSs should be reported and evaluated 18, and how patient and treatment characteristics influence RCTs 19. Some information from RCTs was not relevant for OSs (e.g., blinding of the randomization process), some was relevant but not available in OSs (e.g., protocols for administering the primary treatment and managing intermediate outcomes), and some was not relevant to comparing studies (e.g., power of the study when data have already been collected).

Based on previous literature and our own experience with OSs, we developed the schema in Table 1 for study characteristics that could influence results by influencing either the study's applicability or validity. Factors that could influence applicability include specific characteristics of the treatments, outcomes, or subjects; results that apply to studies using specific types of treatments, outcome measurements or subjects may be valid, but may not be reproduced by other studies using different types of treatments, outcome measurements or subjects.

Factors that could influence validity include those that could contribute to confounding, selection, or information (also called measurement) bias 20. Confounding arises when subjects who receiving one treatment differ in risk from subjects receiving another, independent of the effect of treatment. Selection bias occurs when the association between exposure and disease differs between those who complete a study and those in the target population. In cohort studies of medical treatments, such as those reviewed below, pre-treatment selection bias leads to confounding and post-treatment selection bias results from incomplete follow-up that differs according to both outcome and treatment. Information bias occurs when errors are made in assessing which treatment or outcome a patient had. Although post-treatment selection bias and information bias distort estimates of effect size, they were difficult to assess in the papers reviewed and were not recorded in our analysis.

This schema guided the type of information abstracted from the reviewed articles. Although it does not include all 27 items considered important for measuring the quality of OSs in one schema 18, it is conceptually simple and should include most study aspects that influence interpretation of results. For each article reviewed, we noted critical data elements omitted from the article.

We deemed that the likelihood of confounding would be increased if treatment choice were related to time, so that recent patients generally received one treatment, whereas patients from several years previously received another. Confounding could also be more likely if treatment was allocated on the basis of patient characteristics that contribute to treatment failure, either by the physician or through patient self-selection. Confounding was considered less likely if the physicians treating the patients used only one procedure. An implicit assumption in this criterion is that patient risk and quality of care are similar across physicians; this assumption may not be valid in all cases, but we wanted to judge the studies as generously as possible so that reports of deficiencies in these studies would be conservative. Another criterion for decreased likelihood of confounding was an abrupt change in patient care, so that all patients received one treatment before the change and all patients received another treatment after the change.

All data abstraction was from the original articles. Although the majority of articles were assessed independently by two different reviewers, some articles were only assessed by the same reviewer several months apart. Disagreements between reviewers, between reviews at different times, or between our reviewers and the published meta-analysis were resolved through discussion.

Results were reported as statistically significant if p < 0.05, although p-values were often much lower. We used a 2-by-2 χ²-test for contingency tables to compare OS and RCT subjects for pooled failure rates on the same treatment. Significantly different failure rates for OS and RCT studies of one treatment in a comparison but not the other is a sensitive indication of possible confounding in the OSs. Significantly different failure rates for both treatments suggest that the two types of studies may differ with respect to features that influence failure rates (e.g., patients, outcome measures, specifics of the treatment, or uses of ancillary treatments). We also evaluated whether it might be worthwhile to search for important study factors that caused heterogeneity by examining variation among OSs for failure rates of a given treatment. The p-value for the statistical significance of this variation was determined using a 2-by-k χ²-test for contingency tables, where k was the number of studies that evaluated a given treatment. Pearson's correlation coefficient, denoted r, was used to compute the p-value for the association between the failure rates in the treatment group and the failure rates in the control group at the 0.05 significance level.

Statistical methods were used to combine odds ratios from several studies and to test the difference between the summary odds ratios from the observational and randomized studies. To combine odds ratios from several studies and to find the standard error of the combined odds ratio, we used a fixed-effects calculation 21. By using fixed- rather than random-effects calculations 22, we obtained smaller standard errors and decreased the chances of missing true differences. However, this method may increase the likelihood of finding spurious differences.

We tested the difference between two odds ratios using the equation

Z = (Ln₁ - Ln₂) / √(SE₁² + SE₂²)

where Z has a normal distribution with mean zero and variance 1, Ln₁ and Ln₂ are the logarithms of the two odds ratios, and SE₁ and SE₂ are the standard errors of these logarithms. Heterogeneity in odds ratios was tested with the Breslow-Day test for homogeneity at the 0.05 significance level.

The selected analyses are shown in Table 2. These analyses addressed 10 topics: anticoagulants for treatment of myocardial infarction, quinidine for atrial fibrillation, trial of labor for patients with a breech delivery, colposuspension compared to anterior colporrhaphy for urinary incontinence, colposuspension compared to needle suspension for urinary incontinence, transcutaneous electrical nerve stimulation (TENS) for treatment of postsurgical pain, early discharge following childbirth, hip screws for hip fracture, local anesthesia for patients with carotid endarterectomy, and hysterosalpingography (HSG) media on pregnancy.

With five exceptions, we considered as observational all studies that were considered observational in the reviewed meta-analyses 13: three of these were excluded because they used alternately assigned controls 232425, and two were RCTs 2627. We did not exclude studies that used historical controls. Seven additional studies that were not in English were excluded because we were not able to accurately abstract detailed information about them.

Some studies assessed more than one outcome. With one exception, we reported results for the same outcomes that were assessed in the study by Ioannidis et al 13. The exception was the meta-analysis of quinidine 28. The outcome used by Ioannidis et al. from that analysis was mortality, which was zero or near zero for most studies. We used relapse of atrial fibrillation following cardioversion, which was used by our other source of meta-analyses 14. Failure rates were used to compute odds ratios not computed by the original studies. For some studies the success rates and odds ratios in the primary studies 29303132 differed from those reported by Ioannidis et al. 13 or the meta-analysis 28. When there was a discrepancy, we used the rates reported in the primary studies. Rates in primary studies for positive endpoints (e.g. pregnancy) were converted to failure rates (e.g. no pregnancy).

The comparison of the combined odds ratios for the two types of studies are shown in Figure 1. In general, the confidence intervals were wider for the RCTs than for the OSs, reflecting the larger sample sizes for the OSs. Wide confidence intervals for randomized controlled studies of trial of labor, transcutaneous electrical nerve stimulation (TENS), early discharge, and local anesthesia prevented meaningful comparisons for these treatment areas. The only treatment area for which the odds ratios differed significantly was studies of anticoagulants following an acute myocardial infarction.

The observational and randomized studies differed with respect to several failure rates (see Table 3). For some treatment comparisons there were dissimilar failure rates between the types of studies for both treatment and control groups (TENS for postoperative pain and early discharge following childbirth), and for other treatment comparisons there were significant differences between study designs with respect to the rates for patients with the new treatments, but not for patients on the older treatments (quinidine for the treatment of atrial fibrillation and colposuspension versus two older treatments for urinary incontinence).

As indicated in Table 3, several studies show considerable heterogeneity in results among OSs. For each treatment comparison there was statistically significant variation in failure rates for at least one of the treatments. There was statistically significant heterogeneity in the odds ratios for studies of anticoagulants, colporrhaphy, needle suspension, and hysterosalpingography. Despite small numbers of studies in each treatment area, failure rates were significantly correlated for studies of anticoagulants (r = 0.79, p = 0.01), trial of labor (r = 0.75, p = 0.08), early discharge (r = 0.99, p = 0.01), hip screws (r = 0.93, p = 0.02), and local anesthesia (r = 0.66, p = 0.03). This correlation might be explained by substantial differences among study features that influence failure rates.

Reporting details of the primary treatment, ancillary treatments, and management of intermediate outcomes was uniformly poor. Most aspects of outcome were also poorly reported. However, outcome definitions were generally well reported. Even here there were exceptions: one study of surgical treatment for incontinence defined the subjective outcome only as "cured" 33, and another defined it as "symptom free" 34.

Length of follow-up, which may substantially influence outcome and comparisons of treatments, was usually not well reported. Of the studies reviewed, only two studies of hysterosalpingography and five of surgical treatment for stress incontinence provided both the mean (or median) and range (or other measure of spread) of follow-up times. Eleven studies provided no follow-up information, and the remainder provided only one number (median, minimum, or undefined).

Even though the majority of studies were based on chart abstraction, none described methods for reducing selection or information bias.

Results from studies were sometimes combined, even though they differed with respect to potentially important patient characteristics. For example, studies of surgical treatment for incontinence varied with respect to exclusions due to previous surgery for incontinence, detrussor instability, and other pathologic findings. Another example is that criteria for studies of local anesthesia for carotid endarterectomy varied on the basis of whether patients were included who were simultaneously undergoing a coronary artery bypass grafting procedure or who had an acute stroke. Among studies of early discharge, one unique inclusion criteria was caesarian delivery 35 and another was primiparity 36. Of the two studies of HSG that provided detail on inclusion and exclusion criteria, one required infertility for at least two years 37 and a second required infertility for only one year 38.

In Table 4 articles are rated for their reporting of patient characteristics in a descriptive table. Articles were rated as 'A' if they reported at least one item in each of the categories of medical history, demographics, and clinical assessment. Even with these minimal criteria a minority of studies were categorized as 'A'; the only treatment areas that had primarily 'A's were local anesthesia for carotid endarterectomy and colporrhaphy or needle suspension for incontinence. For one treatment area, early versus conventional discharge, none of the OSs provided information on maternal comorbidities or other relevant aspects of medical history.

Table 4 also describes how study characteristics were reported that could influence confounding. Confounding was more likely in two studies because subjects on one treatment were treated several years previously compared with subjects on another. Confounding was also more likely in other studies (the majority of trials of labor and surgery for incontinence and half the studies of early discharge 3233) because treatment was allocated on the basis of patient characteristics likely to influence the possibility of treatment failure. Confounding may have been less likely if the physicians treating the patients used only one procedure. This occurred in a few studies of local anesthesia for carotid endarterectomy, hip screws for hip fracture, and contrast media for HSG. Confounding was considered less likely in another study because of an abrupt change in patient care 35. In several studies it was not possible to assess how patient preferences may have influenced confounding 39404142.

Table 4 shows whether studies assessed the possibility of confounding by comparing patients on the two treatments with respect to at least one variable from the categories of medical history, demographics, and clinical assessment. Most studies did not make these comparisons; the few that did should have evaluated additional potential confounders. In addition, once potential confounders were identified, the studies made only minimal use of statistical methods to control for confounding. A few studies attempted to control for confounding by stratifying on the basis of some risk factors, but only one study performed a regression analysis that adjusted for multiple risk factors 43.

We found evidence that variation in outcome definition and length of follow-up caused heterogeneity in results. For example, in studies of trial of labor, the study with the lowest failure rate 44 was also the study that defined poor outcome in the newborn as a five-minute Apgar score less than five, instead of less than seven as used in other studies (the lower the score the more likely the newborn is to require resuscitation). For studies of early discharge, the lowest failure rate came from a study that examined post-operative complications of C-section patients, and the highest rates came from a study that included many common symptoms in the definition of maternal morbidity (e.g., cold, flu, and constipation). In the studies comparing colposuspension to either needle suspension or colporrhaphy, the lowest failure rates in the colposuspension groups came from studies in which follow-up was less than one-year, and those low failure rates were very similar to the RCTs, both of which had follow-up of one year. For the study of HSG with the lowest odds ratio (0.98) 37, the duration of follow-up was two years, as compared to the other studies which had follow-up of one year or less. It is possible that infertility problems that improved with oil-contrast media may have resolved in any case over a two-year period.

Differences for the studies of TENS for postoperative pain and early discharge following childbirth may have been due in part to dissimilar definitions of failure. The OSs of TENS defined this as whether or not a patient received post-operative medications. The randomized controlled trials used verbal ratings of pain that were dichotomized into "satisfactory" or "unsatisfactory". For studies of early discharge, the randomized controlled trial defined failure as maternal problems requiring physician referral. These problems were primarily infections: urinary tract infections, episiotomy infection, mastitis, subinvolution, and endometritis. Most OSs defined failure as maternal problems determined from physical assessment or self-report. Since these problems included constipation, flu-like symptoms, and lethargy as well as infections, failure rates were generally higher for observational than for randomized studies. The one exception was an observational study that examined outcomes post-caesarian section and defined failure as fever, wound infection etc 35. The failure rates for this study were 6% for early discharge and 7% for the conventional group, which are similar to the rates from randomized studies. Without this study of C-section patients, the overall failure rates for the observational and randomized studies would have differed even more.

The primary concern about OSs is confounding. There was evidence of obvious confounding that was not taken into account in three treatment comparisons: 1) influence of anticoagulants on survival of myocardial infarction (historical controls treated several years earlier 454647 and anticoagulants preferentially given to younger patients and patients at lower risk for other reasons 4849) 2) quinidine for the treatment of arrhythmias (significantly higher 5051 rates of valvular heart disease in the quinidine group), and 3) colposuspension versus anterior colporrhaphy (substantially and significantly higher rates in the colposuspension group of severe pre-surgery incontinence 52). 3234 In no study showing obvious confounding did the authors assess or adjust for confounding, or even raise it as a concern.