Comparing different interventions for the same condition is often difficult. Large direct comparisons are uncommon, and usually represent only a fraction of the total randomised trial data available. Instead we often have large numbers of randomised trials comparing different interventions with the same or similar comparators, like placebo or an active comparator. These may be direct comparisons, but not the direct comparisons we want.

Indirect comparison of interventions using a common comparator is a valuable alternative because it uses more of the published data 1, and has been done before for analgesics in acute pain 2 and migraine treatments 3. Even this approach can be devalued because clinical trials in published papers are not consistent in the outcomes they report, or the way they report them 4.

The ideal should be to compare interventions of equivalent intensities or dose, in the same condition, at similar disease severity, using the same outcomes properly reported, over the same period of time. In this study we use the example of published studies of phosphodiesterase-5 (PDE-5) inhibitors for male erectile dysfunction to examine the problems of comparing therapies. PDE-5 inhibitors were chosen because they have been introduced within the past decade, by three different pharmaceutical companies, and in an era of good clinical trial practice. There are no good quality trials directly comparing PDE-5 inhibitors at equivalent doses.

We sought randomised trials of three PDE-5 inhibitors (sildenafil, tadalafil, vardenafil), with placebo or active comparator, in men with erectile dysfunction of any causation. Previous systematic reviews 56789 were used to source trials or trial data, supplemented by electronic searches of PubMed (to June 2005) and the Cochrane Library (issue 1, 2005) using drug names and randomis(z)ed trial.

For inclusion a trial had to be randomised and double blind, use one of the three oral PDE-5 inhibitors in men with erectile dysfunction, be conducted in the home setting, last three weeks or longer, have a minimum of 10 men per group, and report useful information on efficacy or adverse events. Abstracts were read, and potentially useful reports retrieved in full paper copy. Decisions on inclusion or exclusion were made by consensus. No information was taken from posters or abstracts, and studies were read carefully to avoid including duplicate material. Studies were scored for reporting quality using a common method 10 utilising reporting of randomisation, blinding and withdrawals. The maximum score possible was 5 points, and no study could be included with fewer than 2 points.

Information extracted from studies included the number of men studied, the cause of erectile dysfunction, and country where the study was performed. Any type of outcome was initially extracted from the studies, in continuous or dichotomous form, and with any dispersion information available. Outcomes could be reported in tables, in graphs, or in text. The following outcomes were sought particularly:

There were 35 included studies using sildenafil 2021222324252627282930313233343536373839404142434445464748495051525354, treating 7135 men with a mixture of conditions causing erectile dysfunction (Table 1); one study 40 was analysed as two trials. Of the men, 3279 received placebo and 3856 sildenafil, of whom all but 90 took licensed doses of 25 to 100 mg. Optimised dosing with 25–100 mg was most used (2546 men), followed by 100 mg (506) or 50 mg (370). All were placebo-controlled with no active comparator. Quality scores were high, 11 trials scoring 5 of 5 points, 17 scoring 4 points, six 3 points, and two 2 points. One further study 55 used a withdrawal model after successful treatment, and information from the 205 men in this study was not included in this analysis.

There were eight included studies using tadalafil 5657585960616263, treating 2071 men with a mixture of conditions causing erectile dysfunction (Table 1). Of these, 632 received placebo and 1439 tadalafil, of whom all but 35 took licensed doses of 5 to 20 mg. All were placebo-controlled with no active comparator. Doses of 20 mg (1258) or 10 mg (109) were most frequent, with no dose-optimised studies. Quality scores were high, two trials scoring 5 of 5 points, one scoring 4 points, four 3 points, and one 2 points. One further trial 64 was a population dose-response study and was not included.

There were seven included studies with vardenafil 65666768697071, treating 3374 men with a mixture of conditions causing erectile dysfunction (Table 1). Of these, 1067 received placebo and 2307 vardenafil, all at licensed doses of 5 to 20 mg. Fixed dosing was most used with 10 mg (809) or 20 mg (698), together with some dose optimised studies of 5–20 mg (382). All were placebo-controlled with no active comparator. Quality scores were high, three trials scoring 5 of 5 points, and four scoring 4 points. One further trial 72 had no placebo group and enriched enrolment and was not included.

Details of the included studies are in additional files 1: conditions, country, treatment, dose, duration and quality score [see Additional file 1]; 2: efficacy outcomes, withdrawals, and adverse events (patients with any adverse event, and severe, serious and treatment-related adverse events) [see Additional file 2]; and 3: details of particular adverse events [see Additional file 3].

All studies provided background information on participants. Typically men had to have a history of erectile dysfunction of at least three to six months, and the average age of men was generally in the mid 50s or older. Some studies had an enriched enrolment in which previous unsuccessful treatment with a PDE-5 inhibitor was an exclusion criterion. This applied to five of eight tadalafil studies, and six of seven vardenafil studies [see Additional file 1], but none of the sildenafil studies. One vardenafil study 70 included only men previously unresponsive to sildenafil.

Studies almost always documented that PDE-5 inhibitor was to be taken as needed to a maximum of one treatment a day, with the additional instruction for sildenafil and vardenafil that the dose be taken about an hour before intercourse. Some studies gave information on the number of doses actually taken. Many of the dose optimised studies reported the proportion of men on maximum dose at the end of the study, which was always over 50%, and typically 60–80%. Trial duration was typically four to 12 weeks, with 12 weeks the commonest duration.

A number of different conditions causing erectile dysfunction were studied (Table 1). All three PDE-5 inhibitors had a similar proportion of men with mixed aetiology (organic, psychogenic, or mixed) and diabetes. Erectile dysfunction after prostatectomy was studied for tadalafil and vardenafil, but not sildenafil; none of a number of studies of sildenafil after prostatectomy could be included, mainly because they were not randomised. With sildenafil, a variety of additional conditions were studied, including depression, spinal cord injuries, multiple sclerosis, coronary heart disease, radiotherapy for prostate cancer, renal failure and haemodialysis patients, rectal surgery and spina bifida.

Many studies were multicentre, often performed in different countries (Table 2). Europe, North America and Australia accounted for most of the men in the studies for tadalafil and vardenafil. Sildenafil studies in addition were reported from South America, Asia, and Africa.

The percentage of men in trials about whom information was provided for various outcomes is shown in Table 3. Reporting of efficacy was mixed. Almost all trials used the outcome of improved erections, and most used the final score or change from baseline in the erectile function domain of the IIEF. Also commonly reported was the percentage of attempts at sexual intercourse that were successful, usually as an average. Final scores on questions 3 and 4 of the IIEF, or change from baseline, were reported in over 90% of sildenafil trials, but in less than 30% of trials of tadalafil or vardenafil.

Study withdrawal for any cause, or because of lack of efficacy or adverse events was commonly reported. The reporting of men with any adverse event, with severe or serious adverse events according to recognised adverse event reporting criteria, or those adverse events considered treatment related by investigators, was not consistent. Most studies reported specific adverse events only if they occurred in a certain proportion of men, usually between 2% and 5%. As a consequence, some adverse events (headache, dyspepsia, flushing, nasal congestion or rhinitis, or visual disturbance) were consistently reported while others were not.

The first stage was a detailed analysis of outcomes for each intervention, followed by a comparison between interventions where there was sufficient information to make the comparison valid. To make comparisons between different interventions, similar outcomes and similar intensities of intervention have to be compared in similar patients. Only those outcomes with a reasonably high and consistent reporting frequency were available for comparison.