Comparing different interventions for the same condition is often difficult. Large direct comparisons are uncommon, and usually represent only a fraction of the total randomised trial data available. Instead we often have large numbers of randomised trials comparing different interventions with the same or similar comparators, like placebo or an active comparator. These may be direct comparisons, but not the direct comparisons we want.

Indirect comparison of interventions using a common comparator is a valuable alternative because it uses more of the published data 1, and has been done before for analgesics in acute pain 2 and migraine treatments 3. Even this approach can be devalued because clinical trials in published papers are not consistent in the outcomes they report, or the way they report them 4.

The ideal should be to compare interventions of equivalent intensities or dose, in the same condition, at similar disease severity, using the same outcomes properly reported, over the same period of time. In this study we use the example of published studies of phosphodiesterase-5 (PDE-5) inhibitors for male erectile dysfunction to examine the problems of comparing therapies. PDE-5 inhibitors were chosen because they have been introduced within the past decade, by three different pharmaceutical companies, and in an era of good clinical trial practice. There are no good quality trials directly comparing PDE-5 inhibitors at equivalent doses.

We sought randomised trials of three PDE-5 inhibitors (sildenafil, tadalafil, vardenafil), with placebo or active comparator, in men with erectile dysfunction of any causation. Previous systematic reviews 56789 were used to source trials or trial data, supplemented by electronic searches of PubMed (to June 2005) and the Cochrane Library (issue 1, 2005) using drug names and randomis(z)ed trial.

For inclusion a trial had to be randomised and double blind, use one of the three oral PDE-5 inhibitors in men with erectile dysfunction, be conducted in the home setting, last three weeks or longer, have a minimum of 10 men per group, and report useful information on efficacy or adverse events. Abstracts were read, and potentially useful reports retrieved in full paper copy. Decisions on inclusion or exclusion were made by consensus. No information was taken from posters or abstracts, and studies were read carefully to avoid including duplicate material. Studies were scored for reporting quality using a common method 10 utilising reporting of randomisation, blinding and withdrawals. The maximum score possible was 5 points, and no study could be included with fewer than 2 points.

Information extracted from studies included the number of men studied, the cause of erectile dysfunction, and country where the study was performed. Any type of outcome was initially extracted from the studies, in continuous or dichotomous form, and with any dispersion information available. Outcomes could be reported in tables, in graphs, or in text. The following outcomes were sought particularly:

There were 35 included studies using sildenafil 2021222324252627282930313233343536373839404142434445464748495051525354, treating 7135 men with a mixture of conditions causing erectile dysfunction (Table 1); one study 40 was analysed as two trials. Of the men, 3279 received placebo and 3856 sildenafil, of whom all but 90 took licensed doses of 25 to 100 mg. Optimised dosing with 25–100 mg was most used (2546 men), followed by 100 mg (506) or 50 mg (370). All were placebo-controlled with no active comparator. Quality scores were high, 11 trials scoring 5 of 5 points, 17 scoring 4 points, six 3 points, and two 2 points. One further study 55 used a withdrawal model after successful treatment, and information from the 205 men in this study was not included in this analysis.

There were eight included studies using tadalafil 5657585960616263, treating 2071 men with a mixture of conditions causing erectile dysfunction (Table 1). Of these, 632 received placebo and 1439 tadalafil, of whom all but 35 took licensed doses of 5 to 20 mg. All were placebo-controlled with no active comparator. Doses of 20 mg (1258) or 10 mg (109) were most frequent, with no dose-optimised studies. Quality scores were high, two trials scoring 5 of 5 points, one scoring 4 points, four 3 points, and one 2 points. One further trial 64 was a population dose-response study and was not included.

There were seven included studies with vardenafil 65666768697071, treating 3374 men with a mixture of conditions causing erectile dysfunction (Table 1). Of these, 1067 received placebo and 2307 vardenafil, all at licensed doses of 5 to 20 mg. Fixed dosing was most used with 10 mg (809) or 20 mg (698), together with some dose optimised studies of 5–20 mg (382). All were placebo-controlled with no active comparator. Quality scores were high, three trials scoring 5 of 5 points, and four scoring 4 points. One further trial 72 had no placebo group and enriched enrolment and was not included.

Details of the included studies are in additional files 1: conditions, country, treatment, dose, duration and quality score [see Additional file 1]; 2: efficacy outcomes, withdrawals, and adverse events (patients with any adverse event, and severe, serious and treatment-related adverse events) [see Additional file 2]; and 3: details of particular adverse events [see Additional file 3].

All studies provided background information on participants. Typically men had to have a history of erectile dysfunction of at least three to six months, and the average age of men was generally in the mid 50s or older. Some studies had an enriched enrolment in which previous unsuccessful treatment with a PDE-5 inhibitor was an exclusion criterion. This applied to five of eight tadalafil studies, and six of seven vardenafil studies [see Additional file 1], but none of the sildenafil studies. One vardenafil study 70 included only men previously unresponsive to sildenafil.

Studies almost always documented that PDE-5 inhibitor was to be taken as needed to a maximum of one treatment a day, with the additional instruction for sildenafil and vardenafil that the dose be taken about an hour before intercourse. Some studies gave information on the number of doses actually taken. Many of the dose optimised studies reported the proportion of men on maximum dose at the end of the study, which was always over 50%, and typically 60–80%. Trial duration was typically four to 12 weeks, with 12 weeks the commonest duration.

A number of different conditions causing erectile dysfunction were studied (Table 1). All three PDE-5 inhibitors had a similar proportion of men with mixed aetiology (organic, psychogenic, or mixed) and diabetes. Erectile dysfunction after prostatectomy was studied for tadalafil and vardenafil, but not sildenafil; none of a number of studies of sildenafil after prostatectomy could be included, mainly because they were not randomised. With sildenafil, a variety of additional conditions were studied, including depression, spinal cord injuries, multiple sclerosis, coronary heart disease, radiotherapy for prostate cancer, renal failure and haemodialysis patients, rectal surgery and spina bifida.

Many studies were multicentre, often performed in different countries (Table 2). Europe, North America and Australia accounted for most of the men in the studies for tadalafil and vardenafil. Sildenafil studies in addition were reported from South America, Asia, and Africa.

The percentage of men in trials about whom information was provided for various outcomes is shown in Table 3. Reporting of efficacy was mixed. Almost all trials used the outcome of improved erections, and most used the final score or change from baseline in the erectile function domain of the IIEF. Also commonly reported was the percentage of attempts at sexual intercourse that were successful, usually as an average. Final scores on questions 3 and 4 of the IIEF, or change from baseline, were reported in over 90% of sildenafil trials, but in less than 30% of trials of tadalafil or vardenafil.

Study withdrawal for any cause, or because of lack of efficacy or adverse events was commonly reported. The reporting of men with any adverse event, with severe or serious adverse events according to recognised adverse event reporting criteria, or those adverse events considered treatment related by investigators, was not consistent. Most studies reported specific adverse events only if they occurred in a certain proportion of men, usually between 2% and 5%. As a consequence, some adverse events (headache, dyspepsia, flushing, nasal congestion or rhinitis, or visual disturbance) were consistently reported while others were not.

The first stage was a detailed analysis of outcomes for each intervention, followed by a comparison between interventions where there was sufficient information to make the comparison valid. To make comparisons between different interventions, similar outcomes and similar intensities of intervention have to be compared in similar patients. Only those outcomes with a reasonably high and consistent reporting frequency were available for comparison.

This analysis comprised 50 randomised comparisons of PDE-5 inhibitors with placebo, in more than 12,000 men. Trials included were all randomised and double blind, and almost all (47/50) were of sufficiently high quality (score of 3 or more out of 5) to avoid major known sources of bias 73.

There was no direct comparison between one PDE-5 inhibitor and another that satisfied our inclusion criteria. Three studies did make a direct comparison between sildenafil and tadalafil, with no placebo. They are not included in the analysis because one 74 was open, another 75 was not convincingly double-blind, and the third 76 used only four tablets per patient. All three studies looked at duration of effect, and two looked at patient preferences. In these less than adequate trials, both drugs seem to be equally effective up to 12 hours after dosing, and both drugs were well tolerated.

Published reports of three PDE-5 inhibitors for treatment of male erectile dysfunction, while largely similar, had interesting differences. These were principally in the underlying aetiology of erectile dysfunction, regions of the world where studies were performed, and in exclusion criteria used to select men for the individual trials. Sildenafil was studied in 10 different clinical conditions, compared with only three for tadalafil and vardenafil (Table 1). The only condition missing for sildenafil was after prostatectomy, where studies could not be included because they were not randomised. Most studies were performed in North America and Europe (Table 2), though sildenafil was studied in men in every inhabited continent, with 27% of men studied being in Asia, South America, or Africa.

Perhaps the most obvious difference between trials of different drugs was the use of different exclusion criteria in individual studies. Five of eight tadalafil studies, and six of seven vardenafil studies excluded men previously unresponsive to PDE-5 inhibitors, thus permitting enrolment to be enriched by responders compared with sildenafil studies, in which such an exclusion would not have been used because it was the first available PDE-5 inhibitor. It is not clear how this major difference might have affected the measured performance of the drugs. Exclusion of non-responders to other PDE-5 inhibitors might be expected to enhance the measured performance of any other PDE-5 inhibitor under test, making it look better in indirect comparisons. However, one vardenafil study including men previously unresponsive to sildenafil 70 was not greatly different from those overall.

The other major difference was in the reporting of outcomes of studies, which varied greatly between the three PDE-5 inhibitors. Common outcomes were responses to a global question about improved erections ("Has the treatment you have been taking over the past four weeks improved your erections?"), and scores and change in score for the erectile function domain. Some outcomes were frequently reported in trials of one treatment, but not others. For example, final scores and changes from baseline for IIEF questions 3 and 4, were reported for most sildenafil, but not tadalafil or vardenafil trials.

There were clear differences in the philosophy of reporting of efficacy and harm, principally between the first PDE-5 inhibitor sildenafil, and the subsequent ones, tadalafil and vardenafil (Table 3). We could find no explanation for this, nor any philosophical discussion about the clinical and practical importance of different outcomes. Few studies provided an estimate of how many men had an outcome approximating erections sufficiently rigid for penetration followed by successful intercourse. This simple pragmatic outcome is more relevant to affected men, their partners, and their professional advisers than an average movement on a scale like IIEF, which itself comprises a number of different questions. Useful outcomes, like the number of men in whom the proportion of successful attempts at sexual intercourse was more than 40%, or 60% or more, known to be recorded in clinical trial reports for sildenafil 6, were almost never reported in published papers. Incomplete reporting of efficacy outcomes has been reported before, for chronic 4 and acute 77 pain.

Withdrawal from studies for any cause, because of lack of effect, or because of adverse events was commonly reported, while numbers of men with at least one adverse event and occurrence of severe adverse events, was recorded for a minority of trials. Serious adverse events were recorded more frequently in tadalafil and vardenafil trials than sildenafil trials, perhaps reflecting recency of studies. The use of a cut-off level for reporting individual adverse events limited the available information for these outcomes.

Analysis of the individual PDE-5 inhibitors (Tables 4, 5, 6, 7) showed that the two top doses (including dose-optimisation schedules) had very similar efficacy for all three interventions. This consistency justified pooling information from the two top doses.

Using this strategy to compare the three PDE-5 inhibitors demonstrated remarkable consistency between them on the basis of available data for any commonly reported outcome. Absolute rates for placebo varied little, showing no major difference between patients studied. The only exception to this was for withdrawals, where tadalafil and vardenafil studies had higher withdrawal rates with placebo and PDE-5 inhibitor than did sildenafil. There was no obvious reason for this. It was perhaps surprising that sildenafil compared well with tadalafil and vardenafil given the much greater number of conditions studied, the wider geographical spread, and that tadalafil and vardenafil studies used a form of enriched enrolment. Overall, sildenafil offered most information, and a trend towards better efficacy and lower adverse events.

How useful is this method of indirect comparison of equivalent doses of PDE-5 inhibitors? It is clearly superior to examining the little inadequate information on direct comparison, and to any superficial examination of individual trials, subject, as each will be, to the random play of chance 13. Large collections of data from high quality, valid, trials are less subject to the vagaries of chance than smaller individual studies. Large direct comparisons may be better, but the evidence is that when large amounts of trial data exist, direct comparisons give no different result from indirect comparisons 1. Further investigation of differences between comparable doses would require access to detailed clinical trial reports to report outcomes available but not published.

More important than direct or indirect comparison of different PDE-5 inhibitors is the question of utility of outcomes. With over 12,000 men in clinical trials over a decade, it is surprising that we have no consensus of what is a useful outcome, and how a useful outcome is reported. Individual patient analysis of clinical trial data illustrates how standard trials could be better reported 677, and this has already been done for PDE-5 inhibitors 6.

RAM & HJM have received lecture fees from pharmaceutical companies. All authors have received research support from charities and government sources at various times. This work was supported by an unrestricted educational grant from Pfizer Ltd. The terms of the financial support from Pfizer included freedom for authors to reach their own conclusions, and an absolute right to publish the results of their research, irrespective of any conclusions reached. Pfizer did have the right to view the final manuscript before publication, and did so. No author has any direct stock holding in any pharmaceutical company.

RAM was involved with planning the study, data extraction, analysis, and preparing a manuscript; SD with data extraction, analysis, and writing; HJM with planning, analysis and writing. All authors read and approved the final manuscript.