In contemporary medical research, randomised controlled trials are seen as the gold standard for establishing treatment effects where it is ethical and practical to conduct them. In palliative care, randomised controlled trials may be impractical, unethical, or extremely difficult, with multiple methodological problems. The fact and nature of these issues with palliative care trials has been frequently commented upon 123. Frequently encountered problems include recruitment and attrition, insufficient numbers of patients for any comparison, clinical heterogeneity between patients (condition palliated, comorbidity), heterogeneity in treatments (intervention, dose, duration), different outcomes reported, and use of non-standard scales. A palliative care Outcomes Working Group has recently made recommendations on outcomes they consider to be important in this context and how they might be sought in clinical trials 4.

Trials that have been done in palliative care are often small, diverse in nature and outcomes, and with high attrition rates, making meta-analysis, and even qualitative systematic review, impractical, unsatisfactory, or both. Moreover, some aspects of palliative care are difficult to capture, given the nature of palliative care as a person-centred approach, in which individual packages of care are often the norm 5.

With this background, the value of systematic reviews of randomized trials in palliative care might be questioned. One side of the argument would be that without a sufficiency of trials satisfying criteria of quality, validity, and size 6 systematic reviews are worthless. Another would see systematic reviews as a necessary first step to obtaining more evidence; despite their limitations, they at least tell us what we don't know, and may indicate how to improve.

This review set out to examine a subset of Cochrane reviews published under the auspices of the Pain, Palliative, and Supportive Care Review Group, to ascertain the number of successfully completed palliative care systematic reviews from this source over the last nine years, to assess their quality and the strength of the evidence presented to guide clinical practice.

A list of published reviews relating to palliative care and registered with the Cochrane Pain, Palliative and Supportive Care Group was obtained from the Review Group Coordinator as of December 2007. Copies of each review were obtained from the Cochrane Database of Systematic Reviews using the most recent upload, issue 1, 2008.

The following information was extracted from each review:

• Number of studies included

• Number of patients included

• Condition palliated

• Intervention

• Trial design (randomized, observational)

• Measures of quality and/or validity used

• Whether exclusions due to poor quality were made, or a sensitivity analysis presented

• Whether a pooled analysis was done

• Original authors' conclusion on efficacy

• Original authors' conclusion on strength of evidence

• Original authors' implications for future research.

Two reviewers (GH, SD) independently carried out data extraction, using a standard form, and assessed the quality of each review using the Oxman & Guyatt Index of Scientific Quality 7. To determine the strength of the evidence presented, for each review we assessed the quality of the included studies, based on randomization and blinding since these characteristics are known to affect potential bias 8, and the number of patients available for any analysis, because small numbers are prone to random error 910. Any discrepancies were resolved by consensus.

There was no prior intention to perform any statistical analyses. What was intended was an evaluation of this set of systematic reviews in palliative care based on the quantity and quality of primary studies available, and the quality of the review process itself, in order to determine their utility for informing clinical practice.

Details of the 25 published systematic reviews 11121314151617181920212223242526272829303132333435 are in Additional file 1, together with the conclusions of the original authors. The first of these Cochrane reviews was published in 1999, and the most recent in 2007. Sixteen of the reviews concerned drug interventions for pain or other reasons, three involved radiotherapy, three complementary therapy, and one each for a mineral supplement, supportive care, and pleurodesis. Only five of the studies were published before 2003, and the rate of publication was five per year since 2004.

This systematic review of systematic reviews in palliative care was to question the utility of systematic reviews for informing clinical practice in this area of medicine. It found that 25 reviews were published in the Cochrane Database of Systematic Reviews over nine years, a rate of about 2.7 per year overall, though almost double that rate occurred in the three years to 2007. Despite a respectable level of productivity from this prestigious source, 22/25 reviews could produce only weak evidence of the benefits of any intervention, and even of the two where the evidence was considered to be strong there were caveats.

The review processes themselves appeared adequate. Deficiencies lay in the primary studies, which were either missing or scant, or were characterized by heterogeneity in the methods, interventions, patients, and outcomes, which made an overall assessment of benefit or harm impossible. These deficiencies are similar to those identified previously 1235. The authors of the reviews commonly commented on these deficiencies, and others. The biggest single issue was that of inadequate trials or inadequate patient numbers in high quality trials. In making even this point, the reviews and the reviewers make an important contribution.

It is likely that these observations are general to systematic reviews in palliative care. We limited our investigation to reviews from the Cochrane Database published through the auspices of the Palliative Care group, but we would expect such reviews to be no worse, and perhaps better, than non-Cochrane reviews 4546. The restriction to Cochrane reviews should not limit any generalisability of these findings, especially as this reasonably sized body of reviews consistently makes the same, or very similar, points.

These findings are not a surprise. The dearth of good quality primary studies in the field of palliative care is widely accepted, and those trials that have been done are often known to have weaknesses 1235. Together, these factors underline the limitations of the knowledge base upon which palliative care has to draw. Whether new guidance about outcomes to be measured in palliative care trials would make a difference 4 remains to be seen, but given the difficulties in design and conduct of palliative care trials, rapid change in the corpus of evidence is unlikely.

The challenge for palliative care is the lack of evidence that is available to support it and the inordinate difficulties in obtaining evidence, for example difficulties with recruitment and attrition in an ill and vulnerable population. This has led to calls for a different framework for examining evidence 47. Part of the problem is that nearly all randomised controlled trials examine single interventions, while in clinical practice that intervention will often form a small part of a much larger overall package of care 5. Randomised trials of overall packages of care with small or incremental differences between them are unlikely to be able to measure small improvements, and an evaluation of systematic reviews of palliative care services 48 highlighted similar problems to those of palliative care interventions. High patient losses also make interpretation of randomised trials difficult. It is important that palliative care research moves away from dependence on randomized trials, and explores alternative study designs to identify the most effective treatments and packages of care for its patients.

There may be alternatives. Nearly all the Cochrane reviews included only randomised trials, and the small number of reviews that did consider non-randomised studies found them to have many of the same problems as randomised trials, with an additional increased risk of bias. We know, from other areas of medicine, that high quality, well-formulated, and impeccably conducted large observational studies, can provide equivalent results to those obtained from randomised controlled trials 64950. To overcome the play of chance these good quality studies need to be large, and to minimise bias they need to be both prospective and inclusive (i.e. a whole population, or all patients attending a clinic in a defined time). Registry studies are studies based on information from registers that systematically record information from all individuals in a defined population. They can be entire populations, as in the death register in the UK, or all patients with a specific characteristic (eg twins) or condition (eg breast cancer) within a defined population. At least one large registry-based programme for continuous quality improvement aimed at cancer pain is ongoing in Italy 51. An extensive search for observational studies in palliative care has been undertaken, with the aim of identifying good quality observational studies and aspects of their design that make them reliable and useful (Hadley et al., manuscript in preparation). The proven limitation of controlled trials in palliative care may make registry studies a more acceptable option in future.

Cochrane reviews in palliative care are well performed, but fail to provide good evidence to guide clinical practice because the primary studies are few in number, small, clinically heterogeneous, and of poor quality and external validity. These reviews do, however, tell us how limited the evidence base is, and highlight common deficiencies in primary studies. There are well-documented problems with conducting valid randomised trials in this area, and it may be that for some questions more, and more clinically relevant, information can be obtained from other types of primary study, such as large registry studies.

BW and SD have received research support from charities, government and industry sources at various times, but no such support was received for this work. No author has any direct stock holding in any pharmaceutical company.

GH and SD were involved with planning the study, data extraction, analysis, and preparation of the manuscript. BW was involved with planning the study and preparation of the manuscript. All authors read and approved the final manuscript.