Efforts to develop an effective pneumococcal vaccine date from the beginning of the last century. Polyvalent pneumococcal polysaccharide vaccine has now been available for many years, yet controversy persists as to its clinical efficacy [1,2]. At the time the vaccine was licensed there were only two published randomised trials, both carried out in unique populations of young, healthy people at extraordinarily high risk of pneumococcal disease. The dearth of randomised trials led to several retrospective studies using case-control and indirect cohort methods [3]. Although these types of studies can provide useful data on the efficacy of a vaccine, they are limited by the problems inherent in these methods. Furthermore, the available retrospective studies have examined different outcomes, by different methods, and reached substantially different conclusions regarding the vaccine's efficacy for various subgroups of patients at risk for pneumococcal infection.

Since the current pneumococcal vaccine was licensed in 1979 several trials have been performed on populations more representative of those for whom the vaccine is recommended in the Western world. Thus an alternative way to examine the value of the vaccine is to conduct a meta-analysis of the available randomised trials. One such meta-analysis published in 1994 included nine randomised trials conducted in adults with vaccines of 12 to 17 valencies [4]. The authors concluded that pneumococcal vaccination significantly reduced the risk of definitive (or bacteraemic) pneumococcal pneumonia, but only in low-risk populations, i.e., those younger than 55 years old and without chronic medical or immunosuppressing conditions. Vaccination did not reduce the incidence of pneumonia of all causes, bronchitis, mortality due to pneumonia or pneumococcal infection, or mortality of all causes. The inconsistency between the lack of efficacy of the vaccine in elderly and high-risk patients and the nearly universal recommendations for its use in those populations is obvious. This conflict has important implications for both individual clinicians and health policy organisations. A subsequent meta-analysis [5] of 13 studies published up to 1986 included three quasi randomised studies. It concluded that pneumococcal vaccination was effective, but that 2520 people would have to be vaccinated to prevent one case of pneumococcal bacteraemia per year.

Although the price of an individual dose of vaccine is relatively low the aggregate financial and administrative costs of providing pneumococcal vaccination (and often re-vaccination) to the many subgroups of patients for whom it is currently recommended is substantial. Unlike in the United States, the UK Department of Health's recommendations for pneumococcal vaccination excludes those at risk through age alone [6]. Some advocate extending the policy to include universal vaccination of everyone aged 65 years and over [7]. Before additional efforts are made to encourage increased use of this vaccine it is crucial to try to determine its value. Several additional randomised trials, two with the newer 23-valent vaccine, have been conducted in elderly or high-risk populations. We therefore conducted another meta-analysis of all available randomised trials of polyvalent pneumococcal vaccine to evaluate its efficacy in populations that are immunocompetent and those that are likely to have an impaired immune system.

We attempted to locate all randomised comparisons of pneumococcal vaccines with placebo in any type of population. Databases searched included the Cochrane Library, (issue 2, 2000), MEDLINE (1966-August 2000), PubMed (to August 2000) and EMBASE (to August 2000) using "Pneumococcal", "vaccine", and variations on these as free text terms. Abstracts were examined and copies of qualifying papers (any language) were obtained and read by all authors. Reference lists from papers obtained and two previous overviews [4,5] were also examined.

To be included in the analysis, a study had to have been a prospective randomised comparison of a polysaccharide pneumococcal vaccine (any valency) and to have a placebo or no treatment comparison group. Because of the potential for bias, a prior decision was taken not to include studies with quasi-randomised design (alternate, date of birth etc). Papers had to report at least one clinical outcome of interest, such as rates of pneumonia, pneumococcal pneumonia, lower respiratory tract infections, pneumonia deaths or bacteraemia. The intention was to analyse outcomes according to whether vaccine recipients were likely to have healthy or impaired immune responses for whom pneumococcal vaccination is indicated in guidelines. Serological outcomes were not sought.

Each report was read independently by all authors and scored using a validated three-item quality scale [8]. Discussion and review achieved consensus. The maximum score for an included study was 5 and the minimum score was 1.

Relative benefit and relative risk estimates were calculated with 95% confidence intervals using a fixed effects model [9] because of the unreliability of statistical tests of heterogeneity [10]. Number-needed-to-treat with 95% confidence intervals was calculated by the method of Cook and Sackett [11]. A statistically significant difference from control was assumed when the 95% confidence interval of the relative risk did not include 1. Calculations were performed using Excel98 on a Power Macintosh G3.

We found 12 reports of 13 randomised comparisons of polyvalent polysaccharide pneumococcal vaccines for inclusion with a total of 45,226 subjects ([12,13,14,15,16,17,18,19,20,21,22,23] Table 1). The report by Austrian et al, 1980 [15] comprised two separate trials. All but one [16] of the reports was in English. Eleven papers were published in scientific journals and one was available only as final report to the US National Institute of Health [15]. Four reports were excluded, three (with 54,863 patients) had a quasi-randomised design [24,25,26] and one [27] described the use of pneumococcal vaccine in children for protection against acute otitis media.

Three comparisons [12,13,14] reported outcomes in 21,152 healthy, mostly young, adults likely to have a normal immune system. Ten [15,16,17,18,19,20,21,22,23] reported outcomes on 24,074 people likely to have an impaired immune system because of age or ill health. Because of the potential importance of a competent immune system in determining the efficacy of pneumococcal vaccines, we divided the studies by this criterion.

The comparisons used a number of different pneumococcal polysaccharide vaccines. Five [12,13,14,15] administered vaccines that were not licensed. Vaccine valencies varied; one used a 6 or 12-valent vaccine, two a 12-valent, six a 14-valent, one a 15-valent, one a 17-valent and two a 23-valent vaccine. The follow-up period for outcomes was a minimum of one year and a maximum of five years; most were two or three years (Table 1). Quality of comparisons was mixed; one scored 5 points, three scored 4, four scored 3, four scored 2 and one scored 1. The major flaw was in lack of double blinding in six comparisons (Table 1).

In the Western world pneumococcal vaccine is almost universally recommended for adults who are at high risk for infections caused by Streptococcus pneumoniae because of an immuno-compromised state or certain diseases. Furthermore, in many of the nine countries in which it is used it is recommended for all elderly persons. This is in spite of the fact that most recent publications allow that "results of published controlled trials of pneumococcal vaccination in people with risk factors for pneumococcal infection are unclear" [7]. Several, but not all, retrospective studies suggest that it may be effective in preventing invasive pneumococcal disease, at least in immunocompetent people. In 1994 the results of prospective randomised trials suggested the lack of efficacy in most Western populations [4]. Despite this, another systematic review [5] using randomised and quasi-randomised studies, and making no distinction as to types of patients likely to be vaccinated in Western populations, concluded that pneumococcal vaccination was effective. Even so, 2520 older people would need to be vaccinated to prevent one case of bacteraemia at one year [5]. In their sensitivity analysis, excluding studies both quasi-randomised and unblinded studies, statistical significance was lost.

Three additional trials [21,22,23] have been published since 1996 (the date of the last search of the most recent review was November 1996 [5]), and we have included two studies missed by the previous review [19,20]. Improper randomisation has been shown consistently to lead to bias [28,29,30]. We chose not to include three large studies, both old [24,25] and new [26], that were quasi randomised. The two quasi-randomised studies from the 1930s and 1940s (26,000 patients) concluded the vaccine was effective. A recent quasi-randomised study in 27,000 over 65s in Finland found pneumococcal vaccination ineffective.

Though it is alleged that the vaccine is inexpensive, the cost to the NHS in England in 1999 was still a minimum of £5.4 million in prescription cost alone [31]. Additional expense, time and effort are required to implement vaccination policies. Obviously these costs would increase with expanded use of the vaccine. Much emphasis has been placed on the possible beneficial effects of pneumococcal vaccines in reducing pneumococcal bacteraemia. This is a rare event, occurring in about 7/100,000 of the general population [32]. Yet the data from prospective trials in high-risk patients show no statistically significant decrease in pneumococcal bacteraemia in vaccinated patients (0.8% versus 1.4%).

The modern pneumococcal vaccine has now been licensed for two decades, yet the debate over its efficacy persists [1,2]. Part of the reason for the controversy is that the current vaccines were not subjected to randomised controlled trials before their release. It is therefore crucial, despite assertions to the contrary [33] that the new protein-conjugated pneumococcal vaccines are properly evaluated.

Some advocates of the vaccine argue that logistical difficulties, great expense, and ethical concerns make randomised trials unsuitable for resolving the pneumococcal vaccine controversy [34]. They suggest that properly conducted non-experimental studies offer attractive alternative strategies. But all observational designs are weakened by their reliance on allocation of vaccine by physician and patient preference. They also suffer from the potential for prognostic susceptibility bias, including inequalities of unknown risk factors, and the degree of reliability and availability of information on risk factors. Other sources of bias include those related to exposure-ascertainment, migration, and detection. Furthermore, all non-RCT designs require analytic assumptions to translate results into estimates of protective efficacy. The change from statistically significant results for pneumococcal vaccines when quasi-randomised trials were excluded [5] and the controversy over mammography trials [30] are reasons enough why properly conducted randomised trials are needed.

Even advocates of the pneumococcal vaccine agree, however, that randomised controlled trials are the most powerful design for preventing bias and estimating protection by the vaccine directly, without the need for assumptions [34]. They contend that the prospective controlled trials have been "inconclusive," and that failure to demonstrate efficacy should not be taken as evidence that the vaccine lacks efficacy [2]. There have been 13 RCTs of the multivalent pneumococcal vaccine, and the meta-analysis technique affords a means of combining the data on the 45,295 subjects enrolled in these studies. Our analysis of the available randomised suggests that the trials should be divided by the type of patients enrolled. Three studies have shown that the vaccine is effective (with low numbers needed to treat) in unique populations of healthy young adults at high risk for pneumococcal infection, such as South African gold miners and New Guinea highlanders. These subjects are capable of mounting a vigorous antibody response to the polysaccharide vaccine. They are also exposed to respiratory irritants (mining dust or fireplace smoke), which increases the risk of developing pneumonia. Finally, they share close living and sleeping quarters, which enables easy spread of a virulent strain of Streptococcus pneumoniae in the community [35]. This set of unusual circumstances is ideal for demonstrating the potential efficacy of the pneumococcal vaccine. Most pneumococcal infections are, however, sporadic; outbreaks in the antibiotic era caused by a single serotype are rare [36].

In more commonly encountered circumstances the vaccine is not effective. Adults with immunosuppressing conditions, and many with chronic medical disorders, are unable to mount an adequate antibody response to the vaccine [7]. Antibody response and vaccine efficacy are also reduced in the elderly [37],especially after age 75 [38], and wane more quickly [39]. Furthermore, since pneumonia in the elderly is usually caused by aspiration of oropharyngeal secretions, pneumococcal vaccine may prevent infection with S. pneumoniae but not pneumonia of other causes. Thus in non-epidemic situations the vaccine may be less effective in preventing pneumococcal infections. Even if the vaccine were effective, vaccinating millions of people in the UK in the hope of preventing perhaps 60% of the 7 cases of pneumococcal bacteraemia/100,000 [32] persons is of dubious value.

In the absence of strong data supporting the efficacy of pneumococcal vaccine, advocates have based their recommendations on the fact that the vaccine is relatively inexpensive and safe. Vaccination routinely causes discomfort at the injection site for a few days, and in perhaps 5-10% of patients this may be sufficient to interfere with daily activities [40]. The results of several of the RCTs, especially that in HIV-infected Ugandans [23] suggest that the vaccine may in fact have adverse effects. If all of the elderly persons and patients with various chronic medical conditions and immunosuppressing diseases that are targeted were to be vaccinated, the aggregate financial and administrative costs would be great. Moreover, since vaccine protection is thought to last only about five years (and perhaps 2-3 years in patients at highest risk), re-immunisation would be necessary for most recipients [35]. This has lead some to recommend monitoring vaccinees' antibody response a few weeks after administration, and annually thereafter, to ensure adequate response and to determine the appropriate time for revaccination [35]. This would greatly add to the cost of a vaccination program, and any potential benefits would have to be balanced against the known harm of vaccination and revaccination [40,41].

In the end it comes down to this: prospective randomised trials in the types of patients targeted in the Western world show the vaccines to be ineffective while retrospective studies show that it may be effective. If ten studies involving over 24,000 high-risk individuals fail to show any benefits, then programmes for mass pneumococcal vaccination need to be re-thought.