Clinical practice guidelines are "systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances" 1. In many respects this is an ideal time to develop guidelines in the management of postmenopausal osteoporosis, since large randomized controlled trials of various therapies with clinically important outcome measures such as fractures have been conducted 2. In the past few years, numerous agencies and organizations have developed clinical practice guidelines (CPGs) for the management of post-menopausal osteoporosis. For the primary care physician, it is difficult to assimilate all the information contained in the numerous guidelines and decide which guideline to use in clinical practice. Recent research demonstrated that the application of two different guidelines 3 and 4 to a cohort of women who had sustained a fracture resulted in quite different estimates of the proportion of women who should be treated (87.9 versus 58.9%) 5.

CPGs may have the potential to improve the use of research findings and ultimately the care of patients, but only if the guidelines are associated with rigorous development. Special consideration should be given to the implementation of guidelines, including the needs of the users. A recent needs assessment conducted by Jaglal et al. revealed that family physicians wanted guidelines or information aids about osteoporosis that have significant input from family physicians 6. Additionally, with looming health care shortages, CPGs are increasingly used as the basis for the rationing of resources. It is therefore important that the guidelines are developed with methodological rigour, then appropriately disseminated and implemented 7.

The objective of this study was to identify and conduct a systematic assessment of the most recent postmenopausal osteoporosis guidelines produced from 1998–2001 that made recommendations on therapy. This process involved the systematic evaluation of the quality of the guidelines using a validated quality assessment tool 7.

We identified 26 unique guidelines from 1998–2001: 21 were retrieved and on assessment met our inclusion criteria 349101112131415161718192021222324252627. Five were excluded for the following reasons: three did not include clinical recommendations or were review articles 282930 and two were not retrievable 3132. Consultation with content experts identified seven guidelines, results of a Medline search identified four further guidelines; search of the World Wide web yielded three guidelines, and known guideline websites yielded six potential guidelines and review of references a further seven guidelines. Eight of the 21 guidelines reported receiving outside funding, and 6/8 of the guidelines acknowledged funding from industry.

Twelve guidelines were published, seven were organizational reports, and two were accessible only from the web. Guideline developers included medical societies (n = 8), national groups (n = 6), international groups (n = 1), and government agencies (n = 6). Table 1 presents the percentage of the guidelines that satisfied each of the 37 quality criteria. Figure 1 illustrates the distribution of the guidelines by dimension quality score. Twenty of the 21 guidelines did not include patients/consumers on the development panel. Only 11 of the 21 included family physicians.

Half or more of the 20 items assessing the rigor of guideline development were met by 15% of the guidelines (median quality score 23%, (95% CI 16.5, 34.7, range 5–80%), 81% met at least half of the 12 items assessing guideline content and context (median score 58%, (95% CI 50.8, 65.5), range 17–83%), and none met half or more of the items assessing guideline application (median score 0%, (95% CI -0.5, 12.6) range 0–47%).

Table 2 presents the mean dimension quality scores for the 21 osteoporosis CPGs. Eight guidelines described the method used to interpret and assess the strength of evidence. Six of the 21 guidelines provided an explicit link between major recommendations and supporting evidence.

The mean global quality score was 5.23 (95% CI 4.52, 5.94; median 5.33, standard deviation 1.56), suggesting that the appraisers felt the guidelines to be of medium quality.

The mean global assessment rating demonstrated that 11 of the 21 guidelines were recommended for use in practice (1 without change, 10 with modifications) and 10 guidelines were not recommended for use in practice.

Assessment of the reliability of the guideline appraisal instrument showed that D1 and D3 had good internal consistency, and D2 was borderline. The Cronbach's alpha was 0.87 for rigour of guideline development, 0.64 for context and content, and 0.71 for application.

For agreement between appraisers, the intraclass correlation coefficients were 0.91 for D1, 0.63 for D2 and 0.93 for D3. The appraisers' scores were within 20 percentage points for 95%, 67% and 100% of the guidelines for D1, 2, and 3, respectively. The Pearson's correlation coefficients between appraisers' dimension scores and their global quality scores (0–10) were 0.71 for D1, 0.60 for D2 and 0.46 for D3, with all coefficients being significant. The Pearson's correlation coefficients between appraisers' dimensions scores and their overall global assessment (recommend or not) were 0.59, 0.61 and 0.49 for D1, D2 and D3 respectively, with all coefficients being significant.

Eighteen of 21 guidelines included both pharmaceutical and non-pharmaceutical options (fall prevention, exercise or hip protectors). Ten of the guidelines included recommendations on fall prevention, and nine included recommendations on hip protectors.

There was no significant difference in the mean quality assessment when analyzed according to industry funded versus non-funded guidelines, type of publication (published, web or report), or the type of organization. There was no significant difference in quality according to year of publication, although there appeared to be a trend to improved quality from 1998 to 2001. Only 1/8 of the guidelines that acknowledged funding gave evidence that the potential biases were taken into account.

The methodological quality with respect to the rigor of development and implementation of current osteoporosis guidelines is low, although their scores with respect to clinical content are higher. Virtually no guidelines covered the application/dissemination issues. In particular, they did not define outcomes that could be monitored and failed to identify key elements that could be monitored by local guideline groups. Only one guideline was judged acceptable for use in its current format 4. Clinical practice guideline developers are usually clinicians, or methodologists who have little or no training in the behavioural and organization sciences. Tremendous time and effort are devoted to the development of evidence-based guidelines and the dissemination and implementation are often considered to be a separate component which has only recently thought to be the jurisdiction of the CPG developers.

Our findings are similar to other published reviews of guidelines 833. We also chose to limit our study to one clinical area and to focus on recent guidelines from 1998–2001, recognizing that guidelines currently being developed may follow a more rigorous methodology. Our results indicate that it is important to search both the Internet and Medline as some guidelines were only posted on the web and that retrieving all relevant guidelines can be a difficult task that requires a comprehensive search strategy.

Another potential limitation is that very few of the guidelines encouraged patient input into the development of CPGs. Potential benefits of including patients in the guideline development process include incorporation of priority setting from the patient perspective, improved adherence with recommendations, and reinforcement of the shared decision-making process.

Twenty-nine percent of the guidelines acknowledged involvement or support from industry. Choudry et al. conducted a survey of CPG authors and found 87% had some form of interaction with the pharmaceutical industry, with 58% receiving financial support for research. Given the heavy involvement of industry in the osteoporosis field, one would anticipate that the majority of the guideline developers would have relationships with industry that predated the guideline creation process. Industry has a vested interest in supporting the recommendations of osteoporosis guidelines, which raises concerns about industry's potential to influence the guideline development process. Acknowledgement of the degree of support/involvement with industry in guideline development is important and Choudry et al. 34 recommended that there be a formal process for discussing these conflicts prior to CPG development.

In addition to the goal of promoting good clinical practice and enhancing appropriate management, another potential goal of developing clinical practice guidelines is to improve the use of health care resources. However, we found that over two-thirds of guidelines neglected to include costs. The majority of guideline recommendations are based on results from efficacy, not effectiveness studies and therefore are not always useful in clinical practice 35. While the methodological quality of guidelines is a prerequisite to the development of good guidelines, it is not sufficient to ensure improvements in clinical practice. In addition, research methodologies on the impact of evidence-based guidelines in clinical practice would be valuable.

Educational material, including the dissemination of practice guidelines has been shown to have little or no effect on promoting behavioural change among health professionals 36. In contrast, strategies that encourage the development of a local guideline initiative have been shown to result in an increased uptake and dissemination. Only one of the osteoporosis guidelines that we reviewed encouraged development of a local initiative 4.

It is clear from the results of our systematic assessment that the developers of future osteoporosis guidelines need to incorporate greater methodological rigor and patient involvement into the development process. It would also be beneficial if guideline developers could follow common standard of reporting similar to that used by CONSORT for RCTs 37.

The multitude of guidelines dealing with postmenopausal osteoporosis makes it difficult for individual clinicians to decide which guideline to follow. Perhaps greater collaboration and sharing of resources between developer groups would result in fewer guidelines, but with local input as appropriate. More of the resources and energy could then be directed towards the dissemination process.

None declared.

AC participated in design of the study, evaluation of guidelines, data analysis, and drafting the manuscript. LW participated in the design, coordination of the study, evaluation of guidelines and commented on the paper. IG participated in design of the study, statistical analysis and writing of manuscript. MM participated in the evaluation of guidelines and provided feedback on the manuscript. AB participated in the evaluation of guidelines and provided feedback on manuscript. DSO participated in evaluation phase and was involved in preparation of the manuscript. All authors read and approved the final manuscript.