The objective of guidelines development is to assist physicians and patients in making optimal health care decisions, which in turn should improve the quality of clinical practice 1.

Different methods are used to develop guidelines. Some are developed by a consensus of experts while others also use a formal way to appraise the literature and create evidence-based (EB) guidelines. In general, evidence-based guidelines are considered to provide better recommendations for practice than consensus-based guidelines but are time consuming and expensive to create 23. This belief that EB guidelines are superior to other types of guideline is based on our normative views of methods for guidelines development 4 and not on empirical comparison of practice recommendations using different methods for development of guidelines. To date no formal evaluation has been performed to detect if there are differences in the quality and recommendations between evidence-based and consensus-based (CB) guidelines.

If guidelines developed by using consensus or evidence-based methods have the same quality and agree in the recommendations, then obviously resources spent on the laborious and time-consuming process of locating and appraising evidence can be used elsewhere. Otherwise, if evidence based guidelines have a better quality and their recommendations differ from those guidelines produced by consensus, then creation of evidence based guidelines may become the only acceptable method of guideline development.

In this paper, we explore if there are differences in the quality and recommendations between EB and CB guidelines.

To enable meaningful comparison, multiple recommendations produced by a given guideline method should be available. This objective is best met by focusing on the guidelines that comprehensively attempt to guide clinicians in the management of one disorder. Since breast cancer is an important disease and various organizations have produced guidelines using different methods 567, we conducted a comparison study of comprehensive breast cancer guidelines. We assessed both the differences in the quality as measured by using different quality instruments assessment and the level of agreement among guidelines according to the method of development.

1. Identification and assessment of instruments for measurement of the quality of guidelines

Since there is no uniformly accepted instrument for evaluation of the quality of guidelines, we first performed a comprehensive literature search to identify published tools for assessment of clinical practice guideline quality. We searched MEDLINE (1996–2000) using the keywords: guidelines, practice guidelines, quality, "weights and measures", "scale", psychometrics, reproducibility. Any article considered relevant to evaluate quality of guidelines was retrieved. The list of references of each article was also scanned. After an assessment of 14 papers by four of us, four instruments to assess the quality of guidelines were identified 89101112. An additional instrument (SIGN) was identified through Evidence-based Health Discussion Group (Table 1). For additional details on the instruments for evaluation of guidelines readers are referred to the Appendix (see Additional file). To assess their reliability and reproducibility, we applied all identified instruments to each guideline (see below). We calculated the coefficient of agreement (kappa) among evaluators for each guideline 13. A good interobserver agreement was considered if kappa value exceeded 0.4 13. In our evaluation, two instruments 1012 had a kappa interobserver agreement K > 0.4 among all investigators in 6 of 8 guidelines (Table 1). When it comes to evaluation of the quality of breast cancer guidelines these instruments 1012 performed better than others and probably can be recommended for future use.

2. Identification and classification of breast cancer guidelines

A literature search was conducted for published breast cancer guidelines using MEDLINE for the years 1996 – April 2000. The following keywords were used in combination: Guidelines, Practice Guidelines, recommendations, breast neoplasms. An Internet search was also performed, using the method described by Sanders et al 11. 131 articles were retrieved, and reviewed for their content. We considered any article that fit the definition of the National Library of Medicine for practice guidelines: directions or principles presenting current or future rules of policy for the health care practitioner to assist him in patient care decisions regarding diagnosis, therapy, or related clinical circumstances 14. Eight papers referred to breast cancer guidelines 567,1516171819 and were selected for the analysis.

Each guideline was classified as CB, when there was no consideration about the quality of evidence used to make practice recommendations; as EB, when there was an explicit consideration of the quality of evidence in the development of guidelines; or as consensus based with no explicit consideration of evidence (CB-EB) when there were considerations about the evidence, but not in explicit manner. From these eight guidelines, three were classified as EB 17716 three as CB-EB 15186 and two as CB 195 (Table 2).

3. Evaluation of guidelines

Each guideline was evaluated independently by three of us using each of the instruments. All discordances were resolved by a consensus meeting. Each guideline was scored according to the instructions of each instrument. The quality and rank was determined by the quotient of items scored positively by the total items scored for each instrument.

4. Evaluation of agreement among guidelines

Using instruments to evaluate practice guidelines yields conclusions regarding normative aspects of the guidelines development 1, but does not necessarily mean that recommendations provided by guidelines using different methods will produce different management advice to our patients. To assess if recommendations among various guidelines differ, we need to determine the level of agreement among guidelines for each specific decision point.

Since NCCN (National Comprehensive Cancer Network) guidelines 6 were presented in explicit, algorithmic format, we used this one to identify the decision points for matched comparison with other guidelines. These guidelines have been developed by the leading 18 cancer institutions in the US and have been constantly updated and re-evaluated. They have also been developed to closely mimic clinical practice. Therefore, we feel that selection of decision points based on the NCCN guidelines were appropriate. We identified fourteen decision points in the management of stage I and II breast cancer that were linked to specific recommendations in the other guidelines for our comparison. Comparison of recommendations for advanced stages of breast cancer has not been performed since there was only one guideline that included it 6.

Subsequently, four of us evaluated each of these decision points in each guideline examining level of agreement among various guidelines. Since matching between recommendations in the guidelines that were presented in non-algorithmic format was poor, we decided to use NCCN guidelines as a benchmark. We classified agreement of each guideline with the NCCN guidelines as having full agreement, partial agreement and disagreement. It was considered that guidelines agree with the NCCN if the management recommendation was the same; the guidelines were considered to disagree if they provided different recommendations. A partial agreement was judged to exist if the guideline recommended the same management but in a broadly defined sense and not in explicit, clear manner.

Each of these decision points was also classified as supported by high quality evidence or not. High quality evidence was considered to be based on randomized trials (RCT) or systematic reviews (SR)/meta-analysis (MA). If the quality evidence was not based on RCT or SR/MA or was not stated, it was classified as low quality evidence.

Subsequently, we performed a regression analysis to assess the contribution of the quality of evidence to the total score obtained by each instrument for the evaluation of the guidelines quality. Independent variable was the proportion of decisions supported by high quality evidence while dependent variable was score obtained by each instrument. A regression analysis was performed after it has assessed that the distribution of the variables was normal by Wilks-Shapiro test.

Guidelines have been increasingly used in medical decision-making. Different methods have been used in guideline development. Does it matter how guidelines were produced? Most authors believe that it matters very much 3 and that guidelines produced using evidence-based methods are superior to other methodologies of development 249. However, empirical investigations to assess if guidelines produced by different methods have different quality and result in different recommendations have not been performed. Here, we report such a study.

Using formal instruments for evaluation of the quality of guidelines we found that EB-guidelines had substantially higher score than CB-guidelines or guideline that considered evidence in a less formal way (CB-EB). As discussed above (see Results), this is not a surprising result, since the instruments for the guidelines evaluation measure the quality based on the number of desired normative characteristics in a particular guideline. Since appraisal of evidence is considered inherently important for the development of a good guideline, one would then expect that the guidelines that pay more attention to its evidence basis (i.e., those that are evidence-based) would receive higher quality score than other types of the guidelines (i.e. guidelines developed solely by a consensus process) (see Fig 1). This is also evident in our finding that variation in the total quality score can be up to 94% explained by the quality of evidence (see Fig 2).

Not all instruments for evaluation of guidelines performed equally well. Only two of the instruments available to address the quality of guidelines had a good level of agreement among evaluators (k > 0.4) in most of guidelines. This result raises concern about the reproducibility of results using the other instruments reported in the literature. In general, a few studies have been done to evaluate reproducibility of the instruments for assessment of the guidelines quality. Any future study attempting to address the quality of guidelines should take this finding into account.

A more interesting question is to assess if the recommendations among guidelines produced by different methods actually differ. We found no instance of total disagreement among guidelines regardless of the method of development. We also found that EB and CB-EB guidelines had more points of agreement with our benchmark guidelines (NCCN)6 than guidelines developed using exclusively consensus method. We also found that when high-quality evidence existed in the literature (see Results) less disagreement was found among various guidelines. This is not completely surprising because formulation of guidelines does not happen in a vacuum. Most guideline developers are experts in the field who have knowledge of the literature. When evidence is unequivocal, less disagreement may be expected. Consequently, less practice variation may be found when high-quality evidence exists.

In conclusion, EB guidelines have a better quality than CB guidelines as measured by the quality assessment instruments used in this study. The explicit use of high quality evidence is desirable and can lead to a better agreement among recommendations. However, no major disagreement among guidelines was noted regardless of the method for their development.

none declared

We thank Dr.Stephen Edge for reviewing our paper and his helpful comments and constructive critique.