The GuideLine Implementability Appraisal (GLIA): development of an instrument to identify obstacles to guideline implementation

1472-6947-5-23 1472-6947 Research article The GuideLine Implementability Appraisal (GLIA): development of an instrument to identify obstacles to guideline implementation Shiffman N Richard richard.shiffman@yale.edu Dixon Jane jane.dixon@yale.edu Brandt Cynthia cynthia.brandt@yale.edu Essaihi Abdelwaheb abdel.essaihi@yale.edu Hsiao Allen allen.hsiao@yale.edu Michel George george.michel@yale.edu O'Connell Ryan ryan@oconnell.org

Yale Center for Medical Informatics, Yale University School of Medicine, New Haven, CT, USA

Yale School of Nursing, Yale University, New Haven, CT, USA

BMC Medical Informatics and Decision Making 1472-6947 2005 5 1 23 http://www.biomedcentral.com/1472-6947/5/23 16048653 10.1186/1472-6947-5-23 28 2 2005 27 7 2005 27 7 2005 2005 Shiffman et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Background

Clinical practice guidelines are not uniformly successful in influencing clinicians' behaviour toward best practices. Implementability refers to a set of characteristics that predict ease of (and obstacles to) guideline implementation. Our objective is to develop and validate a tool for appraisal of implementability of clinical guidelines.

Methods

Indicators of implementability were identified from the literature and used to create items and dimensions of the GuideLine Implementability Appraisal (GLIA). GLIA consists of 31 items, arranged into 10 dimensions. Questions from 9 of the 10 dimensions are applied individually to each recommendation of the guideline. Decidability and Executability are critical dimensions. Other dimensions are Global, Presentation and Formatting, Measurable Outcomes, Apparent Validity, Flexibility, Effect on Process of Care, Novelty/Innovation, and Computability. We conducted a series of validation activities, including validation of the construct of implementability, expert review of content for clarity, relevance, and comprehensiveness, and assessment of construct validity of the instrument. Finally, GLIA was applied to a draft guideline under development by national professional societies.

Results

Evidence of content validity and preliminary support for construct validity were obtained. The GLIA proved to be useful in identifying barriers to implementation in the draft guideline and the guideline was revised accordingly.

Conclusion

GLIA may be useful to guideline developers who can apply the results to remedy defects in their guidelines. Likewise, guideline implementers may use GLIA to select implementable recommendations and to devise implementation strategies that address identified barriers. By aiding the design and operationalization of highly implementable guidelines, our goal is that application of GLIA may help to improve health outcomes, but further evaluation will be required to support this potential benefit.

Background

Tremendous resources have been invested in the development and implementation of clinical practice guidelines over the past 15 years 123. In spite of these efforts, however, guidelines are not uniformly successful in improving care and several instances of implementation failure have been described, some resulting in substantial waste of time and resources 456. In many cases, implementation failures have been related to factors extrinsic to the guideline itself – e.g., organizational and provider-specific obstacles inherent in a particular system of care that interfere with implementation success. In other cases, however, factors intrinsic to the guideline have contributed to implementation failure, e.g., ambiguity, inconsistency, and incompleteness 78. We believe it is particularly important to identify these intrinsic factors, because in many cases they can be ameliorated or fully remedied by guideline authors while the guideline is being developed. If these problems are not captured during guideline development, they must be addressed during implementation.

Guideline implementation involves "the concrete activities and interventions undertaken to turn policies into desired results" 9. We define implementability to refer to a set of characteristics that predict the relative ease of implementation of guideline recommendations. Measures of successful implementation include improved adherence to guideline-prescribed processes of care and, ultimately, improved patient outcomes. Indicators of implementability, on the other hand, focus on the ease and accuracy of translation of guideline advice into systems that influence care. In this paper, we describe a tool for appraisal of implementability that is intended to help anticipate barriers to implementation success. We first delineate the process by which indicators of implementability were chosen. Then we describe the steps we took in the preliminary validation of the evolving instrument and show an example of its application. Finally, we discuss how the instrument might be used in practice.

Methods

Instrument development

The first step in the measurement of implementability was to define its attributes. From a broad-based literature search, we identified several key papers and book chapters that describe the impact of a variety of factors on success of implementation. The report from the Institute of Medicine 9 included general definitions and several high level constructs relevant to implementation. Thorsen and Mäkelä 10 described critical factors in implementation strategy that facilitate use of guidelines and overcome barriers to adoption. Solberg et al. asked expert implementers about implementation success factors and identified 83 variables grouped into 5 clusters 11. Applying diffusion of innovation theory, Grilli and Lomas identified guideline complexity, trialability, and observability as critical factors for successful implementation 12. Grol and colleagues found that vagueness, controversy, demand for a change in routines, and absence of an evidence-base differentiated guidelines that were not followed from those that were 13. Finally, we examined 3 instruments for appraisal of guideline quality – Cluzeau's 37-item instrument 14, the AGREE instrument 15, and Shaneyfelt's Guideline Quality Appraisal Questionnaire 16– and extracted factors from each that addressed implementability.

The authors eliminated redundant factors, i.e., those that appeared in several sources or represented concepts that were subsumed by others, through open discussion and consensus. Factors that indicated guideline quality but not implementability were excluded. We decided early on to focus the GLIA on factors that were intrinsic to the guideline, because they could be addressed centrally by a guideline development committee. Thus, we eliminated many factors related to Solberg's medical group characteristics, organizational capability for change, infrastructure for implementation, and external environment. Extrinsic items relating to a recommendation's effect on the process of care and items relating to the novelty or innovation of a guideline statement were retained in the instrument because developers can anticipate these barriers and offer potential strategies for implementation success.

All remaining factors were grouped into categories of related constructs, hereafter referred to as dimensions. We then devised specific questions to characterize each dimension and phrased them so that negative responses identified barriers. These questions ultimately became items of the instrument. We iteratively refined the items, further clarified definitions, and re-categorized items into the most appropriate dimensions.

Validation

To explore the construct of implementability and its measurement, we carried out a series of validation activities. The steps are summarized in Table 2 and are described below in the sequence in which they were carried out. Concomitant with these validation activities, the GLIA instrument underwent iterative refinement and revision.

Table 2

Summary of validation activities.

Step

Process

Purpose or Validity Type

Results

Ranking of implementability of 3 recommendations by experts at COGS

Validity of the construct of implementability

Consistent ranking

Guideline review with early GLIA versions

To refine GLIA

Demonstrate feasibility of measurement of implementability

Expert review of GLIA items and dimensions by HL7 experts

Content validity of GLIA

GLIA items rated generally as relevant and clear. Two new items were added, definitions were clarified, and explanatory material was added.

Review of the recommendations ranked in Step 1

Construct validity of GLIA

GLIA assessment of these three guidelines was consistent with experts' rankings of implementability (Step 1). Barriers to implementation were explicitly identified.

Results

Final instrument

The table shown in [Additional file 1] summarizes the 10 dimensions of the instrument and their definitions. (The GLIA instrument is available for download at http://ycmi.med.yale.edu/GLIA.) Of the 31 items in GLIA, the first dimension (Global) contains 7 items that relate to the guideline as a whole. The remainder of the dimensions focus on the individual recommendation as the unit of implementability, since a single guideline may contain recommendations that vary widely in their implementability. For each GLIA item, each recommendation is rated using one of four response options (see Table 1). Additional comments that explain each response are encouraged. GLIA users should discuss all divergent ratings and try to achieve consensus. Any items scored with "?" should be resolved: often, this requires the help of an expert in the guideline's topic area.

Table 1

GLIA response options.

The recommendation meets this criterion fully.

The recommendation does

not meet this criterion.

Rater is unable to address this question because of insufficient knowledge or experience in this area.

Criterion is not applicable to this recommendation.

When any GLIA item is assigned an "N" response, its corresponding barrier to implementation is recorded on the summary sheet with a brief description of why the recommendation failed the criterion (see Figure 1). Examination of the barriers recorded on the summary sheet should provide an understanding of predicted impediments to implementation of the recommendation. The summary sheet also contains a column where suggested remedies can be described.

Figure 1

Example of a GLIA Summary Report on draft recommendations from a guideline for diagnosis and management of otitis media with effusion

Example of a GLIA Summary Report on draft recommendations from a guideline for diagnosis and management of otitis media with effusion.

Validation

Step 1: Evaluation of implementability as a concept that is understood by experts

In April 2002, the Conference on Guideline Standardization (COGS) brought together 23 national and international leaders in guideline development, dissemination and implementation 17. This gathering provided the opportunity to explore the concept of implementability. We selected 3 recommendations for review by the experts based on the following criteria:

• Each recommendation concerned a common clinical problem generally understood by the attendees.

• The three recommendations as a group represented a broad range of implementation challenges.

Using our own global subjective judgment of implementability, one of the recommendations was considered to be straightforward to implement, another was considered to be quite difficult, and the remaining recommendation was considered to occupy an intermediate position. The recommendations we selected concerned: aspirin therapy in management of acute myocardial infarction 18, diagnosis of urinary tract infection in young children with unexplained fever 19, and evaluation of unexplained syncope 20.

Divided into 2 groups (10 individuals with considerable experience in guideline implementation in one group, 11 individuals experienced in guideline development and dissemination in the other), participants were asked to rank the relative implementability of the guideline recommendations. After discussion, each group came to a consensus ranking of the three recommendations' implementability. These rankings were consistent between the two groups and also with the authors' expectation. Both groups agreed that the aspirin recommendation would be easiest to implement and the syncope algorithm would be most challenging.

This step provided support for our basic assumption that the construct of implementability is real. Moreover, implementability varies among recommendations in a way that can be systematically recognized by experts.

Step 2- Initial guideline review with early GLIA versions

The authors next tested an early version of GLIA with a convenience sample of 20 guideline recommendations. These reviews provided our first experience using the GLIA in practice and led us to identify areas of rating controversy. Early attempts to apply a simple numeric scoring system failed because all the items were not of equal importance. In addition, we recognized the advantage of qualitative rating in understanding specific barriers to implementation. We appreciated the importance of including a variety of expertise among members of the GLIA team. This experience led to refinement of the instrument, as well as to a deeper understanding of the optimal process for conducting a review using GLIA.

The 20 replications of our process seemed quite adequate to provide us with assurance that creating an instrument to assess implementability was a feasible goal. They also gave us a view of the problems still to be addressed.

Step 3 - content validation: Expert review of GLIA items and dimensions

To investigate the content validity of the draft GLIA, we systematically examined each item's clarity, its relevance to its superordinate dimension, and the comprehensiveness of the GLIA as a whole 21. We prepared a Questionnaire for Expert Review on which relevance and clarity ratings were scored on a four point scale, with higher numbers representing greater relevance and greater clarity. We distributed the Questionnaire to volunteers attending a Workgroup Meeting of the HL7 Special Interest Group (SIG) on Clinical Guidelines. The SIG includes representatives from academia, vendors of electronic health record software, and healthcare providers, who meet three times each year to discuss standardization of guideline components.

Judgments about the GLIA were obtained from 7 guideline implementers. Mean relevancy ratings by item ranged from 2.7 to 4.0. On average, 26 of 30 items were rated as "moderately" or "highly relevant." No reviewer used a relevancy rating of 1 ("not at all relevant") for any item.

Mean clarity ratings by item ranged from 2.3 to 4.0. On average, 29 of 30 items were rated as "clear" or "very clear." The correlation of relevancy ratings and clarity ratings by item was .23 (non-significant), indicating that the two ratings of each item provided different information.

Raters were also asked for suggestions about the items and numerous comments were provided. Based on this feedback from implementers, GLIA was again revised. Items that received low clarity or relevance scores were carefully reviewed, definitions were clarified, and explanatory material was added to better communicate the meaning of the item. Based on suggestions from the reviewers, two new items that explored recommendation sequencing and the internal consistency of the guideline were added to the Global dimension.

This process of obtaining judgment data from implementers proved to be a rich source of ideas for refinement of items. It also provided supportive evidence for the general content validity of the developing instrument.

Step 4 -Review of ranked recommendations

Next, we used GLIA to formally assess the three recommendations whose implementability had been ranked previously by experts at the COGS Meeting (described above in Step 1). Results using GLIA were consistent with the earlier expert rankings, i.e., more critical implementation barriers were identified in the recommendation ranked least implementable and no barriers were identified in the recommendation ranked most implementable. Moreover, appraisal with GLIA allowed us to itemize specific obstacles, thereby clarifying particular impediments to implementation in contrast to the global subjective evaluation performed in Step 1.

This finding of agreement with expert rankings indicated that GLIA results reflect the construct of implementability as conceptualized by experts. This consistency provided preliminary evidence that supports the construct validity of GLIA assessments.

Step 5 – Application to a draft guideline from the American Academy of Pediatrics

To assess the value of application of GLIA in a real world guideline development effort, we applied a late version of the appraisal to a draft clinical guideline for the management of otitis media with effusion (OME) that was in preparation by a joint committee of national professional societies – the American Academy of Pediatrics, the American Academy of Family Physicians, and the American Academy of Otolaryngology. We received an intermediate draft of the guideline for quality appraisal and evaluation of implementability.

After independently rating the draft guideline, we met as a group to discuss the barriers to implementation that we identified. While not every barrier was identified by every rater, no single barrier was uniquely identified. Remarkably little discussion was needed to reach a consensus on anticipated barriers to implementation.

Our report to the guideline authors identified several instances of problems with decidability and executability of individual recommendations. For example, the draft recommended, "During the initial assessment of the child with OME, the clinician should document (a detailed set of physical findings)" (italics added). The GLIA report drew attention to the fact that the guideline's users might not consistently determine when in the course of continuous care an assessment is initial. Adding to the confusion, the draft guideline text later states that these findings should be ascertained at every medical encounter. The recommendation that was ultimately published stated: "Clinicians should document (the physical findings) at each assessment of the child with OME." Vagueness was also inherent in use of the terms "academic risk", "when necessary", and "individualized management" without clear definitions. Following the GLIA report, each of these was clarified in the final guideline publication 22.

GLIA appraisal also identified extrinsic barriers to implementation that were reported to the Joint OME Implementation Committee of the professional societies. These barriers included:

• Recommendations to perform pneumatic otoscopy, tympanometry, hearing screening, and language assessment would require acquisition of new equipment and skills on the part of many physicians.

• Recommendations against prescription of antihistamines, decongestants, antimicrobials and corticosteroids for effusions may not be compatible with patient expectations.

The committee considered these potential barriers in their design of a guideline implementation strategy.

Discussion

Many clinical guidelines – developed at substantial cost and effort – have proven to be difficult or impossible to operationalize. We developed the GuideLine Implementability Appraisal to facilitate guideline implementation. The instrument is designed to systematically highlight barriers to implementation.

The GLIA is intended to provide feedback about a guideline's implementability to two distinct audiences: the authors of the guideline and those individuals who choose guidelines for application within a health care delivery system. As a guideline is being developed, GLIA can provide feedback to guideline authors about potentially remediable defects. Developers may choose to make modifications to the guideline document before it is finalized and disseminated. Implementers can use GLIA to help select a guideline, to identify potential obstacles, and to target efforts toward addressing identified barriers. Thus, GLIA can be used to help select guidelines that are more easily implementable and also to devise implementation strategies that address identified barriers.

Two GLIA dimensions are of particular importance because failure to address them adequately will result in inconsistent implementation 2324. Any recommendation that does not clearly communicate what to do (i.e., it fails executability criteria) or when to do it (i.e., fails decidability criteria) is not fully ready for implementation. If possible, guideline authors should revise such a recommendation before it is disseminated for implementation. If problems in decidability and executability are not corrected prior to dissemination, different implementers may well interpret the guideline authors' intent in a discordant manner.

GLIA incorporates an optional dimension – computability – to indicate the ease with which a recommendation might be operationalized in an electronic information system. Guideline implementation strategies – e.g., education, academic detailing, audit and feedback, administrative sanctions – need not necessarily involve computers 25. Because of the success of electronic implementations in influencing clinician behavior 262728, the wide variation in electronic information systems, and the current lack of guidance regarding computability, these extrinsic considerations were retained in the instrument. Items in the computability dimension address the availability of data to trigger the recommendation, the level of specificity of the triggers and recommended actions, and whether there is a clear path from recommendation to electronic implementation.

Implementability must be differentiated from guideline quality. Quality assessments relate primarily to determining the scientific validity of guidelines and, generally, quality is assessed for the guideline as a whole. Implementability, on the other hand, is one component of guideline quality, but its assessment is applied largely to individual recommendations within a guideline.

In a comprehensive review of 13 tools for guideline quality appraisal, Graham 29 identified instruments developed by Cluzeau 14 and Shaneyfelt et al 16 as the best developed. Since that time, the AGREE Instrument 15 has emerged as the leading exemplar of guideline quality appraisal and it has been endorsed by the Guidelines International Network 30. Application of GLIA can complement quality appraisal in identifying guideline deficiencies. Several GLIA items overlap with items in the AGREE scale. However, to the best of our knowledge, GLIA is the only tool that emphasizes implementation concerns at the level of the individual recommendation.

Application of GLIA to measurement of decidability and executability requires that users translate guideline recommendations into statements comprising conditions and actions 31. Training and practice may be required to assure consistency of logical analysis.

Application of GLIA requires team effort and a consequent resource investment. The team that applies GLIA should include members with skills in guideline implementation as well as members with specific understanding of the clinical domain. Our experience has demonstrated that rating a guideline that contains 15 recommendations might require several hours of an individual's time. Additional time must be spent in resolving divergent ratings, although this effort usually yields an improved understanding of implementation issues. We are currently developing an electronic version of GLIA that will provide a more efficient means of rating, scoring, and reporting results.

During development, GLIA is best applied once evidence has been synthesized and draft recommendations have been formulated. When applied too late in the authoring process, GLIA may have limited impact because authors may have already become committed to recommendations as written and thus not open to making modifications. Assuring that guideline authors understand the importance of implementability early on may help to overcome premature commitment.

The classic challenge of instrument development is to arrive at the correct number of items to minimize burden and avoid redundancy, while including a sufficient number to be comprehensive. GLIA contains 7 global items that are applied once to each guideline, 20 items that are applied to each rated recommendation, and 4 optional items rating computability that are applicable when an electronic implementation is planned. Further use of GLIA is likely to result in clarification and perhaps modification of the number of items.

Limitations

We have performed a series of activities to provide preliminary evidence of GLIA's validity. However, neither the inter-rater reliability of GLIA, the test-retest reliability, the factor structure of the dimensions, nor its predictive validity has yet been established. Plans for this testing are underway.

It should be noted that the authors performed the activities described as Step 4. Until verified by an independent panel, these results should be considered preliminary and subject to potential bias.

In addition, GLIA addresses primarily factors intrinsic to the guideline. It is clear that extrinsic factors are critical in a successful implementation. Future extensions to GLIA will help to identify extrinsic barriers to implementation.

Conclusion

The GuideLine Implementability Appraisal provides a tool designed to help developers and implementers better understand and anticipate barriers to successful implementation. By aiding the design and operationalization of highly implementable guidelines, our goal is that application of GLIA may help improve health outcomes. Demonstration of GLIA's effectiveness will require prospective testing.

Competing interests

The author(s) declare that they have no competing interests.

Authors' contributions

RS conceived the instrument, performed the literature review, participated in all phases of instrument development and validation, and edited the manuscript.

JD participated in all phases of instrument development and validation and edited the manuscript.

CB, AE, AH, GM, and RO participated in all phases of instrument development and validation and reviewed the manuscript.

Additional File 1

GLIA dimensions and example items

Click here for file

Acknowledgements

The work was supported through grants R01-LM-07199, R29- LM05552, and T15-LM07065 from the U.S. National Library of Medicine and by R13 HS10962 from the Agency for Healthcare Research and Quality.

Effectiveness and efficiency of guideline dissemination and implementation strategies. Grimshaw JM Thomas RE MacLennan G Fraser C Ramsay CR Vale L Whitty P Eccles MP Matowe L Shirran L Wensing M Dijkstra R Donaldson C Health Technol Assess 2004 8 6 1 72 Review Status of practice guidelines in the United States: CDC guidelines as an example. [Review] [64 refs] Larson E Preventive Medicine 2003 36 5 519 524 10.1016/S0091-7435(03)00014-8 12689796 Towards evidence-based clinical practice: an international survey of 18 clinical guideline programs Burgers JS Grol R Klazinga NS Makela M Zaat J AGREE Collaboration Int J Qual Health Care 2003 15 1 31 45 10.1093/intqhc/15.1.31 12630799 Effect of computerised evidence-based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial Eccles M McColl E Steen N Rousseau N Grimshaw J Parkin D BMJ 2002 325 941 948 130060 12399345 10.1136/bmj.325.7370.941 Effects of computerized guidelines for managing heart disease in primary care Tierney WM Overhage JM Murray MD Harris LE Zhou XH Eckert GJ Smith FE Nienaber N McDonald CJ Wolinsky FD J Gen Intern Med 2003 18 12 967 976 10.1111/j.1525-1497.2003.30635.x 14687254 Physician awareness and self-reported use of local and national guidelines for community-acquired pneumonia Switzer GE Halm EA Chang CC Mittman BS Walsh MB Fine MJ J Gen Intern Med 2003 18 10 816 823 10.1046/j.1525-1497.2003.20535.x 14521644 Computerizing guidelines to improve care and patient outcomes: the example of heart failure Tierney WM Overhage JM Takesue BY Harris LE Murray MD Vargo DL McDonald CJ J Am Med Inform Assoc 1995 2 316 322 116272 7496881 Barriers between guidelines and improved patient care: an analysis of AHCPR's unstable angina clinical practice guideline Katz DA Health Serv Res 1999 34 1 377 389 10199682 Guidelines for clinical practice: from development to use Institute of Medicine Washington, DC , National Academy Press Field MJ, Lohr KN 1992 Changing professional practice: theory and practice of clinical guidelines implementation Thorsen T Mäkelä M Copenhagen , Danish Institute for Health Services Research and Development 1999 216 Lessons from experienced guideline implementers:attend to many factors and use multiple strategies Solberg LI Brekke ML Fowles J Jacobsen DN Kottke TE Mosser G O'Connor PJ Ohnsorg KA Rolnick SJ Jt Comm J Qual Improv 2000 26 4 171 188 10749003 Evaluating the message: the relationship betweenm the compliance rate and the subject of a practice guideline Grilli R Lomas J Medical Care 1994 32 202 213 8145598 Attributes of clinical guidelines that influence use of guidelines in general practice: observational study Grol R Dalhuijsen J Thomas S Veld C Rutten G Mokkink H BMJ 1998 317 858 861 31096 9748183 Development and application of a generic methodology to assess the quality of clinical guidelines Cluzeau FA Littlejohns P Grimshaw JM Feder G Moran SE Int J Qual Health Care 1999 11 1 21 28 10.1093/intqhc/11.1.21 10411286 Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project AGREE Collaboration Qual Saf Health Care 2003 12 18 23 10.1136/qhc.12.1.18 12571340 Are guidelines following guidelines? the methodological quality of clinical practice guidelines in the peer-reviewed medical literature Shaneyfelt TM Mayo-Smith MF Rothwangl J JAMA 1999 281 1900 1905 10.1001/jama.281.20.1900 10349893 Standardized reporting of clinical practice guidelines: a proposal from the Conference on Guideline Standardization Shiffman RN Shekelle P Overhage JM Slutsky J Grimshaw J Deshpande AM Ann Intern Med 2003 139 493 498 13679327 1999 Update: ACC/AHA guidelines for the management of patients with acute myocardial infarction Ryan TJ Antman EM Brooks NH Califf RM Hillis LD Hiratzka LF rapaport E Riegel B Russell RO Smith EE weaveer WD http://www.acc.org/clinical/guidelines/nov96/1999/index.htm Practice parameter: the diagnosis, treatment, and evaluation of the initial urinary tract infection in febrile infants and young children. American Academy of Pediatrics. Committee on Quality Improvement. Subcommittee on Urinary Tract Infection [published errata appear in Pediatrics 1999 May;103(5 Pt 1):1052, 1999 Jul;104(1 Pt 1):118 and 2000 Jan;105(1 Pt 1):141] [see comments] Anonymous Pediatrics 1999 103 4 Pt 1 843 852 10103321 Diagnosing syncope: Part 2. unexplained syncope Linzer M Yang EH Estes M Wang P Vorperian VR Kapoor WN Ann Intern Med 1997 127 76 86 9214258 Determination and quantification of content validity Lynn MR Nursing Research 1986 35 382 385 3640358 Otitis media with effusion Anonymous Pediatrics 2004 113 5 1412 1429 10.1542/peds.113.5.1412 15121966 Changing clinical behaviour by making guidelines specific Michie S Johnston M BMJ 2004 328 343 345 10.1136/bmj.328.7435.343 14764503 Are nonspecific practice guidelines potentially harmful? a randomized trial of the effect of nonspecific versus specific guidelines on physician decision making Shekelle PG Kravitz RL Beart J Marger M Wang M Lee M Health Services research 2000 34 1429 1448 10737446 Changing physicians' practices Greco PJ Eisenberg JM N Engl J Med 1993 329 1271 1274 10.1056/NEJM199310213291714 8413397 Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review Hunt DL Haynes RB Hanna SE Smith K JAMA 1998 280 1339 1346 10.1001/jama.280.15.1339 9794315 Computer-based guideline implementation systems: a systematic review of functionality and effectiveness Shiffman RN Liaw Y Brandt CA Corb GJ J Am Med Inform Assoc 1999 6 104 114 61349 10094063 Evaluating informatics applications--clinical decision support systems literature review Kaplan B Int J Med Inform 2001 64 1 15 37 Review 10.1016/S1386-5056(01)00183-6 11673100 A comparison of clinical practice guideline appraisal instruments Graham ID Calder LA Hebert PC Carter AO Tetroe JM Int J Tech Assess Health Care 2000 16 1024 1038 10.1017/S0266462300103095 Board of Trustees 2002, Guidelines International Network (G-I-N) Improving the quality of health care: using international collaboration to inform guideline programmes - by founding the Guidelines International Network (G-I-N) Ollenschlager G Marshall C Qureshi S Rosenbrand K Slutsky J Burgers J Makela M Qual Safe Health Care 2004 13 455 460 10.1136/qshc.2003.009761 Representation of clinical practice guidelines in conventional and augmented decision tables Shiffman RN J Am Med Inform Assoc 1997 4 5 382 393 61256 9292844

Pre-publication history

The pre-publication history for this paper can be accessed here:

http://www.biomedcentral.com/1472-6947/5/23/prepub