Email updates

Keep up to date with the latest news and content from BMC Health Services Research and BioMed Central.

Open Access Highly Accessed Research article

Do computerised clinical decision support systems for prescribing change practice? A systematic review of the literature (1990-2007)

Sallie-Anne Pearson1*, Annette Moxey2, Jane Robertson2, Isla Hains1, Margaret Williamson3, James Reeve3 and David Newby2

Author Affiliations

1 UNSW Cancer Research Centre, University of New South Wales and Prince of Wales Clinical School, Sydney, Australia

2 Discipline of Clinical Pharmacology, School of Medicine and Public Health, The University of Newcastle, Australia

3 National Prescribing Service Limited, Sydney, Australia

For all author emails, please log on.

BMC Health Services Research 2009, 9:154  doi:10.1186/1472-6963-9-154

The electronic version of this article is the complete one and can be found online at: http://www.biomedcentral.com/1472-6963/9/154


Received:30 November 2008
Accepted:28 August 2009
Published:28 August 2009

© 2009 Pearson et al; licensee BioMed Central Ltd.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Background

Computerised clinical decision support systems (CDSSs) are used widely to improve quality of care and patient outcomes. This systematic review evaluated the impact of CDSSs in targeting specific aspects of prescribing, namely initiating, monitoring and stopping therapy. We also examined the influence of clinical setting (institutional vs ambulatory care), system- or user-initiation of CDSS, multi-faceted vs stand alone CDSS interventions and clinical target on practice changes in line with the intent of the CDSS.

Methods

We searched Medline, Embase and PsychINFO for publications from 1990-2007 detailing CDSS prescribing interventions. Pairs of independent reviewers extracted the key features and prescribing outcomes of methodologically adequate studies (experiments and strong quasi-experiments).

Results

56 studies met our inclusion criteria, 38 addressing initiating, 23 monitoring and three stopping therapy. At the time of initiating therapy, CDSSs appear to be somewhat more effective after, rather than before, drug selection has occurred (7/12 versus 12/26 studies reporting statistically significant improvements in favour of CDSSs on = 50% of prescribing outcomes reported). CDSSs also appeared to be effective for monitoring therapy, particularly using laboratory test reminders (4/7 studies reporting significant improvements in favour of CDSSs on the majority of prescribing outcomes). None of the studies addressing stopping therapy demonstrated impacts in favour of CDSSs over comparators. The most consistently effective approaches used system-initiated advice to fine-tune existing therapy by making recommendations to improve patient safety, adjust the dose, duration or form of prescribed drugs or increase the laboratory testing rates for patients on long-term therapy. CDSSs appeared to perform better in institutional compared to ambulatory settings and when decision support was initiated automatically by the system as opposed to user initiation. CDSSs implemented with other strategies such as education were no more successful in improving prescribing than stand alone interventions. Cardiovascular disease was the most studied clinical target but few studies demonstrated significant improvements on the majority of prescribing outcomes.

Conclusion

Our understanding of CDSS impacts on specific aspects of the prescribing process remains relatively limited. Future implementation should build on effective approaches including the use of system-initiated advice to address safety issues and improve the monitoring of therapy.

Background

Computerised clinical decision support systems (CDSSs) are espoused as powerful tools for influencing health care provider performance to improve quality of care and patient outcomes. As such, proprietary and locally developed systems have been used to target provider behaviour across a range of clinical circumstances including preventive, acute and chronic care and to target specific test ordering and prescribing practices.

In their simplest form CDSSs provide narrative information requiring further processing and analysis by providers before clinical decisions are made. However, CDSSs have become increasingly sophisticated by matching patient characteristics with computerised knowledge bases and using algorithms to generate patient-specific assessments or treatment recommendations. [1] CDSSs support a range of prescribing practice activities relating broadly to initiating therapy, including the judicious selection of drug treatments, adhering to best practice care and checking allergies and drug interactions. In addition, CDSSs can target other aspects of the prescribing process, namely monitoring (with dose adjustments) and stopping or tapering therapy.

The most comprehensive systematic reviews in the field [1,2] established the benefits of CDSSs in changing practitioner performance and to a lesser extent, patient outcomes. Garg et al [1] reviewed 100 randomised and non-randomised studies of CDSSs across a broad range of clinical targets including diagnosis, prevention, disease management, drug dosing and drug prescribing while the Kawamoto et al [2] review was restricted to randomised controlled trials (RCTs) but assessed the impact of both electronic and non-electronic systems. Both reviews also examined system and organisational features predicting the success of decision support.

Notably, improved CDSS performance was associated with systems prompting providers automatically (system-initiated) rather than requiring providers to activate the decision support themselves (user-initiated).[1] Kawamoto et al also reported the benefits of computerised over paper-based systems and those providing advice within the clinical workflow and at the time and location of decision-making.[2] A subsequent review has also demonstrated systems implemented in hospital practice are more effective in improving outcomes than those targeting chronic disease in ambulatory care.[3]

Other systematic reviews have taken a narrower prescribing focus and demonstrate CDSSs can reduce toxic drug levels and time to achieving therapeutic control [4,5]; reduce medication errors [6,7]; change prescribing in accordance with guideline recommendations and increase diagnostic testing. [8]

In a more recent systematic review, Mollon et al[9] examined the effects of CDSSs on prescribing practices and evaluated whether particular system features could predict successful implementation, provider behaviour change or improved patient outcomes. The review was restricted to RCTs of computer generated advice (delivered in electronic or paper-based formats) and targeted decisions presented before drug therapies were chosen or directly after drug selection had taken place. The authors noted few high quality studies showed improvement in patient outcomes and there was insufficient detail on system features in the original studies to inform future CDSS design and implementation. None of the reviews conducted to date have examined the effects of CDSSs across all aspects of the prescribing process nor addressed clinical targets and how they may relate to the relative success of CDSSs.

Providers' information needs are likely to vary across clinical targets and the therapeutic process (initiating, monitoring and stopping therapy), and it is highly likely that some aspects of prescribing practice are more amenable to change than others. Previous research has suggested physicians are more reluctant to change practice if advice is perceived as threatening to professional autonomy.[10,11] Thus, recommendations attempting to influence initial therapeutic choices or suggestions to stop long-standing treatment may be more difficult to change than attempts to flag safety issues or to fine-tune existing therapy. Further, prescribing involves a complex set of behaviours and there is strong evidence demonstrating the benefits of multi-faceted, compared with single strategy approaches to prescriber behaviour change. [12-14]

Therefore, the objective of this systematic review is to examine the impact of CDSSs in targeting specific aspects of prescribing, namely initiating treatment (before and after drug selection has taken place), monitoring patients on existing therapy and stopping treatment. Further, within each domain, we will explore whether some of the key technical and organisational features shown to be effective in previous CDSS reviews change practice in line with the intent of the CDSS, namely system versus user-initiated advice, CDSSs implemented in institutional settings versus ambulatory care, and multi-faceted versus stand alone CDSS interventions. Finally, we will explore the clinical areas targeted in the CDSS interventions.

Methods

Hypotheses Tested

We hypothesised that CDSSs for prescribing, where advice is generated

    and
delivered electronically, are more likely to change prescribing practice in line with the intent of the CDSS when decision support is:

• Provided to fine-tune existing therapy compared with recommendations to influence initial drug choices or cease long-standing therapy.

Within the various prescribing domains we also hypothesised that CDSS is more likely to change prescribing practice in line with the intent of the CDSS when decision support is:

• Implemented in institutional (inpatient) settings compared with ambulatory care

• System-initiated compared with user-initiated

• Implemented with other intervention strategies compared with implemented alone.

In addition, we were interested in exploring the clinical areas targeted in CDSS interventions. While it is unlikely there would be sufficient studies to test a formal hypothesis relating to clinical target, we aimed to document the clinical targets within prescribing domain and establish if computerised CDSS is more effective in specific clinical areas.

Studies Eligible for Review

We included English-language studies, published since 1990, reporting RCTs and strong quasi-experiments (non-randomised studies with comparison groups or interrupted time series designs with or without comparison groups). We stipulated the studies had to: compare the performance of computerised CDSS to routine care and/or paper-based decision support; provide information applicable to a specific patient that is reviewed by a provider at the time of prescribing (e.g. provide advice to prescribe a particular drug, to monitor a drug or adjust the dose, or to perform laboratory tests related to safe prescribing); generate

    and
deliver information to the provider in electronic formats; and report data on at least one outcome relating to initiating, monitoring or stopping therapy (Table 1).

Table 1. Definitions and examples of initiating, monitoring and stopping therapy

Studies were excluded if the CDSS targeted only medical students, pharmacists or nurses, interventions were based around hypothetical scenarios rather than actual clinical practice, and the advice provided by CDSS was feedback about groups of patients (e.g. audit activities). Further studies not undertaking statistical analyses and/or reporting only patient outcomes (e.g. quality of life, satisfaction with care, greater physical activity levels) or cost data were also excluded. A list of excluded studies is available on request from the authors.

Study Identification

We searched Medline (1990 - November Week 3, 2007), PreMedline (30 November, 2007), Embase (1990 - Week 47, 2007), CINAHL (1990 - November Week 4, 2007) and PsycINFO (1990 - November Week 4, 2007). We combined keywords and subject headings to identify computer-based decision support (e.g. decision support systems clinical, decision making computer assisted), medicines use (e.g. prescription drug, drug utilization) and medical practice (e.g. physicians practice patterns, medical practice). We also searched INSPEC (November 2007) and the Cochrane Database of Systematic Reviews (November 2007) including reviews and protocols published under the Effective Practice and Organisation of Care Group (EPOC). Finally, we hand-searched the reference lists of retrieved articles and reviews. Additional File 1 details the full search strategy.

Additional file 1. Full search strategy.

Format: DOC Size: 38KB Download file

This file can be viewed with: Microsoft Word ViewerOpen Data

Pairs of reviewers evaluated independently the study titles and abstracts identified in the search. Full-text articles were retrieved if any reviewer considered a citation potentially relevant. The studies were further assessed for their relevance using a screening tool based on study design, intervention target and type of intervention.

Data Extraction

Studies deemed eligible for review underwent data extraction by pairs of reviewers independent of one another (AM and IH, IH and SP). Disagreements were resolved by discussion to reach consensus. We developed a data extraction instrument based on checklists used in previous systematic reviews [1,13-15] but refined for the specific aims of this study. We extracted the following information: study objectives, clinical setting (ambulatory or institutional care), details of decision support intervention (e.g. system- or user-initiated, multi-faceted or CDSS alone, clinical target). Given the lack of uniformity in relation to terminology about prompts, alerts and reminders we extracted details as they were reported in the manuscripts. We recorded details of the study design (including the unit of intervention, comparison groups or measures, group assignment, statistical analyses), participant numbers, outcomes of interest and results pertaining to these outcomes.

Quality Assessment

Two reviewers (JR and DN) rated experimental and quasi-experimental studies for methodological quality on a 10-point scale consisting of five potential sources of bias (as described by Garg and colleagues [1]). The items used to assess study quality are described along with the outcomes of this assessment in the results section. We also noted whether statistical analyses were adjusted for clustering (this was not part of the Garg assessment form).

Reporting

Due to heterogeneity in study methodology, comparison groups, setting, intervention targets, and outcomes, we could not use traditional meta-analytic approaches to combine individual study results.

The studies in this review differed substantially in the type and number of outcomes reported to measure intervention impacts. In particular, clinical outcomes are not reported across all studies, most likely due to the short-term nature of the trials. This review only reports on the impact of CDSSs on measures relating to prescribing and includes laboratory or monitoring tests relevant for the safe and appropriate use of particular medicines. While these are intermediate outcomes, where studies have demonstrated the use of a particular drug leads to improvements in important patient outcomes, changes in prescribing consistent with this evidence are likely to be a reasonable surrogate for patient outcomes. We do, however, detail additional outcomes and provide a more extensive description of our research methods in a related report entitled 'Improving the uptake of evidence-based drug information and decision support' http://www.nps.org.au webcite.

Outcomes in this review are summarised separately for each study and coded according to the following scheme:

• "+ (NS)": intervention favoured CDSS (prescribing was more consistent with the intentions of the CDSS) but was not statistically significant;

• "- (NS)": intervention favoured the comparison group (prescribing of comparison groups was more consistent with the intentions of the CDSS) but was not significant;

• "++": intervention favoured CDSS (prescribing was more consistent with the intentions of the CDSS) and was statistically significant;

• "- -": intervention favoured the comparison group (prescribing of comparison groups was more consistent with the intentions of the CDSS) and was statistically significant;

• "0": no difference between groups

• "U": this code was used where it was difficult to ascertain the direction of the effect (usually due to lack of information).

Individual studies were stratified according to prescribing domain. Within domain we also categorised studies by clinical area. Studies appear more than once if they address decision support across more than one prescribing domain (e.g. initiating and monitoring therapy) or report prescribing outcomes across more than one clinical area. In the latter case, only the prescribing outcomes pertinent to the specific clinical area are presented in each category.

We summarised studies in the manuscript by reporting whether they demonstrated at least one positive prescribing outcome (general trend in favour of CDSS) and statistically significant improvements in favour of CDSS on the majority (≥ 50%) of prescribing outcomes (as used by Garg and colleagues [1]). We chose to report general trends as well as significant results given the likelihood some studies were underpowered to detect statistically significant differences in outcomes. Additional File 2 and Additional File 3 detail results for four summary measures: at least one outcome in favour of CDSS; majority of prescribing outcomes in favour of CDSS; at least one statistically significant prescribing outcome; and statistically significant improvement on the majority of prescribing outcomes.

Additional file 2. Table S1 - Full summary of results - Initiating treatment. a Other clinical areas include: salicylates or paracetamol in patients with history of GI bleed; erythropoietin low Hb; HIV medications; various medications. b Other clinical areas include: various medications interacting with warfarin; various conditions in children (e.g. croup, otitis media)

Format: DOC Size: 52KB Download file

This file can be viewed with: Microsoft Word ViewerOpen Data

Additional file 3. Table S2 - Full summary of results - Monitoring treatment. c Other clinical areas include: hormone treatment for infertility; etretinate for psoriasis; HIV medications; various conditions (e.g. epilepsy, gout, diabetes).

Format: DOC Size: 41KB Download file

This file can be viewed with: Microsoft Word ViewerOpen Data

Results

Characteristics of Included Studies (56 studies)

We screened the titles and abstracts of 7,243 articles and reviewed 372 full-text articles. Fifty-six articles were included in the review (Figure 1). Table 2 details the characteristics of these studies.

Table 2. Characteristics of included studies

thumbnailFigure 1. Process of study inclusion for the systematic review. * 8 Studies provided data relevant to more than one prescribing domain.

Most studies (n = 38) focussed on decision support at the time of initiating therapy (26 before and 12 after drug selection), 23 on monitoring treatment and three on stopping therapy. Most studies (n = 39) were undertaken in North America, 35 targeted physicians exclusively and 21 targeted physicians with medical students and/or other health care professionals such as nurses, nurse practitioners, pharmacists and physician assistants.

The majority of studies (n = 39) used system-initiated decision support and 37 studies were conducted in ambulatory care (hospital outpatients, Veterans' Medical Centres, Health Maintenance Organisations), 16 in institutional care (hospital inpatient, long-term care) and three across both settings. Nineteen were multi-faceted in that clinicians received combinations of academic detailing, audit and feedback, didactic lectures and written guidelines in addition to decision support.

Cardiovascular disease management was the most common clinical focus (n = 19). Other clinical areas included anticoagulant therapy (n = 12), antibiotic prescribing (n = 9), vaccinations (n = 9), respiratory conditions such as asthma and chronic obstructive pulmonary disease (n = 9), osteoporosis (n = 2) and prescribing for the elderly (n = 4). Fifteen studies addressed more than one clinical area.

Methodological Quality

Table 3 details the outcomes of the quality assessment. Fifty of the 56 trials were RCTs, six were quasi-experiments, five of which were interrupted time series designs. Fifty studies used random or quasi-random allocation of study groups, 20 reported randomisation by cluster (to minimise contamination) and 24 accounted for clustering in statistical analyses. Thirty six studies reported no baseline differences between study groups or made the appropriate statistical adjustments to account for baseline differences and 53 used objective study outcomes or subjective outcomes with blinded assessment. Fifty trials included at least 90% of baseline participants in final data analyses. On the 10-point methods scale, the mean study quality score was 7.9 (SD 1.9) with a range of 4 to 10.

Table 3. Quality assessment

Impact of CDSSs according to Prescribing Domain

Table 4 summarises the number of studies reporting at least one positive outcome and statistically significant improvement in favour of the CDSS on the majority (≥ 50%) of prescribing outcomes. Additional File 4, Additional File 5, Additional File 6 and Additional File 7 display the key features of individual interventions, outcome measures, quality scores and results, organised by the study's primary clinical focus.

Additional file 4. Table S3: Key study features and results (Initiating treatment - Before drug selection). * Unless otherwise stated, number of patients is close to or equal to that specified in the "participants" column, or was not reported. + (NS) indicates intervention favoured the CDSS but was not statistically significant; - (NS) indicates intervention favoured comparison group but was not statistically significant; 0 = no difference between groups; ++ indicates intervention favoured CDSS and was statistically significant; - - indicates intervention favoured comparator and was statistically significant; U = unclear. ACE = angiotensin-converting enzyme; BMD = bone mineral density; BP = blood pressure; CAD = coronary artery disease; CDSS = computerised clinical decision support system; CHD = coronary heart disease; CHF = congestive heart failure; CME = continuing medical education; CPOE = computerised provider order entry; COPD = chronic obstructive pulmonary disease; CVD = cardiovascular disease; EMR = electronic medical record; GI = gastro intestinal; GP = general practitioner; Hgb = haemoglobin; HIV = human immuno-deficiency virus; HMO = Health Maintenance Organisation; IHD = ischemic heart disease; LDL = low-density lipoprotein; MI = myocardial infarction; NSAIDs = non-steroidal anti-inflammatory drugs; NYHA = New York Heart Association; PCP = P carinii pneumonia; RCT = randomised controlled trial; UTI = urinary tract infection;

Format: DOC Size: 294KB Download file

This file can be viewed with: Microsoft Word ViewerOpen Data

Additional file 5. Table S4: Key study features and results (Initiating treatment - After drug selection). * Unless otherwise stated, number of patients is close to or equal to that specified in the "participants" column, or was not reported. + (NS) indicates intervention favoured the CDSS but was not statistically significant; - (NS) indicates intervention favoured comparison group but was not statistically significant; 0 = no difference between groups; ++ indicates intervention favoured CDSS and was statistically significant; - - indicates intervention favoured comparator and was statistically significant; U = unclear. CDSS = computerised clinical decision support system; CPOE = computerised provider order entry; COPD = chronic obstructive pulmonary disease; CNS = central nervous system; CVD = cardiovascular disease; ED = emergency department; EMR = electronic medical record; GP = general practitioner; HMO = Health Maintenance Organisation; ICU = intensive care unit; IV = intravenous; MRSA = methicillin-resistant Staphylococcus aureus; NSAIDs = non-steroidal anti-inflammatory drugs; RCT = randomised controlled trial; TCA = tertiary amine tricyclics antidepressant.

Format: DOC Size: 189KB Download file

This file can be viewed with: Microsoft Word ViewerOpen Data

Additional file 6. Table S5: Key study features and results (Monitoring existing therapy). * Unless otherwise stated, number of patients is close to or equal to that specified in the "participants" column, or was not reported. + (NS) indicates intervention favoured the CDSS but was not statistically significant; - (NS) indicates intervention favoured comparison group but was not statistically significant; 0 = no difference between groups; ++ indicates intervention favoured CDSS and was statistically significant; - - indicates intervention favoured comparator and was statistically significant; U = unclear. ACE = angiotensin-converting enzyme; BP = blood pressure; ALT = alanine aminotransferase; AST = aspartate aminotransferase; CBC = complete blood count; CDSS = computerised clinical decision support system; CHD = coronary heart disease; CME = continuing medical education; CPOE = computerised provider order entry; COPD = chronic obstructive pulmonary disease; EMR = electronic medical record; FSH = follicle-stimulating hormone; HIV = human immuno-deficiency virus; HMO = health maintenance organisation; INR = international normalised ratio; IV = intravenous; LFT = liver function test; LH = luteinizing hormone; NYHA = New York Heart Association; PTR = prothrombin time ratio; RCT = randomised controlled trial; TSH = thyroid stimulating hormone;

Format: DOC Size: 236KB Download file

This file can be viewed with: Microsoft Word ViewerOpen Data

Additional file 7. Table S6: Key study features and results (Stopping treatment). * Unless otherwise stated, number of patients is close to or equal to that specified in the "participants" column, or was not reported. + (NS) indicates intervention favoured the CDSS but was not statistically significant; - (NS) indicates intervention favoured comparison group but was not statistically significant; 0 = no difference between groups; ++ indicates intervention favoured CDSS and was statistically significant; - - indicates intervention favoured comparator and was statistically significant; U = unclear. CDSS = computerised clinical decision support system; CPOE = computerised provider order entry; COPD = chronic obstructive pulmonary disease; EMR = electronic medical record; FSH = follicle-stimulating hormone; GP = general practitioner; LH = luteinizing hormone; NSAID = non-steroidal anti-inflammatory drugs; NYHA = New York Heart Association; RCT = randomised controlled trial.

Format: DOC Size: 47KB Download file

This file can be viewed with: Microsoft Word ViewerOpen Data

Table 4. Studies reporting at least one positive outcome and ≥ 50% significant outcomes in favour of CDSS

Initiating Treatment (38 studies) [16-41]

Overall, 36 studies demonstrated at least one positive prescribing outcome in favour of CDSSs, and 19 reported significant improvements on the majority of outcomes.

Before Drug Selection (26 studies)

Almost all studies (n = 24) showed improvements in at least one outcome and 12 demonstrated significant improvements on the majority of outcomes. Importantly, one study involving preventive care recommendations for increased use of angiotensin converting enzyme (ACE) inhibitors [33] found the CDSS was significantly inferior to its comparator on at least one outcome measure.

Nineteen of the 20 system-initiated decision support interventions showed improvements in at least one outcome and 12 demonstrated significant improvements on the majority of outcomes. Two of the three user-initiated systems had a positive effect on at least one outcome but none demonstrated statistically significant improvements.

All but two of the 20 studies conducted in ambulatory care demonstrated at least one positive outcome in favour of CDSS. However, only eight demonstrated significant improvements in the majority of outcomes. All five of the studies conducted in institutional care reported at least one positive outcome in favour of CDSSs, and three demonstrated statistically significant improvements in the majority of measures.

Multi-faceted interventions did not appear to have any advantage over interventions using CDSS alone (with 5 of 15 and 7 of 11 reporting statistically significant results on the majority of outcomes respectively).

Cardiovascular disease was the most common clinical target for CDSS interventions, followed by vaccinations. Sixteen studies addressed aspects of cardiovascular disease management, such as primary and/or secondary prevention. However, only four demonstrated significant positive CDSS impacts on the majority of outcomes measured. All four used system-initiated advice to prompt physicians about the therapeutic management of patients determined to be 'at risk' of cardiovascular events. [21,24,26,28] Five of the nine studies targeting vaccinations demonstrated statistically significant benefits in favour of CDSS for the majority of outcomes [20,21,36,40,41], and they too used system-initiated advice to increase vaccination rates. There were too few studies across the other clinical domains to draw any conclusions about the impact of CDSS in specific clinical areas.

After Drug Selection (12 studies) [42-53]

All twelve studies reported improvements in at least one outcome and seven demonstrated favourable results in favour of CDSSs on the majority of outcomes. All of the studies used system-initiated decision support. Four of the five studies conducted in institutional care demonstrated significant improvements in the majority of outcomes and three of the seven studies conducted in ambulatory care demonstrated significant improvements in the majority of outcomes. None of the interventions in this category were multi-faceted.

All six studies examining the effect of CDSS on antibiotic prescribing reported at least one positive outcome, however, four studies demonstrated significant impacts on the majority of outcomes. In each case, the intervention involved system-initiated advice to target the rational use of specific antibiotics.[44,47,50,53] Four studies evaluated the effectiveness of CDSS for prescribing in the elderly. Two demonstrated significant improvements in the majority of outcomes and both studies focussed specifically on safety issues.[43,45] Based on patient specific information, the first intervention customised initial dosing recommendations for sedatives, neuroleptics, anti-emetics and skeletal muscle relaxants. [45] The second used alerts to prompt physicians after prescribing a particular medication to consider alternative therapy. [43]

Monitoring Patients on Existing Therapy (23 studies) [20,35,36,39,49,54-71]

Eighteen studies demonstrated positive results on at least one outcome, and eight reported statistically significant results in favour of CDSS on the majority of outcomes. Five studies used CDSSs to advise on maintenance prescribing, such as dose changes, in chronic therapy for drugs including statins, anti-hypertensives, and asthma medications. Twelve studies used dose calculators, eight of which related to warfarin therapy and the remaining studies investigated prescribers' responses to prompts for laboratory tests, such as liver function tests, for a range of medicines; one of these measured adherence to a warfarin monitoring schedule.

Nine of the 11 system-initiated decision support interventions showed improvements in at least one outcome and six demonstrated significant improvements on the majority of outcomes. Nine of the 11 user-initiated systems had a positive effect on at least one outcome but only two demonstrated significant improvements on the majority of outcomes.

Decision support interventions undertaken in institutional care appeared to perform somewhat better than those conducted in ambulatory care (3 of 7 versus 4 of 14 demonstrated the majority of outcomes were statistically significant in favour of CDSS respectively).

We did not find any differences in the effectiveness of multifaceted interventions versus CDSS only (with 2 of 7 and 6 of 16 reporting statistically significant results on the majority of outcomes respectively).

All nine studies involving warfarin (anticoagulant) therapy reported at least one positive outcome. Only two of these demonstrated significant improvements in the majority of outcomes. There were too few studies across the other clinical domains to draw any conclusions about the impact of CDSS in specific clinical areas. However, there was some evidence to support CDSS in the realm of reminders for laboratory tests with four of the seven studies reporting statistically significant results favouring CDSS on the majority of outcomes. All of the effective studies used system-initiated advice to increase therapeutic monitoring of a range of drugs. [36,58,60,66]

Stopping Treatment (3 studies) [39,42,61]

Studies in this domain were too few to draw any specific conclusions relating to our hypotheses. Two studies were conducted in institutional care. One addressed stopping ipratropium therapy and found the intervention inferior to standard care (although the result was not statistically significant).[39] The second hospital-based study involved patients undergoing ovarian stimulation for infertility using software to predict whether additional cycles of ovulatory stimulants should be administered [61] however the impact on the prescribing outcome (number of cancelled cycles) was unclear. There were no statistically significant changes in any of the prescribing outcomes for the third study that examined alerts relating to discontinuing potentially inappropriate medications in the elderly (e.g. NSAIDs and benzodiazepines). [42]

Discussion

The current review explored the impact of CDSSs on key decision points in the prescribing process and highlights the diversity of studies, intervention targets and methods reported to date. While the range and variety of studies adds great breadth to the field it makes synthesis challenging and creates difficulties in providing clear guidance on where CDSSs are likely to be most effective. We did however find some general trends supporting the effectiveness of CDSSs across specific aspects of the prescribing process.

Key Findings

In the realm of initiating therapy we found some indication of greater effectiveness of CDSSs after, rather than before, drug selection. The effective interventions implemented after drug selection had taken place all used system-initiated advice to flag key safety issues (such as alerting providers to high severity drug interactions, contraindications with other medications and cautions against prescribing particular medications for the elderly) or provided quality use of medicine messages (such as alterations to durations of therapy and/or form of prescribed drugs). This supports our hypothesis that it is easier to influence practice in relation to fine-tuning therapy rather than attempting to influence initial therapeutic choices. We also found some evidence to support the benefits of CDSSs in increasing the laboratory testing rates for patients on long-term therapy including cardiovascular and respiratory medicines. Further, consistent with previous reviews, system-initiated CDSSs appeared to be more effective than user-initiated systems as did CDSSs implemented in institutional as opposed to ambulatory settings.

We did not however find any evidence to support the notion that CDSSs implemented with other strategies such as education are more successful in changing prescribing than stand alone interventions. It is notable that few studies addressed stopping or tapering therapy, none of which demonstrated impacts in favour of CDSSs over comparators on the majority of outcomes. While many of the interventions addressed the prevention and management of cardiovascular disease only a small number of these demonstrated significant improvements on the majority of prescribing outcomes. Studies were too few in the other clinical domains to draw firm conclusions about the benefits of CDSS in managing specific clinical areas.

Limitations

This review had a number of limitations. Despite our intensive efforts, the collection of intervention studies is likely to be incomplete as some evaluations may not be available in the public domain, others may be published outside the peer-reviewed academic literature or published in languages other than English. Of the available studies, it was difficult to synthesise the material presented due to use of ambiguous terminology and more particularly there was little detail provided about the way in which the CDSSs were integrated into existing and/or new software systems. We did however manage to extract some key information about system and organisational factors demonstrated in previous reviews to influence outcomes.

While our review is not unique to the field, the overlap between studies in our review and others published previously is surprisingly small (see Additional File 8). This is due primarily to the variability in search terms and dates and more specifically inclusion and exclusion criteria (for example, RCTs versus RCTs plus quasi-experiments; physicians only versus all providers; prescribing only versus the full spectrum of clinical care; computerised advice delivered in electronic formats versus other electronic and/or paper-based formats). Prescribing is a key aspect of modern medicine therefore there is a strong case to add another review to the existing evidence by examining where CDSSs have the greatest impact across the prescribing process. Importantly, as the evidence base grows and reporting of CDSS and organisational features improves, a clear enhancement to the current review would be assessment of the impact of additional system and organisational features on user acceptability and likelihood of changes in practice in line with CDSS recommendations.

Additional file 8. Table S7: Key differences between the studies included in the current review and previous reviews. * Studies were not prescribing focused (screening, preventive care/disease management). † Studies addressed automated drug dosing studies without decision support. # Review excluded studies below a defined quality rating. ‡ Review did not include monitoring or ceasing studies.

Format: DOC Size: 34KB Download file

This file can be viewed with: Microsoft Word ViewerOpen Data

Previous reviews in prescribing have been more restricted than ours and even the most recent and extensive of these reviews [9] only targets initiation of therapy and ignores other prescribing dimensions associated with reviewing/monitoring existing therapies and stopping therapies. CDSS interventions are highly complex and their effectiveness is dependent largely on how well they are designed and implemented. This review was not able to consider the full impact of factors such as system design, usability and integration with clinician workflow which may have influenced the ability of systems to deliver anticipated prescribing outcomes.

There are also fundamental difficulties in comparing interventions with diverse objectives, measurement methods and outcomes. We managed this issue by reporting differences in favour of CDSSs on at least one prescribing outcome and significant differences on the majority of prescribing outcomes. Many results were positive, but not statistically significant, suggesting inadequate sample sizes and under powering of studies. There is also the possibility that there is some bias in publishing 'positive' studies, however many studies included in this review did not demonstrate significant outcomes in favour of CDSS. We replicated the summary measure used by Garg et al [1] (≥ 50% of significant outcomes), but it may be influenced by the number of outcomes reported and the appropriateness of the outcomes used. For example, the more prescribing outcomes reported, the more difficult it may be to achieve significant positive effects on the majority of outcomes measured. However, had we based our findings on the outcome 'at least one statistically significant outcome in favour of CDSS', our conclusions would change only for setting. Using this less stringent measure, we would have concluded there was no evidence indicating that CDSS was more effective in institutional compared with ambulatory care settings.

Many of the studies in this review used outcomes such as prescribing volume or rates (i.e. number of prescriptions for a particular drug or drug class). While these measures are clearly indicative of a desired change in prescribing practice, previous research has demonstrated that changes in these measures do not always equate to more 'appropriate' prescribing and better outcomes for all patient subgroups. [72,73] Thus, the choice of outcome measures sensitive to such issues would be of greater benefit in establishing intended and unintended consequences of any intervention.

While relatively few studies achieved significant positive effects on the majority of outcomes, considerably more studies did demonstrate at least one positive intervention effect. Many of the studies in this review addressed common conditions and drugs so any change in outcome may equate to substantial clinical and fiscal benefits. Further, while this review did not specifically address the impact of CDSSs on patient outcomes, if the change in a prescribing outcome has been demonstrated in randomised controlled trials to translate into clinical benefit for specific patient populations (e.g. prescribing medicines to lower blood pressure or to lower cholesterol levels leads to fewer myocardial infarctions and strokes) this adds further weight to the benefits of these interventions.

Further, 'neutral' studies should not necessarily be viewed negatively. While it was not the intent of the interventions in this review to establish 'equivalence' it may be worth considering the benefits of CDSSs in releasing health care professionals from activities they need not undertake, such as actively documenting when tests and follow-up should be performed. Finally, few studies reported unintended negative consequences of CDSS interventions such as undesirable drug substitution effects. While our review did not raise any particular concerns about the negative effect of CDSS, its full impact remains unknown.

Commentary

Limitations notwithstanding, this study has contributed to the literature by detailing the specific areas within the prescribing process where CDSSs tend to be more effective. The most consistently effective approaches used system-initiated decision support to fine-tune existing therapy by making recommendations to improve patient safety, adjust the dose, duration or form of prescribed drugs or increase the laboratory testing rates for patients on long-term therapy. This finding remains consistent with other literature. Patient safety is a predominant feature of clinical culture and any system highlighting potential problems is welcomed and adopted by clinicians. [1,7] These triggers and reminders simply add safeguards and enhance continuity and long-term care rather than posing any threat to professional autonomy.

There is currently insufficient evidence to draw conclusions about the impact of CDSSs with respect to stopping therapy but changing behaviour in this domain will likely bring its own challenges. Prescribers may be unwilling to cease drugs that appear to be working and importantly doing no harm; and patients are unlikely to request the drug be stopped. The result is that therapy once started may be continued long term. Further, reminders for this kind of clinical review may be perceived as undermining professional autonomy. [74] Clearly, more studies need to be conducted in this domain to determine if these issues apply.

Our findings that system-initiated decision support systems are more effective than user-initiated systems, and institutional interventions are more effective than those conducted in community practice, are not surprising. Compared with decision support where users are required to initiate systems manually, automatic prompts are likely to dovetail more closely with physician workflow and provide opportunities to work on fine-tuning decisions or correct shortfalls or errors in care.[1,2] The success of institutional interventions is likely attributed to the type of conditions managed in this setting, the stricter controls on the practices of health care professionals in institutional settings and a potentially greater willingness to abide by 'externally' imposed rules and management suggestions.

Our finding that multi-faceted interventions appeared no more effective than CDSS interventions alone conflicts with a number of other reviews [12-14] and also appears counterintuitive from a behavioural perspective. However, in a review of more than 200 studies assessing the effectiveness of guideline implementation strategies, Grimshaw and colleagues [75] found multi-faceted interventions did not appear to be more effective than single strategy interventions but emphasised the difficultly in drawing generalisable conclusions about multi-faceted interventions owing to the large number of different combinations used. Further, our results are likely to be confounded by the setting in which the interventions occurred, clinical condition being targeted and the system design and implementation.

Conclusion

With the ever increasing enthusiasm for, and implementation of CDSSs in medical practice it is paramount we determine their benefits and risks in the short, medium and longer term. As such, it is important to promote the rigorous testing of these systems to produce high quality evidence about their clinical and economic impacts. Our review and others demonstrate CDSSs have merit in supporting specific aspects of the prescribing process. However, despite the substantial number of CDSS interventions in this review our understanding of the impacts of these interventions in particular clinical domains and settings is still relatively limited. Further, interventions to date have largely focussed on supporting clinicians in initiating and monitoring rather than ceasing therapy.

System developers and policy makers should build on the consistently effective CDSS approaches identified in this research such as prompts and alerts incorporating 'do no harm messages', reminders for patients on long-term therapy and care suggestions for 'at risk' patients. In addition, further studies are required to determine the relative merits of particular types of CDSS and the factors related to the successful implementation of these systems. Our understanding of the impact of CDSSs in clinical practice would be much enhanced by detailed reporting on the individual features of the systems, the way in which they were developed and the specifics of the environment in which they were deployed.

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

SP conceived and designed the study, participated in data collection and analysis, interpreted findings and drafted the manuscript. AM contributed to the study design, data collection and analysis and interpretation of findings. JR conceived and designed the study and participated in data collection and analysis and interpretation of findings. IH contributed to the study design, data collection and analysis and interpretation of findings. MW contributed to the study design and interpretation of findings. JR contributed to the study design and interpretation of findings. DN conceived and designed the study and participated in data collection and analysis and interpretation of findings. All authors read and approved the final manuscript.

Acknowledgements

This project was funded by the National Prescribing Service (NPS) Ltd as part of a research partnership with the Universities of Newcastle and New South Wales. The NPS established a Study Guidance Group comprising representatives from each partner organisation. The Group took oversight of the development and implementation of the research. We acknowledge the contributions of other Study Guidance Group Members: Dr Bryn Lewis, Mr Malcolm Gillies, Mr Adi Smith, Ms Michelle Toms, Ms Michelle Sweidan.

References

  1. Garg AX, Adhikari NKJ, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, Sam J, Haynes RB: Effects of computerized clinical decision support systems on practitioner performance and patient outcomes. A systematic review.

    JAMA 2005, 293:1223-1238. PubMed Abstract | Publisher Full Text OpenURL

  2. Kawamoto K, Houlihan CA, Balas EA, Lobach DF: Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success.

    BMJ 2005, 330:765. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  3. Sintchenko V, Magrabi F, Tipper S: Are we measuring the right end-points? Variables that affect the impact of computerised decision support on patient outcomes: a systematic review.

    Med Inform Internet Med 2007, 32:225-240. PubMed Abstract | Publisher Full Text OpenURL

  4. Walton RT, Harvey E, Dovey S, Freemantle N: Computerised advice on drug dosage to improve prescribing practice.

    Cochrane Database Syst Rev 2001, (1):CD002894. PubMed Abstract | Publisher Full Text OpenURL

  5. Durieux P, Trinquart L, Colombet I, Nies J, Walton R, Rajeswaran A, Rege Walther M, Harvey E, Burnand B: Computerized advice on drug dosage to improve prescribing practice.

    Cochrane Database Syst Rev 2008, CD002894. PubMed Abstract | Publisher Full Text OpenURL

  6. Eslami S, Abu-Hanna A, de Keizer NF: Evaluation of outpatient computerized physician medication order entry systems: a systematic review.

    J Am Med Inform Assoc 2007, 14:400-406. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  7. Kaushal R, Shojania KG, Bates DW: Effects of computerized physician order entry and clinical decision support systems on medication safety. A systematic review.

    Arch Intern Med 2003, 163:1409-1416. PubMed Abstract | Publisher Full Text OpenURL

  8. Bennett JW, Glasziou PP: Computerised reminders and feedback in medication management: a systematic review of randomised controlled trials.

    Med J Aust 2003, 178:217-222. PubMed Abstract | Publisher Full Text OpenURL

  9. Mollon B, Chong J, Holbrook A, Sung M, Thabane L, Foster G: Features predicting the success of computerized decision support for prescribing: A systematic review of randomized controlled trials.

    BMC Medical Informatics & Decision Making 2009, 9:11. BioMed Central Full Text OpenURL

  10. Patterson ES, Doebbeling BN, Fung CH, Militello L, Anders S, Asch SM: Identifying barriers to the effective use of clinical reminders: bootstrapping multiple methods.

    Journal of Biomedical Informatics 2005, 38:189-199. PubMed Abstract | Publisher Full Text OpenURL

  11. Varonen H, Kortteisto T, Kaila M: What may help or hinder the implementation of computerized decision support systems (CDSSs): a focus group study with physicians.

    Fam Pract 2008, 25:162-167. PubMed Abstract | Publisher Full Text OpenURL

  12. Grimshaw JM, Shirran L, Thomas R, Mowatt G, Fraser C, Bero L, Grilli R, Harvey E, Oxman A, O'Brien MA: Changing provider behavior: an overview of systematic reviews of interventions.

    Med Care 2001, 39:II2-45. PubMed Abstract | Publisher Full Text OpenURL

  13. Lu C, Ross-Degnan D, Soumerai S, Pearson S: Interventions designed to improve the quality and efficiency of medication use in managed care: a critical review of the literature - 2001-2007.

    BMC Health Serv Res 2008, 8:75. PubMed Abstract | BioMed Central Full Text | PubMed Central Full Text OpenURL

  14. Pearson SA, Ross-Degnan D, Payson A, Soumerai SB: Changing medication use in managed care: A critical review of the available evidence.

    Am J Manag Care 2003, 9:715-731. PubMed Abstract | Publisher Full Text OpenURL

  15. Bero L, Deane K, Eccles M, Grimshaw J, Gruen R, Mayhew A, Oxman AD, Pantoja T, Paulsen E, Shepperd S, et al.: Cochrane Effective Practice and Organisation of Care Group.

    About The Cochrane Collaboration (Cochrane Review Groups (CRGs)) 2009. OpenURL

  16. Ansari M, Shlipak MG, Heidenreich PA, Van Ostaeyen D, Pohl EC, Browner WS, Massie BM: Improving guideline adherence: a randomized trial evaluating strategies to increase beta-blocker use in heart failure.

    Circulation 2003, 107:2799-2804. PubMed Abstract | Publisher Full Text OpenURL

  17. Apkon M, Mattera JA, Lin Z, Herrin J, Bradley EH, Carbone M, Holmboe ES, Gross CP, Selter JG, Rich AS, Krumholz HM: A randomized outpatient trial of a decision-support information technology tool.

    Arch Intern Med 2005, 165:2388-2394. PubMed Abstract | Publisher Full Text OpenURL

  18. Bloomfield HE, Nelson DB, van Ryn M, Neil BJ, Koets NJ, Basile JN, Samaha FF, Kaul R, Mehta JL, Bouland D: A trial of education, prompts, and opinion leaders to improve prescription of lipid modifying therapy by primary care physicians for patients with ischemic heart disease.

    Qual Saf Health Care 2005, 14:258-263. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  19. Cobos A, Vilaseca J, Asenjo C, Pedro-Botet J, Sanchez E, Val A, Torremade E, Espinosa C, Bergonon S: Cost effectiveness of a clinical decision support system based on the recommendations of the European Society of Cardiology and other societies for the management of hypercholesterolemia: report of a cluster-randomized trial.

    Dis Manag Health Outcomes 2005, 13:421-432. Publisher Full Text OpenURL

  20. Demakis JG, Beauchamp C, Cull WL, Denwood R, Eisen SA, Lofgren R, Nichol K, Woolliscroft J, Henderson WG: Improving residents' compliance with standards of ambulatory care. Results from the VA Cooperative Study on Computerized Reminders.

    JAMA 2000, 284:1411-1416. PubMed Abstract | Publisher Full Text OpenURL

  21. Dexter PR, Perkins S, Overhage JM, Maharry K, Kohler RB, McDonald CJ: A computerized reminder system to increase the use of preventive care for hospitalized patients.

    N Engl J Med 2001, 345:965-970. PubMed Abstract | Publisher Full Text OpenURL

  22. Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D, Purves I: Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial.

    BMJ 2002, 325:941. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  23. Feldstein A, Elmer PJ, Smith DH, Herson M, Orwoll E, Chen C, Aickin M, Swain MC: Electronic medical record reminder improves osteoporosis management after a fracture: a randomized, controlled trial.

    J Am Geriatr Soc 2006, 54:450-457. PubMed Abstract | Publisher Full Text OpenURL

  24. Filippi A, Sabatini A, Badioli L, Samani F, Mazzaglia G, Catapano A, Cricelli C: Effects of an automated electronic reminder in changing the antiplatelet drug-prescribing behavior among Italian general practitioners in diabetic patients: an intervention trial.

    Diabetes Care 2003, 26:1497-1500. PubMed Abstract | Publisher Full Text OpenURL

  25. Flottorp S, Oxman AD, Havelsrud K, Treweek S, Herrin J: Cluster randomised controlled trial of tailored interventions to improve the management of urinary tract infections in women and sore throat.

    BMJ 2002, 325:367. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  26. Fretheim A, Oxman AD, Havelsrud K, Treweek S, Kristoffersen DT, Bjorndal A: Rational prescribing in primary care (RaPP): a cluster randomized trial of a tailored intervention.

    PLoS Med 2006, 3:e134. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  27. Kralj B, Iverson D, Hotz K, Ashbury FD: The impact of computerized clinical reminders on physician prescribing behavior: evidence from community oncology practice.

    Am J Med Qual 2003, 18:197-203. PubMed Abstract | Publisher Full Text OpenURL

  28. Krall MA, Traunweiser K, Towery W: Effectiveness of an electronic medical record clinical quality alert prepared by off-line data analysis. In Medinfo 2004: Proceedings of the 11th World Congress on Med Inform; San Francisco, California. Edited by Fieschi M, Coiera E, Li Y-CJ. IOS Press; 2004:135-139. OpenURL

  29. Kucher N, Koo S, Quiroz R, Cooper JM, Paterno MD, Soukonnikov B, Goldhaber SZ: Electronic alerts to prevent venous thromboembolism among hospitalized patients.

    N Engl J Med 2005, 352:969-977. PubMed Abstract | Publisher Full Text OpenURL

  30. Kuperman GJ, Teich JM, Tanasijevic MJ, Ma'Luf N, Rittenberg E, Jha A, Fiskio J, Winkelman J, Bates DW: Improving response to critical laboratory results with automation. Results of a randomized controlled trial.

    J Am Med Inform Assoc 1999, 6:512-522. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  31. Montgomery AA, Fahey T, Peters TJ, MacIntosh C, Sharp DJ: Evaluation of computer based clinical decision support system and risk chart for management of hypertension in primary care: randomised controlled trial.

    BMJ 2000, 320:686-690. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  32. Murray MD, Harris LE, Overhage JM, Zhou XH, Eckert GJ, Smith FE, Buchanan NN, Wolinsky FD, McDonald CJ, Tierney WM: Failure of computerized treatment suggestions to improve health outcomes of outpatients with uncomplicated hypertension: results of a randomized controlled trial.

    Pharmacotherapy 2004, 24:324-337. PubMed Abstract | Publisher Full Text OpenURL

  33. Overhage JM, Tierney WM, McDonald CJ: Computer reminders to implement preventive care guidelines for hospitalized patients.

    Arch Intern Med 1996, 156:1551-1556. PubMed Abstract | Publisher Full Text OpenURL

  34. Paul M, Andreassen S, Tacconelli E, Nielsen AD, Almanasreh N, Frank U, Cauda R, Leibovici L, Group TS: Improving empirical antibiotic treatment using TREAT, a computerized decision support system: cluster randomized trial.

    J Antimicrob Chemother 2006, 58:1238-1245. PubMed Abstract | Publisher Full Text OpenURL

  35. Roumie CL, Elasy TA, Greevy R, Griffin MR, Liu X, Stone WJ, Wallston KA, Dittus RS, Alvarez V, Cobb J, Speroff T: Improving blood pressure control through provider education, provider alerts, and patient education: a cluster randomized trial.

    Ann Intern Med 2006, 145:165-175. PubMed Abstract OpenURL

  36. Safran C, Rind DM, Davis RB, Ives D, Sands DZ, Currier J, Slack WV, Makadon HJ, Cotton DJ: Guidelines for management of HIV infection with computer-based patient's record.

    Lancet 1995, 346:341-346. PubMed Abstract | Publisher Full Text OpenURL

  37. Sequist TD, Gandhi TK, Karson AS, Fiskio JM: A randomized trial of electronic clinical reminders to improve quality of care for diabetes and coronary artery disease.

    J Am Med Inform Assoc 2005, 12:431-437. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  38. Tierney WM, Overhage JM, Murray MD, Harris LE, Zhou XH, Eckert GJ, Smith FE, Nienaber N, McDonald CJ, Wolinsky FD: Effects of computerized guidelines for managing heart disease in primary care.

    J Gen Intern Med 2003, 18:967-976. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  39. Tierney WM, Overhage JM, Murray MD, Harris LE, Zhou XH, Eckert GJ, Smith FE, Nienaber N, McDonald CJ, Wolinsky FD: Can computer-generated evidence-based care suggestions enhance evidence-based management of asthma and chronic obstructive pulmonary disease? A randomized, controlled trial.

    Health Serv Res 2005, 40:477-497. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  40. Flanagan JR, Doebbeling BN, Dawson J, Beekmann S: Randomized study of online vaccine reminders in adult primary care.

    AMIA Annual Symposium Proceedings 1999, 755-759. OpenURL

  41. Tang PC, LaRosa MP, Newcomb C, Gorden SM: Measuring the effects of reminders for outpatient influenza immunizations at the point of clinical opportunity.

    J Am Med Inform Assoc 1999, 6:115-121. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  42. Tamblyn R, Huang A, Perreault R, Jacques A, Roy D, Hanley J, McLeod P, Laprise R: The medical office of the 21st century (MOXXI): effectiveness of computerized decision-making support in reducing inappropriate prescribing in primary care.

    Can Med Assoc J 2003, 169:549-556. OpenURL

  43. Smith DH, Perrin N, Feldstein A, Yang X, Kuang D, Simon SR, Sittig DF, Platt R, Soumerai SB: The impact of prescribing safety alerts for elderly persons in an electronic medical record. An interrupted time series evaluation.

    Arch Intern Med 2006, 166:1098-1104. PubMed Abstract | Publisher Full Text OpenURL

  44. Shojania KG, Yokoe D, Platt R, Fiskio J, Ma'Luf N, Bates DW: Reducing vancomycin use utilizing a computer guideline: results of a randomized controlled trial.

    J Am Med Inform Assoc 1998, 5:554-562. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  45. Peterson JF, Rosenbaum BP, Waitman LR, Habermann R, Powers J, Harrell D, Miller RA: Physicians' response to guided geriatric dosing: initial results from a randomized trial.

    Stud Health Technol Inform 2007, 129:1037-1040. PubMed Abstract | Publisher Full Text OpenURL

  46. Martens JD, Weijden T, Severens JL, de Clercq PA, de Bruijn DP, Kester ADM, Winkens RAG: The effect of computer reminders on GPs' prescribing behaviour: a cluster-randomised trial.

    Int J Med Inform 2007, 76:S403-S416. PubMed Abstract | Publisher Full Text OpenURL

  47. Madaras-Kelly KJ, Remington RE, Lewis PG, Stevens DL: Evaluation of an intervention designed to decrease the rate of nosocomial methicillin-resistant Staphylococcus aureus infection by encouraging decreased fluoroquinolone use.

    Infect Control Hosp Epidemiol 2006, 27:155-169. PubMed Abstract | Publisher Full Text OpenURL

  48. Kuilboer MM, van Wijk MA, Mosseveld M, van der DE, de Jongste JC, Overbeek SE, Ponsioen B, van der LJ: Computed critiquing integrated into daily clinical practice affects physicians' behavior - a randomized clinical trial with AsthmaCritic.

    Methods Inf Med 2006, 45:447-454. PubMed Abstract | Publisher Full Text OpenURL

  49. Judge J, Field TS, DeFlorio M, Laprino J, Auger J, Rochon P, Bates DW, Gurwitz JH: Prescribers' responses to alerts during medication ordering in the long term care setting.

    J Am Med Inform Assoc 2006, 13:385-390. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  50. Hulgan T, Rosenbloom ST, Hargrove F, Talbert DA, Arbogast PG, Bansal P, Miller RA, Kernodle DS: Oral quinolones in hospitalized patients: an evaluation of a computerized decision support intervention.

    J Intern Med 2004, 256:349-357. PubMed Abstract | Publisher Full Text OpenURL

  51. Feldstein AC, Smith DH, Perrin N, Yang X, Simon SR, Krall MA, Sittig DF, Ditmer D, Platt R, Soumerai SB: Reducing warfarin medication interactions. An interrupted time series evaluation.

    Arch Intern Med 2006, 166:1009-1015. PubMed Abstract | Publisher Full Text OpenURL

  52. Davis RL, Wright J, Chalmers F, Levenson L, Brown JC, Lozano P, Christakis DA: A cluster randomized clinical trial to improve prescribing patterns in ambulatory pediatrics.

    PLoS Clinical Trials 2007, 2:e25. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  53. Christakis DA, Zimmerman FJ, Wright JA, Garrison MM, Rivara FP, Davis RL: A randomized controlled trial of point-of-care evidence to improve the antibiotic prescribing practices for otitis media in children [abstract].

    Pediatrics 2001, 107:392. Publisher Full Text OpenURL

  54. Ageno W, Turpie AGG: A randomized comparison of a computer-based dosing program with a manual system to monitor oral anticoagulant therapy.

    Thromb Res 1998, 91:237-240. PubMed Abstract | Publisher Full Text OpenURL

  55. Ageno W, Johnson J, Nowacki B, Turpie AGG: A computer generated induction system for hospitalized patients starting on oral anticoagulant therapy.

    Thromb Haemost 2000, 83:849-852. PubMed Abstract | Publisher Full Text OpenURL

  56. Burton ME, Ash CL, Hill DP, Handy T, Shepherd MD, Vasko MR: A controlled trial of the cost benefit of computerized bayesian aminoglycoside administration.

    Clin Pharmacol Ther 1991, 49:685-694. PubMed Abstract OpenURL

  57. Casner PR, Reilly R, Ho H: A randomized controlled trial of computerized pharmacokinetic theophylline dosing versus empiric physician dosing.

    Clin Pharmacol Ther 1993, 53:684-690. PubMed Abstract OpenURL

  58. Feldstein AC, Smith DH, Perrin N, Yang X, Rix M, Raebel MA, Magid DJ, Simon SR, Soumerai SB: Improved therapeutic monitoring with several interventions. A randomized trial.

    Arch Intern Med 2006, 166:1848-1854. PubMed Abstract | Publisher Full Text OpenURL

  59. Fihn SD, McDonell MB, Vermes D, Henikoff JG, Martin DC, Callahan CM, Kent DL, White RH: A computerized intervention to improve timing of outpatient follow-up: a multicenter randomized trial in patients treated with warfarin. National Consortium of Anticoagulation Clinics.

    J Gen Intern Med 1994, 9:131-139. PubMed Abstract | Publisher Full Text OpenURL

  60. Koide D, Ohe K, Ross-Degnan D, Kaihara S: Computerized reminders to monitor liver function to improve the use of etretinate.

    Int J Med Inform 2000, 57:11-19. PubMed Abstract | Publisher Full Text OpenURL

  61. Lesourd F, Avril C, Boujennah A, Parinaud J: A computerized decision support system for ovarian stimulation by gonadotrophins.

    Fertil Steril 2002, 77:456-460. PubMed Abstract | Publisher Full Text OpenURL

  62. Lester WT, Grant RW, Barnett GO, Chueh HC: Randomized controlled trial of an informatics-based intervention to increase statin prescription for secondary prevention of coronary disease.

    J Gen Intern Med 2006, 21:22-29. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  63. Manotti C, Moia M, Palareti G, Pengo V, Ria L, Dettori AG: Effect of computer-aided management on the quality of treatment in anticoagulated patients: a prospective, randomized, multicenter trial of APROAT (Automated PRogram for Oral Anticoagulant Treatment).

    Haematologica 2001, 86:1060-1070. PubMed Abstract | Publisher Full Text OpenURL

  64. McCowan C, Neville RG, Ricketts IW, Warner FC, Hoskins G, Thomas GE: Lessons from a randomized controlled trial designed to evaluate computer decision support software to improve the management of asthma.

    Med Inform Internet Med 2001, 26:191-201. PubMed Abstract | Publisher Full Text OpenURL

  65. Mitra R, Marciello MA, Brain C, Ahangar B, Burke DT: Efficacy of computer-aided dosing of warfarin among patients in a rehabilitation hospital.

    Am J Phys Med Rehabil 2005, 84:423-427. PubMed Abstract | Publisher Full Text OpenURL

  66. Overhage JM, Tierney WM, Zhou XH, McDonald CJ: A randomized trial of "corollary orders" to prevent errors of omission.

    J Am Med Inform Assoc 1997, 4:364-375. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  67. Palen TE, Raebel M, Lyons E, Magid DM: Evaluation of laboratory monitoring alerts within a computerized physician order entry system for medication orders.

    Am J Manag Care 2006, 12:389-395. PubMed Abstract | Publisher Full Text OpenURL

  68. Poller L, Wright D, Rowlands M: Prospective comparative study of computer programs used for management of warfarin.

    J Clin Pathol 1993, 46:299-303. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  69. Poller L, Shiach CR, MacCallum PK, Johansen AM, Munster AM, Magalhaes A, Jespersen J: Multicentre randomised study of computerised anticoagulant dosage.

    Lancet 1998, 352:1505-1509. PubMed Abstract | Publisher Full Text OpenURL

  70. Vadher B, Patterson DL, Leaning M: Evaluation of a decision support system for initiation and control of oral anticoagulation in a randomised trial.

    BMJ 1997, 314:1252-1256. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  71. Verner D, Seligmann H, Platt S, Dany S, Almog S, Zulty L, Halkin H, Ezra D: Computer assisted design of a theophylline dosing regimen in acute bronchospasm: serum concentrations and clinical outcome.

    Eur J Clin Pharmacol 1992, 43:29-33. PubMed Abstract | Publisher Full Text OpenURL

  72. Pearson SA, Soumerai S, Mah C, Zhang F, Simoni-Wastila L, Salzman C, Cosler LE, Fanning T, Gallagher P, Ross-Degnan D: Racial disparities in access after regulatory surveillance of benzodiazepines.

    Arch Intern Med 2006, 166:572-579. PubMed Abstract | Publisher Full Text OpenURL

  73. Simoni-Wastila L, Ross-Degnan D, Mah C, Gao X, Brown J, Cosler LE, Fanning T, Gallagher P, Salzman C, Soumerai SB: A retrospective data analysis of the impact of the New York triplicate prescription program on benzodiazepine use in medicaid patients with chronic psychiatric and neurologic disorders.

    Clin Ther 2004, 26:322-336. PubMed Abstract | Publisher Full Text OpenURL

  74. Tierney WM, Miller ME, Overhage JM, McDonald CJ: Physician inpatient order writing on microcomputer workstations: effects on resource utilization.

    JAMA 1993, 269:379-383. PubMed Abstract | Publisher Full Text OpenURL

  75. Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, Vale L, Whitty P, Eccles MP, Matowe L, Shirran L, et al.: Effectiveness and efficiency of guideline dissemination and implementation strategies.

    Health Technol Assess 2004, 8:iii-iv.

    1-72

    PubMed Abstract | Publisher Full Text OpenURL

Pre-publication history

The pre-publication history for this paper can be accessed here:

http://www.biomedcentral.com/1472-6963/9/154/prepub