In consultation with a Library Information Specialist familiar with the field, we searched Medline, CINAHL, Healthstar, ERIC, Cochrane Central Register of Controlled Trials, and Psychinfo from inception to February 2006 (Table 1). Ancestry searches were conducted on relevant studies, and systematic reviews indexed in the Cochrane Database of Systematic Reviews and elsewhere 67891011. We searched grey literature using the System for Information on Grey Literature database (SIGLE), the New York Academy of Medicine, and the Sarah Cole Hirsch Institute. We retrieved the majority of relevant studies from our database search from the Journal of Nursing Care Quality, MEDSURG Nursing, Journal of Clinical Nursing and Journal of Gerontological Nursing. We manually searched these journals from 1990 (or their inception) to 2006.

A study was eligible for inclusion if: 1) it was a randomized controlled trial (RCT) or controlled before and after (CBA) design, 2) authors evaluated interventions aimed at increasing research use or evidence-based practice, 3) participants were nurses, and 4) outcomes directly and explicitly captured research use. Only studies in English were assessed.

For criterion one, we defined RCT and CBA using Cochrane definitions. To meet criterion two, investigators must have explicitly stated that the research purpose was to test an intervention aimed at increasing research or evidence-based practice. For criterion three, we included both registered and student nurses and did not exclude based on type of nurse (i.e., psychiatric nurse, license practical nurse, etc). However, we did not include studies of nurse practitioners because, we argue, their practice has more similarities to medical practice than nursing. To meet criterion four, investigators must have explicitly described how their chosen outcomes represented research use or have used an instrument designed explicitly to measure research use. We excluded studies unless authors were explicitly clear as to how chosen outcomes captured a conceptualization of research use. This was a clear decision when authors used a tool designed to measure research use. However, to be included when a change in provider behavior was the outcome, the investigator had to have clearly described how the behavior reflected research use. For example, in evaluating the implementation of a clinical practice guideline, the investigator needed to measure all recommended behaviors outlined in the guideline, identify the percentage of recommended behaviors that signified research use, or illustrate how outcomes reflected their conceptualization of research use. If this was not done, we could not be certain the investigators were measuring research use and so we excluded the study.

The search resulted in over 8,000 titles. One author reviewed titles, abstracts and selected studies. Two reviewers each screened 20% of the titles and abstracts. Inter-rater reliability between reviewers was greater than 90%. The initial screening process resulted in 117 studies. Manual and ancestry searching produced an additional 21 studies. Further review of the 138 studies narrowed them to 14 and the final result was four studies meeting the inclusion criteria 24252627: three RCTs and one CBA (Figure 1).

In the final exclusion of studies, ten of the studies were excluded for two reasons: uncertainty that the outcomes were measuring research use 28293031, and interventions not explicitly aimed at increasing research use or evidence-based practice 323334353637 (Table 2).

We evaluated the studies for methodological quality using two tools available from the Cochrane Collaboration Effective Practice and Organization of Care Group (EPOC) 38. The RCT tool consisted of items related to unit of analysis, power, baseline measure, concealment of allocation, blinded or objective assessment of outcome(s), protection against contamination, reliable outcome(s), and completeness of follow-up. The CBA tool consisted of items related to unit of analysis, power, baseline measure, comparability of groups, blinded or objective assessment of outcome(s), protection against contamination, reliable outcome(s), and completeness of follow-up. In both tools, unit of analysis errors were determined using the unit of allocation and unit of analysis items. That is, if authors allocated by cluster and analyzed by individual without reporting appropriate statistical measures to account for clustering, we reported unit of analysis errors. If in these cases the authors reported power calculations and did not account for intra-cluster correlations, we scored the power calculation item as done but accounted for the error in the overall rating. We report results in Table 3.

Two reviewers assessed each study and discrepancies were resolved through discussion. Each item was scored as: done, not done, and not clear. A quality rating was assigned to each study as low, medium, or high depending whether it scored done on zero to four, five to six, or seven to eight items respectively. Unit of analysis errors and incorrect power calculations were noted. We did not use quality assessment ratings to exclude studies because we sought to explore the general state of the science in this field.

We extracted data from four studies representing five experimental cohorts where an intervention was compared to a control. One reviewer independently extracted data from all four studies while two reviewers each extracted data from two of the studies. We used extraction tools and dictionaries available from EPOC 38. Data on design, subjects, setting, interventions and outcomes were extracted.

To facilitate comparison and discussion, we classified interventions using an EPOC classification system 38. Interventions were classified as: educational meetings, multidisciplinary committees and local opinion leaders. The EPOC classification is used throughout the text (Table 4).

Several studies in this review reported additional outcomes, for example, on predictors of research use, changes in knowledge or attitudes, or patient outcomes. These were not extracted or reported on as we did not consider them as measures of research use per se.

Overall, the quality of the studies was low (Table 3). Two had unit of analysis errors where the investigators allocated by group but did not account for clustering in the analysis 2425. Of the two studies without unit of analysis errors, the investigators of one study allocated by unit and accounted for clustering 26, while the other allocated and analyzed at the provider level 27. No authors presented power calculations. Two studies had substantial differences in outcomes prior to the intervention 2426. Allocation concealment was not reported in two RCTs 2426. None of the investigators used blinded or reliable outcome assessments. The CBA investigators did not protect against contamination of the intervention across study groups 27. However, the RCT investigators all randomized by ward and attempted to protect against contamination 242526. The CBA investigator reported adequate provider follow up 27. However, the RCT investigators either used separate samples 2526, or did not report on follow up 24.

Four studies representing five intervention cohorts in Canada, USA, Taiwan, and Hong Kong met our inclusion criteria (Table 4). Three were RCTs (four intervention cohorts) 242526, and one was a CBA (one intervention cohort) 27. All studies included nurses from inpatient clinical settings; oncology, medicine, surgery and multiple specialties.

Investigators assessed educational meetings delivered to nurses in three studies 252627. In one study, the investigators compared two investigator-provided educational interventions to a control 26. Because these interventions varied in content and duration, we identified this study as having two cohorts. Another study used a combination of local experts and educators to deliver the intervention 27. The third study that assessed educational meetings used local opinion leaders identified by the study participants to conduct a demonstration tutorial which was supplemented with education delivered by a local expert 25. One study investigated the formation of a multidisciplinary team of practitioners and researchers 24. Within this intervention there were components of education and marketing. However, the investigators based their conclusions on the entire intervention (the multidisciplinary team) rather than the components, therefore, we did not separate the components of this intervention.

The investigators of three studies used nurse-administered instruments to measure research use. Dufault 24 used Kim's 39 13-item Likert-type scale that asked participants to rate their research utilization competency on a one to seven scale. Tranmer 26 used the Research Utilization Questionnaire (RUQ) developed by Champion and Leach 4041. This 42-item Likert-type questionnaire measured attitudes towards research, access to research, support of the use of research and research use. The questionnaire was divided into corresponding subscales. Because Tranmer 26 reported and analyzed the results of each subscale, we extracted only the data that pertained to the use of research subscale. Finally, using an instrument based on her previous work, Tsai 27 assessed whether research utilization was implemented in nursing practice and to what degree. Tsai's instrument consisted of 11 items including single-choice, multiple-choice and open-ended questions.

In the final study by Hong, investigators used self-reporting and participant observation to assess practice compliance with all the recommendations from a clinical practice guideline 25. This study differed from many of the excluded studies that assessed provider behavior change. Specifically, the investigators linked all eight outcomes to the eight practices recommended by the clinical guideline, which was referenced to research, thus providing support that the outcomes did reflect research use.

Methodological weaknesses, varied interventions and outcomes across health contexts, incomplete reporting, and the small samples prevented meta-analysis. Instead, we present narrative results. The characteristics and findings of the four studies included in this review are summarized in Tables 5 and 6. All findings must be interpreted with significant caution given the low quality of studies.

Two studies representing three cohorts tested the effect of interactive educational meetings on research utilization 2627. Tranmer measured research use both in nurses who participated and nurses from the same unit as those who participated 26. There were no significant changes in research utilization scores in either group. This suggests that, based on this study, educational meetings are ineffective whether a nurse participates directly (attending education meetings) or indirectly (working with nurses who attended educational meetings but not attending themselves). However, no definite conclusions can be drawn due to design limitations.

Educational meetings of varying content, frequency and duration (Table 6) were also found to be ineffective. Tranmer, who did not describe frequency of their intervention, reported non-significant changes in research utilization scores regardless of whether the intervention was twenty hours and focused on literature critiquing, research design, and protocol implementation, or eight hours and focused solely research design and implementation 26. These results are supported by Tsai's study, in which she tested a series of educational strategies focused on research use totaling 65 hours and delivered over eight weeks 27.

Interactive educational meetings did not have a delayed effect on research utilization. Tsai measured research use at two points: immediately and six months following the intervention. In both cases, there were no significant changes in research utilization 27. Similar findings were reported by Tranmer who measured research utilization only once, one year following the start of the intervention and also reported non-significant results 26.

In summary, based on this review, educational meetings of varying content, duration, and frequency cannot be said to be effective research utilization interventions in nursing. The studies were few in number and were of poor quality. Clearly, there is inconclusive evidence and educational meetings require more rigorous investigations to determine their effect in nursing.

One study tested the effect of interactive educational meetings combined with a local opinion leader, and found that nurses who attended both the lecture and the tutorial (led by a local opinion leader) reported increased research utilization related to urinary catheter practices 25. It was not possible to determine whether the positive effect was due to the local opinion leader, the educational meeting, or a combination of both. The intervention consisted of a 30 minute lecture by an educator, followed one week later by a demonstration tutorial conducted by a local opinion leader (Table 6). The length of the demonstration tutorial was not reported. No data were collected during the lapse between interventions. Outcomes were assessed twice: two weeks and two months following the intervention. The authors used a practice survey at two weeks, and direct observation at two months. Longitudinally, education and local opinion leaders appeared to sustain an increase in research utilization, but this study was also of low quality and represents inconclusive evidence for educational interventions combined with a local opinion leader.

One study was found in which formation of multidisciplinary committees was reported to be effective at increasing nurses' research use related to oncology pain 24. The intervention lasted 28 weeks and was divided into six stages (Table 6). Each stage was sequential and lasted between two and nine weeks. Stages were constructed around collaboration of members of the multidisciplinary team working to operationalize an existing research utilization process (the Conduct and Utilization of Research in Nursing Project) 42. Unlike other interventions, education was not the primary component. Outcomes were assessed at one point using a research utilization scale. The investigators did not report the duration between the intervention and outcome measurement. Multidisciplinary committees require further investigation.

In summary, the four studies reviewed were of poor quality 24252627. The findings of this review represent a lack of evidence to support or refute the benefit of educational meetings for increasing research utilization in nursing, and further study is required.

In a review of organizational interventions aimed at increasing evidence-based nursing practice, Foxcroft and Cole could locate no rigorous studies 11. Grimshaw and colleagues 678 published comprehensive reviews of provider behavior change reviews and guideline dissemination strategies. While we were interested specifically in nurses' research utilization and Grimshaw and colleagues 678 examined broader outcomes (provider behavior change and guideline dissemination), these reviews were all aimed at improving understanding of how to translate research findings into practice. Grimshaw and colleagues 7 concluded that interventions with different educational strategies showed mixed effects depending upon a combination of strategies. We report inconclusive evidence compared to these results 24252627. The educational interventions included in our review were small interactive group sessions. In medicine, these types of educational strategies showed the most promise. We did find some limited support for two interventions: multidisciplinary committees and local opinion leaders. Grimshaw and colleagues 7 also found that multidisciplinary collaboration was effective and that use of local opinion leaders showed mixed effects.

Similarities and differences between these reviews can be attributed to multiple factors. Perhaps the most obvious is in the review methods. Grimshaw and colleagues 8 had a more robust dataset and derived a single effect size for each of the 235 studies reviewed, as well as summarizing the range of effects and median effects across studies for each intervention. In contrast, we were only able to locate four studies, and were limited to a narrative analysis based on the number of positive and negative results (vote counting).

Secondly, investigators of existing reviews have focused on changing provider behavior 67, implementing practice guidelines 810 and conducting continuing education 9. We located no review focusing specifically on research utilization. Instead, authors have relied upon changes in patient outcomes, provider behavior, or a combination of both. For example, in a systematic review of guidelines in professions allied to medicine, 24% of nursing studies measured patient outcomes, 24% measured provider behavior, and 53% measured both provider and patient outcomes 10. Additionally, all four nursing studies included in a review of continuing education and workshops measured provider outcomes 9. Findings from these reviews 678910 are increasingly being used to guide strategies aimed at increasing research use in clinical practice 13. Hakkennes and Green suggested that authors must choose between assessing changes in provider practice or changes in patient outcomes when evaluating interventions aimed at implementing evidence or changing practice 13. While this is true for the latter, we argue there is a third and perhaps more accurate choice: assessing changes in research use. By not assessing research use and moving directly to changes in provider or patient outcomes, investigators are treating research utilization interventions as a 'black box' phenomenon that somehow produces a change in clinician behavior or patient outcomes. Clearly, there are additional factors that influence both clinician and patient outcomes. The assumption that research use is the only mediating factor represents a large gap in the literature, and thus our understanding of how to increase the use of research in practice.

A major conceptual issue we identified is related to outcome measurement. We excluded several studies due to unclear conceptualizations of research use (Table 2). Investigators have commonly aligned themselves with a model of evidence-based practice consisting of five steps: 1) converting information needs to an answerable question, 2) locating the evidence, 3) critically appraising the evidence, 4) implementing the evidence in practice, and 5) evaluating care performances 43. Most often, evaluations of interventions designed to increase evidence-based practice (or in our case, research use) relied upon behavior change outcomes 13. Using the model outlined above, this approach is synonymous with evaluating care performances (Step five). However, research use is only one of the factors that influence care performance or behavior change 212223. Reliance on behavior change outcomes has treated the implementation stage (Step four) also as a 'black box' and led to less understanding of how implementation of research in practice occurs. We and others argue that the key to increasing research use in practice lies in the implementation stage 6224445.

There is a lack of clarity and uncertainty, in fact a near silence, in the research community about what constitutes an appropriate measure of research use 464748. This silence can be partly to blame for a lack of research examining the intervention stage of research use and can be attributed to a poor understanding of the conceptual structure of research utilization 4649. Ideally, selecting outcomes to assess the effectiveness of an intervention aimed at increasing research use should be informed by an explicit conceptualization of research use 2250. Only two authors in our review explicated how they conceptualized research utilization 2426; both offered different conceptualizations, and it was not clear from either how their conceptualization informed outcome selection. Rich argues that misconceptions of how research-based knowledge enters the decision-making process lead to inaccurate measures of research use 51. Estabrooks and colleagues suggested that "unresolved measurement challenges present an important and practical problem" to advancing the field of research utilization 22. The findings of this review support these claims and suggest such issues persist.

Investigators are interested in the link between using research in practice and improving patient outcomes. The abundance of studies focusing on behavior change and patient outcomes as a result of research uptake points at this interest. However, establishing this link is best accomplished if we first develop sufficient evidence to support the relationship between specific interventions and research use. From this, we can explore the relationship between effective research use interventions and behavior change or patient outcomes. If studies aim to evaluate an intervention to increase research use, outcomes must be structured to capture changes in research use. More attention to the fit between study outcomes and the conceptual structure of research use will advance the field by peering into the 'black box' and producing more accurate results.

Common problems with the instruments used in the studies we reviewed, and elsewhere, include lack of theory (measurement or research utilization theory), lack of construct clarity, lack of psychometric assessment, a presumption of linearity, lack of longitudinal work, and influential yet unacknowledged assumptions 22. Until more reliable and valid instruments are developed, investigators should present explicit statements outlining the conceptual and practical basis for chosen outcomes. Making use of available conceptualizations to operationalize research use would increase the validity and ability to compare results across studies 1617, 515253.

The studies in our review were published between 1990 to 2003. Methodological quality (Table 3) was low in all four 24252627. This suggests that the field is not developing within nursing as would be expected. We present what we believe are the most urgent methodological challenges facing the field.

A primary outcome helps determine the key endpoint signifying the effectiveness of an intervention 54. Explicit reporting of the primary outcome enables the reader to determine whether the study results provide sufficient evidence for an intervention and to whom the study results apply. In our review, we extracted only research use outcomes. However, three investigators in this review also reported outcomes additional to research use and all three assessed attitude towards research 242627. The relationship between such characteristics and research use is not well-supported 55. When authors report on multiple outcomes without discussing why particular measures were chosen or what constitutes the primary outcome, it is difficult to interpret study findings in the context of research utilization.

The challenge in using multiple outcomes to evaluate research utilization interventions is determining the number that must be changed to indicate effectiveness 33. We excluded many studies due to uncertainty that the investigators were actually measuring research use (Table 2). In these cases, rationale or support for multiple outcomes in the context of research utilization was not provided. It is challenging to determine whether an intervention was effective at increasing research use if there were changes in some, but not all, of the outcomes. Also challenging is determining how many recommendations from clinical practice guidelines must be met to indicate research use, and for this review we included only one study that measured all recommended practices 25. Measuring all outcomes may not be the most accurate or feasible approach, especially if guidelines recommend large numbers of practices or procedures.

Two studies 2427 measured longitudinal outcomes; one illustrated a benefit of intervention over time (two months) 24 and the other illustrated no effect either immediately or six months following 27. Longitudinal outcome measurements will advance our understanding of the optimal timing and frequency of outcome evaluation and are needed to establish the sustainability of research use. Titler has described two challenges in assessing sustainability of research utilization interventions: defining the boundary between the end of the intervention phase and the start of the sustainability phase, and timing the outcome measurement to differentiate between sustained improvements and residual effects 48. Compartmentalizing these stages becomes difficult when multiple and overlapping interventions are tested. Thus far, the literature on research utilization provides little guidance on the optimal timing or length of outcome measurement for different interventions. Hong 25 and Tsai 27 did not report why they assessed outcomes at two and six months. Future investigators should clearly describe intervention characteristics such as duration and frequency, deliverer and receiver, and mode of delivery. Guidelines such as the Consolidation of Standards for Reporting of Trials (CONSORT) 56 or the CONSORT statement for cluster RCTS 57 should be followed. Future reviews would also benefit from using a common classification system for interventions. We used a classification system proposed by EPOC. However, this approach may require adaptation for use in nursing and needs to be examined and validated.

Two RCTS included in our review had unit of analysis errors (Table 3) 2425. Unit of analysis errors occur when investigators assign clusters or groups of individuals to a study group (i.e., intervention or control), and then analyze as if each individual had an equal chance of being assigned to either group 58. When this occurs, outcomes for each individual are not independent of others within the same group. This is a unit of analysis error because people within clusters share similarities (i.e., burn unit and psychiatric nurses are each familiar with different practices) that may not be accounted for during analysis. When clustering is ignored, the number of participants required (sample size) is underestimated and the level of study significance (p value) is overestimated, resulting in an over-estimate of the precision of the result 59. Future studies should be designed and analyzed using methods that account for clustering if allocation is done by groups of individuals, and allocation procedures should be unbiased (i.e., central randomization) and explicitly outlined in study reports.

This systematic review has some limitations. First, we did not conduct a meta-analysis because of lack of effect sizes and a small size. The method we used (vote counting) is a crude estimate of effectiveness. Second, we used the EPOC classification that was developed for broad use 38. Its applicability specifically to nursing has yet to be established. Third, the four studies included were all of low quality. Including studies of low quality limits any conclusions. Fourth, we included only studies published in English. While researchers have reported that language restrictions do not change the results of systematic reviews on conventional medicine interventions 6061, the impact of language restriction on research utilization intervention systematic reviews is unknown. However, it is possible that we missed potential studies due to our language restriction. Finally, because we focused on research use as the outcome of interest specific to intervention effectiveness, comparisons between our review and those of others must be made cautiously.