Training practitioners in preparing systematic reviews: a cross-sectional survey of participants in the Australasian Cochrane Centre training program

Abstract

Background

Although systematic reviews of health care interventions are an invaluable tool for health care providers and researchers, many potential authors never publish reviews. This study attempts to determine why some people with interest in performing systematic reviews do not subsequently publish a review; and what steps could possibly increase review completion.

Methods

Cross-sectional survey by email and facsimile of the 179 participants in Australasian Cochrane Centre training events between 1998 and 2000.

Results

Ninety-two participants responded to the survey (51 percent). Response rate of deliverable surveys was 82 percent (92/112). The remainder of the participants had invalid or no contact information on file. More than 75 percent of respondents felt that the current workshops met their needs for training. The most critical barriers to completion of a Cochrane review were: lack of time (80 percent), lack of financial support (36 percent), methodological problems (23 percent) and problems with group dynamics (10 percent).

Conclusions

Strategies to protect reviewer time and increase the efficiency of the review process may increase the numbers of trained reviewers completing a systematic review.

Background

Increasing emphasis is being placed on evidence-based medicine. [1, 2] The best evidence for treatment interventions comes from systematic reviews of randomised controlled trials, [3] however, systematic reviews with meta-analysis of published and unpublished data are time- and labour-intensive.

A major player in the evidence-based medicine movement has been the Cochrane Collaboration, an international organisation committed to 'preparing, maintaining and promoting the accessibility of systematic reviews of the effects of health care interventions.' [4] The Australasian Cochrane Centre is one of the component centres of the Cochrane Collaboration, providing ongoing training and support to people in the Australasian region who prepare systematic reviews that are subsequently published on the Cochrane Library. [5] The ultimate goal of the training program is to increase the number and quality of completed Cochrane reviews, and ensure that they are routinely updated.

Currently, training occurs over two days, with the first day focusing on developing a protocol for a systematic review and the second, statistical analysis and interpretation of the review. Training sessions occur several times per year in locations throughout Australasia. In May 2001, the Australasian Cochrane Centre surveyed all of the participants of its training workshops during the years 1998 to 2000. We undertook this survey to:

• 1. determine why some people who attend Cochrane training workshop do not go on to publish a systematic review; and

• 2. what steps could increase systematic review completion.

Methods

Survey Population and Contact Information

Our survey population included the 179 participants of the Australasian Cochrane Centre's training workshops from 1998 to 2000. Contact information for the year 2000 participants was drawn from the Australasian Cochrane Centre database (initiated in January 2000) of all workshop participants and current Cochrane reviewers in Australasia. Several partial data systems provided contact information for 1998–1999 participants.

Questionnaire

Participants were asked to respond to a cross-sectional survey of 21 'yes' or 'no' questions with a provision for open-ended comments. [See Additional File for the questionnaire used in this study.] The format of our internally developed questionnaire did not lend itself to statistical testing for internal consistency. No demographic details were collected, so that responses could be deidentified after receipt of the questionnaire.

The initial questionnaire was sent via email or fax to all participants. A second email or fax was sent four weeks later, with a third survey sent via post to those participants who had not responded to the first two requests and had a postal address on file.

Analysis

Both quantitative and qualitative methods were used in data analysis. The structured questions were analysed by SPSS for Windows software, release 10.0.5, and results were presented in the form of frequencies and percentages. Content analysis was used to analyse the qualitative responses.

Publication status of workshop participants was determined by searching the Cochrane Library by each workshop participants' last name and first initial.

Results

Response rate

Thirty-five questionnaires were returned as undeliverable (20 percent), with an additional 32 participants having no contact information on file with the Australasian Cochrane Centre (18 percent). [see Figure 1] Of the 112 deliverable questionnaires (63 percent of participants), 54 participants (47 percent) responded after the first two mailings. With the third mailing, an additional 38 participants responded, creating an overall response rate for deliverable surveys of 82 percent. Eighteen people returned the survey without response, which provided us with 74 valid surveys (41 percent of participants) for analysis.

Figure 1

Survey Response of Workshop Participants

Publication status

Approximately 40 percent of these 1998–2000 workshop participants have published a protocol or review on the Cochrane Library, as of 2001 issue 4. However, due to issues of anonymity of questionnaire response, we are unable to stratify the results based upon publication status.

Findings from Questionnaire: Factors that Interfere with Systematic Review Completion

Lack of time emerged as the most critical barrier to completion of Cochrane reviews, with 80 percent of respondents citing this factor. Twenty respondents (36 percent) acknowledged lack of financial support, and twelve (21 percent), lack of institutional encouragement and support, as barriers to completion of their systematic review. Of the 31 people who offered qualitative comments, 11 suggested a need for dedicated time and support, with two needing better time management, and one wanting a grant to supply a research assistant. In a similar tone, seven people stated that other work or personal commitments take priority over Cochrane activities. [see Table 1]

Table 1 Barriers for Training in, and Completion of, Systematic Review of the Literature

Methodological barriers affected 23 percent of respondents. Three respondents specifically mentioned problems with translation of non-English papers, and two had no access to EMBASE (Excerpta Medica). In addition, one person was frustrated that a similar title for a Cochrane review had been registered, but not completed in a timely manner. Ten percent of respondents had problems with group dynamics, with two stating this factor as a significant delay to publication, and one complaining of contradictory editorial commentary. Thirteen respondents (22 percent) had yet to start a review. Nine respondents (16 percent) were working on a systematic review outside of the Cochrane Collaboration, with one qualitative commentator stating that the employer insisted on publication of the review prior to submission to Cochrane.

A substantial minority (38 percent) of respondents felt that follow-up communication, especially via email, would help them complete their review. From the qualitative comments, this email could occur every one to six months, with 11 of the 12 respondents suggesting a time period of every three months or less.

Findings from Questionnaire: Training Issues

Scheduling and content of the training workshops appeared to meet the needs of participants. Over 75 percent of respondents felt that the workshops were not too infrequent, too inconvenient, too far away, assumed too much knowledge, or were too long. Qualitative comments stressed the need for ongoing support after training. Three respondents requested a specific contact person, whereas another four asked for support to discuss general problems, statistical analysis issues, help selecting trials, and the creation of a bulletin board-style mentoring system. Three more respondents requested advanced training.

In January 2000, the Australasian Cochrane Centre began collecting formal immediate evaluations from workshop participants. Analysis of these data indicates that the average participant self-ranks only slightly above an absolute beginner in systematic review methodology. Mean satisfaction with content, presentations, opportunity to ask questions, and overall satisfaction range between 4.5 to 4.7, on a scale from 1 = highly dissatisfied to 5 = highly satisfied.

Finally, 83 percent of respondents stated that their Cochrane training had made them more likely to use the Cochrane Library to answer clinical questions. Only 45 percent, however, felt that their practice patterns had changed.

Discussion

Lack of time is by far the most commonly reported barrier to completion of a systematic review. Financial, institutional, personal, and methodological issues are not problematic for the majority of respondents. These findings have important implications for both reviewers and entities commissioning systematic reviews. In the absence of funding for all or, at least, part of a systematic review, it will continue to be difficult for reviewers to do this research by taking more time out of their busy lives. The challenge becomes the ability to support researchers in minimising the time they spend on the process of the review, without affecting quality.

Limitations in this study are several. First of all, the subjective nature of this study may have lead reviewers to state that lack of time was a barrier to completion of the project, when another reason, such as change in professional priorities was the 'true' barrier. Second, searching publications by last name and first initial may lead to either overstatement or understatement of publication status, as reviewers may have common last names, have changed their last names, or use a different first initial to the first name they use routinely.

Finally, we had a large number of invalid addresses on file at the Australasian Cochrane Centre. The disruption of a move of the Australasian Cochrane Centre from Adelaide to Melbourne in 1999, with resultant staff turnover and loss of records, is responsible for much of our difficulty. However, we still had invalid contact information for almost one-quarter of the 2000 participants, whose training occurred less than 18 months prior to this survey. This lack of contact information is problematic as this survey suggests email communication could facilitate review completion. Strategies we plan to implement to improve the currency of our database include:

• 1. discussion at training events of the importance of informing the Australasian Cochrane Centre of change of address,

• 2. reminder notices on newsletters, and

• 3. liaison with other Cochrane entities.

The year 2000 also saw a change in enrolment policy for the training workshops, allowing people to attend the course only if they had a title or protocol registered with the relevant Cochrane entity. Therefore, our results are likely to be more representative of recent participants of our program, committed to the Cochrane system.

Conclusions

Lack of time, not skill, is the most common barrier to completion of a systematic review for authors who have attended such training given by the Australasian Cochrane Centre. One solution lies in advancements in methodology to make systematic reviews more efficient, such as automating aspects of the review or providing authors with specialised, centralised services in literature searches, project management, and the like.

We expect that these advances in systematic review methodology, increased capacity for completing systematic reviews in Australasia, and realistic expectations regarding required investment of time for valid evidence will lead to an increased number and quality of systematic reviews published on the Cochrane Library. Ultimately, consumers and practitioners of health care around the globe should benefit from this increased knowledge.