Skip to main content

The quality of reporting of RCTs used within a postoperative pain management meta-analysis, using the CONSORT statement

Abstract

Background

Randomized controlled trials (RCTs) are routinely used in systematic reviews and meta-analyses that help inform healthcare and policy decision making. The proper reporting of RCTs is important because it acts as a proxy for health care providers and researchers to appraise the quality of the methodology, conduct and analysis of an RCT. The aims of this study are to analyse the overall quality of reporting in 23 RCTs that were used in a meta-analysis by assessing 3 key methodological items, and to determine factors associated with high quality of reporting. It is hypothesized that studies with larger sample sizes, that have funding reported, that are published in journals with a higher impact factor and that are in journals that have adopted or endorsed the CONSORT statement will be associated with better overall quality of reporting and reporting of key methodological items.

Methods

We systematically reviewed RCTs used within an anesthesiology related post-operative pain management meta-analysis. We included all of the 23 RCTs used, all of which were parallel design that addressed the use of femoral nerve block in improving outcomes after total knee arthroplasty. Data abstraction was done independently by two reviewers. The two main outcomes were: 1) 15 point overall quality of reporting score (OQRS) based on the Consolidated Standards for Reporting Trials (CONSORT) and 2) 3 point key methodological item score (KMIS) based on allocation concealment, blinding and intention-to-treat analysis.

Results

Twenty-three RCTs were included. The median OQRS was 9.0 (Interquartile Range = 3). A multivariable regression analysis did not show any significant association between OQRS or KMIS and our four predictor variables hypothesized to improve reporting. The direction and magnitude of our results when compared to similar studies suggest that the sample size and impact factor are associated with improved key methodological item reporting.

Conclusions

The quality of reporting of RCTs used within an anesthesia related meta-analysis is poor to moderate. The information gained from this study should be used by journals to register the urgency for RCTs to be clear and transparent in reporting to help make literature accessible and comparable.

Peer Review reports

Background

Transparent and well designed, conducted and reported randomised controlled trials (RCTs) are the gold standard in evaluating healthcare interventions [1]. The quality of reporting of RCTs in publications is important because it acts as a proxy for health care providers and researchers to appraise the quality of the methodology, conduct and analysis expected of an RCT [26]. The evaluation of the quality of reporting can be compromised when RCTs lack methodological rigour, thus generating biased results possibly resulting in exaggeration or shrinking of treatment effects [1, 714]. Poor conduct in the quality of reporting of RCTs may allow the introduction of treatments that are less effective than the studies themselves report [13]. Hence, it is important to know the quality of RCTs used in meta-analyses in order to identify the reliability of the evidence that may inform treatment and care in medicine, including treatment in the realm of anesthesiology and pain management [4, 7, 12, 15, 16].

Readers and reviewers of published RCTs need complete, clearly written, and transparent information on a study’s methodology and findings in order to assess its quality [1]. Attempts in assessing the quality of reporting recurrently fail because authors of RCTs neglect to provide comprehensive and clear descriptions of critical methodological information [1, 1719]. In response to inadequate reporting the Consolidated Standards of Reporting Trials (CONSORT) group created the CONSORT statement and published its latest revision of the statement in 2010 [1, 6]. The CONSORT 2010 Statement includes a 25 item checklist and a flow diagram. These are recommended items which should be incorporated into an RCT [1, 6]. The statement provides guidance for reporting all RCTs with a focus on individually randomised, two groups, parallel trials [1].

Following the CONSORT initial statement publication in 1996, there has been improvement in the quality of reporting of some RCTs [1, 10, 2022]. However, despite improvements noted, there continues to be observational studies reporting the inadequate quality of reporting of many RCTs [1, 17], including RCTs in specialized journals [1, 21, 2325]. There are a few other studies that have investigated the quality of reporting of RCTs specifically in anesthesia specialized journals. Many of them have evaluated the quality of reporting using items similar to items on the CONSORT 2010 checklist [7, 2628]. Some of the specialty journals in anesthesiology have also adopted or supported the CONSORT statement, yet the quality of reporting of parallel-group RCTs still remains poor [26]. Improvements made in the quality of reporting in the last decade may indicate a demand on the “quality, consistency and transparency” [26] of RCT reporting as well as endorsement of the CONSORT statement by medical journals which publish clinical trials [1, 26]. The quality of reporting of RCTs may be improved in specialty journals pertaining to anesthesiology by clearly illustrating methodology. Areas in methodology requiring attention are allocation concealment (randomization blinding), blinding of data collectors and “discussion of Type II error in negative trials” [26].

We have two primary aims for this study. The first aim of this study is to analyse the overall quality of reporting and reporting of 3 key methodological items of 23 RCTs used in a meta-analysis that investigated analgesia outcomes in RCTs comparing femoral nerve block (FNB) with epidural and patient-controlled analgesia (PCA) after total knee arthroplasty [29]. Secondly, we aim to determine factors associated with higher quality of reporting. We hypothesized that studies with larger sample sizes, that have funding reported, that are published in journals with a higher impact factor and that are in journals that have adopted, endorsed or supported the CONSORT statement will be associated with better overall quality of reporting and reporting of the 3 key methodological items. Our study is unique in that we evaluated the quality of reporting of RCTs used within a meta-analysis using 15 items plus 3 methodologically related items, all of which were directly from the CONSORT 2010 checklist. It has been suggested that the quality of studies included in meta-analyses should be regularly evaluated as studies show that inadequate quality of trials may distort the results of systematic reviews and meta-analyses [9]. We consider it important to do an update on the status of the quality of reporting of RCTs published within anesthesia specialized journals as this will contribute more evidence around the quality of reporting of RCTs used. Much of our study’s method was modelled upon a previous study assessing the quality of reporting of RCTs in general endocrinology literature [3].

Methods

Study selection

The 23 RCTs included in this study for quality of reporting and key methodological item evaluation came from a meta-analysis that compared FNB, with and without sciatic nerve block, with epidural and PCA after total knee arthroplasty [29]. This meta-analysis included 23 RCTs for its own analysis. It was of interest to this study to know the reporting quality and reporting of key methodology items included in this meta-analysis. All 23 RCTs assessed were parallel-design and addressed the issue of comparative treatment effectiveness with respect to morphine consumption, incidence of side effects and pain experiences, which was scored on a pain scale both at rest and with activity [29].

Data abstraction process

An electronic Excel standardized data collection form was used to extract data from each of the 23 RCTs. One pair of trained reviewers (VBD and SZ), both of whom were blinded to each other’s ratings, abstracted data independently. This form was pilot tested and subsequently modified for final revision. Any disagreement between the reviewers was rectified by consensus and chance-adjusted inter-rater agreements were calculated.

The rating of overall reporting quality

We defined the overall quality of reporting as the degree to which the rationale, design, conduct and analysis of each RCT, evaluated in this study, were reported. In order to do this, 15 relevant items from the 2010 CONSORT statement were adopted in this quality of reporting evaluation of RCTs (Table 1). These 15 items were chosen because of literature indicating that an absence in their reporting is associated with a higher level of bias [6]. The CONSORT discussion items were excluded because of the subjective nature in assessing them. Three key methodological qualities that are part of the CONSORT statement were also included in a separate assessment. An Overall Quality of Reporting Score (OQRS), with 15 items in total (score out of 15), was established. Each of the 15 items was scored 1 if it was reported and 0 if it was not clearly stated or definitely not stated. Each RCT assessed could have a possible score between 0 and 15 inclusively. Two additional items that are considered important to be present in RCT publications were also assessed (Table 1), however they were not included in the calculation of the OQRS. This included a participant flow chart, as indicated by the CONSORT statement [6], and an ethical item, as it can improve a clinical study’s review of its methodology and patient safety [30].

Table 1 Description of overall quality of reporting items

Rating of key methodology items

Appropriate concealment of allocation mechanisms, blinding, and numbers analysed (intention-to-treat principle) were assessed separately because they are highly important in avoiding bias and distortions of the effect estimates [4, 9, 26, 3135]. The methodological items are also underreported in studies with a high overall reporting quality score hence emphasizing the need to assess them separately [34, 35].

Appropriate concealment of allocation is a process that assigns patients in an RCT into alternative treatment groups in a way that will prevent the people who manage the allocation of patients as well as the patients themselves from foreknowledge of the allocation mechanism. This is done in order to avoid selection bias [8]. Obvious and easily manipulated variables such as medical record numbers or birthdays are not adequate for use in allocation concealment mechanisms. In order to avoid selection bias, allocation concealment mechanisms considered appropriate by the Cochrane Handbook are centralized randomization schemes, numbered, coded vehicles and sequentially numbered, sealed and opaque envelopes [5]. Hence, we considered allocation concealment to be appropriate in this study if centralized randomization, numbered, coded vehicles, or opaque, sealed, and sequentially numbered envelopes were reported [8].

Blinding refers to the process by which study participants, health care providers, outcome assessors, data assessors and manuscript writers are kept unaware of the intervention to which patients are randomized. Blinding helps to reduce the risk of knowing which intervention was received and this is critical, as knowledge of the assigned intervention can affect outcomes and assessments of outcomes [8, 36]. Physicians and textbooks vary in their understanding and definitions of single, double, and triple blinding thus making the sole use of these terms in a study ambiguous in describing the blinding method. Despite this, single, double and triple blinding are terms commonly and singularly used to report blinding status without the accompaniment of descriptors indicating who was indeed blinded [3638]. A meta-analysis indicated that double blinding is associated with potential bias reduction; double blinded studies reported a 15% lower treatment effect than non-blinded studies [4]. Sometimes there are cases where the blinding of patients and health care providers is not feasible. This is especially true in instances where there is a comparison of the effectiveness of a minimally invasive procedure with a procedure that is more invasive, as with surgery. In this case, patients or investigators cannot easily be blinded [38]. In these instances, where blinding is not feasible, we considered blinding to have been done appropriately if at least one specific group was unambiguously reported as blinded [3]. In RCTs where blinding did not have any barriers, at least two groups had to be unambiguously reported as blinded to fulfill this methodological item criteria [3, 8, 38].

We defined the numbers analyzed based on the intention-to-treat (ITT) principle. In this study we define ITT analysis to include all participants randomized into a trial and its analysis regardless if treatment or the intervention was administered, if patients satisfied the entry criteria, if patients withdrew from the study, or if there were deviations from the protocol [8, 3941]. Though the meaning of ITT has been interpreted differently by numerous RCTs, the Cochrane Handbook has compiled evidence indicating that the definition we have used prevents biased treatment effects and is a pragmatic way to understand the effects of a clinical intervention [41, 42].

Each of the three above mentioned methodological items was scored 1 point if the method was appropriately reported and performed and 0 points if it was inappropriate or ambiguous. A combined methodological score of these three items was calculated for each trial by adding the scores of each item with a possible score ranging from 0 to 3 inclusively. This score was termed Key Methodological Item Score (KMIS). An inter-rater agreement was calculated for each of the three included methodological items as done in a previous study [3].

Definition of predictor variables

Sample size was defined as the number of patients randomized in each trial. The journal impact factor refers to an index number assigned and calculated by Thomson Reuters to a particular journal catalogued in Thomson Reuter's Journal Citation Reports that indicates the frequency with which the "average article" in a specific journal has been cited in a particular year or period [43]. Funding reported was defined as funding mentioned and provided for the conduct of the study. Journal adoption of the CONSORT statement at the time of data abstraction was defined as the article’s journal endorsing, encouraging or enforcing the CONSORT statement for study submissions at the time the article was reviewed for this study.

Hypotheses

Journal impact factor, sample size and declared funding have been correlated with better reporting [3, 7, 25, 4448]. Journal adoption of the CONSORT statement has also been associated with improved reporting of RCTs [6, 7, 10, 11, 22]. We hypothesized that studies with larger sample sizes, that have funding reported, that are published in journals with a higher impact factor and that are in journals that have adopted, endorsed or supported the CONSORT statement will be associated with better quality of reporting and reporting of key methodological items.

Statistical analysis

The statistical analysis was modeled based on the analysis done in a previous study looking at the quality of reporting of general endocrinology literature [3]. The predictor variables, journal impact factor and sample size were also assessed in this study [3]. The new predictor variable included for our study is journal adoption, endorsement or support of the CONSORT statement at the time of data abstraction and funding reported.

Categorical data were reported as the number of counts. The number of trials that clearly stated the element and the associated 95% confidence interval (CI) were calculated. For the element that had “zero” count or “full” counts (i.e. when none of or all of the included trials reported that element respectively), the 95% CI was calculated by adopting the rule of three [49]. If none of the n studies showed the element that we were interested in, we could be 95% confident that the chance of this event occurring is at most 3 in n [49]. For the other elements, the 95% CI of the count was calculated by assuming that the number of trials that clearly stated the element followed a binomial distribution. The probability that a trial had clearly stated the element was set to be the observed probability in the sample. The Cohen’s Kappa (κ) statistics were used to calculate the chance-adjusted agreement between the two raters for the 15 quality of reporting items, the 2 additional items (ethical issues and flow chart) and the 3 key methodological items. Agreement was interpreted as poor if κ ≤ 0.2; fair if 0.21 ≤ κ ≤ 0.4; moderate if 0.41 ≤ κ ≤0.6; substantial if 0.61 ≤ κ ≤ 0.8; and good if κ > 0.8 [50].

A multivariable regression analysis with the OQRS as an outcome variable was conducted to determine if a higher sample size, a higher impact factor, the presence of funding reported and the journal adoption, endorsement or support of the CONSORT statement was independently associated with a better OQRS. These four variables were used as predictors. The sample size was transformed by the logarithm function with base 10 to improve interpretability. The OQRS (discrete, ranged from 0–15) was assumed to follow a Poisson distribution. Incidence rate ratios (IRR) were used to express the results of the analysis. Variables were considered to be statistically significant at alpha (α) = 0.05. The same approach was followed for regression analysis with the KMIS as the outcome variable. It is also important to note that we used Variance Inflation Factors (VIFs) to assess collinearity between predictors and none of the VIFs were greater than 10 showing no major collinearity among the variables. SPSS© 19.0.0.0 (IBM Corporation, 2010) was used to perform the statistical analysis.

Results

Rating of quality of reporting

The inter-rater agreement (κ) for the quality of reporting items varied from 0.646 to 1, indicating substantial to perfect agreement. The ratings of items for the OQRS are shown in Table 2. Some factors that constitute the overall quality of reporting score were consistently well reported. These were items 2, 3, 5, and 13 (Background, Objectives, Intervention and Baseline Data respectively) and they were properly reported in over 90% of the studies. The other items were poorly reported in the majority of the RCTs evaluated. Seven of the 15 items of the overall quality of reporting score were correctly reported in fewer than 50% of the studies. These items were Title and Abstract, Participants, Outcomes, Randomization: Sequence Generation, Randomization: Implementation, Recruitment, and Outcomes and Estimates. The mean OQRS was 8.43 (SD = 1.93), and the median was 9.0 (IQR = 3), with the minimum and maximum score of 4 and 11 respectively. Among the additional items considered, ethical issues were consistently well reported in 21 out of the 23 studies while a participant flow chart was provided in only 2 of the RCTs evaluated (Table 2).

Table 2 Overall Quality of Reporting: Rating using items from the CONSORT Statement and two additional items

Rating of key methodological items

The inter-rater agreement (κ) for the quality of key methodological items was 1 for all items, thus there was perfect agreement. The number of studies that reported each key methodological item is provided in Table 3. Among the 23 RCTs, 7 (30.4%) did not report any of the three key methodological items and none of the 23 studies evaluated reported all three methodological items. The reporting of the key methodological items among studies with an OQRS above the 75th percentile was also poor. The average key methodological score of articles with an OQRS above the 75th percentile was 1. Of the articles above the 75th percentile for the overall quality of reporting score only 3 articles describe an appropriate allocation concealment mechanism, 1 article describes blinding and 1 article describes intention-to-treat analysis. Consequently, those studies scoring a higher OQRS did not guarantee a good report of key methodological items.

Table 3 Reporting Quality of Key Methodology Items

An appropriate method of allocation concealment was reported in only 8 of the 23 RCTs (34.8%), with opaque, sealed and sequentially numbered envelopes being the only method reported by all of these 8 RCTs. Most reports, 15 of 23 (65.2%) did not give any information about this key methodological item.

Only 6 of the 23 RCTs (26.1%) were blinded according to the study definition. Among the 18 studies where blinding of patients was considered feasible, only 4 of those 18 studies specifically reported at least two blinded groups. Among the 5 studies where blinding of patients was considered not feasible, 2 reported blinding of another group. Table 3 shows the number of studies reporting key methodological items.

Factors associated with reporting quality

Table 4 displays the results of the multivariable analysis of factors associated with the OQRS. Though the regression model did not show a significant association (p < 0.05) for any of these four predictor variables (sample size, impact factor, journal adoption of the CONSORT statement at the time of data collection and funding reported) the model did show a trend for funding reported, where the presence of funding reported in a study showed an observed 26.2% more quality of reporting items present or higher than studies without funding reported, on average.

Table 4 Regression analysis for the Overall Reporting Quality a in 23 RCTs

The multivariable analysis of factors associated with better reporting of KMIS also did not show a significant association for any of these four predictor variables (Table 5). Nonetheless the model did show a trend for funding reported as was shown with the regression for the overall quality score, where the presence of funding reported in a study showed an observation of 76.6% more key methodological items present or higher than studies without funding reported, on average. In addition, the model showed a trend for impact factor, where a 1 unit increment in impact factor showed an observation of 102.2% more key methodological items present.

Table 5 Regression analysis for the Key Methodology Item Score a in 23 RCTs

Discussion

In this study we assessed the RCT reporting quality of 23 RCTs used within a FNB meta-analysis [29]. Most of these articles came from anesthesia specialty journals, except for 3 of the RCTs. All RCTs were related to post-operative management of analgesia after total knee arthroplasty. The overall quality of reporting of the RCTs assessed was poor to moderate, with only 4 of the 15 items being reported in over 90% of the articles. We have identified several areas where reporting of a particular item was insufficient in most of the studies assessed. These areas included: Item 1) Title and Abstract, Item 4) Participants, Item 6) Outcomes, Item 8) Randomization: Sequence Generation, Item 9) Randomization: Implementation, Item 11) Participants Flow, Item 12) Recruitment and Item 14) Outcomes and Estimation. Areas where items were reported moderately well were: Item 7) Sample size, Item 10) Statistical Methods and Item 15) Harms. Also very noteworthy was that the reporting of the key methodological items was poor with less than 50% of the articles reporting any of the 3 key methodological items (Table 3). Our results agree with many studies which have assessed the quality of reporting of RCTs published in medical and subspecialty journals, including studies which have focused on journals specialized in anesthesia [3, 17, 21, 2329, 4446]. All of these studies showed there is poor quality of reporting, with the 3 key methodological items assessed in this study (appropriate allocation concealment mechanism, blinding and numbers analysed by ITT) being highlighted as poorly reported [17, 21, 2327, 35, 44]. Even though there is evidence to suggest that the quality of reporting has significantly improved over time, especially with the onset, use and publication of the CONSORT statement, the poor quality of reporting overall suggests that the statistically significant improvements are not enough to be clinically important as higher quality of reporting is needed to reduce bias and properly support clinicians’ decisions about treatment management [13, 20, 22, 35]. Also, the overall quality of reporting does not necessarily correspond with the proper reporting of the 3 key methodological items assessed for in this study; 3 items which are deemed important in preventing bias [9, 35]. Allocation concealment is important in avoiding selection bias, proper blinding is important in avoiding performance and detection bias, and numbers analyzed (ITT principle) is important in avoiding attrition bias [9]. Any method of randomization, allocation concealment or blinding which allows the investigator to control the treatment group to which the study participant will be assigned should be avoided [27]. Recent studies suggest that deficiency and poor quality of reporting of these 3 key methodological items often causes bias in the estimate of treatment effects, though the extent of bias can be unpredictable [27, 31, 32]. The overall poor quality of reporting in anesthesia literature is important to highlight as it suggests that there is a need to standardize the reporting of RCTs in anesthesia publications where information is retrieved, disseminated, and used for clinical practice. Although the CONSORT group was created to help improve reporting, there seems to be a need to register the urgency for completeness, clarity and transparency of reporting [1] in order to make publications more accessible and easily comparable with one another. This is apparent with the 23 RCTs assessed for this study as well since all of these articles were published after the CONSORT statement was established in 1996. The introduction of the CONSORT among journals and enforcing its use by requiring authors to submit a CONSORT checklist has the potential to greatly improve the quality of reporting for its readers and researchers needing to use the important information that RCTs provide for future application and meta-analysis. This notion is supported by evidence shown in 3 recent observational studies demonstrating that the introduction of the CONSORT statement is associated with an improvement in the quality of reporting [20, 22, 51].

Limitations

There are many limitations to this study. One of the key limitations is the small sample size of RCTs used to evaluate the quality of reporting and key methodological items. The reason for this is that our intention was to evaluate the reporting quality of RCTs used in an actual meta-analysis. In hindsight, our study may not have had enough power to detect any significant association with our four predictor variables. In order to compensate for this we looked at trends among our findings with two other studies [3, 25] that looked at analogous predictor variables using comparable methods of statistical analysis, and their association with the overall quality of reporting or reporting of key methodological items in RCTs. We found that two of the predictor variables we looked at (Sample Size and Impact Factor) had shown a significant association with reporting quality in these two studies. We were not able to compare our other two predictor variables (Journal adoption of CONSORT statement at the time of data collection and funding reported) with these two comparator studies and with other studies that were reviewed because of one of three reasons: 1) the variable was not used as a predictor variable, 2) the findings were not reported in a numerical and statistical manner that could be used for trend comparison, or 3) it was not reported at all in publication. For the OQRS there were a few noted trends. Though the direction and the magnitude of the effect of sample size was not seen as similar to the two comparator studies, our 95% CI range for sample size encompassed the IRR of both comparator studies suggesting that there is a possibility that our results would have showed association had the sample size been larger. However, for the predictor variable of impact factor, which was measured in our study on a continuous scale, there were no trends observed in the one study we could use as a comparison. It should be noted that the comparator paper we could use for impact factor measured impact factor association in a categorical manner. Most of the impact factors of the RCT journals in our study fell into only one of those categories suggesting that our CI range may have encompassed their model had we been able to compare on a continuous scale. We also had 4 RCTs that came from journals with no impact factor evaluation for 2010. This decreased our RCT sample size from 23 to 19, further reducing our power to detect a difference.

For the KMIS there were also a few noted trends with one of the comparator studies. Both the sample size and impact factor of our results matched the direction and range found in the study by Lai et al. suggesting that our results would have reached significance had the sample size been larger. This trend is also more likely to be apparent with KMIS, even with the sample size we had, because of noted studies suggesting that key methodological items, such as the three used in this study, are associated with improved quality of reporting [3, 7, 25, 4446] and the importance the 3 key methodological items assessed (Blinding, Allocation Concealment and Numbers Analysed) have in reducing bias [6, 31, 32, 36, 52]. In the future it might be of interest to assess the quality of RCTs from more than one meta-analysis on a related topic or even a random sample of RCTs in recent years in primary journals. Another limitation of our study is the fact that we cannot verify the degree with which the quality of reporting reflects the true methodological quality of the RCTs we assessed. The lack of reporting of a particular item in an RCT does not mean that the study in truth had a poor methodology. It is possible that the protocols of many RCTs include important aspects of methodology however important methodological details were left out of the published report [53, 54]. In one study it was reported that the proportion of double blinded trials with a clear description of the blinding of participants increased from 19%, with reliance on the publication alone, to 67% when the protocol for the RCT was reviewed and supplemented for assessment with the publication [55]. Although the reporting on the key methodological item of blinding was found to be inadequate in both trial protocols and publications, the results show that there is still a possibility that the methodological and reporting quality of an RCT is greater than what is assessed in the published report [55, 56]. Soares et al. also showed that adequate concealment of allocation was achieved in all protocols but was only reported in 42% of the published papers [56]. Reviewing protocols may improve the quality of reporting assessment however the published report is an important proxy in determining the validity and clinically relevant effect estimate as it is the source most accessible to clinicians and researchers. The quality of the report itself is important for this purpose. An additional limitation was the fact there is no standard instrument to evaluate the quality of reporting of RCTs. The majority of scales have not been thoroughly developed or tested for in regards to validity and reliability as a gold standard (criterion validity) is needed to compare it against. A gold standard does not exist for assessing the quality of RCTs and so checklists demarcating important features of quality and scales based on a quality of reporting checklist are assessed based on face validity and content validity from a theoretical model founded on accepted criteria [57, 58]. Quality scores based on checklists are also unreliable and may introduce bias as scores were found to differ depending on the scoring system that was used [59, 60]. Our results may also not entirely represent the quality of reporting of RCTs used in meta-analyses related to the intervention of the meta-analysis we used [29]. Therefore the assessment of the quality of reporting of the RCTs used in other anesthesia related meta-analyses may help in compiling a sufficient pool of studies providing a sample size with enough power to detect a significant difference. Even with our limitations our results have good internal validity as we had two qualified reviewers and good inter-rater correlation despite that there was a lack of clarity with the reporting of evaluated items. In the future, one important item to add to the OQRS is trial registration. Trial registration is another recently enforced part of any published RCT article in order to ensure accessibility of all research. Studies have shown that the presence of registration with a trial registry is suboptimal [61, 62] and trial registration is associated with improved reporting in RCTs within the highest ranked journals [62]. It is also important to note that the year of publication definitely plays a role in the reporting quality of the RCTs chosen for our study. The CONSORT statement was established in 1996 and the speed of implementation of the guideline from the CONSORT group has been substandard as there continue to be observational studies concluding inadequate quality of reporting of many RCTs dated from 1996 till now [1, 17, 21, 2325]. Indeed, it has been shown that the total quality of reporting score is significantly associated with factors such as trial size, publication year, and the impact factor which the RCT is published within [44, 47, 48]. With that said the medical research community should also take notice that impact factor has been shown to be statistically significant as a predictor for better reporting quality on several occasions [25, 47, 48] and may be a feature to look for when wanting to find RCT publications that are of high reporting quality. It is possible that a large proportion of journals that have a higher impact factor have made initiatives to have authors submitting RCT manuscripts to their journal to adhere to the CONSORT statement. This may be worth investigating in the future to see if such a relationship exists.

Conclusions

In conclusion our study showed that the quality of reporting and reporting of key methodological items within anesthesia RCTs used for a post-operative pain management meta-analysis is poor to moderate. The knowledge gained from this study should be taken as another opportunity for journals to register the urgency for RCT publications to be complete, clear and transparent in reporting in order to make the literature accessible and comparable. Enforcing the use of the CONSORT statement by requiring authors to submit a CONSORT checklist can help greatly to improve the quality of reporting and get authors in the frame of reference needed to properly report studies or even structure study design.

Abbreviations

CI:

Confidence Interval

CONSORT:

Consolidated Standards for Reporting Trials

FNB:

Femoral Nerve Block

IRR:

Incidence Rate Ratio

IQR:

Inter-Quartile Range

KMIS:

Key Methodological Item Score

PCA:

Patient-Controlled Analgesia

RCT:

Randomized Controlled Trial

OQRS:

Overall Quality of Reporting

SPSS:

Statistical Package for the Social Sciences.

References

  1. Schulz KF, Altman DG, Moher D, CONSORT Group: CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMC Med. 2010, 8: 18-10.1186/1741-7015-8-18.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Rios LP, Ye C, Thabane L: Association between framing of the research question using the PICOT format and reporting quality of randomized controlled trials. BMC Med Res Methodol. 2010, 10: 11-10.1186/1471-2288-10-11.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Rios LP, Odueyungbo A, Moitri MO, Rahman MO, Thabane L: Quality of reporting of randomized controlled trials in general endocrinology literature. J Clin Endocrinol Metab. 2008, 93 (10): 3810-3816. 10.1210/jc.2008-0817.

    Article  CAS  PubMed  Google Scholar 

  4. Schulz KF, Chalmers I, Hayes RJ, Altman DG: Empirical evidence of bias Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995, 273 (5): 408-412. 10.1001/jama.1995.03520290060030.

    Article  CAS  PubMed  Google Scholar 

  5. Cochrane Handbook for Systematic Reviews of Intervention. , , , http://www.cochrane-handbook.org,

  6. Moher D, Hopewell S, Schulz KF, Montori V, Gotzsche PC, Devereaux PJ, Elbourne D, Egger M, Altman DG: CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ. 2010, 340: c869-10.1136/bmj.c869.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Can OS, Yilmaz AA, Hasdogan M, Alkaya F, Turhan SC, Can MF, Alanoglu Z: Has the quality of abstracts for randomised controlled trials improved since the release of Consolidated Standards of Reporting Trial guideline for abstract reporting? A survey of four high-profile anaesthesia journals. Eur J Anaesthesiol. 2011, 28 (7): 485-492. 10.1097/EJA.0b013e32833fb96f.

    Article  PubMed  Google Scholar 

  8. Higgins JPT, Altman DG, Sterne JAC: Chapter 8 Assessing risk of bias in included studies. Cochrane Handbook for Systematic Reviews of Interventions. Edited by: Higgins JPT, Green S. 2011, The Cochrane Collaboration, , 8.0-510

    Google Scholar 

  9. Juni P, Altman DG, Egger M: Systematic reviews in health care: Assessing the quality of controlled clinical trials. BMJ. 2001, 323 (7303): 42-46. 10.1136/bmj.323.7303.42.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  10. Moher D, Jones A, Lepage L, CONSORT Group (Consolitdated Standards for Reporting of Trials): Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. JAMA. 2001, 285 (15): 1992-1995. 10.1001/jama.285.15.1992.

    Article  CAS  PubMed  Google Scholar 

  11. Moher D, Schulz KF, Altman DG: The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet. 2001, 357 (9263): 1191-1194. 10.1016/S0140-6736(00)04337-3.

    Article  CAS  PubMed  Google Scholar 

  12. Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, Tugwell P, Klassen TP: Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?. Lancet. 1998, 352 (9128): 609-613. 10.1016/S0140-6736(98)01085-X.

    Article  CAS  PubMed  Google Scholar 

  13. Moher D, Altman DG, Schulz KF, Elbourne DR: Opportunities and challenges for improving the quality of reporting clinical research: CONSORT and beyond. CMAJ. 2004, 171 (4): 349-350. 10.1503/cmaj.1040031.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Higgins JPT, Green S: Cochrane Collaboration: Cochrane handbook for systematic reviews of interventions. 2008, Wiley-Blackwell, Chichester, West Sussex; Hoboken NJ

    Book  Google Scholar 

  15. Balk EM, Bonis PA, Moskowitz H, Schmid CH, Ioannidis JP, Wang C, Lau J: Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. JAMA. 2002, 287 (22): 2973-2982. 10.1001/jama.287.22.2973.

    Article  PubMed  Google Scholar 

  16. Kjaergard LL, Villumsen J, Gluud C: Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med. 2001, 135 (11): 982-989.

    Article  CAS  PubMed  Google Scholar 

  17. Chan AW, Altman DG: Epidemiology and reporting of randomised trials published in PubMed journals. Lancet. 2005, 365 (9465): 1159-1162. 10.1016/S0140-6736(05)71879-1.

    Article  PubMed  Google Scholar 

  18. Glasziou P, Meats E, Heneghan C, Shepperd S: What is missing from descriptions of treatment in trials and reviews?. BMJ. 2008, 336 (7659): 1472-1474. 10.1136/bmj.39590.732037.47.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan AW, Cronin E, Decullier E, Easterbrook PJ, Von Elm E, Gamble C, Ghersi D, Ioannidis JP, Simes J, Williamson PR: Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One. 2008, 3 (8): e3081-10.1371/journal.pone.0003081.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Hopewell S, Dutton S, Yu LM, Chan AW, Altman DG: The quality of reports of randomised trials in 2000 and 2006: comparative study of articles indexed in PubMed. BMJ. 2010, 340: c723-10.1136/bmj.c723.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Mills EJ, Wu P, Gagnier J, Devereaux PJ: The quality of randomized trial reporting in leading medical journals since the revised CONSORT statement. Contemp Clin Trials. 2005, 26 (4): 480-487. 10.1016/j.cct.2005.02.008.

    Article  PubMed  Google Scholar 

  22. Plint AC, Moher D, Morrison A, Schulz K, Altman DG, Hill C, Gaboury I: Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. Med J Aust. 2006, 185 (5): 263-267.

    PubMed  Google Scholar 

  23. Lai TY, Wong VW, Lam RF, Cheng AC, Lam DS, Leung GM: Quality of reporting of key methodological items of randomized controlled trials in clinical ophthalmic journals. Ophthalmic Epidemiol. 2007, 14 (6): 390-398. 10.1080/09286580701344399.

    Article  PubMed  Google Scholar 

  24. Mills E, Loke YK, Wu P, Montori VM, Perri D, Moher D, Guyatt G: Determining the reporting quality of RCTs in clinical pharmacology. Br J Clin Pharmacol. 2004, 58 (1): 61-65. 10.1111/j.1365-2125.2004.2092.x.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Lai R, Chu R, Fraumeni M, Thabane L: Quality of randomized controlled trials reporting in the primary treatment of brain tumors. J Clin Oncol. 2006, 24 (7): 1136-1144. 10.1200/JCO.2005.03.1179.

    Article  PubMed  Google Scholar 

  26. Greenfield ML, Mhyre JM, Mashour GA, Blum JM, Yen EC, Rosenberg AL: Improvement in the quality of randomized controlled trials among general anesthesiology journals 2000 to 2006: a 6-year follow-up. Anesth Analg. 2009, 108 (6): 1916-1921. 10.1213/ane.0b013e31819fe6d7.

    Article  PubMed  Google Scholar 

  27. Greenfield ML, Rosenberg AL, O'Reilly M, Shanks AM, Sliwinski MJ, Nauss MD: The quality of randomized controlled trials in major anesthesiology journals. Anesth Analg. 2005, 100 (6): 1759-1764. 10.1213/01.ANE.0000150612.71007.A3.

    Article  PubMed  Google Scholar 

  28. Pua HL, Lerman J, Crawford MW, Wright JG: An evaluation of the quality of clinical trials in anesthesia. Anesthesiology. 2001, 95 (5): 1068-1073. 10.1097/00000542-200111000-00007.

    Article  CAS  PubMed  Google Scholar 

  29. Paul JE, Arya A, Hurlburt L, Cheng J, Thabane L, Tidy A, Murthy Y: Femoral nerve block improves analgesia outcomes after total knee arthroplasty: a meta-analysis of randomized controlled trials. Anesthesiology. 2010, 113 (5): 1144-1162. 10.1097/ALN.0b013e3181f4b18.

    Article  PubMed  Google Scholar 

  30. Chung KC, Kotsis SV: The ethics of clinical research. J Hand Surg Am. 2011, 36 (2): 308-315. 10.1016/j.jhsa.2010.11.012.

    Article  PubMed  Google Scholar 

  31. Nuesch E, Reichenbach S, Trelle S, Rutjes AW, Liewald K, Sterchi R, Altman DG, Juni P: The importance of allocation concealment and patient blinding in osteoarthritis trials: a meta-epidemiologic study. Arthritis Rheum. 2009, 61 (12): 1633-1641. 10.1002/art.24894.

    Article  PubMed  Google Scholar 

  32. Nuesch E, Trelle S, Reichenbach S, Rutjes AW, Burgi E, Scherer M, Altman DG, Juni P: The effects of excluding patients from the analysis in randomised controlled trials: meta-epidemiological study. BMJ. 2009, 339: b3244-10.1136/bmj.b3244.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Wood L, Egger M, Gluud LL, Schulz KF, Juni P, Altman DG, Gluud C, Martin RM, Wood AJ, Sterne JA: Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ. 2008, 336 (7644): 601-605. 10.1136/bmj.39465.451748.AD.

    Article  PubMed  PubMed Central  Google Scholar 

  34. Schulz KF, Grimes DA, Altman DG, Hayes RJ: Blinding and exclusions after allocation in randomised controlled trials: survey of published parallel group trials in obstetrics and gynaecology. BMJ. 1996, 312 (7033): 742-744. 10.1136/bmj.312.7033.742.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  35. Huwiler-Muntener K, Juni P, Junker C, Egger M: Quality of reporting of randomized trials as a measure of methodologic quality. JAMA. 2002, 287 (21): 2801-2804. 10.1001/jama.287.21.2801.

    Article  PubMed  Google Scholar 

  36. Montori VM, Bhandari M, Devereaux PJ, Manns BJ, Ghali WA, Guyatt GH: In the dark: the reporting of blinding status in randomized controlled trials. J Clin Epidemiol. 2002, 55 (8): 787-790. 10.1016/S0895-4356(02)00446-8.

    Article  PubMed  Google Scholar 

  37. Haahr MT, Hrobjartsson A: Who is blinded in randomized clinical trials? A study of 200 trials and a survey of authors. Clin Trials. 2006, 3 (4): 360-365.

    PubMed  Google Scholar 

  38. Miller LE, Stewart ME: The blind leading the blind: use and misuse of blinding in randomized controlled trials. Contemp Clin Trials. 2011, 32 (2): 240-243. 10.1016/j.cct.2010.11.004.

    Article  PubMed  Google Scholar 

  39. Newell DJ: Intention-to-treat analysis: implications for quantitative and qualitative research. Int J Epidemiol. 1992, 21 (5): 837-841. 10.1093/ije/21.5.837.

    Article  CAS  PubMed  Google Scholar 

  40. Lewis JA, Machin D: Intention to treat–who should use ITT?. Br J Cancer. 1993, 68 (4): 647-650. 10.1038/bjc.1993.402.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  41. Higgins JPT, Deeks JJ, Altman DG: Chapter 16: Special Topics in statistics. Cochrane Handbook for Systematic Reviews and Interventions. Edited by: Higgins JPT, Green S. 2011, The Cochrane Collaboration,, 16.0-16.2. http://www.cochrane-handbook.org, 2011,

    Google Scholar 

  42. Hollis S, Campbell F: What is meant by intention to treat analysis? Survey of published randomised controlled trials. BMJ. 1999, 319 (7211): 670-674. 10.1136/bmj.319.7211.670.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  43. The Thomson Reuters Impact Factor. , , , http://thomsonreuters.com/products_services/science/free/essays/impact_factor,

  44. Farrokhyar F, Chu R, Whitlock R, Thabane L: A systematic review of the quality of publications reporting coronary artery bypass grafting trials. Can J Surg. 2007, 50 (4): 266-277.

    PubMed  PubMed Central  Google Scholar 

  45. Thomas O, Thabane L, Douketis J, Chu R, Westfall AO, Allison DB: Industry funding and the reporting quality of large long-term weight loss trials. Int J Obes (Lond). 2008, 32 (10): 1531-1536. 10.1038/ijo.2008.137.

    Article  CAS  Google Scholar 

  46. Balasubramanian SP, Wiener M, Alshameeri Z, Tiruvoipati R, Elbourne D, Reed MW: Standards of reporting of randomized controlled trials in general surgery: can we do better?. Ann Surg. 2006, 244 (5): 663-667. 10.1097/01.sla.0000217640.11224.05.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Toulmonde M, Bellera C, Mathoulin-Pelissier S, Debled M, Bui B, Italiano A: Quality of Randomized Controlled Trials Reporting in the Treatment of Sarcomas. J Clin Oncol. 2011, 29 (9): 1204-1209. 10.1200/JCO.2010.30.9369.

    Article  PubMed  Google Scholar 

  48. Ivers NM, Taljaard M, Dixon S, Bennett C, McRae A, Taleban J, Skea Z, Brehaut JC, Boruch RF, Eccles MP, Grimshaw JM, Weijer C, Zwarenstein M, Donner A: Impact of CONSORT extension for cluster randomised trials on quality of reporting and study methodology: review of random sample of 300 trials, 2000–8. BMJ. 2011, 343: d5886-10.1136/bmj.d5886.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  49. Eypasch E, Lefering R, Kum CK, Troidl H: Probability of adverse events that have not yet occurred: a statistical reminder. BMJ. 1995, 311 (7005): 619-620. 10.1136/bmj.311.7005.619.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  50. Landis JR, Koch GG: The measurement of observer agreement for categorical data. Biometrics. 1977, 33 (1): 159-174. 10.2307/2529310.

    Article  CAS  PubMed  Google Scholar 

  51. Egger M, Juni P, Bartlett C, CONSORT Group (Consolidated Standards of Reporting of Trials): Value of flow diagrams in reports of randomized controlled trials. JAMA. 2001, 285 (15): 1996-1999. 10.1001/jama.285.15.1996.

    Article  CAS  PubMed  Google Scholar 

  52. Montori VM, Guyatt GH: Intention-to-treat principle. CMAJ. 2001, 165 (10): 1339-1341.

    CAS  PubMed  PubMed Central  Google Scholar 

  53. Devereaux PJ, Choi PT, El-Dika S, Bhandari M, Montori VM, Schunemann HJ, Garg AX, Busse JW, Heels-Ansdell D, Ghali WA, Manns BJ, Guyatt GH: An observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure to report these methods. J Clin Epidemiol. 2004, 57 (12): 1232-1236. 10.1016/j.jclinepi.2004.03.017.

    Article  CAS  PubMed  Google Scholar 

  54. Pildal J, Chan AW, Hrobjartsson A, Forfang E, Altman DG, Gotzsche PC: Comparison of descriptions of allocation concealment in trial protocols and the published reports: cohort study. BMJ. 2005, 330 (7499): 1049-10.1136/bmj.38414.422650.8F.

    Article  PubMed  PubMed Central  Google Scholar 

  55. Hrobjartsson A, Pildal J, Chan AW, Haahr MT, Altman DG, Gotzsche PC: Reporting on blinding in trial protocols and corresponding publications was often inadequate but rarely contradictory. J Clin Epidemiol. 2009, 62 (9): 967-973. 10.1016/j.jclinepi.2009.04.003.

    Article  PubMed  Google Scholar 

  56. Soares HP, Daniels S, Kumar A, Clarke M, Scott C, Swann S, Djulbegovic B: Radiation Therapy Oncology Group: Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. BMJ. 2004, 328 (7430): 22-24. 10.1136/bmj.328.7430.22.

    Article  PubMed  PubMed Central  Google Scholar 

  57. Verhagen AP, de Vet HC, de Bie RA, Boers M, van den Brandt PA: The art of quality assessment of RCTs included in systematic reviews. J Clin Epidemiol. 2001, 54 (7): 651-654. 10.1016/S0895-4356(00)00360-7.

    Article  CAS  PubMed  Google Scholar 

  58. Olivo SA, Macedo LG, Gadotti IC, Fuentes J, Stanton T, Magee DJ: Scales to assess the quality of randomized controlled trials: a systematic review. Phys Ther. 2008, 88 (2): 156-175. 10.2522/ptj.20070147.

    Article  PubMed  Google Scholar 

  59. Juni P, Witschi A, Bloch R, Egger M: The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999, 282 (11): 1054-1060. 10.1001/jama.282.11.1054.

    Article  CAS  PubMed  Google Scholar 

  60. Herbison P, Hay-Smith J, Gillespie WJ: Adjustment of meta-analyses on the basis of quality scores should be abandoned. J Clin Epidemiol. 2006, 59 (12): 1249-1256.

    Article  PubMed  Google Scholar 

  61. Agha R, Cooper D, Muir G: The reporting quality of randomised controlled trials in surgery: A systematic review. Int J Surg. 2007, 5 (6): 413-422. 10.1016/j.ijsu.2007.06.002.

    Article  PubMed  Google Scholar 

  62. Reveiz L, Cortés-Jofré M, Asenjo Lobos C, Nicita G, Ciapponi A, Garcìa-Dieguez M, Tellez D, Delgado M, Solà I, Ospina E: Influence of trial registration on reporting quality of randomized trials: Study from highest ranked journals. J Clin Epidemiol. 2010, 63 (11): 1216-1222. 10.1016/j.jclinepi.2010.01.013.

    Article  CAS  PubMed  Google Scholar 

Pre-publication history

Download references

Acknowledgements

LT is the clinical trials mentor for the Canadian Institutes of Health Research. The study was funded in part by the CANNeCTIN programme and by Drug Safety and Effectiveness Cross-Disciplinary Training (DSECT) Program.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Lehana Thabane.

Additional information

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

VBD was involved with the conception of the study, participated in the study design, the acquisition of data, helped to perform the statistical analysis, made substantial contributions to the interpretation of the data, and drafted the manuscript. SZ was substantially involved with data acquisition and critical revisions to the paper. CY participated in the statistical design of the study, performed and made substantial contribution to the statistical analysis and interpretation of data and helped critically revise the manuscript. LT was involved with the conception of the study and made substantial contributions to the statistical analysis, interpretation of data and critically revised the manuscript. JP, LH, AA and YM were involved with contributing data and helped draft and edit the final manuscript. All authors read and approved the final manuscript.

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Authors’ original file for figure 2

Rights and permissions

Open Access This article is published under license to BioMed Central Ltd. This is an Open Access article is distributed under the terms of the Creative Commons Attribution License ( https://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article

Borg Debono, V., Zhang, S., Ye, C. et al. The quality of reporting of RCTs used within a postoperative pain management meta-analysis, using the CONSORT statement. BMC Anesthesiol 12, 13 (2012). https://doi.org/10.1186/1471-2253-12-13

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/1471-2253-12-13

Keywords