BMC Medical Research Methodology - Most accessed articles

The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews

Penny Whiting — 2003-11-10T00:00:00Z

Background: In the era of evidence based medicine, with systematic reviews as its cornerstone, adequate quality assessment tools should be available. There is currently a lack of a systematically developed and evaluated tool for the assessment of diagnostic accuracy studies. The aim of this project was to combine empirical evidence and expert opinion in a formal consensus method to develop a tool to be used in systematic reviews to assess the quality of primary studies of diagnostic accuracy. Methods: We conducted a Delphi procedure to develop the quality assessment tool by refining an initial list of items. Members of the Delphi panel were experts in the area of diagnostic research. The results of three previously conducted reviews of the diagnostic literature were used to generate a list of potential items for inclusion in the tool and to provide an evidence base upon which to develop the tool. Results: A total of nine experts in the field of diagnostics took part in the Delphi procedure. The Delphi procedure consisted of four rounds, after which agreement was reached on the items to be included in the tool which we have called QUADAS. The initial list of 28 items was reduced to fourteen items in the final tool. Items included covered patient spectrum, reference standard, disease progression bias, verification bias, review bias, clinical review bias, incorporation bias, test execution, study withdrawals, and indeterminate results. The QUADAS tool is presented together with guidelines for scoring each of the items included in the tool. Conclusions: This project has produced an evidence based quality assessment tool to be used in systematic reviews of diagnostic accuracy studies. Further work to determine the usability and validity of the tool continues.

The CONSORT statement: revised recommendations for improving the quality of reports of parallel group randomized trials

David Moher — 2001-04-20T00:00:00Z

To comprehend the results of a randomized controlled trial (RCT), readers must understand its design, conduct, analysis and interpretation. That goal can only be achieved through complete transparency from authors. Despite several decades of educational efforts, the reporting of RCTs needs improvement. Investigators and editors developed the original CONSORT (Consolidated Standards of Reporting Trials) statement to help authors improve reporting by using a checklist and flow diagram. The revised CONSORT statement presented in this paper incorporates new evidence and addresses some criticisms of the original statement.The checklist items pertain to the content of the Title, Abstract, Introduction, Methods, Results and Discussion. The revised checklist includes 22-items selected because empirical evidence indicates that not reporting the information is associated with biasedestimates of treatment effect or the information is essential to judge the reliability or relevance of the findings. We intended the flow diagram to depict the passage of participants through an RCT. The revised flow diagram depicts information from four stages of a trial (enrolment, intervention allocation, follow-up, and analysis). The diagram explicitly includes the number of participants, for each intervention group, included in the primary data analysis. Inclusion of these numbers allows the reader to judge whether the authors have performed an intention-to-treat analysis.In sum, the CONSORT statement is intended to improve the reporting of an RCT, enabling readers to understand a trial's conduct and to assess the validity of its results.

Comparing recruitment strategies in a study of acupuncture for chronic back pain

Karen Sherman — 2009-10-27T00:00:00Z

Background: Meeting recruitment goals is challenging for many clinical trials conducted in primary care populations. Little is known about how the use of different recruitment strategies affects the types of individuals choosing to participate or the conclusions of the study. Methods: A secondary analysis was performed using data from participants recruited to a clinical trial evaluating acupuncture for chronic back pain among primary care patients in a large integrated health care organization. We used two recruitment methods: mailed letters of invitation and an advertisement in the health plan's magazine. For these two recruitment methods, we compared recruitment success (% randomized, treatment completers, drop outs and losses to follow-up), participant characteristics, and primary clinical outcomes. A linear regression model was used to test for interaction between treatment group and recruitment method. Results: Participants recruited via mailed letters closely resembled those responding to the advertisement in terms of demographic characteristics, most aspects of their back pain history and current episode and beliefs and expectations about acupuncture. No interaction between method of recruitment and treatment group was seen, suggesting that study outcomes were not affected by recruitment strategy. Conclusion: In this trial, the two recruitment strategies yielded similar estimates of treatment effectiveness. However, because this finding may not apply to other recruitment strategies or trial circumstances, trials employing multiple recruitment strategies should evaluate the effect of recruitment strategy on outcome.Trial registrationClinical Trials.gov NCT00065585.

Parametric versus non-parametric statistics in the analysis of randomized trials with non-normally distributed data

Andrew Vickers — 2005-11-03T00:00:00Z

Background: It has generally been argued that parametric statistics should not be applied to data with non-normal distributions. Empirical research has demonstrated that Mann-Whitney generally has greater power than the t-test unless data are sampled from the normal. In the case of randomized trials, we are typically interested in how an endpoint, such as blood pressure or pain, changes following treatment. Such trials should be analyzed using ANCOVA, rather than t-test. The objectives of this study were: a) to compare the relative power of Mann-Whitney and ANCOVA; b) to determine whether ANCOVA provides an unbiased estimate for the difference between groups; c) to investigate the distribution of change scores between repeat assessments of a non-normally distributed variable. Methods: Polynomials were developed to simulate five archetypal non-normal distributions for baseline and post-treatment scores in a randomized trial. Simulation studies compared the power of Mann-Whitney and ANCOVA for analyzing each distribution, varying sample size, correlation and type of treatment effect (ratio or shift). Results: Change between skewed baseline and post-treatment data tended towards a normal distribution. ANCOVA was generally superior to Mann-Whitney in most situations, especially where log-transformed data were entered into the model. The estimate of the treatment effect from ANCOVA was not importantly biased. Conclusion: ANCOVA is the preferred method of analyzing randomized trials with baseline and post-treatment measures. In certain extreme cases, ANCOVA is less powerful than Mann-Whitney. Notably, in these cases, the estimate of treatment effect provided by ANCOVA is of questionable interpretability.

Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews

Beverley Shea — 2007-02-15T00:00:00Z

Background: Our objective was to develop an instrument to assess the methodological quality of systematic reviews, building upon previous tools, empirical evidence and expert consensus. Methods: A 37-item assessment tool was formed by combining 1) the enhanced Overview Quality Assessment Questionnaire (OQAQ), 2) a checklist created by Sacks, and 3) three additional items recently judged to be of methodological importance. This tool was applied to 99 paper-based and 52 electronic systematic reviews. Exploratory factor analysis was used to identify underlying components. The results were considered by methodological experts using a nominal group technique aimed at item reduction and design of an assessment tool with face and content validity. Results: The factor analysis identified 11 components. From each component, one item was selected by the nominal group. The resulting instrument was judged to have face and content validity. Conclusion: A measurement tool for the 'assessment of multiple systematic reviews' (AMSTAR) was developed. The tool consists of 11 items and has good face and content validity for measuring the methodological quality of systematic reviews. Additional studies are needed with a focus on the reproducibility and construct validity of AMSTAR, before strong recommendations can be made on its use.

A multidisciplinary systematic literature review on frailty: Overview of the methodology used by the Canadian Initiative on Frailty and Aging

Sathya Karunananthan — 2009-10-12T00:00:00Z

Background: Over the past two decades, there has been a substantial growth in the body of literature on frailty in older persons. However, there is no consensus on its definition or the criteria used to identify frailty. In response to this lack of consensus, the Canadian Initiative on Frailty and Aging carried out a set of systematic reviews of the literature in ten areas of frailty research: biological basis; social basis; prevalence; risk factors; impact; identification; prevention and management; environment and technology; health services; health and social policy. This paper describes the methodology that was developed for the systematic reviews. Methods: A Central Coordination Group (CCG) was responsible for developing the methodology. This involved the development of search strategies and keywords, article selection processes, quality assessment tools, and guidelines for the synthesis of results. Each review was conducted by two experts in the content area, with the assistance of methodologists and statisticians from the CCG. Results: Conducting a series of systematic literature reviews involving a range of disciplines on a concept that does not have a universally accepted definition posed several conceptual and methodological challenges. The most important conceptual challenge was determining what would qualify as literature on frailty. The methodological challenges arose from our goal of structuring a consistent methodology for reviewing literature from diverse fields of research. At the outset, certain methodological guidelines were deemed essential to ensure the validity of all the reviews. Nevertheless, it was equally important to permit flexibility in the application of the proposed methodology to capture the essence of frailty research within the given fields. Conclusion: The results of these reviews allowed us to establish the status of current knowledge on frailty and promote collaboration between disciplines. Conducting systematic literature reviews in health science that involve multiple disciplines is a mechanism to facilitate interdisciplinary collaboration and a more integrated understanding of health. This initiative highlighted the need for further methodological development in the performance of multidisciplinary systematic reviews.

Evaluating the informatics for integrating biology and the bedside system for clinical research.

Vikrant Deshmukh — 2009-10-28T00:00:00Z

Background: Selecting patient cohorts is a critical, iterative, and often time-consuming aspect of studies involving human subjects; informatics tools for helping streamline the process have been identified as important infrastructure components for enabling clinical and translational research. We describe the evaluation of a free and open source cohort selection tool from the Informatics for Integrating Biology and the Bedside (i2b2) group: the i2b2 hive. Methods: Our evaluation included the usability and functionality of the i2b2 hive using several real world examples of research data requests received electronically at the University of Utah Health Sciences Center between 2006 - 2008. The hive server component and the visual query tool application were evaluated for their suitability as a cohort selection tool on the basis of the types of data elements requested, as well as the effort required to fulfill each research data request using the i2b2 hive alone. Results: We found the i2b2 hive to be suitable for obtaining estimates of cohort sizes and generating research cohorts based on simple inclusion/exclusion criteria, which consisted of about 44% of the clinical research data requests sampled at our institution. Data requests that relied on post-coordinated clinical concepts, aggregate values of clinical findings, or temporal conditions in their inclusion/exclusion criteria could not be fulfilled using the i2b2 hive alone, and required one or more intermediate data steps in the form of pre- or post-processing, modifications to the hive metadata, etc. Conclusion: The i2b2 hive was found to be a useful cohort-selection tool for fulfilling common types of requests for research data, and especially in the estimation of initial cohort sizes. For another institution that might want to use the i2b2 hive for clinical research, we recommend that the institution would need to have structured, coded clinical data and metadata available that can be transformed to fit the logical data models of the i2b2 hive, strategies for extracting relevant clinical data from source systems, and the ability to perform substantial pre- and post-processing of these data.

Clinical trial participant characteristics and saliva and DNA metrics

Denise Nishita — 2009-10-29T00:00:00Z

Background: Clinical trial and epidemiological studies need high quality biospecimens from a representative sample of participants to investigate genetic influences on treatment response and disease. Obtaining blood biospecimens presents logistical and financial challenges. As a result, saliva biospecimen collection is becoming more frequent because of the ease of collection and lower cost. This article describes an assessment of saliva biospecimen samples collected through the mail, trial participant demographic and behavioral characteristics, and their association with saliva and DNA quantity and quality. Methods: Saliva biospecimens were collected using the Oragene® DNA Self-Collection Kits from participants in a National Cancer Institute funded smoking cessation trial. Saliva biospecimens from 565 individuals were visually inspected for clarity prior to and after DNA extraction. DNA samples were then quantified by UV absorbance, PicoGreen®, and qPCR. Genotyping was performed on 11 SNPs using TaqMan® SNP assays and two VNTR assays. Univariate, correlation, and analysis of variance analyses were conducted to observe the relationship between saliva sample and participant characteristics. Results: The biospecimen kit return rate was 58.5% among those invited to participate (n = 967) and 47.1% among all possible COMPASS participants (n = 1202). Significant gender differences were observed with males providing larger saliva volume (4.7 vs. 4.5 ml, p = 0.019), samples that were more likely to be judged as cloudy (39.5% vs. 24.9%, p < 0.001), and samples with greater DNA yield as measured by UV (190.0 vs. 138.5, p = 0.002), but reduced % human DNA content (73.2 vs. 77.6 p = 0.005) than females. Other participant characteristics (age, self-identified ethnicity, baseline cigarettes per day) were associated with saliva clarity. Saliva volume and saliva and DNA clarity were positively correlated with total DNA yield by all three quantification measurements (all r > 0.21, P < 0.001), but negatively correlated with % human DNA content (saliva volume r = -0.148 and all P < 0.010). Genotyping completion rate was not influenced by saliva or DNA clarity. Conclusion: Findings from this study show that demographic and behavioral characteristics of smoking cessation trial participants have significant associations with saliva and DNA metrics, but not with the performance of TaqMan® SNP or VNTR genotyping assays.Trial registrationCOMPASS; registered as NCT00301145 at clinicaltrials.gov.

Alternatives for logistic regression in cross-sectional studies: an empirical comparison of models that directly estimate the prevalence ratio

Aluisio Barros — 2003-10-20T00:00:00Z

Background: Cross-sectional studies with binary outcomes analyzed by logistic regression are frequent in the epidemiological literature. However, the odds ratio can importantly overestimate the prevalence ratio, the measure of choice in these studies. Also, controlling for confounding is not equivalent for the two measures. In this paper we explore alternatives for modeling data of such studies with techniques that directly estimate the prevalence ratio. Methods: We compared Cox regression with constant time at risk, Poisson regression and log-binomial regression against the standard Mantel-Haenszel estimators. Models with robust variance estimators in Cox and Poisson regressions and variance corrected by the scale parameter in Poisson regression were also evaluated. Results: Three outcomes, from a cross-sectional study carried out in Pelotas, Brazil, with different levels of prevalence were explored: weight-for-age deficit (4%), asthma (31%) and mother in a paid job (52%). Unadjusted Cox/Poisson regression and Poisson regression with scale parameter adjusted by deviance performed worst in terms of interval estimates. Poisson regression with scale parameter adjusted by χ2 showed variable performance depending on the outcome prevalence. Cox/Poisson regression with robust variance, and log-binomial regression performed equally well when the model was correctly specified. Conclusions: Cox or Poisson regression with robust variance and log-binomial regression provide correct estimates and are a better alternative for the analysis of cross-sectional studies with binary outcomes than logistic regression, since the prevalence ratio is more interpretable and easier to communicate to non-specialists than the odds ratio. However, precautions are needed to avoid estimation problems in specific situations.

Meta-analysis: Neither quick nor easy

Nancy Berman — 2002-08-09T00:00:00Z

Background: Meta-analysis is often considered to be a simple way to summarize the existing literature. In this paper we describe how a meta-analysis resembles a conventional study, requiring a written protocol with design elements that parallel those of a record review. Methods: The paper provides a structure for creating a meta-analysis protocol. Some guidelines for measurement of the quality of papers are given. A brief overview of statistical considerations is included. Four papers are reviewed as examples. The examples generally followed the guidelines we specify in reporting the studies and results, but in some of the papers there was insufficient information on the meta-analysis process. Conclusions: Meta-analysis can be a very useful method to summarize data across many studies, but it requires careful thought, planning and implementation.