Email updates

Keep up to date with the latest news and content from BMC Medical Research Methodology and BioMed Central.

Open Access Research article

Newcastle-Ottawa Scale: comparing reviewers’ to authors’ assessments

Carson Ka-Lok Lo1, Dominik Mertz234 and Mark Loeb345*

Author Affiliations

1 Department of Medicine, University of Toronto, Toronto Ontario, Canada

2 Department of Medicine, McMaster University, Hamilton Ontario, Canada

3 Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton Ontario, Canada

4 Michael G. DeGroote Institute for Infectious Diseases Research, McMaster University, Hamilton Ontario, Canada

5 Department of Pathology and Molecular Medicine, McMaster University, MDCL 3203, 1200 Main St W, Hamilton, Ontario L8N 3Z5, Canada

For all author emails, please log on.

BMC Medical Research Methodology 2014, 14:45  doi:10.1186/1471-2288-14-45

Published: 1 April 2014



Lack of appropriate reporting of methodological details has previously been shown to distort risk of bias assessments in randomized controlled trials. The same might be true for observational studies. The goal of this study was to compare the Newcastle-Ottawa Scale (NOS) assessment for risk of bias between reviewers and authors of cohort studies included in a published systematic review on risk factors for severe outcomes in patients infected with influenza.


Cohort studies included in the systematic review and published between 2008–2011 were included. The corresponding or first authors completed a survey covering all NOS items. Results were compared with the NOS assessment applied by reviewers of the systematic review. Inter-rater reliability was calculated using kappa (K) statistics.


Authors of 65/182 (36%) studies completed the survey. The overall NOS score was significantly higher (p < 0.001) in the reviewers’ assessment (median = 6; interquartile range [IQR] 6–6) compared with those by authors (median = 5, IQR 4–6). Inter-rater reliability by item ranged from slight (K = 0.15, 95% confidence interval [CI] = −0.19, 0.48) to poor (K = −0.06, 95% CI = −0.22, 0.10). Reliability for the overall score was poor (K = −0.004, 95% CI = −0.11, 0.11).


Differences in assessment and low agreement between reviewers and authors suggest the need to contact authors for information not published in studies when applying the NOS in systematic reviews.

Newcastle Ottawa Scale; Inter-rater; Reliability; Validity; Risk of bias; Observational studies