Email updates

Keep up to date with the latest news and content from BMC Medical Research Methodology and BioMed Central.

Open Access Research article

Intracluster correlation coefficients for the Brazilian Multicenter Study on Preterm Birth (EMIP): methodological and practical implications

Giuliane J Lajos1*, Samira M Haddad1, Ricardo P Tedesco1, Renato Passini Jr1, Tabata Z Dias1, Marcelo L Nomura1, Patrícia M Rheder1, Maria H Sousa2, Jose G Cecatti12 and for the Brazilian Multicenter Study on Preterm Birth study group

Author Affiliations

1 Department of Obstetrics and Gynecology, School of Medical Sciences, University of Campinas, Campinas, Brazil

2 Center for Studies in Reproductive Health of Campinas (Cemicamp), Campinas, Brazil

For all author emails, please log on.

BMC Medical Research Methodology 2014, 14:54  doi:10.1186/1471-2288-14-54

The electronic version of this article is the complete one and can be found online at: http://www.biomedcentral.com/1471-2288/14/54


Received:27 November 2013
Accepted:14 April 2014
Published:22 April 2014

© 2014 Lajos et al.; licensee BioMed Central Ltd.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Abstract

Background

Cluster-based studies in health research are increasing. An important characteristic of such studies is the presence of intracluster correlation, typically quantified by the intracluster correlation coefficient (ICC), that indicate the proportion of data variability that is explained by the way of clustering. The purpose of this manuscript was to evaluate ICC of variables studied in the Brazilian Multicenter Study on Preterm Birth.

Methods

This was a multicenter cross-sectional study on preterm births involving 20 referral hospitals in different regions of Brazil plus a nested case–control study to assess associated factors with spontaneous preterm births. Estimated prevalence rates or means, ICC with 95% confidence intervals, design effects and mean cluster sizes were presented for more than 250 maternal and newborn variables.

Results

Overall, 5296 cases were included in the study (4,150 preterm births and 1,146 term births). ICC ranged from <0.001 to 0.965, with a median of 0.028. For descriptive characteristics (socio-demographic, obstetric history and perinatal outcomes) the median ICC was 0.014, for newborn outcomes the median ICC was 0.041 and for process variables (clinical management and delivery), it was 0.102. ICC was <0.1 in 78.4% of the variables and <0.3 for approximately 95% of them. Most of ICC >0.3 was found in some clinical management aspects well defined in literature such as use of corticosteroids, indicating there was homogeneity in clusters for these variables.

Conclusions

Clusters selected for Brazilian Multicenter Study on Preterm Birth had mainly heterogeneous findings and these results can help researchers estimate the required sample size for future studies on maternal and perinatal health.

Keywords:
Intracluster correlation coefficient; Preterm birth; Spontaneous preterm labor; Premature rupture of membranes; Indicated preterm delivery; Neonatal morbidity

Background

Cluster-based studies involving aggregated units such as hospitals, health centers, schools or medical practices are increasingly being used in healthcare evaluation, especially in cluster randomized trials, which are perhaps the most high impact form of public health research/evaluation study design that can benefit from good extent estimates of ICC. In such situations, population groups (specific geographical areas), healthcare units (hospitals) or healthcare sectors are considered primary sampling units and generally all subjects belonging to each group are included to obtain data of interest [1,2].

However, depending on the method of selection, data obtained from clusters may not be sufficiently representative to allow for generalization. Population observed in clusters can present a large degree of similarity in some characteristics (homogeneity), unlike when there is a simple random sampling (SRS), in which each individual has the same probability of being selected in the general population, with more heterogeneity [2].

Therefore, an important characteristic of cluster-based studies is to evaluate the proportion of data variability that is explained by means of clustering, and this reliability may be analyzed by measuring inter and intracluster variance [3].

Intracluster correlation coefficient (ICC), denoted by ρ, is defined as the ratio of the between-cluster variance to the total variance (both between and within clusters), and therefore has a value between 0 and 1 [4,5]. Its value depends on the type of variable, cluster size and the prevalence of the condition [6]. Coefficients close to zero indicate that individuals within clusters are no more similar to each other than individuals from different clusters (the variable is randomly distributed among clusters); otherwise the values close to 1 reflect the homogeneity in a sample [7]. In other words, for cluster based population studies this heterogeneity (ICC close to zero) is desired as a proxy to the subjects being randomly selected.

The increase in variance due to clustering, compared to what would be obtained if sampling had been carried out by the SRS method, is calculated by design effect (Deff) [8]. It is given by 1 + (m-1) ICC, where m is the average cluster size [9]. Deff value is directly proportional to ICC and to the size of a cluster [10].

The ICC estimate in cluster studies is very useful for the development of new studies in the same field, because values obtained could be used as a correction factor for the calculation of sample size needed, thus avoiding underestimates, since in studies in which SRS is used, the sample size required to achieve sufficient statistical power is usually smaller [4].

The purpose of this manuscript is to evaluate the ICC of variables studied in the Brazilian Multicenter Study on Preterm Birth, a multicenter cross-sectional study on preterm births involving 20 referral hospitals in different regions of Brazil plus a nested case–control study. Estimated prevalence rates or means, ICC with 95% confidence intervals, design effects and average cluster sizes were also objectives for this study and they are presented for more than 250 maternal and neonatal variables.

Methods

The Brazilian Multicenter Study on Preterm Birth consisted of a multicenter cross-sectional study plus a nested case–control study to assess their associated factors implemented in referral obstetrical units (clusters) from several states of the country. The full research proposal has already been published elsewhere [11].

A single-stage cluster sampling was used. Clusters were selected by an invitation to 27 healthcare institutions that build a national network called Brazilian Network for Studies on Reproductive and Perinatal Health. They are located in the five geographical regions of the country, almost all of them are public institutions, and all of them receive both low and high risk pregnant women. Initially 26 centers accepted to participate, but 20 selected institutions were able to fully take part in the study.

The sample size was calculated using the official prevalence of preterm births in Brazil of around 6.5% [12]. Considering an acceptable absolute difference of about 0.25% between the sample and the population prevalence, and a type I error of 5%, initial surveillance of a sample size of 37,000 deliveries was necessary. For the case–control study component, the estimated sample size was 1,055 women in each group (cases and controls). The total number of preterm births estimated to be followed in both components of the study was around 3,600.

The participating centers performed a prospective surveillance of all patients admitted to give birth in order to identify preterm births. For this purpose and according to standard international definitions, preterm birth was considered that occurring before 37 completed weeks of gestational age evaluated by an ultrasound scan performed early in pregnancy, by a known date of the last menstrual period, or alternatively by the evaluation of the somatic age of the newborn. During the first months of the study, in order to complete the sample for the appropriate analysis of the factors associated with spontaneous preterm birth, a random sample of women who had full-term birth was also selected.

Data was collected during six to twelve months for each center, from April 2011 to March 2012, in a detailed form called “Questionnaire” including 306 variables from four sources: interview with women in the postpartum period, medical records and prenatal chart of the mother (before hospital discharge), and newborn medical records (within sixty days after birth, even if it remained in hospital for longer period). An electronic system of data entry called OpenClinica® was selected and a proper clinical research form (CRF) was designed for the input of data after the questionnaire of each case was completed and reviewed.

High quality data and reliable information was guaranteed by several steps: preparatory meetings, development of detailed manuals of operation, monitoring technical site visits to the centers, close monitoring of data collection and data entry, concurrent query management, checking for logical inconsistencies, and correction of database. The research proposal was firstly approved by the Institutional Review Board of the coordinating center and then confirmed by IRB of each other participating center.

Data analysis

In this study, each of the 20 participating centers (hospital) was considered a primary sampling unit (PSU) and there was no stratification of the PSU or weighting of the data. The subject (unit of analysis) was woman who delivered preterm (case) or at term (control).

Estimated prevalence (categorical variables) or means (continuous numeric variables), intracluster correlation coefficients (ICC), their respective 95% confidence intervals (CI), design effects (Deff) and mean cluster size of each variable were calculated. Software programs used for analysis were SPSS® version 20.0 [13] and Stata version 7.0 [14], taking into consideration the cluster sampling plan (centers) for data analysis.

According to Kish [2], ICC (Roh) is: ρ = (s2a − s2b/b)/sˆ2, where s2a is the variance between clusters; s2b is the variance within clusters, b is the size of clusters and sˆ2 is the estimate of S2 (variance in individual level). The estimate sˆ2 is obtained by: sˆ2 = s2a + [(b − 1)/b]s2b. Stata’s equivalent computing formula for ICC [14] is: ICC = [(F − 1)a/n]/1 + (F − 1)a/n, where ‘F’ is the Snedecor’s F-value from the ANOVA table and ‘a’ is the number of groups. The variance estimate for ICC is obtained by an extensive asymptotic formula and because this it was not showed.

For this study, the Design effect - DEFF [2] is Deff = varactual(r)/varSRS(r) = s2a/a/s2/n) where varactual(r) is the estimated variance according to the complex design being studied and varSRS(r) is the variance in the estimator considering the design as if it were calculated using a SRS of the same size, n.

Results

During fifteen months, 5,296 births were included in the study, 4,150 of them being preterm births (1,491 due to spontaneous preterm labor, 1,191 due to a prelabor premature rupture of membranes and 1,468 due to a therapeutic interruption of pregnancy either for a maternal or fetal condition) and a sample of 1,146 term births to be used as controls for the case–control component.

Clustering was not stratified by region. Proportionally more centers were located in the Southeast of the country and consequently over half of births were from this region (11/20 – 53.5%). The other centers were from Northeast region (7/20 – 35%), contributing with 34.8% of births studied, South region (2/20 – 10%), with 11.7% of births. The mean size of each cluster was 265 cases.

Estimated ICCs

Estimated ICCs are presented in Tables 1, 2, 3, 4, 5, 6, 7, 8, 9 and 10 for each of 261 variables. Tables 2 and 8 show results for continuous numeric variables, while other tables present results for categorical variables or were categorized for analysis. In addition to ICC, the 95% confidence interval (CI), the design effect (Deff) and the mean cluster size (na), as well as the estimated prevalence (or mean) are presented. ICC ranged from <0.001 to 0.965, with a median of 0.028. ICC was < 0.1 in 78.5% of the variables and < 0.3 for 95% of them.

Table 1. Estimates of prevalence (P), intracluster correlation coefficients (ICC), their respective 95% CI, design effect (Deff), and mean cluster size (na) for categorical maternal characteristics

Table 2. Estimates of mean, intracluster correlation coefficients (ICC), their respective 95% CI, design effect (Deff), and mean cluster size (na) for numeric maternal characteristics

Table 3. Estimates of prevalence (P), intracluster correlation coefficients (ICC), their respective 95% CI, design effect (Deff), and mean cluster size (na) for maternal socio-demographic characteristics

Table 4. Estimates of prevalence (P), intracluster correlation coefficients (ICC), their respective 95% CI, design effect (Deff), and mean cluster size (na) for categorical variables of process during pregnancy

Table 5. Estimates of prevalence (P), intracluster correlation coefficients (ICC), their respective 95% CI, design effect (Deff), and mean cluster size (na) for categorical variables of process during pregnancy

Table 6. Estimates of prevalence (P), intracluster correlation coefficients (ICC), their respective 95% CI, design effect (Deff), and mean cluster size (na) for categorical variables of process during labor

Table 7. Estimates of prevalence (P), intracluster correlation coefficients (ICC), their respective 95% CI, design effect (Deff), and mean cluster size (na) for categorical newborn outcome variables

Table 8. Estimates of mean, intracluster correlation coefficients (ICC), their respective 95% CI, design effect (Deff), and mean cluster size (na) for numeric newborn outcome variables

Table 9. Estimates of prevalence (P), intracluster correlation coefficients (ICC), their respective 95% CI, design effect (Deff), and mean cluster size (na) for categorical management variables in spontaneous labor conditions or preterm due to pPROM

Table 10. Estimates of prevalence (P), intracluster correlation coefficients (ICC), their respective 95% CI, design effect (Deff), and mean cluster size (na) for diagnosis and management among categorical variables related to therapeutic preterm delivery

Tables 1 and 2 presents some variables related to maternal characteristics, including clinical and obstetrical history. ICCs ranged from <0.001 to 0.145 (median 0.011). Table 3 shows the socio-demographic variables studied, and ICC ranged from 0.017 to 0.191 (median 0.041). Tables 4 and 5 presents variables related to pregnancy characteristics with ICC ranging from 0.001 to 0.386 (median 0.015). The variables related to labor conditions were presented in Table 6. It can be observed that ICC ranged from 0.002 to 0.384, with a median of 0.022. Tables 7 and 8 shows variables related to perinatal outcomes and ICC were < 0.1 in 81% of them. The most important outcome variable, newborn morbidities, is presented in Table 7. Tables 9 and 10 present some variables analyzed specifically for preterm births and are related to management. Most variables in Table 9 showed ICC greater than 0.3 and the greatest ICC of this study (0.965) was relative to the variable “corticosteroids use”, a management aspect well defined and well-established in all obstetric protocols, so there were high degree of homogeneity in clusters in these variables. The median of ICCs was 0.274. The median ICC in Table 10 was 0.079.

Estimated deffs

Estimated Deffs are presented in Tables 1, 2, 3, 4, 5, 6, 7, 8, 9 and 10 for each of 261 variables. Deff ranged from 0.6 to 148.0, with a median of 6.1.

Deff were under 5.0 in 74% of variables in Tables 1 and 2, ranging from 1.2 to 42.2 (median 3.65). Table 3 presents Deff values ranging from 2.8 to 60 (median 13.0). In variables related to gestational process (Tables 4 and 5), Deff values ranged from 0.6 to 101.5 (median 4.9). The variables related to labor conditions (Table 6) showed Deff ranging from 1.1 to 148 (median 6.6), with 60% of them under 8.0. In Tables 7 and 8, related to perinatal outcomes, Deff values ranged from 1.1 to 84.8 (median 7.8). Tables 9 and 10 presented Deff median of 16.35 and 7.0, respectively.

We can observe that greater Deff median is present in process variables (Table 9), and greater ICCs.

Discussion

This study presents a large number of intracluster correlation coefficients whose values can be considered low (close to zero) in most variables, showing intracluster heterogeneity.

The greater ICC values were found in process variables, especially management in spontaneous preterm labor conditions, as corticosteroids use, Group B streptococcus screening, use of tocolytic agents and use of antibiotic. Indeed, the mean ICC value for these variables was 10 times higher than the mean ICC of the study. The variable with the highest ICC was “corticosteroids – betamethasone”, with a value of 0.965. The prevalence of this variable was 85%, showing a high degree of homogeneity in this management for preterm labor. These findings are in accordance with the literature that describes ICC values generally higher for variables related to process compared to those variables related to outcome [15,16].

In the field of maternal and perinatal healthcare, Taljaard et al. calculated ICC values based on data obtained from secondary/tertiary services [16]. Comparing with our study, they found an overall median ICC of 0.067 versus 0.028. For maternal and newborn outcome variables, their median ICCs was 0.011 (versus 0.014), and 0.054 (versus 0.041), respectively. The findings of those investigators showed that, for variables associated to process, ICC values tend to be > 0.07. The present findings are in agreement with this observation.

Pagel et al.[17] estimated ICC for a range of outcomes using data from five community-based clusters randomized controlled trials in three low-income countries. Estimated ICC values for mortality outcomes were lower than those for process outcomes, with narrower confidence intervals throughout for trials with larger number of clusters.

All comparisons show that the smaller the cluster size, the higher the ICC and the opposite occurs regarding the prevalence of the condition. Estimates of intracluster correlation are much less reliable for rare outcomes and the size of the cluster had a greater impact than the number of clusters on the reliability of estimates for rare outcomes [17].

Furthermore, higher healthcare levels tend to increase the degree of homogeneity [18,19]. The size of ICC increases if the ICC represents data from secondary rather than primary care. This may be a reflection of the underlying heterogeneity of the datasets under consideration as the conditions represented across the different datasets were diverse. Although numerically small (average 0.01), such differences can have a substantial effect on sample size, even when the average of cluster is small [15]. The clusters in this study are secondary and tertiary hospitals, most of them are teaching hospitals, with the majority of procedures performed in conformity with evidence-based healthcare protocols.

Stratified randomization had the effect of reducing estimates of cluster correlation [15]. However, in the same way that in Brazilian Network for Surveillance of Severe Maternal Morbidity Study [20], which found ICC values close to zero, the selection of clusters did not performed stratification by region. The distribution of centers in this study, with almost half located in southeast region, is in accordance with the actual distribution of healthcare institutions and the proportionality of births per region in the country [21,22].

The large number of intracluster correlation coefficients presented in this study, considered low (close to zero) in most of variables, can probably be seen as a good parameter of variance for calculating sample size in new studies in the field of perinatal and maternal health [15].

We can, however, to identify some possible limitations of the study, including the fact that we used a non-probabilistic sample from the centers (hospitals). Therefore, strictly speaking, the findings cannot be generalized to other populations. However, the majority of hospitals included in the study was third level referral hospitals taking care of high risk pregnancies and preterm babies. Probably the results would be applicable to other centers with such characteristics, irrespective of being private or public, especially in middle income countries like Brazil.

Conclusions

The Brazilian Multicenter Study on Preterm Birth, developed as part of the Brazilian Network for Studies on Reproductive and Perinatal Health, to the best of our knowledge is the first cross sectional multicenter study on this topic in the country. It represents a planned comprehensive assessment of preterm birth in Brazil and ICC values calculation and analysis of more than 250 maternal and newborn variables, showed heterogeneity of data in selected clusters. These findings increase reliability of study estimates and allow the use of these results to calculate the required sample size for future research studies in maternal and perinatal health.

Abbreviations

CRF: Clinical research form; Deff: Design effect; ICC: Intraclass correlation coefficient; SRS: Simple random sampling.

Competing interests

The authors declare that they have no competing interests.

Authors’ contributions

The idea for the study first arose in a discussion among RPJ, JGC and RPT and then was developed and implemented by the whole research team. The first version of the manuscript was drafted by GJL and SMH, and then complemented with the suggestions of the others. RPJ and JGC supervised the entire process. All authors contributed to the development of the study protocol and approved the final version of manuscript.

Acknowledgements

The authors thank the CNPq (Brazilian National Research Council) and Fapesp (Foundation for Support to Research of the State of Sao Paulo) for the financial sponsorship of this study, Process Fapesp 2009/53245-5 (Call AP.PPSUS-1). This publication was also sponsored by Fapesp (Processo 2014/08183-0)

The Brazilian Multicenter Study on Preterm Birth study group: Sergio T Marba, Ruth Guinsburg, Francisco E Martinez, Vilma Zotarelli, Lucio T Gurgel, Francisco E Feitosa, George N Chaves, Ana M Porto, Isabela C Coutinho, Antonio C Barbosa Lima, Elias F Melo Jr, Débora F Leite, Melania M Amorim, Adriana SO Melo, Fabiana O Melo, Marília G Martins, Marynea V Nunes, Cláudio S Paiva, Moises D Lima, Djacyr M Freire, Edson G Tristão, Denis J Nascimento, Renato T Souza, Carlos A Menezes, Marcelo Aquino, Janete Vettorazzi, Cintia E Senger, Augusta MB Assumpção, Marcela AF Guedes, Maria EL Moreira, Vera T Borges, Nelson L Maia Filho, Jacinta P Mathias, Eduardo Souza, Ana CP Zamarian, Silvana M Quintana, Patrícia PS Melli, Fátima A Lotufo, Kaliane Uzilin, Elvira A Zanette, Carla B Andreucci, Tenilson A Oliveira, Laércio R Oliveira, Nelson Sass, Mirian RF Silveira, Pedro R Coutinho, Luciana Siqueira.

References

  1. Ukoumunne OC, Gulliford MC, Chinn S, Sterne JA, Burney PG: Methods for evaluating area-wide and organisation-based interventions in health and health care: a systematic review.

    Health Technol Assess 1999, 3(5):iii-92. PubMed Abstract | Publisher Full Text OpenURL

  2. Kish L: Survey sampling. New York: John Wiley & Sons; 1965. OpenURL

  3. Donner A, Koval JJ: Design considerations in the estimation of the intraclass correlation.

    Ann Hum Genet 1982, 46:271-277. PubMed Abstract | Publisher Full Text OpenURL

  4. Campbell MK, Grimshaw JM, Elbourne DR: Intracluster correlation coefficients in cluster randomised trials: empirical insights into how should they be reported.

    BMC Med Res Methodol 2009, 4:9. OpenURL

  5. Killip S, Mahfoud Z, Pearce K: What is an intracluster correlation coefficient? Crucial concepts for primary care researchers.

    Ann Fam Med 2004, 2:204-208. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  6. Adams G, Gulliford MC, Ukoumunne OC, Eldridge S, Chinn S, Campbell MJ: Patterns of intra-cluster correlation from primary care research to inform study design and analysis.

    J Clin Epidemiol 2004, 57:785-794. PubMed Abstract | Publisher Full Text OpenURL

  7. Hayes R, Moulton LH: Cluster Randomised Controlled Trials. New York: Chapman &Hall/CRC Press; 2009. OpenURL

  8. US Census Bureau: Technical paper 63RV: current population survey - design and methodology. TP63RV 2002, 14–18. Available at http://www.census.gov/prod/2002pubs/tp63rv.pdf webcite. Accessed on October 02, 2013

  9. Hayes RJ, Benett S: Simple size calculation for cluster-randomised trials.

    Int J Epidemiol 1999, 28(2):319-326. PubMed Abstract | Publisher Full Text OpenURL

  10. Eldridge SM, Ukoumunne OC, Carlin JB: The intra-cluster correlation coefficient in cluster randomized trials: a review of definitions.

    Int Stat Rev 2009, 77(3):378-394. Publisher Full Text OpenURL

  11. Passini R Jr, Tedesco RP, Marba ST, Cecatti JG, Guinsburg R, Martinez FE, Nomura ML, Brazilian Network of Studies on Reproductive and Perinatal Health: Brazilian multicenter study on prevalence of preterm birth and associated factors.

    BMC Pregnancy Childbirth 2010, 10:22. PubMed Abstract | BioMed Central Full Text | PubMed Central Full Text OpenURL

  12. Tedesco RP, Passini R Jr, Cecatti JG, Camargo RS, Pacagnella RC, Sousa MH: Estimation of preterm birth rate, associated factors and maternal morbidity from a Demographic and Health Survey in Brazil.

    Matern Child Health J 2013, 17(9):1638-1647. PubMed Abstract | Publisher Full Text OpenURL

  13. SPSS for Windows: Release 20.0. New York: SPSS Inc; 2011. OpenURL

  14. StataCorp: Stata Statistical Software: Release 7. College Station, TX: StataCorp LP; 2001. [College Station] OpenURL

  15. Campbell MK, Fayers PM, Grimshaw JM: Determinants of the intracluster correlation coefficient in cluster randomised trials: the case of implementation research.

    Clin Trials 2005, 2:99-107. PubMed Abstract | Publisher Full Text OpenURL

  16. Taljaard M, Donner A, Villar J, Wojdyla D, Velazco A, Bataglia V, Bataglia V, Faundes A, Langer A, Narváez A, Valladares E, Carroli G, Zavaleta N, Shah A, Campodónico L, Romero M, Reynoso S, de Pádua KS, Giordano D, Kublickas M, Acosta A, World Health Organization 2005 Global Survey on Maternal and Perinatal Health Research Group: Intracluster correlation coefficients from the WHO Global Survey on Maternal and Perinatal Health: implications for implementation research.

    Paediat Perinatal Epidemiol 2005, 2008(22):117-125. OpenURL

  17. Pagel C, Prost A, Lewycka S, Das S, Colbourn T, Mahapatra R, Azad K, Costello A, Osrin D: Intracluster correlation coefficients and coefficients of variation for perinatal outcomes from five cluster-randomised controlled trials in low and middle-income countries: results and methodological implications.

    Trials 2011, 12:151. PubMed Abstract | BioMed Central Full Text | PubMed Central Full Text OpenURL

  18. Piaggio G, Carroli G, Villar J, Pinol A, Bakketeig L, Lumbiganon P, Bergsjø P, Al-Mazrou Y, Ba'aqeel H, Belizán JM, Farnot U, Berendes H, WHO Antenatal Care Trial Research Group: Methodological considerations on the design and analysis of an equivalence stratified cluster randomization trial.

    Stat Med 2001, 20:401-416. PubMed Abstract | Publisher Full Text OpenURL

  19. Campbell SM, Braspenning J, Hutchinson A, Marshall M: Research methods used in developing and applying quality indicators in primary care.

    Qual Safe Health Care 2002, 11:358-364. Publisher Full Text OpenURL

  20. Haddad SM, Sousa MH, Cecatti JG, Parpinelli MA, Costa ML, Souza JP, and for the Brazilian Network for Surveillance of Severe Maternal Morbidity Group: Intraclass correlation coefficients in the Brazilian network for surveillance of severe maternal morbidity study.

    BMC Pregnancy Childbirth 2012, 12:101. PubMed Abstract | BioMed Central Full Text | PubMed Central Full Text OpenURL

  21. Brazilian Ministry of Health: National Registry of Health Facilities.

    2013.

    Available at http://cnes.datasus.gov.br/Lista_Tot_Es_Estado.asp. Accessed on November 27th, 2013

  22. Brazilian Ministry of Health: SINASC – National Information System on Live Births.

    2011.

    Available at http://tabnet.datasus.gov.br/cgi/tabcgi.exe?sinasc/cnv/nvuf.def. Accessed on November 27th, 2013

Pre-publication history

The pre-publication history for this paper can be accessed here:

http://www.biomedcentral.com/1471-2288/14/54/prepub