Comparing estimates of child mortality reduction modelled in LiST with pregnancy history survey data for a community-based NGO project in Mozambique

Abstract

Background

There is a growing body of evidence that integrated packages of community-based interventions, a form of programming often implemented by NGOs, can have substantial child mortality impact. More countries may be able to meet Millennium Development Goal (MDG) 4 targets by leveraging such programming. Analysis of the mortality effect of this type of programming is hampered by the cost and complexity of direct mortality measurement. The Lives Saved Tool (LiST) produces an estimate of mortality reduction by modelling the mortality effect of changes in population coverage of individual child health interventions. However, few studies to date have compared the LiST estimates of mortality reduction with those produced by direct measurement.

Methods

Using results of a recent review of evidence for community-based child health programming, a search was conducted for NGO child health projects implementing community-based interventions that had independently verified child mortality reduction estimates, as well as population coverage data for modelling in LiST. One child survival project fit inclusion criteria. Subsequent searches of the USAID Development Experience Clearinghouse and Child Survival Grants databases and interviews of staff from NGOs identified no additional projects. Eight coverage indicators, covering all the project’s technical interventions were modelled in LiST, along with indicator values for most other non-project interventions in LiST, mainly from DHS data from 1997 and 2003.

Results

The project studied was implemented by World Relief from 1999 to 2003 in Gaza Province, Mozambique. An independent evaluation collecting pregnancy history data estimated that under-five mortality declined 37% and infant mortality 48%. Using project-collected coverage data, LiST produced estimates of 39% and 34% decline, respectively.

Conclusions

LiST gives reasonably accurate estimates of infant and child mortality decline in an area where a package of community-based interventions was implemented. This and other validation exercises support use of LiST as an aid for program planning to tailor packages of community-based interventions to the epidemiological context and for project evaluation. Such targeted planning and assessments will be useful to accelerate progress in reaching MDG4 targets.

Background

Although there are encouraging trends in some key countries, meeting Millennium Development Goal (MDG) 4 for reduction of child mortality will be challenging, given current trends.[1] Community-based intervention packages are not commonly implemented at large scale, although recent evidence demonstrates that they are effective for neonatal and child mortality reduction at moderate scale in various resource-constrained settings. [2, 3] This has prompted calls for greater emphasis on community-level delivery, especially preventive interventions and integrated strategies. [2, 4, 5]

Analysis of effectiveness of this type of programming is hampered by its cost and complexity. It is difficult to estimate the mortality impact of packages of interventions in realistic field settings, as well as effectiveness of component interventions within packages. [6] Projects implementing interventions under these conditions usually lack the resources necessary to carry out mortality impact evaluations. The Lives Saved Tool (LiST) produces mortality reduction estimates by modelling the mortality effect of increases in population coverage for key child health interventions. LiST calculates this by combining coverage change data with data on effectiveness of each intervention against common serious child illnesses, and country-specific cause of death profiles. This is explained in detail elsewhere. [7] By producing intuitive and equivalent outputs from otherwise disparate data, such as the percentage reduction in mortality rates and number of deaths averted, LiST facilitates comparisons that are otherwise difficult to make. Population based surveys in which mortality is directly measured are costly, difficult, and time-consuming, and LiST modelling could be an attractive alternative to estimate mortality reduction.

In order to validate LiST-produced estimates of child mortality reduction in community-based NGO programming, a search was done of such projects with complete coverage data for their child health interventions and independent child mortality reduction estimates. One met criteria for inclusion.

Methods

Search for community-based NGO projects

Projects with data available for validation of LiST were sought based on the following criteria: 1. Study was of a community-based NGO child health project; 2. Baseline and endline population coverage indicators were available for at least two child health interventions; 3. Mortality data were available at least at baseline and endline and independently verified. A comprehensive search of the published literature had been run on PubMed by one of the authors (HP) for effectiveness of community-based interventions. The 3,000 articles from this search were reviewed and one project was identified that fit selection criteria, a USAID-funded child survival project implemented by World Relief in Mozambique from 1999-2003.[8] A search for similar projects not published in the peer-reviewed literature was then run on USAID’s Development Experience Clearinghouse database (http://dec.usaid.gov) and Child Survival and Health Grants database (http://www.mchipngo.net). Five additional candidate projects were identified. Project documentation was reviewed and knowledgeable staff interviewed. None of these additional projects met inclusion criteria. Table 1 shows selected key characteristics of this project which was not a research project and had no control or comparison group. The project intervened on all major causes of under-five mortality in the area (neonatal conditions, malaria, pneumonia, diarrhea, and measles) except HIV/AIDS. Direct mortality data were available from an independent retrospective mortality assessment carried out in 2004 by a research team from the Mozambican National Institute of Statistics, Ministry of Health, World Relief and other NGOs, and designed by collaborators from Johns Hopkins School of Public Health. This research team used a pregnancy history survey adapted from the birth history in the women’s questionnaire of the 2003 Mozambique Demographic and Health Survey.

Table 1 Key characteristics, strategies, interventions, and results of World Relief Mozambique Vurhonga II project (explained in detail in Edward, et. al.[7])

Coverage data used for modeling in LiST

The project collected population coverage data on seven LiST interventions pertaining to its areas of intervention. Coverage for an eighth LiST intervention (education for complementary child feeding) was not available, but data collected for increased food intake during previous pregnancy, another nutrition education intervention included in its nutrition education package (Table 2), was used as a proxy. These eight indicators cover all project technical intervention areas.

Table 2 World Relief Mozambique Vurhonga II project coverage data and mapping to LiST indicators for modelling

Coverage data was collected at baseline and endline using a small-sample survey instrument known as the Knowledge, Practices, and Coverage (KPC) survey, based on DHS questions. Households were selected according to a standard cluster sampling methodology, with 30 independent clusters and 10 households in each cluster. Cluster selection was based on village level population data, with probability of selection proportional to population size. The method used is explained in detail elsewhere [9] The project target geographic area remained invariant from baseline to endline and comprised the entire district of Chokwe except Chokwe town – 48 villages with an estimated population of 119,467 at baseline). The KPC survey collects data on multiple indicators important to the project, and the sample is designed to detect statistically significant baseline/endline differences of at least ± 16% (alpha = 0.05, beta = 0.20) if no sub-sampling is done and an indicator starts at a baseline of 50%. Ninety five percent confidence intervals for project data used for LiST modelling are shown in Table 2. KPC surveys were carried out in October 1999 and July 2003. Project activities started in March 2000, so this is taken as the baseline year for LiST modelling.

KPC surveys cover mothers/caretakers of children 0-23 months of age. The surveys were carried out by the project staff themselves. In order to minimize possible bias, interviewers were not assigned clusters in which they themselves were working in their day-to-day project activities. The data was checked for consistency by an independent team at ICF Macro before being entered in a publically available database (http://www.mchipngo.net).

The other child health indicators in LiST for which the project did not have data were reviewed. Coverage data were estimated for most of the interventions being implemented at the time. The values of and sources for non-project data are shown in Table 3. Most data are from the 1997 and 2003 Demographic and Health Surveys (DHS). When 1997 data is used, its value assumed by LiST to change linearly toward its 2003 value, and the 2000 value assigned by LiST is used as the baseline to estimate mortality reductions.

Table 3 LiST indicators not in project data – estimated values and data sources

LiST modeling

LiST is a cohort model of child survival from 0-59 months of age. Its structure and assumption are described in detail elsewhere.[7, 10] LiST provides estimates of the cause-specific child mortality impact of over 40 interventions with strong evidence of effect on child survival. The user must supply the values of changes in coverage for these interventions. LiST has country-specific baseline under-five and infant mortality rates and cause of death profiles needed for its calculations. These parameters can be manipulated by the user if desired. The Child Health Epidemiology Reference Group (CHERG) meets periodically to weigh published evidence, determine which interventions to include in the model and what effect sizes to assign them.[11] The under-five mortality modeling is contained within the Spectrum platform which models demographic trends, given assumptions about population growth rates and prevalence of use of family planning methods. [12] Version 4.2 of the LiST tool was used for modeling and was downloaded from the Johns Hopkins Institute for International Programs web site. [7] The under-five and infant mortality rates used were those specific to the project area at baseline, as measured in the pregnancy history and described in detail elsewhere. [8] National cause of death profiles, population structure, and fertility data were used.

All available coverage data both from the World Relief Mozambique project and other sources were examined to determine which coverage indicators matched those in LiST. The authors discussed the indicator definitions and corresponding coverage data that best fit the interventions in LiST. Eight project indicators (Table 2) were mapped to LiST interventions. The fit between project indicators and LiST was exact for seven of the indicators. For one LiST indicator (complementary feeding) the project had no direct data, but had intervened for a package of behavior change practices that included both maternal nutritional practices during pregnancy and child feeding practices. The project had data on the coverage for increased food intake during last pregnancy, and this was used as a proxy for child complementary feeding practices. Of the other 21 indicators in LiST for interventions being implemented in Mozambique at the time, information was available from other sources for eight; LiST estimates the value of nine others from available data (e.g. LiST estimates coverage for syphilis screening from ANC coverage). Non-project data used for LiST modelling is summarized in Table 3. In summary, data was available for all but four of the indicators in LiST for interventions being implemented in Mozambique in the relevant time period.

Sensitivity analyses were run for the LiST estimates, by varying all the parameters used in the model: Coverage data was varied within the limits of the 95% confidence intervals. The values assigned for intervention effectiveness, baseline mortality figures, and cause of death profiles were varied by ± 10% as well.

Pregnancy history survey

Infant and child mortality were estimated for the period from two years before the start of project activities through one year after it ended (1998-2004), with calculations for these parameters at one year intervals. The information for these calculations was derived from a complete pregnancy history survey of 998 women in the project area, performed in May 2004. The questionnaire used was adapted from the birth history in the DHS 2003 women’s questionnaire and was implemented by a group of surveyors that included personnel from the National Institute of Statistics who had implemented the 2003 DHS, as described in detail in Edward, et.al. [8] The pregnancy history covered all pregnancies, but the period analyzed and reported was the period from March 1998 to February 2004. The baseline period used to match LiST estimates was March 2000 – February 2001 and the endline period used was March 2003-Feb. 2004.

An unpublished Fortran program written by one of the authors of Edward, et. al. takes as input the time period (beginning and end dates in months) and age group (minimum and maximum) for mortality estimation and calculates m(x) for this age group in the time period. Using the formula of Chiang [13] and the calculation of mean time lived in the interval for those dying in the interval, ₁q₀ and ₅q₀ were calculated and the data plotted in a lexis diagram. Standard errors were calculated assuming a Poisson distribution.

Results

LiST modeled mortality estimates and the corresponding directly measured mortality estimates are shown in Table 4. The project had several measures of under-five and infant mortality reduction, derived both from a project-implemented community-based vital events registration system and from the independent pregnancy history survey. [8] The latter was felt to be the most accurate mortality measure for use as a comparison to LiST results. The directly measured under-five mortality rate (U5MR) from the birth history survey demonstrated a baseline value of 180 per 1,000 live births (95% CI, 130 – 230) in the 12 month period from March 2000 to February 2001 decreasing to an endline value of 114 per 1,000 live births (95% CI, 75 – 153) for the period from March 2003 to February 2004. This represents a 37% reduction in U5MR. The estimation of under-five mortality based on coverage changes modeled in LiST was 110, an estimated U5MR reduction of 39%. The directly measured infant mortality rate (IMR) from the birth history survey demonstrated a baseline value of 102 per 1,000 live births (95% CI, 64 – 141) for the period from March 2000 to February 2001 decreasing to an endline value of 53 per 1,000 live births (95% CI, 25 – 81) for the period from March 2003 to February 2004. This represents a 48% reduction in IMR. The estimation of IMR based on coverage changes modeled in LiST gave an endline IMR of 67, for an estimated IMR reduction of 34%. The LiST estimates for both under-five and infant mortality reductions are within the 95% confidence limits of the directly measured mortality estimates obtained from the pregnancy history survey.

Table 4 Under-five and infant mortality, comparison of measured and LiST modelled changes

Discussion

Accuracy and completeness of coverage data

We used survey data generated as part of standard program monitoring and evaluation activities to model mortality impacts using LiST. Although the available data was not collected as part of a research project, the coverage data input into LiST was of sufficient quality to generate relatively accurate estimates within the limits of the tool. A standard survey instrument was used; data was collected by professional project staff; the potential for bias reduced by avoiding having interviewers collect information from villages where they worked; supervisory spot checks were performed for reliability of information; and data was reviewed for quality by technical support staff from ICF Macro on entry into the online child survival project database.

Although project interventions targeted children 0-59 month olds, which is the cohort modeled in LiST and whose mortality was measured directly in the pregnancy history, the coverage data used for LiST was collected for children 0-23 months old. The inaccuracy caused by this is likely to be small for the following reasons: (1) Even though the KPC measures are collected for children 0-23 months of age, in fact the project implemented interventions for the entire 0-59 month cohort of children, so we expect that the coverage for 0-59 month olds to be substantially the same. (2) we expect that 79% of deaths in children 0-59 months occurred in 0 to 23 month olds, so coverage of 0-23 month olds is the most critical. This calculation was done using the Model Life Tables for Developing Countries published by the United Nations [14] for an area with the project’s baseline U5MR and IMR. (3) LiST assigns the same effect size to relevant age groups 0-23 months and 24-59 months for all modelled project interventions. As part of the sensitivity analysis presented in Table 5, the effect was examined of halving the coverage change among 24-59 month olds compared to that measured in the KPC for 0-23 month olds. This dropped the estimate of U5MR reduction by 4.8%.

This project had several nutrition education interventions for well, sick, and malnourished children as well as pregnant mothers, but only an estimate for complementary feeding matched the nutrition interventions in LiST. The CHERG has not included other project interventions in LiST like continued feeding during diarrheal episodes because of a lack of published high quality data needed to accurately estimate an effect size, even though they are likely to have an effect on child mortality.

Table 5 Sensitivity analysis results for LiST modelling

Accuracy of modelled mortality estimates

The accuracy of the U5MR reduction estimate (39% LiST; 37% Pregnancy History) was better than the IMR reduction estimate (34% LiST; 48% Pregnancy History). Both were within the 95% CI of the parameter, but LiST’s underestimation of the reduction in IMR may be caused by the fact that the only nutritional intervention with probable infant mortality impact that could be modelled in LiST was complementary feeding.

The results of a sensitivity analysis of the LiST model are shown in Table 5. LiST estimates of mortality reduction are calculated based on several inputs: The baseline mortality rate, cause of death profile, change in coverage for each of the interventions in the model, and their effect sizes. Table 5 shows the effect on LiST’s estimates of the reductions in U5MR and IMR caused by changing one of the most critical examples of each of these parameters by 10%. The manipulations of the diarrhea and malaria parameters are shown in the table, as the project had large coverage changes for highly effective interventions for these causes of death. The modelled changes in mortality are more sensitive to changes in parameters that affect the calculation of the overall baseline mortality than they are to changes in the estimation of coverage or intervention effectiveness. This is not surprising, as the value of the baseline mortality affects the calculations for all interventions in the model.

One of the potential strengths of LiST is its ability to simplify analysis of situations in which multiple interventions are implemented simultaneously. Yet to date there have been few published reports on the accuracy of LiST estimates for mortality reductions in areas where packages of community-based child health interventions are being implemented. The LiST validation with data from the evaluation of Accelerated Child Survival and Development programs is similar to this one [15] and to some extent the national level exercises with DHS data.[16] The current analysis shows that even in the context of relatively complex community-based NGO programming with interventions designed to affect less proximate determinants of child health like level of community organization and women’s empowerment, LiST accurately estimated mortality changes.

Limitations of validation analysis

Although coverage data for project interventions was fairly complete and the time periods coincided well with the mortality estimates calculated from the pregnancy history, the main limitations of the current exercise are (1) that the data was not available from the project for 21 relevant LiST indicators, and had to be estimated mainly from consecutive DHS surveys and (2) the 95% confidence intervals are quite wide for the mortality estimates derived from the pregnancy history.

There are cautions that must be kept in mind when using LiST. The accuracy of its estimates is dependent on having accurate information on the causes of death in the program area. National cause of death profiles are now available through CHERG, but there may be important variation from one region of a country to another. The outputs from LiST must also be interpreted in light of complementary considerations. For example, when used in planning LiST could mistakenly give the impression that mature interventions like vaccination that already have achieved high levels of coverage are not important, as simply maintaining high coverage yields no additional lives saved. LiST also does not take account of the mode of delivery and the fact that delivery of some interventions like antenatal care or vaccination establishes a platform that can serve for adding other interventions, like ITN or vitamin A distribution. Even with an awareness of these limitations and caveats, LiST can be a valuable aid in prioritizing choices for deployment of scarce resources.

Although only a single project was identified for study, it is typical of integrated, community-based NGO programming and implemented under realistic field conditions in a resource-constrained setting typical of conditions of other community-based NGO programming and the type of settings in which greater progress needs to be made to reach MDG4 targets.

Conclusions

A validation exercise has confirmed that in a relatively routine field setting of an NGO child survival project implementing a package of community-based interventions in Mozambique, the Lives Saved Tool (LiST) provides a reasonably accurate estimate of under-five and infant mortality reduction when compared to independent directly measured mortality estimates. These are the kinds of routine programming conditions that LiST attempts to simulate with its modeling. These findings support the use of LiST as a practical tool for estimating the mortality effect of NGO community-based child health programs that is less costly than direct mortality measurement. These findings also support the use of LiST as a planning tool for choosing among child survival interventions in an attempt to maximize mortality impact in pursuit of MDG4.

Role of the funding source

Several authors (JR, DP, LR) have been associated with USAID’s Child Survival and Health Grants Program during manuscript preparation. Several others (MM, PE) were associated with World Relief, whose Vurhonga II project was funded by USAID through this mechanism. The study sponsors had no role in the study, data collection, or analysis. The corresponding author had final decision-making authority over interpretation of the results and the decision to submit this paper.