Can hospital audit teams identify case management problems, analyse their causes, identify and implement improvements? A cross-sectional process evaluation of obstetric near-miss case reviews in Benin

Abstract

Background

Obstetric near-miss case reviews are being promoted as a quality assurance intervention suitable for hospitals in low income countries. We introduced such reviews in five district, regional and national hospitals in Benin, West Africa. In a cross-sectional study we analysed the extent to which the hospital audit teams were able to identify case management problems (CMPs), analyse their causes, agree on solutions and put these solutions into practice.

Methods

We analysed case summaries, women’s interview transcripts and audit minutes produced by the audit teams for 67 meetings concerning one woman with near-miss complications each. We compared the proportion of CMPs identified by an external assessment team to the number found by the audit teams. For the latter, we described the CMP causes identified, solutions proposed and implemented by the audit teams.

Results

Audit meetings were conducted regularly and were well attended. Audit teams identified half of the 714 CMPs; they were more likely to find managerial ones (71%) than the ones relating to treatment (30%). Most identified CMPs were valid. Almost all causes of CMPs were plausible, but often too superficial to be of great value for directing remedial action. Audit teams suggested solutions, most of them promising ones, for 38% of the CMPs they had identified, but recorded their implementation only for a minority (8.5%).

Conclusions

The importance of following-up and documenting the implementation of solutions should be stressed in future audit interventions. Tools facilitating the follow-up should be made available. Near-miss case reviews hold promise, but their effectiveness to improve the quality of care sustainably and on a large scale still needs to be established.

Background

Maternal mortality in sub-Saharan Africa stagnates at a high level, its contribution to the global maternal death toll increasing from 23% to 52% between 1980 and 2008 [1]. Direct complications of pregnancy and childbirth contribute, in Africa, approximately 68% to this mortality [2]. Such complications usually require emergency obstetric care in district hospitals, where IV therapy, blood transfusion, general anaesthesia, caesarean section etc. are available [3]. Evidence suggests quality of care in hospitals often to be poor, and that women incur considerable delays before obtaining emergency treatment [4]. Reasons include lack of essential resources, sub-standard case management and unsatisfactory provider-patient interaction [5].

Audits may have potential to address case management problems (CMPs) in hospitals [6]. Audits come in various guises, but have in common that they seek to identify and analyse shortcomings systematically, and to design and implement interventions to prevent such shortcomings in the future [7]. They operate on the assumptions that shortcomings are learning opportunities and ought to be used as such, that health workers in general wish to perform well but need support for doing so, and that, except in cases of gross negligence or malevolence, encouragement and assistance are more effective than punishment.

In this paper we report the experience of introducing audits in hospitals in Benin, West Africa, and document the extent to which audit teams could identify CMPs, analyse their causes, agree on solutions and put these solutions into practice.

Methods

The audit intervention

In 1999 we introduced obstetric near-miss case reviews in five hospitals in Southern Benin. A near-miss case was defined as a woman suffering from a life-threatening complication during pregnancy or six weeks after delivery, whose “immediate survival is threatened and who survive by chance or because of the hospital care they receive” [8]. For each type of near-miss (uterine pre-rupture/rupture; obstetric haemorrhage; severe pre-eclampsia/eclampsia; sepsis; anaemia), a panel of national and international experts agreed on an operational definition (Table 1). In the absence of appropriate national guidelines, and with the relevant WHO manual [9] not yet published, the project developed guidelines on the management of major complications, and distributed them to participating hospitals.

Table 1 Definition of near-miss complications

In each hospital a core audit team was constituted, consisting of a senior physician involved in obstetric care and a senior midwife; both received a financial incentive for assisting the research activities. Staff members of maternity units and other relevant hospital departments (laboratory, pharmacy, anaesthesiology, intensive care, hospital administration) and from referring health centres were invited to attend audit meetings on a case-by-case basis, and received a small amount for transport and refreshments. Audit teams received an initial five days training on objectives and conduct of audits, and a three day refresher training after 12 months. All hospitals received a small financial contribution for their running costs.

Audit teams in each hospital were requested to identify near-miss cases continuously, choosing one per month for discussion during the monthly audit meeting. We advised the teams to choose particularly informative cases, making sure that the scope of major pathologies was represented over time. A staff member prepared a summary of the selected case’s management on the basis of clinical records. The hospital’s social worker conducted an in-depth interview with the woman, preferably at home, sometimes involving relatives, to obtain additional information from the woman’s perspective, possibly not apparent from clinical records.

The audit meeting was chaired by a member of the core audit team. Participants were reminded of the meeting’s objectives. The staff member who had prepared the case summary presented the case, and the social worker reported on the interview with the woman and her relatives. Subsequently the participants were invited to identify and discuss the positive aspects of the case management, and the CMPs. The discussion followed the course of events chronologically: from before admission to hospital to admission, diagnosis finding, near-miss treatment, further case management and discharge (“gate-to-gate” approach) [8]. Once all CMPs were thought to be identified, participants were invited to identify the cause which led to the CMP, and a solution which could prevent a similar CMP in the future. Audit teams were encouraged to discuss CMPs, causes and solutions that were within their sphere of influence. For each solution, teams were asked to nominate a staff member responsible for implementing it, to report during the next audit meeting whether this had been successful, and if not, to re-discuss the CMP. A secretary took notes and wrote structured minutes, recording positive aspects, CMPs, causes, solutions and responsibilities.

The type of audit we implemented could be labelled “hospital-based multi-professional patient-centred near-miss case review”. Choosing this type of audit was based on the following assumptions: “case review” - discussing authentic examples of severe obstetric complications interests and engages health workers; “hospital-based” - having the hospital teams carry out the audit themselves enhances their sense of ownership and their willingness to make an effort in implementing solutions; “multi-professional” - involving staff from different professions improves communication between them and thus the quality of care; “patient-centred” - interviewing women provides additional information and an opportunity to confront professionals with women’s experiences and views, a possible eye opener for the humane dimension of emergency obstetric care; “near-miss” - discussing complications that have been survived includes, by definition, positive aspects, thus facilitates frank discussions and bears less risk for depression or unhelpful attribution of blame. Since near-miss cases occur much more often than maternal deaths, reviewing them offers learning opportunities for staff more frequently.

Benin’s health system

Benin is a francophone West African country with a population of about 6 million. In 1999, the maternal mortality ratio was estimated at 498 maternal deaths per 100.000 live births [10]. Benin was divided into 34 health districts, of which 28 had a public or religious district hospital, in addition to six departmental and one national hospital. Specialised health professionals were few (1 obstetrician per 36,162, 1 midwife per 17,846 women in reproductive age) and concentrated in large cities. The use of maternal services was high: 78% of women delivered in a health facility. The high use of services combined with high maternal mortality suggests problems with the quality of care.

Participating hospitals

The five participating hospitals vary in size, ownership and position in the health system pyramid (Table 2): four government and one private non-profit hospitals; one national, two regional and two district hospitals; 1,000 to 6,000 deliveries per year, 4 to 14 midwives per 1000 deliveries, and deliveries by Caesarean section between 15% and 33%. The incidence of maternal near-miss morbidity and maternal mortality did not vary much across Hospitals 1, 2, 4 and 5 (7.6% to 10.4%, 700 to 1200 per 100,000 live births, respectively). The exception was Hospital 3, a regional referral hospital where the near-miss incidence reached 22.9% and the maternal mortality ratio 3,200 per 100,000 live births.

Table 2 Characteristics of hospitals

Hospitals performed between 7 and 17 audits between October 1999 and March 2001. Due to different local policies the percentage of audits moderated by a physician as opposed to a midwife varied from 6.3% to 100%.

Audit process evaluation

To carry out an audit, various documents were established: participants’ list, case summary with extracts from clinical records, summaries of the women’s interviews, and structured minutes of the audit meeting. For research, verbatim minutes were produced from audio-tapes of women’s interviews, and audit meetings were observed by social scientists, who did not intervene in the discussions but gave feedback at the end of the meeting and produced reports on their observations. Information was extracted from all these documents, and coded based on criteria as they appeared in the documents. Original clinical records were not available. The extracted information included: women’s age; obstetric history; date and time of the near-miss onset, admission, obstetric interventions, discharge, audit; type of referral and transport; type of near-miss complication and underlying cause; type of obstetric interventions; neonatal outcome if appropriate; woman’s satisfaction; number and profession of audit participants; and difficulties observed during the audit meeting.

Two external assessors (MB and TD, physician and obstetrician, respectively, both with experience in practising obstetrics in West Africa) used project guidelines, WHO guidelines [11] and occasionally a textbook [12] to decide what constituted a CMP. They independently identified CMPs from case summaries and women’s interview minutes. They extracted from minutes of the audit meetings the CMPs identified by the audit teams, assessed to what extent they were valid CMPs, and identified CMPs missed by the audit teams. Some clinical practices consistently deviated from guidelines without this ever being discussed by audit teams. It turned out that audit teams were unaware of the clinical standard, and could not be expected to identify and discuss these particular CMPs (Table 3); missing such CMPs was therefore “excused”. A complication caused or aggravated by inappropriate health seeking behaviour was not eligible as CMP, unless it happened after admission to hospital and staff’s failure to communicate effectively had apparently contributed to this behaviour. We categorised CMPs by type of CMP (diagnostic procedure missing, inadequate drug dosage, woman not informed etc.), by the timing or “gate-to-gate” phase in which they occurred, and by severity (i.e. the clinical significance for maternal survival or avoidance of long-term maternal morbidity). Categories for severity were “minor” (not directly relevant, e.g. dirty showers), “intermediate” (relevant, e.g. scarred uterus not diagnosed at admission, antibiotics underdosed) and “major” (immediate danger, e.g. failure to diagnose transverse lie, no rapid volume replacement for haemorrhagic shock). In case of disagreement, the assessors sought and mostly achieved consensus.

Table 3 Case management problems presumably due to standards unknown to audit teams*

From the minutes of the audit meetings one author (MB) assessed the performance of the audit team by constructing a score based on marks for the following achievements: 20 marks for ascertaining a valid CMP, regardless of whether the external assessors had identified the CMP from case summary or interview minutes; 10 marks each for analysing the causes of this CMP in a plausible way, in appropriate depth (profound enough to facilitate finding a solution vs. too superficial to be helpful), and taking into account their realm of influence; 10 marks each for agreeing on solutions that were plausible, addressed the problem in appropriate depth, and were within reach; 10 marks for assigning responsibility to one team member; 10 marks for reportedly having put the solution into practice or re-discussing it on the next audit meeting. The audit performance score was then computed per audit meeting as a percentage of achieved marks over achievable marks, thus taking into account that the achievable marks varied with the number of valid CMPs per meeting.

We analysed the performance of audit teams by individual CMP and by audit meeting. When analysing the performance by individual CMP, we either insisted on a valid analysis of the cause, or were satisfied when an appropriate solution was proposed regardless whether a valid cause had been identified. We used Chi²-tests or Fisher exact tests for comparing proportions as appropriate, and F-tests for comparing means. We used multiple linear regression and t-tests to verify the independence of determinants of audit team performance.

Ethics approval

The ethics committee of the London School of Hygiene and Tropical Medicine approved the study (12 November 1997, no. 470), and the Ministry of Health in Benin gave permission to carry it out. Hospital managers were approached and agreed to let their hospitals participate in the study.

Results

Patients, complications, treatments

During the study period, 77 near-miss cases were reviewed. The analysis is restricted to 67 cases, for which case summary and audit minutes could be retrieved. The age of these women ranged from 16 to 46 (median 28); 18% were primigravidae, 18% grand multipara.

Seventy percent of the women had entered the hospital as emergency, either referred by another health facility (56%) or on their own initiative (14%, Table 4). The most important emergency transport means were private cars (59%), only 16% arrived by ambulance. The most common near-miss complications were haemorrhage (42%), followed by hypertensive disorders (25%), obstructed labour (19%), and infections (9%). Two thirds of the patients received whole blood transfusions, one third colloid fluids (Table 5). Fourteen out of 17 patients with near-miss hypertensive disorders received anticonvulsive drugs, 16 antihypertensive drugs. Of 41 women not delivered when the near-miss situation began, 63% were delivered by caesarean section or laparatomy, none by ventouse or forceps. 16% of all women had a hysterectomy. Fifty-three percent of the children were born alive. A third was born healthy, another third showed signs of intrapartum suffering leading to neonatal asphyxia (17%) or resulting in fresh stillbirth (15%).

Table 4 Characteristics of obstetric complications

Table 5 Management of obstetric complications

Audits

Audits took place on average one month after admission (Table 6). On average, meetings lasted 1.5 hours and had 15 participants. A third of the participants were health workers not involved in the clinical management of the reviewed case. In virtually all meetings, physicians, midwives, nurses and social workers were present, while laboratory technicians participated in half, and administrative staff in a quarter of the audits. Half of the audits were chaired by a physician, the others by a midwife or nurse. Most case presentations were done by a midwife or nurse (85% vs. 15% by a doctor). Discussions were lively and in general followed the recommended steps. The most frequent problems observed during the audit meetings were heated arguments between participants (45%), poor moderation (39%), inattentive (25%) or dominant participants (19%); poor recall of case management details was rare (3%).

Table 6 Characteristics of audit meetings

Factors facilitating patients’ survival

The most important facilitators for women’s survival identified by audit teams were those linked to human resources (95%), with commitment, willingness and availability (73%) ranking before competence (67%) and good teamwork (9%). Financial resources came second (73%), with patients having funds available (57%) mentioned more often than hospital funds for destitute patients (16%); in three cases, staff reported having paid for patients from their own pockets. Material resources ranked third (54%) and included the availability of blood (12%) and drugs (10%) from emergency kits in the hospital (trousse d’urgence) or in pharmacies with 24/24 opening hours. Other facilitating factors were clinical guidelines (19%), timely referral (19%) and good collaboration with and understanding by the women’s families (16%).

Case management problems

During 67 audit meetings, the audit teams identified 413 case management problems (CMPs), of which the external assessors found 376 (91.0%) to be valid; invalid CMPs included women not attending antenatal care, families not providing moral support to the woman, and health workers using outdated medical terminology for the diagnosis. As mentioned before, audit teams consistently missed certain CMPs, being presumably unaware of the relevant clinical standard; these CMPs were exempt from further analysis (Table 3). The external assessors identified 338 CMPs not detected by the audit teams and not exempt. Further analysis of CMPs is carried out on the total of 714 CMPs.

Problems in the treatment of near-miss complications contributed most to the total of 714 CMPs (35%, Table 7), with delayed treatment reported in 11%, necessary drugs not given in 5.5%, unnecessary drugs given in 3.9% and therapeutic procedures not carried out satisfactorily in 3.5% of CMPs. Problems relating to diagnosis or follow-up were identified in 24% of CMPs, with insufficient monitoring in 9.0% and unsatisfactory physical examination in 5.9% of CMPs. Managerial problems were reported in 19% of CMPs, with insufficient documentation in 7.7% and unsatisfactory financial arrangements in 2.7% of CMPs. Unsatisfactory interaction with the patient constituted 14% of the CMPs, with insufficient provision of information (6.6%) leading before lack of friendliness and moral support (4.6%).

Table 7 Types of case-management problems, in detail

Audit teams identified 53% of all 714 CMPs classified as valid by the external assessors (Table 8). Audit teams were particularly proficient in identifying CMPs related to referral (88%) and hospital management (71%), and weak in identifying shortcomings in treatment of the near-miss complication (30%, p<0.0005). The capability of audit teams to identify CMPs did not vary with severity of the CMP (p=0.3).

Table 8 Type, timing and severity of valid case management problems

Audit teams were able to attribute causes for 83% of 354 CMPs they had identified (Table 9). Most causes were considered to be plausible (98%); an implausible cause was, for instance, the lack of attending ANC in the first trimester as a cause for the uterine rupture following a face presentation. While about 40% of the plausible causes were judged to be profound, i.e. to touch on the root cause of the CMP, almost 60% of the causes were judged as too superficial to be helpful in identifying an appropriate solution; examples include causes like “staff is negligent, staff has an inappropriate attitude”. Hospital teams were assumed to have some control over most plausible causes (90%); causes beyond reach included the lack of an ambulance at another health facility, or the assertion that “patient lacks funds”. Less than a quarter of all 376 valid CMPs identified by the audit teams had their causes appropriately analysed (found to be plausible, profound and within reach). Identification of appropriate causes worked better for CMPs related to near-miss treatment and worse for CMPs related to establishing the diagnosis and interaction with the women and their families, but not significantly so (p=0.071); the causal analysis of CMPs became more appropriate with increasing severity (p=0.001) and passed the test in 41% of the most severe cases.

Table 9 Appropriateness of CMP cause identified by audit team, by type and severity of CMP

Audit teams recorded their suggested solutions for 202 (54%) of the CMPs they had identified (Table 10). Almost two thirds (129/202, 64%, data not shown) related to the way work was organised, most frequently by improving clinical documentation (12%), interaction with patients (11%), feed-back provided to referring health facilities (7%), and collaboration within the obstetrics team and with other hospital services (5%). While 18% of the solutions called for additional resources, mainly for training (8%) or improved availability of consumables (5%), only 16% of the solutions directly required changes in the medical procedures, for examination and monitoring (12%) or treatment (4%).

Table 10 Appropriateness of CMP solution suggested by audit team, by type and severity of CMP

Almost all solutions were plausible (Table 10). Three quarters of the 197 plausible solutions were judged to be profound; solutions that failed this test included those specifying an objective but not how it could be reached: “improve communication with laboratory”, “respect treatment protocol” etc. Virtually all 197 plausible solutions were within the reach of the hospital teams who ran the audits. In total, for 38% of the 376 valid CMPs identified by the audit team appropriate solutions were proposed. Audit teams performed better in proposing appropriate solutions for managerial CMPs and worse for CMPs relating to the diagnosis (p=0.002). Audit teams were also better, albeit not significantly so (p=0.078), in proposing appropriate solutions for the most severe CMPs (52%).

Audit teams recorded having identified a staff member as responsible to implement the solution for 37% of all solutions; this proportion did not vary by type of CMP (p=0.5) nor by severity (p=0.6). We have found documentation on whether or not solutions have been implemented for only 97 of 202 CMPs with proposed solutions (48%). For 44% of these, solutions have been reported to have been implemented, without variation between types and severity of CMP (Table 10).

A comprehensive definition for handling a CMP overall appropriately requires that the CMP is recognised as such, and a valid cause identified, and an appropriate solution proposed and implemented by the audit team. For a total of 714 CMPs we found evidence in 376 (53%) that they were recognised, in 90 CMPs (13%) that valid causes had also been identified, in 55 CMPs (7.7%) that also appropriate solutions had been proposed and in 13 CMPs (1.8%) that these solutions had been implemented (Figure 1). Using only the CMPs recognised by audit teams as denominator, the percentage of CMPs with valid causes is 25%, and of CMPs for which solutions also have been proposed and implemented is 15% and 3.5%, respectively.

Figure 1

Performance of near-miss audits, requiring that a valid cause* has been identified for each CMP. *For a cause to be valid it had to be plausible, profound and within reach of the audit team. **For a solution to be appropriate it had to be plausible, profound and within reach of the audit team.

A less challenging definition for overall appropriate handling requires that a CMP is recognised by the audit team, and an appropriate solution proposed and implemented regardless whether or not valid causes have been identified. Following this definition, for a total of 714 CMPs we found evidence in 143 CMPs (20%) that appropriate solutions have been proposed, and in 32 CMPs implemented (4.5%, Figure 2). Making the optimistic assumption that the proportion of implemented solutions (44%) is the same in CMPs with and without documentation on implementation, the proportion of CMPs for which an appropriate solution has supposedly been implemented increases to 10%. Taking only those CMPs recognised by the audit teams as denominator, the percentages for CMPs with appropriate solutions proposed and implemented are 38% and 8.5%, respectively. Making the same optimistic assumption that the proportion of CMPs with an implemented solution is identical for CMPs with and without documentation about implementation, the proportion of recognised CMPs with a supposedly implemented solution amounts to 19%.

Figure 2

Performance of near-miss audits, not requiring that a valid cause* has been identified for each CMP. *For a cause to be valid it had to be plausible, profound and within reach of the audit team. **For a solution to be appropriate it had to be plausible, profound and within reach of the audit team.

We found the performance of audit teams to vary between hospitals (p=0.028, Table 11), but not by type of near-miss (p=0.7), nor that it improved over time (p=0.5). Teams moderated by a physician performed significantly better than those moderated by a midwife (p=0.030). This association persisted in multiple linear regression after controlling for type of near-miss and project maturation (t-test, p=0.028), but not when the hospital ID was added to the model instead (t-test, p=0.6). The effect of the moderator’s profession can therefore not be disentangled from other characteristics of the hospital.

Table 11 Determinants of the performance of audit teams

Discussion

After introducing near-miss case reviews in five hospitals in Benin, we performed a process evaluation to assess to what extent audit teams were able perform near-miss case reviews according to the rules, that is, to identify CMPs and underlying causes, propose solutions, define responsibilities and document the implementation of solutions. Our study attempts to assess neither enhancements of quality of care, nor improvements of health outcomes.

Hospital teams readily adopted the audit intervention, organised meetings regularly and with appropriate frequency, and participated in large numbers – meetings were rather too large than too small, possibly resulting from incentives offered to participants, albeit modest (3 EUR). To preserve a frank exchange of views in large hospitals it may be necessary to restrict participation to those who were directly involved in the audited case’s management, and to brief the rest of the team on proposed solutions separately. The composition of participants was appropriately multi-professional. Midwives and nurses played crucial roles in the meetings by often moderating them and virtually always presenting the case summary. Not surprisingly, there was an important potential for conflict in these meetings, which underlines the importance of the moderators’ skills [13].

Audit participants acknowledged the overwhelming importance of health workers’ attitudes and competence for a positive patient outcome. Health workers are aware that available resources allow providing life saving obstetric care, if used by motivated and skilled health workers. The importance of financial arrangements for the poor and of the availability of blood and emergency drugs was acknowledged, which points to possible targets of interventions to improve the capability of Benin’s hospitals to respond to obstetric emergencies.

Most CMPs discussed by the audit teams merited auditing. Worrisome, however, is the high proportion of CMPs related to the treatment of complications (70%) that was missed by the audit teams. It appears that audit teams were more able or willing to discuss managerial shortcomings than medical ones. As most treatment decisions are either taken or approved by physicians, it is possible that the majority of participants did not feel competent or entitled to challenge these decisions. This may be an inherent limitation of internal case reviews when carried out in district hospitals with few physicians involved in emergency obstetric care; possibly the involvement of a higher level, e.g. the obstetrician from the regional referral hospital, could help, but one would have to look out for any threatening or intimidating effect this may have on the audit team. The lack of detailed evidence-based guidelines on the management of obstetric complications at the time of the study has probably contributed to the under-representation of therapeutic CMPs in the audit meetings. In the meanwhile WHO has published such guidelines [11] in several languages (e.g. Arabic, English, French, Indonesian, Russian and Spanish), so that case reviews should now use these or equivalent guidelines. The guidelines may also raise the awareness in Benin that certain practices (Table 3) are no longer in line with international recommendations. The emerging consensus on near-miss case definitions may help developing curricula on how to diagnose and manage life-threatening obstetric complications [14].

Many causes recorded by the audit teams appeared to be too superficial to be of value for identifying the most appropriate solution: to note that something has not been done because of “negligence” without analysing where motivational problems come from hardly points into the direction remedial action should take. It is possible that the discussion had more substance than what has been recorded in the minutes. In any case the “why? why?” root cause analysis described by Weeks et al. [15] may constitute an improvement over the audit practice described here: it involves questioning the CMP’s cause that comes first to mind in order to get to the root cause of the problem. It is encouraging, however, that audit teams in general accepted responsibility for the CMP, and rarely located its cause outside their sphere of influence.

Audit teams proposed solutions for only half of the CMPs they identified themselves. It is not clear whether this represents an oversight, a decision to focus on some CMPs and not to seek solutions for others, or a failed attempt to find and agree on solutions. In any case, the high proportion of CMPs identified but not tackled by the audit teams is worrying. As reported from neighbouring Burkina Faso [16], many solutions addressed organisational issues, fewer medical procedures, and very few treatment, possibly for the same reasons why audit teams missed many CMPs related to treatment in the first place. It is reassuring that most solutions were plausible, directed towards the CMP’s root cause, and within reach of the audit team; many did not appear to require significant additional resources.

A sobering finding is the low proportion of CMPs with implemented solutions. Solutions were at best implemented for 19.3% of the CMPs, while the proportion of CMPs for which implemented solutions have been documented was 8.5% only. We may have underestimated this proportion by not following up for long enough; in Burkina Faso it was found that it may take several months to implement a solution [16]. However, this alone is unlikely to explain the considerable gap between proposed and implemented solutions. Future audit trainings should underline the importance of implementation and follow-up. Tools should be used that facilitate the follow-up of agreed solutions, for instance an audit register with the columns “problem | why | why | solution proposed | responsible | deadline | follow-up”. If the follow-up reveals that solutions have not been implemented by the agreed deadline, the audit team needs to reconsider the course of action. Possibly, incentives should be tied to the implementation of solutions, and not to the participation in audit meetings. If action does not follow the talking, audits risk degenerating into a fruitless palaver which bears few prospects to improve quality of care.

The number of participating hospitals (N=5) was too small for a meaningful analysis of hospital factors determining the variable performance of audit teams. We have found that audit teams perform better if moderated by physicians, which may be due to their more profound medical knowledge and their greater authority, but we cannot rule out confounding by other hospital factors we were unable to elicit.

The variety of case reviews carried out in Benin included interviews with patients and their relatives, whose results were than reported back to the audit team. Whether these interviews had an overall engaging or disruptive effect on the discussions of the audit team remains unclear and merits further research. A considerable variation in quality, from rich to superficial, from encouraging and respectful to confrontational and judgemental, transpired from the interview transcripts. This demonstrates that training and supervising staff carrying out these interviews, e.g. social workers, is essential.

Another open question is whether near-miss case reviews should complement or replace maternal death reviews. The exclusion of maternal deaths from case reviews is likely to facilitate the audit process, as the question of guilt is less burning when the woman has survived. Certain shortcomings in case management, e.g. unacceptable delays, may only be recognisable from maternal deaths but not from near-miss cases; if maternal deaths concentrate during certain times of the day or the week this may point at underlying causes, which will be missed by excluding maternal deaths from review. Operational research should investigate whether experienced case review teams can cope with the additional challenge to review a maternal death and remain faithful to the principles of frankness and non-accusation.

Finally, the question of sustainability of audits arises. Often, while there is special funding, audits tend to be implemented and conducted, but without that funding audits are less likely to happen. Once the effectiveness of audits has been demonstrated hospital management, with the support of the Ministry of Health, should advocate that they are an internal quality improvement technique and as such part of the staff’s duty, and not an optional activity that needs to be paid extra. When Hutchinson et al. found that all hospitals participating in this study had stopped doing audits within two years after completing this feasibility study, lack of resources was one, but not the most prominent reason [17]. Support by Ministries of Health has increased recently, and many national plans now include audits of maternal deaths as a compulsory activity to improve quality of care.

Limitations of the study

This study has several limitations. Most importantly, no primary data like clinical records or tape recordings of patient interviews or audit meetings were available; the data sources were limited to documents produced by health workers to prepare and document the audits. We are likely to have missed some CMPs if they have not transpired from case summaries or patient interview transcripts. We may also not have fully captured the audit teams’ reasoning on causes of and solutions for CMPs if it is not reflected in the audit minutes. We may have missed solutions that were implemented but not documented. Because of the difficulty to distinguish between shortcomings in the audits’ conduct and weaknesses in their documentation, their quantitative assessment should be interpreted with caution, but we have confidence in the principal patterns that have emerged.

The presence of researchers as observers, i.e. social scientists and epidemiologists, in many audit sessions may have positively influenced the way audits were conducted, but audits were still far from perfect and allowed many shortcomings to be identified.

It may be argued that the observation time was too short, and that audit teams would have learned to perform audits better over time. However, our data do not support this assumption, and we doubt that, without further training, audit teams would by themselves have been able to improve significantly.

Because of the purposive sampling of audit cases by the audit teams, selected cases do not constitute a representative sample of all near-miss cases treated in the participating hospitals. The audit teams may have chosen patients with particularly poor case management to maximise learning, or those who were relatively well looked after to limit embarrassment. No conclusion can be drawn on the relative frequency of type of near-miss, admission, CMP etc. in those hospitals. However, the validity of our analysis of how well the audit teams dealt with the cases they have selected is not compromised by this non-representativity of the sample.

We have not studied hospital environment or team dynamics in detail. It is therefore difficult to conclude to which extent the results are specific to these five hospitals or can be generalised to hospitals in similar settings. We have some confidence in the external validity of our results, as the participating hospitals represent various levels of the health system pyramid and both public and private sectors.

The data of this study are now 10–12 years old. However, we have continued working on the quality of obstetric care in this setting and believe that the results are still relevant for current efforts to implement audits.

Conclusions

Quality assurance approaches based on near-miss morbidity remain highly topical [18]. Near-miss case reviews have the potential to improve the quality of managing life-threatening obstetric complications in Benin’s hospitals, a setting where resources are scarce and must be used with consideration. The reviews were popular with staff and carried out regularly, and the audit teams were able to analyse shortcomings and make suggestions as to how they can be avoided in future. While the principle of such reviews is deceptively simple, they are not a quick fix, and their implementation needs thorough training and follow-up. Ultimately, their capacity to improve the quality of care in settings similar to the one studied here still needs to be demonstrated – something the ongoing AUDOBEM trial (ISRCTN67206260) attempts to do.