- Research article
- Open access
- Published:
Core indicators evaluation of effectiveness of HIV-AIDS preventive-control programmes carried out by nongovernmental organizations. A mixed method study
BMC Health Services Research volume 11, Article number: 176 (2011)
Abstract
Background
The number of nongovernmental organizations working on AIDS has grown. There is great diversity in the type of activities and population groups that have been targeted. The purposes of this study are: to describe and analyze the objectives and HIV-AIDS preventive activities that are carried out by the AIDS-NGOs that work with AIDS in Catalonia and that receive subsidies from the Department of Health; and to develop a comprehensive proposal for measurable and agreed upon core quality evaluation indicators to monitor and assess those objectives and activities that can have an impact on the fight against inequalities and stigmatization, and incorporate the perspectives of the service providers and users.
Methods
A mixed method study has been carried out with professionals from the 36 NGOs that work with HIV/AIDS in Catalonia, as well as their users. This study achieved the completeness model using the following phases:
1. A systematic review of AIDS-NGOs annual reports and preparation of a catalogue of activities grouped by objectives, level of prevention and AIDS-NGOs target population; 2. A transversal study through an ad-hoc questionnaire administered to the AIDS-NGOs representatives; 3. A qualitative study with a phenomenological approach through focus groups, individual interviews and observations; 4. Consensus meetings between AIDS-NGOs professionals and the research team using Haddon matrices in order to establish a proposal of evaluation indicators.
Results
The information was classified according to level of prevention and level of intervention. A total of 248 objectives and 258 prevention activities were identified. 1564 evaluation indicators, addressed to 7 target population groups, were produced. Thirty core activities were selected. The evaluation indicators proposed for these activities were: 76 indicators for 15 primary prevention activities, 43 for 5 secondary prevention activities and 68 for 10 tertiary prevention activities.
Conclusions
The results could help to homogeneously assess the preventive-control activities carried out of AIDS-NGOs. The proposed indicators could help the professionals to improve the evaluation of the preventive-control AIDS-NGOs activities. Furthermore, the Haddon matrix enables us to identify deficiencies of activities at intervention levels and strategies to bear in mind in order to enhance the future AIDS prevention programs.
Background
The 2001 Declaration of Commitment on HIV/AIDS and the 2006 Political Declaration on HIV/AIDS adopted by the United Nations General Assembly are the guiding forces of the global response to AIDS. The process of preparing country progress reports should involve all partners in AIDS response and should provide an opportunity for reflection on the national response, its achievements as well as its shortcomings, in order to reach universal goals. Monitoring the response to the AIDS epidemic is essential to ensure that investments in AIDS achieve the expected results in health and society [1].
HIV-related Non Governmental Organizations (AIDS-NGOs) work to promote prevention and to provide care and help to people affected by HIV in most countries of the world. The role of these AIDS-NGOs has been critical in the fight against HIV infection since the onset of the epidemic [2]. The activities of the AIDS-NGOs are important both around the world and in Catalonia (one of 17 Autonomous Communities in the North-East of Spain with 7,364,078 inhabitants in 2008) [3]. These activities are complementary and are often leaders of public and private health systems in various countries. AIDS-NGOs have lead the initiative against HIV and act as a bridge and "communications space" between the most vulnerable sectors of the population (because of socioeconomic inequalities) and health services [4]. AIDS-NGOs are the largest providers of preventive activities against HIV/AIDS, particularly amongst high-risk behavior groups: commercial sex workers, injecting drug users, men who have sex with men, youths in high-risk situations, prisoners and immigrants and persons living with HIV/AIDS [5, 6]. Furthermore, they have wide experience of working at the community level, and their autonomous nature allows them to respond more quickly and employ innovative methods [7].
Since the inception of the AIDS National Program in Catalonia, the promotion and coordination of activities carried out by AIDS-NGOs has been emphasized. To support the AIDS-NGOs, since 1992 the Department of Health has provided annual subsidies that have progressively increased to reach 1.4 million Euros in 2007 [8–10]. There has been an increase in the prevention, health promotion and care activities of AIDS-NGOs [6, 11].
The number of AIDS-NGOs and their activities has grown to meet the demand and the needs of individuals in preventive materials, determination of serostatus and HIV care. For this reason there is great diversity in the type of activities, as well as in the social levels and population groups that have been targeted.
In Spain, Peiró et al. systematically classified and evaluated the activities of national AIDS programs, including AIDS-NGOs activities using the Haddon matrix. This study concluded that there is much heterogeneity in the structure and indicators of the various programs. Concepts such as goals, objectives and priorities are poorly defined. Moreover, in all countries the implementation of preventive programs shows a predominance of individual interventions over social ones [12].
A more recent report, the Health Plan of the Community of Valencia 2005-2009 [13] includes a description of objectives and lines of action for HIV prevention based on the Haddon matrix axes [14–16]. Other autonomous communities in Spain such as the Canary Islands and Asturias have also provided a basis for developing evaluation indicators for all activities [17, 18]. In Spain, the "AIDS National Programme" of the Ministry of Health addresses the development of preventive programs for HIV and other sexually transmitted infections in individuals that work as commercial sex workers. The plan suggests both qualitative and quantitative evaluation indicators for these activities [9].
Moreover, the methodology that has been proposed to compare the different ways to prioritize issues and to forecast a policy's application in terms of results and outcomes is the Haddon Matrix. This matrix is a way to synthesizes the enormous amount of information available in strategy documents, shorten the time needed to produce an assessment, and improve a policy's value by comparing it with external models. In this paper we use this theoretical framework developed with the explicit intention of covering the need of the AIDS National Program of Catalonia to know what preventive and control interventions are done by AIDS-NGOs with the objective of improving the praxis of these activities and establishing new strategies for the AIDS National program [12]. Furthermore, a proposal of measurable indicators to monitor and assess each activity done by these organizations will be developed.
It is important to provide key constituents who are actively involved in a country's response to AIDS with essential information on core indicators that measure the effectiveness of the national response. The indicators will help to ensure the consistency and transparency of the process used by national governments. Increasingly, countries are strongly encouraged to integrate the core indicators into their ongoing monitoring and evaluation activities [19].
The key motivation for undertaking this work is to propose quality indicators which include users' satisfaction, psychosocial aspects that can have an impact on the fight against inequalities and stigmatization, and the perspectives of professional service providers. Until now, in many evaluation proposals, these aspects were not really addressed in the HIV literature, while their consideration would permit an important step toward the incorporation of strategies for evaluation of NGO prevention and more comprehensive control activities.
The purposes of this study are: to describe and analyze the objectives of and the HIV-AIDS preventive activities carried out by the AIDS-NGOs that work with AIDS in Catalonia and that receive subsidies from the Department of Health; and to develop a more comprehensive set of measurable and agreed upon core quality evaluation indicators to monitor and assess those objectives and activities that can have an impact on health and the fight against inequalities and stigmatization, and incorporate the perspectives of service providers and users.
Methods
A mixed method strategy was used in order to achieve the objectives of the study. The way to combine quantitative and qualitative research was by completeness, in order to achieve more complete and comprehensive answers to research questions. The strategy was divided into four phases and there was full data integration of each phase (each new phase includes the findings of previous phases) [20–23] (Figure 1).
Methodological Strategy.
Phase 1-Systematic and exhaustive review of the AIDS-NGOs annual reports
A systematic reading and review of the 36 annual reports from the all AIDS-NGOs working in Catalonia and which received subsidies from the Catalan Government in 2006. The annual reports of AIDS-NGO were provided by the AIDS National Program exclusively for this study.
From this systematic reading, we found objectives and activities of prevention were classified according to the following criteria [12, 15, 16]: AIDS-NGOs target population (commercial sex workers, injecting drug users, men who have sex with men, youths in high risk situations and persons living with HIV/AIDS), levels of prevention (primary, secondary and tertiary), and levels of intervention (individuals, the individual's immediate environment, the external environment and the social environment). This classification was the first step in constructing Haddon matrices. The Haddon matrix offers a conceptual framework that facilitates the analysis of health problems, taking into consideration a three dimensional grid: on the x-axis, the time dimension: primary prevention, secondary prevention and tertiary prevention; on the y-axis, the setting of the intervention: individual factors, immediate environment, external environment and a fourth level that includes the social system and the current regulations. The third axis or dimension includes actions designed to decrease gender, age, ethnic, socio-economic and cultural inequalities [12, 15, 16].
From this synthesis, a database relating the objectives and activities described in the AIDS-NGOs annual reports was created.
What new information does this methodological phase of the study provide?
This phase provides new basic information on the objectives of and the activities carried out by the AIDS-NGOs as described in the annual reports these organisations prepare for the Health Department. A Haddon matrix has also been constructed with the activities classified by strategy, level of prevention, and level of intervention.
Phase 2-Transversal study
An ad-hoc questionnaire was constructed in order to identify activities and objectives classified according to AIDS-NGOs target populations that were not explicit in the AIDS-NGOs annual reports.
A pilot study was carried out in 2008, using a cognitive interview which allowed assessment of the understanding and applicability of the questionnaire [24, 25].
The questionnaire was self-administered and it was sent by mail to all the 36 AIDS-NGOs. It consisted of 15 questions, (5 open questions, 7 closed questions and 3 multiple choice questions with non-mutually exclusive answers) and included the followed sections: (1) Profile of the individual responding to the questionnaire, (2) Information regarding the AIDS-NGOs target population, (3) Objectives of the AIDS-NGOs and preventive activities of the program, (4) Preventive activities carried out by the AIDS-NGOs, (5) Agreement to participate in the future consensus meetings between AIDS-NGOs professionals and the research team to establish an evaluation indicators proposal. In each section, there was one open question to express opinions and experiences that were not covered by the questionnaire.
To facilitate the highest percentage of responses, two rounds of confirmatory phone calls were conducted and a descriptive analysis (absolute frequencies and percentages) was performed.
What new information does this methodological phase of the study provide?
The results of the transverse study complement those of the first phase of the elaboration of the Haddon matrix based on the systematic study of the reports. In this methodological phase we have also been able to establish a relationship between objectives and activities.
Phase 3-Qualitative study of phenomenological perspective
A qualitative study was conducted in order to identify the experiences that professionals in the Catalan AIDS-NGOs had in preventive activities, the potential areas of improvement of the activities, the professionals' evaluation of their activities, the experiences and practices of AIDS-NGOs with regard to HIV infection and prevention, and elements that enhanced their relationship with the AIDS-NGOs.
A theoretical sampling (professional) and opportunistic sample (participants) was proposed. Professionals and users from 36 AIDS-NGO funded by the Health Department to take part in the study. For the professionals, a theoretical sampling based on the previous definition of the characteristics of the participants has been carried out to obtain the greatest variety and discursive wealth to reach data saturation. Variables used to define the informant profile for the professionals of AIDS-NGO are: AIDS-NGO target population (commercial sex workers, injecting drug users, men who have sex with men, youths in high-risk situations, prisoners and immigrants and persons living with HIV/AIDS), age, sex, professional profile, setting (urban or rural) and years of experience.
Due to the difficulties of the theoretical sampling, an opportunistic sampling was finally chosen for the AIDS-NGO users. However, heterogeneity criteria were taken into account. Variables used to define the users are: AIDS-NGO target population, age, sex, nationality, serostatus, and time in contact with the NGO [26].
Different techniques have been used [27]. For the professionals, focus group [28] and triangular groups [29]. In the focus groups, interaction is the instrument to stimulate the individual speech [30]. For the users, semi-structured interviews (because during the interviews some sensitive issues may arise) [30] and open, focused and non-systematic observation of theatre performances in teenagers and young adults have been carried out [31].
Data collection was between February and June 2008. Focus groups took place in a neutral place (IDIAP Jordi Gol) and they included a moderator and an observer. The semi-structured interviews took place in the working place of the users or at the AIDS-NGO places. The observations were made in two secondary schools.
A thematic interpretive content analysis was conducted by three analysts. More details about the methodology of this phase have been published elsewhere [32].
What new information does this methodological phase of the study provide?
This methodological phase of the study has permitted a deeper knowledge of how AIDS-NGO professionals carry out their activities. It has also allowed exploration of the barriers professionals encounter in carrying out evaluation. Users also took part in this methodological phase of the study, in which their knowledge and perception of HIV-AIDS risks was explored.
Phase 4-Consensus meetings between NGO professionals and the research team
From the synthesis of the information obtained in phases 1, 2 and 3, the Haddon matrices described in phase 1 were completed. Later, consensus meetings were held between five AIDS-NGOs professionals and four members of the research team, in order to review and redevelop the Haddon matrices and to identify new preventive activities when possible.
The AIDS-NGOs professional selection was carried out considering the inclusion of different professional profiles, work with different AIDS-NGOs target groups, years of antiquity and the geographical setting.
From the Haddon matrices, a proposal of qualitative and quantitative evaluation indicators was developed, both for process and outcomes of activities carried out by AIDS-NGOs. The structure and methodology that was followed in order to develop the indicators proposal is shown in tables 1 and 2 [33].
Finally, core indicators were selected according to the following criteria: (i) relevance of the indicator (ii) adequacy of the indicator in the AIDS-NGOs context (iii) ability of AIDS-NGOs to change the indicator outcome (iv) feasibility of obtaining the information needed to construct the indicator and (v) activity common to all AIDS-NGOs target group [34, 35].
What new information does this new methodological phase of the study provide?
This new methodological phase, through consensus meetings between AIDS-NGO professionals and the IDIAP Jordi Gol research team, has allowed the elaboration of a Haddon matrix for each target group as well as evaluation indicators for each activity carried out by the AIDS-NGOs with regard to the objective of this activity.
Ethical aspects
The study has been conducted according to Guidelines of the Helsinki Declaration of Good Clinical Research Practice. The project has been approved by the Ethical and Clinical Research Committee of the Institute of Research in Primary Health Care (IDIAP) Jordi Gol.
Informed Consent: The information for the study has been provided orally as well as in writing. Study subjects had sufficient opportunity to ask questions regarding details of the study. Written informed consent following the guidelines of the Helsinki Declaration was obtained.
Data confidentiality: Confidentiality and anonymity of the data have been ensured according to the Spanish law 15/1999 of data confidentiality, both in the implementation phase of the project and in presentations or publications resulting there from. Individual data was encoded to ensure anonymity. Only researchers and monitors have access to the data.
Results
Phase 1-Systematic and exhaustive review of the annual AIDS-NGOs reports
The objectives and activities identified using the review of the reports were aimed at 5 target populations established by the AIDS Program of Catalonia (commercial sex workers, injecting drug users, men who have sex with men, youth in high risk situations, persons living with HIV/AIDS), and 2 more identified target population established for this study (general population and women). Consequently, objectives and activities were collected into seven Haddon matrices.
248 objectives were detected, 118 for Primary Prevention, 44 for Secondary Prevention and 86 for Tertiary Prevention (Figure 2). A total of 242 activities were identified in relation to the objectives in this phase. Some activities were similar or repeated for different objectives and different levels of prevention (Figure 2).
Flow chart from the total objectives and evaluation indicators to core objectives and evaluation indicators.
What new results were obtained?
In this phase we know the number of objectives and activities carried out by the AIDS-NGOs and the relationship between them. We also observe that many activities are common to different target groups, that different activities share the same objective, and vice versa.
Phase 2-Transversal study
The total response to the questionnaires was 72% (26/36). Women accounted for 57.7% of the respondents.
The AIDS-NGO carried out activities for different target population. It was seen that 2/26 of the AIDS-NGOs target their activities to 7 different target groups, 12/26 to 4, 5 or 6 and 11/26 to 2 or 3 target groups.
Regarding the position of the person answering the questionnaire, "coordinator of HIV prevention activities" was the most frequent answer (15/26), followed by "person with most administrative knowledge of the AIDS-NGOs" (11/26). The category "others" was also frequently chosen (9/26). This last category included administrative personnel, managers and individuals responsible for specific programs within the AIDS-NGOs.
Regarding the question about what professionals work in AIDS-NGOs, the answer was that they have interdisciplinary teams, psychologists (64%), social workers (31%), doctors (27%), educators (23%), nurses (17%), and affected volunteers (20%) being the most frequent professional profiles.
What new results were obtained?
The number of identified activities increased to 258 in total, while seventeen activities (6.6%) that were not covered in the annual reports were detected. Three were especially addressed at the primary prevention level (activities that encourage closer ties with NGO services, activities that promote development leisure and interaction activities, legal advice), 3 at secondary prevention (viral load tests, risk pregnancy tests and subsequent referral, detection of tuberculosis and clinical follow-up), 5 for tertiary prevention (individualized psychological care to prison inmates, delivery of diapers to HIV+ families, websites and chats for seropositive individuals, forums and e-mail consultations and at home health care) and 6 activities present at the different levels of prevention (coordination with Spanish, European and African entities that work with HIV+ infants, adolescents and youth, guidelines to improve the doctor-patient relationship, participation in research projects, training courses and continuous education).
Phase 3. A qualitative study with a phenomenological perspective
The main results achieved through the qualitative study were: AIDS-NGOs offered different activities adapted to the needs of the users (people who use AIDS-NGOs' services); The NGO professionals perceived that users were satisfied. The users were satisfied and felt comfortable with the education and health promotion model; the preventive activities of the AIDS-NGOs were based on a participatory health education model adjusted to the people's needs, based on empowerment and focusing on the ideas of knowledge and skills; AIDS-NGOs professionals worked in interdisciplinary groups using a holistic approach with cultural and professional competence; they were flexible, innovative and motivated; and they traveled to where the users are.
Regarding the evaluation of the AIDS-NGOs objectives and activities, the main conclusions were: evaluating the work of AIDS-NGOs is difficult (a concern voiced in most group meetings); some activities require great effort and dedication and quantitative indicators such as the number of cases are not enough to evaluate them; to improve the assessment of the activities and objectives of the AIDS-NGOs, the professionals suggest looking for quantitative and qualitative indicators that provide a reliable measure of the process and the results. More details about results in this phase have been published elsewhere [34].
What new information do the results of this study provide?
AIDS-NGO professionals carry out their activities on a holistic basis, maintaining confidentiality, with professional competence and cultural awareness and respecting the principle of equality. The relationship between AIDS-NGO professionals and users is based on attention, a holistic focus (both socially and in health terms), and user needs.
Users often apply to NGOs for help not only with their basic needs (food, shelter and hygiene) but also for legal aid and assistance with employment and health. Activities aimed at the prevention of HIV-AIDS infection are more effective if they take these basic needs into account. NGOs carry out specific activities aimed at promoting the accessibility of and user links with these associations. NGO professionals form flexible, motivated and innovative interdisciplinary teams ready to go wherever their users are. They require continued training and coordination between the different fields.
NGO professionals recognise the need for an adequate system for the collection of information in order to facilitate information management. They also emphasise the need to support and improve the planning and execution of the current process of the evaluation of objectives and activities. NGO professionals cite perception of a high level of satisfaction among users with regard to the attention received, as well as a lack of recognition of their activities by other professional and institutional groups.
Phase 4. Consensus meetings between the research team and the NGO professionals
Seven Haddon matrices were developed with the results achieved in the three previous phases. In total, 1564 evaluation indicators, 720/1564 of primary prevention, 255/1564 of secondary prevention and 589/1564 of tertiary prevention were proposed (Figure 2). All the information containing the core objectives, activities and indicators are shown in detail in Additional File 1. All the indicators have been published elsewhere [36].
Finally, from all the activities and all the evaluation indicators proposed, the research team selected the core objectives and indicators based on the criteria specified above [34] as shown in table 3 and in Figure 2. Thirty core activities were undertaken: 15 for primary prevention activities (76 indicators), 5 for secondary prevention (43 indicators) and 10 for tertiary prevention (68 indicators) (Table 4).
An example of the core preventive activities and proposed evaluation indicators are described in tables 5, 6 and 7 for each level of prevention. Primary prevention indicators are mainly related to the use of barrier methods for preventing HIV infection and to the increased knowledge of HIV and Sexually Transmitted Infections (STI) (Table 5). Secondary prevention indicators are related to early detection of HIV, STI and opportunistic infections (Table 6). Tertiary prevention indicators are related with comprehensive counselling to achieve effective reintegration of individuals infected with HIV/AIDS (Table 7).
All the core evaluation indicators proposed can be seen in Additional Files 1.
Activities were carried out to achieve all of the proposed objectives by the various AIDS-NGO, although there were some objectives covered by more than one activity.
Discussion
The present study followed a mixed method strategy that contributes to improving knowledge of activities undertaken by AIDS-NGOs and serves as a basis from which to develop a final proposal of indicators to assess these prevention-control based activities. As shown in the results, this methodological strategy implies that the different phases complement one another and progressively richer information is obtained [20, 37].
The results allowed us to know all the preventive-control based activities done by AIDS-NGOs in Catalonia and how they could be assessed. The first two phases showed us a relationship between objectives and activities, and what other activities were not registered in the AIDS-NGOs annual reports. The qualitative phase showed how the preventive-control activities are done and what the main barriers and limitations to evaluate it are. Finally, the indicators consensus meetings enabled us to talk with AIDS-NGOs professionals and establish a consensus on the indicators proposed to evaluate the activities in the daily work flow.
The preventive activities of the AIDS-NGO are based on a participatory health education model adjusted to the people's needs, based on empowerment and focusing on the ideas of knowledge and skill. This education strategy follows the principles put forward by the WHO [38].
Results emphasise the importance of offering ancillary services [39] to people in need of HIV-AIDS prevention or treatment. Another study showed that the holistic person-based approach is essential to achieve a change in behaviour [40]. On the other hand, the WHO Report 2008 [38] underlines the impact of person-based care in health improvement, quality of life, user trust and treatment adherence.
In this relationship model, the professional takes into account the values and perspective of the user, and therefore incorporates them in the decision-making process [41]. The applicability of specific programmes requires a community approach to adjust them to match the needs of the target population [39, 42].
The results confirm that the AIDS-NGOs perform most of their activities in relation to target groups at risk of social exclusion or because they are socially vulnerable. Moreover, it reduces social inequalities due to socioeconomic status, gender and social orientation. It also confirms the role of NGO-AIDS as a bridge acting as a "communications space" between health services and the population) and also with other services (legal, social, employment, etc.).
Evaluation is a continuous process that facilitates the identification of areas for improvement. It should also contribute to the recognition of tasks that have been satisfactorily carried out by professionals. The professionals belonging to the participating AIDS-NGOs of this study consider evaluation to be an activity which is relevant to them and share the evaluation needs. Therefore, there is concordance between the necessities detected by the HIV-AIDS program (Public Health Department) and those identified through a revision of international publications [1, 43–45].
In order to evaluate the objectives and activities of AIDS-NGOs, it is essential to use good indicators bringing together characteristics such as: acceptability, feasibility, reliability, sensitivity to change, validity and meaningful and possible communicability [19, 46].
The latest update of the "Compendium of Evidence-Based HIV Prevention Interventions", elaborated by the CDC, as well as the "Community guide" website, both show a broad range of evidence-based preventive activities. However, these activities are included in programs aimed at very specific groups (female condom skills training; many men, many voices; personalized cognitive risk-reduction counseling), while our study presents a broader evaluation and suggests multiple indicators to evaluate concrete activities, which can be applicable to diverse intervention programs and to different groups. Thus, this proposal of indicators can help provide new evidence about the effectiveness of specific activities according to the level of prevention and intervention [47].
Through the development of the first phase of this study, two more target groups of population were identified: inmates and immigrants. These two target populations had specific characteristics deserving of special consideration and their own preventive programs on the part of the AIDS-NGOs, and they were added as target populations for purposes of this study. Specific objectives and activities addressed to these groups were identified, and specific evaluation indicators proposed through the seven final Haddon matrices, available via the following link (http://www.gencat.cat/salut/depsalut/html/ca/dir2068/informefinal_actongsida2009.pdf) [36].
Our proposal agrees with some of the quality measures of HIV care proposed by the CDC and UNAIDS [1, 45, 47]. What is new and constitutes a strong point in our study is that we have added other quality indicators that measure user satisfaction (an essential aspect of the evaluation of service quality) and more psychosocial factors (legal advice) that can have an impact on the fight against inequalities and stigmatization [6, 48].
Another important aspect to note is that, by using the Haddon matrix, many preventive interventions and evaluation indicators have been identified and defined on the individual, external and social levels, while less have been directed towards the individual's immediate environment. These findings are consistent with those observed by Peiró et al [12]. A possible interpretation of this phenomenon is the concern of AIDS-NGOs professionals about data confidentiality; in order to act in the immediate environment of the affected person, he or she must agree to share the problem with individuals in his or her intimate environment.
The consensus phase among AIDS-NGOs professionals and the research team to develop the indicators has meant the cooperation and participation of the professionals in this study and constitutes a key element in facilitating the acceptance and implementation of the improvements proposed in the evaluation. It is also worth mentioning that professionals belonging to the AIDS-NGOs that participated in the consensus phase make up a multidisciplinary team much like the AIDS-NGOs. This gives the proposed indicators wide variability and richness, encompassing all of the activities undertaken by the AIDS-NGOs [34, 35, 45].
The main limitation of this proposal of indicators is that it has not been applied. It would be important to share and disseminate it among professionals of the various programs and organizations that work with HIV in order to test its applicability, usefulness, validity and practical relevance.
A possible selection bias could also be argued, given that these indicators have been designed with the participation of professionals from AIDS-NGOs that work in Catalonia and are financed by this autonomous community's Department of Health. Nonetheless, as mentioned above, the enormous variability in the number of activities and proposed evaluation indicators minimizes this bias and raises their potential usefulness for AIDS-NGOs located in other geographical areas. Furthermore, one limitation of the study was not to have studied the views of other health providers that work with HIV-AIDS prevention or control-programmes (Primary Care, Public Health, Reproductive and Sexual Health, Faith-Based Organizations and Community Based Organizations). Although their contributions would have been of great interest, for reasons of practicality and insufficient resources this aspect could not be included.
We also want to mention that indicators for global results such as HIV mortality, rate of hospitalizations and approach to other health problems (diabetes, hypertension...) have not been proposed. However, the improvements in these epidemiological indicators can be attributed to multiple factors (improvements in security and effectiveness of drug treatments, healthcare, etc.) and not only to the AIDS-NGOs' activities. Nonetheless, studies that directly involve classic systems of epidemiological surveillance for HIV-AIDS would be of interest.
For the selection of indicators in future evaluations of preventive activities in AIDS-NGOs, the elements that make up the third dimension of the Haddon matrix should be taken into consideration: effectiveness, cost, freedom, equity, stigmatization, preferences of the affected community or individual and viability. These factors can be considered from different points of view and can have greater or lesser weight when selecting indicators, based on the general objectives of future health policies and on HIV-AIDS [12, 15, 16].
It is also important that the strategic plan of future HIV-AIDS policies emphasize activities that address the population that makes up the affected or at-risk person's immediate environment [12]. Although activities directed towards the social environment can have an indirect impact and their execution can be more difficult in some cases, it is necessary to emphasize the key role of the AIDS-NGOs in the development of these activities and the important social impact they can have.
Conclusions
This study showed the importance of methodological combination for a better understanding of results and the mechanisms of changes in the evaluation of AIDS-NGO activities. Although the study had some limitations, the combination of quantitative and qualitative data gave the opportunity to construct core evaluation activities and the corresponding indicators. The results could help to homogeneously assess the preventive-control activities carried out by several AIDS-NGOs. Moreover, the indicators could help the professionals to understand and therefore to improve the evaluation of the preventive-control AIDS-NGO activities. The Haddon matrix enables us to detect the gaps in intervention levels and, as a consequence, to keep in mind strategies that could enhance AIDS programs in the future. A further important message of the research was that qualitative approaches are essential in order to identify underlying processes and to detect activity procedures in order to better define the indicators.
References
UNAIDS: Monitoring the Declaration of Commitment of HIV/AIDS: Guidelines on construction of core indicators: 2010 reporting. 2010, Geneva: UNAIDS
UNAIDS: 2008. Report on the global AIDS epidemic. 2009
Anuari estadístic de Catalunya. 2010 Població. 1900-2010. Províncies. 2011, Sèrie temporal. Institut d'Estadística de Catalunya Ed. Jule 2010 Barcelona, ISSN: 1130-166X. Available in http://www.idescat.cat/pub/?id=aec&n=245
de SR: Creating "communicative spaces": a case of NGO community organizing for HIV/AIDS prevention. Health Commun. 2009, 24: 692-702. 10.1080/10410230903264006.
Crane SF, Carswell JW: A review and assessment of non-governmental organization-based STD/AIDS education and prevention projects for marginalized groups. Health Educ Res. 1992, 7: 175-194. 10.1093/her/7.2.175.
Kelly JA, Somlai AM, Benotsch EG, Amirkhanian YA, Fernandez MI, Stevenson LY, Sitzler CA, McAuliffe TL, Brown KD, Opgenorth KM: Programmes, resources, and needs of HIV-prevention nongovernmental organizations (NGOs) in Africa, Central/Eastern Europe and Central Asia, Latin America and the Caribbean. AIDS Care. 2006, 18: 12-21. 10.1080/09540120500101757.
Sehgal PN: Prevention and control of AIDS: the role of NGOs. Health Millions. 1991, 17: 31-33.
Generalitat de Catalunya: Pla de salut de Catalunya a l'horitzó 2010. Els 5 eixos estratègics que articulen les polítiques de salut. Servei del Pla de Salut. 2010, Departament de salut. 2007, Accessible a: http://www.gencat.cat
Plan Nacional sobre el Sida, Grupo de trabajo sobre prevención del VIH/Sida y otras ITS en personas que ejercen la prostitución: Prevención del VIH/Sida y otras infecciones de transmisión sexual en personas que ejercen la prostitución: Elementos clave para el desarrollo de programas. 2003, Ministerio de Sanidad y Consumo, 2003. Madrid
RESOLUCIÓ SLT/1948/2007, de 21 de juny per la qual s'adjudiquen les subvencions per finançar activitats de prevenció de la infecció pel VIH/Sida a entitats de caràcter no lucratiu corresponents a l'any 2007. 2010, (DOGC núm. 4915 de 29/6/2007, pàg. 21947)
Sullivan PS, Hamouda O, Delpech V, Geduld JE, Prejean J, Semaille C, Kaldor J, Folch C, Op de Coul E, Marcus U, Hughes G, Archibald CP, Cazein F, McDonald A, Casabona J, van Sighem A, Fenton KA: Reemergence of the HIV epidemic among men who have sex with men in North America, Western Europe, and Australia, 1996-2005. Ann Epidemiol. 2009, 19: 423-431. 10.1016/j.annepidem.2009.03.004.
Peiró R, Alvarez-Dardet C, Plasencia A, Borrell C, Colomer C, Moya C, Pasarín MI, Zafra E: Rapid appraisal methodology for 'health for all' policy formulation analysis. Health Policy. 2002, 62: 309-328. 10.1016/S0168-8510(02)00053-2.
Generalitat Valenciana: Plan de Salud de la Comunidad Valenciana 2005-2009. 2010, Conselleria de Salut. 2006, ISBN: 84-482-4365-X
Barnett DJ, Balicer RD, Blodgett D, Fews AL, Parker CL, Links JM: The application of the Haddon matrix to public health readiness and response planning. Environ Health Perspect. 2005, 113: 561-566.
Runyan CW: Using the Haddon matrix: introducing the third dimension. Inj Prev. 1998, 4: 302-307. 10.1136/ip.4.4.302.
Runyan CW: Introduction: back to the future--revisiting Haddon's conceptualization of injury epidemiology and prevention. Epidemiol Rev. 2003, 25: 60-64. 10.1093/epirev/mxg005.
Dirección General de Salud Pública y Participación: Evaluación del Programa de Prevención y Atención a las Personas Afectadas por el VIH-sida en Asturias 2003-2008. 2010, Consejería de Salud y Servicios Sanitarios. Asturias, 2009
García Ruiz M, Suárez Álvarez O: Programa de Educación Sexual y Prevención del VIH en Centros de Menores. Experiencias en Asturias y País Vasco. 2000, Consejería de Salud y Servicios Sanitarios del principado de Astúrias. edn
NHS Institute for Innovation and improvement: The good indicators guide: Understanding how to use and choose indicators. 2011, Coventry: 2008
Bryman A: Mixed methods research: combining quantitative and qualitative research. In Social Research Methods. 2011, Oxford: Oxford University Press; 2008
Lingard L, Albert M, Levinson W: Grounded theory, mixed methods, and action research. BMJ. 2008, 337: a567-10.1136/bmj.39602.690162.47.
O'Cathain A, Murphy E, Nicholl J: Three techniques for integrating data in mixed methods studies. BMJ. 2010, 341: c4587-10.1136/bmj.c4587.
Sandelowski M: Combining qualitative and quantitative sampling, data collection, and analysis techniques in mixed-method studies. Res Nurs Health. 2000, 23: 246-255. 10.1002/1098-240X(200006)23:3<246::AID-NUR9>3.0.CO;2-H.
Herdman M, Fox-Rushby J, Badia X: A model of equivalence in the cultural adaptation of HRQoL instruments: the universalist approach. Qual Life Res. 1998, 7: 323-335. 10.1023/A:1008846618880.
Valderas JM, Ferrer M, Alonso J: [Health-related quality of life instrumentsand other patient-reported outcomes]. Med Clin (Barc). 2005, 56-60.
Tuckett AG: Qualitative research sampling: the very real complexities. Nurse Res. 2004, 12: 47-61.
Tong A, Sainsbury P, Craig J: Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care. 2007, 19: 349-357. 10.1093/intqhc/mzm042.
Krueger R, Casey MA: Focus Group: A Practical Guide for Applied Research. 2011, London: Sage Publications; 2000
Conde F: Los grupos triangulares como espacios transicionales para la producción discursiva. La vivienda en Huelva. Culturas e identidades urbanas. 1996, Sevilla: Junta de Andalucia-Fundación El Monte, 275-307. 2011
Bryman A: Social Research Methods. 2008, Oxford: Oxford University Press; 2008, 3
Gilbert N: Researching Social life. 2011, London: Sage Publications; 2001, 2
Berenguera A, Pujol-Ribera E, Violan C, Romaguera A, Mansilla R, Giménez A, et al: Experiences about HIV-AIDS preventive-control activities. Discourses from non-governmental organizations professionals and users: qualitative study. Gac Sanit. 2011, 25 (3): 184-190. 10.1016/j.gaceta.2010.10.015.
Asch SM, McGlynn EA, Hogan MM, Hayward RA, Shekelle P, Rubenstein L, et al: Comparison of quality of care for patients in the Veterans Health Administration and patients in a national sample. Ann Intern Med. 2004, 141: 938-945.
Plaza TA, Guarga RA, Farres QJ, Zara YC: [Consensus on a process of benchmarking in primary care in Barcelona]. Aten Primaria. 2005, 35: 130-139. 10.1157/13071938.
Plaza TA, Zara YC, Guarga RA, Farres QJ: [Results of a benchmarking exercise for primary care teams in Barcelona, Spain]. Aten Primaria. 2005, 35: 122-127. 10.1157/13071936.
Almeda J, Berenguera A, Pujol-Ribera E, Romaguera A, Violan C: Proposta d'indicadors d'avaluació de les activitats de les ONG-SIDA. 2008
Popay J: Qualitative research and the epidemiological imagination: a vital relationship. Gac Sanit. 2003, 17 (Suppl 3): 58-63.
World Health Organization: The World Health Report 2008 - primary Health Care (Now More Than Ever). 2008
Conviser R: Catalyzing system changes to make HIV care more accessible. J Health Care Poor Underserved. 2007, 18: 224-243.
Estrada JH: Prevention models in the fight against HIV/AIDS. Acta Bioethica. 2006, 12: 91-100.
Ruiz MR, Rodriguez JJ, Epstein R: [What style of consultation with my patients should I adopt? Practical reflections on the doctor-patient relationship]. Aten Primaria. 2003, 32: 594-602.
Rhodes SD, Hergenrather KC, Wilkin A, Alegria-Ortega J, Montano J: Preventing HIV infection among young immigrant Latino men: results from focus groups using community-based participatory research. J Natl Med Assoc. 2006, 98: 564-573.
de Souza EM: Evaluation methods in health promotion programmes: the description of a triangulation in Brazil. Cien Saude Colet. 2010, 15: 2521-2532. 10.1590/S1413-81232010000500026.
Duber HC, Coates TJ, Szekeras G, Kaji AH, Lewis RJ: Is there an association between PEPFAR funding and improvement in national health indicators in Africa? A retrospective study. J Int AIDS Soc. 2010, 13: 21-10.1186/1758-2652-13-21.
Wilson IB, Landon BE, Hirschhorn LR, McInnes K, Ding L, Marsden PV, et al: Quality of HIV care provided by nurse practitioners, physician assistants, and physicians. Ann Intern Med. 2005, 143: 729-736.
Campbell SM, Braspenning J, Hutchinson A, Marshall MN: Research methods used in developing and applying quality indicators in primary care. BMJ. 2003, 326: 816-819. 10.1136/bmj.326.7393.816.
CDC 2009 Compendium of Evidence-Based Interventions. 2009, [http://www.cdc.gov/hiv/topics/research/prs/]
Kelly JA, Somlai AM, Benotsch EG, McAuliffe TL, Amirkhanian YA, Brown KD, et al: Distance communication transfer of HIV prevention interventions to service providers. Science. 2004, 305: 1953-1955. 10.1126/science.1100733.
Pre-publication history
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6963/11/176/prepub
Acknowledgements
This study has been funded by the AIDS Programme, Public Health Department of the Ministry of Health, Government of Catalonia. We would like to thank all the AIDS-NGO professionals, and in particularly Kati Zaragoza, Anna Rafael, Ricardo Fuertes, Alba Mestre and Anna Lara, who took part in the consensus meetings. We specially acknowledge all users who have taken part in the qualitative study. We thank the IDIAP Jordi Gol for the funding of the study and translation of the manuscript into English.
Author information
Authors and Affiliations
Corresponding author
Additional information
Competing interests
The authors declare that they have no competing interests.
Authors' contributions
AB, EPR, CV, AR, RM, AG and JA have been involved in writing the manuscript, and all authors critically revised and approved the final manuscript. AB, EPR, CV, AR and JA designed the methodology and contributed to the description of the background. AB, EPR, CV, AR, CA and JA have done the fieldwork and the analysis and interpretation of data. All authors have participated sufficiently in the work to take public responsibility for appropriate portions of the content.
Electronic supplementary material
12913_2011_1691_MOESM1_ESM.DOC
Additional file 1: Complete information about core objectives, activities and indicators. NGO: Nongovernmental organizations. STI: Sexual Transmitted Diseases. QOL: Quality of life questionnaire. HAART: Highly Active Antiretroviral treatment. DO: Direct Observation. FG: Focus Groups. (DOC 474 KB)
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.
Rights and permissions
Open Access This article is published under license to BioMed Central Ltd. This is an Open Access article is distributed under the terms of the Creative Commons Attribution License ( https://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
About this article
Cite this article
Berenguera, A., Pujol-Ribera, E., Violan, C. et al. Core indicators evaluation of effectiveness of HIV-AIDS preventive-control programmes carried out by nongovernmental organizations. A mixed method study. BMC Health Serv Res 11, 176 (2011). https://doi.org/10.1186/1472-6963-11-176
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/1472-6963-11-176