Open Access Research article

Attempt to increase the transparency of fourth hurdle implementation in Central-Eastern European middle income countries: publication of the critical appraisal methodology

András Inotai1*, Márta Pékli2, Gabriella Jóna2, Orsolya Nagy2, Edit Remák3 and Zoltán Kaló45

Author Affiliations

1 University Pharmacy, Department of Pharmacy Administration, Semmelweis University, Hőgyes E. u. 7-9, Budapest, 1092, Hungary

2 Office of Health Technology Assessment, National Institute for Quality- and Organizational Development in Healthcare and Medicines, Diós árok 3, Budapest, 1125, Hungary

3 United BioSource Corporation, Bég u 3-5, Budapest, 1022, Hungary

4 Health Economics Research Centre, Eötvös Loránd Science University, Pázmány P. 1a, Budapest, 1117, Hungary

5 Syreon Research Institute, Thököly út 119, Budapest, 1146, Hungary

For all author emails, please log on.

BMC Health Services Research 2012, 12:332  doi:10.1186/1472-6963-12-332

Published: 21 September 2012



In middle income countries the number of trained health technology assessment specialists is limited and the public budget for health technology assessment is considerably lower compared to developed countries. These countries therefore must develop their own solutions to improve the quality and efficiency of health technology assessment implementation in reimbursement decisions. Our study aimed to develop a scientifically rigorous and detailed appraisal checklist for economic evaluations of pharmaceuticals in the single health technology assessment process.


The research design entailed a review of economic evaluations, submitted for reimbursement of pharmaceuticals, by two independent academic reviewers to identify the most common methodological problems. Fifty economic evaluations submitted in 2007-2008, randomly selected by the Health Technology Assessment Office served as data sources. The new checklist was developed by an iterative working process: first by assessing ten economic evaluations, then improving the checklist by generating new question items, then employing the improved checklist to assess the next ten economic evaluations. After appraising 25 documents, the reviewers reconciled their opinions and improved the checklist with the researchers of the Health Technology Assessment Office during an expert panel discussion. The reviewers scrutinized the second 25 economic evaluations, after which the expert panel finalized the checklist with consensus.


The final checklist consists of 91 yes or no questions in 11 main topics concerning comparator selection, efficacy, effectiveness, costs, sensitivity analysis, methodological approach, transparency, and interpretation of results. The new checklist is based on current Hungarian evaluation practice. As the published checklist will be part of the official single health technology assessment process of pharmaceuticals, submitters will be able to assure the quality of their economic evaluation.


The transparent critical appraisal method should improve the consistency of pharmaceutical reimbursement decisions and facilitate the utilization of economic evaluations in other fields of health care decision-making in other Central-Eastern European countries.