An economic evaluation is a comparison of outcomes among alternative ways of achieving common objectives. These analyses are conducted according to explicit, systematic, and consistent criteria, and take into account both the positive and negative consequences of each alternative. Consequences may include economic, clinical, and humanistic outcomes, known as the ECHO model 11. Economic outcomes represent the consumption and production of resources and their monetary value from the perspective of a decision maker. Clinical outcomes are medical events that are professionally meaningful. Humanistic outcomes are a broad category of intangible personal attributes, typically collected through self-report. Humanistic outcomes include quality of life characteristics such as sense of safety, physical comfort, enjoyment, meaningful activity, relationships, functional competence, dignity, privacy, individuality, autonomy, and spiritual well-being. Conventionally, clinical and humanistic outcomes are considered health outcomes, and we follow this convention for the remainder of the article.

There are several forms of economic evaluations that can be performed (cost-effectiveness analysis being only one of these) and each differs based on the selection and measurement of health outcomes. The perspective (or point of view) taken for the analysis also influences the selection and measurement of consequences, because not all outcomes are important to all decision makers. Generally, there are three perspectives for economic analysis: individual (eg, patient), institutional (eg, health maintenance organization), or societal. The societal perspective accumulates all outcomes, while individual and institutional analyses are more selective. Regardless of perspective, the objective of an economic evaluation is to provide information on consequences relating to alternatives faced by a decision maker.

The most basic form of economic evaluation is a table that lists the individual economic and health outcomes of alternative interventions. This table is known as a cost-consequence study. Cost-identification studies and cost-minimization analyses only address economic outcomes and are discussed below in that section. The remaining forms of economic evaluations summarize economic and health outcomes into a single result (Table 1).

The advantages of performing cost-benefit and cost-utility analyses are that multiple outcomes are summarized into a single unit, either monetary units such as dollars (CBA) or QALYs (CUA) and that therapies with different sets of health outcomes can be compared based on the differences in the summary measures. Cost-benefit analysis has the additional benefit of directly indicating whether the therapy pays for itself.

The disadvantages of CBA and CUA come from the techniques required to produce a summary measure. Cost-benefit analysis requires putting a monetary value on all health outcomes (and ultimately on life), and cost-utility analysis assigns value to health outcomes based on their contribution to quality of life under the presumption of population-based preferences. An extensive literature addresses the methodological and theoretical issues involved in the construction of these summary measures. The process usually occurs in two steps. In the first step, health outcomes of the intervention are measured, and in the second the outcomes are valued in summary units and aggregated. Cost-benefit analyses often assess the monetary value of health outcomes based on willingness-to-pay using a technique called conjoint analysis 121314. Willingness-to-pay inherently places a lower values of life on individuals with low income, because they can not pay what they do not have. Cost-utility analyses have multiple methods to place quality of life values on health outcomes, also known as social tariffs. Summary measures of quality of life may not be sensitive enough to pick up short-term changes such as for acute conditions and will not pick up specific clinical outcomes like blood pressure control 15. Examples of instruments used to capture these general health states include the EuroQoL (EQ-5D) 16 and the Health Utilities Index 17.

Cost-effectiveness analysis (CEA) is the current standard in the literature, and has the most straight forward interpretation. Under CEA, therapies useful for a specific disease or condition can be directly compared using a metric of effectiveness relevant to that condition, such as blood pressure control. Although these types of analyses do not allow a summary of multiple outcomes they tend to respond well to the most urgent questions, such as how much would it cost to reduce the number of gestational diabetes cases by 10%? Clearly, a reduction in gestational diabetes cases has measurable implications in quality of life and economic units, but the creation of a summary measure is not necessary to address the decision maker's question.

No matter the approach taken, it is recommended that the estimated outcomes (economic, clinical and humanistic) of health care alternatives used in economic evaluation are best estimated in pragmatic clinical trials that directly and realistically compare the therapies of interest 10. Rarely are the results of placebo-controlled trials appropriate 14181920. Also, since many CAM therapies target chronic disease, it is important that the study period be long enough to capture the full benefits and costs of each therapy, and that future costs and benefits be discounted to the present for comparison. Finally, all economic evaluations should include some type of sensitivity analysis to test the robustness of results to the various assumptions made 12021.

Economic outcomes are the net bundle of resources forgone due to an intervention valued at the opportunity cost of those resources (the value of their next best use or "opportunity"). Since the cost of a therapy differs depending on whether you are a patient, a health plan, or a health care provider, the economic outcomes (ie, costs) of each therapy depend on the perspective of the study. Studies that only measure the economic outcomes of interventions are known as cost-identification studies. A study that describes the economic and health outcomes of a single therapy can also be called a cost-identification study. These studies inform full economic evaluations. That is, they provide the data needed to better design future studies that consider both the economic and health outcomes of two or more alternative therapies. A cost-minimization analysis (CMA) explicitly assumes equivalence in health outcome among alternative therapies, and examines only economic outcomes. In practice, it appears the same as a cost-identification study, but under the assumption of equivalence, a CMA is a full economic evaluation.

Table 2 has been summarized from other references 12022 and gives a list of the types of economic outcomes and the perspective of analysis where each is considered. Note that these types of economic outcomes should be inclusive of both the full costs of the therapy and of any treatment for adverse effects, which can be expensive. In economic evaluations, the safety of a therapy is addressed through accounting for the cost of treating these adverse events as well as through their impact on clinical and quality of life outcomes.

It is recommended that economic outcome data are best collected prospectively as part of a pragmatic clinical trial 141920. Inclusion and exclusion criteria for cost data should be established in the protocol, as for clinical outcome measurements, but provision must be made to add extra categories of costs which only become apparent after the trial has commenced 120. Many studies try to collect cost data retrospectively, often after a therapy has shown clinical effectiveness. However, retrospective data collection is seldom fertile, adapted, or exhaustive, and it is subject to bias 1820.

Our systematic review of the CAM economic evaluation literature (presented below) revealed no cost-consequence studies and no cost-benefit analyses. However, we did find examples of a cost-identification study, cost-minimization analysis, cost-effectiveness analysis, and cost-utility analysis. These examples are presented below.

We searched the following electronic databases from January 1999 to October 2004: Medline, AMED, Alt-HealthWatch, and the Complementary and Alternative Medicine Citation Index via NCCAM and the National Library of Medicine (NLM). Searching was restricted to English language journals and human studies with the keywords: complementary medicine or alternative medicine, and costs or cost analysis or cost-benefit or cost-effective or economic analysis or economic evaluation.

We removed duplicates from the search results and selected papers that reported original data on specific CAM therapies from any form of standard economic analysis, analysis of costs, or economic modeling. Studies were then excluded if they were cited in the White and Ernst review 4, or if they were case studies or case series of five or fewer subjects.

The following data were extracted from each of the included studies: full citation information (author(s), date, title, journal, etc), form of economic evaluation (stated or inferred), the therapies being compared and whether the CAM therapies were being used in addition to usual care (complementary) or instead of usual care (alternative), the perspective of analysis (stated or inferred), the study design, the sample size, and summary results.

The studies were categorized as either full economic evaluations (defined as a comparison between two or more alternatives and considering both costs and consequences 10) or partial economic evaluations (those studies that did not contain a comparison, or only addressed costs). Studies that estimated resource utilization were included as full economic evaluations even if resources were not valued.

We captured data on quality of the full economic evaluations using two approaches. The first approach was to gather from each study the data needed to assess quality according to a 35-item checklist developed by the BMJ Economic Evaluation Working Party 30. This checklist was developed to improve the quality of published economic evaluations, and was chosen because it is thorough, and entails an objective assessment of whether essential components of an economic evaluation are reported in the article. Therefore, the checklist is mainly a measure of reporting quality and not necessarily of study quality. We also report available results from several other general reviews of economic evaluations of conventional therapies that use this checklist for comparison.

As the purpose of economic evaluations is to inform clinical practice and health policy decisions, the best evaluations are timely and use the best data available at the time 10. On the other hand, an evaluation is only as good as the data upon which it is based. It has been suggested that the ideal situation for data collection is to collect economic data along side health outcomes in a randomized pragmatic trial 10. Pragmatic trials offer a compromise between the goals of internal and external validity. To assess study quality, we went beyond White and Ernst's 4 criteria of randomization (to reduce bias by creating comparable groups) and prospective collection of economic outcome data (to ensure all costs are captured) to include two additional indicators of whether a pragmatic (effectiveness or "real world") rather than efficacy trial was conducted. The first is that the comparison group was usual care, and the second was that the study was not blinded and not mandatory – ie, that physicians and patients could react realistically to the therapy 10. These criteria relate to the external validity or generalizability of the study. Other indicators of a study's generalizability, such as the determination of whether study participants could be assumed to represent a normal case load, were not used as they required detailed knowledge as to the appropriateness of the inclusion and exclusion criteria for each condition studied – a level of expertise not held by the study's authors.

Based on the study quality criteria, we report summarized results of the exemplary studies – ie, those meeting all four study quality criteria. If the health outcomes for one therapy are better than that of its alternative and the economic outcomes are better or equal (lower or equal costs), that therapy is said to dominate (be clearly better than) its alternative. This is also the case if both therapies have equal health outcomes and one has lower costs. In all other cases, the decision maker must elect whether the increase (loss) in health benefits is worth the increase (savings) in cost.

Table 4 shows the results of the application of the BMJ 35-item quality checklist 30 to the 39 full economic evaluations. For comparison, Table 4 also contains comparable results from systematic reviews in conventional medicine 68384.

Twenty-seven studies (69%) gathered cost data prospectively and 21 (54%) used randomly assigned comparison groups. In 32 studies (82%) the physicians and patients were not blinded to the treatment received and participation was not mandatory (a worksite intervention 57), and therapies were compared to usual care in 34 (87%) of studies. Fourteen studies 2527323435505153545568747682 met all study quality criteria, and a summary of their results is shown in Table 5.

A possible explanation for the paucity of studies is that there may be less of an incentive to perform economic evaluations of CAM. Consumers are already spending a large amount of their disposable income on CAM without formal proof of effectiveness or cost effectiveness. Economic evaluations are typically required for the incorporation of therapies under traditional financing mechanisms and for adjustment of coverage under these mechanisms. Therefore, the market for economic evaluation in CAM may be small due to reduced involvement of third-party payers in CAM financing.

Some CAM practitioners do not see the need for economic evaluations. An interesting study by Kelner et al 85 asked chiropractors, homeopaths, and Reiki practitioners about the need to demonstrate the effectiveness, safety and cost effectiveness of their therapies. The chiropractors agreed that high quality economic evaluations are essential to their practice, but Reiki practitioners could see no reason for this research, and the homeopaths were divided on these issues. There may be good reason why some practitioners resist economic evaluation. If studies are performed that show economic benefit of CAM therapies, third party reimbursement may follow which could reduce practitioner autonomy. Coverage may also be restricted to the standardized forms of botanical medicines, nutritional supplements, or protocols used in the studies 86. This could dramatically change how CAM is practiced by decreasing the use of multidimensional multicomponent interventions, by institutionalizing care into conventional health care systems, and by limiting the individualization of care.

The reporting quality was poor for certain items, but was comparable to the quality found by systematic reviews of economic evaluations in conventional medicine 68384. Although the BMJ checklist was mostly objective (ie, required the least amount of judgment compared to the other checklists available), a fair amount of interpretation was still required for many items. For example, in our review we interpreted Item 1 as whether the study stated either a specific research question or study objectives in terms of economic and health outcomes. Three-quarters of the full economic evaluations of CAM met this criterion. However, in Jefferson et al, 1998 83, only 16% of the 43 economic evaluations of conventional medicine reviewed where identified as fulfilling Item 1. It is likely that Jefferson et al took a more restrictive interpretation of this quality criterion.

Several studies have shown that at least some aspects of quality in economic evaluations improve over time 687. Our findings suggest a trend of quality improvement in these studies in CAM. We found that 69% (27 of 39) of the cohort of full economic evaluations collected cost data prospectively as compared to 45% (5 of 11) in White and Ernst's review. Similarly, we found that 54% (21 of 39) of our studies used randomization to create the comparison groups as compared to 45% (5 of 11) in White and Ernst's review.

We found that 14 (36%) of full economic evaluations met all four study quality criteria and were identified as exemplars. However, the evidence from these criteria must be interpreted cautiously; meeting all study quality criteria does not guarantee an adequate study design. Some aspects of what makes a good pragmatic trial could not be judged by what was reported. For example, pragmatic trials enroll patients typical of normal caseload in typical settings with average physicians following them under routine conditions 10. Judgments as to whether these criteria were met were not possible because of vague reporting. It is also not generally agreed across all health economists that a pragmatic trial, even a well-designed one, can fully represent the real world of health care. These economists advocate for the collection of cost data using an observational study design.

A study may also be of "poor" quality because it applied the CAM therapy inappropriately. This can happen when a study is designed by researchers not familiar with a therapy. In response to this problem researchers and practitioners of several CAM therapies have begun development of standards for research and reporting. Reporting standards do not guarantee that the therapy was used appropriately, but they at least allow determination of what was done. One such set of reporting standards are the STRICTA recommendations for acupuncture 88. Of the four full evaluations of acupuncture, two (one of which was included in Table 532) met STRICTA reporting standards. As these types of guidelines are not yet available for all CAM therapies, we did not assess whether CAM therapies were applied appropriately in the studies reviewed.

The exemplary studies summarized in Table 5 indicate that a number of CAM therapies may be considered cost-effective compared to usual care for a number of conditions: acupuncture for migraine, manual therapy for neck pain, spa therapy for Parkinson's, complementary guided imagery for cardiac surgery patients, complementary relaxation therapy for patients with previous myocardial infarction, complementary self-administered stress management for cancer patients undergoing chemotherapy, complementary pre- and post-operative oral nutritional supplementation for lower gastrointestinal tract surgery, potassium-rich diet (rather than potassium supplements) for postoperative cardiac patients, and biofeedback for patients with "functional" disorders such as irritable bowel syndrome. Acupuncture and homeopathy were both found to be equivalent in terms of effects and costs to usual care for dyspepsia. The attractiveness of the other CAM therapies shown in Table 5 depends on whether the increased health benefits are worth the additional cost, or whether other aspects of the therapy make them attractive, such as patient preference. Only one of the studies summarized in Table 5 reported results of a CAM therapy being dominated by (clearly inferior to) usual care. The use of professionally-administered stress management for cancer patients undergoing chemotherapy was shown to have higher costs, but no additional health benefits over usual care. It is important for CAM that this contradictory evidence is also known for best clinical practice and the efficient use of CAM resources.

On the surface one might expect that therapies that substitute for usual care (alternative medicine) would be much more likely to be cost effective. In this sample of exemplar studies, of the nine study comparisons where CAM therapies were shown to be superior to usual care (better effects and lower costs, similar effects and lower costs, or better effects and similar costs), four were studies of complementary therapies. Therefore, there is evidence that even though complementary therapies are given in addition to usual care, they can improve clinical outcomes without increasing costs.

In many ways the economic evaluation of CAM therapies is similar to that of conventional medicine. However, there are a number of issues specific to CAM that must be considered. These issues can roughly be divided into three groups: those involved with the impact of economic evaluation on CAM in general, those involving the estimation of health outcomes (ie, issues involved with estimating the efficacy or effectiveness of CAM), and those specific to CAM's economic and humanistic outcomes. The first group of issues has already been addressed above under the potential reasons for paucity.

The methodological challenges involved in determining the clinical effectiveness of CAM have been discussed at length in a number of papers. These include the appropriateness of population-based studies when individualized treatments are used and individualized outcomes are expected 899091, reductionist focus on one therapy for one outcome when that therapy comes from a holistic healing system 929394, the difficulties with blinding when no appropriate placebo is available 9495, and the requirement for randomization when most CAM users have strong preferences for their therapy of choice and will often either refuse to be randomized, or will bypass the randomization if it is not to their liking 94. These challenges are relevant to economic evaluations since they are dependent on effectiveness studies for health outcomes. Also since humanistic and economic outcomes are ideally measured alongside health outcomes in the same trials 141920, the challenges above are also relevant to their measurement.

However, there are several additional issues specific to CAM humanistic and economic outcome measurement which must be considered. First, although CAM therapies can be used to treat acute conditions, they are more commonly used to treat chronic disease, to prevent future disease (risk reduction), and to optimize health and well-being. Using CAM for those indications requires that long term studies be performed 96. However, there are a number of challenges inherent in long term studies in addition to the increase in cost (eg, increased loss to follow-up through patient attrition) 97. In our systematic review we found only two clinical trials that followed patients prospectively longer than one year: a five-year study of relaxation therapy for patients with a previous myocardial infarction 54, and a 3.5-year study of n-3 PUFA as secondary prevention for patients with previous myocardial infarction 25.

Economic evaluations in CAM must recognize that the process of healthcare itself can be effective for patients. Attributes of the process of using CAM that may have value include patient empowerment, the operationalization of patient preference for a particular type of intervention, the length and process of the consultation, and still having treatment options open when other medical approaches have failed 498. Therefore, economic evaluation of CAM needs to measure and include this value where appropriate.

Optimizing health, maximizing wellness, and enhancing well-being are patient-centered outcomes – ones that by definition require subjective measurement 99. Economic evaluation of CAM must include appropriate measurement of these humanistic outcomes to account for the full value of CAM therapies. Our systematic review found five studies where humanistic outcomes were captured. The more well-known instruments used to measure health status in these studies included the SF-6D 35 and the EuroQoL (EQ-5D), and health status was translated into quality of life units using population-based preferences 2751. Sensitivity of these instruments to the changes in quality of life is an important concern for the evaluation of CAM therapies. Although the use of the EuroQoL for manual therapy for neck pain 27 resulted in a statistically insignificant change in quality of life, two other studies demonstrated small, but statistically significant differences in quality of life using the SF-6D for acupuncture for chronic headache 35, and using the EuroQoL for spa therapy for ankylosing spondylitis 51. Therefore, it is possible to measure a change in humanistic outcomes for CAM therapies with these instruments.

The collection of economic outcome data is complicated by that fact that in the United States and other countries many CAM therapies are available over the counter and/or are often paid for out-of-pocket. The lack of administrative claims data on CAM therapies in countries where these costs are not covered or reimbursed means that cost studies require primary data collection (eg, patient self-report instruments) 100. In their study on manual therapy for neck pain, Korthals-de Bos and colleagues used weekly cost diaries to obtain economic outcomes 27. The second, related challenge is that many over-the-counter products, such as certain botanical medicines and nutritional supplements, are not standardized and of inconsistent quality. Standardization and quality will affect both the costs of the therapy and its outcomes. Finally, since there is often no provider "gatekeeper" controlling access to CAM therapies, monitoring of patient use can be complicated and labor intensive.

Despite the challenges described for economic evaluations of CAM therapies, these studies ought to be done. Every planned trial of CAM therapies should at least consider the feasibility of including an evaluation of economic impacts. Observational studies should also include these data, and as information accumulates regarding economic impacts, these costs and cost savings can be estimated more accurately. Although in the ideal every cost category shown in Table 2 should be measured and outcomes should include a measure of quality-adjusted life-years, the estimation of direct medical costs and savings associated with the therapy (eg, practitioner fees, lab fees, and the cost of herbs or other supplements prescribed) will be fairly straightforward for most studies, and the planned primary outcome of the study can serve as the measure of effects to determine cost effectiveness. Even if the clinical outcomes of a CAM therapy are similar or slightly less beneficial than those of usual care, a lower cost of care can still make these therapies attractive to decision makers. However, if no cost data are available, even highly effective therapies can be easily overlooked.

The limitations of this study are similar to those of the other reviews. First, the reader was not blinded to journals and article authors, which may have influenced results. Second, our measures of study quality depend on the information reported in an article, and no attempt was made to judge the merits of clinical or modeling assumptions made in the analyses. Third, only one reader read all the papers and extracted all the data. This may have lead to inaccurate reporting of results, and/or a biased interpretation of study quality. To maximize accuracy, data extraction was performed at least twice for each paper with several months break between extractions. Also, the approach and assumptions used to determine study quality were discussed at length with the other authors. These discussions led to a homogeneous approach being taken to both the application of the reporting quality criteria and the definition as to what constitutes an economic evaluation.