Is the impact of hospital performance data greater in patients who have compared hospitals?

Abstract

Background

Public information on average has limited impact on patients' hospital choice. However, the impact may be greater in consumers who have compared hospitals prior to their hospital choice. We therefore assessed whether patients who have compared hospitals based their hospital choice mainly on public information, rather than e.g. advice of their general practitioner and consider other information important than patients who have not compared hospitals.

Methods

337 new surgical patients completed an internet-based questionnaire. They were asked whether they had compared hospitals prior to their hospital choice and which factors influenced their choice. They were also asked to select between four and ten items of hospital information (total: 41 items) relevant for their future hospital choice. These were subsequently used in a hospital choice experiment in which participants were asked to compare hospitals in an Adaptive Choice-Based Conjoint analysis to estimate which of the hospital characteristics had the highest Relative Importance (RI).

Results

Patients who have compared hospitals more often used public information for their hospital choice than patients who have not compared hospitals (12.7% vs. 1.5%, p < 0.001). However, they still mostly relied on their own (47.9%) and other people's experiences (31%) rather than to base their decision on public information. Both groups valued physician's expertise (RI 20.2 [16.6-24.8] in patients comparing hospitals vs. 16.5 [14.2-18.8] in patients not comparing hospitals) and waiting time (RI 15.1 [10.7-19.6] vs. 15.6 [13.2-17.9] respectively) as most important public information. Patients who have compared hospitals assigned greater importance to information on wound infections (p = 0.010) and respect for patients (p = 0.022), but lower importance to hospital distance (p = 0.041).

Conclusion

Public information has limited impact on patient's hospital choice, even in patients who have actually compared hospitals prior to hospital choice.

Background

Information on performance of hospitals is increasingly available within the public domain in various European countries and worldwide [1], and includes information provided by hospitals and information from previous patients. One of the purposes of this information is to inform consumers about the variation between hospitals so that they can use this information to make informed choices.

The current literature shows that consumers indeed want hospital performance information [2] and that the expected social trend is that use of hospital information will increase over time [3]. However, the current literature also shows that this information only has a limited impact on consumers' choices [4, 5]. Part of the limited impact may be that only a small group of patients have looked up and compared hospital information. Various studies showed that only a small group of patients looked up information to compare hospitals, and that the impact on decision making was small for the total group of patients [6, 7]. It is likely that the impact on decision making is greater in consumers who have actually compared hospitals since one needs to have looked at information for it to have an effect. It is important to assess whether the impact of public information is greater in consumers who have compared hospitals, since this would suggest it may be possible to enlarge the impact of available information on decision making in some groups. If the impact is similarly small in both groups, then other factors may be more important in decision making and we may have to adjust our expectations about the potential impact of public information as suggested by Marshall et al. [8].

If the impact of public information is greater in consumers who have compared hospitals it is important to explore differences between the two groups. There may be several reasons why people have not compared hospitals. First, consumers may not have a need for comparative information because they always go to the same hospital, so they may only look for information for their preferred hospital [3] or do not trust or understand the provided information [5, 9, 10]. These reasons may be difficult to influence. Second, consumers may not have compared hospitals because they could not find the information they consider important. In other words, if we would provide all consumers with all the available public information so that they do not have to find the information themselves, to what extent do consumers who have not compared hospitals consider other information important than consumers who have compared hospitals? If they consider other information important, then we may look for reasons why consumers have not found the information, which does not play a role if they consider the same information important.

The present study therefore aims to assess the extent to which patients who have compared hospitals, more often based their hospital choice on public information, and focus on different information than patients who have not compared hospitals. This can be viewed as a first step to assess whether it may be possible to increase the impact of public information in some groups.

Methods

Study population

The study population consisted of a group new surgical patients from three hospitals (a university hospital, a general teaching hospital, and a general non-teaching hospital) in the Western region of the Netherlands. They were included only if they visited the surgical outpatient clinic for the first time, as identified by a specific code in the hospital information system. They thus did not have any experience with the surgical department, but may have previous experience with other departments of the hospital.

Questionnaire

The questionnaire consisted of four parts. The first part included questions about visiting alternative hospitals, the factors influencing their current hospital choice, the awareness of the available hospital information to compare hospitals and the knowledge where to find hospital information. The second and third part of the questionnaire included an Adaptive Choice-Based Conjoint (ACBC) analysis (see also heading methods). Within the ACBC, patients were first asked to select four to ten attributes they would use for future hospital choice from the list of 41 attributes (selection phase in the second part of the questionnaire). Then, in the third part of the questionnaire, the selected hospital attributes were used to present the respondents with different hospitals. For each hospital the respondent had to indicate whether he/she would consider this hospital for treatment or not (screening phase). Only the considered hospitals are taken forward into a final step to identify the overall best hospital (choice phase).

The hospitals compared in the ACBC are not actual hospitals with their data but fictive hospitals designed to assess the importance of hospital information without any distortion being introduced due to naming the hospitals. In the ACBC all participants were presented with all the information available in the public domain, so that they did not have to find this information by themselves. In this way we could assess whether patients who have compared hospitals consider other information important than patients who have not compared hospitals without this being influenced by difficulty finding the information or reputation associated with certain hospital names. In the last part patients were asked for demographic characteristic like age, gender and education.

Selection of attributes

Hospital attributes and their different levels were selected from available Dutch websites, so that realistic choices between hospitals were presented to patients, closely resembling the actual Dutch situation in which patients may compare hospitals based on public performance data. To identify websites, we searched for hospital information available in the public domain with keywords (e.g. choosing a hospital, performance indicators) and expert advice.

The following websites contained relevant information regarding hospital characteristics and variation between hospitals: http://www.kiesbeter.nl, http://www.independer.nl, http://www.prismant.nl, http://www.nfu.nl, http://www.centrumklantervaringzorg.nl, http://www.rivm.nl, http://www.ziekenhuizentransparant.nl http://www.mediquest.nl, http://www.nvz-ziekenhuizen.nl. The information collected by the Netherlands Health Care Inspectorate (NHI) was also presented on these websites, as well as other hospital information. We only included data about hospital attributes if data were available on variation between hospitals. In addition, information had to apply to all surgical patients rather than disease or surgery-specific information, to be relevant for most of the participants.

A part of the information is based on experiences of previous patients, systematically collected using the Consumer Quality-index (CQ-index) hospital admission [11]. In the CQ-index patients are asked to evaluate the quality of care they received during their hospital stay using a scale with grades from zero to ten. This grading system is the same as the Dutch education report card grading system ranging from 0-10, used throughout the school years and in further education in the Netherlands. A grade below 6 is considered insufficient quality of care, consistent with the education system in which a student would fail a test or exam.

Methods

The study was designed as an experimental choice study, in which we developed an internet-based questionnaire including an Adaptive Choice-Based Conjoint (ACBC) analysis. The questionnaire was developed using SSI Web from Sawtooth Software. Standard CBC is an effective method for understanding how consumers choose between different products or services (in this case hospitals). Simply asking individuals to rate or choose their preferred item from a list may yield no more information than that they want all the benefits and none of the costs. Instead, in a choice experiment respondents are forced to make a trade-off between two or more options. We have previously shown this to be a feasible approach in surgical patients [12]. Recently, Sawtooth Software developed a new approach called Adaptive CBC (ACBC) [13].

The main advantage of ACBC is that each respondent first selects hospital characteristics (attributes) from a list that are relevant for him/her personally when choosing a hospital, so that attribute sets may differ between individuals. In a CBC on the other hand, the investigator chooses a fixed number of attributes and presents these to respondents with the possibility that some of the attributes are not relevant for the respondent. Consequently, the results of a CBC may not accurately reflect which information is important when consumers choose a hospital if they are not able to select the attributes relevant for their situation.

The hospital attributes selected by the respondent in the ACBC were then used to present the respondent with different hospitals. For each hospital the respondent had to indicate whether he/she would consider this hospital for treatment or not. Forty-one hospital attributes were selected for the questionnaire. The order in which attributes were presented to patients was randomized to prevent that first presented attributes received more attention and thereby a higher importance. Based on empirical data, three levels were constructed for each attribute given the minimum, median and maximum estimate. The only exception was the attribute 'type of hospital', for which two levels was constructed (academic hospital and general hospital).

During the screening phase the Sawtooth software notices when a respondent systematically avoids an attribute level or if the respondent expresses interest in only one attribute level. The respondent is asked whether that level is completely unacceptable or an absolute requirement ('must have'). In case the respondent identifies unacceptable levels or 'must have' levels, then all further 'hospitals' that are shown from that point onwards will satisfy those requirements. In this screening phase respondents were presented with a maximum of eight screening tasks.

Considered hospitals are taken forward into a final step (choice phase) to identify the overall best hospital. Respondents had to choose one out of three hospitals, with a maximum of eight choice tasks (the exact number depending on the number of attributes selected and the use of 'unacceptables' and 'must haves').

In the ACBC the respondent is presented with relevant comparisons only through the preselection of attributes and the use of 'unacceptables' and 'must haves' in the screening phase. At the same time fewer comparisons can be presented. The ACBC questionnaire therefore is experienced as a realistic and personalized questionnaire, which is more engaging. Moreover, ACBC requires smaller sample sizes than standard CBC, because more information is captured from each individual [13, 14]. Therefore, this method is even more effective than the previously used CBC [15] to assess which hospital information determines consumers' hospital choice, particularly when there is conflicting information regarding which hospital is best, since it forces consumers to choose which attribute takes precedence over others.

From the choices made by the respondent, the individual's utility for each attribute level can be inferred. Based on the ranges of these utilities the relative importance (RI) of the attributes can be determined (see heading analysis).

Procedure

The patients received written information about this study and a response form before visiting the outpatient clinic. On this form patients could indicate if they wanted to participate or not. If not, they were asked for their reason of non-participation. At their first visit to the outpatient clinic they were asked to hand in the response form, to ensure that we received all response forms.

All patients received the link to the website and a Response ID number by email. If the questionnaire was not filled in after ten days, a reminder was sent. The study was approved by the Medical Ethical Committee of the Leiden University Medical Center.

Analyses

Hierarchical Bayes (HB) estimation was used to estimate the individual utilities for each attribute level using the ACBC Sawtooth software [12]. This method uses an iterative process, along with information from the patients, to estimate the utilities for each subject. Using Gibbs sampling, attribute level utilities are estimated that best fit each patient, borrowing information from other patients to stabilize the model [15].

The RI of each attribute was used as a measure of the influence of each attribute on decision making. The RI of each attribute (A) for each individual is calculated according to:

$$\mathsf{\text{RI}}=\frac{\mathsf{\text{Range }}{\mathsf{\text{A}}}_i}{\sum _{i=1\mathsf{\text{to}}n}\mathsf{\text{Range }}{\mathsf{\text{A}}}_i}\times 100$$

Where Range A_i is the difference between the highest and lowest utility for the i^th attribute, n is the number of attributes and A_i is the i^th attribute.

Based on the choice tasks in the choice phase, after preselection of must haves and unacceptables, the utilities of the attribute levels and the RIs of the attributes are calculated for each individual respondent. The sum of all RIs is thus 100 for each individual. Must haves result in more extreme utilities indicating a stronger preference, and thereby in a higher RI, consistent with the notion that the respondent is saying that this attribute is very important. So a high RI indicates a stronger preference of one attribute over another level, relative to other attributes and thus indicates a high impact on decision making, whereas a low RI indicates a low impact on decision making. When an attribute is not selected by a respondent the RI of this attribute is set to zero for this individual.

To assess which attribute on average is most and least influential in this population, as well as the order in between, we calculated the average RI of an attribute over all respondents with its 95% confidence interval.

The Goodness-of-fit was determined by calculating the Root Likelihood (RLH) [16]. This can be compared to the null RLH i.e. the RLH expected by chance, defined as one divided by the number of alternatives, which is 0.33 in this study given that three hospitals are presented to patients.

Respondents and non-respondents were compared on possible differences in demographic variables. Differences between groups were tested using chi-square tests or using Fisher's exact test in case cells had an expected count less than five.

The study population was divided into patients who compared hospitals (regardless whether patients have compared two or more options) and patients who did not compare hospitals. This was based on a single question by which patients could indicate whether they had compared hospitals for their current hospital choice, or not. We assessed whether patients who compared hospitals differed in age, gender and educational level from patients who did not compare hospitals. We distinguished two age groups (< 65 years and 65+ years) and three educational level groups: basic education (no or only primary education), intermediate education (pre-vocational secondary education, senior secondary vocational training, senior secondary general education or university, pre-university education) or high education (higher professional education or university (Master, Bachelor or PhD)).

Patients not comparing hospitals still might have looked up public information (e.g., only for their preferred hospital) or may have deliberately chosen to go to the same hospital they always go to. With additional questions patients were therefore asked which factors influenced their choice and whether they looked up public information. Based on this information we assessed whether patients who compared hospitals more often used public information for their hospital choice than patients who did not compare hospitals, and differed in other factors influencing their current hospital choice.

Furthermore, we assessed whether the two groups considered other hospital information as most important, by investigating whether the RI of attributes differed between the two groups of patients. We first compared the average RI between the two groups using univariate linear regression analysis. Second, we assessed whether differences between the two groups remained after adjusting for age, gender, level of education using multivariate linear regression analyses. In all statistical analyses a p-value less than 0.05 was considered statistically significant.

Results

643 patients were included of which 461 (71.7%) patients consented to participate and 337 (52.4%) patients had complete data. The most important reason given for non-participation was not having a computer with internet connection (43.2%). The participating patients did not differ in gender (X² = 1.36, p = 0.24), but were younger (X² = 38.93, p < 0.01) than non-participating patients. Level of education was not available for non-participating patients, so that a comparison was not possible. Compared to the general Dutch population [17], our study population included relatively higher educated individuals, but was similar with respect to age and gender (Table 1).

Table 1 Demographic characteristics of the study population and the general Dutch population

From the participating patients, only 71 (21.1%) patients had compared hospitals. No differences were found between patients who did and did not compare hospitals regarding gender, age and level of education (Table 1). At least half of the patients indicated that they were aware of information on hospitals to compare hospitals and most of them knew were to find this information. However few patients actually looked up this information to compare hospitals (Table 2).

Table 2 Differences between patients who have and have not compared hospitals regarding factors influencing hospital choice

Patients who compared hospitals more often chose deliberately for the current hospital (p < 0.05) and more often relied on experiences from other people and on public information in the media compared to patients who did not compare hospitals (p < 0.05) (Table 2). However, their choice still was mainly based on their own previous experience with other departments of the hospital (47.9%) or the advice of their general practitioner (29.6%), as it did in patients who did not compare hospitals (43.2% and 35.3% respectively). These results are not explained by e.g. more patients with higher education in the group comparing hospitals, given that the groups did not differ in age, gender or educational level.

Relative importance of the attributes in the total population

'Report card grade regarding physician's expertise' had the highest relative importance (RI) followed by 'Waiting time for appointment at the outpatient clinic' (Table 3). This information clearly had a higher RI than the next important characteristics 'Waiting time for surgery' and 'Positive judgment about physician communication', given the non-overlapping confidence intervals. These four attributes were most important in both men and women, in patients aged below 65 years and the two highest educational groups (data not shown).

Table 3 Selected attributes, levels and relative importance (RI) between patients who have and have not compared hospitals

In the lowest educational level group 'Waiting time for outpatient clinic appointment' had the highest RI (20.83 [5.66-35.99]), followed by 'Waiting time for surgery' (15.12 [0.68-29.55]), 'Physicians communication' (7.70, [2.07-13.33]) and 'Operation with textbook outcome' (5.79 [-1.85-13.44]). In patients aged above 65 years 'Report card grade regarding physician's expertise' (12.84 [8.12-17.55]) had the highest RI, followed by 'Positive judgment about physician communication' (9.75 [7.24-12.27]), 'Attention to pain management' (5.78 [2.86-8.30]) and 'Waiting time for appointment at the outpatient clinic' (5.38 [3.02-7.75]).

In the total population, the attributes 'Access to information about medication use at the outpatient clinic' (0.18 [0.04-0.31]) and 'Pressure sores' (0.10 [-0.01-0.21]) had the lowest RI (Table 3).

Differences between patients who did and did not compare hospitals

Both patients who did and did not compare hospitals assigned the greatest importance to the same four attributes (Table 3). However, patients comparing hospitals assigned greater importance to 'Wound infections' and 'Report card grade regarding respect for patients' than patients who did not compare hospitals. At the same time, they assigned lower importance to 'Hospital distance' (Table 3).

After adjustment for age, gender and educational level, only differences in 'Wound infections' (β = 0.06 [0.11-1.06]) and 'Report card grade regarding respect for patients' (β = 0.75 [0.07-1.42]) remained.

Discussion

This study has shown that patients who have compared hospitals more often based their hospital choice on public information than patients who have not compared hospitals, but still their decision was mostly based on their own and other people's experiences. Both patients who have compared hospitals and patients who have not compared hospitals value the same public hospital information as most important, that is information regarding physician's expertise, waiting time and communication. However, patients who have compared hospitals more often focus on information on wound infections and report card grade regarding respect for patients and they seem to be prepared to travel further to visit their hospital of choice, given the lower importance assigned to hospital distance.

Our results may have been biased due to the selection of participating patients. This affects our results when these patients choose differently than the non-participants. The participating patients were younger than the non-participating patients and our study population was higher educated than the Dutch population, most likely explained by the use of an internet-based questionnaire. The availability of a computer and internet tends to be higher in individuals from higher socioeconomic groups and in younger individuals [17]. In the group of less educated individuals, waiting time was considered more important than in the group of individuals with higher educational levels. However, this concerned a small number of less educated individuals so that we have to be careful with our conclusions regarding the effect this may have on the outcomes of our study. Assuming that the higher importance of waiting time is true in a larger group of less educated individuals, then the importance of waiting time may have been underestimated in our study due to selective participation. Similarly, in patients aged above 65, physicians' communication and attention to pain management were considered more important than in patients aged younger than age 65. The importance of physicians' communication and attention to pain management may have been underestimated in this study, due to selective participation of younger patients.

Regarding the type of public information that is considered important, both patient groups focus on information regarding physician's expertise, waiting time and physician's communication when choosing a hospital. Other studies also showed that doctor communication was the most important item that influenced the choice of hospital besides friendly staff [2, 18, 19]. Waiting time was also considered most important in a previous study among surgical patients [3]. These results regarding the type of information are thus consistent with other studies. Previous research also showed that information on outcomes of care hardly influenced patients' choice of health care provider [6, 7], but no comparison was made between patients who have and have not compared hospitals and whether they choose differently. The present study thus adds that patients who have compared hospitals consider the same hospital information important as patients who have not compared hospitals, so that we can assume that even in the group actually comparing hospitals, public information only has a minor impact on their hospital choice.

A minority of our study population used public information to compare hospitals. This is consistent with the study of Schwartz et al. who found that few respondents (11%) looked up information to compare hospitals [7]. In the current study patients mostly relied on their own experiences or experiences of other people with the hospital and on the advice of the general practitioner when choosing a hospital. These results seem to be similar to that of other studies showing that comparative consumer information is not the primary information consumers base their choice on, but that they rely mostly on advice of their physician, family or friends [9, 20, 21].

There may be several explanations for our findings that public information is hardly used to compare hospitals and therefore has little influence on decision-making. These can be found in either not searching for information and/or not using this information to compare hospitals. First, many patients rely on the advice of their general practitioner, possibly assuming that their physician uses hospital information to recommend a hospital as suggested by Schwartz et al. [7], so that they are less likely to search for information to compare hospitals by themselves. Second, as suggested in the literature, patients are often unaware that information is available [4]. Only half of our study population indicated that they were aware that public hospital information is available and even a smaller proportion of the patients knew were to find this information. This situation may change in the future, as consumers become more familiar with the internet in general and with websites providing comparative consumer information. Third, patients may be aware of hospital information, but do not use this information to compare hospitals since they always go to the same hospital. It is possible that they only search for information for their preferred hospital. In our study population only a small part (14-15%) checked information regarding the hospital they visited, so that this does not seem to be the entire explanation. Another possibility is that patients do know where to find information, but do not understand, trust or believe this information. If consumers do not understand certain information they are more likely to ignore it or to consider it as unimportant and consequently do not use it to compare hospitals [10]. This would explain why medically oriented quality of care information, like percentage of complications or percentage of wound infections are valued lower than measures on report card grade, which is familiar to everybody since these are used throughout the educational system. It is also possible that the information is not easily available or that patients do not like the way the information is presented on the websites. Studies indicate that the way information is presented affects whether patients understand and use this information [10]. Currently, consumers have to visit different websites for relevant information and have to group the available information on their own. A more structured presentation of the information (e.g. in groups) may facilitate finding the information and thereby the chances that it is used in patients decision making. Further research should explore how patients' decision making is supported in the best possible way.

Conclusions

Even in patients who used public information to compare hospitals, hospital choice was mostly based on their own and other people's experiences. Patients who have compared hospitals prior to their visit focus on the same public hospital information as patients who have not compared hospitals. Therefore, it seems likely that choosing a hospital is not a rational process based on the cognitive assessment of information. In line with a recent analysis by Marshall & McLoughin [8], we believe that hospital choice may be a more social process in which experiences are important. They often have a strong sense of loyalty and indebtedness, which can transcend hard evidence about performance. For example, a person might decide not to choose highly rated hospital only because their grandmother died there even though CQ-data suggest otherwise. It may still be possible to further increase the impact of public information on decision making, but this probably requires a different type of hospital information. For instance, presenting reviews of other people in combination with a score may better enable consumers to interpret the score themselves, because they get more detailed information on the reasons behind a low or high score, similar to current practice regarding e.g. hotel reviews. People may be more likely to trust and thus use this information in decision making.