Accuracy of prognosis estimates by four palliative care teams: a prospective cohort study

Abstract

Background

Prognosis estimates are used to access services, but are often inaccurate. This study aimed to determine the accuracy of giving a prognosis range.

Methods and measurements

A prospective cohort study in four multi-professional palliative care teams in England collected data on 275 consecutive cancer referrals who died. Prognosis estimates (minimum – maximum) at referral, patient characteristics, were recorded by staff, and later compared with actual survival.

Results

Minimum survival estimates ranged <1 to 364 days, maximum 7 – 686 days. Mean patient survival was 71 days (range 1 – 734). In 42% the estimate was accurate, in 36% it was over optimistic and in 22% over pessimistic. When the minimum estimate was less than 14 days accuracy increased to 70%. Accuracy was related, in multivariate analysis, to palliative care team and (of borderline significance) patient age.

Conclusions

Offering a prognosis range has higher levels of accuracy (about double) than traditional estimates, but is still very often inaccurate, except very close to death. Where possible clinicians should discuss scenarios with patients, rather than giving a prognosis range.

Background

Doctors are commonly asked 'how long have I got?'. But prognostic error is widespread, only around 20% of estimates are accurate [1, 2]. Errors of both optimism and pessimism occur; the former is more common, in one study 63% of 468 estimates were optimistic [3]. Doctors and other health care professionals are not trained in prognostication even though they may need this information to plan future care, particularly referral to hospice care or for other benefits. Prognostic information can be very important for patients and families in decision making, if they wish to organise their affairs or plan special meetings, trips or events [4, 5]. Younger patients often want full information [6]. Predictive indices, developed to improve estimates of shorter survival, have included factors such as: falling functional status, as indicated by the Karnofsky or Bartell index [7], asthenia [8], weight loss, cognitive failure, dysphagia [9], high total white blood cell count, low lymphocyte percentage and low pseudocholinesterase [10]. However, their sensitivity and specificity is poor. One way of dealing with the difficulties in prognostication is to discuss a possible range of scenarios and survival. But we do not know whether giving a prognosis range improves accuracy, and, if it does, by how much. In this study we tested the accuracy of giving an estimated range of prognosis, rather than a specific point in time.

Methods

Design

Prospective cohort study, as part of a larger assessment of palliative care [11, 12].

Setting

Four multi-professional palliative care teams in the South of England, UK. Teams worked either in home care and/or hospitals [11]. Teams Z and W were basedin the community, but both saw patients in hospitals as well. Teams Y and X were based in acute hospitals but also cared for patients who were at home. All teams included nurses (total 10 full-time equivalent nurses, team Z - 2, Y - 3, X - 2, W - 3) all ofwhom had been trained in palliative care. Some also had oncology and community nurse training. Two teams (Z and W) had their own physicians (1 full-time physician, team Z, 0.6 team W) who visited patients both in hospital and at home, but Y and X (the hospital-based teams) had support from physicians working within the hospital. Teams Z and W also had social workers as part of the team.

Patients

409 consecutive patient referrals known to have died at follow-up one year after the end of the study.

Measurements

After their first full assessment of the patient, the team members gave an estimate of minimum and maximum prognosis. The instructions were simply to estimate the boundaries of minimum to maximum survival. No limit to the range was defined. Team members could refuse to give an estimate if they felt very unsure – which was recorded no estimate given. The key worker assigned to the patient, usually a nurse, made the estimate. This was then usually discussed at the team meeting

Analysis

First, we compared the characteristics and survival of those patients with and without estimates. Then, for those with estimates, the prediction of survival was defined as accurate if survival fell between minimum and maximum estimates. Associations between possible explanatory variables and accuracy were examined using, in the univariate analysis, chi-square tests. To evaluate the independent role of each variable after adjusting for all the others, a multivariate logistic regression model was fitted to the data.

Results

Of the 409 patients, for 133 (33%) no survival estimates were given, and for 1 a minimum prognosis was missing, leaving 275 patients with estimates. There were significant differences between teams in whether they gave a survival estimate, with team W recording an estimate in 93% of cases and team X recording an estimate in only 20% (table). There was slightly shorter survival among patients without estimates, but this did not reach significance. No other differences in patient characteristics were detected.

Table 1 Comparison of the characteristics of those patients with and without prognosis estimates. The percentages are row percentages. Test results are given when P < 0.2. NS = Not significant.

Of the 275 patients with estimates, all had cancer, with a mean age of 66 years (median 67, range 32–92), 52% were male, 61% were married, 28% had cancers of the lung, 31% of gastro-intestinal tract, 12% genito-urinary, 9% breast, 4% lymphatic or haemopoetic, 8% primary unknown, 8% others.

Minimum survival estimates ranged from <1 to 364 days (median 28 days), and maximum from 7 to 686 days (median 84 days). The mean range between minimum and maximum estimates was 57 days, median 42 (range 6–504). Mean actual patient survival was 71 days; median 42 (range 1–734). In 115 patients (42%) the predicted range was accurate (falling between minimum and maximum estimates), in 100 (36%) the minimum was over optimistic; and 60 (22%) lived longer that the maximum estimate (see figure).

Figure

Estimated survival in days (maximum and minimum) versus actual survival.

The minimum estimate was closer to actual survival than the maximum estimate in many, but not all, instances (see figure). The median (range) difference between survival and the minimum estimate (i.e. survival-minimum estimate) was 5 days, but there was a wide range from -156 to 491. Thus, many minimum estimates were close to actual survival and, on average, were 5 days shorter. However, in the most extreme case (-156), the team estimated a minimum survival of 175 days but the patient survived 19 days. The median (range) difference between the maximum estimate and survival (i.e. maximum estimate-survival) was 29 days (-360 to 572). Thus, in the most extreme case (-360), the team estimated a maximum survival of 168 days but the patient survived 528 days.

Accuracy did not appear to be related to gender, status (married or not), whether there was a lay caregiver or their relationship to the patient, or the size of the prognosis range, from minimum to maximum. In univariate analysis accuracy was significantly related to: diagnosis (chi-squared, 9.6, df 4, p < 0.05, accuracy was lowest (24%) for breast cancer patients and was similar for all the others) and palliative care team (chi-squared, 32.2, df 6, p < 0.0001, accuracy was; Z - 33%, Y - 78% X - 15%, W - 42%). Although the least accurate team (X) completed the smallest proportion of survival estimates, further exploration of the team data found no consistent relationship between accuracy and the proportion of missing estimates. However, the most accurate team (Y) cared for patients with the shortest survivals; survival among the other three teams was similar.

There was a trend of reducing accuracy with increasing patient age (chi-squared for trend, 5.0, df 1, p = 0.025, from 52% accuracy in those aged < 55 years, to 31% in those over 75). Accuracy was better in those cases when the minimum estimate was lower (chi-squared, 35.2, df 4, p < 0.001, accuracy was 70% in cases where the minimum estimate was 2 weeks or less). Similar trends were found for the maximum estimate and for survival (although clearly the latter can not be used prospectively).

The logistic model was fitted to the data by including: palliative care teams (3 df), diagnoses (4 df), age (1 df) and, to adjust for case-mix, survival. After adjusting for all these variables, only palliative care teams (p = 0.001) remained significantly associated with accuracy of prediction, age was borderline, (p = 0.054), and the association with the diagnosis observed in the univariate analysis disappeared (p = 0.192).

Discussion

One criticism of the study is that the team members had little time to get to know the patient. Had we recorded estimates later in care, they might have been better. However, a strong doctor-patient relationship is associated with lower prognosis accuracy and prognostic accuracy is higher in more experienced professionals [3]. All of the staff in this study were professionals working in palliative care for some time. Therefore, the staff in this study should be among the most accurate.

The team members gave best guesses about their patient's survival prospects. These were communicated to the investigators, and not to the patients themselves, unless the patient asked for information about prognosis, when a sensitive discussion with the team member followed. In a significant proportion of cases (33%), team members were reluctant to give estimates at all, indicating their own uncertainty in this field and the growing opinion that prognosis estimates were dangerous because of error.

Staff varied in their willingness to provide estimates. One team provided estimates on only 20% of patients, whereas another team provided estimates on over 90%. Willingness to provide estimates did not appear to be related to team characteristics, presence of a doctor, home or hospital based, or accuracy. Nor was it related to patient characteristics. Our discussions with the teams suggested it was more of a philosophical stance of some staff. It may be that if we had asked the question in a different way, for example: 'would you be surprised if..X.. was alive before or after these dates..' we could have found higher completion rates.

However, in some instances staff are required to provide survival estimates. Eligibility for some health care and benefits, for example, Medicare Hospice Benefit in the USA, admission to some hospice programs in Italy and the welfare benefit 'Attendance Allowance' in the UK, is dependent on an estimated prognosis of less than six months. We found that these experienced palliative care staff, primarily nurses, giving a range, had higher levels of accuracy than in earlier studies of doctors or other staff. Gross over-optimism was less common. However, in 58% of cases, the range was wrong. This is likely to be an upper bound to their accuracy, because in one third of patients the team refused to record an estimate. Sensitivity analysis, including these unrecorded estimates as all either 'inaccurate' or 'accurate', would give a range of 28 – 61% accuracy. It is likely that true accuracy is in the range of 28 – 58%, as estimation is likely to be more difficult among those patients where survival was unrecorded.

Estimates tended to be less accurate among older patients. This may be due to the difficulties of multiple health problems in older patients. It might also be one cause of the disadvantaged access of older people to hospice and palliative care services [13, 14]. Despite the relative homogeneity of the teams, differences in accuracy existed and remained after adjusting for the other variables in the logistic model. This suggests that it may be possible to improve estimates with training and experience. During the study, IJH observed that some team members began to change practice. The following quote from one team member, noted during the study, illustrates this: 'about two months ago when I saw a patient, I would think of my normal minimum estimate and then halve it, because I got so many over. But I think that is still too long. So now I take my minimum estimate, halve it, and halve it again, and that's what I write down'. Other team members may have done this, without the investigators knowledge. It may be that monitoring and estimating prognosis estimates and feeding this information back within the team may be an important part of improving practice. This warrants further study. Further research is needed to identify more reliable prognostic indicators, to identify the features of a more accurate estimate or assessor and instances when assessments should or should not be made.

This finding supports earlier work, that prognosis estimates are unreliable and best avoided. However, a range is slightly better than a one point estimate. Estimates became only slightly more reliable (70%) when the minimum survival estimate was less than 2 weeks. This finding may explain why the median survival after enrollment to hospice care in the USA, where a prognosis estimate is a criteria for admission, is 36 days [15], and falling, although a few patients live for longer than one year.

We suggest that if asked about prognosis, we should acknowledge their likely inaccuracies, explore reasons for needing one, discuss possible scenarios and appropriate indications to reappraise the situation, and if appropriate consult with more experienced clinicians. However, this approach would bring into question the criteria for eligibility for palliative and hospice care in many countries. The consistent trend to over estimate survival and therefore delay referral based on these guidelines may be one reason why such short median survivals are observed by palliative care teams. Perhaps short survival after referral should become a negative quality control outcome, since patients with very short survival have very limited access for themselves or their families to the benefits of palliative care teams. Our work supports calls for new eligibility criteria for service access, not reliant highly unreliable prognosis estimates of six months or less [16, 17]. We propose, a more flexible approach is required, centered on a patient and family's condition, needs and likelihood of benefit.