Email updates

Keep up to date with the latest news and content from BMC Medical Informatics and Decision Making and BioMed Central.

Open Access Research article

Measuring decision quality: psychometric evaluation of a new instrument for breast cancer chemotherapy

Clara N Lee1*, Matthew H Wetschler2, Yuchiao Chang3, Jeffrey K Belkora4, Beverly Moy5, Ann Partridge6 and Karen R Sepucha7

Author Affiliations

1 Department of Surgery, Lineberger Comprehensive Cancer Center, Sheps Center for Health Services Research, University of North Carolina, UNC Plastic Surgery CB 7195, 27599-7195 Chapel Hill, NC, USA

2 Department of Emergency Medicine, Stanford University, Palo Alto, USA

3 General Medicine Division, Massachusetts General Hospital, Harvard Medical School, Boston, USA

4 Institute for Health Policy Studies, University of California, San Francisco, CA, USA

5 Massachusetts General Hospital Cancer Center, Harvard Medical School, Boston, USA

6 Dana-Farber Cancer Institute, Brigham and Women’s Hospital, Harvard Medical School, Boston, USA

7 Health Decision Sciences Center, Massachusetts General Hospital, Harvard Medical School, Boston, USA

For all author emails, please log on.

BMC Medical Informatics and Decision Making 2014, 14:73  doi:10.1186/1472-6947-14-73

The electronic version of this article is the complete one and can be found online at: http://www.biomedcentral.com/1472-6947/14/73


Received:24 February 2014
Accepted:13 August 2014
Published:20 August 2014

© 2014 Lee et al.; licensee BioMed Central Ltd.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Abstract

Background

Women diagnosed with early stage (I or II) breast cancer face a highly challenging decision – whether or not to undergo adjuvant chemotherapy. We developed a decision quality instrument for chemotherapy for early stage breast cancer and sought to evaluate its performance.

Methods

Cross-sectional, mailed survey of recent breast cancer survivors, providers, and healthy controls and a retest survey of survivors. The decision quality instrument includes questions on knowledge and personal goals. It results in a knowledge score and concordance score, which reflects the percentage of patients who received treatments that match their goals. Hypotheses related to acceptability, feasibility, validity, and reliability of the survey instrument were examined.

Results

Responses were received from 352 patients, 89 providers and 35 healthy controls. The decision quality instrument was feasible to implement with few missing data. The knowledge scores had good retest reliability (intraclass correlation coefficient (ICC) =0.75). Knowledge scores discriminated between providers and patients (mean difference 31.1%, 95% CI 26.9, 35.3) and between patients and healthy controls (mean difference 11.2, 95% CI 5.4, 17.1). Most providers reported that the knowledge items covered essential content. Two of the five goal items had a ceiling effect, and one goal had low content validity. The goal items had moderate retest reliability (ICC’s 0.57 to 0.78). In the multivariable model of treatment, none of the patient goals was associated with receipt of chemotherapy. Age and hormone receptor status were the only variables independently associated with chemotherapy. Most patients (77.6%) had treatment concordant with that predicted by the model. Patients who had concordant treatment had similar levels of confidence and regret as those who did not.

Conclusions

The Decision Quality Instrument is a reliable and valid measure of patient knowledge about chemotherapy, but its ability to measure concordance with patient goals is limited. In this sample, patient goals were not associated with treatment, and most patients reported they were not asked their preference, suggesting that goals were not adequately considered in decision making.

Keywords:
Decision; Regret; Breast cancer; Shared decision making; Quality of care; Decision quality; Chemotherapy; Adjuvant therapy

Background

Women diagnosed with early stage (I or II) breast cancer face a highly challenging decision – whether or not to undergo adjuvant chemotherapy. Chemotherapy can provide gains in survival after breast cancer, but it also poses risks of serious toxicity. Because of this tradeoff between survival gains and toxicity risks, clinical guidelines leave the decision about chemotherapy open to the patient and provider, for most patients with stage I disease [1,2]. For Stage I patients, five year survival is quite high, and chemotherapy has a small absolute impact on 10-year survival [3-5]. Thus, the decision about chemotherapy for such patients is a “preference sensitive” one, for which medical evidence is necessary but not sufficient, and the right choice depends heavily on the patient’s personal preferences [6]. For patients with stage II disease, guidelines clearly recommend chemotherapy [1]. For those patients, the decision about chemotherapy should be an informed one, for which the patient fully understands the risks and benefits of the treatment.

Breast cancer patients report challenges and dissatisfaction in making decisions about their treatment [7], and some patients have significant gaps in knowledge or unrealistic expectations of the benefits of systemic therapy [8-12]. Patients tend to rely heavily on provider recommendations about chemotherapy [13], but providers do not always know or elicit patients’ preferences [14] and they may convey information in ways that patients do not understand [15]. Overall, the evidence suggests that early-stage patients may not be meeting acceptable standards for informed consent when making decisions with their providers about adjuvant chemotherapy.

Reducing the rate of uninformed or preference-insensitive chemotherapy decisions will require better measures of decision quality, due to a lack of reliable measures in these domains. A recent international consensus process has defined decision quality as the extent to which patients are informed and receive treatments that match their goals[6]. Valid measures of patient knowledge and preferences for breast cancer chemotherapy decisions, however, are lacking. Most studies of patient preferences have focused on the minimum survival gain that patients require to accept chemotherapy and have not examined the full set of goals and concerns that patients express, including risks of complications and side effects [8,16-19]. Studies of knowledge have relied on ad hoc measures [8].

To address the need for better assessment of quality in chemotherapy decisions, we developed a survey instrument to measure patient knowledge and preferences about chemotherapy for early stage breast cancer, and evaluated its psychometric properties in a field test in breast cancer survivors. The instrument results in two scores, a knowledge score, indicating the extent to which patients are informed, and a concordance score, indicating the percentage of patients who received treatments that matched their goals. To compare the instrument’s performance in groups with different levels of knowledge, we also surveyed breast cancer providers and healthy volunteers. The purpose of the field test was to evaluate feasibility, validity, and reliability of a new instrument that has never been used with breast cancer patients. Given the stress and vulnerability of newly diagnosed patients, we decided to conduct the field test with patients who had recovered from treatment. This approach would provide sufficient data to determine whether the questions were performing as intended, identify problematic wording or other issues, and allow some freedom to include more items than might be reasonably asked of newly diagnosed patients.

Methods

Initial instrument development

The approach to measuring decision quality draws from normative and behavioral decision making theories and is based on the conceptual framework of shared decision making, as described by Mulley [20] and extended by Mulley and Sepucha [21]. Decision quality is defined as the extent to which patients are informed and receive treatments that match their goals [6]. Thus the two primary domains of the instruments are knowledge of key facts and incorporation of patient goals and concerns into the decision.

Development of the Decision Quality Instrument (BCS-DQI©) followed best practices of survey research methods [22-24]. We derived the initial set of facts and goals from a thorough review of the clinical literature and from focus groups of breast cancer patients. The candidate facts and goals were then rated by convenience samples of patients (n = 17) and a multidisciplinary group of providers (n = 27) to evaluate accuracy, importance and completeness [25,26]. Experts in survey research drafted items to cover the key content and conducted cognitive interviews with patients (n = 6) in order to ensure comprehension and acceptability. Of the patients who participated in cognitive interviews, two had Stage I disease, 3 had Stage II, and one had unknown stage. Time since diagnosis ranged from 6 months to 5 years.

Study design

The instrument was tested in a cross-sectional survey of breast cancer patients and providers at four National Cancer Institute-designated cancer centers in 2008, along with healthy volunteers from one of the sites. The four cancer centers were academically affiliated and located in urban areas in the Northeast, Southeast, and West Coast of the United States. The study was approved by the Institutional Review Boards at the Dana Farber Cancer Institute, Massachusetts General Hospital, University of California San Francisco, and University of North Carolina Chapel Hill.

Samples and procedures

Eligible patients were adult women, diagnosed in 2005 or 2006 with early stage breast cancer, as reported in the cancer registry at each site. Exclusion criteria were a diagnosis prior to 2005, Stage 3 or higher, ductal carcinoma in situ only, bilateral breast cancer, receipt of neoadjuvant(before surgery) chemotherapy, and inability to read or speak English. The patients’ treating physicians approved contact.

Eligible patients received the study packet, including an informed consent form, in the mail. Respondents returned their signed informed consent form, signed authorization to access the medical record, and completed study questionnaire by mail. After two weeks, the study staff called those who did not opt out. After four weeks, non-respondents received a reminder mailing and follow up phone call. Respondents received a thank-you letter and small compensation (e.g., $8 book of stamps) for completing the survey. A subset of respondents across all sites received a second survey by mail approximately four weeks after responding, to assess test-retest reliability of the decision quality instrument.

A multidisciplinary group of breast cancer providers (medical oncologists, radiation oncologists, surgical oncologists, plastic surgeons, and general surgeons) was identified by the principal investigator at each site. Healthy volunteers who had no history of breast cancer and were not clinicians were recruited through employee email bulletins of Partners Healthcare in Massachusetts. Providers and healthy volunteer participants received the knowledge portion of the survey by mail. All potential participants received an email reminder after two weeks and a mailed reminder after four weeks. Providers received $50 cash or a gift card upon completion of the survey. Healthy volunteers received a book of stamps upon completion.

Measures

We collected data on demographics and treatment history from each site’s cancer registry and the patient’s report on the survey. When patient self-report conflicted with the registry on a medical or treatment issue (e.g., stage of disease), we examined the medical record to resolve the issue. The survey included the following measures:

1. Breast Cancer Systemic Therapy Decision Quality Instrument (BCS-DQI ©, see Additional file 1). Items cover decisions about adjuvant chemotherapy and hormone therapy. Here we report on the chemotherapy items only. Findings regarding the hormone therapy items will be reported separately. The instrument has two primary domains:

Additional file 1. Breast Cancer Systemic Therapy Decision Quality Instrument.

Format: DOC Size: 128KB Download file

This file can be viewed with: Microsoft Word ViewerOpen Data

a. Knowledge: 8 multiple choice and 4 open-ended items about recurrence, survival, treatment processes, and risks associated with chemotherapy.

b. Goals and concerns: 5 items rating on a scale from 0 (not at all important) to 10 (extremely important).

2. Decision process: 4 items about the nature of the interaction with health care providers including discussion of chemotherapy, pros and cons, and patients’ preferences. In addition, 1 item asked patients about who made the final decision about treatments (mainly the doctor, mainly me, both equally).

3. Patient’s preferred treatment: A single item asked, “Which treatment was your personal preference?” with 4 possible responses: chemotherapy only, hormone therapy only, both chemotherapy and hormone therapy, not have chemotherapy and not have hormone therapy.

4. Treatment received: A single item asking “Did you have chemotherapy?”

5. Informed: A single item asking “How informed did you feel about chemotherapy treatments?” with a 0 to 10 rating scale where 0 (not informed at all) and 10 (extremely well informed).

Confidence: A single item asking “How confident were you that the decision about chemotherapy was the right one?” with a 0 to 10 rating scale where 0 (not confident at all) and 10 (extremely confident). This item has been used in large national studies of decision making. Those studies found that patients reporting a stronger treatment preference and more involvement in decision making had higher confidence [27,28].

6. Regret: A single item asking “If you had to do it again, would you have the same treatments?” with a 5-point Likert scale. The response options were: definitely have the same, probably have the same, not sure, probably not have the same, and definitely not have the same. This item has been used in national studies of decision making that found higher preference concordance was associated with lower regret [29].

Healthy Volunteer survey: Healthy volunteers completed the knowledge questions and some demographic items.

Provider survey: Providers completed the knowledge questions, reported how important each item was (essential, important, not important), and rated how well the knowledge questions covered key content (extremely well, very well, somewhat well, not at all well). We did not ask providers to suggest additional knowledge items, since we had done that in prior stages of instrument development [26].

Analysis

Sample evaluation

We began by evaluating demographics and response rates of the three samples (patient, provider, healthy volunteer).

Item retention and deletion

We evaluated each item in terms of problems with format, level of difficulty, redundancy, and floor and ceiling effects. A problem with format was defined as more than 5% of respondents having a problem, such as marking two answers when one was expected. A knowledge item was identified as possibly too difficult if fewer than 50% of providers responded correctly, or too easy if more than 80% of healthy volunteers answered correctly. An item was identified as possibly redundant if inter-item correlation was greater than 0.8. A steering committee composed of six experts in survey research, decision sciences, and breast cancer care (including two of the authors KS and CL) reviewed these analyses and recommended items for revision or deletion.

Knowledge score

A total knowledge score for each patient was calculated, using the items that were retained after the item retention and deletion process. The knowledge score was defined as the percentage of knowledge questions answered correctly. Missing responses to the knowledge items were considered incorrect, and any respondent who did not complete at least half of the knowledge items did not receive a knowledge score. An aggregate total knowledge score (0-100%) was computed for each sample.

Concordance score

We limited the concordance analysis to Stage I patients since chemotherapy has modest absolute impact on their year survival [3-5], and guidelines recommend considering (rather than definitely providing) chemotherapy, so their decisions are more preference-sensitive. The concordance analysis began by examining each goal to determine whether or not it distinguished between those who did and did not have chemotherapy. Then we included the goals, along with age, and hormone receptor status (defined as estrogen and progesterone receptor positive or not) in a multivariable logistic regression model, with receipt of chemotherapy as the dependent variable. To take into account correlation of observations from the same hospital, a Generalized Estimating Equations (GEE) approach was used with study site as a clustering variable. A patient who had predicted probability ≥0.5 and received chemotherapy, or who had predicted probability <0.5 and did not receive chemotherapy, was defined as having care that “matched” her goals. A summary concordance score (0-100%) was calculated by dividing the number of patients who matched by the total number of patients.

In addition to this concordance analysis in all Stage I patients, we performed an additional analysis in Stage I patients with hormone receptor-negative tumors, since guidelines provide stronger recommendations for chemotherapy in these patients [1].

Acceptability and feasibility

We examined acceptability using instrument completion time, response rates, and written comments. We only had an estimate of completion time because participants had reported completion time for the whole survey, which included scales other than the systemic therapy DQI. We evaluated feasibility using item non-response patterns. Any item with more than 5% missing responses was subject to revision.

Assessment of reliability

To examine retest reliability, we calculated the intra-class correlation coefficient for the total knowledge score and for the individual goals and concerns. The target was to exceed 0.7. Because the knowledge items were not based on a single underlying construct, we did not calculate Cronbach’s alpha or evaluate internal consistency.

Validity of knowledge score

For discriminant validity of the knowledge items, we tested the hypothesis that the provider total knowledge score would be higher than the patient score, and the patient score would be higher than the healthy volunteer score (using ANOVA with planned comparison). For content validity, we evaluated whether or not at least 70% of providers reported that the knowledge items covered key content.

Validity of concordance score

This analysis was limited to Stage I patients. For construct validity of the concordance model, we tested whether or not patients who stated a preference for chemotherapy had a higher model-predicted probability of chemotherapy than those who were unsure, using a two-sample t-test. We examined the hypothesis that concordance would be associated with less regret using a Chi-squared test. We also tested the hypothesis that concordance would be associated with more confidence in decisions using a two-sample t-test. For content validity of the goal items, we evaluated whether or not at least 20% of patients included each item in their top three most important goals.

Results

Sample characteristics

This study was part of a larger study of breast cancer treatment decisions, in which some patients received the survey on systemic therapy, and others received a survey on breast reconstruction. In the overall study, 456 out of 768 patients (59.4%) participated in the initial survey, and 125/137 (91.2%) in the retest survey. This paper includes 352 who completed the systemic therapy initial survey and 89 systemic therapy retest surveys. Many of those patients (237 of 352, 67.3%) had Stage I disease and were included in the concordance analyses. Among the Stage I patients, 35.4% received chemotherapy, alone or in combination with hormone therapy (Table 1). The rate of chemotherapy did not vary by site, for Stage I or Stage II patients. The mean length of time since diagnosis was 30.8 months (SD 9.8). The sample of 35 healthy volunteers was similar to the patient sample, in terms of race, education, marital status, and reported health status, but was younger on average (42 vs. 57 years, p <0.001) (Table 1). Most providers who received the survey completed it (89 of 116, 77%). The providers’ average age was 45 years (SD 9), most were female (65%), and they had been in practice an average of 15 years (SD 11). The provider sample was multidisciplinary, including medical oncologists (33%), nurses (29%), surgeons/surgical oncologists(17%), radiation oncologists (8%) and plastic surgeons (8%).

Table 1. Characteristics of patients and healthy volunteers

Item retention and deletion

Two knowledge items were deleted for being too easy – “Can breast cancer come back after chemotherapy?” (correct answer “yes”) and “Mark whether or not some women have this side effect from chemotherapy: a. hair loss” (correct answer“yes”). We also recommended reducing the number of open-ended quantitative estimate questions (e.g., “If 100 women with breast cancer like yours had chemotherapy, about how many would be cancer free in ten years? ___ write in number of people”) from 4 to 2, because of feedback that patients had difficulty distinguishing among the four recurrence/survival scenarios, and because the questions did not add more information. We kept some of the open-ended questions because they were on critical topics, such as recurrence.

We recommended removal of the response option “I am not sure” because patients used it so frequently (frequency ranging from 11.4 to 49.4%) (Table 2), and it appeared to result in underestimation of patient knowledge.

Table 2. Selected knowledge items and responses (of 12 total items)

Two of the goal items had a ceiling effect. For “Live as live as long as possible,” 284 of 352 patients (79.5%) rated the importance as 10 out of 10. For “Lower chance of recurrence”, 311 of 354 (88.4%) rated the importance as 10 out of 10. We kept these items for the concordance analyses because most patients included them in their top three issues. However, the steering committee recommended revision in future versions to improve their ability to discriminate between patients who had chemotherapy and those who did not.

At least 20% of patients selected each of the goals, except one, in their top three. One goal, “Avoid the costs of having additional treatments”, was deleted for lack of content validity -- only 5.4% of respondents selected it in their top three, and its selection did not vary by treatment. Patients who did not undergo chemotherapy had a slightly different frequency of selecting goals in their top three, compared to those who did undergo chemotherapy. For example, fewer patients who did not have chemotherapy selected “live as long as possible” (84.1 for those who did not have chemotherapy versus 92.4% for those who did, p = 0.021) and “lower the chance of having cancer come back” (85.7% vs. 98.2%, p = 0.001), in their top three goals.

Acceptability and feasibility

The estimated mean completion time for the DQI (including the knowledge items, goals and decision process items) was 7.2 minutes (range 2.3, 22.9 minutes). The knowledge items had a mean of 2.5% missing responses (range 1.4 to 4.5%). Six respondents (1.7%) did not answer enough knowledge items to generate a total knowledge score. For the goal items, the missing responses were also infrequent, with a mean of 2.8% (range 0.4 to 4.4%).

Reliability

The total knowledge score had an intra-class correlation coefficient (ICC) of 0.75.

The goal items had coefficients ranging from 0.57 to 0.78. The individual coefficients were as follows: “avoid serious risks of chemotherapy such as heart problems and other cancers” ICC = 0.58, “have a treatment that was convenient to complete” ICC = 0.66, “avoid having to take medicine for several years” ICC = 0.67, “avoid the costs of having additional treatments” ICC = 0.69, “live as long as possible” ICC = 0.69, “lower the chance of having cancer come back” ICC = 0.78.

Validity of knowledge score

The average patient knowledge score, among the 346 (98.3%) patients who completed at least half of the knowledge questions, was 40.1 (SD 20.9). The average provider knowledge score (71.2, SD 17.0) was higher than the patient score (mean difference 31.1, 95% confidence interval 26.9, 35.3). The average healthy volunteer knowledge score (28.8, SD 16.0) was lower than the patient score (mean difference 11.2, 95% confidence interval 5.4, 17.1). Most providers (62%) reported that the knowledge items covered key content extremely or very well. An additional 33% reported that the items covered key content somewhat well. In the written comments, providers did not note any specific missing content.

Validity of concordance score

These analyses were limited to Stage I patients (n = 232). On multivariable analysis, none of the patient goals was meaningfully associated with receipt of chemotherapy. Age and hormone receptor status were associated with chemotherapy in the multivariable model, with large odds ratios (Table 3). On bivariable analysis, several goals were associated with chemotherapy treatment. For “live as long as possible”, the mean importance score was 9.87 in patients who had chemotherapy and 9.64 in patients who did not (p = 0.008). For “lower the chance of having cancer come back”, the mean importance score was 9.93 in patients who had chemotherapy and 9.77 in those who did not (p = 0.02). For “avoid serious risks of chemotherapy such as heart problems and other cancers”, the mean importance score was 7.01 in patients who had chemotherapy and 7.76 in those who did not (p = 0.076).

Table 3. Multivariable logistic regression model of characteristics associated with chemotherapy in Stage I patients

In the analysis limited to patients with Stage I and hormone receptor positive tumors, no goal was associated with having chemotherapy. Only age was associated with chemotherapy (odds ratio 0.478, 95% confidence interval 0.332, 0.687).

With the multivariable model of treatment, which includes age, hormone receptor status, and goals, 77.6% of patients had treatment that was concordant with the model’s prediction (Table 4). Patients who stated a preference for chemotherapy had higher model-predicted probability of chemotherapy (0.52), compared to those who stated a preference for no chemotherapy or were unsure (0.28, p < 0.0001). Having concordance was not associated with less decisional regret (11.7% for concordant versus 15.4% for non-concordant, p = 0.48) or with more confidence in decisions (8.4 for concordant versus 8.2 for non-concordant, p = 0.67).

Table 4. Concordance between predicted and actual treatment for Stage I patients

Decision process

Patients’ reports about their involvement in the decision process were variable (Table 5). A minority of patients (42.9%) reported that their doctor asked their preference. When asked who made the final decision, 45.7% reported “mainly me”, 22.7% reported “mainly the doctor”, and 29.5% reported “both equally”.

Table 5. Decision process by stage

Discussion

The Breast Cancer Systemic Therapy Decision Quality Instrument is designed to measure the degree to which patients’ chemotherapy decisions are informed and concordant with personal goals. The knowledge score was reliable, and it discriminated among providers, patients, and healthy volunteers. The goal items had modest retest reliability and strong content validity. Patients were able to complete the instrument quickly with few problems.

Concordance

We were somewhat surprised to find that none of the patients’ goals was meaningfully associated with treatment in the concordance model. Although one goal (“Have a treatment that was convenient to complete”) was statistically significant in the multivariable model, the odds ratio was so close to 1 (1.03) that we did not consider it clinically meaningful. One clinical factor, namely hormone receptor status, was the strongest predictor of receipt of chemotherapy. Hormone receptor status, as well as other clinical and patient features, such as age and tumor stage, should, in fact, guide care decisions. Some clinicians might even argue that providers should recommend chemotherapy to their patients with hormone receptor negative disease. However, for hormone receptor positive patients, it is reasonable for patients’ goals to play a role in determining whether or not to undergo chemotherapy. When we restricted the concordance analyses to stage I, hormone receptor positive participants, we still found that only age was associated with receipt of chemotherapy. None of the goals appeared to have any impact on receipt of chemotherapy, even for hormone receptor positive patients. One interpretation of these results is that providers use clinical features to guide treatment choices, with little regard for patients’ goals and preferences.

Another possible interpretation is that the goal items in this instrument may not have captured aspects that were driving patients’ decisions and as a result, the model may not reflect the tailoring of treatments to patients’ goals. Also, the responses to the goal items had little variability, limiting our ability to discriminate among the treatment options. Although the differences in the mean responses by treatment were statistically significant, the actual magnitude of the differences was quite small and would be difficult to use to guide clinical decisions. Future studies could explore alternative item and response formats for eliciting patients’ goals to increase variability.

A limitation of the concordance analysis is that a few of the Stage I patients may have met guideline criteria for chemotherapy (Her2 neu positive tumor larger than 1 centimeter, triple negative tumor larger than 1 centimeter, or high 21-gene assay score) [1,2]. We did not collect clinical data at that level of detail, however, so we could not remove those patients from the concordance analysis.

Decision process

Our data on the decision process suggest that providers may not be taking patients’ goals and preferences into account. Less than half of the patients reported that their provider asked about their preferences for chemotherapy. Discussing patients’ goals and preferences is a core element of shared decision making and is important for patient-centered care. Patients and providers may not be viewing chemotherapy as a preference-sensitive decision for stage 1 patients, but instead considering it as effective care. Treatment guidelines, however, clearly state that use of chemotherapy for most patients with stage I disease is dependent upon patients’ preferences [1]. Additional studies may be warranted to understand factors driving providers’ recommendations for use of chemotherapy, including financial and organizational incentives [30].

Knowledge

The results provide evidence of the reliability and validity of the knowledge score. Few other validated measures of knowledge about chemotherapy are available. A 2003 randomized controlled trial of a decision aid about chemotherapy used a 25-item questionnaire that included true-false items about risk of breast cancer recurrence and risks and benefits of chemotherapy [31]. The questionnaire was based on those used in prior studies of surgery and radiation decisions [32,33], but its psychometric properties have not been reported. Other ad hoc knowledge measures have focused on the accuracy of patients’ estimates of their personal risk of breast cancer recurrence or their estimate of the survival benefit of chemotherapy [17-19,34].

Patient knowledge about key facts related to chemotherapy was quite low in our population, with average overall knowledge of 40.1%. Knowledge about the natural history of early stage breast cancer was particularly low, with only 32.6% of patients answering correctly. Most patients who answered incorrectly overestimated mortality. Knowledge about survival with chemotherapy was low, with 27% of patients answering correctly. Most patients who answered incorrectly overestimated the benefit of chemotherapy on survival.

Prior research suggests that patients tend to overestimate baseline recurrence, mortality rates, and the benefit of chemotherapy; while providers emphasize relative benefit of chemotherapy in terms of impact on 5 or 10 year recurrence [10-12]. This relative benefit is often in the range of 30%, which sounds very high to both patients and providers. When providers educate patients about the absolute benefit of chemotherapy on ten-year mortality (which can range from 2-11%), both parties appear to recognize that the decision is preference-sensitive, and fewer patients elect chemotherapy [9,10,17].

The study had some important limitations. Patients were an average of 2.6 years (SD 9.8) out from treatment, so they may have forgotten some information they knew at the time of decision making. Their responses to questions about goals may have been subject to recall bias, with a tendency to favor goals that were more consistent with the treatment they had. The study population was primarily white, educated, and higher-income than average, so the performance of the instrument in other populations remains unclear. This study was part of a larger study to validate other breast cancer decision quality measures, and the overall response rate for the larger study was lower among non-white patients [35]. We conducted the study at four academic comprehensive cancer centers, where select patients may be referred, and practice styles may differ from that in the community, which may further limit the generalizability of our findings. Healthy volunteers were from one of the sites, so their responses to the knowledge questions may not be representative of healthy persons from other locations.

Conclusions

The Breast Cancer Systemic Therapy Decision Quality Instrument demonstrated strong reliability and validity as a measure of patient knowledge about chemotherapy for early stage breast cancer in this sample, but its ability to measure concordance was limited. In this population, no patient goal was associated with treatment, and most patients reported they were not asked their preference, suggesting that patients’ goals may not have been adequately considered in treatment decision making about chemotherapy. Further testing will be important to understand the performance of the survey in newly diagnosed patients who are actively making the decision, to evaluate alternative approaches to eliciting patients’ goals, to examine the responsiveness of the knowledge score to interventions, such as patient decision aids, and to gather data to help set benchmarks for knowledge and concordance scores.

Competing interests

The authors declare that they have no competing interests.

Authors’ contributions

CL contributed to the study design and participated in the data collection, data interpretation, drafting of the manuscript, and revision of the manuscript. MW participated in the data collection and drafting of the manuscript. YC contributed to the study design and participated in the data management, data analysis, data interpretation, and drafting of the manuscript. JB participated in the data collection, data interpretation, and drafting of the manuscript. BM participated in the data collection, data interpretation, and drafting of the manuscript. AP participated in the data collection, data interpretation, and drafting of the manuscript. KS contributed to the conception and design of the study and participated in the data collection, data interpretation, drafting of the manuscript, and revision of the manuscript. All authors read and approved the final manuscript.

Acknowledgements

The authors would like to acknowledge the patients and providers who participated in the study and the study’s project manager, Sandra Feibelmann.

The study was funded by the Informed Medical Decision Foundation (IMDF). The IMDF contributed to the study design. Karen Sepucha received salary support from the IMDF. Clara Lee received salary support from IMDF and National Institutes of Health KL2RR025746 and 5K07CA154850-02.

References

  1. Carlson RW, Allred DC, Anderson BO, Burstein HJ, Carter WB, Edge SB, Erban JK, Farrar WB, Goldstein LJ, Gradishar WJ, Hayes DF, Hudis CA, Jahanzeb M, Keil K, Ljung BM, Marcom PK, Mayer IA, McCormick B, Nabell LM, Pierce LJ, Reed EC, Smith ML, Somlo G, Theriault RL, Topham NS, Ward JH, Winer EP, Wolff AC: Breast cancer: clinical practice guidelines in oncology.

    J Natl Compr Canc Netw 2009, 7(2):122-192. PubMed Abstract | Publisher Full Text OpenURL

  2. Theriault RL, Carlson RW, Allred C, Anderson BO, Burstein HJ, Edge SB, Farrar WB, Forero A, Giordano SH, Goldstein LJ, Gradishar WJ, Hayes DF, Hudis CA, Isakoff SJ, Ljung BM, Mankoff DA, Marcom PK, Mayer IA, McCormick B, Pierce LJ, Reed EC, Schwartzberg LS, Smith ML, Soliman H, Somlo G, Ward JH, Wolff AC, Zellars R, Shead DA, Kumar R: Breast cancer, version 3.2013: featured updates to the NCCN guidelines.

    J Natl Compr Canc Netw 2013, 11(7):753-760. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  3. Berry DA, Cirrincione C, Henderson IC, Citron ML, Budman DR, Goldstein LJ, Martino S, Perez EA, Muss HB, Norton L, Hudis C, Winer EP: Estrogen-receptor status and outcomes of modern chemotherapy for patients with node-positive breast cancer.

    JAMA 2006, 295(14):1658-1667. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  4. Berry DA, Cronin KA, Plevritis SK, Fryback DG, Clarke L, Zelen M, Mandelblatt JS, Yakovlev AY, Habbema JD, Feuer EJ, Cancer intervention and surveillance modeling network collaborators: Effect of screening and adjuvant therapy on mortality from breast cancer.

    N Engl J Med 2005, 353(17):1784-1792. PubMed Abstract | Publisher Full Text OpenURL

  5. Peto R, Davies C, Godwin J, Gray R, Pan HC, Clarke M, Cutter D, Darby S, McGale P, Taylor C, Wang YC, Bergh J, Di Leo A, Albain K, Swain S, Piccart M, Pritchard K, Early Breast Cancer Trialists’ Collaborative G: Comparisons between different polychemotherapy regimens for early breast cancer: meta-analyses of long-term outcome among 100,000 women in 123 randomised trials.

    Lancet 2012, 379(9814):432-444. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  6. Elwyn G, O’Connor A, Stacey D, Volk R, Edwards A, Coulter A, Thomson R, Barratt A, Barry M, Bernstein S, Butow P, Clarke A, Entwistle V, Feldman-Stewart D, Holmes-Rovner M, Llewellyn-Thomas H, Moumjid N, Mulley A, Ruland C, Sepucha K, Whelan T: Developing a quality criteria framework for patient decision aids: online international Delphi consensus process.

    BMJ 2006, 333(7565):417. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  7. Degner L, Kristijanson L, Bowman D, Sloan J, Carriere K, O’Neil J, Bilodeau B, Watson P, Mueller B: Information needs and decisional preferences in women with breast cancer.

    JAMA 1997, 277(18):1485-1492. PubMed Abstract | Publisher Full Text OpenURL

  8. Ravdin PM, Siminoff IA, Harvey JA: Survey of breast cancer patients concerning their knowledge and expectations of adjuvant therapy.

    J Clin Oncol 1998, 16(2):515-521. PubMed Abstract | Publisher Full Text OpenURL

  9. Belkora JK, Hutton DW, Moore DH, Siminoff LA: Does use of the adjuvant! model influence use of adjuvant therapy through better risk communication?

    J Natl Compr Canc Netw 2011, 9(7):707-712. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  10. Hutton DW, Belkora JK, Shachter RD, Moore DH: Are patients getting the “gist” in risk communication? Patient understanding of prognosis in breast cancer treatment.

    J Cancer Educ 2009, 24(3):194-199. PubMed Abstract | Publisher Full Text OpenURL

  11. Belkora JK, Rugo HS, Moore DH, Hutton DW, Chen DF, Esserman LJ: Oncologist use of the Adjuvant! model for risk communication: a pilot study examining patient knowledge of 10-year prognosis.

    BMC Cancer 2009, 9:127. PubMed Abstract | BioMed Central Full Text | PubMed Central Full Text OpenURL

  12. Belkora JK, Rugo HS, Moore DH, Hutton D, Esserman LJ: Risk communication with breast cancer patients: a cautionary note about printing adjuvant! Screenshots.

    Lancet Oncol 2008.

    In Press

    OpenURL

  13. Siminoff LA, Fetting JH, Abeloff MD: Doctor-patient communication about breast cancer adjuvant therapy.

    J Clin Oncol 1989, 7(9):1192-1200. PubMed Abstract | Publisher Full Text OpenURL

  14. Bruera E, Willey JS, Palmer JL, Rosales M: Treatment decisions for breast carcinoma: patient preferences and physician perceptions.

    Cancer 2002, 94(7):2076-2080. PubMed Abstract | Publisher Full Text OpenURL

  15. Siminoff LA: Improving communication with cancer patients.

    Oncology 1992, 6(10):83-87.

    discussion 87–89

    OpenURL

  16. Siminoff L, Ravdin P, Colabianchi N, Sturm C: Doctor-patient communication patterns in breast cancer adjuvant therapy discussions.

    Health Expect 2000, 3(1):26-36. PubMed Abstract | Publisher Full Text OpenURL

  17. Ravdin PM, Siminoff LA, Davis GJ, Mercer MB, Hewlett J, Gerson N, Parker HL: Computer program to assist in making decisions about adjuvant therapy for women with early breast cancer.

    J Clin Oncol 2001, 19(4):980-991. PubMed Abstract | Publisher Full Text OpenURL

  18. Simes RJ, Coates AS: Patient preferences for adjuvant chemotherapy of early breast cancer: how much benefit is needed?

    J Natl Cancer Inst Monogr 2001, 30:146-152. PubMed Abstract | Publisher Full Text OpenURL

  19. Duric VM, Stockler MR, Heritier S, Boyle F, Beith J, Sullivan A, Wilcken N, Coates AS, Simes RJ: Patients’ preferences for adjuvant chemotherapy in early breast cancer: what makes AC and CMF worthwhile now?

    Ann Oncol 2005, 16(11):1786-1794. PubMed Abstract | Publisher Full Text OpenURL

  20. Mulley AG: Assessing patients’ utilities: can the ends justify the means?

    Med Care 1989, 27(3 Suppl):S269-S281. PubMed Abstract OpenURL

  21. Sepucha K, Mulley A: A perspective on the patient’s role in treatment decisions.

    Med Care Res Rev 2008.

    in press

    OpenURL

  22. Fowler F: Improving Survey Questions. Volume 38. Thousand Paks, CA: Sage Publiations, Inc.; 2002. OpenURL

  23. Fowler F: Survey Research Methods. Volume 1. Thousand Oaks, CA: Sage Publications, Inc; 1993. OpenURL

  24. DeVellis R: Scale Development Theory and Applications. Volume 26. 2nd edition. Thousand Oaks: Sage Publications; 2003. OpenURL

  25. Sepucha KR, Levin CA, Uzogara EE, Barry MJ, O’Connor AM, Mulley AG: Developing instruments to measure the quality of decisions: early results for a set of symptom-driven decisions.

    Patient Educ Couns 2008, 73(3):504-510. PubMed Abstract | Publisher Full Text OpenURL

  26. Lee CN, Dominik R, Levin CA, Barry MJ, Cosenza C, O’Connor AM, Mulley AG, Sepucha K: Development of instruments to measure the quality of breast cancer treatment decisions.

    Health Expect 2010, 13(3):258-272. PubMed Abstract | PubMed Central Full Text OpenURL

  27. Singer E, Couper MP, Fagerlin A, Fowler FJ, Levin CA, Ubel PA, Van Hoewyk J, Zikmund-Fisher BJ: The role of perceived benefits and costs in patients’ medical decisions.

    Health Expect 2014, 17(1):4-14. PubMed Abstract | Publisher Full Text OpenURL

  28. Zikmund-Fisher BJ, Couper MP, Singer E, Levin CA, Fowler FJ Jr, Ziniel S, Ubel PA, Fagerlin A: The DECISIONS study: a nationwide survey of United States adults regarding 9 common medical decisions.

    Med Decis Making 2010, 30(5 Suppl):20S-34S. PubMed Abstract | Publisher Full Text OpenURL

  29. Fowler FJ Jr, Gallagher PM, Drake KM, Sepucha KR: Decision dissonance: evaluating an approach to measuring the quality of surgical decision making.

    Jt Comm J Qual Patient Saf 2013, 39(3):136-144. PubMed Abstract | PubMed Central Full Text OpenURL

  30. Malin JL, Weeks JC, Potosky AL, Hornbrook MC, Keating NL: Medical oncologists’ perceptions of financial incentives in cancer care.

    J Clin Oncol 2013, 31(5):530-535. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  31. Whelan T, Sawka C, Levine M, Gafni A, Reyno L, Willan A, Julian J, Dent S, Abu-Zahra H, Chouinard E, Tozer R, Pritchard K, Bodendorfer I: Helping patients make informed choices: a randomized trial of a decision aid for adjuvant chemotherapy in lymph node-negative breast cancer.

    J Natl Cancer Inst 2003, 95(8):581-587. PubMed Abstract | Publisher Full Text OpenURL

  32. Whelan TJ, Levine MN, Gafni A, Lukka H, Mohide E, Patel M, Streiner D: Breast irradiation post lumpectomy: development and evaluation of a decision instrument.

    J Clin Oncol 1995, 13:847-853. PubMed Abstract | Publisher Full Text OpenURL

  33. Whelan T, Levine M, Gafni A, Sanders K, Willan A, Mirsky D, Schnider D, McCready D, Reid S, Kobylecky A, Reed K: Mastectomy or lumpectomy? Helping women make informed choices.

    J Clin Oncol 1999, 17(6):1727-1735. PubMed Abstract | Publisher Full Text OpenURL

  34. Thewes B, Meiser B, Duric VM, Stockler MR, Taylor A, Stuart-Harris R, Links M, Wilcken N, McLachlan SA, Phillips KA, Beith J, Boyle F, Friedlander ML: What survival benefits do premenopausal patients with early breast cancer need to make endocrine therapy worthwhile?

    Lancet Oncol 2005, 6(8):581-588. PubMed Abstract | Publisher Full Text OpenURL

  35. Lee CN, Chang Y, Adimorah N, Belkora JK, Moy B, Partridge AH, Ollila DW, Sepucha KR: Decision making about surgery for early-stage breast cancer.

    J Am Coll Surg 2012, 214(1):1-10. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

Pre-publication history

The pre-publication history for this paper can be accessed here:

http://www.biomedcentral.com/1472-6947/14/73/prepub