Training satisfaction for subspecialty fellows in internal medicine: Findings from the Veterans Affairs (VA) Learners' Perceptions Survey

Abstract

Background

Learner satisfaction assessment is critical in the design and improvement of training programs. However, little is known about what influences satisfaction and whether trainee specialty is correlated. A national comparison of satisfaction among internal medicine subspecialty fellows in the Department of Veterans Affairs (VA) provides a unique opportunity to examine educational factors associated with learner satisfaction. We compared satisfaction across internal medicine fellows by subspecialty and compared factors associated with satisfaction between procedural versus non-procedural subspecialty fellows, using data from the Learners' Perceptions Survey (LPS), a validated survey tool.

Methods

We surveyed 2,221 internal medicine subspecialty fellows rotating through VA between 2001 and 2008. Learners rated their overall training satisfaction on a 100-point scale, and on a five-point Likert scale ranked satisfaction with items within six educational domains: learning, clinical, working and physical environments; personal experience; and clinical faculty/preceptor.

Results

Procedural and non-procedural fellows reported similar overall satisfaction scores (81.2 and 81.6). Non-procedural fellows reported higher satisfaction with 79 of 81 items within the 6 domains and with the domain of physical environment (4.06 vs. 3.85, p <0.001). Satisfaction with clinical faculty/preceptor and personal experience had the strongest impact on overall satisfaction for both. Procedural fellows reported lower satisfaction with physical environment.

Conclusions

Internal medicine fellows are highly satisfied with their VA training. Nonprocedural fellows reported higher satisfaction with most items. For both procedural and non-procedural fellows, clinical faculty/preceptor and personal experience have the strongest impact on overall satisfaction.

Background

The quality of training provided in physician training programs is an important focus of health systems, hospitals and undergraduate and graduate medical education leaders. Trainee satisfaction is one element of quality in clinical education. Its relationship with different components of clinical, learning and work experiences is often explored to identify elements associated with high satisfaction within individual training programs, but not across training programs nationally or across disciplines. Understanding which factors contribute to trainee satisfaction, and how they contribute, is critical to the design of education programs that will meet the needs of trainees across different specialties.

An ideal setting for examining components of trainee satisfaction is the Department of Veterans Affairs (VA). VA is the second largest funding source of United States (US) physician training positions, with over 9500 physician resident positions of which 1600 internal medicine subspecialty fellow positions were funded in 2008-2009. Each year, nearly one-third of the nation's physician trainees rotate at 120 VA centers and three independent outpatient clinics through affiliations with medical schools and teaching hospitals.

Previous studies of satisfaction with VA training have used data from the Learners' Perceptions Survey (LPS), a validated survey instrument that measures satisfaction across multiple domains. Since 2001, VA has administered the LPS annually to all learners who train at VA medical facilities. Prior work using data from the LPS established differences in perception of satisfaction for learners in different stages of training [1] and between physician trainees in different specialties [2]. One hypothesis to explain differences between types of trainees is that such variability may be related to differences in daily experiences. To test this hypothesis, we examined the degree to which predominance of procedural experiences explain dissimilarities among learners, focusing on fellows in different subspecialties in internal medicine.

Specifically, we measured satisfaction across internal medicine fellows by subspecialty and compared satisfaction between procedural versus non-procedural subspecialties. We also identified factors associated with satisfaction and compared how these factors differ between fellows in these two groups.

Methods

Survey development

The LPS was developed to examine and measure learner satisfaction for all healthcare trainees working in VA. Survey development began in 1999 using standard psychometric procedures [2]. An initial item pool was derived based on an exhaustive review of the literature on learner satisfaction and refined based on feedback from 15 focus groups of VA faculty and clinical trainees. Items were grouped into six domains: clinical faculty or preceptor (13 items), learning environment (15), clinical environment (15), working environment (13), physical environment (12) and personal experience (13). For each item, respondents were asked to rate satisfaction with VA training using a five-point Likert scale. The survey was piloted on over 1,000 trainees from 22 geographically diverse VA medical centers. Confirmatory factor analysis upheld the integrity of each domain. On the basis of pilot testing, the survey was refined to items that contributed to an overall training satisfaction and to satisfaction ratings in six educational domains. Items and the corresponding domains are listed in Additional file 1.

Since its 2001 rollout, the LPS has been administered annually to assess perceptions of clinical learners toward their VA experiences. The LPS consistently shows domain content stability, integrity and Alpha reliability in the .90s for both the overall survey and its sub-domains [2].

Over time, questions have been added to the LPS to explore the impact of changes in the clinical education environment on trainee satisfaction. For the current analysis, we evaluated only those components of the survey that have been unchanged since 2001, including scores for overall training satisfaction, educational domains and their associated items.

LPS Outcome Measures

To derive an overall satisfaction score, learners are asked: "On a scale of 0 to 100, where 100 is a perfect score and 70 is a passing score, what numerical score would you give your most recent VA clinical training experience?" The response to this question is the "overall training satisfaction score" and the primary outcome measure used in this analysis.

Study design

The current report presents survey results from an eight-year summary analysis of trainees' satisfaction with training experiences at VA medical centers from 2001 through 2008.

Study population and survey administration

Participants were physician fellows in internal medicine fellowships who rotated through a VA facility during the academic year. In 2001, trainees registered for the LPS survey through a post-card registration process. Registered trainees were then mailed a paper survey or could complete an online version of the survey. For subsequent years the separate registration process was discontinued, and all physician trainees were encouraged to participate in the survey through a combination of national and local recruitment efforts. Nationally, letters of information and invitation were sent out from the Office of Academic Affiliations to all physician trainees for whom addresses were available. In addition, individual VA facilities were encouraged to develop complementary local processes to encourage trainee participation in the survey. Local processes for trainee recruitment varied. The survey was available in both paper and online versions in 2001 through 2003. Since 2004, the survey has been administered exclusively online.

We categorized fellows in internal medicine-based programs as participating in procedural fellowship (cardiology, gastroenterology and pulmonary/critical care medicine) or non-procedural fellowships (endocrinology, geriatrics, hematology/oncology, infectious diseases, nephrology and rheumatology).

Analyses

We used mixed-effects models to adjust scores by subspecialty and to compute both the effect of individual items on domain scores and the effect of domain scores on overall satisfaction for the VA hospital. We adjusted all estimates to account for each subspecialty, calculated to reflect a PGY-4 fellow, and corrected for year of survey and facility nesting. Adjustment was necessary to permit comparisons across subspecialties when responders may be distributed across survey year and facilities differently. No adjustments were made to reflect characteristics of individual responders because the information was limited due to individual responder anonymity. Furthermore, we wanted to measure total, rather than partial, differences across specialties. We computed item effect sizes from the estimated coefficients to mean-centered domain item × subspecialty indicator interaction terms. Domain and domain items were scaled between one and five, where five indicates very satisfied, and one indicates very dissatisfied. We measured effect sizes (item on domain scores) to reflect the increase in domain score per unit increase in item score. Effect sizes were adjusted to reflect a PGY-4 and corrected for year of survey and facility nesting. We did not adjust to reflect changes in the other items since our purpose was to compute a total effect size. All procedures were performed using SPSS. To account for multiple comparisons, we consider statistically significant only factors where p ≤ 0.001.

Ethical considerations

The U.S. Office of Management and Budget, which reviews and approves federal government sponsored surveys, approved the LPS. We maintained confidentiality by keeping respondent information in a separate database and reviewing only aggregate data. Participation in the survey was voluntary. The confidential nature of the data collection and voluntary participation were fully disclosed to survey participants.

Results

Overall training satisfaction

There were 2,221 responses by fellows between 2001 and 2008 (Table 1). The distribution of respondents by fellowship was very similar to the distribution of funded positions by the OAA for each fellowship type (data not shown). There were 1,026 responses in the procedural group and 1,195 responses in the non-procedural group. Cardiology fellows provided the largest number of responses for procedural fellows (n = 459), and Hematology-Oncology fellows provided the largest number of responses for non-procedural fellows (n = 283).

Table 1 Number of responders by year of survey, level of training and discipline

Overall satisfaction scores were similar for all internal medicine subspecialties, revealing only minor differences in adjusted mean scores that did not achieve statistical significance (Table 2). Additionally, the mean adjusted overall satisfaction scores for procedural fellows (81.2) and non-procedural fellows (81.6) were not statistically significantly different (p = 0.59).

Table 2 Adjusted overall training satisfaction scores for resident fellows by subspecialty

Satisfaction at the domain level

Procedural and non-procedural fellows reported similar satisfaction with the following domains: clinical faculty/preceptors, personal experiences, learning, working and clinical environments. There were differences in the reported satisfaction with overall physical environment, with non-procedural fellows reporting higher satisfaction compared to procedural fellows (4.06 vs. 3.85, p < 0.001) (Table 3). Rank order of domain satisfaction was similar for procedural and non-procedural fellows, with trainees reporting highest satisfaction with clinical faculty/preceptors, followed by personal experience, learning environment and working environment.

Table 3 Adjusted mean scores for domains and selected* items for non-procedural and procedural fellows

To measure the impact of domains on overall satisfaction, we measured the effect of domains on overall satisfaction score as the change in overall satisfaction (range 1 to 100) associated with each one-point change in the mean scale score for the domain (Likert scale with range 1 to 5). Each domain provided a statistically significant contribution to the overall score for both procedural and non-procedural trainees (data not shown). A comparison of the impact of each domain on overall satisfaction revealed differences between procedural and non-procedural fellows, with learning environment and personal experience contributing more to overall satisfaction for procedural fellows than for non-procedural fellows (Table 4). However, the rank order of the impact of each domain on overall satisfaction was similar, with both procedural and non-procedural trainees ranking personal experience the highest, followed by learning environment.

Table 4 Adjusted effects of each domain score on overall satisfaction for non-procedural and procedural fellows

Satisfaction with items within domains

Results for non-procedural fellows showed a consistent pattern of greater satisfaction on individual items, both in number of items and in magnitude (Additional file 1). Non-procedural fellows reported significantly higher satisfaction with: accessibility and availability of clinical faculty and preceptors; timeliness of feedback; fairness in evaluation; clinical faculty and preceptors' patient-oriented nature; time for learning; teaching conferences; morale of ancillary and support staff; laboratory services; ancillary and support staff; maintenance of equipment; availability of food on call; and continuity of relationship with patients (Additional file 1). Procedural fellows reported higher satisfaction with only two of 81 items (timely performance of necessary procedures/surgeries and degree of autonomy); these differences did not achieve statistical significance.

Within the domain of clinical faculty and preceptor, the rank order of satisfaction with items was similar, with highest satisfaction reported for approachability and openness, clinical skills, patient-oriented, and quality of faculty. For the domain of personal experience, the rank order of satisfaction with items was also similar, with highest satisfaction reported for relationship with patients, appreciation of respondent's work by patients, personal reward and personal responsibility for patient care.

We calculated the contribution of each item to its domain satisfaction score, measuring the change in the respective domain score associated with each one-point change on the Likert scale for individual items. Those items with larger effect sizes are listed in Table 5 and a more comprehensive listing is found in Additional file 2. For both types of trainees, most items within clinical faculty/preceptors were strongly associated with domain satisfaction, as were many items within the domain of personal experience. All items were found to have a statistically significant association with the respective domain score, both for procedural and non-procedural fellows (Additional file 2). Only a few items had statistically significant differences in impact on satisfaction for the two types of fellows (Additional file 2). Cleanliness of facilities and housekeeping, balance of personal and professional life, parking and level of job stress contributed more strongly to overall satisfaction for procedural fellows; however, these items contributed only modestly to the overall score and many were associated with lower overall domain satisfaction scores.

Table 5 Domain items with the largest* adjusted effect sizes on domain scores

Discussion

Our study is the first to comprehensively survey internal medicine subspecialty fellow satisfaction across multiple programs and compare perceptions between procedural and non-procedural fellows. Overall satisfaction with VA training is similar between procedural and non-procedural fellows in internal medicine, but differences exist at the item and domain-level.

The LPS is a validated survey instrument with robust psychometric properties and content and face validity that has been used to evaluate trainee satisfaction across a large, relatively uniform healthcare system. Previous studies have demonstrated its usefulness in comparing trainee perceptions across disciplines. Keitz et al found that the LPS functioned well in discriminating differences between different types of learners in VA [2]. Cannon et al extended the scope of the LPS by comparing satisfaction between medical students and residents [1]. Analysis of differences in subspecialty fellow satisfaction provides valuable information to GME leaders for program development and extends the utility of the LPS for evaluating satisfaction in graduate medical education.

There were similarities between our findings and those of Keitz et al [2] and Cannon et al [1] with respect to overall satisfaction and domain satisfaction. Overall satisfaction was similar for fellows, different types of residents [1, 2] and medical students [1]. With the exception of physical environment, procedural and non-procedural fellows had similar domain satisfaction, as was seen with both medical students and residents [1]. Like medical students [1] and residents [1, 2], fellows reported highest satisfaction with the domain of clinical faculty and preceptors. All three studies found that learning environment contributed highly to overall satisfaction, although in our study this was more so for procedural fellows. Unlike in the previous studies utilizing the LPS, our study examined the personal experience domain and found that personal experience was the domain most strongly associated with overall satisfaction for both procedural and non-procedural fellows, with both reporting similarly high levels of satisfaction with relationship with patients, appreciation of respondent's work by patients, personal reward and personal responsibility for patient care.

While domain satisfaction was similar across a wide spectrum of learners in all three studies, individual items within domains and their contribution to overall domain score provided more distinction between different types of learners. For example, procedural fellows reported lower satisfaction with several items in the clinical faculty/preceptor domain including accessibility/availability, timeliness of feedback, fairness of evaluation and patient orientation. Keitz et al found similar results for surgical residents who reported lower satisfaction with accessibility and availability of faculty and preceptors as compared to less procedural residents [2]. For subspecialty fellows, item differences were seen in equipment maintenance and food on-call, with procedural fellows expressing lower satisfaction. These results may reflect differing needs of procedural fellows who use diagnostic and therapeutic equipment more frequently and may be more likely to be on-call overnight in the hospital.

Within the working environment, non-procedural fellows reported higher satisfaction with ancillary/support staff morale, laboratory services, and ancillary/support staff. In a previous study, differences were also found between medical students and residents [1], and among residents, with internal medicine residents least satisfied with these items [2]. The relatively lower levels of satisfaction with these items among procedural fellows and internal medicine residents may reflect higher intensity of interactions with these services and therefore a different level of expectation compared to other types of learners. Whereas past studies suggested that such differences in satisfaction may be related to the differences in types of training programs [2], current results showing differences between procedural and non-procedural fellows may reflect degrees to which the same parts of the VA healthcare system intersect differently with the training goals of different program types and available training infrastructure.

Levels of satisfaction may reflect those attitudes and values that influenced physicians' choice of specialty training. A number of studies have evaluated predictors of subspecialty choice among residents and satisfaction with career choice among practicing physicians. In a study of factors associated with subspecialty choice of Canadian internal medicine residents, Horn et al found that residents pursuing non-procedural fellowships were more concerned with issues related to lifestyle, stress, work hours, leisure hours, and patient populations than those pursuing procedural fellowships [3]. Other studies have found that lifestyle [4–7], mentorship [3], faculty influence [4, 5, 8], role models [3, 9, 10], resident clinical experience [3–5, 8] and high sense of satisfaction of fellows [8] are important factors in trainee selection of specialty training. Our study showed that personal experience (including lifestyle, stress and fatigue) and clinical faculty/preceptors (including mentoring and role models) contributed most significantly to overall satisfaction for both types of trainees, suggesting that improvements in these areas could lead to higher learner satisfaction and possibly more successful recruitment. Furthermore, differences in satisfaction with career choice have been noted between primary care and specialty residents [11] as well as procedure and non-procedure-based practicing physicians [12], suggesting that data on fellow satisfaction may provide useful information for residents in guiding career choices.

Quality in graduate medical education programs is complex and has many crucial components such as curriculum, trainee competency, and faculty development. Learner satisfaction with training is another crucial component for individual training programs [13], hospital systems [14], and national organizations [15]. With significant national focus on both changes in health care systems and regulatory requirements, particularly given forthcoming changes in Accreditation Council of Graduate Medical Education's Common Program Requirements, careful analysis of trainees' satisfaction with educational and work environments is essential to improving quality. Byrne et al advocated for the use of a comprehensive survey tool to examine residents' satisfaction with training, and demonstrated the use of such a tool to effect improvements within affiliated hospitals from one GME program [14]. The authors argued for the importance of a national, comprehensive survey tool to monitor trainee experience with the expressed goal of improving the training environment. In addition, monitoring the experience of trainees may provide advantages to individual programs in the form of longer accreditation cycle length [16]. Changes in trainee satisfaction should be monitored over time, and ideally both aggregate and facility-level data would be available to allow for analysis of variation between facilities.

Our study has several strengths. First, the LPS is a validated, comprehensive survey tool, providing key information regarding the work and learning environments in which the majority of physicians train. Second, the LPS targets learners within one large healthcare system, potentially limiting the degree to which the differences between healthcare systems in which trainees learn may affect the measured outcomes. Third, the LPS is designed to assess trainee perceptions across a full complement of subspecialty training programs, program training years and academic years. Finally, the results of this study are likely representative of fellow perceptions throughout the VA, since the distribution of respondents across subspecialties was nearly identical to the distribution of VA funded positions nationally for each subspecialty.

The study has several limitations. First, there were multiple comparisons performed in this study, which may have led to statistical error. For this reason, we set a threshold for statistical significance at p ≤ 0.001. Secondly, because the data are collected anonymously, we were unable to evaluate the changes in individual respondents over time, limiting the precision of our results. Thirdly, we were unable to determine the total number of fellows within the system and made the survey available but did not distribute the survey to them directly. Consequently, the estimated response rate was relatively low, and this strategy could result in selection bias. However, as mentioned, the distribution of fellow respondents mirrors the distribution of VA funding for each specialty. In addition, a comparison of common questions on the LPS and the American Academy of Medical Colleges (AAMC) medical student survey, which has a greater than 90% response rate, have demonstrated similar results suggesting that the sampling method for the LPS may not be subject to major selection bias. Finally, we collected the LPS data only for VA experiences, which may limit the applicability of the findings to other training settings. While most physicians train in a VA setting, further information about non-VA training settings would allow better understanding of how VA experience compares to the other sites where fellows train.

Conclusions

VA internal medicine fellows of all subspecialties are generally satisfied with their VA training. For procedural and non-procedural fellows, satisfaction with the domains of clinical faculty and preceptors and personal experience contributed highly to the overall satisfaction score. Differences in satisfaction were evident in the comparison, and further study is needed to clarify whether the sources of these differences arise from dissimilarities in the learners themselves, the training needs of various disciplines or the parts of the healthcare system or infrastructure with which trainees interact. Better understanding of the factors associated with satisfaction among different types of fellows may assist residents in selecting training programs and may assist program directors and GME leaders in improving programs, thereby enhancing recruitment, improving fellow satisfaction with their training experience and enhancing educational outcomes.