Undergraduate educational environment, perceived preparedness for postgraduate clinical training, and pass rate on the National Medical Licensure Examination in Japan

Abstract

Background

We investigated the views of newly graduating physicians on their preparedness for postgraduate clinical training, and evaluated the relationship of preparedness with the educational environment and the pass rate on the National Medical Licensure Examination (NMLE).

Methods

Data were obtained from 2429 PGY-1 physicians-in-training (response rate, 36%) using a mailed cross-sectional survey. The Dundee Ready Education Environment Measure (DREEM) inventory was used to assess the learning environment at 80 Japanese medical schools. Preparedness was assessed based on 6 clinical areas related to the Association of American Medical Colleges Graduation Questionnaire.

Results

Only 17% of the physicians-in-training felt prepared in the area of general clinical skills, 29% in basic knowledge of diagnosis and management of common conditions, 48% in communication skills, 19% in skills associated with evidence-based medicine, 54% in professionalism, and 37% in basic skills required for a physical examination. There were substantial differences among the medical schools in the perceived preparedness of their graduates. Significant positive correlations were found between preparedness for all clinical areas and a better educational environment (all p < 0.01), but there were no significant associations between the pass rate on the NMLE and perceived preparedness for any clinical area, as well as pass rate and educational environment (all p > 0.05).

Conclusion

Different educational environments among universities may be partly responsible for the differences in perceived preparedness of medical students for postgraduate clinical training. This study also highlights the poor correlation between self-assessed preparedness for practice and the NMLE.

Background

One of the principal roles of medical schools is to equip newly graduating physicians for postgraduate clinical training [1–3]. Lack of preparedness for this training has been linked to greater stress in junior physicians and may also lead to poorer quality of patient care [4]. To improve clinical skills that can increase readiness for postgraduate clinical training, undergraduate medical education in Japan has recently begun implementing the Model Core Curriculum. This is proposed to restructure all areas of undergraduate medical education by requiring the implementation of the core curriculum for 70% of an entire curriculum and unique curriculum for remaining 30% in each medical school in order to facilitate learning by active participation in clinical clerkships [5]. The core curriculum emphasizes clinical practice through use of a problem-based learning (PBL), organ-based integrated curriculum (rather than a department-based approach), and implementation of the objective structured clinical examination (OSCE) [6].

In UK studies, self-perceived preparedness for postgraduate clinical training was found to differ substantially among medical schools [1, 2]. Crucial factors influencing these differences may include a good educational environment for medical students, curricular changes with greater attention to preparedness for practice, adherence to modern educational theory, integration of basic and clinical sciences, and appropriate selection of students who will be successful medical graduates [7]. However, the extent, to which newly graduating physicians in Japan are prepared for postgraduate clinical training and differences among medical schools in terms of preparedness, has not been evaluated.

Preparedness for postgraduate clinical training of medical students is likely to depend on the engagement of medical students for learning clinical medicine. Educational environment is now considered as one of the most important factors determining engagement of medical students [8, 9]. The educational environment is also considered to be the most significant manifestation and conceptualization of a curriculum, since it embraces everything that happens in a medical school [10]. Factors such as self-perceptions of learning, self-perceptions of teachers, academic self-perceptions, self-perceptions of atmosphere, and social self-perceptions are currently considered as major domains that comprise the educational environment of medical schools [11]. For instance, competitive, authoritarian, stressful, or intimidating environments may de-motivate students and weaken their engagement for learning clinical medicine. Environments that are collaborative, collegial, and supportive may enhance greater engagement of medical students for learning clinical medicine and the greater engagement may lead to improved preparedness for postgraduate clinical training. Thus, it is important to investigate whether a better environment is likely to help graduating physicians to be better prepared for postgraduate clinical training.

Preparedness for postgraduate clinical training could be assessed by different measures and self-perceived preparedness may be one for conducting nationwide studies like the previous UK report [1]. Thus, in the current study, we first sought the views of newly graduating physicians on their preparedness for postgraduate clinical training to obtain an understanding of this preparedness at different medical schools. Next, we evaluated the relationship between the educational environment and students' self-perceived preparedness at all 80 medical schools in Japan. We also evaluated the relationship between the pass rate on the National Medical Licensure Examination (NMLE) and self-perceived preparedness for postgraduate clinical training.

Methods

Subjects

In November 2008, a cross-sectional survey was mailed to 6725 1^st-year resident physicians at 427 teaching hospitals with five or more 1^st-year resident physicians, including 80 university hospitals and 347 non-university teaching hospitals throughout Japan. The program director at each hospital was asked to encourage the residents to complete the self-administered questionnaire. The academic calendar in Japan starts on April 1 and ends on March 31 of the following year; thus, the survey was conducted at the midpoint (November) of the 2008 academic year. We obtained ethical approval for the study from the ethics committee of St. Luke's International Hospital (the institution that the principal investigator was affiliated with at the time of the study) in Tokyo, Japan.

Survey Contents

Data collected in the survey included the demographics of the respondents and their responses to the Dundee Ready Education Environment Measure (DREEM) inventory, which were used to assess the learning environment of the medical schools from which the respondents had graduated [11]. The earlier Japanese version of the DREEM was developed by Nishigori et al [12, 13]. The original British version of the DREEM is in the public domain and was translated forward and backward by native-English speakers and native Japanese speakers, respectively, for development of a version for this study, the item translation of which was similar to that of the earlier version [12, 13]. The DREEM inventory consists of 50 items, each of which is scored from 1 to 4 on a Likert scale, giving a maximum score of 200. A score of 100 or lower indicates a poor environment for learning medicine. The five DREEM domains include students' perception of learning, perception of teachers, academic self-perceptions, perception of atmosphere, and social self-perceptions. The DREEM has been validated as a measure of learning environment effectiveness in several previous studies [8, 14].

The survey also included six questions related to preparedness for clinical competencies taken from the Association of American Medical Colleges Graduation Questionnaire (AAMC-GQ) after forward and backward translation into Japanese, for which we received permission from the AAMC. This questionnaire has been validated in several studies [15–17]. The questions represented a broad range of competencies for the following six clinical areas: 1) general clinical skills, 2) basic knowledge of diagnosis and management of common conditions, 3) communication skills, 4) skills for applying evidence-based medicine to clinical care, 5) professionalism, and 6) basic skills for physical examination. Participants rated their confidence in each of these areas using a five-point Likert scale: 1 (strongly agree), 2 (agree), 3 (neither agree nor disagree), 4 (disagree), and 5 (strongly disagree).

Data Analysis

Participant responses were grouped by university that they graduated from. Spearman correlation coefficients and linear regression were used for analysis of relationships between the mean DREEM score for each medical school and the proportion of respondents from that school with confidence in their preparedness for each of the six clinical skills (as indicated by an answer of "strongly agree" or "agree"). Similar analyses were performed for the relationships between the pass rates on the 2008 NMLE (data were obtained from the Ministry of Health, Welfare and Labor) and the proportion of respondents with confidence in their preparedness. Individual performance scores were not available for analysis. Statistical analyses were conducted using SPSS 15.0J (Tokyo, Japan), with a two-tailed value of p < 0.05 considered to be statistically significant.

Results

A questionnaire was sent to 6725 1st-year residents in 427 teaching hospitals in November 2008. A total of 2429 PGY-1 physicians-in-training (927 women, 38%) completed the questionnaire (response rate, 36%). Among the 80 medical schools from which the participants in the survey had graduated, the mean DREEM scores ranged from 95 to 137 (pooled mean 112; median 111; standard deviation 7.2). Three medical schools had a mean DREEM score of less than 100. The highest mean DREEM score was 137 (Table 1). The five subscales of the DREEM were highly inter-correlated (Table 2).

Table 1 DREEM score and confidence in preparedness for postgraduate training in six clinical areas in 80 medical schools (DREEM = Dundee Ready Educational Environment Measure, EBM = evidence-based medicine, PE = physical examination)

Table 2 Spearman correlation coefficients among total score and subscales of the DREEM*

Of the 2429 respondents to the survey, 406 (17%) agreed or strongly agreed that they were confident that they had acquired the general clinical skills required to begin postgraduate training. Overall, confidence in preparedness for postgraduate training in other clinical areas was indicated by 686 respondents (29%) for basic knowledge of diagnosis and management of common conditions, 1173 (48%) for communication skills, 465 (19%) for skills for applying evidence-based medicine to clinical care, 1317 (54%) for professionalism, and 904 (37%) for basic skills for physical examination.

Based on data for each school, the proportions of participants in the survey who were confident in their preparedness for postgraduate training ranged from 0% to 43% for general clinical skills; from 3% to 56% for basic knowledge of diagnosis and management of common conditions; from 19% to 77% for communication skills; from 3% to 42% for skills for applying evidence-based medicine to clinical care; from 27% to 83% for professionalism; and from 9% to 70% for basic skills for physical examination (Table 1).

A Spearman correlation analysis of the relationships between mean DREEM scores for medical schools and the proportions of participants with confidence in preparedness indicated significant relationships (all p < 0.01) in all six clinical areas (Figures 1, 2, 3, 4, 5 and 6), with positive coefficients of 0.425 for general clinical skills (R² = 0.218), 0.377 for basic knowledge of diagnosis and management of common conditions (R² = 0.160), 0.452 for communication skills (R² = 0.169), 0.402 for skills for applying evidence-based medicine to clinical care (R² = 0.236), 0.522 for professionalism, R² = 0.294), and 0.568 for basic skills for physical examination (R² = 0.393). A Spearman correlation analysis of the relationships between each of all five DREEM subscales for medical schools and the proportions of participants with confidence in preparedness also indicated significant relationships (all p < 0.05) in all six clinical areas.

Figure 1

The DREEM score and confidence in preparedness for postgraduate training in general clinical skills in 80 medical schools (DREEM = Dundee Ready Educational Environment Measure).

Figure 2

The DREEM score and confidence in preparedness for postgraduate training in basic knowledge of diagnosis and management of common conditions in 80 medical schools (DREEM = Dundee Ready Educational Environment Measure).

Figure 3

The DREEM score and confidence in preparedness for postgraduate training in communication skills in 80 medical schools (DREEM = Dundee Ready Educational Environment Measure).

Figure 4

The DREEM score and confidence in preparedness for postgraduate training in skills for applying evidence-based medicine to clinical care in 80 medical schools (DREEM = Dundee Ready Educational Environment Measure, EBM = evidence-based medicine).

Figure 5

The DREEM score and confidence in preparedness for postgraduate training in professionalism in 80 medical schools (DREEM = Dundee Ready Educational Environment Measure).

Figure 6

The DREEM score and confidence in preparedness for postgraduate training in basic skills for physical examination in 80 medical schools (DREEM = Dundee Ready Educational Environment Measure).

The pass rates for the 2008 NMLE ranged from 85% to 99%. A Spearman correlation analysis of the relationships between these pass rates and the proportion of participants with confidence in their preparedness indicated no significant relationship in any of the six clinical areas, with coefficients of 0.035 for general clinical skills (p = 0.758), 0.039 for basic knowledge of diagnosis and management of common conditions (p = 0.733), 0.123 for communication skills (p = 0.277), -0.077 for skills for applying evidence-based medicine to clinical care (p = 0.496), 0.078 for professionalism (p = 0.492), and -0.012 for basic skills for physical examination (p = 0.917). The mean DREEM scores and pass rates for the universities also showed no significant relationship (Spearman correlation coefficient = 0.209, p = 0.063).

Discussion

The current study showed that only 17% of our responding newly graduated physicians from all medical schools in Japan felt prepared by their schools in general clinical skills for their first postgraduate clinical training. Perceived preparedness for other clinical areas was similarly unsatisfactory. These results are in line with those of our recent pilot report indicating that Japanese medical graduates perceive that they are not well prepared clinically to start working as physicians-in-training, and that self-reported preparedness is significantly lower in Japanese graduates than in their US counterparts [18]. However, because of a limited ability to accurately self-assess among professionals [19], external assessment is needed for evaluating a real difference in preparedness between Japan and US.

The Model Core Curriculum has been introduced in undergraduate medical education throughout Japan. But medical schools in Japan vary greatly in the extent to which they have adopted the Model Core Curriculum, and there are still considerable differences in the duration and contents of clinical clerkships among medical schools [20, 21]. In many schools, students have limited opportunities to examine and interact with patients and little direct responsibility for patient care [22]. In this study, we found substantial differences among Japanese medical schools in self-perceived preparedness of their graduates, with significant positive correlations between self-assessed preparedness for all clinical areas and a better educational environment. These findings suggest that differences in educational environments among medical schools may be partly responsible for the differences in perceived preparedness of graduates for postgraduate clinical training. The implications of this finding and the need for improvement of the educational environment for medical students should be of concern to medical educators.

Engagement is a crucial step in learning that depends not only on the motivation and learning style of students, but also on the environment or "climate" in which the learning is taking place [23, 24]. Genn and Harden suggested that the educational environment is the soul and spirit of the medical curriculum and that establishing an effective environment is the most important single task of medical educators [10, 25]. Components of the educational environment related to the curriculum include the style (e.g., PBL vs. a traditional approach) and quality of teaching, signposting and clarity of the process, outcomes, assessment, and support mechanisms for students. Components of the educational environment related to individual teachers include teaching style, enthusiasm, physical environment, and role modeling [23]. Thus, our finding of significant positive correlations between perceived preparedness for all clinical areas and a better educational environment in medical schools reflects the importance of this environment in affecting the extent of engagement of medical students. Further research is needed to determine the components in an educational environment that are most strongly related to preparedness.

We found no significant association between the pass rate on the NMLE and perceived preparedness for any of the six clinical areas addressed in our survey. Traditionally in Japan, medical students have been required to attend lectures that were aimed at equipping the students to pass the NMLE (a paper-only test with a major emphasis on cognitive domains), even in the last two years of clinical training in a six-year medical school program. Some schools still emphasize this traditional curriculum, with little emphasis on the new Model Core Curriculum [20, 21]. However, the result of our study showing lack of association between the pass rate on the NMLE and perceived preparedness may indicate poor predictive validity of students' perceived preparedness for their first postgraduate clinical training against the NLME outcomes. This may also reflect relative insensitivity for identifying any association between the NMLE and perceived preparedness when using data about the pass rate. Thus, this result should be cautiously interpreted and there would be a need for further investigations such as a study using actual scores of the exam rather than pas-fail rates.

If our results would be confirmed using actual scores of the NMLE, a change of the contents and assessment method of the NMLE, such as introduction of clinical skills assessment or an OSCE-type test, could be considered for improving preparedness by helping schools to focus their training on this aspect. Another possibility for a change of the exam would be to administer the cognitive domain tests of the exam prior to entry into the clinical clerkship period in the last two years of medical school, similar to the United States Medical Licensing Exam (USMLE) Step 1.

There are several limitations in this study. First, our results might have been influenced by sampling bias, since the response rate was relatively low and many residents may have been too busy to respond to the survey. As we have conducted a cross-sectional survey for 1st-year resident physicians at teaching hospitals with five or more 1st-year resident physicians, we did not obtained data from teaching hospitals with four or less 1st-year resident physicians. Thus, we did not know the range of response rates across schools and gender ratio in all PGY-1 population. However, our sample size was large and we were able to examine a nationwide sample. Although Japanese Ministry of Health, Welfare and Labor does not open data related to individual residents, based on data about 46,800 medical students in 2006, 33% were women [26]. So our sample (38% were women) seemed not much different from the total population.

Second, our survey only investigated each resident's perception of their preparedness, rather than using clinical or objective assessment of preparedness. Thus, we cannot determine the differences in outcomes between universities objectively. Finally, because of the cross-sectional observational nature of the survey, we are unable to determine a causal link between the educational environment and preparedness, since confounding factors such as personal characteristics of students that may be related to both environment and preparedness may have influenced the results.

Conclusions

In summary, Japanese medical graduates perceive that they are not well prepared clinically to start working as physicians-in-training. There were substantial differences among Japanese universities in the perceived preparedness of their graduates. Significant positive correlations were found between perceived preparedness for all clinical areas and a better educational environment, but there was no significant association between the pass rate on the National Medical Licensure Examination and perceived preparedness for any clinical areas. The link between self-perceived and independently observed preparedness needs to be established. Then, further research will be required to evaluate the most important components in the educational environment that influence preparedness. Japanese medical schools may need to consider establishment of minimum requirements for clinical competencies for their medical graduates.