BMC Medical Education - Latest Comments

Does association mean causality?

George Somers — 2013-04-26T14:47:42Z

Thank you for this paper. Unfortunatley, only the abstract is available at this stage, the full paper, when it becomes available, might explain my question.
From the abstract, it seems that when you conclude that increasing the length of rural rotations to three years 'significantly increases the likelihood of rural career intentions of non-rural students' you are implying causality.
If you merely found that an increased association was observed, have you shown that it was the rural experience that caused the increase in intention? Other readers of your abstract might also infer that causality was shown. Perhaps this is only my interpretation of your sentence, however such ambiguity cannot be ignored.
In a recent study I found that the intention of those who chose to undertake longer rural rotations was already higher at entry to the rural program. I found that the decrease in rural intention observed in the whole cohort over the three years was not influenced by length of rural rotaton. I concluded that it was this stronger interest in a rural career that motivated students to undertake longer rural rotations not the other way around. (Somers GT, Spencer RJ. Nature or nurture: The effect of undergraduate rural clinical rotations on pre-existent rural career choice likelihood as measured by the SOMERS Index. Aust J Rural Health. 2012;20:80-7. Epub 7 January 2012.)
I raise this issue because most papers written on this subject have been produced by and refereed by employees of Schools of Rural Health with a vested interest, raising the risk of publication bias.

The true costs of simulation equipment

Kieran Walsh — 2013-02-15T14:54:35Z

Dear Editor,
Tran and colleagues provide a fascinating insight into how best to deliver simulation training that is both highly effective and low cost (1). They give a clear outline of some of the potential short-comings of the study in the discussion but surprisingly do not discuss costs in any great detail. The authors claim that the cost of producing their model was 5 USD and compared this with the commercial model which was available at a cost of 300-400 USD. However this may not be a fair comparison. The 300-400 USD is likely to be the price of the commercially available model ¿ the cost of producing it is likely to be less than this. Also the cost of producing their model at 5 USD seems remarkably low. It would be helpful to see a breakdown of such costs. Is this simply the cost of manufacture or does it include the cost of research, development, design, planning, road-testing, evaluation and quality assurance of the new model? Does it include the costs of space or facilities required for manufacture (even if these rooms were part of the medical school)? These are all costs that the commercial manufacturer would have to account for and so it seems only fair that the medical school should account for these also. Also the teacher made model was used by attaching it to patients - did the authors take into account the cost of patients in their analyses? It is likely that producing the home-grown model was done at bulk and that this enabled them to get the cost per model so low, however it would be helpful to know how many models needed to be produced to achieve this level of cost.

It is almost certain that the teacher made model cost less than the commercially available model ¿ but we need more data to be able to say for certain whether the cost of the teacher made model was less than 1.7% of the commercially available one.
Yours Sincerely,
Kieran Walsh

References

1. Tran TQ, Scherpbier A, Van Dalen J, Wright PE. Teacher-made models: the answer for medical skills training in developing countries? BMC Med Educ. 2012 Oct 19;12(1):98. [Epub ahead of print]

CREATE Slide for public use

Julie K. Tilson — 2012-09-18T17:05:01Z

Use this link to access a slide of the CREATE framework from this paper:

http://pt.usc.edu/files/create_slide_from_Tilson_et_al.pptx

A bigger study is required to confirm the hypothesis

Suvash Shrestha — 2012-05-16T09:49:47Z

I read the article with great interest and felt it addressed very important and relevant topic. However, I would like to point out few things about the study that I believe have reduced the credibility of the findings.

1. The sample selection poses a huge problem. Not only the sample size is small but it is also biased. The findings or the opinions are of the residents in one center among many in New York, leave aside the other hospitals in other states. Residents in community hospitals in other places may have different opinion about the fellowship.

2. The number of residents in "Interested in fellowship" and "Not interested in fellowship" is highly unequal with 37 out of 45 (82%) in the "Interested in fellowship" group. The fact that these two groups are highly unequal makes the comparison between these two groups highly biased and unrealistic.

3. As the authors have also mentioned, currently the research respondents are residents, they are yet to join fellowships, and whatever answers they gave are just ideas and not necessarily what they are going to do in future. This difference in thought and action also reduces the significance of the findings. In my opinion, a retrospective case control study which would enroll doctors doing fellowships at the moment and doctors not doing fellowships and look into their past examination reports would better answer the hypothesis.

4. As the authors have also mentioned, the association should not mean causal relation. The relation could be both ways - academically sound residents go for sub speciality fellowship, while academically poor residents are left behind or residents with interest for fellowships work harder so they score high in the exams.

Comment on: BMC Med Educ. 2011 Oct 3;11(1):76. [Epub ahead of print] The effect of an intercalated BSc on subsequent academic performance. Mahesan N, Crichton S, Sewell H, Howell S.

Martin Ho Yin Wong — 2011-11-08T15:49:08Z

To the editor:

I read the recent paper by Mahesan et al1 with great interest, and wondered if the authors had considered several possible confounders to their study. Indeed, it is logical that there may be an association between the two variables in the title of this article, but one has to be cautious whether there is causation.

For example, at King’s College London where this study is based, doing an intercalated BSc (iBSc) is optional for all medical students, and so one may think that those who choose to do one are already more academically-oriented, determined and motivated than those who do not, and so may score higher in subsequent exams and be positioned in a higher academic quartile because of this principle rather than the BSc itself leading to subsequent higher exam performance.

At Barts and The London School of Medicine and Dentistry, the iBSc is not offered to all medical students; there is a selection criteria based on their prior academic performance at medical school.2 This means that these intercalating students have already been ‘pre-selected’ and may be likely to score higher in subsequent exams and end up in a higher academic quartile.

At University College London3 and Imperial College School of Medicine4, the iBSc is compulsory for all medical students, and so possibly the next step to this study would be to analyse similar data from these two universities and to compare and contrast the results. This would remove the optional element of the variable and may help to determine whether the iBSc itself does in fact lead to subsequent higher exam performance or not.

Regarding the exact correlation between having done an iBSc and subsequent overall Foundation School outcome as mentioned in this study, one has to first understand exactly how the Foundation applications are scored.5 Currently for Foundation programmes commencing in August 2012, students will be ranked according to their overall application score and jobs allocated accordingly. The overall score is out of 100 and is divided into 3 main sections: (1) academic quartiles – max 40 points (1st quartile 40, 2nd quartile 38, 3rd quartile 36, 4th quartile 34), (2) other educational achievements other than the primary medical degree – max 10 points, (3) ‘white-space questions’ – max 50 points. In relation to the terms the authors have used in their article relative to the terms above, ‘academic quartile score’ refers to (1) and ‘application form score’ refers to (2) and (3) combined. In this study, the authors did not separately analyse (2) and (3).

Of particular relevance is section (2), ‘other educational achievements’. This comprises of ‘additional degrees’, where a doctoral degree (PhD, DPhil etc) is allocated 5 points (not relevant here), a 1st class honours is allocated 4 points, 2.1 class allocated 3 points, 2.2 class allocated 2 points and 3rd class allocated 1 point. The fact that a medical student may have an iBSc automatically means that they will be allocated more points in this category compared to a student without an iBSc, and so is more likely to have a higher overall application score and subsequently more likely to secure their first choice Foundation School. One can say that a statistical analysis was not needed to arrive at this conclusion, due to the fact that this is simply how the scoring criteria works. Furthermore, the other 5 points in section (2) relate to PubMed ID publications, national/international presentations and national/international educational first prizes.5 Again, one can state that undertaking an iBSc opens up many opportunities for a student to obtain these additional ‘educational achievements’, which may otherwise not be as easy for a student who had not undertaken an iBSc, and that a statistical analysis was not needed to arrive at this conclusion. Perhaps one way to overcome this ‘confounder’ is to just analyse similar data relating to sections (1) and (3) taken from a university where the iBSc is compulsory, which may give a more accurate answer to the authors’ study aims.

In conclusion, although I agree that an iBSc and its associated opportunities will increase the likelihood of a student obtaining a higher overall Foundation score and securing their first choice Foundation School, simply due to the way the Foundation Programme scoring system currently works, I believe more research needs to be carried out to conclude exactly whether ‘doing an intercalated BSc leads to an improvement in subsequent exam results’, as stated by the authors. Furthermore, the ‘white-space questions’ could be analysed separately from the ‘educational achievements’ section in order to discover if iBSc students do develop skills that enable them to obtain higher scores in this section of the Foundation application process.

References

1. Mahesan N, Crichton S, Sewell H, et al. The effect of an intercalated BSc on subsequent academic performance. BMC Med Educ [Online] 2011; 11(1):76. Available from: http://www.ncbi.nlm.nih.gov/pubmed/21967682. [Accessed 8 October 2011].
2. Barts and The London School of Medicine and Dentistry. Intercalated degrees. [Online]. Available from: http://www.smd.qmul.ac.uk/undergraduate/intercalated/apply/index.html. [Accessed 8 October 2011]
3. UCL Medical School. BSc and IBSc. [Online]. Available from: http://www.ucl.ac.uk/medicalschool/bsc-ibsc. [Accessed 8 October 2011]
4. Imperial College London Faculty of Medicine. Studying Medicine – 6 year MBBS/BSc. [Online]. Available from: http://www1.imperial.ac.uk/medicine/teaching/undergraduate/medicine/ [Accessed 8 October 2011]
5. The UK Foundation Programme Office. FP 2012 Applicant’s Handbook. [Online]. Available from: http://www.foundationprogramme.nhs.uk/pages/home. Accessed 8 October 2011]

Correction to Table 3

Michael Fetters — 2011-01-11T15:18:23Z

Michael, I found your recently published article in BMC Medical Education (2010, 10:50) very interesting. I have started using Second Life, an online virtual world, for crime scene simulations in my forensic science class.

I am writing because I am having a little difficulty interpreting table 3 in your article. I printed your article from biomedcentral.com. Even though it is not clear, it looks like the first three columns are those
students who agreed, and the next three columns are for those who disagreed. I think some data for the fourth survey question, "I feel that video games can have educational value" are transposed. The totals
for males and females do not add up to 100%, e.g., 89% (for males that agree) + 27% (for males that disagree).

In the text you state that females are only about 39% as likely as males to believe in the potential educational value of video games, and females are about 31% as likely as males to want to be part of a team to design an educational video game. Can you explain how you determined these percentages?

thanks,
--

Don Lehman, Ed.D., MT(ASCP), SM(NRM)
Dept. Medical Technology
University of Delaware
Newark, DE

Dr. Lehman is correct. With reference to Table 3: thank you for pointing it out. There is indeed a typo in Table 3: Q4 (I feel that video games can have educational value) two percentages (11 and 27) under Disagree M and F are interchanged by mistake.

As for your question how we determined the percentages towards the statement that females are only about 39% as likely as maless ... etc, please consider the following: Binary logistic regression has been used to model the dichotomous outcomes with gender, institutional affiliation, familiarity level (basic/intermediate or advanced), age as independent variables.

Thanks to Dr. Lehman on behalf of all the authors.

Michael D. Fetters, MD, MPH, MA

Comments and Questions

Anees Sindi — 2009-01-23T16:48:28Z

Dear Dr. Bajammal and Colleagues,

Thank you for publishing this well written article. I have some comments and questions:

1. Page 14:

“Given the comprehensiveness and standardization in the planning, writing and conduct of these rigorous examinations, these licensing examinations can eventually replace the multiple exams required currently from medical students which include the final year examinations, the SCHS’ “Acceptance Test”, and the specialty-specific screening exams.”

Do you think replacing the final year examination is something doable given that the grading system is different in each university? If yes, how?

Should it just replace the SLE and the specialty-specific exams instead?

2. Page 15

“The established medical schools are at a disadvantage due to relocation of some of their experienced medical educators to the new schools, while the new medical schools are often disadvantaged due to being “new” in the business.”

One more point, this test will help us evaluate the newly established private medical schools and compare their graduates to the government-based medical schools.

3. Page 17:

“Scholarships for undergraduate medical students:

The expansion of international scholarship of undergraduate medical studies will bring many Saudi physicians with variable qualities of undergraduate medical education and from different cultural backgrounds into the country. The licensing exam will serve as a method of unifying the minimum standard of acceptable competency in the health service among these graduates.”

I think the test should be done before they go abroad instead of doing it after years of practice outside the country. It may not be easy to convince somebody who is already board certified to write such a basic test after 7-10 years of training in what is supposed to be a highly qualified center.

4. Page 19:

“The standardization of the assessment of medical students prior to their entry into medical practice, whether they join residency or not, is needed to complete the first and last step of the standardization process already in practice.”

Strongly agree since we started to see more Saudi graduates who are not interested in joining a local program or waiting to be accepted somewhere abroad.

5. Page 21:

“ The SCHS’ “Acceptance Test” is not suitable to provide this information for medical schools because only students who want to enroll in a Saudi residency program sit for this test.”

Is there any data/study looked at the score of this exam among different medical schools graduates who were enrolled in a local program?

6. Page 21:

“Exposing potential weakness in medical schools education system:

Medical schools might resist this call for change fearing that they will be exposing the weaknesses of their medical schools publicly.”

I personally don’t think that this should be an issue. I think it is going to be a very good assessment tool for the Ministry of Higher Education (MOHE) to evaluate our medical schools, new vs. old, government vs. private.

7. Page 21:

“The fear of impeding flexibility within medical school’s curriculum:

Medical schools educators might resist the change because they believe that national learning outcomes and a national standardized examination will restrict the flexibility of customizing their curriculum and choice of assessment methods.”

If this test got approved by the MOHE as an assessment tool then no body can complain or resist.

8. Page 22:

“Language barrier for non-Arabic speaking physicians

English is the language of instruction and examination in all medical schools in Saudi Arabia. Implementing a mandatory OSCE part which entails communicating with patients, who are mostly Arabic-speaking, might be a major issue for non-Arabic speaking physicians.”

Excellent point. Strongly agree. This may avoid a lot of the miscommunication we have been seeing in our hospitals. I would add the following point:

6. Religious Aspects for non-Muslims physicians:

This should include how to communicate with a female Muslim patient from the religious point of view, end of life decisions from the religious point of view.

Sincerely,

Anees A.Ramadani Sindi, MBChB, ABIM

Clinical Fellow, Division of Respirology

McMaster University

Hamilton,ON

Canada

Demonstrator, Division of Critical Care Medicine

Faculty of Medicine, King Abdulaziz Univeristy

Jeddah, Saudi Arabia

Congratulation and a comment

Abdulwahab Telmesani — 2009-01-22T17:17:47Z

Good job for Dr Sohail Bajammal and the group. A well written article in the sense that the idea was discussed from different angles and a semi SWOT analysis was conducted. I totally agree with the idea of having a licensing examination in Saudi Arabia. I can give one more reason that is very difficult to argue against. Few days back, one of the local news papers reported the discovery of many health care workers ,from doctors to nurses, who were holding fake certificates and practicing medicine for sometime i.e. fake doctors and nurses?!!. A licensing exam would have identified and checked such a crime. The proposed format of the exam is excellent, but we need to take in consideration the contest of the governing bodies of the heath care and practice i.e., choosing the appropriate format and implantation of the exam that fits the unique circumstances of country. I believe that the SCHS is taking a steady steps in the right direction toward the licensing exam. It is a matter of time and triggers like your proposal. I expect the SCHS giving the proposal the right attention and probably form a task force to put together a format(the proposed or modified one) for a licensing exam for any physician wish to practice in the Saudi Arabia. Abdulwahab Telmesani School of Medicine Umm Al-Qura University Makkah, Saudi Arabia

Computer adaptive testing

Mohammed saqr — 2009-01-22T17:16:42Z

The adaptive mastery testing (AMT) procedure developed by Kingsbury and Weiss (1979) is designed to make high-precision classifications concerning students’ mastery of specific content areas within a course of instruction. The procedure is also intended to minimize the number of test questions needed to make these classifications in order to increase the amount of class time available for actual instruction. The AMT procedure makes use of item response theory (IRT; Lord, 1980; Lord & Novick, 1968) to adapt the test items administered to suit each student. That has not become practical until computers became more prevalent with time. And generated what is known as computer adaptive testing. IRTs probability-based estimation strategy is what makes CAT possible. A measure is adaptive only when the selection of items is adapted to prior estimates of the respondent’s level of the construct being measured. After thorough evaluation of their clarity, content, sensitivity, and other properties, have been calibrated to an IRT model. The developmental items must be administered to a large sample of persons who are representative of the population of interest. These responses are used to tests the assumptions of IRT and to calibrate the items to an IRT model. CAT relies on large banks of previously collected and calibrated responses as described, which means that banks will need time to develop first and would be exposed to large number of students, and hence will be consumed earlier contrary to what is proposed in the paper. The preparation of the bank is a very complicated process and requires lots of time, technical experience, time and technology. Much to our dismay only few models do actually exist in real life. A fact that complicates matters further. Unfortunately, the technology and infrastructure needed to implement CAT are currently beyond the reach, Existing CAT software is highly technical. And needs much expertise to run, Most CATs are developed “in house.” Which means every new implementation will start from the ground up, an unwarranted and arduous process to begin a bank with. The relatively high start-up costs which includes research, programming, bank development, training and? Outcome assessment of the methodology makes CAT impractical as a startup project. Another expense in implementing CAT is the “delivery device.” Because CATs are administered by computer, either respondents or an interviewer must interface with a computer. In addition to being expensive, these interfaces have other drawbacks. Although the methodology is there more than three decades ago, it has not gained much acceptance in academic settings,. Nonetheless few examples do exist; the NCLEX is an example. However the NCLEX is mainly a pass/fail exam and level of proficiency is hard to get using their methodology. Using computer adaptive testing Increases measurement efficiency. And in many occasions saves time, we argue here that using it in a Saudi Bank for medical examination is not only difficult, but could be problematic and may not be the best option. Further research is needed and time before effective banks and exams could be created using CAT. References 1-Karon F Cook, Kimberly J O'Malley, and Toni S Roddey. Dynamic Assessment of Health Outcomes: Time to Let the CAT Out of the Bag? Health Serv Res. 2005 October; 40(5 Pt 2): 1694–1711. 2-Jette AM, Haley SM, Ni P, Olarsch S, Creating a computer adaptive test version of the late-life function and disability instrument. Moed R.J Gerontol A Biol Sci Med Sci. 2008 Nov;63(11):1246-56.Click here to read Links 3-McGlohen M, Chang HH. Combining computer adaptive testing technology with cognitively diagnostic assessment. Behav Res Methods. 2008 Aug;40(3):808-21.

The references don't show that CFS "affects at least 4 million adults in the United States"

Tom Kindlon — 2009-01-15T16:56:13Z

This is not necessarily a major point to do with the methodology. However given the paper involves CFS medical education, and gives few facts about CFS, one would think that the statements that are made are at least reasonably accurate.

However, this statement clearly isn't: "CFS affects at least 4 million adults in the United States [2-4]."

The references given are the first three references below. The second study found a prevalence of 422 per 100,000 adults and the third study found a prevalence of 235 per 100,000 adults. These aren't even close to 4 million adults - the second one suggests a figure of at least 400,000 adults. This study involved one of the authors (William C Reeves).

The other study[1], also involved Reeves, did find a prevalence of 2540 per 100,000 adults, which would equate to over 4 million adults. As can be seen, this is much higher than previous estimates of the prevalence of CFS in the US. The reason for the huge discrepancy is largely because it used a different method of defining CFS[4].

There has been some criticism of this new definition[5].

Unlike previous times when the CDC produced definitions for CFS[6,7], the definition used in this study is generally only being used by the CDC-funded CFS research team [the cohorts from the Wichita 2-day study in 2003 (not to be confused with the Reyes study) and the Georgia prevalence study[3]]. So it's far from clear that most people would accept that CFS "affects at least 4 million adults in the United States". But certainly two of the three studies given to back up this reference did not find anything close to such a prevalence.

[1] Reeves WC, others: Prevalence of chronic fatigue syndrome in metropolitan, urban, and rural Georgia. Population Health Metrics 2007, 5:5.

[2] Jason LA, Richman JA, Rademaker AW, Jordan KM, Plioplys AV, Taylor RR, McCready W, Huan CF, Plioplys S: A community-based study of chronic fatigue syndrome. Arch Int Med 1999, 159:2129-2137.

[3] Reyes M, Nisenbaum R, Hoaglin DC, Emmons C, Stewart G, Randall B, Stewart JA, Abbey S, Jones JF, Gantz N, Minden S, Reeves WC: Prevalence and incidence of chronic fatigue syndrome in Wichita, Kansas. Arch Intern Med 2003, 163:1530-1536.

[4] Reeves WC, Wagner D, Nisenbaum R, Jones JF, Gurbaxani B, Solomon L, Papanicolaou DA, Unger ER, Vernon SD, Heim C: Chronic Fatigue Syndrome – A clinically empirical approach to its definition and study. BMC Medicine 2005, 3:19 (15 December 2005)

[5] Jason LA, Richman JA: How Science Can Stigmatize: The Case of Chronic Fatigue Syndrome. Journal of Chronic Fatigue Syndrome, Vol. 14(4), 2007

[6] Fukuda, K., Straus, S.E., Hickie, I., Sharpe, M.C., Dobbins, J.G., & Komaroff, A. (1994). The chronic fatigue syndrome: A comprehensive approach to its definition and study. Annals of Internal Medicine, 121 (12):953-959. http://www.annals.org/cgi/content/full/121/12/953

[7] Holmes GP, Kaplan JE, Gantz NM, Komaroff AL, Schonberger LB, Straus SE, et al. Chronic fatigue syndrome: a working case definition. Ann Intern Med. 1988; 108:387-9.