Email updates

Keep up to date with the latest news and content from BMC Health Services Research and BioMed Central.

Open Access Research article

Clinical decision support must be useful, functional is not enough: a qualitative study of computer-based clinical decision support in primary care

Tiina Kortteisto1*, Jorma Komulainen2, Marjukka Mäkelä3, Ilkka Kunnamo4 and Minna Kaila5

Author affiliations

1 School of Health Sciences, University of Tampere, Tampere, 33014, Finland

2 The Finnish Medical Society Duodecim, PO Box 713, Helsinki, 00101, Finland

3 Finnish Office for Health Technology Assessment at National Institute for Health and Welfare, PO Box 30, Helsinki, 00271, Finland

4 Duodecim Medical Publications Ltd, PO Box 713, Helsinki, 00101, Finland

5 The Hjelt Institute, University of Helsinki, PO Box 41, Helsinki, 00014, Finland

For all author emails, please log on.

Citation and License

BMC Health Services Research 2012, 12:349  doi:10.1186/1472-6963-12-349


The electronic version of this article is the complete one and can be found online at: http://www.biomedcentral.com/1472-6963/12/349


Received:13 December 2011
Accepted:5 October 2012
Published:8 October 2012

© 2012 Kortteisto et al.; licensee BioMed Central Ltd.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Background

Health information technology, particularly electronic decision support systems, can reduce the existing gap between evidence-based knowledge and health care practice but professionals have to accept and use this information. Evidence is scant on which features influence the use of computer-based clinical decision support (eCDS) in primary care and how different professional groups experience it. Our aim was to describe specific reasons for using or not using eCDS among primary care professionals.

Methods

The setting was a Finnish primary health care organization with 48 professionals receiving patient-specific guidance at the point of care. Multiple data (focus groups, questionnaire and spontaneous feedback) were analyzed using deductive content analysis and descriptive statistics.

Results

The content of the guidance is a significant feature of the primary care professional’s intention to use eCDS. The decisive reason for using or not using the eCDS is its perceived usefulness. Functional characteristics such as speed and ease of use are important but alone these are not enough. Specific information technology, professional, patient and environment features can help or hinder the use.

Conclusions

Primary care professionals have to perceive eCDS guidance useful for their work before they use it.

Background

Health information technology (HIT), particularly electronic decision support systems [1,2], can reduce the existing gap between evidence-based knowledge and health care practice [3,4]. Benefits of computer-based clinical decision support (eCDS) to the quality of care and patient safety depend on the health care professionals’ acceptance and use of the eCDS in their decision making [5]. Automatic provision of eCDS in the workflow was found to be an independent predictor of improved clinical practice [6]. This means that eCDS guidance is provided automatically via electronic patient records (EPR) without extra effort by the professional [7]. Speed, real time delivery, and fitting in the workflow are important features for effective eCDS [8]. Too much guidance, e.g. drug interaction alerts, can be intrusive [9,10]. There is little evidence on eCDS in primary care or in settings where a variety of clinical areas need to be covered [11,12].

Health care professionals prefer patient-specific and relevant guidance, provided in a way that does not interfere with care or require inordinate effort and time [13,14]. In practice, plenty of factors influence the use of guidance [15]. Characteristics of the patient and the environment may be the most important factors for the acceptance of eCDS guidance [16]. Many barriers have been identified [17,18], particularly, too frequent or false alarms, lack of co-ordination between nurses and physicians, poor interface usability, time pressures, and inadequate training [19-22]. Facilitators include limiting the number of reminders, their ease of use and utility, documenting system problems and receiving feedback [21,23].

In Finland, electronic patient records and national guideline databases have long been used, which makes piloting a new HIT in primary care environment feasible [24]. Despite this, our implementation study of 18 months indicated that actual use of eCDS in the primary care context was only modest [25]. Therefore, we examine in this paper the professionals’ feedback more thoroughly. The main purpose was to increase our understanding for improving the application of the eCDS in the workflow of physicians, nurses, and other professionals [26]. The aim was to assess and describe in depth the specific reasons for using or not using the eCDS in primary care.

Methods

Study context

In one Finnish primary health care centre, patient-specific decision support (consisting of drug interaction alerts, reminders, and links to diagnosis-based guidelines) was shown automatically via EPR to physicians, nurses, and other professionals at the time of opening the record, entering a new diagnosis, or prescribing a drug (Table 1, see screen shots in additional files 1, 2,and 3). In addition, by selecting a specific function in the EPR, the physicians could see all reminders specific to the patient under their names on the relevant day’s appointment list (= virtual health check, VHC). (see description of the eCDS system [27] in additional file 4). Practical inclusion criteria for selecting the health centre were stable use of the EPR which included laboratory measurements, and recording of core patient information by the professionals, e.g. diagnoses and medications. The study protocol was approved by the ethics committee of the Pirkanmaa hospital district.

Additional file 1. Patient specific automatic reminders and guideline links on the electronic patient record screen: a patient with diabetes as an example.

Format: PDF Size: 83KB Download file

This file can be viewed with: Adobe Acrobat ReaderOpen Data

Additional file 2. Patient specific drug interaction alert function on the electronic patient record screen.

Format: PDF Size: 40KB Download file

This file can be viewed with: Adobe Acrobat ReaderOpen Data

Additional file 3. Virtual health check results within the physician’s appointment schedule.

Format: PDF Size: 121KB Download file

This file can be viewed with: Adobe Acrobat ReaderOpen Data

Additional file 4. Computer-based Clinical Decision Support: An overview of the Evidence-Based Medicine electronic Decision Support (EBMeDS) architecture and decision support rules (= scripts).

Format: PDF Size: 43KB Download file

This file can be viewed with: Adobe Acrobat ReaderOpen Data

Table 1. Computer-based clinical decision support (eCDS) functions (drug alerts, reminders, guideline links, and virtual health check = VHC) and procedures: examples

Participants

We targeted the 48 professionals of the health centre; 15 physicians, 24 nurses (ward nurses, general practice nurses, and public health nurses) and nine other professionals (physiotherapists, head nurses, and a psychologist). All signed an informed consent form. During the study, four physicians left and four physicians entered the health centre.

Data collection

Data collection took place between July 2009 and February 2011, by means of focus group interviews, a questionnaire, and spontaneous feedback.

Focus groups

In January 2010, six months after introduction of the eCDS to the primary care professionals, we convened three focus groups with the help of the chief officers. We aimed at involving as many physicians as possible, with at least one representative from preventive care, nurse practice, physiotherapy, and the two inpatient wards. Six physicians, five nurses, and one physiotherapist participated in one of two profession-specific groups or in one multidisciplinary group. The agenda was planned in advance by the moderator (TK) and facilitator (MK) and tailored to each group [28]. The aim was an open, informal discussion among participants about their experiences of the eCDS [29]. Broad themes such as general ideas about the eCDS, experiences of the use, practical problems, advantages /disadvantages for work, barriers to use and facilitators, and development issues guided the discussion. Each group discussion was audio recorded, transcribed, and the facilitator took notes on the general atmosphere and various pertinent issues [30].

In February 2011, at the end of the implementation period, a fourth focus group was convened for physicians. At this point, we wanted to gather opinions and experiences on the eCDS functionalities that were only available to physicians; 13 of the 15 physicians participated. The agenda was again planned ahead, using the previous broad themes, adding content and thresholds of reminders, by the moderator (JK) and facilitator (TK). JK acted as moderator, since his medical background was expected to facilitate discussion [31]. The aim was to obtain authentic feedback on the eCDS in physicians’ work. The group discussion was audio recorded and notes were made in a memo by a secretary.

Questionnaire

A questionnaire was sent to professionals in September 2010. After one reminder, 28 of the 45 professionals (62%) had responded; nine physicians, twelve nurses, two public health nurses, and five others. The six structured questions reported in this paper dealt with the system’s capacity and quality, as well as its perceived usefulness and ease of use, according to the technology acceptance model (TAM) [32]. It hypothesizes that perceived usefulness and ease of use are the main determinants of behavioural intention to use IT, which is expected to lead to actual use [33]. In this study, the perceived usefulness indicates the professional’s subjective probability for increasing her or his work performance by using the eCDS system. The perceived ease of use indicates the professional’s feelings for new system to be free of effort. In addition, reasons for using or not using the eCDS functions were queried in an open question: twelve out of 28 respondents (43%) elaborated their reasons.

Spontaneous feedback

The professionals were encouraged from the beginning to provide feedback via personal email, whenever they recognized specific issues or questions for system developers or researchers. The vendor added a feedback channel within the EPR-system from January 2010. Twelve spontaneous feedback messages were submitted by physicians; two via the personal email and ten via the EPR.

Analyses

Two researchers (TK and MK) analyzed the qualitative data (focus group interviews, open question, and spontaneous feedback) by using deductive content analysis for testing previous theoretical issues in order to enhance understanding of the phenomenon [34]. The principle derived from the literature [35] was to identify features that helped or hindered the use of the eCDS, and how this was justified by the participants [36]. We first independently coded a categorization matrix consisting of the helping and hindering features. Next, we discussed the initial themes and grouped emerging themes by theory-based category (Table 2). The information in each category was condensed, reflected on, and interpreted jointly. Quantitative data were analyzed by using the software SPSS for Windows, version 15.0. The small sample size did not permit robust statistical analysis.

Table 2. A theory-based framework of analysis with emerging sub-themeshelping (+) or hindering (−), quotations are described in text

Results

In the end, the qualitative and quantitative analyses were combined. The main categories were the content of the eCDS guidance, the functionality of the eCDS system, the features related to the professions involved, the features of the patient groups, and environmental factors. A unique category of development issues emerged.

The content of the eCDS guidance

The content of the guidance appeared to be a significant factor in the professionals’ decision to use eCDS in their work. Perceived usefulness was an essential reason for using eCDS in general or its particular function, e.g. drug alerts. Five out of nine physicians perceived the guidance as useful in their practice and as affecting their decisions, while one physician did not (Table 3). When asked about the usefulness of the eCDS, the majority of the nurses responded ‘I cannot say’. The nurses in preventive care, the physiotherapists, and the psychologist considered the eCDS guidance, mainly based on drug alerts, as unhelpful for their work. Reliability and quality were perceived as facilitating use (Table 3).

Table 3. Perceived functionality and usefulness of the computer-based clinical decision supportnumber of responses in the questionnaire

The physicians’ consensus was that the guideline links and VHC were not useful; the guideline texts were too long, and use of the VHC function would have necessitated more time for paper work. Only physiotherapists considered the guideline links useful.

From the physicians:

"‘Good system; it does not disturb, some reminders are useful.’"

"‘Interaction alerts with a prescription are useful.’"

"‘Drug alerts motivate to clear up medication lists, but it takes time.’"

"‘I think that there is too much text … too many links.’"

"‘Use of virtual health checks requires more time for paperwork.’"

From the other professionals:

"‘Guideline links are useful for the physiotherapist.’"

"‘For our work [health promotion] and customer relationships, there aren’t necessarily relevant things.’"

"‘Mainly drug interaction alerts, and I [nurse] do not prescribe.’"

"‘Okay, good information but it does not influence my work much in practice, how you use it depends on the group of professionals you belong to.’"

"‘Most of the drug alerts are not relevant for physiotherapists’ work.’"

"‘Reminders do not support psychologist’s work’"

The functionality of the eCDS system

The majority of the physicians and other professionals found the ease of use and speed good, while most of the nurses expressed no opinion, stating that they could not answer these questions (Table 3). The physicians reported that specific features of the system (EPR, eCDS, or their integration) were hindering the use. These included known problems with the EPR system, reminders’ position on the screen, irritating drug interaction and contraindication alerts, and a lack of structured recording of smoking status (causing inappropriate reminders). The nurses experienced the eCDS as impractical for work at a busy call centre.

From the physicians:

"‘There was generally a problem with the functioning of the electronic patient record system … yes, big problems with the computer.’"

"‘Reminders’ position on the left side of the screen.’"

"‘If a patient has 20 reminders, I just go past them quickly.’"

"‘Reminders’ texts are sometimes too strict in the short version. If you don’t move the cursor over the text and see the whole reminder, the wording doesn’t work.’"

"‘Too low triggering threshold with drug interaction alerts. I never bother to read them.’"

"‘Irritating alerts.’"

"‘Excess alerts – e.g., asthma and opiate, warfarin and paracetamol.’"

"‘The reminder “great cardiovascular risk [SCORE], clarify the patient’s smoking status…”. It is not possible to record smoking status in a structured manner in the EPR.’"

From the nurses:

"‘We [general-practice nurses] receive such a staggering number of phone calls that we really don’t look for anything, just do exactly what they [patients] call about.’"

The features related to professionals

The health care professionals’ motivation to learn to use a new IT system in practice seemed to vary. Some nurses discussed inadequate skills or interest in using IT in general. Primary care professionals seemed to have generally positive attitudes toward future use of eCDS, but more training and time to learn to use a new system were described as necessary. In addition, it was suggested that a local opinion leader could be useful to teach others.

From the physicians:

"‘I think that we should use it more.’"

"‘Do not need. I use Terveysportti [national health portal for professionals].’"

"‘I get an exact search when I use Terveysportti.’"

"‘Should there be an enthusiast or someone who uses it and thinks about it, and then tells us what the tricks are, what is good?’"

From the nurses:

"‘We have certainly more potential than we have been used to … Yes, it is absolutely a good thing … Yeah, I would say the same, learn to use it and intensify its use then’"

"‘All experienced this training as necessary.’"

"‘The individual differences, where you are used to searching for information… e.g., poor IT skills or no interest in the computer.’"

Features of the patient groups

The patient’s agenda for the visits was the most important issue for the physicians. It was common that the eCDS guidance was ignored because it did not address the reasons for the encounter. The physicians’ spontaneous feedback indicated another factor, in overly sensitive patient-specific triggering cut points and irrelevant reminders.

"‘The patients usually have a reason for their visit. This has to be the primary focus for the physician. If there are ten VHC reminders even before the patient arrives, there is in no way enough time.’"

"‘My patient had a fasting plasma glucose value of 6.0 once, previously normal. Decision support suggests diabetes. To my mind, a single FPG value of 6.0 does not justify diagnosing diabetes.’"

Environmental factors

An unforeseeable external factor arose during the study, the swine-flu epidemic with concomitant large-scale vaccination of the population, resulting in changes to at least nursing practice for many months and clearly causing extra work. A common barrier to use was busy practice in primary care. In particular, nurses reported that changes in their practice would have been needed before they could use the system.

"‘The swine-flu epidemic has consumed an awful lot [of time], we didn’t even have any general-nurse practice appointments available for a long time.’"

"‘When I am busy, I don’t look for anything really.’"

"‘Nothing more than simply doing what I have to do.’"

"‘In that situation, I could imagine that once you get acquainted with one [specific disease], it will be more useful than it is now.’"

Development issues

The physicians hoped that the interface design would become more visually oriented. They also proposed some new alerts or eCDS functions that could better assist in their work or make the eCDS system consistent.

"‘Make the system more visual; nice and desirable like an iPad… to give the user the feeling “Oh, what else could I look for? I want more of this.”’"

"‘If it were possible, a visual sign like when Microsoft™ e-mail shows pop-ups for new mail.’"

"‘A transient box, in the middle enough, for a short enough time, and automatically [closing]. If there would be a movement, then it would be noticed. […] it should be neutral on the screen and definitely should not disturb.’"

"‘Request for a new interaction alert: enoxaparin or dalteparin prescription when a patient has warfarin medication. This alert reminds physician to check the appropriateness of continued simultaneous use.’"

"‘Patient-tailored threshold values, different from guideline-based’"

"‘In the virtual health check retrospective testing, one slim patient got an obesity warning, because of a flawed weight or height recording. It was related to a missing digit, a recording of the height as 52 cm instead of the correct 152 cm. These types of typos should be guarded against by the system.’"

Discussion

The main findings of our study are twofold: perceived usefulness results in professionals using eCDS guidance while perceived non-helpfulness leads to non-use in primary health care. The guidance has to be designed for each profession and tailored, so that also e.g. nurses and physiotherapists can find it relevant for their practice. The profession groups have their own duties and specific practices; therefore their information needs vary greatly [37-40]. Our study indicates that even within one profession there are differences in the perceived usefulness of eCDS guidance. Lugtenberg et al. [41] reported similarly that perceived barriers for Dutch general practitioners varied greatly with the set of guideline recommendations involved.

Our results are in accordance with Davis’s technology acceptance theory [32], wherein perceived usefulness is considered a determinant with the main determinants of attitude and behavioural intention to use IT. Our results are strengthened by our use of multiple study methods: both open and structured questions, as well as focus group discussions with participants describing their specific reasons for using or not using the eCDS system or guidance. According to McDermott [14], perceived usefulness is a key characteristic in acceptance of a computer-based intervention among physicians. We discovered in addition that, while the majority of the physicians perceived the eCDS guidance to be useful for their work, the opposite was true for nurses and others, who sometimes appeared unable even to answer the question.

It is interesting that perceived usefulness showed in the data more consistently than perceived ease of use. It may be that the eCDS system basically was easy to use, since it had been integrated to the EPR system and the guidance popped on screen without specific effort, indicating perceived ease of use in TAM model [32]. Seven of nine physicians replied ‘yes’ to the question of easy to use, and the one negative answer came from a nurse (Table 3). It is conceivable that perceived ease of use, together with the main determinants of attitude and behavioural intention to use, was influencing in the background to the professionals’ perceived usefulness of the eCDS as TAM suggests [32].

The functionality of the eCDS system seemed good. Most responded positively to the questions dealing with this, with only one physician and nurse answering in the negative. However, the majority of the nurses found themselves unable to answer these functionality questions, which seemingly confirms the failure of the implementation among nurses [25]. In general, the professionals did not report any eCDS-system-based problem encumbering their work. By contrast, they seemed satisfied, because the eCDS system did not disturb them or force them to do anything. Indeed, this has been cited elsewhere as a key feature of a successful eCDS system [13]. According to the physicians, a more visually oriented system might be more desirable to them.

The results reaffirm that plenty of barriers exist to the use of reminders, even automatic ones, in primary care [17,19-22]. These were related to the studied eCDS system, inaccurate data in the EPR or code mismatch between the EPR and the eCDS system, the usability of the EPR itself, the groups of professionals, and patient preferences. The list bears a remarkable resemblance to the list of hurdles for use of clinical information systems as well as clinical guidelines [15,42]. One exception was seen: lack of agreement with the recommendations was evaluated as the most common perceived barrier to use of guideline recommendations among Dutch physicians [43], while this was not brought up as a barrier in our study. On the contrary, all respondents reported reliance on the reliability of the eCDS guidance.

The swine-flu epidemic and full-scale vaccination effort caused clear and measurable changes, especially in the nurses’ practice, for several months. It probably had a negative influence on the eCDS implementation and use, as busy practice was a common environmental reason for non-use of the eCDS. A recent usability evaluation study of a CDS tool for osteoporosis disease management [44] also indicated that physicians’ lack of time was a major challenge to point-of-care use of the tool.

A facilitating factor seems to be a generally positive attitude toward eCDS among primary care professionals. They did report needing more training and more time to learn to use the eCDS system. In particular, nurses expected some changes to their general nursing practice before actual implementation of the eCDS. Indeed, good training has been seen to be associated with favourable assessment of IT among nurses [45].

Strengths and limitations

Generalization of qualitative results can be problematic by the standards of the positivist approaches typical of quantitative research, in view of issues such as the representativeness of the sample [31]. Generalization can, however, be approached from a theoretical standpoint that takes into account the nature of the study question and the framework. So we can postulate that our results strengthen existing knowledge of the meaningful elements behind eCDS use among health care professionals. By using multiple methods and a long time period, we were able to extend our understanding of primary care realities during the implementation phase [46]. We believe that these results have validity for application in other primary care settings where a new automatic eCDS system integrated with EPR is being introduced.

Malterud [46] raises another issue, surrounding the researchers’ commitment to reflexivity. Our preconceptions stemming from previous studies [47,48] directed our initial choices and influenced the study process. We therefore chose to use deductive methods in the qualitative analyses and to describe the process as explicitly as possible. Four of us have a background in medicine and one in physiotherapy, which might aid us in understanding the different perspectives of the professionals. Two of us (JK and IK) also participated in development of the target eCDS system, which could have affected the study. From the beginning of the eCDS development, a study project was in progress to describe and evaluate what was going on in practice when the eCDS was piloted [49,50] and implemented [25]. The same choice has been made in many previous studies that involve IT developers as researchers [5,51].

A limitation is our use of a non-standardized questionnaire. However, the questions reported here were planned based on the theory [32] and the evidence of applied research [52,53]. Limited sample size and potential selection bias in responses due to participated versus non-participated professionals decrease a generalization of the results.

Conclusions

Perceived usefulness seems to be decisive for the use of eCDS guidance in primary care practice, and therefore the content of the eCDS is a critical issue. Information needs of profession groups in various environments (e.g., preventive care) have to be determined, and the eCDS guidance tailored according to those needs. Functionality-related characteristics such as speed and ease of use of the eCDS system and reliability of guidance are important but not sufficient in relation to uptake by primary care professionals.

Although this study targeted multi-profession groups in primary care, the studied eCDS system had been essentially based on medical evidence and to aid in physicians’ work. In this respect, there are several issues for future studies, e.g. targeting to patient’s reason for visit and tailoring with patient-specific threshold values. Unless future research considers in depth all professions’ needs, experiences and opinions the full potential of eCDS services may not become a reality.

Competing interests

Authors TK and MM declare that they have no competing interests. JK is Editor-in-chief of Current Care Guidelines, Finnish Medical Society Duodecim, and a member of the editorial board for EBMeDS, Duodecim Medical Publications Ltd. IK is a salaried employee of Duodecim Medical Publications Ltd, the company that develops and licenses the EBMeDS decision support service. MK chairs the Current Care Guidelines board at Finnish Medical Society Duodecim.

Authors’ contributions

All authors were involved in conceiving the study and designing the questionnaire and interview forms. TK was responsible for data collection and analysis. TK led the writing process, supervised by MK, and all authors commented on sequential drafts and approved the final version of the manuscript.

Acknowledgements

This study was funded by the Finnish Funding Agency for Technology and Innovation (Tekes), the National Institute for Health and Welfare (THL), Duodecim Medical Publications Ltd, ProWellness Ltd., and Doctoral Programs in Public Health (DPPH). We are grateful to the staff at the Sipoo health centre who gave their time to participate in the study process. The authors would like to thank other members of the EBMeDS study group for their help.

References

  1. Car J, Black A, Anandan C, Cresswell K, Pagliari C, McKinstry B, Procter R, Majeed A, Sheikh A: The impact of eHealth on the quality & safety of health care:a systematic overview & synthesis of the literature. Report for the NHS Connecting for Health Evaluation Programme. London: Imperial College London; 2008. OpenURL

  2. Lau F, Kuziemsky C, Price M, Gardner J: A review on systematic reviews of health information system studies.

    J Am Med Inform Assoc 2010, 17(6):637-645. OpenURL

  3. Grol R, Grimshaw J: From best evidence to best practice: effective implementation of change in patients' care.

    Lancet 2003, 362(9391):1225-1230. OpenURL

  4. McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A, Kerr EA: The quality of health care delivered to adults in the United States.

    N Engl J Med 2003, 348(26):2635-2645. OpenURL

  5. Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, Morton SC, Shekelle PG: Systematic review: impact of health information technology on quality, efficiency, and costs of medical care.

    Ann Intern Med 2006, 144(10):742-752. OpenURL

  6. Kawamoto K, Houlihan CA, Balas EA, Lobach DF: Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success.

    BMJ 2005, 330(7494):765. OpenURL

  7. Varonen H, Kaila M, Kunnamo I, Komulainen J, Mantyranta T: [Computerised decision support: the way towards advisory patient record].

    Duodecim 2006, 122(10):1174-1181. OpenURL

  8. Bates DW, Gawande AA: Improving safety with information technology.

    N Engl J Med 2003, 348(25):2526-2534. OpenURL

  9. Ahearn MD, Kerr SJ: General practitioners’ perceptions of the pharmaceutical decision-support tools in their prescribing software.

    Med J Aust 2003, 179(1):34-37. OpenURL

  10. Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS: Physicians’ decisions to override computerized drug alerts in primary care.

    Arch Intern Med 2003, 163(21):2625-2631. OpenURL

  11. Bryan C, Boren SA: The use and effectiveness of electronic clinical decision support tools in the ambulatory/primary care setting: a systematic review of the literature.

    Inform Prim Care 2008, 16(2):79-91. OpenURL

  12. Gosling AS, Westbrook JI, Spencer R: Nurses’ use of online clinical evidence.

    J Adv Nurs 2004, 47(2):201-211. OpenURL

  13. Greenes RA: Clinical decision support: The Road Ahead. Boston, MA: Elsevier; 2007. OpenURL

  14. McDermott L, Yardley L, Little P, Ashworth M, Gulliford M: Developing a computer delivered, theory based intervention for guideline implementation in general practice.

    BMC Fam Pract 2010, 11:90. OpenURL

  15. Francke AL, Smit MC, de Veer AJ, Mistiaen P: Factors influencing the implementation of clinical guidelines for health care professionals: a systematic meta-review.

    BMC Med Inform Decis Mak 2008, 8:38. OpenURL

  16. Sittig DF, Krall MA, Dykstra RH, Russell A, Chin HL: A survey of factors affecting clinician acceptance of clinical decision support.

    BMC Med Inform Decis Mak 2006, 6:6. OpenURL

  17. Rousseau N, McColl E, Newton J, Grimshaw J, Eccles M: Practice based, longitudinal, qualitative interview study of computerised evidence based guidelines in primary care.

    BMJ 2003, 326(7384):314. OpenURL

  18. Zheng K, Padman R, Johnson MP, Diamond HS: Understanding technology adoption in clinical care: clinician adoption behavior of a point-of-care reminder system.

    Int J Med Inform 2005, 74(7–8):535-543. OpenURL

  19. Patterson ES, Doebbeling BN, Fung CH, Militello L, Anders S, Asch SM: Identifying barriers to the effective use of clinical reminders: bootstrapping multiple methods.

    J Biomed Inform 2005, 38(3):189-199. OpenURL

  20. Patterson ES, Nguyen AD, Halloran JP, Asch SM: Human factors barriers to the effective use of ten HIV clinical reminders.

    J Am Med Inform Assoc 2004, 11(1):50-59. OpenURL

  21. Saleem JJ, Patterson ES, Militello L, Render ML, Orshansky G, Asch SM: Exploring barriers and facilitators to the use of computerized clinical reminders.

    J Am Med Inform Assoc 2005, 12(4):438-447. OpenURL

  22. Short D, Frischer M, Bashford J: Barriers to the adoption of computerised decision support systems in general practice consultations: a qualitative study of GPs’ perspectives.

    Int J Med Inform 2004, 73(4):357-362. OpenURL

  23. Fung CH, Woods JN, Asch SM, Glassman P, Doebbeling BN: Variation in implementation and use of computerized clinical reminders in an integrated healthcare system.

    Am J Manag Care 2004, 10(11 Pt 2):878-885. OpenURL

  24. Hamalainen P, Reponen J, Winblad I: eHealth of Finland: check point 2006. In Stakes reports. Helsinki, Finland: Valopaino Oy; 2007. OpenURL

  25. Kortteisto T, Komulainen J, Kunnamo I, Mäkelä M, Kaila M: Implementing clinical decision support for primary care professionals – the process.

    Finnish Journal of eHealth and eWelfare 2012, 4(3):153-164. OpenURL

  26. Fung CH, Tsai JS, Lulejian A, Glassman P, Patterson E, Doebbeling BN, Asch SM: An evaluation of the Veterans Health Administration’s clinical reminders system: a national survey of generalists.

    J Gen Intern Med 2008, 23(4):392-398. OpenURL

  27. EBMeDS context-sensitive guidance at the point of care.

    http://www.ebmeds.org/ebmeds/ebmeds_home.asp webcite.

    OpenURL

  28. Steward DW, Shamdasani PN: Focus groups: Theory and Practice. London: Sage Publications; 1990.

    Volume 20

    OpenURL

  29. Kitzinger J: Qualitative research: introducing focus groups.

    BMJ 1995, 311(7000):299-302. OpenURL

  30. Barbour RS: Doing Focus Groups. London: Sage; 2007. OpenURL

  31. Sim J: Collecting and analysing qualitative data: issues raised by the focus group.

    J Adv Nurs 1998, 28(2):345-352. OpenURL

  32. Davis FD: Perceived usefulness, perceived ease of use and user acceptance of information technology.

    MIS Q 1989, 13(3):319-340. OpenURL

  33. Lapointe L, Rivard S: Clinical information systems: understanding and preventing their premature demise.

    Healthc Q 2005, 8(2):92-100. OpenURL

  34. Elo S, Kyngas H: The qualitative content analysis process.

    J Adv Nurs 2008, 62(1):107-115. OpenURL

  35. Fleuren M, Wiefferink K, Paulussen T: Determinants of innovation within health care organizations: literature review and Delphi study.

    Int J Qual Health Care 2004, 16(2):107-123. OpenURL

  36. Silverman D: Doing Qualitative Research: A Practical Handbook. 2nd edition. London: Sage Publications; 2005. OpenURL

  37. Goossens A, Bossuyt PM, de Haan RJ: Physicians and nurses focus on different aspects of guidelines when deciding whether to adopt them: an application of conjoint analysis.

    Med Decis Making 2008, 28(1):138-145. OpenURL

  38. Hayrinen K, Saranto K, Nykanen P: Definition, structure, content, use and impacts of electronic health records: a review of the research literature.

    Int J Med Inform 2008, 77(5):291-304. OpenURL

  39. Kloda LA, Bartlett JC: Clinical information behavior of rehabilitation therapists: a review of the research on occupational therapists, physical therapists, and speech-language pathologists.

    J Med Libr Assoc 2009, 97(3):194-202. OpenURL

  40. Randell R, Dowding D: Organisational influences on nurses’ use of clinical decision support systems.

    Int J Med Inform 2010, 79(6):412-421. OpenURL

  41. Lugtenberg M, Burgers JS, Besters CF, Han D, Westert GP: Perceived barriers to guideline adherence: A survey among general practitioners.

    BMC Fam Pract 2011, 12:98. OpenURL

  42. Van Der Meijden MJ, Tange HJ, Troost J, Hasman A: Determinants of success of inpatient clinical information systems: a literature review.

    J Am Med Inform Assoc 2003, 10(3):235-243. OpenURL

  43. Lugtenberg M, Zegers-van Schaick JM, Westert GP, Burgers JS: Why don’t physicians adhere to guideline recommendations in practice? An analysis of barriers among Dutch general practitioners.

    Implement Sci 2009, 4:54. OpenURL

  44. Kastner M, Lottridge D, Marquez C, Newton D, Straus SE: Usability evaluation of a clinical decision support tool for osteoporosis disease management.

    Implement Sci 2010, 5:96. OpenURL

  45. Alpay L, Needham G, Murray P: The potential of information technology for nurses in primary care: a review of issues and trends.

    Prim Health Care Res Dev 2000, 1:5-13. OpenURL

  46. Malterud K: Qualitative research: standards, challenges, and guidelines.

    Lancet 2001, 358(9280):483-488. OpenURL

  47. Kortteisto T, Kaila M, Komulainen J, Mantyranta T, Rissanen P: Healthcare professionals’ intentions to use clinical guidelines: a survey using the theory of planned behaviour.

    Implement Sci 2010, 5:51. OpenURL

  48. Varonen H, Kortteisto T, Kaila M: What may help or hinder the implementation of computerized decision support systems (CDSSs): a focus group study with physicians.

    Fam Pract 2008, 25(3):162-167. OpenURL

  49. Korhonen M, Miettinen M: Information quality as data content and coherence in a regional information system for diabetes care.

    IADIS: 2008 2008. OpenURL

  50. Miettinen M, Korhonen M: Information quality in healthcare: coherence of data compared between organization’s electronic patient records. In 21st IEEE International Symposium on Computer-Based Medical Systems: 2008. Jyväskylä, Finland: IEEE Computer Society; 2008. OpenURL

  51. Garg AX, Adhikari NK, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, Sam J, Haynes RB: Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review.

    JAMA 2005, 293(10):1223-1238. OpenURL

  52. Gagnon MP, Godin G, Gagne C, Fortin JP, Lamothe L, Reinharz D, Cloutier A: An adaptation of the theory of interpersonal behaviour to the study of telemedicine adoption by physicians.

    Int J Med Inform 2003, 71(2–3):103-115. OpenURL

  53. Raitoharju R: Information technology acceptance in the Finnish social and healthcare sector. In Exploring the effects of cultural factors. A-4 edition. Turku, Finland: School of Economics; 2007. OpenURL

Pre-publication history

The pre-publication history for this paper can be accessed here:

http://www.biomedcentral.com/1472-6963/12/349/prepub