Email updates

Keep up to date with the latest news and content from BMC Medical Informatics and Decision Making and BioMed Central.

Open Access Research article

Validation of the Work Observation Method By Activity Timing (WOMBAT) method of conducting time-motion observations in critical care settings: an observational study

Mark A Ballermann1, Nicola T Shaw2*, Damon C Mayes3, RT Noel Gibney4 and Johanna I Westbrook5

Author Affiliations

1 Department of Family Medicine, University of Alberta, Edmonton, Alberta, Canada

2 Health Informatics Institute, Room SH 500. Algoma University, 1520 Queen Street East, Sault Ste. Marie, Ontario, P6A 2G4, Canada

3 Alberta Health Services, Edmonton, Alberta, Canada

4 Division of Critical Care Medicine, Faculty of Medicine & Dentistry, University of Alberta, Edmonton, Alberta, Canada

5 Centre for Health Systems and Safety Research, Australian Institute of Health Innovation, Faculty of Medicine, University of New South Wales, Sydney, Australia

For all author emails, please log on.

BMC Medical Informatics and Decision Making 2011, 11:32  doi:10.1186/1472-6947-11-32

The electronic version of this article is the complete one and can be found online at: http://www.biomedcentral.com/1472-6947/11/32


Received:2 December 2010
Accepted:17 May 2011
Published:17 May 2011

© 2011 Ballermann et al; licensee BioMed Central Ltd.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Background

Electronic documentation handling may facilitate information flows in health care settings to support better coordination of care among Health Care Providers (HCPs), but evidence is limited. Methods that accurately depict changes to the workflows of HCPs are needed to assess whether the introduction of a Critical Care clinical Information System (CCIS) to two Intensive Care Units (ICUs) represents a positive step for patient care. To evaluate a previously described method of quantifying amounts of time spent and interruptions encountered by HCPs working in two ICUs.

Methods

Observers used PDAs running the Work Observation Method By Activity Timing (WOMBAT) software to record the tasks performed by HCPs in advance of the introduction of a Critical Care clinical Information System (CCIS) to quantify amounts of time spent on tasks and interruptions encountered by HCPs in ICUs.

Results

We report the percentages of time spent on each task category, and the rates of interruptions observed for physicians, nurses, respiratory therapists, and unit clerks. Compared with previously published data from Australian hospital wards, interdisciplinary information sharing and communication in ICUs explain higher proportions of time spent on professional communication and documentation by nurses and physicians, as well as more frequent interruptions which are often followed by professional communication tasks.

Conclusions

Critical care workloads include requirements for timely information sharing and communication and explain the differences we observed between the two datasets. The data presented here further validate the WOMBAT method, and support plans to compare workflows before and after the introduction of electronic documentation methods in ICUs.

Background

Some of the most acute patients in hospital settings are treated in Intensive Care Units (ICUs) by specialized Health Care Providers (HCPs). HCPs use diverse information sources to prioritize their tasks and make decisions about patient care. These sources include other HCPs, bedside equipment, and laboratories located at some distance from the patient [1-3]. HCPs coordinate care over time and across HCP roles in a way that is consistent with patient needs, and thus strive for continuity of care [4]. Continuity of care is thought a crucial determinant of patient outcome, and depends on the timely availability of patient information or informational continuity. Patient charts contain critical information about patient status and care plans, and support medical decision making.

HCPs working in ICUs must manage many information sources to ensure that the documentation in each patient chart is correct, current, and complete. With paper charts in place, HCPs transcribe information from bedside equipment and laboratory reports. Transcribing information into patient charts may not be the best use of time for highly specialized and expensive HCPs. An electronic medical record designed for the ICU environment, a Critical Care clinical Information System (CCIS), can automate some transcription tasks and aid in informational continuity between HCPs [5]. The results of studies to date examining whether a CCIS will reduce time spent on documentation have been equivocal [6,7]. Moreover, high rates of failure have been reported for many electronic systems in healthcare, in which organizations have either adopted other systems or reverted to paper charts [8,9].

Evaluation methods of HCP work can be used before and after a system introduction to collect more objective data about its impact on ICU operations. Impacts on ICU operations will include impacts to the workflows of HCPs including physicians, nurses, and others. Observational and self-reporting techniques have been used to study work in hospitals and other settings [10,11]. A practical advantage of self report studies is the lower cost resulting from participants recording their own activities when they are prompted by a reminder device [10,12]. However, participants giving self reports may ignore the prompts during busy work periods, which can result in incomplete data [10,11,13]. In ICU environments, tasks performed during busy work periods may be particularly critical for patient outcomes and thus may be systemically neglected by work sampling methods. In comparison, observational techniques employ a researcher to observe and record the behaviours of interest, which significantly reduces concerns about incomplete data [10]. Others have used continuous observation to examine the overall time spent by health care providers (such as physicians) on tasks in hospital ward settings [14,15]. However, little is currently known about the proportions of time that HCPs spend on their various tasks while working in ICUs.

Furthermore, comparisons between ICUs and general hospital wards have shown that there are higher rates of adverse medical events in ICUs [16]. Interruptions and their associated shifts in cognitive focus may be a factor in medical error [17]. A clear picture of how HCPs working in ICUs manage interruptive communications is needed to better understand the relationship between interruptions and errors [17-19]. The CCIS may facilitate access to information that is crucial to decision making associated with patient care. This paper is part of a larger study that will examine whether the use of a CCIS represents a positive step for ICU patient care and for HCPs working in ICUs. The availability of information in stored patient charts may improve after the CCIS introduction, but this is presently not known. The capture of myriad data sources may be facilitated by the introduction of a CCIS, but valid comparisons will benefit from more complete descriptions of methods for quantifying workflow (including proportions of time spent on tasks, and interruptions encountered) of HCPs in ICUs. Initial descriptions of the larger CCIS study design have been reported [20,21], and other reports have compared pre- and post-CCIS data [22-25].

Rationale

This paper provides part of a methodological foundation for a larger mixed-methods study assessing the impact of a CCIS. We are conducting a study investigating two academic tertiary-care ICU units that introduced a CCIS in early 2009. The CCIS implementation is likely to impact both the quality and availability of information in patient charts, and it is likely to also affect HCP work.

This paper documents the methodology used to observe ICU nurses, physicians, unit clerks and respiratory therapists working with paper charts. We recorded the amounts of time they spent on their tasks using Personal Digital Assistants (PDAs) running Work Observation Method By Activity Timing (WOMBAT) software. We assert that this observation method allows us to derive valid and reliable measures of amounts of time HCP spend on tasks and interruption rates when compared with previous data [13,15]. Previous studies using the WOMBAT software have described task data definitions for nurses and physicians working in general hospital settings [13,15]. We extend these task data definitions to quantify the time spent on tasks by respiratory therapists and unit clerks. We collected observational data at shift changes, which are times when maintaining continuity of care is challenging, and balanced our data collection across all days of the week and times of day to capture a representative picture of work in the ICUs. Finally, we show evidence suggesting that HCPs do not alter their activities based on the presence of an observer, an effect commonly referred to as the Hawthorne effect [26]. This evidence may help to address concerns that care providers alter their activities as a result of being observed.

Objective

In this paper we compare our time-motion baseline results with those previously published to provide supporting evidence that the WOMBAT method provides valid results when quantifying amounts of time spent on different tasks and interruptions encountered by critical care providers. We discuss differences observed between the reported data and previously published results from Australian general hospital wards [13,15] and emergency wards [27], based on the nature of critical care.

Methods

Research setting

The University of Alberta Human Research Ethics Board (File #B-241107) and Northern Alberta Clinical Trials and Research Centre (File #6035) granted approval for this study prior to the commencement of data collection. We conducted our study in the Pediatric ICU (PICU) at the Stollery Children's Hospital and the General Systems ICU (GSICU) at the University of Alberta hospital in Edmonton, Alberta, Canada. The PICU has 17 beds. The GSICU has 30 beds, with 24 operational during the observations due to staff shortages. These are busy critical care units in academic tertiary referral hospitals. At the time of the observations, the units operated with paper charts in place, with internet-enabled computers at nursing stations and throughout the unit. Laboratory data are available through these computers. The ratio of nurses to patients is 1:1 in the PICU, and 1:1 70% of the time and 1:2 30% of the time in the GSICU, depending on patient acuity.

Participants

Members of the staff were informed of our study through presentations given by members of the research team and by posters distributed around the units. Participants were then approached by members of the research team for their consent to be observed.

Of 215 nurses in permanent staff positions, 87 agreed to participate (40%) and 47 were observed. Of 35 attending physicians and fellows, 32 agreed to participate (91%) and 18 were observed. Fellows included physicians at their third year of post medical degree training or above, working on the unit in a full-time capacity. Of 72 respiratory therapists working on the units, 46 agreed to participate (64%) and 25 were observed. We obtained consent from 14 out of 16 unit clerks working on the units (88%), and 10 were observed. Observations were randomly selected from the participants working in the ICUs, so participants who were scheduled for fewer shifts would be less likely to be observed. Informed consent and demographic data (age, sex, time employed in ICU settings, number of shifts per month, and self-assessed familiarity with computers) were obtained from participants.

Observers

Observers were trained for at least 12 hours before starting observations. During training sessions, observers were oriented to the PDA software, the work definitions, and the ICUs. Observers would then follow a HCP alongside an experienced observer, simultaneously scoring the same tasks. The experienced observers were nurses working on the PICU. Inter-rater reliability scores were calculated for total time spent and numbers of tasks scored. Values of 85% or higher were obtained between the observers before trainees conducted their own observations.

Work definitions

Observers carried a list of work definitions to assist in classifying tasks they observed into one of the categories in the PDA. These work definitions were initially provided by Westbrook and colleagues [15]. The work definitions were further refined to include tasks specific to the observed units, and tasks specific to the respiratory therapists and unit clerks. The complete work definitions for the nurses, physicians and respiratory therapists are provided in Table 1. Unit clerks collect and disseminate a great deal of information within the units. Separate work definitions were created for them as their tasks were fairly different from other HCP roles. The work definitions for unit clerks are provided in Table 2.

Table 1. Physician, Nurse, and Respiratory Therapist work definitions

Table 2. Unit clerk work definitions

Time motion observation tool

Observers carried Hewlett-Packard iPAQ hx2490 Personal Digital Assistants (PDAs) running the Work Observation Method By Activity Timing (WOMBAT) software [15]. Observers scored the start of tasks by selecting the task category (what was done), the people who were present (who the task was done with), and any information tools used and pressing the 'enter' button. The time that the enter button was pressed was then recorded as the start time for that task. Subsequent tasks could be scored by selecting new information and pressing enter. The end time of the previous task and the start time of the new task were both entered into the database automatically.

Participants were frequently observed performing more than one task simultaneously. The WOMBAT program allowed observers to score multiple simultaneous tasks using the multitasking function. New tasks could be added to those that were already underway if observers pressed the 'add' button prior to pressing the 'enter' button. The presence of different 'tabs' allowed the observer to see the tasks that were being scored at the current time. If one or more tasks ended, individual tasks could be scored as stopped using an 'end multi' button. The software required that at least one task be scored at all times.

If any external factor appeared to cause a HCP to stop performing one or more tasks, and start another task, an 'interruption' was recorded. Observers entered the task information for the task that was started and pressed the 'interrupt' button. The first task was moved into a background tab as a 'pending' task that the care provider may or may not return to. Observers could remove pending tasks if it appeared that the care provider would not come back to them.

Data were extracted into Excel spreadsheets via a laptop computer. Proportions of time spent on the different task categories and the rates of interruption were calculated for each observation. Interruptions were characterized by the task the participant initiated after the interruption was scored.

Observations

Observations were carried out for 90 minutes with no advance warning to the participant. The 90 minute time limit is based on the notion that this may represent an upper limit to the length of time in which observers may be capable of recording all the tasks they observe. Equal numbers of observations were performed in 4 conditions: mid-shift during the day (07:00-19:00), mid-shift at night (19:00-07:00), during the morning shift change (06:30-08:00), and evening shift change (18:30-20:00). Observations were also balanced between 4 day types: midweek, weekend, Mondays, and Fridays. Workload is likely to vary with the day of the week as the number of operating rooms in use impacts the mix of admitted ICU patients. Mondays and Fridays would likely have more admission and discharge activity, respectively. These factors would impact the workload encountered by participants. Observers kept field notes for recording contextual information, such as their impression of how busy the unit was, whether the unit was short-staffed, and whether there were students present on the unit. Observers maintained a following distance of approximately 3 metres.

Observations were suspended when participants left the unit or went on a break. The goal of this study was to collect observational data in the unit environment where the CCIS would be put into place, rather than tasks performed elsewhere. The observations in the PICU were carried out between September and November, 2008. The observations in GSICU were carried out between January and February 2009. We performed 62 hours of nurse observations, 58 hours of physician observations, 55 hours of respiratory therapist observations and 57 hours of unit clerk observations.

Statistical analysis

Although participants were assured that their personal work was not being evaluated when informed consent was obtained, the possibility exists that participants may not recollect that reassurance. As such, participants may have an incentive to avoid tasks that would be scored as 'social' time. Participants may habituate to the presence of an observer, so differing proportions of social time during the early parts of an observation compared with the overall proportions may indicate a phenomenon known as participant reactivity or the Hawthorne effect [26]. Proportions of time spent on social tasks in the first minute, first five minutes, and first ten minutes were compared to the similar proportions from the entire observation with Mann-Whitney U tests. These tests were completed to examine whether HCPs tended to avoid socializing when an observation started, but eventually habituated to the presence of an observer.

Results

We observed 14,928 separate tasks across all HCP roles. The mean time spent per task was 78 seconds, with a median task time of 34 seconds. The maximum time spent on one task was 39 minutes when a respiratory therapist attended a meeting. ICU physicians, nurses, respiratory therapists and unit clerks were observed for 58 hours, 62, 55, and 57 hours respectively.

Percentages of time spent on tasks

Total percentages of time spent on task categories by physicians, nurses, respiratory therapists, and unit clerks, were calculated and presented in Figure 1. Comparable time percentages from two Australian datasets [13,27] of observations of physician work (Figure 1A) and one dataset [15] of nurse work (Figure 1B) were superimposed to facilitate comparisons. Critical care providers spent large percentages of time performing more than one task at a time, or multitasking, a finding that has been previously reported [23]. We found that the WOMBAT method yields data that are generally consistent with the Australian datasets [13,15,27], with only minor differences. Physicians and nurses in the Australian datasets spent less time on professional communication tasks than ICU nurses and physicians. For nurses, these findings may be accounted for by ICU patient to nurse ratios being closer to 1:1, compared with much higher ratios on general hospital wards. Due to the highly acute nature of ICU patients, we frequently observed nurses working together to perform patient care tasks. For example, several nurses may be required to bathe a heavier sedated patient. High levels of teamwork in ICUs likely account for the greater proportions of professional communication among nurses. Similarly, ICU physicians frequently participate in morning rounds, afternoon "sign-out" rounds, and nightly rounds as part of a multidisciplinary team, including nurses, respiratory therapists, resident physicians, nutritionists, pharmacists, and social workers. Additionally, consulting physicians often arrive to speak with the attending physicians about patients in the ICU. This team-based approach to providing health care in the ICU likely accounts for the very high values we observed in professional communication.

thumbnailFigure 1. Time percentages spent performing different tasks by critical care providers. Values represent means and error bars represent 95% confidence intervals. Plus signs (+) in A represent values for percentages of time spent by physicians in general hospital wards on tasks taken from [13], and in B corresponding values for nurses were adapted from [15]. Asterisks in A represent percentage of time spent by physicians working in EDs, adapted from [27]. P. Communication = Professional Communication.

Nurses spent greater amounts of time on documentation tasks in ICUs, a difference from the Australian general ward dataset [15]. This may be partly attributable to the greater requirements for regular monitoring and documentation of ICU patients [28]. Rapidly changing clinical conditions of ICU patients may necessitate more frequent and detailed documentation than is the case for patients on general hospital wards. The myriad information sources in the ICU, including bedside telemetry, lab results, observations, and procedures performed by HCPs all need to be documented from a medico-legal standpoint as well as to inform care providers of the current and possibly rapidly changing state of the patient.

Interruptions

We recorded mean interruption rates for ICU physicians of 3.8 times per hour (once every 15.8 minutes on average; Figure 2A). Nurses were interrupted 3.3 times per hour, an average of an interruption every 18.3 minutes (Figure 2B). Respiratory therapists were interrupted 3.5 times per hour, on average, corresponding to an average time between interruptions of 17 minutes (Figure 2C). Unit clerks were interrupted 4.4 times per hour on average, corresponding to an interruption every 13.8 minutes (Figure 2D). For all roles observed, the tasks initiated after an interruption were most often professional communication tasks. For physicians, nurses, respiratory therapists, and unit clerks, interruptions where these tasks were initiated occurred at rates of 2.0 interruptions/hour (or hr-1), 1.8 hr-1, 1.9 hr-1, and 1.8 hr-1, respectively. HCPs may need to collect and disseminate information in a timelier manner in ICUs than on other units. As a consequence, interruptions may be more frequent in the ICU setting than on general hospital wards [13,15]. As a part of our larger study we have examined HCP perceptions of interruptions through interviews and focus groups. HCPs of all types reported that interruptions are pervasive in ICUs [25]. The reports of more frequent interruptions are confirmed by the observational data provided here. ICU physicians were interrupted 3.8 times per hour whereas comparable physicians on hospital wards experienced 2.9 interruptions per hour [13]. Interruptions in EDs have also been studied using the WOMBAT method, where ED physicians encountered 6.6 interruptions/hour, which is comparable to other observations of ED physicians [29,30]. We find that ICU physicians were interrupted less often than ED physicians, which is axiomatic.

thumbnailFigure 2. Interruption rates for critical care providers. Values represent mean interruption rates for 90 minute observations. Error bars represent 95% confidence intervals. Black bars represent overall rates of interruption. White bars represent the rates of interruptions where HCPs where the secondary (interrupting) task was the task named at the bottom. P. Communication = Professional Communication.

Participant reactivity

If participants change their behaviour as a result of being observed [26], such changes should be most evident at the start of an observation when participants first learn that they will be observed for the next 90 minutes. As participants habituate to the presence of the observer, we might expect Hawthorne-like effects to lessen. As an approach to exploring whether Hawthorne-like effects exist within our data, we examined amounts of time recorded as 'social' tasks. All participants were assured that their personal performance would not be evaluated, yet some individuals expressed some scepticism to the assertion that they were not being personally evaluated. As such, individual participants may have felt uncomfortable engaging in social tasks in the presence of observers. To examine the possibility of participants altering their behaviour due to the presence of an observer, the percentages of time spent on social activities were calculated per session for the first minute, 5 minutes, 10 minutes, and for the entire 90 minute observation (Figure 3). There were no significant changes between the mean proportions of time spent on activities scored as 'social' for the early periods of the observation compared to the entire observation, a result that may be inconsistent with participants altering their behaviour as a result of being observed.

thumbnailFigure 3. Percentages of time spent on social tasks. Percentages of observed time spent on social time were calculated across all health care provider roles for the first minute, 5 minutes, 10 minutes and entire 90 minute observation. Values represent mean percentages of time spent for each sampling period. Error bars represent 95% confidence intervals.

Discussion

We report the percentages of time that HCPs working in two Canadian ICUs spent on different categories of tasks, and the interruptions encountered using a method previously applied to quantify the work of physicians [13,27] and nurses [15] in Australian hospitals. Based on the results reported here, we posit that this method is a valid approach for collecting time and motion data in health care settings to assess the amounts of time spent on different tasks.

We make strong comparisons between the Australian datasets [13,15,27] and our own data from ICUs as we used nearly identical data definitions, with minor changes to account for tasks specific to the two units participating in our study and those tasks specific to the respiratory therapist and unit clerk roles, which were not assessed previously. Where our findings vary from those in the literature, they make theoretical sense based on the ICU observational setting, where complex patients are managed by teams of specialized care providers. We applied this methodology to observe respiratory therapists and unit clerks, to obtain a more complete picture of how critical care changes after a CCIS introduction. We conducted observations during nurse, respiratory therapist, and unit clerk shift changes, to examine workflows during times when informational continuity was challenged [31]. The results demonstrate the amounts of time critical care providers spend dealing with information to address the needs of highly complex and acute patients and provide a basis to assess the impacts of a CCIS on critical care. Future studies will examine whether the introduction of a CCIS facilitates the access to information and the completion of documentation tasks associated with patient care.

Importantly, we show for the first time the proportions of time spent by respiratory therapists and unit clerks in the critical care setting. We applied the WOMBAT method to observe respiratory therapists and unit clerks, and reported the amounts of time spent by HCPs in each role on the different task categories. We refined the work definitions to include the tasks performed by these HCPs. Much like the physicians and nurses in the ICUs, respiratory therapists spent high percentages of time performing professional communication tasks. Indirect and direct patient care and documentation were additional major categories of tasks. Respiratory therapists often care for multiple patients on the units, depending on the patient load. Due to the physical layout of the GSICU, respiratory therapists on GSICU spent more time 'In transit' than any other HCP role we observed. These findings objectively show the quantities of time spent on each task category, and illustrate the nature of workflow of respiratory therapists on the two units.

Unit clerks spent a significant proportion of their time on professional communication and administration tasks. The unit clerk is a central focus of information flow throughout the unit. Unit clerks assist by providing information that is critical to patient care by communicating with other HCPs to manage the timing of family visits and ensure the delivery of effective patient care.

ICU care providers encounter interruptions at rates that are between those of care providers working in EDs and general hospital wards. Of these interruptions, half are followed by professional communication tasks. As an association between interruptions and medication errors has been shown [32], understanding the reasons why interruptions are pervasive in certain environments is a prerequisite to improving patient safety. As many interruptions in ICUs are related to the management of information, the introduction of a CCIS may result in changes to the rates and types of interruptions encountered by critical care providers. Future studies will focus on the rates and types of interruptions and how they are both perceived and managed by HCPs on ICUs [24,25].

Participant reactivity

One concern that is frequently raised about observational studies concerns the phenomenon of participant reactivity, when a participant may change their behaviour as a result of being aware that they are being studied [26]. Also called the Hawthorne effect, its original appearance has been called into question by a more recent careful re-examination of the original data [33]. We examined the proportions of time spent on 'social' tasks during the first minute, 5 minutes, and 10 minutes across observations for all roles, based on the likelihood that if participants changed their work significantly due to being observed, a likely change may be avoiding actions that would be scored as 'social', especially at the outset of an observation. We found no significant differences between the proportions of time spent on 'social' activities during any of these time periods and the entire observation periods, which is not consistent with a Hawthorne-like effect. This interpretation of the data depends on the assumption that Hawthorne-like effects would extinguish as the observation continues, but if this assumption is incorrect, the results reported would fail to identify a Hawthorne-like effect. However, various observational studies of clinicians in situ have suggested that the extent of behaviour change is minimal [10,34,35]. Although it is difficult to completely rule out Hawthorne-like effects as factors in general, these findings are inconsistent with the Hawthorne effect being a factor in our data. It is important to note that the nature of the critical care environment requires staff to quickly adjust to the presence of many different members of the health care team. HCPs on ICUs within teaching hospitals, in particular, may be less likely to demonstrate Hawthorne-like effects because of their experience in performing tasks in the presence of many observers. Finally, our observers included ICU nurses, some of whom were familiar to the participants.

Strengths and limitations

This study provides valuable information about the tasks performed by HCPs in ICU settings. Our study follows a wider variety of HCP roles than previous studies in order to examine in more depth how a system like the CCIS may impact the various roles differently. The CCIS introduction has included some types of bedside telemetry (patient monitors of vital signs) but not others (ventilators, for example), which may impact different HCP roles uniquely. Our study is ideally positioned to capture these myriad effects. Additionally, our study specifically examines staff tasks around shift change, at night, and on different days of the week. As informational continuity may be challenged during shift change [31], different factors may alter unit operations at night (e.g. different HCPs, different availability of staff), and on different days of the week (differing workloads based on numbers of operating rooms in use), this approach represents a more complete method of examining the effects of the CCIS on the ICU.

We have examined one perceived weakness of observational studies, participant reactivity. A second potential weakness surrounds the use of a 90-minute time limit for observations, which is aimed at limiting observer fatigue. At times, observers would perform two observations consecutively. We encouraged observers to take a short break in between observations to improve their alertness during each observation to help ensure high data quality. In fact, the literature supporting the use of a 90-minute time limit on observations is not entirely robust. We have been unable to find published findings supporting the use of a 90 minute time-limit, and it is very likely that this limit will vary from observer to observer. In the present study, we take a somewhat conservative approach in using a 90 minute limit, but we also allow observers the flexibility to complete up to three observations each day. In our experience, three daily observations can be completed accurately, as long as the observer takes longer breaks between each observation. The time of day that each observation is completed may also impact potential effects of observer fatigue. Multiple observations carried out during the middle of the night, for example, may be more likely to impact data quality. Our observers did not perform multiple observations at night.

A remaining question is the frequency with which interobserver reliability scores ought to be assessed. Although observers completed 'buddy' shifts alongside an experienced observer during training, there is some question in the literature as to how often observers should repeat these tests to ensure data quality. We ensure 85% agreement between observers before allowing trainees to complete their own observations. Data quality may be best determined by performing interobserver reliability scores among all pairs of observers. Depending on the number of observers, this task may or may not be feasible given resource constraints.

An additional potential weakness surrounds the use of the 'indirect patient care' category. The design of this category is such that activities like monitoring patient status, collecting medical equipment, and hand washing are included with activities based around information management such as reviewing patient charts and finding medical records. The time spent on these 'information tasks' may be important to investigate more closely as HCPs work with the CCIS to find the information that they need to provide patient care. If the process of finding and reviewing information is made more or less efficient for HCPs, the current indirect care category system would be unlikely to capture those changes. Effects involving time spent on reviewing documents may be diluted by other tasks in the 'indirect patient care' category. In addition, other categories could be enhanced to collect more detailed information about the tasks being performed, such as what information is being discussed during professional communication tasks. This consideration needs to be balanced with the possibility that any change that increases the complexity of the data definitions may make training the observers more difficult and could potentially impact the data quality. Additionally, significant changes to the data definitions may make comparison with other studies difficult or impossible. We would caution investigators that the work definitions they use at the outset of their studies may result in more or less valuable data depending on the aspects of HCP workflow they wish to examine.

The results showing time spent on documentation should be carefully interpreted due to the medication tasks category. The ICUs use medication orders as part of the patient charts. Unless an observer is standing much closer than the 3 metre following distance we employ, it may not be possible to identify every instance when medication orders are being written in the chart. If an observer stood closer to obtain more accurate records of medication related documentation, they would risk obstructing their participant. Thus, the values we have found may underreport medication prescribe and chart events, which may instead be captured in the documentation section. In general, we have sought to openly report potential weaknesses in this approach such that future investigators may benefit from this information.

Conclusions

Canadian critical care providers spend greater proportions of time communicating with each other than do physicians and nurses working in Australian general hospital wards. This is consistent with specialized, coordinated, team based care. These results help to validate the previously published findings of the WOMBAT method on hospital wards as well as demonstrating the amounts of time front line critical care providers spend accessing and disseminating information for patient care. We describe a truly blended method with quantitative data resulting from subjective observations. Future studies will employ this method to examine how the CCIS impacts the time HCPs spend on their tasks, the interruptions they encounter, and whether the CCIS introduction is broadly a positive step for patient care.

List of abbreviations

CCIS: Critical Care clinical Information System; ED: Emergency Department; GSICU: General System Intensive Care Unit; HCP: Health Care Provider; ICU: Intensive Care Unit; PDA: Personal Digital Assistant; PICU: Pediatric Intensive Care Unit; WOMBAT: Work Observation Method By Activity Timing

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

MAB coordinated the observations, wrote the manuscript, and performed the data analysis. NTS is the senior author of the CCIS research study. She designed the study and provided editorial advice. DCM, RTNG, and JIW provided editorial advice. All authors provided direction and input into the study design, and reviewed and approved the final manuscript.

Acknowledgements & Funding

We wish to acknowledge the assistance of the staff and management of the University of Alberta GSICU and Stollery Children's Hospital PICU, as well as research observers Tineke Chattargoon, Sara Belton, and Kelly Speer. Funding support was provided by Alberta Health Services and a Partnership in Health System Improvement Grant from Canadian Institutes of Health Research (Grant identifier: PHE 91423). Sally Ho, Michela Brown, Nerida Creswick, and Krish Thiru contributed to planning early versions of the research work plan. Audrey Seehagen and Aireen Wingert provided editorial assistance. The study sponsors had no role in the study design, data collection, analysis and interpretation of data, writing of the manuscript, nor in the decision to submit the manuscript for publication.

References

  1. Brown M, Shaw NT, Mador RL: Mapping the sociotechnical healthcare ecosystem: expanding the horizons of sociotechnical inquiry.

    AMIA Annu Symp Proc 2008, 1233-1235. OpenURL

  2. Noble MA: The critical care clinical nurse specialist: Need for hospital and community.

    Clin Nurse Spec 1988, 2:30-33. PubMed Abstract | Publisher Full Text OpenURL

  3. Rosenberg DI, Moss M, Section on Critical Care and Committee on Hospital Care: Guidelines and levels of care for pediatric intensive care units.

    Pediatrics 2004, 114:1114-1125. PubMed Abstract | Publisher Full Text OpenURL

  4. Haggerty JL, Reid RJ, Freeman GK, Starfield BH, Adair CE, McKendry R: Continuity of care: a multidisciplinary review.

    BMJ 2003, 327:1219-1221. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  5. Varon J, Marik PE: Clinical information systems and the electronic medical record in the intensive care unit.

    Curr Opin Crit Care 2002, 8:616-624. PubMed Abstract | Publisher Full Text OpenURL

  6. Mador RL, Shaw NT: The impact of a Critical Care Information System (CCIS) on time spent charting and in direct patient care by staff in the ICU: a review of the literature.

    Int J Med Inform 2009, 78:435-445. PubMed Abstract | Publisher Full Text OpenURL

  7. Poissant L, Pereira J, Tamblyn R, Kawasumi Y: The impact of electronic health records on time efficiency of physicians and nurses: A systematic review.

    J Am Med Inform Assoc 2005, 12:505-516. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  8. Heeks R: Health information systems: Failure, success and improvisation.

    International Journal of Medical Informatics 2006, 75:125-137. PubMed Abstract | Publisher Full Text OpenURL

  9. Wears RL, Berg M: Computer technology and clinical work: still waiting for Godot.

    JAMA 2005, 293:1261-1263. PubMed Abstract | Publisher Full Text OpenURL

  10. Ampt A, Westbrook J, Creswick N, Mallock N: A comparison of self-reported and observational work sampling techniques for measuring time in nursing tasks.

    Journal of Health Services Research and Policy 2007, 12:18-24. PubMed Abstract | Publisher Full Text OpenURL

  11. Murray MD, Loos B, Tue W, Eckert GJ, Zhou XH, Tierney WM: Work patterns of ambulatory care pharmacists with access to electronic guideline-based treatment suggestions.

    Am J Health Syst Pharm 1999, 56:225-232. PubMed Abstract | Publisher Full Text OpenURL

  12. Abdellah FG, Levine E: Work sampling applied to the study of nursing personnel.

    Nurs Res 1954, 3:11-16. PubMed Abstract OpenURL

  13. Westbrook JI, Ampt A, Kearney L, Rob MI: All in a day's work: an observational study to quantify how and with whom doctors on hospital wards spend their time.

    Med J Aust 2008, 188:506-509. PubMed Abstract | Publisher Full Text OpenURL

  14. Cullen DJ, Sweitzer BJ, Bates DW, Burdick E, Edmondson A, Leape LL: Preventable adverse drug events in hospitalized patients: A comparative study of intensive care and general care units.

    Critical Care Medicine 1997, 25:1289-1297. PubMed Abstract | Publisher Full Text OpenURL

  15. Westbrook JI, Ampt A: Design, application and testing of the Work Observation Method by Activity Timing (WOMBAT) to measure clinicians' patterns of work and communication.

    Int J Med Inform 2009, 78(Suppl 1):S25-33.

    Epub:2008 Oct 31.:S25-S33.

    PubMed Abstract | Publisher Full Text OpenURL

  16. Einstein GO, McDaniel MA, Williford CL, Pagan JL, Dismukes RK: Forgetting of intentions in demanding situations is rapid.

    J Exp Psychol Appl 2003, 9:147-162. PubMed Abstract | Publisher Full Text OpenURL

  17. Biron AD, Loiselle CG, Lavoie-Tremblay M: Work Interruptions and Their Contribution to Medication Administration Errors: An Evidence Review.

    Worldviews Evid Based Nurs 2009, 6:70-86. PubMed Abstract | Publisher Full Text OpenURL

  18. Grundgeiger T, Sanderson P: Interruptions in healthcare: theoretical views.

    Int J Med Inform 2009, 78:293-307. PubMed Abstract | Publisher Full Text OpenURL

  19. Brixey JJ, Robinson DJ, Turley JP, Zhang J: The roles of MDs and RNs as initiators and recipients of interruptions in workflow.

    Int J Med Inform 2010, 79:e109-e115. PubMed Abstract | Publisher Full Text OpenURL

  20. Mador RL, Brown M, Shaw NT, Mayes D: Interruptions in workflow prior to the implementation of a Critical Care Information System (CCIS) in two intensive care settings.

    AMIA Annu Symp Proc 2008, 1037. OpenURL

  21. Shaw NT, Mador RL, Ho S, Mayes D, Westbrook JI, Creswick N, Thiru K, Brown M: Understanding the impact on intensive care staff workflow due to the introduction of a critical care information system: a mixed methods research methodology.

    Stud Health Technol Inform 2009, 143:186-91.

    186-191

    PubMed Abstract | Publisher Full Text OpenURL

  22. Ballermann M, Shaw NT, Mayes DC, Gibney RTN: Critical care providers refer to information tools less during communication tasks after a critical care clinical information system introduction. In In Proceedings of the Information Technology and Communications in Health Conference. Edited by Borycki, E and Bartle-Clar, J. Victoria, British Columbia; 2011. OpenURL

  23. Ballermann M, Shaw NT, Arbeau KJ, Mayes DC, Gibney RTN: Intensive care unit health care providers spend less time multitasking after the introduction of a critical care clinical information system. In In Proceedings of the HIC2010 Health Informatics Society of Australia. Edited by Hansen, D., Schaper, L., and Rowlands, D. Melbourne, Australia; 2010. OpenURL

  24. Ballermann MA, Shaw NT, Arbeau KJ, Mayes DC, Noel Gibney RT: Impact of a critical care clinical information system on interruption rates during intensive care nurse and physician documentation tasks.

    Stud Health Technol Inform 2010, 160:274-278. PubMed Abstract | Publisher Full Text OpenURL

  25. Shaw NT, Ballermann M, Hagtvedt R, Ho S, Mayes DC, Gibney RTN: Impact of a critical care clinical information system on interruptions experienced by intensive care unit health care providers: "Our whole lives are interruptions." [abstract]. In 5th International Mixed Methods Conference. Harrogate, United Kingdom; 2009. OpenURL

  26. Roethlisberger FJ, Dickson WJ: Management and the Worker. Harvard University Press; 1939. OpenURL

  27. Westbrook JI, Coiera E, Dunsmuir WT, Brown BM, Kelk N, Paoloni R, Tran C: The impact of interruptions on clinical task completion.

    Qual Saf Health Care 2010, 19:284-289. PubMed Abstract | Publisher Full Text OpenURL

  28. Manor-Shulman O, Beyene J, Frndova H, Parshuram CS: Quantifying the volume of documented clinical information in critical illness.

    J Crit Care 2008, 23:245-250. PubMed Abstract | Publisher Full Text OpenURL

  29. Chisholm CD, Collison EK, Nelson DR, Cordell WH: Emergency department workplace interruptions: are emergency physicians "interrupt-driven" and "multitasking"?

    Acad Emerg Med 2000, 7:1239-1243. PubMed Abstract | Publisher Full Text OpenURL

  30. Coiera EW, Jayasuriya RA, Hardy J, Bannan A, Thorpe ME: Communication loads on clinical staff in the emergency department.

    Med J Aust 2002, 176:415-418. PubMed Abstract | Publisher Full Text OpenURL

  31. Shaw NT, Ballermann MA, Hagtvedt R, Ho S, Mayes DC, Gibney RTN: Intensive care unit nurse workflow during shift change prior to the introduction of a critical care clinical information system. electronic.

    Journal of Health Informatics 2011, 6:e5. OpenURL

  32. Westbrook JI, Woods A, Rob MI, Dunsmuir WT, Day RO: Association of interruptions with an increased risk and severity of medication administration errors.

    Arch Intern Med 2010, 170:683-690. PubMed Abstract | Publisher Full Text OpenURL

  33. Jones SRG: Was There a Hawthorne Effect?

    The American Journal of Sociology 1992, 98:451-468. Publisher Full Text OpenURL

  34. Dean B, Barber N: Validity and reliability of observational methods for studying medication administration errors.

    Am J Health Syst Pharm 2001, 58:54-59. PubMed Abstract | Publisher Full Text OpenURL

  35. Schnelle JF, Ouslander JG, Simmons SF: Direct observations of nursing home care quality: Does care change when observed?

    J Am Med Dir Assoc 2006, 7:541-544. PubMed Abstract | Publisher Full Text OpenURL

Pre-publication history

The pre-publication history for this paper can be accessed here:

http://www.biomedcentral.com/1472-6947/11/32/prepub