Email updates

Keep up to date with the latest news and content from BMC Nursing and BioMed Central.

Open Access Highly Accessed Research article

Conceptualizing performance of nursing care as a prerequisite for better measurement: a systematic and interpretive review

Carl-Ardy Dubois1*, Danielle D’Amour1, Marie-Pascale Pomey2, Francine Girard1 and Isabelle Brault1

Author Affiliations

1 Faculty of Nursing, University of Montreal, Montreal, Canada

2 Department of Health Administration, Faculty of Medicine, University of Montreal, Montreal, Canada

For all author emails, please log on.

BMC Nursing 2013, 12:7  doi:10.1186/1472-6955-12-7

The electronic version of this article is the complete one and can be found online at: http://www.biomedcentral.com/1472-6955/12/7


Received:15 July 2012
Accepted:7 February 2013
Published:7 March 2013

© 2013 Dubois et al; licensee BioMed Central Ltd.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Background

Despite the critical role of nursing care in determining high-performing healthcare delivery, performance science in this area is still at an early stage of development and nursing’s contribution most often remains invisible to policy-makers and managers. The objectives of this study were: 1) to develop a theoretically based framework to conceptualize nursing care performance; 2) to analyze how the different components of the framework have been operationalized in the literature; and 3) to develop a pool of indicators sensitive to various aspects of nursing care that can be used as a basis for designing a performance measurement system.

Methods

We carried out a systematic review of published literature across three databases (MEDLINE, EMBASE and CINAHL), focusing on literature between 1990 and 2008. Screening of 2,103 papers resulted in final selection of 101 papers. A detailed template was used to extract the data. For the analysis, we used the method of interpretive synthesis, focusing first on 31 papers with theoretical or conceptual frameworks; the remaining 70 articles were used to strengthen and consolidate the findings.

Results

Current conceptualizations of nursing care performance mostly reflect a system perspective that builds on system theory, Donabedian’s earlier works on healthcare organization, and Parsons’ theory of social action. Drawing on these foundational works and the evidence collated, the Nursing Care Performance Framework (NCPF) we developed conceptualizes nursing care performance as resulting from three nursing subsystems that operate together to achieve three key functions: (1) acquiring, deploying and maintaining nursing resources, (2) transforming nursing resources into nursing services, and (3) producing changes in patients’ conditions. Based on the literature review, these three functions are operationalized through 14 dimensions that cover 51 variables. The NCPF not only specifies core aspects of nursing performance, it also provides decision-makers with a conceptual tool to serve as a common ground from which to define performance, devise a common and balanced set of performance indicators for a given sector of nursing care, and derive benchmarks for this sector.

Conclusions

The NCPF provides a comprehensive, integrated and theoretically based model that allows performance evaluation of both the overall nursing system and its subsystems. Such an approach widens the view of nursing performance to embrace a multidimensional perspective that encompasses the diverse aspects of nursing care.

Keywords:
Performance measurement; Nursing care; Systems theory; Quality of care; Nursing structure; Nursing processes; Nursing sensitive outcomes; Systematic review.

Background

Nurses represent the largest occupational group in the healthcare workforce, providing the most care at all levels of the care continuum and accounting for a significant proportion of hospitals’ operating costs [1,2]. Yet nursing’s contribution most often remains virtually invisible to policy-makers and healthcare managers, and many analysts consider it undervalued and understudied [3,4]. Symptomatic of the situation, data considered sensitive to nursing care are, in most cases, not represented in databases that are routinely examined for performance analysis in healthcare organizations and health policy decisions [5,6]. The causes attributed to these symptoms include poor conceptualization of nursing performance, inadequate measures of nursing contribution, inadequate information systems to capture and manipulate nursing performance data, and absence of a standardized language [7,8].

Concerns about the shortage of nurses and its potential adverse effect on patient safety, coupled with mounting public expectations regarding the value of services consumed, have created a sense of urgency about the need for monitoring performance of nursing services [9]. In the current context of health service reforms, driven partly by resource constraints and consumer pressures, increasing demands are being placed on nursing administrators and nursing care providers. They must ensure the performance of nursing services, give a more comprehensive and accurate picture of what they do, and demonstrate the value and benefits of their services in line with established objectives and standards [10,11]. This renewed interest has resulted in an accelerated expansion of a range of initiatives within and outside the nursing profession to make explicit those aspects of care outcomes directly attributable to nursing practice. Specifically, indicators to quantify nursing’s contribution to healthcare performance have been considered in several countries by influential organizations such as the Agency for Healthcare Research and Quality and the National Quality Forum in the United States, the Ontario Hospital Association in Canada, and the Council on Health Care Standards in Australia [12,13].

Despite such recent efforts, the research base on performance measurement in the area of nursing care is still considered somewhat embryonic [14], and no common theory-driven schema guides the nursing discipline, the regulatory agencies and the provider organizations in their efforts to define, organize and operationalize the dimensions of nursing care performance [15]. There are many fragmented pieces of knowledge, but what is critically missing is a comprehensive framework in which to fit the pieces together and that can guide the implementation of performance assessment activities in nursing. In many cases, the terminology of performance measurement can itself be confusing and is characterized by a wide array of terms and concepts that include productivity, outcomes, effectiveness, efficiency, quality, etc. As there is no agreed-upon definition, performance measurement systems are also conceptualized in multiple ways [16].

The overall aim of this research was to draw on the most recent developments in conceptual modelling to help build a common orientation on how to capture nursing care performance. More specifically, the objectives of the research were: 1) to develop a theoretically based framework to conceptualize nursing care performance; 2) to analyze how the different components of the framework have been operationalized in the literature; and 3) to develop a pool of indicators sensitive to various aspects of nursing care that can be used as a basis for designing a performance measurement system.

While there is already a substantial amount of literature on the topic of healthcare performance, the focus of our study was on performance metrics potentially sensitive to nursing care. Although other reviews are available in the literature [17,18], none have been as exhaustive and none have systematically analyzed the existing models using a structured template, as we did. Furthermore, while the notions of quality and performance are frequently used interchangeably, our study acknowledges both their conceptual linkages and differences. A system can only be said to be performing if it delivers high quality interventions, care or services [19]. However, performance and quality are not necessarily identical and interchangeable concepts. A widely accepted definition of quality proposed by the Institute of Medicine conceptualizes quality as the degree to which health services for individuals and populations are consistent with current professional knowledge and increase the likelihood of desired health outcomes [20]. From this perspective, quality is at best only a proxy of healthcare performance, which is understood as a much broader concept. To be consistent with developments in contemporary literature, the notion of performance in this study is embedded in a general vision of key functions that an organizational entity must fulfill in a given environment. It refers to the ability of a health unit, organization, or system to perform its diverse functions most effectively and efficiently and to ensure the coordination and equilibrium among these functions needed to achieve its goals. From this perspective, nursing care performance can be measured based on those attributes or dimensions related to the functioning of organizational entities involved in nursing care provision. Those dimensions are not necessarily direct measures of quality, but rather, they cover what an organization does to provide nursing care, how it does it in relation to articulated goals and functions, and the resulting outcomes.

This article presents the Nursing Care Performance Framework (NCPF), describes its key components and identifies indicators related to each component. This NCPF will provide policy-makers and managers with an effective tool to structure performance evaluation systems that are consistent with their overall goals. This framework will widen the view of nursing performance to encompass the diverse aspects of nursing care.

Methods

To achieve the above-mentioned objectives, our research built upon developments in conceptual modelling and advances in systematic review methods as applied to organizational research [21]. Our approach drew on explicit, systematic and reproducible methodology to locate, select and appraise the available literature and to extract and analyze the relevant data [22,23]. It can be defined as an interpretive synthesis. It is interpretive in the sense that the focus was not on simply aggregating or summarizing data reported in the literature but rather drawing on a large and complex body of evidence to build a conceptual framework grounded in the studies included in the review [24,25]. As summarized in Figure 1, the review followed the main phases that have been systematized in previous works: literature searches, screenings and appraisals, data extraction, and analysis and interpretative synthesis [21,22].

thumbnailFigure 1. Methodological steps.

Literature search

The search for relevant literature was made across three databases. MEDLINE, EMBASE and CINAHL were selected, based on preliminary tests applied to a broader range of databases. Other potential sources, such as the Cochrane and Francis Costello libraries, were rejected after preliminary tests found them unsatisfactory, as they generated hundreds of irrelevant items. After several iterations and advice from a professional librarian, we adopted key words and applied them to the three selected databases. The key words were nursing care performance, nursing performance, nursing-sensitive performance measurement, nursing-sensitive measures, and quality indicators and nursing care. The robustness of the strategy was confirmed by comparison with a predetermined list of key articles referenced in a recent review article [17], all of which were captured by the search when they were clearly related to performance in the area of nursing care. After removal of duplicates, the papers collated at this phase totalled 2,103. The search was limited to literature between 1990 and 2008, to focus on most recent developments in this area. However, reference chaining was used to uncover additional key articles published before this period.

Screening and appraisal

Our six researchers were paired into three teams and the articles’ abstracts split among the teams. Inclusion criteria were tested on a sample of 15 abstracts. Each abstract was systematically reviewed independently by two researchers based on criteria of: 1) relevance (clear focus on the theme of performance; clear focus on the domain of nursing care); 2) potential contribution to the conceptualization or measurement of performance (development of a theoretical model, elaboration of performance measures, performance evaluation, empirical analysis); and 3) use of explicit and valid methods. For each case, a decision was taken to either exclude the paper or select it for the next step. When there was disagreement, the two researchers met to discuss their views and find common ground. Letters, comments, editorials and anecdotal papers were systematically excluded. The criterion on methodological quality was intentionally broad for this initial screening. At this step, we aimed to include as many relevant papers as possible that met basic methodological standards or were likely to contribute at the level of concepts, while excluding studies that were irrelevant or fatally flawed. One hundred and eighty-one papers passed this screening and advanced to the next step.

Final screening and data extraction

The 181 papers were downloaded. We designed a data extraction template that was intended, first, to confirm the relevance of each paper (clear focus on performance conceptualization and/or measurement in the domain of nursing care) and its scientific quality (theoretical foundation, design, sampling, data collection, explicit criteria of validity) and then to extract further data. Again, the six researchers were grouped into teams of two. We pretested the data extraction tool on 45 articles split among the three teams, refined the tool, and then applied it to the entire sampling. In-depth examination of each article and data extraction were first completed by one member of a team and then validated by the other. It was still possible at this step to exclude a paper if it was deemed irrelevant or methodologically flawed. Again, when there was disagreement, the two researchers met to discuss their views and find common ground. Of the 181 articles, only 101 merited complete data extraction and analysis.

Information extracted from all the 101 papers included country, type of study, appropriate criteria of validity based on the type of study (ex: representative sampling, data saturation), type of setting, study focus (development of framework, development of indicators, empirical analysis of links between variables) and a list of performance indicators (variables or markers proposed or used to measure performance). The main context of interest was acute care for 65% of the papers, long-term care for 27% and primary care for 8%. This information was supplemented with grey literature, reference chaining and scanning of key journals specializing in performance issues. The three pairs of reviewers were also asked to flag papers with theoretical or conceptual frameworks; 31 such papers were identified on the basis of two criteria: 1) papers with an explicit framework that specified the dimensions of the concept of nursing care performance and/or how these dimensions can be operationalized; and 2) reviews that used a coherently organized structure to present the various aspects of nursing care performance.

Data analysis

Our approach to analyzing the data involved constructing a general interpretation grounded in the findings of separate studies and then integrating evidence from across studies into a coherent theoretical framework comprising a network of constructs [25,26]. For this phase, the analysis focused first on the 31 papers with theoretical frameworks. Our approach was similar to that often undertaken in primary qualitative research. We began with detailed inspection of the papers, gradually identifying recurring variables used to measure performance. We then grouped these variables into different subdimensions and dimensions, constantly comparing the conceptual structures we were developing against the data in the papers and the models collated, and attempting to specify the relationships between the different variables, dimensions or subdimensions identified. The information extracted from the 70 other papers were incorporated in a second stage to strengthen and consolidate the findings drawn from the first 31 papers, confirm the performance indicators identified and their groupings into dimensions and subdimensions, look for any additional variable or dimension that would not have been captured in the models, and document the relationships between the variables identified. All team members were involved in this analysis and the continual dialogue between them helped to ensure a reflexive account of the processes followed all along the different steps.

Results and discussion

A diversity of perspectives with a common thread

A first observation that emerged from the corpus of papers examined was that there are a significant number of activities under way to standardize measures of diverse aspects of nursing work and capture nursing’s contribution to care. The review revealed three parallel streams of activities that have contributed to varying degrees to conceptualizing and measuring performance in nursing care.

A first stream of activities focuses on encoding and classifying all aspects of nursing activities, including nursing diagnoses, nursing interventions and nursing outcomes [27,28]. Those classification systems are viewed as measurable and standardized ways to sort out those health conditions or clinical situations that are amenable to nursing interventions and for which nursing holds the ultimate responsibility for patient outcomes. A second stream of activities, grounded in the patient safety movement, is dominated by a search for evidence relating given factors in nursing structures or processes (e.g. nurse staffing, nursing unit characteristics) to safety failures or poor outcomes for patients [29,30]. A third stream of activities, closely associated with the quality assurance movement, is primarily concerned with developing evidence-based standards of care and measuring resources, processes and interventions that must be in place to optimize care quality and outcomes [31].

While these activities follow different directions, the review also showed they share a common thread. In the three streams, nursing care performance involves the analysis of multiple interacting elements that relate to the diverse aspects of nursing services, their antecedents and their results. This reflects a conception of nursing care as a complex, aggregate entity, comprised of multiple interrelated and interdependent subsystems and components that are logically coordinated and oriented toward the achievement of common goals [32]. As the review unfolded, a core finding that emerged from the analysis was that these diverse streams of activities, to varying extents, built upon Donabedian’s model of healthcare organization and delivery, which in turn owes much to Parsons’ theory of social action and systems theory. Taken together, these theoretical foundations help us understand how inputs are acquired from the nursing care environment and fed into the service production cycle, where transformation of resources results in changes in patient conditions [33]. As such, they provide a strong basis for organizing and integrating the diverse and often fragmented perspectives of nursing care performance.

Donabedian’s structure, processes and outcome (SPO) framework provides a tool that efficiently articulates three subdimensions representing three key components of the healthcare supply chain [34]. In many cases, the three components of the SPO model have been used as a taxonomy to identify and classify those factors or building blocks that must be put together to assess nursing care performance. However, while the SPO triad highlights the different components and the underlying anatomy that define the performance of a given system, it may be not sufficient for understanding the operating mechanisms of these diverse components and their interactions. Parsons’ framework for social analysis provides a functional organismic perspective that affords further insight into the performance of a given system by conceptualizing the social system as a system of interactions between different subsystems, mostly defined in terms of functions [35]. These functions that any system needs to align and balance in order to perform well include goal attainment, production, adaptation to the environment, and values’ maintenance. As such, Parsons’ framework may be useful to understand the physiology of a performance system, conceptualized as a set of specialized functions that must be achieved by autonomous functional subsystems. The Parsons functions provide a generic foundation that has been used to define the criteria on which human services performance must be assessed [36]. This is also congruent with systems theory. A systems approach to nursing care assumes that the nursing system is composed of interrelated subsystems and components that carry out specialized functions. If any of these subsystems performs inadequately, it will affect the performance of the whole system. This means that a highly complex nursing care system must first be broken into subsystems so that each can be analyzed and understood before being reassembled into a whole. Each subsystem must achieve high performance in its own functions but also join with other subsystems and units to optimize the performance of the whole [37]. Although these functional subsystems may be relatively self-sufficient, a systems perspective recognizes the interaction and reciprocal interconnection among their different functions [37].

We used the complementarity of these foundational works to analyze the selected articles and determine the current state of performance measurement in nursing care. Drawing upon these works and the evidence collated, we developed the Nursing Care Performance Framework (NCPF) to integrate diverse perspectives and take into account the diverse aspects of the nursing system. Donabedian’s SPO triad provides the underlying anatomy of this framework. Parsons’ social theory helps to understand the operating mechanisms and functions that underlie each of the constitutive components. The system perspective takes into account the linkages between those different components or subsystems and the environmental factors influencing them. The 31 models we reviewed differ widely in the translation of these theoretical foundations and reveal diverse gaps in measuring nursing care performance. We observed wide differentials in the existing models and approaches regarding how performance is conceptualized and subsequently operationalized. Some models exhibit a plethora of indicators reflecting diverse theoretical, methodological and practical perspectives. Other frameworks focus on selected and restricted aspects of nursing practice, while neglecting others. Structural indicators that may be more easily measurable and more readily available in existing administrative data sets are often put forward to the detriment of more complex process indicators that could better reflect nursing activities. Several frameworks or models reviewed are simply a list of indicators or broad domains and do not clearly indicate the conceptual dimensions and subdimensions that underlie these indicators. While many models built on the Donabedian structure–process–outcomes (SPO) framework, often each component of the triad is taken separately, without attention to the linkages between them. Variations among the existing models also reflect the lack of a common terminology and the use of a plethora of terms and concepts with inconsistent definitions. Despite these limitations, our analysis is built on the hypothesis that conceptualization of performance in the domain of nursing may benefit from the combination of different perspectives and models. Taken together and used in combination, guided by the theoretical foundations mentioned above, existing models may provide a more accurate picture of nursing system performance than would each one separately.

The nursing care performance framework

The NCPF, illustrated in Figure 2, conceptualizes nursing care performance as resulting from three nursing subsystems that operate together to achieve three key functions: (1) acquiring, deploying and maintaining nursing resources, (2) transforming nursing resources into nursing services, and (3) producing positive changes in a patient’s condition as a result of providing nursing services. These three functions, derived initially from the theoretical foundations mentioned above, served as an analytical tool to capture the main dimensions emerging from efforts to conceptualize and measure performance in the articles reviewed. Table 1 shows how our proposed framework relates to Donabedian’s SPO triad, Parsons’ functional analysis and systems theory and builds on these foundations to develop a broad view of nursing care performance. In the NCPF, each of the three subsystems is composed of multiple components that define the structure, processes and outcomes of the overall system. Within each subsystem, the components are involved in multiple processes of interaction to achieve a specialized function. The underlying systemic perspective highlights the linkages between these functions.

thumbnailFigure 2. The nursing care performance framework.

Table 1. Theoretical foundations for the NCPF (Nursing Care Performance Framework)

Building upon the perspectives mentioned above, the NCPF suggests a broader view of nursing care performance that can be defined as the capacity demonstrated by an organization or an organizational unit to acquire the needed nursing resources and use them in a sustainable manner to produce nursing services that effectively improve patients’ conditions. Contrary to many prevailing definitions, nursing care performance is not restricted to the end goals or outcomes of the nursing system, but refers also to the effectiveness of those upstream functions that provide the means necessary to achieve nursing system goals.

As shown in Tables 2, 3 and 4, the three functions of the NCPF summarize the universe of dimensions and subdimensions used to conceptualize nursing care performance in the 31 models collated (see the models list, Table 5). The same dimensions and subdimensions were found in the 70 additional articles (empirical analyses, descriptive articles) retained after the screening exercise. Tables 2, 3 and 4 show not only the extent to which the 31 performance models inventoried integrate the three subsystems, but also how each subsystem is operationalized in each model. Assuming these models summarize the universe of content commonly accepted as the main components of nursing care performance, we listed all variables present in the 31 models. Variables with similar meanings but differences in denominations were renamed to facilitate their groupings under one heading. We present below the three nursing care subsystems and their components. We also provide evidence on interrelations both between each subsystem’s components and between the subsystems themselves.

Table 2. Operationalization of the subsystem of acquiring, deploying and maintaining nursing resources

Table 3. Operationalization of the subsystem of transforming nursing resources into relevant nursing services

Table 4. Operationalization of the subsystem of producing changes in patients’ condition

Table 5. Models list

Acquiring, deploying and maintaining nursing resources

No system for healthcare delivery can fulfill its objective of providing care and improving patient health without deploying the necessary human and material resources. Therefore, as revealed by our review of the 31 models, the first key function of a nursing system is to acquire, deploy and maintain the resources needed to provide nursing care. Twenty-six of the models include indicators related to this function, which has been defined as the ability to ensure a sound and efficient stewardship in acquiring and managing the needed nursing resources [67]. To achieve this function, the nursing system must perform on four key dimensions (see Table 2).

Supply of nursing staff

Nursing care is labour-intensive, and delivering effective nursing services depends on the availability of staff with the skills and competencies necessary to address patients’ specific needs in a timely manner. The supply of nursing staff, as a performance measure, thus reflects the effectiveness of diverse activities that govern nursing staff intake (planning, recruitment, selection) and deployment and must ensure an adequate balance with the demand for nursing services. Supply is concerned with not only the quantity of available staff, but also their types (educational preparation, qualifications, and experience) and their mix. It must also be examined in the capacity of available staff to address the specific needs of patients they care for, and this raises the importance of a resource allocation system that takes into account the acuity of patients’ conditions.

Nurses’ working conditions

Two aspects of this dimension emerged from the literature reviewed. One was the diverse types of material resources that are required to support the work of nursing staff. These include physical facilities, technologies, organizational configurations and financial resources, all factors identified in several models as precursors that define the overall organization of nursing care and determine the extent to which nursing staff are able to perform their roles. The other aspect consisted of employment characteristics, which encompass a number of issues ranging from workload, scheduling and employment status to system-wide issues such as labour relationships. This aspect reflects the ability to create conditions that may attract nursing workers and ensure their stability in the workforce. This aspect is captured in the inventoried models by a series of indicators, including balance between temporary and permanent staff, use of agency personnel, ratios of floating staff, balance between full-time and part-time staff, and ratios of overtime hours.

Maintenance and economic sustainability of the nursing staff

These two dimensions can be considered to a certain extent as contingent upon activities developed in the two first dimensions (supply and working conditions). Poor working conditions and inadequate staffing, often reflected in heavy workload, unpredictable and inflexible schedules, excessive overtime, and threat of job loss, influence workforce maintenance by limiting the capacity to recruit new nurses and retain those already employed. It has been demonstrated that the same factors are significant predictors of work-related illnesses, injuries and job dissatisfaction, which in turn lead to increased absenteeism and turnover [68-71]. The costs associated with these adverse effects come in diverse forms: productivity costs, wage replacements and disability payouts.

Beyond the ability to manage the inevitable outflow of resources, a perennial challenge for every organization is to sustain its economic capacity to acquire and maintain the resources needed. This sustainability reflects a unit’s or organization’s ability to obtain needed inputs as economically as possible (that is, obtaining appropriate quality resources at least cost). Other indicators, such as cost per case-mix, highlight the issue of productivity and the imperative of maximizing the outputs produced from a given set of inputs or, alternatively, minimizing the inputs of nursing tasks, materials, and equipment to produce nursing services.

Thus, with regard to this first function, nursing system performance covers a set of four dimensions that were operationalized into 14 variables. Five of the 31 models have no clear variables related to this subsystem. In the remaining 26, the number of variables used to conceptualize this subsystem varies widely, from one to 11. The most prevalent variables were related to nursing staff supply: quantity/intensity (17 times), quality (18) and skill mix (18). The issue of cost per case-mix figures in 11 models.

The usefulness of indicators related to this first subsystem for assessing nursing care performance may be justified on several grounds:

Their potential to detect latent failures at the “blunt end” of the nursing system

The indicators mentioned above may reveal a broad set of dysfunctions and deficiencies, such as insufficient supply of basic material resources, gaps between nurse staffing and patients’ needs, heavy workload and staff instability. Such deficiencies are considered as latent failures that do not involve the practitioner and do not result from direct contact with the patients but that may have potentially delayed consequences on both nursing care processes and outcomes. As an example, inappropriate nurse staffing may not necessary result in immediate harm for patients but reveals a latent failure that may force nurses to reduce the surveillance function, thus creating unsafe delivery conditions that may ultimately result in adverse events among patients.

Their sensitivity to nursing management

These indicators’ sensitivity to nursing management (e.g. staffing decisions) gives them an explanatory value for differing performances across care units or organizations. The four dimensions that define this subsystem are all modifiable factors that result from managerial decisions and can be enhanced through policy initiatives at the unit, organizational and system levels.

Their linkage with nursing care processes

Assessing this function makes it possible to judge whether nursing care is being provided under conditions that are conducive to optimal nursing processes. Although the link between structural attributes and care processes has been shown to be inconsistent, the literature reviewed contains a number of examples illustrating the potential application of structural indicators to predicting the performance of nursing processes. Research has shown that nursing inputs influence RNs’ ability to care for patients, therefore affecting surveillance and other processes of care [72]. Higher patient-to-RN ratios and nursing workload have been associated with more tasks left undone by RNs and rationing of nursing care [49,73]. Low levels of staffing or heavy workload may result in reductions in time spent by nurses collaborating and communicating with other providers and patients, thereby affecting the quality of both interprofessional coordination [74] and nurse-patient communication [75].

Their linkage with patient outcomes

Measures of such structural aspects may be also useful to predict patient outcomes or to specify the conditions under which outcomes are produced. A number of studies and systematic reviews suggest that higher staffing levels and a richer staff-mix and skill-mix may be associated with better outcomes and fewer adverse events for patients [76,77]. Conversely, poor working conditions, inadequate staffing resulting in heavy workload or extended working hours, and problems reflective of staff maintenance, such as high turnover and excessive utilization of temporary staff from external agencies, have been linked with increasing risks to patient safety [78-81].

Transforming nursing resources into relevant nursing services

The effective stewardship of nursing resources, as described above, is not an end state. A second key function of the nursing system is to transform available resources into nursing services that address patients’ needs. This function involves a broad set of processes and mechanisms that reflect not only what staff nurses do for, with, and on behalf of patients and their families, but also what nursing managers do to support nurses’ work and create an appropriate practice environment. It encompasses, as well, how patients are engaged in their own care processes and how both staff and patients live their experience of care. Among the 31 models inventoried, 28 include indicators related to this function. Those are covered in four interrelated dimensions, described below.

The nursing processes

Nursing care provision involves a range of individual, family, community and population-based approaches. It is materialized through interventions and processes that reflect the deployment of nurses’ scope of practice, including assessment, planning and evaluation, problem and symptom management, health promotion and illness prevention, care coordination, and discharge planning. From a provider perspective, they capture the technical aspects of care and reflect the degree to which staff are able to use all their competencies and deploy their full scope of practice. Such processes also reflect the ability to provide care that meets the patient’s overall needs.

Nurses’ practice environments

While effective nursing interventions are considered antecedents of health outcomes, nurses’ ability to perform these interventions is closely and consistently associated with organizational processes that define the nursing practice environment and mediate the outcomes [54,82]. These processes have been conceptualized as interventions to support nursing work and create a professional environment for nurses [83]. Many of the indicators for capturing these processes that emerged from this review echo the characteristics of magnet organizations. These indicators include: support for nurses from co-workers and administration; communication and collaboration among nursing staff; collegiality between nurses and other professional groups; a motivating work climate; decentralization of decision-making with responsibilities for nursing services devolved to the nursing unit; and autonomy for nurses in their role.

Patient experience and professional satisfaction

These two dimensions are to a certain extent contingent upon the first two. Patient experience can be considered the result of clinical and organizational processes that should optimally ensure patients receive the right nursing care, at the right time, and in the right way for them. Such a measure is essential to assess the acceptability and appropriateness of nursing care from the patient perspective. A number of frameworks have been developed to conceptualize patient experience [20,84-87]. Yet the central theme, overarching all others in the nursing literature reviewed, relates to the patient-centredness of nursing care. Holistic, patient-centred care has often been seen as the essence of nursing care and interpreted as nurses’ ability to communicate effectively with patients, work with patients’ expectations and values, engage patients in their care, deliver services that are sensitive to patients’ needs, and ensure patients’ smooth transition across the care system [88,89]. This notion covers not only the technical and organizational aspects of nursing processes, but also their interpersonal dimension. It is reflected in the 31 models through a number of indicators that we grouped inductively into three attributes: 1) care continuity, which includes diverse facets, such as coordination of care, patient transition, and intra- and interprofessional collaboration; 2) patient/family involvement, which is concerned with engaging patients and their families in their care, and the types of support provided to them to empower them and foster self-care; and 3) responsiveness, which also encompasses several facets, including respect for patients’ preferences, quality of communication with patients and their families and degree of comprehensiveness of care.

Nurses’ professional satisfaction may be also considered the result of nursing processes. Although nurses can be rewarded through different facets of their work, their professional satisfaction mostly results from specific aspects that influence their perception of their ability to accomplish their daily assignments and enjoy the work itself. As such, professional satisfaction relates to those Herzberg motivator factors and intrinsic rewards that are directly linked with the work itself (e.g. responsibility, achievement, growth) as opposed to hygiene factors (e.g. salary, working conditions, supervision) [90]. Measures of professional satisfaction have also built upon the work of Hinshaw and colleagues, for whom nurses’ job satisfaction is a combined function of their perception of the quality of care they provide, having the time to do their job, and the enjoyment derived from it [91].

Table 3 summarizes how this second subsystem is conceptualized and operationalized within the 31 conceptual models reviewed. In total, the review identified 10 variables split across the four dimensions. Only three models have no indicator related to this subsystem. For the remaining 28, the number of variables used to conceptualize this subsystem ranges from one to seven. The most prevalent variables in these models are patient/family involvement (13 times); problems and symptoms management (13), nursing work practice environment characteristics (12), assessment, planning and evaluation (8) and care continuity (8).

The usefulness of these four sets of indicators may be justified on three main grounds:

Their potential to detect active or imminent failures at the “sharp end” of the nursing system

An intrinsic advantage of indicators related to this function is that they are more sensitive to differences in quality of care than are structure and outcome indicators. They may reveal dysfunctions and deficiencies in the clinical microsystem, where providers interact directly with patients to deliver care. Inappropriate nurse interventions, miscommunication between nursing team members, and continuity gaps are examples of dysfunctions that create holes in the nursing system’s front line of defense. Such holes are immediate causes of imminent or active failures that undermine the patient’s experience of nursing care, compromise the provision of adequate services, predispose those services to errors, and ultimately may result in adverse outcomes.

Their sensitivity to nurse interventions

These indicators refer to the essence of what nurses do to improve patients’ conditions [92]. They provide insights into the clinical and organizational processes that reflect both how nurses enact their roles and the conditions experienced by patients in their interactions with the nursing system. They also provide information that is actionable. Knowing which processes are ineffective is essential for both readjusting how nursing care is provided and redeploying nursing resources.

Their linkage with patient outcomes

Specific nursing processes, characterized by patient involvement, patient education, self-care assistance and psychological support, have been associated with better outcomes for patients [93-95]. Doran and colleagues provided evidence that patients’ therapeutic self-care ability was predicted by nurses’ coordination of care, communication with other health care providers, and provision of high-quality technical care [38]. Other studies have shown that organizational models of nursing care which result in greater nurse autonomy, more control by nurses of resources, and better relations between nurses and physicians yield better patient satisfaction and outcomes [96,97].

Producing positive changes in patients’ conditions

Beyond the imperatives of effective stewardship of nursing resources and their transformation into services that meet high standards of quality, the desired end result of the interactions between patients, nursing staff, and nursing processes is to produce outcomes that lead to positive changes in a patient’s functional status, disease state or evolving condition. All 31 models include indicators related to this third function. Using an inductive process, we identified six distinct but interrelated dimensions from the plethora of outcome indicators in the literature reviewed. The first five categories are considered nursing-sensitive outcomes, that is, outcomes that are affected, provided, and/or influenced by nursing personnel [80,98], although nursing may not be exclusively responsible for them. The sixth category includes outcomes to which nurses may contribute to some extent, but which are less informative for assessing nursing interventions because these outcomes may be more influenced by other systems.

Outcomes reflecting patient safety

Nursing care services are provided to patients in an environment with complex interactions that can generate harm, errors and unintended outcomes. Because of this, patient safety is largely considered an indicator of high-performing nursing care. This is related to nurses’ roles in integrating care, detecting hazards and, ultimately, preventing errors and adverse events [99,100]. Failures in fulfilling these roles may result in errors of omission and commission, as well as adverse events. This review identified a number of safety-related outcomes considered potentially sensitive to nursing: patient falls, injuries, medication errors, pulmonary infections, pressure ulcers, urinary tract infections, intravenous infections, abuses, and failure to rescue.

Outcomes reflecting patient comfort and quality of life related to care

Beyond harm prevention, the value of nursing care services lies also in their potential to relieve discomfort and improve the quality of life in the context of care. From this standpoint, nursing system performance reflects the extent to which patients’ needs in relation to personal hygiene, nutrition, management of symptoms (pain, dyspnea), and continence are met, unnecessary interventions (physical or chemical restraints, nasogastric tubes, prolonged use of urinary catheters) are avoided, and patients’ respect is ensured throughout the episode of care.

Outcomes reflecting changes in patients’ knowledge, skills, and behaviours

Such outcomes reflect patients’ ability to influence their own conditions through self-care and new behaviours. The indicators here focus on knowledge gained, attitudes changed, skills acquired, and behaviours adopted as a result of nursing interventions. Examples include patients’ knowledge of the prescribed treatment, ability to recognize symptoms, ability to carry out treatments as prescribed, and demonstration of health promoting behaviours.

Outcomes reflecting patients’ functional status

This category of indicators covers essential end results and benefits that reflect what happens in people’s lives as a result of nursing care interventions. In the models examined, these indicators encompass diverse aspects of patients’ general functional status and conditions, including physical, psychosocial and cognitive status.

Outcomes reflecting patient satisfaction with the care experience

These indicators focus on patients’ satisfaction with their care experience and the evolution of their condition. This subjective evaluation by patients reflects the interaction of their expectations of care and their perceptions of actual outcomes resulting from provider services [101]. The nature of nurses’ responsibilities is such that they are potentially well positioned to influence patients’ conditions and how the health system functions across all aspects of patient care. Hence, patients’ satisfaction with nursing care is considered in several models to be both an outcome of nursing services and a primary determinant of overall satisfaction with an episode of care.

Outcomes reflecting a joint contribution of nursing care and other care systems

Nursing as a “whole system” interacts with several other “whole systems” to achieve outcomes. Positive change in patients’ health status is an ultimate outcome that most often reflects the contribution of several systems of care and many healthcare disciplines. Conversely, negative outcomes such as mortality, morbidity, complications, extended length of stay and readmissions reflect failures that may have multiple causes and can originate from one or more systems. Although such events cannot easily be imputed either to nursing or to any specific care system, they still reflect outcomes of each of these systems and warrant consideration in assessing the performance of any system.

Table 4 shows how this third function is conceptualized and operationalized in the models reviewed. In total, the review identified 27 variables, which we grouped into six dimensions. All 31 models include outcome indicators. The number of variables used to conceptualize this subsystem ranges from one to 15. The most prevalent variables cut across the different categories of nursing-sensitive outcomes and include symptom management (22 times), skin integrity (17), falls (17), patient satisfaction (17), physical functional capacity (17), cognitive and psychosocial functional capacity (15), medication management (14), nosocomial infections (12) and ability to achieve appropriate self-care (11). The linkages between these outcomes related indicators and the two other subsystems have been described in the previous sections.

Conclusions

Across the three subsystems, at least 51 variables, grouped into 14 dimensions, have been used to assess these subsystems’ performance in achieving their functions. Cross-validation by examination of the 70 additional papers revealed no additional generic variables, but in some cases, specific indicators related to a specific clinical domain and linkages between those variables. Figure 3 summarizes the dimensions and indicators inventoried for the three subsystems, based on the review.

thumbnailFigure 3. The nursing care performance framework.

While all 51 variables are valid indicators that together represent the entire domain of nursing care performance as a construct, it can be seen in Tables 2, 3 and 4 that no individual model covers this entire domain. Rather, the different models show a variety of combinations of potential variables, reflecting the diversity of options for measuring nursing care. However, it is worth noting that, for nearly all the models, the scope of nursing care performance variables extends to more than one subsystem. In fact, 25 of the 31 models include indicators related to all three subsystems. Of the others, three focus on the second and third subsystems, one includes indicators related to the first and third subsystems, and two restrict their scope to outcome indicators.

From a policy perspective, this review clearly shows that a systems approach provides a useful and powerful tool to build a comprehensive framework for performance evaluation of both the overall nursing system and its subsystems. By incorporating the three functions described above, the NCPF offers an effective way to compile a complete picture of overall nursing care performance in any setting. It suggests a shift away from a narrow view of nursing performance to a broader perspective that encompasses more complex and more complete measures that take into account extended aspects of nursing activities. As such, the NCPF’s systemic perspective may help avoid many of the dysfunctional strategies that often guide performance measurement [102-105]. One such dysfunctional strategy, often described as tunnel vision, consists of concentrating on certain limited areas of performance while neglecting other important but unmeasured areas. Another such strategy is to focus on short-term issues and proximal outcomes that are more easily detectable, to the exclusion of long-term issues or distal outcomes, which may be more difficult to measure or take some time to become measurable. A third strategy is to focus on indicators that are more readily available (in most cases, structural indicators) while overlooking more complex measures. Finally, another strategy is to focus on isolated indicators without taking into account the many interactions between the different components of the system. The NCPF may help mitigate these dysfunctions. This framework offers managers appropriate guidance to devise a set of performance indicators that encompasses all functional areas of the nursing system, covers its structural features as well as processes and outcomes, considers both short-term and long-term issues, and provides a cross-functional view of the nursing system and of the interdependence among its different functions. While many aspects of nursing care may remain irreducibly invisible to policy-makers, the NCPF brings attention to a wide array of definable and measurable dimensions that can be used to assess nursing contribution.

However, while such a framework reflects the multidimensionality of nursing care performance, it poses the serious challenge of selecting a core set of indicators from among multiple options. The challenge is to find a fair balance between a narrow, easily implemented set of indicators that provides a limited picture of performance and a larger, more complex, but unwieldy set of indicators that gives a more complete picture. Some authors warn that having an unbalanced and overabundant set of performance measures can be counterproductive. Pushing staff to deliver an unreasonable number of targets may increase stress, lower morale and ultimately compromise quality and performance [105]. The NCPF may be a useful tool to address such challenges.

However this study only achieves the first requirement in the development of any performance measurement system, which is to develop a robust conceptual framework within which performance indicators can be identified and developed. The foundations provided by the NCPF would ensure that all major areas of nursing care performance are covered by any measurement system, that priorities for developing new measures or instruments can be identified, and that data collection efforts on performance are not misdirected or duplicated.

While the NCPF provides a matrix of indicators that covers the key functions of the nursing system, this framework is based on the principle that performance measurement, by its dynamic nature, must address both short-term and long-term issues and be constantly adjusted to the challenges of particular domains at particular times [106-108]. While the framework specifies what core aspects should be measured to assess nursing care performance, it allows for variations in the combinations of indicators used to measure those aspects in different contexts of care. Many jurisdictions are faced with the challenge of developing a realistic set of performance indicators with sufficient breadth and depth to capture the spectrum of nursing care. The NCPF provides a common ground from which to define performance, decide what is meaningful and important with regard to each nursing subsystem or function, devise a common and balanced set of performance indicators for a given sector of nursing care, and derive benchmarks for this sector. In short, the challenge for policy-makers is to select from among the universe of variables incorporated into this framework to develop, for the different sectors of nursing, an optimal and common portfolio of performance indicators that do not simply focus on measuring what is available or easy to measure, but are consistent with their objectives, account for their context of care and are relevant to the needs of the key stakeholders. Such selection must be informed by explicit criteria for ensuring that performance indicators exhibit the key characteristics of acceptability, feasibility, reliability, sensitivity to change and validity [109-111].

Developing a performance measurement system for nursing care is also about collecting, computing and presenting performance data for the purposes of following up, monitoring and improving organizational and clinical performance. Many of the current efforts to use performance data tend to concentrate on collecting and organizing existing administrative information and disseminating it for limited management applications [112]. To use this framework to best advantage, policy-makers and managers must invest in more creative ways of collecting, analyzing and reporting the data, and ultimately applying that information to improve nursing performance. The diverse nature of the dimensions of nursing care performance, the multiple possible uses of the performance data, and the variety of audiences involved may necessitate a wide variety of data collection methods, analytical techniques and dissemination strategies to assemble this information, make it accessible to different groups, and ensure that positive action results from it.

Despite the contributions mentioned above, some limitations remain in this study. Although we used a highly elaborated and systematic protocol to conduct the review, the approach was challenged by the broad nature of the topic being reviewed. It required a breadth of search and a labour-intensive process that allowed us to identify a rich set of performance-related variables and linkages between them. However, the number of themes and issues to cover may have limited the depth of the review on specific aspects of performance measurement in nursing care. More in-depth complementary reviews will be needed to document specific dimensions or subdimensions of this framework or to compile more thorough evidence on the linkages identified between those dimensions. While the review revealed some indicators that are more prevalent in existing conceptual models or are supported by strong research evidence, it is premature to conclude those elements are more significant than others that have been the subject of less attention or research. This is an unfinished agenda that opens the door for extensive research programs. Further studies will be needed to assess the implementation of the NCPF in different contexts of nursing care, gain further insight into the linkages hypothesized in the framework, and compile more evidence on the indicators that are more sensitive to nursing and that address the needs of the key stakeholders involved in care. Finally, another limitation may be that, while our search focused on the performance literature, we may have overlooked some key dimensions that do not relate directly to performance but may be essential to ensure a system’s performance.

Competing interests

The authors declare they have no competing interests.

Authors’ contributions

The study was conceived and designed by Carl-Ardy Dubois and Danielle D’Amour. All authors made a substantive contribution to the four phases of this systematic review, as described in the methods section. Carl-Ardy Dubois prepared the first draft of this manuscript. All authors contributed to revising the manuscript. All authors read and approved the final manuscript.

Acknowledgements

This study was funded by the Canadian Institutes of Health Research.

References

  1. Jelinek R, Pierce F: A nursing systems approach: productivity and quality of care.

    Public Prod Rev 1982, 6(3):223-240. Publisher Full Text OpenURL

  2. Pappas SH: The cost of nurse-sensitive adverse events.

    J Nurs Adm 2008, 38(5):230-236. PubMed Abstract | Publisher Full Text OpenURL

  3. McCloskey JC: Help to make nursing visible.

    J Nurs Scholarsh 1995, 3:170-175. OpenURL

  4. Lang NM, Mitchell PH, Hinshaw AS, Jennings BM, Lamb GS, Mark BA, Moritz P: Foreword: measuring and improving healthcare quality.

    Med Care 2004, 42(2):1-3. PubMed Abstract | Publisher Full Text OpenURL

  5. Maas ML, Delaney C: Nursing process outcome linkage research: issues, current status, and health policy implications.

    Med Care 2004, 42(2):II40-II48. PubMed Abstract | Publisher Full Text OpenURL

  6. Kennedy MA, Hannah K: Representing nursing practice: evaluating the effectiveness of a nursing classification system.

    Can J Nurs Res 2007, 39(1):58-79. PubMed Abstract OpenURL

  7. Zielstorff RD: Capturing and using clinical outcome data: implications for information systems design.

    J Am Med Inform Assoc 1995, 2(3):191-196. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  8. Hajewski C, Maupin JM, Rapp DA, Sitterding M, Pappas J: Implementation and evaluation of Nursing Interventions Classification and Nursing Outcomes Classification in a patient education plan.

    J Nurs Care Qual 1998, 12(5):30-40. PubMed Abstract | Publisher Full Text OpenURL

  9. Aiken LH, Clarke SP, Sloane DM: Hospital restructuring: does it adversely affect care and outcomes?

    J Nurs Adm 2000, 30(10):457-465. PubMed Abstract | Publisher Full Text OpenURL

  10. Gregg AC: Performance management data systems for nursing service organizations.

    J Nurs Adm 2002, 32(2):71-78. PubMed Abstract | Publisher Full Text OpenURL

  11. Behrenbeck JG, Timm JA, Griebenow LK, Demmer KA: Nursing-sensitive outcome reliability testing in a tertiary care setting.

    Int J Nurs Terminol Classif 2005, 16(1):14-20. PubMed Abstract | Publisher Full Text OpenURL

  12. Riehle AI, Hanold LS, Sprenger SL, Loeb JM: Specifying and standardizing performance measures for use at a national level: implications for nursing-sensitive care performance measures.

    Med Care Res Rev 2007, 64(Suppl 2):64S-81S. PubMed Abstract | Publisher Full Text OpenURL

  13. Doran D, Mildon B, Clarke S: Toward a National Report Card in Nursing: A Knowledge Synthesis. Toronto: Nursing Health Services Research Unit and University of Toronto; 2011. OpenURL

  14. Kurtzman ET, Kizer KW: Evaluating the performance and contribution of nurses to achieve an environment of safety.

    Nurs Adm Q 2005, 29(1):14-23. PubMed Abstract OpenURL

  15. Harris M, Vanderboom C, Hughes R: Nursing-sensitive safety and quality outcomes: the taming of a wicked problem.

    Appl Nurs Res 2009, 22(2):146-151. PubMed Abstract | Publisher Full Text OpenURL

  16. Adair CE, Simpson E, Casebeer AL, Birdsell JM, Hayden KA, Lewis S: Performance measurement in healthcare: part I–concepts and trends from a state of the science review.

    Healthc Policy 2006, 1(4):85-104. PubMed Abstract | PubMed Central Full Text OpenURL

  17. Needleman J, Kurtzman ET, Kizer KW: Performance measurement of nursing care: state of the science and the current consensus.

    Med Care Res Rev 2007, 64(Suppl 2):10S-43S. PubMed Abstract | Publisher Full Text OpenURL

  18. Idvall E, Rooke L, Hamrin E: Quality indicators in clinical nursing: a review of the literature.

    J Adv Nurs 1997, 25(1):6-17. PubMed Abstract | Publisher Full Text OpenURL

  19. Arah OA, Klazinga NS, Delnoij DM, ten Asbroek AH, Custers T: Conceptual frameworks for health systems performance: a quest for effectiveness, quality, and improvement.

    Int J Qual Health Care 2003, 15(5):377-398. PubMed Abstract | Publisher Full Text OpenURL

  20. Institute of Medicine: Crossing the Quality Chasm: A New Health System for the Twenty-first Century. Washington, DC: National Academies Press; 2001. OpenURL

  21. Greenhalgh T, Robert G, MacFarlane F, Bate P, Kyriakidou O: Diffusion of innovations in service organizations: systematic review and recommendations.

    Milbank Q 2004, 82(4):581-629. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  22. Lavis J, Davies H, Oxman A, Denis JL, Golden-Biddle K, Ferlie E: Towards systematic reviews that inform health care management and policy-making.

    J Health Serv Res Policy 2005, 10(Suppl 1):35-48. PubMed Abstract | Publisher Full Text OpenURL

  23. Tranfield D, Denyer D, Smart P: Towards a methodology for developing evidence-informed management knowledge by means of systematic review.

    Brit J Manage 2003, 14(3):207-222. Publisher Full Text OpenURL

  24. Mays N, Pope C, Popay J: Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field.

    J Health Serv Res Policy 2005, 10(Supp 1):6-20. PubMed Abstract | Publisher Full Text OpenURL

  25. Dixon-Woods M, Cavers D, Agarwal S, Annandale E, Arthur A, Harvey J, Hsu R, Katbamna S, Olsen R, Smith L, Riley R, Sutton AJ: Conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups.

    BMC Med Res Methodol 2006, 6:35. PubMed Abstract | BioMed Central Full Text | PubMed Central Full Text OpenURL

  26. Dubois C-A, Singh D: From staff-mix to skill-mix and beyond: towards a systemic approach to health workforce management.

    Hum Resour Health 2009, 7:87. PubMed Abstract | BioMed Central Full Text | PubMed Central Full Text OpenURL

  27. Bowker GC, Star SL, Spasser MA: Classifying nursing work.

    Online J Issues Nurs 2001, 6:2. PubMed Abstract OpenURL

  28. Canadian Nurses Association: International Classification for Nursing Practice: documenting nursing care and client outcomes.

    Nursing Now 2003, 14:1-4. OpenURL

  29. Hughes RG: Patient Safety and Quality: An Evidence-based Handbook for Nurses. Agency for Healthcare Research and Quality: Rockville, MD; 2008. OpenURL

  30. Savitz LA, Jones CB, Bernard S: Quality indicators sensitive to nurse staffing in acute care settings. In Advances in Patient Safety: From Research to Implementation. Edited by Henriksen K, Battles JB, Marks ES, Lewin D. Rockville, MD: Agency for Healthcare Research and Quality; 2005:375-385.

    Volume 4: Programs, Tools, and Products

    OpenURL

  31. Koch T: A review of nursing quality assurance.

    J Adv Nurs 1992, 17(7):785-794. PubMed Abstract | Publisher Full Text OpenURL

  32. Henriksen K, Dayton E, Keyes MA, Carayon P, Hughes R: Understanding adverse events: a human factors framework. In Patient Safety and Quality: An Evidence-based Handbook for Nurses. Edited by Hughes RG. Rockville, MD: Agency for Healthcare Research and Quality; 2008:I-67-I-85. OpenURL

  33. Atchley SJ: A time-ordered, systems approach to quality assurance in long-term care.

    J Appl Gerontol 1991, 10(1):19-34. PubMed Abstract | Publisher Full Text OpenURL

  34. Donabedian A: Models for organizing the delivery of health services and criteria for evaluating them.

    Milbank Q 1972, 50(4):103-154. OpenURL

  35. Parsons T: Structure and Process in Modern Societies. New York: Free Press; 1960. OpenURL

  36. Sicotte C, Champagne F, Contandriopoulos A-P, Barnsley J, Béland F, Leggat SG, Denis J-L, Bilodeau H, Langley A, Brémond M, Baker GR: A conceptual framework for the analysis of health care organizations’ performance.

    Health Serv Manage Res 1998, 1(1):24-41.

    discussion 41–48

    OpenURL

  37. Reid PP, Compton WD, Grossman JH, Fanjiang G: Building a Better Delivery System: A New Engineering/Health Care Partnership. Washington, DC: National Academies Press; 2005. OpenURL

  38. Doran DI, Sidani S, Keatings M, Doidge D: An empirical test of the NursingRole Effectiveness Model.

    J Adv Nurs 2002, 38(1):29-39. PubMed Abstract | Publisher Full Text OpenURL

  39. Irvine D, Sidani S, Hall LM: Finding value in nursing care: a framework for quality improvement and clinical evaluation.

    Nurs Econ 1998, 16(3):110-116,131. PubMed Abstract OpenURL

  40. Wong ST, Stewart AL, Gilliss CL: Evaluating advanced practice nursing care through use of a heuristic framework.

    J Nurs Care Qual 2000, 14(2):21-32. PubMed Abstract | Publisher Full Text OpenURL

  41. Gallagher RM, Rowell PA: Claiming the future of nursing through nursing-sensitive quality indicators.

    Nurs Adm Q 2003, 27(4):273-284. PubMed Abstract OpenURL

  42. Sidani S, Irvine D: A conceptual framework for evaluating the nurse practitioner role in acute care settings.

    J Adv Nurs 1999, 30(1):58-66. PubMed Abstract | Publisher Full Text OpenURL

  43. Pierce SF: Nurse-sensitive health care outcomes in acute care settings: an integrative analysis of the literature.

    J Nurs Care Qual 1997, 11(4):60-72. PubMed Abstract | Publisher Full Text OpenURL

  44. Chang BL, Lee JL, Pearson ML, Kahn KL, Elliott MN, Rubenstein LL: Evaluating quality of nursing care: the gap between theory and practice.

    J Nurs Adm 2002, 32(7–8):405-418. PubMed Abstract | Publisher Full Text OpenURL

  45. Unruh L, Wan TT: A systems framework for evaluating nursing care quality in nursing homes.

    J Med Syst 2004, 28(2):197-214. PubMed Abstract OpenURL

  46. Gerolamo AM: The conceptualization of physical restraint as a nursing-sensitive adverse outcome in acute care psychiatric treatment settings.

    Arch Psychiatr Nurs 2006, 20(4):175-185. PubMed Abstract | Publisher Full Text OpenURL

  47. Attree M: An analysis of the concept “quality” as it relates to contemporary nursing care.

    Int J Nurs Stud 1993, 30(4):355-369. PubMed Abstract | Publisher Full Text OpenURL

  48. Stetler CB, Morsi D, Burns M: Physical and emotional patient safety. A different look at nursing-sensitive outcomes.

    Outcomes Manag Nurs Pract 2000, 4(4):159-165.

    quiz 165–166

    PubMed Abstract OpenURL

  49. Schubert M, Glass TR, Clarke SP, Aiken LH, Schaffert-Witvliet B, Sloane DM, De Geest S: Rationing of nursing care and its relationship to patient outcomes: the Swiss extension of the International Hospital Outcomes Study.

    Int J Qual Health Care 2008, 20(4):227-237. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  50. Rice SM, Van Slobbe A, Rathgeber D: Nursing practice evaluation using an expert panel process.

    Clin Govern Int J 2007, 12(2):93-101. Publisher Full Text OpenURL

  51. McGillis Hall L: Report cards: relevance for nursing and patient care safety.

    Int Nurs Rev 2002, 49(3):168-177. PubMed Abstract | Publisher Full Text OpenURL

  52. McGillis Hall L, Doran D, Laschinger HS, Mallette C, Pedersen C, O’Brien-Pallas LL: A balanced scorecard approach for nursing report card development.

    Outcomes Manag 2003, 7(1):17-22. PubMed Abstract OpenURL

  53. Maas ML, Reed D, Reeder KM, Kerr P, Specht J, Johnson M, Moorhead S: Nursing outcomes classification: a preliminary report of field testing.

    Outcomes Manag 2002, 6(3):112-119. PubMed Abstract OpenURL

  54. Sidani S, Doran DM, Mitchell PH: A theory-driven approach to evaluating quality of nursing care.

    J Nurs Scholarsh 2004, 36(1):60-65. PubMed Abstract | Publisher Full Text OpenURL

  55. Doran DM, Harrison MB, Laschinger HS, Hirdes JP, Rukholm E, Sidani S, Hall LM, Tourangeau AE: Nursing-sensitive outcomes data collection in acute care and long-term-care settings.

    Nurs Res. 2006, 55(2 Suppl):S75-81. PubMed Abstract | Publisher Full Text OpenURL

  56. Given BA, Sherwood PR: Nursing sensitive patient outcomes–a white paper.

    Oncol Nurs Forum 2005, 32(4):773-84. PubMed Abstract | Publisher Full Text OpenURL

  57. Maas ML, Johnson M, Moorhead S: Classifying nursing-sensitive patient outcomes.

    Image J Nurs Sch 1996, 28(4):295-301. PubMed Abstract | Publisher Full Text OpenURL

  58. Carayon P, Gürses AP: A human factors engineering conceptual framework of nursing workload and patient safety in intensive care units.

    Intensive Crit Care Nurs 2005, 21(5):284-301. PubMed Abstract | Publisher Full Text OpenURL

  59. Lancaster DR, King A: The spider diagram nursing quality report card: bringing all the pieces together.

    J Nurs Adm 29(7–8):43-48. OpenURL

  60. Jennings BM, Loan LA, DePaul D, Brosch LR, Hildreth P: Lessons learned while collecting ANA indicator data.

    J Nurs Adm 2001, 31(3):121-9. PubMed Abstract | Publisher Full Text OpenURL

  61. Cho SH: Nurse staffing and adverse patient outcomes: a systems approach.

    Nurs Outlook 2001, 49(2):78-85. PubMed Abstract | Publisher Full Text OpenURL

  62. Jennings BM, Staggers N, Brosch LR: A classification scheme for outcome indicators.

    Image J Nurs Sch 1999, 31(4):381-388. PubMed Abstract | Publisher Full Text OpenURL

  63. Pearson ML, Lee JL, Chang BL, Elliott M, Kahn KL, Rubenstein LV: Structured implicit review: a new method for monitoring nursing care quality.

    Med Care 2000, 38(11):1074-1091. PubMed Abstract | Publisher Full Text OpenURL

  64. NQF (National Quality Forum): National voluntary consensus standards for nursing-sensitive care: an initial performance measure set. Washington, DC: ; 2004. OpenURL

  65. Aydin CE, Bolton LB, Donaldson N, Brown DS, Mukerji A: Beyond nursing quality measurement: the nation’s first regional nursing virtual dashboard. In Advances in Patient Safety: New Directions and Alternative Approaches. Volume 1: Assessment. Edited by Henriksen K, Battles JB, Keyes MA, Grady ML. Rockville, MD: Agency for Healthcare Research and Quality; 2008:217-234. OpenURL

  66. ANA (American Nurses Association):

    Nursing Sensitive Indicators. 2011.

    http://www.nursingworld.org/MainMenuCategories/ThePracticeofProfessionalNursing/PatientSafetyQuality/Research-Measurement/The-National-Database/Nursing-Sensitive-Indicators_1 webcite. Last accessed: March 14, 2013

    OpenURL

  67. Cowing M, Davino-Ramaya C, Ramaya K, Szmerekovsky J: Health care delivery performance: service, outcomes, and resource stewardship.

    Perm J 2009, 13:72-78. PubMed Abstract | PubMed Central Full Text OpenURL

  68. Kirkcaldy BD, Trimpop R, Cooper CL: Working hours, job stress, work satisfaction and accident rates among medical practitioners and allied personnel.

    Int J Stress Manag 1997, 4(2):79-87. OpenURL

  69. Healy C, McKay M: Identifying sources of stress and job satisfaction in the nursing environment.

    Aust J Adv Nurs 1999, 17(2):30-35. PubMed Abstract OpenURL

  70. Hoffman AJ, Scott LD: Role stress and career satisfaction among registered nurses by work shift patterns.

    J Nurs Adm 2003, 33(6):337-342. PubMed Abstract | Publisher Full Text OpenURL

  71. Hayes LJ, O’Brien-Pallas L, Duffield C, Shamian J, Buchan J, Hughes F, Spence Laschinger HK, North N, Stone PW: Nurse turnover: a literature review.

    Int J Nurs Stud 2006, 43:237-263. PubMed Abstract | Publisher Full Text OpenURL

  72. Aiken LH, Clarke SP, Sloane DM: Hospital staffing, organization, and quality of care: cross-national findings.

    Int J Qual Health Care 2002, 14(1):5-13. PubMed Abstract | Publisher Full Text OpenURL

  73. Thomas-Hawkins C, Flynn L, Clarke SP: Relationships between registered nurse staffing, processes of nursing care, and nurse-reported patient outcomes in chronic hemodialysis units.

    Nephrol Nurs J 2008, 35(2):123-130.

    145; quiz 131

    PubMed Abstract | PubMed Central Full Text OpenURL

  74. Baggs JG, Schmitt MH, Mushlin AI, Mitchell PH, Eldredge DH, Oakes D, Hutson AD: Association between nurse-physician collaboration and patient outcomes in three intensive care units.

    Crit Care Med 1999, 27:1991-1998. PubMed Abstract | Publisher Full Text OpenURL

  75. Davis S, Kristjanson LJ, Blight J: Communicating with families of patients in an acute hospital with advanced cancer: problems and strategies identified by nurses.

    Cancer Nurs 2003, 26:337-345. PubMed Abstract | Publisher Full Text OpenURL

  76. Currie V, Harvey G, West E, McKenna H, Keeney S: Relationship between quality of care, staffing levels, skill mix and nurse autonomy: literature review.

    J Adv Nurs 2005, 51(1):73-82. PubMed Abstract | Publisher Full Text OpenURL

  77. Needleman J, Buerhaus P, Mattke S, Stewart M, Zelevinsky K: Nurse-staffing levels and the quality of care in hospitals.

    N Engl J Med 2002, 346(22):1715-1722. PubMed Abstract | Publisher Full Text OpenURL

  78. Rogers AE, Hwang WT, Scott LD, Aiken LH, Dinges DF: The working hours of hospital staff nurses and patient safety.

    Health Aff 2004, 23(4):202-212. Publisher Full Text OpenURL

  79. Hickam DH, Severance S, Feldstein A, Ray L, Gorman P, Schuldheis S, Hersh WR, Krages KP, Helfand M: The Effect of Health Care Working Conditions on Patient Safety. Evidence Reports/Technology Assessments, Number 74. Rockville, MD: Agency for Healthcare Research and Quality; 2003. OpenURL

  80. Institute of Medicine: Keeping Patients Safe: Transforming the Work Environments of Nurses. Washington, DC: National Academies Press; 2004. OpenURL

  81. Prot S, Fontan JE, Alberti C, Bourdon O, Farnoux C, Macher MA, Foureau A, Faye A, Beaufils F, Gottot S, Brion F: Drug administration errors and their determinants in pediatric in-patients.

    Int J Qual Health Care 2005, 17:381-389. PubMed Abstract | Publisher Full Text OpenURL

  82. Aiken LH, Patrician PA: Measuring organizational traits of hospitals: the Revised Nursing Work Index.

    Nurs Res 2000, 49:146-153. PubMed Abstract | Publisher Full Text OpenURL

  83. Lake ET: Development of the practice environment scale of the Nursing Work Index.

    Res Nurs Health 2002, 25(3):176-188. PubMed Abstract | Publisher Full Text OpenURL

  84. Gerteis M, Edgman-Levitan S, Daley J, Delbanco T: Through the Patient’s Eyes: Understanding and Promoting Patient-centered Care. San Fransisco: Jossey-Bass; 1993. PubMed Abstract OpenURL

  85. Cronin C: Patient-centered Care: An Overview of Definitions and Concepts. Washington, DC: National Health Council; 2004. OpenURL

  86. Johnson CL, Cooper PK: Patient-focused care: what is it?

    Holist Nurs Pract 1997, 11(3):1-7. PubMed Abstract OpenURL

  87. Sizmur S, Redding D: Key Domains of the Experience of Hospital Outpatients. Oxford, UK: Picker Institute Europe; 2010. OpenURL

  88. McCormack B, McCance TV: Development of a framework for person-centred nursing.

    J Adv Nurs 2006, 56(5):472-479. PubMed Abstract | Publisher Full Text OpenURL

  89. McCance TV: Caring in nursing practice: the development of a conceptual framework.

    Res Theory Nurs Pract 2003, 17(2):101-116. PubMed Abstract | Publisher Full Text OpenURL

  90. Herzberg F: One more time: how do you motivate employees?

    Harv Bus Rev 1987, 65(5):109-120. OpenURL

  91. Hinshaw AS, Smeltzer CH, Atwood JR: Innovative retention strategies for nursing staff.

    J Nurs Adm 1987, 17(6):8-14. PubMed Abstract OpenURL

  92. Lohr KN: Health outcomes methodology symposium. Summary and recommendations.

    Med Care 2000, 38(II-9):194-208. OpenURL

  93. Brooker C, Repper JM, Booth A: The effectiveness of community mental health nursing: a review.

    J Clin Effect 1996, 1(2):44-49. OpenURL

  94. Singh D: Transforming Chronic Care: Evidence about Improving Care for People with Long-term Conditions. Birmingham, UK: University of Birmingham Health Services Management Centre; 2005. OpenURL

  95. Poole K: The evolving role of the clinical nurse specialist within the comprehensive breast cancer care centre.

    J Clin Nurs 1996, 5:341-349. PubMed Abstract | Publisher Full Text OpenURL

  96. Aiken LH, Sochalski J, Lake ET: Studying outcomes of organizational change in health services.

    Med Care 1997, 35(11):NS6-18. PubMed Abstract | Publisher Full Text OpenURL

  97. Kutney-Lee A, McHugh MD, Sloane DM, Cimiotti JP, Flynn L, Neff DF, Aiken LH: Nursing: a key to patient satisfaction.

    Health Aff 2009, 28(4):669-677. Publisher Full Text OpenURL

  98. National Quality Forum: National Voluntary Consensus Standards for Nursing-Sensitive Care: An Initial Performance Measure Set. Washington, DC: National Quality Forum; 2004. OpenURL

  99. Mitchell PH, Lang NM: Nurse staffing: a structural proxy for hospital quality?

    Med Care 2004, 42(1):1-3. PubMed Abstract | Publisher Full Text OpenURL

  100. Rogers AE, Dean GE, Hwang WT, Scott LD: Role of registered nurses in error prevention, discovery and correction.

    Qual Saf Health Care 2008, 17(2):117-121. PubMed Abstract | Publisher Full Text OpenURL

  101. Eriksen LR: Patient satisfaction with nursing care.

    J Nurs Meas 1995, 3(1):59-76. PubMed Abstract OpenURL

  102. Bouckaert G, Balk W: Public productivity measurement: diseases and cures.

    Public Prod Manage Rev 1991, 15(2):229-235. OpenURL

  103. Smith P: On the unintended consequences of publishing performance data in the public sector.

    Int J Public Admin 1995, 18:277-310. Publisher Full Text OpenURL

  104. Mannion R, Braithwaite J: Unintended consequences of performance measurement in healthcare: 20 salutary lessons from the English National Health Service.

    Intern Med J 2012, 42(5):569-574. PubMed Abstract | Publisher Full Text OpenURL

  105. Goddard M, Smith PC: Performance measurement in the new NHS.

    Health Policy Matters 2001, 3:1-8. OpenURL

  106. Brook RH, McGlynn EA, Shekelle PG: Defining and measuring quality of care: a perspective from US researchers.

    Int J Qual Health Care 2000, 12(4):281-295. PubMed Abstract | Publisher Full Text OpenURL

  107. Mant J: Process versus outcome indicators in the assessment of quality of health care.

    Int J Qual Health Care 2001, 13(6):475-480. PubMed Abstract | Publisher Full Text OpenURL

  108. Mainz J: Defining and classifying clinical indicators for quality improvement.

    Int J Qual Health Care 2003, 15(6):523-530. PubMed Abstract | Publisher Full Text OpenURL

  109. Hurtado MP, Swift EK, Corrigan JM: Envisioning the National Health Care Quality Report. Washington, DC: National Academies Press; 2001. OpenURL

  110. Kelley E, Hurst J: Health Care Quality Indicators Project: Conceptual Framework Paper. OECD Health Working Papers, no.23 . Paris: OECD; 2006. OpenURL

  111. Mueller C, Karon SL: ANA nurse sensitive quality indicators for long-term care facilities.

    J Nurs Care Qual 2004, 19(1):39-47. PubMed Abstract | Publisher Full Text OpenURL

  112. Smith P, Mossialos E, Papanicolas I, Leatherman SL: Performance Measurement for Health System Improvement: Experiences, Challenges and Prospects. 2010 edition. Cambridge: Cambridge University Press; 2010. OpenURL

Pre-publication history

The pre-publication history for this paper can be accessed here:

http://www.biomedcentral.com/1472-6955/12/7/prepub