Teamwork is important in providing high‐quality hospital care. Despite tremendous efforts in the 10 years since publication of the Institute of Medicine's To Err is Human report,1 hospitalized patients remain at risk for adverse events (AEs).2 Although many AEs are not preventable, a large portion of those which are identified as preventable can be attributed to communication and teamwork failures.35 A Joint Commission study indicated that communication failures were the root cause for two‐thirds of the 3548 sentinel events reported from 1995 to 2005.6 Another study, involving interviews of resident physicians about recent medical mishaps, found that communication failures contributed to 91% of the AEs they reported.5

Teamwork also plays an important role in other aspects of hospital care delivery. Patients' ratings of nurse‐physician coordination correlate with their overall perception of the quality of care received.7, 8 A study of Veterans Health Administration (VHA) hospitals found that teamwork culture was significantly and positively associated with overall patient satisfaction.9 Another VHA study found that hospitals with higher teamwork culture ratings had lower nurse resignations rates.10 Furthermore, poor teamwork within hospitals may have an adverse effect on financial performance, as a result of inefficiencies in physician and nurse workflow.11

Some organizations are capable of operating in complex, hazardous environments while maintaining exceptional performance over long periods of time. These high reliability organizations (HRO) include aircraft carriers, air traffic control systems, and nuclear power plants, and are characterized by their preoccupation with failure, reluctance to simplify interpretations, sensitivity to operations, commitment to resilience, and deference to expertise.12, 13 Preoccupation with failure is manifested by an organization's efforts to avoid complacency and persist in the search for additional risks. Reluctance to simplify interpretations is exemplified by an interest in pursuing a deep understanding of the issues that arise. Sensitivity to operations is the close attention paid to input from front‐line personnel and processes. Commitment to resilience relates to an organization's ability to contain errors once they occur and mitigate harm. Deference to expertise describes the practice of having authority migrate to the people with the most expertise, regardless of rank. Collectively, these qualities produce a state of mindfulness, allowing teams to anticipate and become aware of unexpected events, yet also quickly contain and learn from them. Recent publications have highlighted the need for hospitals to learn from HROs and the teams within them.14, 15

Recognizing the importance of teamwork in hospitals, senior leadership from the American College of Physician Executives (ACPE), the American Hospital Association (AHA), the American Organization of Nurse Executives (AONE), and the Society of Hospital Medicine (SHM) established the High Performance Teams and the Hospital of the Future project. This collaborative learning effort aims to redesign care delivery to provide optimal value to hospitalized patients. As an initial step, the High Performance Teams and the Hospital of the Future project team completed a literature review related to teamwork in hospitals. The purpose of this report is to summarize the current understanding of teamwork, describe interventions designed to improve teamwork, and make practical recommendations for hospitals to assess and improve teamwork‐related performance. We approach teamwork from the hospitalized patient's perspective, and restrict our discussion to interactions occurring among healthcare professionals within the hospital. We recognize the importance of teamwork at all points in the continuum of patient care. Highly functional inpatient teams should be integrated into an overall system of coordinated and collaborative care.

TEAMWORK: DEFINITION AND CONSTRUCTS

Physicians, nurses, and other healthcare professionals spend a great deal of their time on communication and coordination of care activities.1618 In spite of this and the patient safety concerns previously noted, interpersonal communication skills and teamwork have been historically underemphasized in professional training.1922 A team is defined as 2 or more individuals with specified roles interacting adaptively, interdependently, and dynamically toward a shared and common goal.23 Elements of effective teamwork have been identified through research conducted in aviation, the military, and more recently, healthcare. Salas and colleagues have synthesized this research into 5 core components: team leadership, mutual performance monitoring, backup behavior, adaptability, and team orientation (see Table 1).23 Additionally, 3 supporting and coordinating mechanisms are essential for effective teamwork: shared mental model, closed‐loop communication, and mutual trust (see Table 1).23 High‐performing teams use these elements to develop a culture for speaking up, and situational awareness among team members. Situational awareness refers to a person's perception and understanding of their dynamic environment, and human errors often result from a lack of such awareness.24 These teamwork constructs provide the foundational basis for understanding how hospitals can identify teamwork challenges, assess team performance, and design effective interventions.

CHALLENGES TO EFFECTIVE TEAMWORK

Several important and unique barriers to teamwork exist in hospitals. Teams are large and formed in an ad hoc fashion. On a given day, a patient's hospital team might include a hospitalist, a nurse, a case manager, a pharmacist, and 1 or more consulting physicians and therapists. Team members in each respective discipline care for multiple patients at the same time, yet few hospitals align team membership (ie, patient assignment). Therefore, a nurse caring for 4 patients may interact with 4 different hospitalists. Similarly, a hospitalist caring for 14 patients may interact with multiple nurses in a given day. Team membership is ever changing because hospital professionals work in shifts and rotations. Finally, team members are seldom in the same place at the same time because physicians often care for patients on multiple units and floors, while nurses and other team members are often unit‐based. Salas and others have noted that team size, instability, and geographic dispersion of membership serve as important barriers to improving teamwork.25, 26 As a result of these barriers, nurses and physicians do not communicate consistently, and often disagree on the daily plan of care for their patients.27, 28 When communication does occur, clinicians may overestimate how well their messages are understood by other team members, reflecting a phenomenon well known in communication psychology related to egocentric thought processes.29, 30

The traditionally steep hierarchy within medicine may also serve as a barrier to teamwork. Studies in intensive care units (ICUs), operating rooms, and general medical units reveal widely discrepant views on the quality of collaboration and communication between healthcare professionals.3133 Although physicians generally give high ratings to the quality of collaboration with nurses, nurses consistently rate the quality of collaboration with physicians as poor. Similarly, specialist physicians rate collaboration with hospitalists higher than hospitalists rate collaboration with specialists.33 Effective teams in other high‐risk industries, like aviation, strive to flatten hierarchy so that team members feel comfortable raising concerns and engaging in open and respectful communications.34

The effect of technology on communication practices and teamwork is complex and incompletely understood. The implementation of electronic heath records and computerized provider order entry systems fundamentally changes work‐flow, and may result in less synchronization and feedback during nurse‐physician collaboration.35 Similarly, the expanded use of text messages delivered via alphanumeric paging or mobile phone results in a transition toward asynchronous modes of communication. These asynchronous modes allow healthcare professionals to review and respond to messages at their convenience, and may reduce unnecessary interruptions. Research shows that these systems are popular among clinicians.3638 However, receipt and understanding of the intended message may not be confirmed with the use of asynchronous modes of communication. Moreover, important face‐to‐face communication elements (tone of voice, expression, gesture, eye contract)39, 40 are lacking. One promising approach is a system which sends low‐priority messages to a Web‐based task list for periodic review, while allowing higher priority messages to pass through to an alphanumeric pager and interrupt the intended recipient.41 Another common frustration in hospitals, despite advancing technology, is difficulty identifying the correct physician(s) and nurse(s) caring for a particular patient at a given point in time.33 Wong and colleagues found that 14% of pages in their hospital were initially sent to the wrong physician.42

ASSESSMENT OF TEAMWORK

One of the challenges in improving teamwork is the difficulty in measuring it. Teamwork assessment entails measuring the performance of teams composed of multiple individuals. Methods of teamwork assessment can be broadly categorized as self assessment, peer assessment, direct observation, survey of team climate or culture, and measurement of the outcome of effective teamwork. While self‐report tools are easy to administer and can capture affective components influencing team performance, they may not reflect actual skills on the part of individuals or teams. Peer assessment includes the use of 360‐degree evaluations or multisource feedback, and provides an evaluation of individual performance.4347

Direct observation provides a more accurate assessment of team‐related behaviors using trained observers. Observers use checklists and/or behaviorally anchored rating scales (BARS) to evaluate individual and team performance. A number of BARS have been developed and validated for the evaluation of team performance.4852 Of note, direct observation may be difficult in settings in which team members are not in the same place at the same time. An alternative method, which may be better suited for general medical units, is the use of survey instruments designed to assess attitudes and teamwork climate.5355 Importantly, higher survey ratings of collaboration and teamwork have been associated with better patient outcomes in observational studies.5658

The ultimate goal of teamwork efforts is to improve patient outcomes. Because patient outcomes are affected by a number of factors and because hospitals frequently engage in multiple, simultaneous efforts to improve care, it is often difficult to clearly link improved outcomes with teamwork interventions. Continued efforts to rigorously evaluate teamwork interventions should remain a priority, particularly as the cost of these interventions must be weighed against other interventions and investments.

EXAMPLES OF SUCCESSFUL INTERVENTIONS

A number of interventions have been used to improve teamwork in hospitals (see Table 2).

CONCLUSIONS

In summary, teamwork is critically important to provide safe and effective care. Important and unique barriers to teamwork exist in hospitals. We recommend the use of survey instruments, such as those mentioned earlier, as the most feasible method to assess teamwork in the general medical setting. Because each intervention addresses only a portion of the barriers to optimal teamwork, we encourage leaders to use a multifaceted approach. We recommend the implementation of a combination of interventions with adaptations to fit unique clinical settings and local culture.

Teamwork is important in providing high‐quality hospital care. Despite tremendous efforts in the 10 years since publication of the Institute of Medicine's To Err is Human report,1 hospitalized patients remain at risk for adverse events (AEs).2 Although many AEs are not preventable, a large portion of those which are identified as preventable can be attributed to communication and teamwork failures.35 A Joint Commission study indicated that communication failures were the root cause for two‐thirds of the 3548 sentinel events reported from 1995 to 2005.6 Another study, involving interviews of resident physicians about recent medical mishaps, found that communication failures contributed to 91% of the AEs they reported.5

Teamwork also plays an important role in other aspects of hospital care delivery. Patients' ratings of nurse‐physician coordination correlate with their overall perception of the quality of care received.7, 8 A study of Veterans Health Administration (VHA) hospitals found that teamwork culture was significantly and positively associated with overall patient satisfaction.9 Another VHA study found that hospitals with higher teamwork culture ratings had lower nurse resignations rates.10 Furthermore, poor teamwork within hospitals may have an adverse effect on financial performance, as a result of inefficiencies in physician and nurse workflow.11

Some organizations are capable of operating in complex, hazardous environments while maintaining exceptional performance over long periods of time. These high reliability organizations (HRO) include aircraft carriers, air traffic control systems, and nuclear power plants, and are characterized by their preoccupation with failure, reluctance to simplify interpretations, sensitivity to operations, commitment to resilience, and deference to expertise.12, 13 Preoccupation with failure is manifested by an organization's efforts to avoid complacency and persist in the search for additional risks. Reluctance to simplify interpretations is exemplified by an interest in pursuing a deep understanding of the issues that arise. Sensitivity to operations is the close attention paid to input from front‐line personnel and processes. Commitment to resilience relates to an organization's ability to contain errors once they occur and mitigate harm. Deference to expertise describes the practice of having authority migrate to the people with the most expertise, regardless of rank. Collectively, these qualities produce a state of mindfulness, allowing teams to anticipate and become aware of unexpected events, yet also quickly contain and learn from them. Recent publications have highlighted the need for hospitals to learn from HROs and the teams within them.14, 15

Recognizing the importance of teamwork in hospitals, senior leadership from the American College of Physician Executives (ACPE), the American Hospital Association (AHA), the American Organization of Nurse Executives (AONE), and the Society of Hospital Medicine (SHM) established the High Performance Teams and the Hospital of the Future project. This collaborative learning effort aims to redesign care delivery to provide optimal value to hospitalized patients. As an initial step, the High Performance Teams and the Hospital of the Future project team completed a literature review related to teamwork in hospitals. The purpose of this report is to summarize the current understanding of teamwork, describe interventions designed to improve teamwork, and make practical recommendations for hospitals to assess and improve teamwork‐related performance. We approach teamwork from the hospitalized patient's perspective, and restrict our discussion to interactions occurring among healthcare professionals within the hospital. We recognize the importance of teamwork at all points in the continuum of patient care. Highly functional inpatient teams should be integrated into an overall system of coordinated and collaborative care.

TEAMWORK: DEFINITION AND CONSTRUCTS

Physicians, nurses, and other healthcare professionals spend a great deal of their time on communication and coordination of care activities.1618 In spite of this and the patient safety concerns previously noted, interpersonal communication skills and teamwork have been historically underemphasized in professional training.1922 A team is defined as 2 or more individuals with specified roles interacting adaptively, interdependently, and dynamically toward a shared and common goal.23 Elements of effective teamwork have been identified through research conducted in aviation, the military, and more recently, healthcare. Salas and colleagues have synthesized this research into 5 core components: team leadership, mutual performance monitoring, backup behavior, adaptability, and team orientation (see Table 1).23 Additionally, 3 supporting and coordinating mechanisms are essential for effective teamwork: shared mental model, closed‐loop communication, and mutual trust (see Table 1).23 High‐performing teams use these elements to develop a culture for speaking up, and situational awareness among team members. Situational awareness refers to a person's perception and understanding of their dynamic environment, and human errors often result from a lack of such awareness.24 These teamwork constructs provide the foundational basis for understanding how hospitals can identify teamwork challenges, assess team performance, and design effective interventions.

CHALLENGES TO EFFECTIVE TEAMWORK

Several important and unique barriers to teamwork exist in hospitals. Teams are large and formed in an ad hoc fashion. On a given day, a patient's hospital team might include a hospitalist, a nurse, a case manager, a pharmacist, and 1 or more consulting physicians and therapists. Team members in each respective discipline care for multiple patients at the same time, yet few hospitals align team membership (ie, patient assignment). Therefore, a nurse caring for 4 patients may interact with 4 different hospitalists. Similarly, a hospitalist caring for 14 patients may interact with multiple nurses in a given day. Team membership is ever changing because hospital professionals work in shifts and rotations. Finally, team members are seldom in the same place at the same time because physicians often care for patients on multiple units and floors, while nurses and other team members are often unit‐based. Salas and others have noted that team size, instability, and geographic dispersion of membership serve as important barriers to improving teamwork.25, 26 As a result of these barriers, nurses and physicians do not communicate consistently, and often disagree on the daily plan of care for their patients.27, 28 When communication does occur, clinicians may overestimate how well their messages are understood by other team members, reflecting a phenomenon well known in communication psychology related to egocentric thought processes.29, 30

The traditionally steep hierarchy within medicine may also serve as a barrier to teamwork. Studies in intensive care units (ICUs), operating rooms, and general medical units reveal widely discrepant views on the quality of collaboration and communication between healthcare professionals.3133 Although physicians generally give high ratings to the quality of collaboration with nurses, nurses consistently rate the quality of collaboration with physicians as poor. Similarly, specialist physicians rate collaboration with hospitalists higher than hospitalists rate collaboration with specialists.33 Effective teams in other high‐risk industries, like aviation, strive to flatten hierarchy so that team members feel comfortable raising concerns and engaging in open and respectful communications.34

The effect of technology on communication practices and teamwork is complex and incompletely understood. The implementation of electronic heath records and computerized provider order entry systems fundamentally changes work‐flow, and may result in less synchronization and feedback during nurse‐physician collaboration.35 Similarly, the expanded use of text messages delivered via alphanumeric paging or mobile phone results in a transition toward asynchronous modes of communication. These asynchronous modes allow healthcare professionals to review and respond to messages at their convenience, and may reduce unnecessary interruptions. Research shows that these systems are popular among clinicians.3638 However, receipt and understanding of the intended message may not be confirmed with the use of asynchronous modes of communication. Moreover, important face‐to‐face communication elements (tone of voice, expression, gesture, eye contract)39, 40 are lacking. One promising approach is a system which sends low‐priority messages to a Web‐based task list for periodic review, while allowing higher priority messages to pass through to an alphanumeric pager and interrupt the intended recipient.41 Another common frustration in hospitals, despite advancing technology, is difficulty identifying the correct physician(s) and nurse(s) caring for a particular patient at a given point in time.33 Wong and colleagues found that 14% of pages in their hospital were initially sent to the wrong physician.42

ASSESSMENT OF TEAMWORK

One of the challenges in improving teamwork is the difficulty in measuring it. Teamwork assessment entails measuring the performance of teams composed of multiple individuals. Methods of teamwork assessment can be broadly categorized as self assessment, peer assessment, direct observation, survey of team climate or culture, and measurement of the outcome of effective teamwork. While self‐report tools are easy to administer and can capture affective components influencing team performance, they may not reflect actual skills on the part of individuals or teams. Peer assessment includes the use of 360‐degree evaluations or multisource feedback, and provides an evaluation of individual performance.4347

Direct observation provides a more accurate assessment of team‐related behaviors using trained observers. Observers use checklists and/or behaviorally anchored rating scales (BARS) to evaluate individual and team performance. A number of BARS have been developed and validated for the evaluation of team performance.4852 Of note, direct observation may be difficult in settings in which team members are not in the same place at the same time. An alternative method, which may be better suited for general medical units, is the use of survey instruments designed to assess attitudes and teamwork climate.5355 Importantly, higher survey ratings of collaboration and teamwork have been associated with better patient outcomes in observational studies.5658

The ultimate goal of teamwork efforts is to improve patient outcomes. Because patient outcomes are affected by a number of factors and because hospitals frequently engage in multiple, simultaneous efforts to improve care, it is often difficult to clearly link improved outcomes with teamwork interventions. Continued efforts to rigorously evaluate teamwork interventions should remain a priority, particularly as the cost of these interventions must be weighed against other interventions and investments.

EXAMPLES OF SUCCESSFUL INTERVENTIONS

A number of interventions have been used to improve teamwork in hospitals (see Table 2).

CONCLUSIONS

In summary, teamwork is critically important to provide safe and effective care. Important and unique barriers to teamwork exist in hospitals. We recommend the use of survey instruments, such as those mentioned earlier, as the most feasible method to assess teamwork in the general medical setting. Because each intervention addresses only a portion of the barriers to optimal teamwork, we encourage leaders to use a multifaceted approach. We recommend the implementation of a combination of interventions with adaptations to fit unique clinical settings and local culture.

Teamwork is important in providing high‐quality hospital care. Despite tremendous efforts in the 10 years since publication of the Institute of Medicine's To Err is Human report,1 hospitalized patients remain at risk for adverse events (AEs).2 Although many AEs are not preventable, a large portion of those which are identified as preventable can be attributed to communication and teamwork failures.35 A Joint Commission study indicated that communication failures were the root cause for two‐thirds of the 3548 sentinel events reported from 1995 to 2005.6 Another study, involving interviews of resident physicians about recent medical mishaps, found that communication failures contributed to 91% of the AEs they reported.5

Teamwork also plays an important role in other aspects of hospital care delivery. Patients' ratings of nurse‐physician coordination correlate with their overall perception of the quality of care received.7, 8 A study of Veterans Health Administration (VHA) hospitals found that teamwork culture was significantly and positively associated with overall patient satisfaction.9 Another VHA study found that hospitals with higher teamwork culture ratings had lower nurse resignations rates.10 Furthermore, poor teamwork within hospitals may have an adverse effect on financial performance, as a result of inefficiencies in physician and nurse workflow.11

Some organizations are capable of operating in complex, hazardous environments while maintaining exceptional performance over long periods of time. These high reliability organizations (HRO) include aircraft carriers, air traffic control systems, and nuclear power plants, and are characterized by their preoccupation with failure, reluctance to simplify interpretations, sensitivity to operations, commitment to resilience, and deference to expertise.12, 13 Preoccupation with failure is manifested by an organization's efforts to avoid complacency and persist in the search for additional risks. Reluctance to simplify interpretations is exemplified by an interest in pursuing a deep understanding of the issues that arise. Sensitivity to operations is the close attention paid to input from front‐line personnel and processes. Commitment to resilience relates to an organization's ability to contain errors once they occur and mitigate harm. Deference to expertise describes the practice of having authority migrate to the people with the most expertise, regardless of rank. Collectively, these qualities produce a state of mindfulness, allowing teams to anticipate and become aware of unexpected events, yet also quickly contain and learn from them. Recent publications have highlighted the need for hospitals to learn from HROs and the teams within them.14, 15

Recognizing the importance of teamwork in hospitals, senior leadership from the American College of Physician Executives (ACPE), the American Hospital Association (AHA), the American Organization of Nurse Executives (AONE), and the Society of Hospital Medicine (SHM) established the High Performance Teams and the Hospital of the Future project. This collaborative learning effort aims to redesign care delivery to provide optimal value to hospitalized patients. As an initial step, the High Performance Teams and the Hospital of the Future project team completed a literature review related to teamwork in hospitals. The purpose of this report is to summarize the current understanding of teamwork, describe interventions designed to improve teamwork, and make practical recommendations for hospitals to assess and improve teamwork‐related performance. We approach teamwork from the hospitalized patient's perspective, and restrict our discussion to interactions occurring among healthcare professionals within the hospital. We recognize the importance of teamwork at all points in the continuum of patient care. Highly functional inpatient teams should be integrated into an overall system of coordinated and collaborative care.

TEAMWORK: DEFINITION AND CONSTRUCTS

Physicians, nurses, and other healthcare professionals spend a great deal of their time on communication and coordination of care activities.1618 In spite of this and the patient safety concerns previously noted, interpersonal communication skills and teamwork have been historically underemphasized in professional training.1922 A team is defined as 2 or more individuals with specified roles interacting adaptively, interdependently, and dynamically toward a shared and common goal.23 Elements of effective teamwork have been identified through research conducted in aviation, the military, and more recently, healthcare. Salas and colleagues have synthesized this research into 5 core components: team leadership, mutual performance monitoring, backup behavior, adaptability, and team orientation (see Table 1).23 Additionally, 3 supporting and coordinating mechanisms are essential for effective teamwork: shared mental model, closed‐loop communication, and mutual trust (see Table 1).23 High‐performing teams use these elements to develop a culture for speaking up, and situational awareness among team members. Situational awareness refers to a person's perception and understanding of their dynamic environment, and human errors often result from a lack of such awareness.24 These teamwork constructs provide the foundational basis for understanding how hospitals can identify teamwork challenges, assess team performance, and design effective interventions.

CHALLENGES TO EFFECTIVE TEAMWORK

Several important and unique barriers to teamwork exist in hospitals. Teams are large and formed in an ad hoc fashion. On a given day, a patient's hospital team might include a hospitalist, a nurse, a case manager, a pharmacist, and 1 or more consulting physicians and therapists. Team members in each respective discipline care for multiple patients at the same time, yet few hospitals align team membership (ie, patient assignment). Therefore, a nurse caring for 4 patients may interact with 4 different hospitalists. Similarly, a hospitalist caring for 14 patients may interact with multiple nurses in a given day. Team membership is ever changing because hospital professionals work in shifts and rotations. Finally, team members are seldom in the same place at the same time because physicians often care for patients on multiple units and floors, while nurses and other team members are often unit‐based. Salas and others have noted that team size, instability, and geographic dispersion of membership serve as important barriers to improving teamwork.25, 26 As a result of these barriers, nurses and physicians do not communicate consistently, and often disagree on the daily plan of care for their patients.27, 28 When communication does occur, clinicians may overestimate how well their messages are understood by other team members, reflecting a phenomenon well known in communication psychology related to egocentric thought processes.29, 30

The traditionally steep hierarchy within medicine may also serve as a barrier to teamwork. Studies in intensive care units (ICUs), operating rooms, and general medical units reveal widely discrepant views on the quality of collaboration and communication between healthcare professionals.3133 Although physicians generally give high ratings to the quality of collaboration with nurses, nurses consistently rate the quality of collaboration with physicians as poor. Similarly, specialist physicians rate collaboration with hospitalists higher than hospitalists rate collaboration with specialists.33 Effective teams in other high‐risk industries, like aviation, strive to flatten hierarchy so that team members feel comfortable raising concerns and engaging in open and respectful communications.34

The effect of technology on communication practices and teamwork is complex and incompletely understood. The implementation of electronic heath records and computerized provider order entry systems fundamentally changes work‐flow, and may result in less synchronization and feedback during nurse‐physician collaboration.35 Similarly, the expanded use of text messages delivered via alphanumeric paging or mobile phone results in a transition toward asynchronous modes of communication. These asynchronous modes allow healthcare professionals to review and respond to messages at their convenience, and may reduce unnecessary interruptions. Research shows that these systems are popular among clinicians.3638 However, receipt and understanding of the intended message may not be confirmed with the use of asynchronous modes of communication. Moreover, important face‐to‐face communication elements (tone of voice, expression, gesture, eye contract)39, 40 are lacking. One promising approach is a system which sends low‐priority messages to a Web‐based task list for periodic review, while allowing higher priority messages to pass through to an alphanumeric pager and interrupt the intended recipient.41 Another common frustration in hospitals, despite advancing technology, is difficulty identifying the correct physician(s) and nurse(s) caring for a particular patient at a given point in time.33 Wong and colleagues found that 14% of pages in their hospital were initially sent to the wrong physician.42

ASSESSMENT OF TEAMWORK

One of the challenges in improving teamwork is the difficulty in measuring it. Teamwork assessment entails measuring the performance of teams composed of multiple individuals. Methods of teamwork assessment can be broadly categorized as self assessment, peer assessment, direct observation, survey of team climate or culture, and measurement of the outcome of effective teamwork. While self‐report tools are easy to administer and can capture affective components influencing team performance, they may not reflect actual skills on the part of individuals or teams. Peer assessment includes the use of 360‐degree evaluations or multisource feedback, and provides an evaluation of individual performance.4347

Direct observation provides a more accurate assessment of team‐related behaviors using trained observers. Observers use checklists and/or behaviorally anchored rating scales (BARS) to evaluate individual and team performance. A number of BARS have been developed and validated for the evaluation of team performance.4852 Of note, direct observation may be difficult in settings in which team members are not in the same place at the same time. An alternative method, which may be better suited for general medical units, is the use of survey instruments designed to assess attitudes and teamwork climate.5355 Importantly, higher survey ratings of collaboration and teamwork have been associated with better patient outcomes in observational studies.5658

The ultimate goal of teamwork efforts is to improve patient outcomes. Because patient outcomes are affected by a number of factors and because hospitals frequently engage in multiple, simultaneous efforts to improve care, it is often difficult to clearly link improved outcomes with teamwork interventions. Continued efforts to rigorously evaluate teamwork interventions should remain a priority, particularly as the cost of these interventions must be weighed against other interventions and investments.

EXAMPLES OF SUCCESSFUL INTERVENTIONS

A number of interventions have been used to improve teamwork in hospitals (see Table 2).

CONCLUSIONS

In summary, teamwork is critically important to provide safe and effective care. Important and unique barriers to teamwork exist in hospitals. We recommend the use of survey instruments, such as those mentioned earlier, as the most feasible method to assess teamwork in the general medical setting. Because each intervention addresses only a portion of the barriers to optimal teamwork, we encourage leaders to use a multifaceted approach. We recommend the implementation of a combination of interventions with adaptations to fit unique clinical settings and local culture.

Acute pulmonary embolism (PE) is associated with significant morbidity and mortality.1 While expeditious diagnosis and management results in reduced mortality, the ability to rapidly and accurately identify those at increased risk for death remains elusive. Multiple studies have utilized various biomarkers as risk stratification tools, however, these approaches have proven to have many limitations. For example, both serum brain natriuretic peptide (BNP) and troponin levels have been studied as possible risk stratification tools. Those with elevated levels of these following a PE may have concomitant right ventricular (RV) dysfunction and/or hemodynamic instability. Thus, they may face a greater risk for cardiovascular collapse and death. The low positive predictive value of these biomarkers (14%‐44%) has limited their clinical utility.24 Furthermore, imaging modalities, such as echocardiography, which is considered the clinical gold standard for determining the presence of acute RV dysfunction in PE, may not be readily available and may require special expertise for interpretation.5

Conversely, the need to identify acute PE patients at low risk for death is just as important. Recent studies suggest that carefully selected patients can successfully be managed as outpatients which can subsequently lead to significant cost savings and patient satisfaction. Movement towards enhanced outpatient resources and the advent of subcutaneous anticoagulants have made outpatient management of acute PE an appealing possibility. However, proper education, close follow‐up, and a rigorous selection process to recognize those at minimal risk for a fatal complication must all be available before clinicians prematurely discharge these patients to home.

Recently, clinical scoring tools have been developed to aid in risk stratifying patients with acute PE to accurately determine patient outcome. The pulmonary embolism severity index (PESI) is a reproducible scoring system that accurately predicts 30‐day and 90‐day mortality.6, 7 It consists of 11 clinical variables that can be quickly assessed at the time of diagnosis (Table 1A). The fact that biomarkers and imaging technology, such as echocardiography, are unnecessary to compute a PESI score demonstrates the appeal of this system. Similar to the PESI, Sanchez et al.8 have proposed the prognosis in pulmonary embolism (PREP) score as an alternate clinical risk tool in PE (Table 1B). Contrary to PESI, the PREP only uses 3 clinical variables to accurately predict vital outcome with an area under the receiver operating characteristic (AUROC) curve of 0.73 (95% confidence interval [CI], 0.65‐0.82). While both scoring systems have been developed to predict 30‐day mortality in acute PE, the comparative validity of these prognostic tools has not been assessed.

We hypothesized that the PESI more precisely risk stratifies the risk for death in acute PE compared to the PREP. Furthermore, we theorized that the PESI more reliably predicts not only 30‐day but also 90‐day mortality. To test our hypothesis, we performed a retrospective analysis, of all consecutive patients diagnosed with acute PE at our hospital, to compare the prognostic accuracy of these 2 scoring systems.

METHODS

RESULTS

The final cohort included 302 subjects (mean age: 59.7 17.2 years; 44.0% males). As Table 3 reveals, the majority of PEs was diagnosed via CT scan (76%). On presentation, 6.6% had cardiogenic shock, while 5.0% had altered censorium. In terms of comorbid conditions, 25.2% had congestive heart failure, 25.2% had cancer, and 22.2% had a prior venous thromboembolic event. Overall, 3.0% and 4.0% met our primary outcomes of death within 30‐days and 90‐days of their acute PEs, respectively.

The rates of 30‐day and 90‐day mortality, respectively, increased with increasing score for both the PESI and the PREP. No patients in PESI class I died by either time point, while 9.2% of PESI class V subjects expired by 30 days (P < 0.0001) and 10.5% died by 90 days (P = 0.003) (Table 2). Based on PESI, 30‐day death rates were 4.6% in the high‐risk cohort versus 0% in the low‐risk group (P = 0.023). Conversely, 7.1% of high‐risk PREP subjects died by day 30 versus 1% of low‐risk subjects (P = 0.004) (Figure 1A). Those stratified into the PESI high‐risk group had a 90‐day mortality of 6.2% versus 0% for the low‐risk group (P = 0.008) versus 9.1% in those deemed high risk by PREP, as compared to 1.5% of those scored as low risk by PREP (P = 0.001) (Figure 1B).

Figure 1

Regarding the 30‐day mortality, the negative predictive value of the PESI was 100% (95% CI, 98.6%‐100%) while that for PREP was 99.0% (95% CI, 97.6%‐99.7%); the ability of the PREP to predict 30‐day mortality was similar to the PESI (Table 4). The AUROCs for PESI and PREP for predicting 30‐day death were also equivalent; for the raw PESI score, this measured 0.858 (95% CI, 0.773‐0.943), compared to 0.719 (95% CI, 0.563‐0.875) for PREP. When these scores were dichotomized to high‐risk versus low‐risk groups, the AUROC for the PESI was 0.684 (95% CI, 0.559‐0.810) and 0.732 (95% CI, 0.571‐0.893) for PREP.

In terms of 90‐day mortality, the negative predictive values of PESI and PREP did not change: 100% (95% CI, 97.4%‐100%) and 98.5% (95% CI, 96.9%‐99.5%), respectively. The ability of PESI and PREP as predictors of 90‐day mortality was equivalent (Table 4). Here, the AUROC for the raw PESI score remained excellent at 0.835 (97% CI, 0.762‐0.907). The AUROC for PREP was akin to that of PESI at 0.704 (95% CI, 0.564‐0.844). Segregating scores into high‐risk versus low‐risk groups demonstrated that the AUROC for PESI was 0.686 (95% CI, 0.576‐0.796) compared to 0.720 (95% CI, 0.574‐0.865) for PREP.

DISCUSSION

This retrospective analysis of patients with acute PE confirms that both the PESI and the PREP are accurate scoring tools for identifying patients at low risk of death. Under both rubrics, as the score increases, the likelihood of death also increases. More importantly, we demonstrate that the negative predictive value for both the PREP and PESI are excellent. Thus, these scoring tools can distinguish those at higher risk for death versus those at low risk in a simple‐to‐apply manner. In comparing these 2 scoring systems, the PREP comparably identifies acute PE patients at risk for death when contrasted with the PESI. Given the fewer required scoring points to calculate PREP and its ability to accurately predict clinically relevant outcomes, this simpler scoring system may have greater clinical utility.

Prior studies have validated the PESI as a risk stratification tool to predict 30‐day and 90‐day mortalities. In their original derivation of the PESI, Aujesky et al. demonstrated that higher PESI scores correlated with death at 30 days.6 Acute PE patients classified into risk class I had a short‐term mortality rate of 1.1% compared to nearly 25% of patients risk stratified into risk class V. The same authors subsequently verified that there is a linear relationship between PESI score and risk of death at 90 days.10 We have also confirmed the accuracy of the PESI for identifying persons at high risk for death and documented the limited interobserver variability in this tool.7 In combination, there is evidence that the PESI can accurately predict vital outcome. Despite the effectiveness of the PESI, it is a somewhat cumbersome scoring system. It requires gathering information on 11 clinical variables, each with a different score allocation to ultimately compute the PESI score. In contrast, the PREP only requires knowing 3 clinical variables: presence of cancer, mental status, and the presence of cardiogenic shock. Akin to the PESI, the PREP and mortality are linearly related, where higher PREP scores result in higher 30‐day and 90‐day mortalities.

Our analysis helps expand the evidence regarding clinical risk stratification in PE in several ways. First, we verify that both the PESI and PREP are accurate predictors of short‐term mortality. While this has been accomplished for the PESI in prior studies, to our knowledge, this is the first confirmatory study for PREP's utility as a risk stratification tool. Second, we demonstrate that PREP is also an accurate predictor of intermediate‐term mortality. If the eventual goal is to develop tools that allow for the initial outpatient management of acute PE, clinicians require data on longer‐term outcomes to ensure that later harms do not arise based on a decision to defer hospitalization. Prior observational studies and randomized controlled clinical trials have proven that appropriately selected individuals face similar rates of complications following acute PE, whether they are managed in or outside of a hospital setting.1116 The key limitation of these earlier efforts, though, was that there was no clear standardized approach to determining whom could be safely managed solely as an outpatient. Finally, our study is unique in that we compare the discriminatory power of these 2 risk‐scoring schemes and illustrate their equivalence. As a scoring system that only requires 3 variables, the PREP is easier and simpler, and may therefore have more clinical utility than the PESI. The high negative predictive value of the PREP suggests that it has potential in identifying patients with acute PE who can safely be managed on an outpatient basis. However, given the complexity of factors associated with the decision for early discharge, these scores should be used in conjunction with, and not supplant, clinical judgment for outpatient management. Of course, formal prospective management trials incorporating both the PREP and PESI are needed to validate this concept.

Why does PREP perform so well despite the fact that it focuses on so few clinical variables? Essentially, the PREP is an effective scoring tool for acute PE because of its ability to identify individuals at risk for progressing to shock. The presence of AMS in acute PE has been associated with a greater likelihood of death, as it likely arises as a consequence of severe shock or RV strain resulting in decreased cerebral blood flow. Alternatively, altered censorium could represent a manifestation of hypoxemia from significant V/Q mismatching and/or pulmonary shunting due to the obstructive clot. This, too, portends a poorer prognosis secondary to impending respiratory failure from hypoxemia. Thus, individuals with an acute PE presenting with altered mentation merit very close observation. Similarly, pending hemodynamic instability is a concerning manifestation that warrants inpatient monitoring.5, 17, 18 At the very minimum, these individuals have RV strain and should therefore be admitted to the hospital to potentially administer more aggressive treatment modalities (ie, thrombolytics or thrombectomy). The last clinical criteria involves the presence of malignancy. The presence of a cancer may serve as a surrogate marker for those at increased risk for early recurrent thromboembolic phenomena, since malignancy is associated with a hypercoagulable state.17, 19 Perhaps there is a threshold whereby accumulating clot resulting in RV strain ensues with subsequent poorer outcomes. Thus, it clinically and physiologically seems logical that, in the absence of any of these findings, patients with acute PE will have lower mortality rates.

Thus far, other methods used for risk stratification may either be expensive, not really obtainable, or not routinely available at the time of presentation. For example, confirmation of RV strain with an echocardiogram requires a skilled technician and interpreter. In contrast, both the PESI and PREP are scored based on multiple clinical findings. Hence, they are not dependent upon a single test to determine outcome, but on various clinical variables making these scoring tools comprehensive, simple, and reliable approaches of recognizing low‐risk patients.

Our analysis has several limitations. First, the retrospective nature of this analysis subjects it to multiple forms of bias. We attempted to eliminate these biases by defining, a priori, the time frame from which vital signs can be used during scoring. We also used all‐cause mortality as our primary endpoint to minimize the possibility of ascertainment bias. However, this type of bias could not be completely eliminated since data collected was not specifically for the purpose of this study. Second, this single‐center study may limit the generalizability of these findings; yet, the diversity of patients admitted to this 900‐bed, tertiary care facility, as well as the inclusion of both inpatients and outpatients, helps to mitigate this concern. Third, the exclusion of individuals with expectant deaths within <30 days limits the applicability of these findings to this group. We chose to exclude persons with anticipated short‐term mortality to reduce the tally of patients who did not receive therapeutic treatment (ie, those transitioned to comfort care). Fourth, the use of the Social Security Death Index objectively determines death status for all‐cause mortality but cannot delineate cause‐specific death. Consequently, death strictly due to PE could not be assessed. Fifth, the original investigators for PREP assessed the PREP score with and without BNP and left‐to‐right ventricular diameter ratios. Although their results demonstrated similar AUROCs for the PREP score with and without BNP to predict 30‐day outcomes, this was a finding we could not confirm due to inconsistencies in measuring BNP and echocardiograms in our cohort. Also, our post hoc power analysis demonstrates that our findings may be limited by sample size. The lack of statistically significant differences between the PESI and the PREP may, in fact, be due to the small sample size versus true effect. Finally, tolerance for medical therapy and compliance with treatment were not documented and, therefore, were immeasurable. Poor compliance to anticoagulants or intolerability increases risk for recurrent PE, while excessive anticoagulation increases likelihood of bleeding.

In summary, the PREP and PESI can both safely predict 30‐day and 90‐day outcomes. However, the simplicity of the PREP renders it more clinician friendly. The fact that only 3 clinical noninvasive variables are required would ultimately make it the preferred bedside tool to risk stratify patients for acute PE. The high negative predictive value and comparable AUROCs establishes the effectiveness of these 2 scoring systems in recognizing low‐risk patients. Irrespective of the clinician's choice to use 1 tool over the other, both have potential for clinical application at the bedside and in clinical trials. Nevertheless, further evidence is required before they are utilized to triage patients for outpatient therapy.

Acute pulmonary embolism (PE) is associated with significant morbidity and mortality.1 While expeditious diagnosis and management results in reduced mortality, the ability to rapidly and accurately identify those at increased risk for death remains elusive. Multiple studies have utilized various biomarkers as risk stratification tools, however, these approaches have proven to have many limitations. For example, both serum brain natriuretic peptide (BNP) and troponin levels have been studied as possible risk stratification tools. Those with elevated levels of these following a PE may have concomitant right ventricular (RV) dysfunction and/or hemodynamic instability. Thus, they may face a greater risk for cardiovascular collapse and death. The low positive predictive value of these biomarkers (14%‐44%) has limited their clinical utility.24 Furthermore, imaging modalities, such as echocardiography, which is considered the clinical gold standard for determining the presence of acute RV dysfunction in PE, may not be readily available and may require special expertise for interpretation.5

Conversely, the need to identify acute PE patients at low risk for death is just as important. Recent studies suggest that carefully selected patients can successfully be managed as outpatients which can subsequently lead to significant cost savings and patient satisfaction. Movement towards enhanced outpatient resources and the advent of subcutaneous anticoagulants have made outpatient management of acute PE an appealing possibility. However, proper education, close follow‐up, and a rigorous selection process to recognize those at minimal risk for a fatal complication must all be available before clinicians prematurely discharge these patients to home.

Recently, clinical scoring tools have been developed to aid in risk stratifying patients with acute PE to accurately determine patient outcome. The pulmonary embolism severity index (PESI) is a reproducible scoring system that accurately predicts 30‐day and 90‐day mortality.6, 7 It consists of 11 clinical variables that can be quickly assessed at the time of diagnosis (Table 1A). The fact that biomarkers and imaging technology, such as echocardiography, are unnecessary to compute a PESI score demonstrates the appeal of this system. Similar to the PESI, Sanchez et al.8 have proposed the prognosis in pulmonary embolism (PREP) score as an alternate clinical risk tool in PE (Table 1B). Contrary to PESI, the PREP only uses 3 clinical variables to accurately predict vital outcome with an area under the receiver operating characteristic (AUROC) curve of 0.73 (95% confidence interval [CI], 0.65‐0.82). While both scoring systems have been developed to predict 30‐day mortality in acute PE, the comparative validity of these prognostic tools has not been assessed.

We hypothesized that the PESI more precisely risk stratifies the risk for death in acute PE compared to the PREP. Furthermore, we theorized that the PESI more reliably predicts not only 30‐day but also 90‐day mortality. To test our hypothesis, we performed a retrospective analysis, of all consecutive patients diagnosed with acute PE at our hospital, to compare the prognostic accuracy of these 2 scoring systems.

METHODS

RESULTS

The final cohort included 302 subjects (mean age: 59.7 17.2 years; 44.0% males). As Table 3 reveals, the majority of PEs was diagnosed via CT scan (76%). On presentation, 6.6% had cardiogenic shock, while 5.0% had altered censorium. In terms of comorbid conditions, 25.2% had congestive heart failure, 25.2% had cancer, and 22.2% had a prior venous thromboembolic event. Overall, 3.0% and 4.0% met our primary outcomes of death within 30‐days and 90‐days of their acute PEs, respectively.

The rates of 30‐day and 90‐day mortality, respectively, increased with increasing score for both the PESI and the PREP. No patients in PESI class I died by either time point, while 9.2% of PESI class V subjects expired by 30 days (P < 0.0001) and 10.5% died by 90 days (P = 0.003) (Table 2). Based on PESI, 30‐day death rates were 4.6% in the high‐risk cohort versus 0% in the low‐risk group (P = 0.023). Conversely, 7.1% of high‐risk PREP subjects died by day 30 versus 1% of low‐risk subjects (P = 0.004) (Figure 1A). Those stratified into the PESI high‐risk group had a 90‐day mortality of 6.2% versus 0% for the low‐risk group (P = 0.008) versus 9.1% in those deemed high risk by PREP, as compared to 1.5% of those scored as low risk by PREP (P = 0.001) (Figure 1B).

Figure 1

Regarding the 30‐day mortality, the negative predictive value of the PESI was 100% (95% CI, 98.6%‐100%) while that for PREP was 99.0% (95% CI, 97.6%‐99.7%); the ability of the PREP to predict 30‐day mortality was similar to the PESI (Table 4). The AUROCs for PESI and PREP for predicting 30‐day death were also equivalent; for the raw PESI score, this measured 0.858 (95% CI, 0.773‐0.943), compared to 0.719 (95% CI, 0.563‐0.875) for PREP. When these scores were dichotomized to high‐risk versus low‐risk groups, the AUROC for the PESI was 0.684 (95% CI, 0.559‐0.810) and 0.732 (95% CI, 0.571‐0.893) for PREP.

In terms of 90‐day mortality, the negative predictive values of PESI and PREP did not change: 100% (95% CI, 97.4%‐100%) and 98.5% (95% CI, 96.9%‐99.5%), respectively. The ability of PESI and PREP as predictors of 90‐day mortality was equivalent (Table 4). Here, the AUROC for the raw PESI score remained excellent at 0.835 (97% CI, 0.762‐0.907). The AUROC for PREP was akin to that of PESI at 0.704 (95% CI, 0.564‐0.844). Segregating scores into high‐risk versus low‐risk groups demonstrated that the AUROC for PESI was 0.686 (95% CI, 0.576‐0.796) compared to 0.720 (95% CI, 0.574‐0.865) for PREP.

DISCUSSION

This retrospective analysis of patients with acute PE confirms that both the PESI and the PREP are accurate scoring tools for identifying patients at low risk of death. Under both rubrics, as the score increases, the likelihood of death also increases. More importantly, we demonstrate that the negative predictive value for both the PREP and PESI are excellent. Thus, these scoring tools can distinguish those at higher risk for death versus those at low risk in a simple‐to‐apply manner. In comparing these 2 scoring systems, the PREP comparably identifies acute PE patients at risk for death when contrasted with the PESI. Given the fewer required scoring points to calculate PREP and its ability to accurately predict clinically relevant outcomes, this simpler scoring system may have greater clinical utility.

Prior studies have validated the PESI as a risk stratification tool to predict 30‐day and 90‐day mortalities. In their original derivation of the PESI, Aujesky et al. demonstrated that higher PESI scores correlated with death at 30 days.6 Acute PE patients classified into risk class I had a short‐term mortality rate of 1.1% compared to nearly 25% of patients risk stratified into risk class V. The same authors subsequently verified that there is a linear relationship between PESI score and risk of death at 90 days.10 We have also confirmed the accuracy of the PESI for identifying persons at high risk for death and documented the limited interobserver variability in this tool.7 In combination, there is evidence that the PESI can accurately predict vital outcome. Despite the effectiveness of the PESI, it is a somewhat cumbersome scoring system. It requires gathering information on 11 clinical variables, each with a different score allocation to ultimately compute the PESI score. In contrast, the PREP only requires knowing 3 clinical variables: presence of cancer, mental status, and the presence of cardiogenic shock. Akin to the PESI, the PREP and mortality are linearly related, where higher PREP scores result in higher 30‐day and 90‐day mortalities.

Our analysis helps expand the evidence regarding clinical risk stratification in PE in several ways. First, we verify that both the PESI and PREP are accurate predictors of short‐term mortality. While this has been accomplished for the PESI in prior studies, to our knowledge, this is the first confirmatory study for PREP's utility as a risk stratification tool. Second, we demonstrate that PREP is also an accurate predictor of intermediate‐term mortality. If the eventual goal is to develop tools that allow for the initial outpatient management of acute PE, clinicians require data on longer‐term outcomes to ensure that later harms do not arise based on a decision to defer hospitalization. Prior observational studies and randomized controlled clinical trials have proven that appropriately selected individuals face similar rates of complications following acute PE, whether they are managed in or outside of a hospital setting.1116 The key limitation of these earlier efforts, though, was that there was no clear standardized approach to determining whom could be safely managed solely as an outpatient. Finally, our study is unique in that we compare the discriminatory power of these 2 risk‐scoring schemes and illustrate their equivalence. As a scoring system that only requires 3 variables, the PREP is easier and simpler, and may therefore have more clinical utility than the PESI. The high negative predictive value of the PREP suggests that it has potential in identifying patients with acute PE who can safely be managed on an outpatient basis. However, given the complexity of factors associated with the decision for early discharge, these scores should be used in conjunction with, and not supplant, clinical judgment for outpatient management. Of course, formal prospective management trials incorporating both the PREP and PESI are needed to validate this concept.

Why does PREP perform so well despite the fact that it focuses on so few clinical variables? Essentially, the PREP is an effective scoring tool for acute PE because of its ability to identify individuals at risk for progressing to shock. The presence of AMS in acute PE has been associated with a greater likelihood of death, as it likely arises as a consequence of severe shock or RV strain resulting in decreased cerebral blood flow. Alternatively, altered censorium could represent a manifestation of hypoxemia from significant V/Q mismatching and/or pulmonary shunting due to the obstructive clot. This, too, portends a poorer prognosis secondary to impending respiratory failure from hypoxemia. Thus, individuals with an acute PE presenting with altered mentation merit very close observation. Similarly, pending hemodynamic instability is a concerning manifestation that warrants inpatient monitoring.5, 17, 18 At the very minimum, these individuals have RV strain and should therefore be admitted to the hospital to potentially administer more aggressive treatment modalities (ie, thrombolytics or thrombectomy). The last clinical criteria involves the presence of malignancy. The presence of a cancer may serve as a surrogate marker for those at increased risk for early recurrent thromboembolic phenomena, since malignancy is associated with a hypercoagulable state.17, 19 Perhaps there is a threshold whereby accumulating clot resulting in RV strain ensues with subsequent poorer outcomes. Thus, it clinically and physiologically seems logical that, in the absence of any of these findings, patients with acute PE will have lower mortality rates.

Thus far, other methods used for risk stratification may either be expensive, not really obtainable, or not routinely available at the time of presentation. For example, confirmation of RV strain with an echocardiogram requires a skilled technician and interpreter. In contrast, both the PESI and PREP are scored based on multiple clinical findings. Hence, they are not dependent upon a single test to determine outcome, but on various clinical variables making these scoring tools comprehensive, simple, and reliable approaches of recognizing low‐risk patients.

Our analysis has several limitations. First, the retrospective nature of this analysis subjects it to multiple forms of bias. We attempted to eliminate these biases by defining, a priori, the time frame from which vital signs can be used during scoring. We also used all‐cause mortality as our primary endpoint to minimize the possibility of ascertainment bias. However, this type of bias could not be completely eliminated since data collected was not specifically for the purpose of this study. Second, this single‐center study may limit the generalizability of these findings; yet, the diversity of patients admitted to this 900‐bed, tertiary care facility, as well as the inclusion of both inpatients and outpatients, helps to mitigate this concern. Third, the exclusion of individuals with expectant deaths within <30 days limits the applicability of these findings to this group. We chose to exclude persons with anticipated short‐term mortality to reduce the tally of patients who did not receive therapeutic treatment (ie, those transitioned to comfort care). Fourth, the use of the Social Security Death Index objectively determines death status for all‐cause mortality but cannot delineate cause‐specific death. Consequently, death strictly due to PE could not be assessed. Fifth, the original investigators for PREP assessed the PREP score with and without BNP and left‐to‐right ventricular diameter ratios. Although their results demonstrated similar AUROCs for the PREP score with and without BNP to predict 30‐day outcomes, this was a finding we could not confirm due to inconsistencies in measuring BNP and echocardiograms in our cohort. Also, our post hoc power analysis demonstrates that our findings may be limited by sample size. The lack of statistically significant differences between the PESI and the PREP may, in fact, be due to the small sample size versus true effect. Finally, tolerance for medical therapy and compliance with treatment were not documented and, therefore, were immeasurable. Poor compliance to anticoagulants or intolerability increases risk for recurrent PE, while excessive anticoagulation increases likelihood of bleeding.

In summary, the PREP and PESI can both safely predict 30‐day and 90‐day outcomes. However, the simplicity of the PREP renders it more clinician friendly. The fact that only 3 clinical noninvasive variables are required would ultimately make it the preferred bedside tool to risk stratify patients for acute PE. The high negative predictive value and comparable AUROCs establishes the effectiveness of these 2 scoring systems in recognizing low‐risk patients. Irrespective of the clinician's choice to use 1 tool over the other, both have potential for clinical application at the bedside and in clinical trials. Nevertheless, further evidence is required before they are utilized to triage patients for outpatient therapy.

Acute pulmonary embolism (PE) is associated with significant morbidity and mortality.1 While expeditious diagnosis and management results in reduced mortality, the ability to rapidly and accurately identify those at increased risk for death remains elusive. Multiple studies have utilized various biomarkers as risk stratification tools, however, these approaches have proven to have many limitations. For example, both serum brain natriuretic peptide (BNP) and troponin levels have been studied as possible risk stratification tools. Those with elevated levels of these following a PE may have concomitant right ventricular (RV) dysfunction and/or hemodynamic instability. Thus, they may face a greater risk for cardiovascular collapse and death. The low positive predictive value of these biomarkers (14%‐44%) has limited their clinical utility.24 Furthermore, imaging modalities, such as echocardiography, which is considered the clinical gold standard for determining the presence of acute RV dysfunction in PE, may not be readily available and may require special expertise for interpretation.5

Conversely, the need to identify acute PE patients at low risk for death is just as important. Recent studies suggest that carefully selected patients can successfully be managed as outpatients which can subsequently lead to significant cost savings and patient satisfaction. Movement towards enhanced outpatient resources and the advent of subcutaneous anticoagulants have made outpatient management of acute PE an appealing possibility. However, proper education, close follow‐up, and a rigorous selection process to recognize those at minimal risk for a fatal complication must all be available before clinicians prematurely discharge these patients to home.

Recently, clinical scoring tools have been developed to aid in risk stratifying patients with acute PE to accurately determine patient outcome. The pulmonary embolism severity index (PESI) is a reproducible scoring system that accurately predicts 30‐day and 90‐day mortality.6, 7 It consists of 11 clinical variables that can be quickly assessed at the time of diagnosis (Table 1A). The fact that biomarkers and imaging technology, such as echocardiography, are unnecessary to compute a PESI score demonstrates the appeal of this system. Similar to the PESI, Sanchez et al.8 have proposed the prognosis in pulmonary embolism (PREP) score as an alternate clinical risk tool in PE (Table 1B). Contrary to PESI, the PREP only uses 3 clinical variables to accurately predict vital outcome with an area under the receiver operating characteristic (AUROC) curve of 0.73 (95% confidence interval [CI], 0.65‐0.82). While both scoring systems have been developed to predict 30‐day mortality in acute PE, the comparative validity of these prognostic tools has not been assessed.

We hypothesized that the PESI more precisely risk stratifies the risk for death in acute PE compared to the PREP. Furthermore, we theorized that the PESI more reliably predicts not only 30‐day but also 90‐day mortality. To test our hypothesis, we performed a retrospective analysis, of all consecutive patients diagnosed with acute PE at our hospital, to compare the prognostic accuracy of these 2 scoring systems.

METHODS

RESULTS

The final cohort included 302 subjects (mean age: 59.7 17.2 years; 44.0% males). As Table 3 reveals, the majority of PEs was diagnosed via CT scan (76%). On presentation, 6.6% had cardiogenic shock, while 5.0% had altered censorium. In terms of comorbid conditions, 25.2% had congestive heart failure, 25.2% had cancer, and 22.2% had a prior venous thromboembolic event. Overall, 3.0% and 4.0% met our primary outcomes of death within 30‐days and 90‐days of their acute PEs, respectively.

The rates of 30‐day and 90‐day mortality, respectively, increased with increasing score for both the PESI and the PREP. No patients in PESI class I died by either time point, while 9.2% of PESI class V subjects expired by 30 days (P < 0.0001) and 10.5% died by 90 days (P = 0.003) (Table 2). Based on PESI, 30‐day death rates were 4.6% in the high‐risk cohort versus 0% in the low‐risk group (P = 0.023). Conversely, 7.1% of high‐risk PREP subjects died by day 30 versus 1% of low‐risk subjects (P = 0.004) (Figure 1A). Those stratified into the PESI high‐risk group had a 90‐day mortality of 6.2% versus 0% for the low‐risk group (P = 0.008) versus 9.1% in those deemed high risk by PREP, as compared to 1.5% of those scored as low risk by PREP (P = 0.001) (Figure 1B).

Figure 1

Regarding the 30‐day mortality, the negative predictive value of the PESI was 100% (95% CI, 98.6%‐100%) while that for PREP was 99.0% (95% CI, 97.6%‐99.7%); the ability of the PREP to predict 30‐day mortality was similar to the PESI (Table 4). The AUROCs for PESI and PREP for predicting 30‐day death were also equivalent; for the raw PESI score, this measured 0.858 (95% CI, 0.773‐0.943), compared to 0.719 (95% CI, 0.563‐0.875) for PREP. When these scores were dichotomized to high‐risk versus low‐risk groups, the AUROC for the PESI was 0.684 (95% CI, 0.559‐0.810) and 0.732 (95% CI, 0.571‐0.893) for PREP.

In terms of 90‐day mortality, the negative predictive values of PESI and PREP did not change: 100% (95% CI, 97.4%‐100%) and 98.5% (95% CI, 96.9%‐99.5%), respectively. The ability of PESI and PREP as predictors of 90‐day mortality was equivalent (Table 4). Here, the AUROC for the raw PESI score remained excellent at 0.835 (97% CI, 0.762‐0.907). The AUROC for PREP was akin to that of PESI at 0.704 (95% CI, 0.564‐0.844). Segregating scores into high‐risk versus low‐risk groups demonstrated that the AUROC for PESI was 0.686 (95% CI, 0.576‐0.796) compared to 0.720 (95% CI, 0.574‐0.865) for PREP.

DISCUSSION

This retrospective analysis of patients with acute PE confirms that both the PESI and the PREP are accurate scoring tools for identifying patients at low risk of death. Under both rubrics, as the score increases, the likelihood of death also increases. More importantly, we demonstrate that the negative predictive value for both the PREP and PESI are excellent. Thus, these scoring tools can distinguish those at higher risk for death versus those at low risk in a simple‐to‐apply manner. In comparing these 2 scoring systems, the PREP comparably identifies acute PE patients at risk for death when contrasted with the PESI. Given the fewer required scoring points to calculate PREP and its ability to accurately predict clinically relevant outcomes, this simpler scoring system may have greater clinical utility.

Prior studies have validated the PESI as a risk stratification tool to predict 30‐day and 90‐day mortalities. In their original derivation of the PESI, Aujesky et al. demonstrated that higher PESI scores correlated with death at 30 days.6 Acute PE patients classified into risk class I had a short‐term mortality rate of 1.1% compared to nearly 25% of patients risk stratified into risk class V. The same authors subsequently verified that there is a linear relationship between PESI score and risk of death at 90 days.10 We have also confirmed the accuracy of the PESI for identifying persons at high risk for death and documented the limited interobserver variability in this tool.7 In combination, there is evidence that the PESI can accurately predict vital outcome. Despite the effectiveness of the PESI, it is a somewhat cumbersome scoring system. It requires gathering information on 11 clinical variables, each with a different score allocation to ultimately compute the PESI score. In contrast, the PREP only requires knowing 3 clinical variables: presence of cancer, mental status, and the presence of cardiogenic shock. Akin to the PESI, the PREP and mortality are linearly related, where higher PREP scores result in higher 30‐day and 90‐day mortalities.

Our analysis helps expand the evidence regarding clinical risk stratification in PE in several ways. First, we verify that both the PESI and PREP are accurate predictors of short‐term mortality. While this has been accomplished for the PESI in prior studies, to our knowledge, this is the first confirmatory study for PREP's utility as a risk stratification tool. Second, we demonstrate that PREP is also an accurate predictor of intermediate‐term mortality. If the eventual goal is to develop tools that allow for the initial outpatient management of acute PE, clinicians require data on longer‐term outcomes to ensure that later harms do not arise based on a decision to defer hospitalization. Prior observational studies and randomized controlled clinical trials have proven that appropriately selected individuals face similar rates of complications following acute PE, whether they are managed in or outside of a hospital setting.1116 The key limitation of these earlier efforts, though, was that there was no clear standardized approach to determining whom could be safely managed solely as an outpatient. Finally, our study is unique in that we compare the discriminatory power of these 2 risk‐scoring schemes and illustrate their equivalence. As a scoring system that only requires 3 variables, the PREP is easier and simpler, and may therefore have more clinical utility than the PESI. The high negative predictive value of the PREP suggests that it has potential in identifying patients with acute PE who can safely be managed on an outpatient basis. However, given the complexity of factors associated with the decision for early discharge, these scores should be used in conjunction with, and not supplant, clinical judgment for outpatient management. Of course, formal prospective management trials incorporating both the PREP and PESI are needed to validate this concept.

Why does PREP perform so well despite the fact that it focuses on so few clinical variables? Essentially, the PREP is an effective scoring tool for acute PE because of its ability to identify individuals at risk for progressing to shock. The presence of AMS in acute PE has been associated with a greater likelihood of death, as it likely arises as a consequence of severe shock or RV strain resulting in decreased cerebral blood flow. Alternatively, altered censorium could represent a manifestation of hypoxemia from significant V/Q mismatching and/or pulmonary shunting due to the obstructive clot. This, too, portends a poorer prognosis secondary to impending respiratory failure from hypoxemia. Thus, individuals with an acute PE presenting with altered mentation merit very close observation. Similarly, pending hemodynamic instability is a concerning manifestation that warrants inpatient monitoring.5, 17, 18 At the very minimum, these individuals have RV strain and should therefore be admitted to the hospital to potentially administer more aggressive treatment modalities (ie, thrombolytics or thrombectomy). The last clinical criteria involves the presence of malignancy. The presence of a cancer may serve as a surrogate marker for those at increased risk for early recurrent thromboembolic phenomena, since malignancy is associated with a hypercoagulable state.17, 19 Perhaps there is a threshold whereby accumulating clot resulting in RV strain ensues with subsequent poorer outcomes. Thus, it clinically and physiologically seems logical that, in the absence of any of these findings, patients with acute PE will have lower mortality rates.

Thus far, other methods used for risk stratification may either be expensive, not really obtainable, or not routinely available at the time of presentation. For example, confirmation of RV strain with an echocardiogram requires a skilled technician and interpreter. In contrast, both the PESI and PREP are scored based on multiple clinical findings. Hence, they are not dependent upon a single test to determine outcome, but on various clinical variables making these scoring tools comprehensive, simple, and reliable approaches of recognizing low‐risk patients.

Our analysis has several limitations. First, the retrospective nature of this analysis subjects it to multiple forms of bias. We attempted to eliminate these biases by defining, a priori, the time frame from which vital signs can be used during scoring. We also used all‐cause mortality as our primary endpoint to minimize the possibility of ascertainment bias. However, this type of bias could not be completely eliminated since data collected was not specifically for the purpose of this study. Second, this single‐center study may limit the generalizability of these findings; yet, the diversity of patients admitted to this 900‐bed, tertiary care facility, as well as the inclusion of both inpatients and outpatients, helps to mitigate this concern. Third, the exclusion of individuals with expectant deaths within <30 days limits the applicability of these findings to this group. We chose to exclude persons with anticipated short‐term mortality to reduce the tally of patients who did not receive therapeutic treatment (ie, those transitioned to comfort care). Fourth, the use of the Social Security Death Index objectively determines death status for all‐cause mortality but cannot delineate cause‐specific death. Consequently, death strictly due to PE could not be assessed. Fifth, the original investigators for PREP assessed the PREP score with and without BNP and left‐to‐right ventricular diameter ratios. Although their results demonstrated similar AUROCs for the PREP score with and without BNP to predict 30‐day outcomes, this was a finding we could not confirm due to inconsistencies in measuring BNP and echocardiograms in our cohort. Also, our post hoc power analysis demonstrates that our findings may be limited by sample size. The lack of statistically significant differences between the PESI and the PREP may, in fact, be due to the small sample size versus true effect. Finally, tolerance for medical therapy and compliance with treatment were not documented and, therefore, were immeasurable. Poor compliance to anticoagulants or intolerability increases risk for recurrent PE, while excessive anticoagulation increases likelihood of bleeding.

In summary, the PREP and PESI can both safely predict 30‐day and 90‐day outcomes. However, the simplicity of the PREP renders it more clinician friendly. The fact that only 3 clinical noninvasive variables are required would ultimately make it the preferred bedside tool to risk stratify patients for acute PE. The high negative predictive value and comparable AUROCs establishes the effectiveness of these 2 scoring systems in recognizing low‐risk patients. Irrespective of the clinician's choice to use 1 tool over the other, both have potential for clinical application at the bedside and in clinical trials. Nevertheless, further evidence is required before they are utilized to triage patients for outpatient therapy.

Healthcare associated pneumonia (HCAP) is defined as pneumonia that is present upon admission, and occurs in patients that have recently been hospitalized, reside in a nursing home, or have had other recent healthcare exposures. Practice guidelines developed by the American Thoracic Society (ATS) and the Infectious Diseases Society of America (IDSA), recommend strategies for the diagnosis and treatment of patients with HCAP.1 A premise of the guidelines is that recent healthcare exposure places patients at risk for infection due to multi‐drug resistant (MDR) pathogens such as methicillin‐resistant Staphylococcus aureus (MRSA) or Pseudomonas aeruginosa. In addition to criteria utilized to define HCAP, the guidelines state that recent immunosuppression and antibiotic exposure are risk factors for pneumonia due to MDR pathogens. In contrast to the treatment of community‐acquired pneumonia (CAP), the guidelines recommend empirical administration of antibiotics with activity against MRSA and Pseudomonas aeruginosa for all patients with HCAP.

We recently reported that antimicrobial resistance to CAP antibiotics (CAP‐resistance) was identified in one‐third of culture‐positive patients with HCAP.2 Data regarding the predictive ability of the guideline‐defined criteria specific to HCAP are limited.3 Evaluation and potential refinement of the criteria to identify patients at risk for MDR pathogens can aid in making antibiotic‐related treatment decisions.

The purposes of this study are to: 1) develop and validate a model to predict CAP‐resistance among patients with HCAP, and to compare the model's predictive performance to a model that includes traditional guideline‐defined risk factors; and 2) develop models to predict recovery of pathogen‐specific etiology (MRSA and Pseudomonas aeruginosa), and to compare the predictive performance of the pathogen‐specific and CAP‐resistance models.

METHODS

Patients with HCAP who were admitted to 6 Veterans Affairs Medical Centers (VAMC) in the northwestern United States between January 1, 2003 and December 31, 2008 were included in the retrospective cohort study. The cohort was identified utilizing medical records data extracted from the Veterans Integrated Service Network (VISN20) Data Warehouse. The Data Warehouse is a centralized open architecture relational database that houses medical and administrative records data for VISN20 patients. This research complies with all federal guidelines and VAMC policies relative to human subjects and clinical research.

Subjects were identified by the following pneumonia‐related discharge International Classification of Diseases (ICD‐9 CM) codes: 1) a primary diagnosis of 480‐483; 485‐487.0 (pneumonia); or 2) a primary diagnosis of 507.0 (pneumonitis), 518.8 (respiratory failure), or 0.38 (septicemia), and a secondary diagnosis of 480‐483; 485‐487.0.4 Eligibility required that patients received antibiotic therapy for pneumonia within 24 hours of admission, continue inpatient treatment for >24 hours, and meet any of the following guideline‐defined criteria: 1) hospitalization during the preceding 90 days; 2) admission from a nursing home; 3) outpatient or home wound care, outpatient or home infusion therapy, or chronic hemodialysis.1 In addition, patients not meeting guideline‐defined criteria, who had frequent healthcare system exposure, defined as 12 Emergency Department, Medicine, or Surgery clinic visits within 90 days of admission, were also included. Patients were excluded if they were directly transferred from another hospital, or had pneumonia‐related ICD‐9 codes but received inpatient care for pneumonia in a non‐VA hospital.

Study data included medical records for the year prior to admission for HCAP through 30 days afterwards. Data included: demographics; domicile preceding admission; healthcare utilization including diagnosis and procedure codes; inpatient medications administered, and outpatient prescription fills; vital signs; and laboratory test results, including cultures and susceptibilities.

Guideline‐defined criteria for predicting CAP‐resistance were similar to those used to identify the study cohort. Nursing home admission included patients who were directly admitted from a nursing home, skilled nursing facility, or domiciliary. Prior hospitalization 2 days within 90 days was calculated by summing the length of stay for all admissions during the preceding 90 days. Outpatient intravenous therapy, chronic hemodialysis, and wound care therapy was determined from medication administration records and relevant Current Procedural Terminology (CPT) or ICD‐9 procedure codes for care administered within 30 days. Antibiotic exposure was defined as administration of 1 dose of antibiotic during inpatient care, or fill of an outpatient prescription for 1 antibiotic dose within 90 days preceding admission. Immunosuppression was defined as: human immunodeficiency virus (HIV) diagnosis; white blood cell (WBC) count of 2500 cells/mm³ within 30 days of admission; corticosteroid ingestion during prior admission, or outpatient prescription fills for a corticosteroid with quantity sufficient to last 14 days preceding admission; or inpatient ingestion of, or outpatient prescription fills for, transplant or rheumatologic‐related immunosuppressants within 90 days preceding admission.

Additional variables assessed to predict CAP‐resistance were obtained as follows. First, modifications of guideline‐defined criteria were constructed. These included: direct nursing home admission, or recent nursing home stay preceding admission; total days of hospitalization within 90 days preceding admission; specific antibiotic exposures, including dates since last exposure preceding admission; and individual components of the immunosuppression criterion. Other cohort‐developed variables included: demographics; substance use history; chronic comorbidity determined by individual and composite measures of Charlson score; pulmonary disease history (eg, bronchiectasis); type and frequency of outpatient visits; consecutive (2) prescription fills for chronic medications of interest; clinical and surveillance culture results preceding admission; admitting ward; vital signs; and relevant hematology and chemistry labs.5

Sputum, blood, and bronchoscopy‐collected cultures obtained within 48 hours after admission were assessed to determine specimen acceptability. Poor sputum specimens were defined by Gram stain quantitative results indicating >10 epithelial cells (EPI) per low power field (LPF), or in the absence of quantitative results, semi‐quantitative results indicating 2‐4+EPI. Single positive blood cultures with results indicating likely contaminants were considered poor specimens. All bronchoscopy‐obtained specimens were considered acceptable. All cultures classified as poor specimens were excluded, and microbiology results were evaluated for the remaining specimens.2, 6 Organisms thought to represent colonization or contamination were excluded: coagulase‐negative (CN) Staphylococcus, Enterococcus sp, Bacillus sp, Proprionibacterium sp, and Candida sp. Recovery of a potential pneumonia pathogen from 1 acceptable culture constituted a culture‐positive admission.

CAP‐resistance was determined for each isolate. CAP‐resistance was defined as non‐susceptibility to non‐pseudomonal third generation cephalosporins (ceftriaxone or cefotaxime) or non‐pseudomonal 8‐methoxy fluoroquinolones (moxifloxacin, gatifloxacin), the VA preferred agents for treatment of CAP.7 There were differences between facilities in susceptibility reporting criteria; therefore, the following approach was used to determine CAP‐resistance. First, MRSA and Pseudomonas aeruginosa isolates were classified as CAP‐resistant. Second, susceptibility results were directly utilized to determine CAP‐resistance if both antibiotic results were available. Third, if only a surrogate antibiotic from a class was reported, a representative antibiotic consistent with Clinical Laboratory Standards Institute reporting criteria was utilized.8 Finally, expert rules determined CAP‐resistance for select potential pneumonia pathogens (eg, Haemophilus sp) if antibiotic susceptibility results for both cephalosporin and fluoroquinolone classes were not reported.815 Presence of 1 CAP‐resistant isolate resulted in a CAP‐resistant classification for an admission. MRSA and Pseudomonas aeruginosa endpoints were defined in a similar manner. Only the first admission for each patient was utilized in the analysis.

The probability of CAP‐resistance was predicted from guideline‐defined criteria (guideline‐defined model) with logistic regression. Next, non‐guideline variables were classified as high, medium, or low interest for association with CAP‐resistance. Variables were assessed for collinearity. A model of CAP‐resistance was developed from variables of high interest. Guideline‐defined criteria were omitted to allow consideration of more specific measures (eg, specific antibiotic exposures as opposed to receipt of antibiotics within the preceding 90 days) during this stage. Next, guideline‐defined criteria, and subsequently variables of lesser interest, were added in an attempt to improve the model. Annual trends and plausible interactions were considered. Model selection was by Akaike's Information Criterion (AIC).16 To promote model reliability, the final model was required to lack evidence of over‐fitting in bootstrapped internal validation.17 The guideline‐defined and cohort‐developed models were compared by difference in area under receiver operating characteristic (aROC) curves. The model development process was repeated for MRSA and Pseudomonas aeruginosa endpoints. Finally, to determine if the CAP‐resistance model sufficiently predicted pathogen‐specific MDR, the CAP‐resistance model was re‐estimated for MRSA and Pseudomonas aeruginosa endpoints. Statistical analysis was performed with R version 2.10.0 (The R Foundation for Statistical Computing, Vienna, Austria).

RESULTS

The cohort was comprised of 1300 patients with HCAP. Of these, 375 (28.8% [26.4‐31.4]) met culture‐positive criteria for potential pneumonia pathogens. CAP‐resistant organisms were identified in 118 (31.5% [26.8‐36.4]) patients within 48 hours of admission. CAP‐resistant organisms included: MRSA (49.2% [40.4‐58.1]), Pseudomonas aeruginosa (29.5% [21.9‐38.1]), Enterobacteriaceae (11.4% [6.5‐18.0]), Gram‐negative non‐enterics (8.3% [4.2‐14.4]), Streptococcus pneumoniae (1.5% [0.2‐5.4]), and opportunistic organisms (eg, Mycobacterium spp) (8.3% [4.2‐14.4]). Differences in select characteristics and exposures between culture‐positive and culture‐negative admissions, as well as CAP‐resistant and CAP‐sensitive admissions, were evident (Table 1).

Of the guideline‐defined criteria, direct admission from a nursing home, prior hospitalization, and recent antibiotic exposure were associated with CAP‐resistance (Table 2). The cohort‐derived CAP‐resistance model included 6 variables. Prior MRSA colonization or infection within 90 days preceding admission was strongly predictive of CAP‐resistance. A composite variable consisting of direct admission from a nursing home or admission from the community after recent discharge from a nursing home was more predictive than direct admission from a nursing home alone. Exposure to cephalosporin antibiotics within the prior year was also predictive of CAP‐resistance. Subcategorizing cephalosporins by class or by most recent exposure in 90‐day increments did not improve the model. The remaining predictors in the model were guideline‐defined infusion therapy criterion, diabetes, and intensive care unit (ICU) admission.

Of the guideline‐defined criteria, direct admission from a nursing home was most predictive of MRSA HCAP (n = 57), followed by prior hospitalization and recent antibiotic exposure (Table 3). The cohort‐developed model of MRSA HCAP included predictors common to the CAP‐resistance model: direct admission from a nursing home or patients who were recently discharged from a nursing home, history of prior MRSA, and diabetes. Positive MRSA status within 90 days preceding admission exhibited the strongest prediction of MRSA HCAP. Exposure to anti‐pseudomonal fluoroquinolones (ciprofloxacin and levofloxacin) within the prior year was also predictive of MRSA HCAP, however, exposure to 8‐methoxy fluoroquinolone was not (crude odds ratio (OR) = 0.7 [0.3‐1.4]; final model adjusted OR = 0.6 [0.2‐1.2]). Exposure to third generation cephalosporins within the previous year was more predictive than other cephalosporin exposures, and more predictive than exposure times categorized in 90‐day increments.

Of the guideline‐defined criteria, only prior hospitalization within 90 days and admission from a nursing home were predictive of Pseudomonas aeruginosa HCAP (n = 36) (Table 4). In the cohort‐developed model of Pseudomonas aeruginosa HCAP, Pseudomonas aeruginosa was predicted by prior cephalosporin exposure within the preceding year, prior culture of Pseudomonas aeruginosa from any anatomical source within the preceding year, and chronic steroid use of 10 mg/day prednisone equivalents. Again, the model was not improved by subcategorizing cephalosporin by class or by most recent exposure time. Finally, a negative annual trend in Pseudomonas aeruginosa HCAP was evident.

The cohort‐developed model of CAP‐resistance was re‐estimated for MRSA and Pseudomonas aeruginosa endpoints. Only positive MRSA status within 90 days preceding admission was associated with both endpoints (OR = 8.7 [3.5‐22.1] for MRSA; OR = 4.3 [1.4‐12.2] for Pseudomonas aeruginosa). Direct or recent nursing home residence (OR = 2.4 [1.3‐4.6]) and diabetes (OR = 2.4 [1.3‐4.5]) were highly predictive of MRSA, but not Pseudomonas aeruginosa (OR = 1.8 [0.8‐3.9] for nursing home residence; OR = 1.3 [0.6‐2.7] for diabetes), respectively. Cephalosporin exposure preceding admission was highly predictive of Pseudomonas aeruginosa (OR = 4.0 [1.9‐9.3]), but not with MRSA (OR = 1.1 [0.6‐2.1]). In these models, all estimated odds ratios were >1.0, consistent with the cohort‐developed model of CAP‐resistance.

For each endpoint, the cohort‐developed model was more predictive than the guideline‐defined model (Table 5) (to view ROC curves see Supporting Figures 1 to 3 in the online version of the article.). The cohort‐developed model of CAP‐resistance re‐estimated for pathogen‐specific endpoints resulted in similar predictive performance. To assess performance of the cohort developed models by facility, aROC was calculated for each of the 3 larger sites separately and for the 3 smaller facilities combined due to limited counts. Site specific aROC ranged from 0.652 to 0.762 for CAP‐resistance, 0.725 to 0.815 for MRSA, and 0.719 to 0.801 for Pseudomonas aeruginosa. The cohort‐developed model of CAP‐resistance re‐estimated for pathogen‐specific endpoints resulted in similar predictive performance.

A nomogram for the cohort‐developed model of CAP‐resistance can provide the predicted probability of culturing a CAP‐resistant organism for an individual patient (Table 6). Point scores assigned to levels of variables, are summed to obtain a total score, and the total score corresponds to a predicted probability of CAP‐resistance. The prevalence of CAP‐resistance (%) from highest to lowest quartile of predicted probability was 92.9, 58.8, 32.9, and 18.5, respectively.

DISCUSSION

In this study, select ATS/IDSA guideline‐defined criteria predicted identification of CAP‐resistant organisms in patients with HCAP. Admission from a nursing home was most predictive of CAP‐resistant organisms, whereas recent hospitalization and antibiotic exposure were predictive to a lesser extent. There was weak evidence of associations between recent infusion and chronic hemodialysis criteria with MDR endpoints. Recent wound care and a composite definition of immunosuppression were not predictive of these endpoints.

The cohort‐developed model resulted in improved prediction of CAP‐resistance endpoints. Culture history, particularly history of MRSA within 90 days preceding admission, was a strong predictor of MDR endpoints. The MRSA history variable definition included cultures from all anatomical sources and nares polymerase chain reaction surveillance results, the latter increasing in 2007‐2008 due to the implementation of the VA MRSA initiative.18 This finding suggests that prior culture results should be considered when selecting empirical antimicrobial therapy, and the rapid proliferation of electronic medical records increases potential to utilize this information routinely. While the guideline‐defined nursing home admission criterion was a strong predictor of CAP‐resistance, admission from the community after recent discharge from a nursing home, in addition to direct admission from a nursing home, was also important.

Similarities in variables included in the pathogen‐specific and CAP‐resistance models reflect the importance of MRSA in defining the CAP‐resistance endpoint. Both CAP‐resistance and MRSA models included prior MRSA status, diabetes, and ICU admission, whereas cephalosporin exposure was common to the Pseudomonas aeruginosa and CAP‐resistance models. Annual trends in CAP‐resistance and MRSA recovery were not identified. The negative annual trend in Pseudomonas aeruginosa HCAP is unexplained and beyond the scope of this study. The percentage of culture‐positive admissions with Pseudomonas aeruginosa HCAP averaged 12% in 2003‐2006, but dropped to <5% in 2007‐2008. A potential explanation is that identification and isolation of patients with MRSA, as a result of the VA‐wide MRSA initiative, may have impacted Pseudomonas aeruginosa colonization by isolating patients co‐colonized with these pathogens during prior healthcare exposures. This is consistent with the observation that when the cohort‐derived CAP‐resistance model was refit with the Pseudomonas aeruginosa endpoint, recent MRSA colonization was strongly predictive of Pseudomonas aeruginosa. Despite differences between variables in pathogen‐specific and CAP‐resistant models, the CAP‐resistance model provided a similar degree of MRSA and Pseudomonas aeruginosa prediction. Finally, as a study purpose included developing best predictive models for each endpoint, and not merely identifying associations, there were other plausible models not reported.

Study strengths included use of the VISN20 Data Warehouse, which provided an integrated outpatient and inpatient medical record. This facilitated analysis of prior healthcare exposures and inpatient study endpoints. In addition, poor blood and sputum specimens and unlikely pneumonia pathogens were not included in establishing MDR endpoints. The variable set explored in regression modeling was extensive and detailed, and analysis included time and intensity‐based components of the variables. Importantly, a standardized approach to regression modeling was specified in advance, which included identification of variables with high potential for association with MDR endpoints, model selection by AIC, re‐evaluation of guideline‐defined criteria and variables of lower interest, and bootstrapped internal model validation.19

Study limitations included the use of ICD‐9 codes to establish a pneumonia diagnosis, which may lack sensitivity and specificity. However, an enhanced ICD‐9based algorithm superior to other claims‐based definitions of pneumonia was utilized.4, 20 Veterans may have received care at non‐VA facilities impacting identification of all healthcare system exposures preceding admission. Data for microbial endpoints were obtained from sterile and non‐sterile site cultures, and it was not possible to determine if the cultured organisms were truly pathogenic. While pathogen‐specific endpoints were not affected, the use of expert rules in select cases to establish CAP‐resistance may have impacted precision for this endpoint. It is also possible that refitting the cohort‐developed CAP‐resistance model for pathogen‐specific endpoints resulted in optimistic aROC due to model over‐fitting. Finally, the cohort was comprised of elderly males, and caution is warranted in extrapolating the results to other populations.

The predictive ability of the guideline‐defined criteria to identify patients with MDR pathogens has been studied. A prospective observational cohort study of 625 consecutive ICU admissions determined that the guideline‐defined criteriaprior antimicrobial treatment, nursing home residence, and prior hospitalizationwere associated with recovery of MDR colonization.21 Shorr et al., investigating a retrospective cohort of 619 patients with HCAP, reported that recent hospitalization, nursing home residence, hemodialysis, and ICU admission were associated with infections caused by CAP‐resistant organisms.22 This study did not report antimicrobial exposures. Our study complements these studies by evaluating existing HCAP guideline criteria, and identifying specific antibiotic exposure, prior culture data, comorbid illness, and immunosuppressive medications that are predictive of MDR infection.

Studies comparing the bacterial etiology of patients with pneumonia in nursing homes relative to CAP, have demonstrated mixed results in recovery of Gram‐negative MDR pathogens, but generally increased MRSA pneumonia.3 Our study suggests that a nursing home stay in the last 6 months is associated with an increased risk for MRSA, but not Pseudomonas aeruginosa, although this was limited by small sample size. Recent infusion therapy has not been previously reported to be associated with MDR pathogens in an HCAP population. In our study, this criterion was predictive of CAP‐resistance in the cohort‐developed model, but not in conjunction with other variables in the guideline‐defined model. Predictors of pathogen‐specific HCAP are limited to an aforementioned single prior study, which identified recent hospitalization, nursing home residence, and ICU admission as risk factors for MRSA HCAP.22

Many studies have investigated risks for infection with MRSA and Pseudomonas aeruginosa outside of the context of HCAP. Predictor variables in cohort‐developed pathogen‐specific models in our study are known risk factors for colonization or infection with these pathogens. For example, antecedent MRSA colonization has been noted as a strong risk factor for MRSA infection, particularly pneumonia.23, 24 Further, patients with diabetes and inhaled corticosteroid exposure are immunosuppressed and at increased risk for colonization with MRSA.25, 26 Likewise, bronchiolar colonization and corticosteroid exposures are known risk factors for pneumonia due to Pseudomonas aeruginosa.27

Many studies have identified prior antibiotic use as a risk factor for infections caused by MRSA and Pseudomonas aeruginosa. However, this criterion is excessively broad and specific antimicrobial exposures carry different magnitudes of risk. Third generation cephalosporins and anti‐pseudomonal fluoroquinolones are commonly reported antibiotics associated with risk for MRSA infection, whereas 8‐methoxy fluoroquinolones appear not to possess the same effect.2831 Likewise, cephalosporins have been reported as risk factors for MDR Pseudomonas aeruginosa infections.32

Several areas of research involving HCAP MDR risk should be investigated. First, the predictive models developed in our and other studies should be evaluated in larger, more diverse populations to establish generalizability. Second, empirical broad‐spectrum antibiotic therapy in all patients with HCAP results in overtreatment of many patients. To date, no reported models provided optimal performance for selecting empirical therapy for unstable ICU patients with HCAP, and many patients do not receive de‐escalation therapy. Thus, models to identify patients with low probability of MDR pathogens upon admission and to aid in de‐escalation are warranted. Finally, the negative trend in Pseudomonas aeruginosa HCAP requires confirmation and further study.

In conclusion, of the ATS/IDSA guideline‐defined criteria for MDR, nursing home admission, recent hospitalization, and antibiotic exposure were predictive of the recovery of CAP‐resistant organisms. Alternative models primarily based on prior culture data, specific antibiotic exposures, and immunosuppression‐related variables improved predictive performance of HCAP associated with MDR.

Healthcare associated pneumonia (HCAP) is defined as pneumonia that is present upon admission, and occurs in patients that have recently been hospitalized, reside in a nursing home, or have had other recent healthcare exposures. Practice guidelines developed by the American Thoracic Society (ATS) and the Infectious Diseases Society of America (IDSA), recommend strategies for the diagnosis and treatment of patients with HCAP.1 A premise of the guidelines is that recent healthcare exposure places patients at risk for infection due to multi‐drug resistant (MDR) pathogens such as methicillin‐resistant Staphylococcus aureus (MRSA) or Pseudomonas aeruginosa. In addition to criteria utilized to define HCAP, the guidelines state that recent immunosuppression and antibiotic exposure are risk factors for pneumonia due to MDR pathogens. In contrast to the treatment of community‐acquired pneumonia (CAP), the guidelines recommend empirical administration of antibiotics with activity against MRSA and Pseudomonas aeruginosa for all patients with HCAP.

We recently reported that antimicrobial resistance to CAP antibiotics (CAP‐resistance) was identified in one‐third of culture‐positive patients with HCAP.2 Data regarding the predictive ability of the guideline‐defined criteria specific to HCAP are limited.3 Evaluation and potential refinement of the criteria to identify patients at risk for MDR pathogens can aid in making antibiotic‐related treatment decisions.

The purposes of this study are to: 1) develop and validate a model to predict CAP‐resistance among patients with HCAP, and to compare the model's predictive performance to a model that includes traditional guideline‐defined risk factors; and 2) develop models to predict recovery of pathogen‐specific etiology (MRSA and Pseudomonas aeruginosa), and to compare the predictive performance of the pathogen‐specific and CAP‐resistance models.

METHODS

Patients with HCAP who were admitted to 6 Veterans Affairs Medical Centers (VAMC) in the northwestern United States between January 1, 2003 and December 31, 2008 were included in the retrospective cohort study. The cohort was identified utilizing medical records data extracted from the Veterans Integrated Service Network (VISN20) Data Warehouse. The Data Warehouse is a centralized open architecture relational database that houses medical and administrative records data for VISN20 patients. This research complies with all federal guidelines and VAMC policies relative to human subjects and clinical research.

Subjects were identified by the following pneumonia‐related discharge International Classification of Diseases (ICD‐9 CM) codes: 1) a primary diagnosis of 480‐483; 485‐487.0 (pneumonia); or 2) a primary diagnosis of 507.0 (pneumonitis), 518.8 (respiratory failure), or 0.38 (septicemia), and a secondary diagnosis of 480‐483; 485‐487.0.4 Eligibility required that patients received antibiotic therapy for pneumonia within 24 hours of admission, continue inpatient treatment for >24 hours, and meet any of the following guideline‐defined criteria: 1) hospitalization during the preceding 90 days; 2) admission from a nursing home; 3) outpatient or home wound care, outpatient or home infusion therapy, or chronic hemodialysis.1 In addition, patients not meeting guideline‐defined criteria, who had frequent healthcare system exposure, defined as 12 Emergency Department, Medicine, or Surgery clinic visits within 90 days of admission, were also included. Patients were excluded if they were directly transferred from another hospital, or had pneumonia‐related ICD‐9 codes but received inpatient care for pneumonia in a non‐VA hospital.

Study data included medical records for the year prior to admission for HCAP through 30 days afterwards. Data included: demographics; domicile preceding admission; healthcare utilization including diagnosis and procedure codes; inpatient medications administered, and outpatient prescription fills; vital signs; and laboratory test results, including cultures and susceptibilities.

Guideline‐defined criteria for predicting CAP‐resistance were similar to those used to identify the study cohort. Nursing home admission included patients who were directly admitted from a nursing home, skilled nursing facility, or domiciliary. Prior hospitalization 2 days within 90 days was calculated by summing the length of stay for all admissions during the preceding 90 days. Outpatient intravenous therapy, chronic hemodialysis, and wound care therapy was determined from medication administration records and relevant Current Procedural Terminology (CPT) or ICD‐9 procedure codes for care administered within 30 days. Antibiotic exposure was defined as administration of 1 dose of antibiotic during inpatient care, or fill of an outpatient prescription for 1 antibiotic dose within 90 days preceding admission. Immunosuppression was defined as: human immunodeficiency virus (HIV) diagnosis; white blood cell (WBC) count of 2500 cells/mm³ within 30 days of admission; corticosteroid ingestion during prior admission, or outpatient prescription fills for a corticosteroid with quantity sufficient to last 14 days preceding admission; or inpatient ingestion of, or outpatient prescription fills for, transplant or rheumatologic‐related immunosuppressants within 90 days preceding admission.

Additional variables assessed to predict CAP‐resistance were obtained as follows. First, modifications of guideline‐defined criteria were constructed. These included: direct nursing home admission, or recent nursing home stay preceding admission; total days of hospitalization within 90 days preceding admission; specific antibiotic exposures, including dates since last exposure preceding admission; and individual components of the immunosuppression criterion. Other cohort‐developed variables included: demographics; substance use history; chronic comorbidity determined by individual and composite measures of Charlson score; pulmonary disease history (eg, bronchiectasis); type and frequency of outpatient visits; consecutive (2) prescription fills for chronic medications of interest; clinical and surveillance culture results preceding admission; admitting ward; vital signs; and relevant hematology and chemistry labs.5

Sputum, blood, and bronchoscopy‐collected cultures obtained within 48 hours after admission were assessed to determine specimen acceptability. Poor sputum specimens were defined by Gram stain quantitative results indicating >10 epithelial cells (EPI) per low power field (LPF), or in the absence of quantitative results, semi‐quantitative results indicating 2‐4+EPI. Single positive blood cultures with results indicating likely contaminants were considered poor specimens. All bronchoscopy‐obtained specimens were considered acceptable. All cultures classified as poor specimens were excluded, and microbiology results were evaluated for the remaining specimens.2, 6 Organisms thought to represent colonization or contamination were excluded: coagulase‐negative (CN) Staphylococcus, Enterococcus sp, Bacillus sp, Proprionibacterium sp, and Candida sp. Recovery of a potential pneumonia pathogen from 1 acceptable culture constituted a culture‐positive admission.

CAP‐resistance was determined for each isolate. CAP‐resistance was defined as non‐susceptibility to non‐pseudomonal third generation cephalosporins (ceftriaxone or cefotaxime) or non‐pseudomonal 8‐methoxy fluoroquinolones (moxifloxacin, gatifloxacin), the VA preferred agents for treatment of CAP.7 There were differences between facilities in susceptibility reporting criteria; therefore, the following approach was used to determine CAP‐resistance. First, MRSA and Pseudomonas aeruginosa isolates were classified as CAP‐resistant. Second, susceptibility results were directly utilized to determine CAP‐resistance if both antibiotic results were available. Third, if only a surrogate antibiotic from a class was reported, a representative antibiotic consistent with Clinical Laboratory Standards Institute reporting criteria was utilized.8 Finally, expert rules determined CAP‐resistance for select potential pneumonia pathogens (eg, Haemophilus sp) if antibiotic susceptibility results for both cephalosporin and fluoroquinolone classes were not reported.815 Presence of 1 CAP‐resistant isolate resulted in a CAP‐resistant classification for an admission. MRSA and Pseudomonas aeruginosa endpoints were defined in a similar manner. Only the first admission for each patient was utilized in the analysis.

The probability of CAP‐resistance was predicted from guideline‐defined criteria (guideline‐defined model) with logistic regression. Next, non‐guideline variables were classified as high, medium, or low interest for association with CAP‐resistance. Variables were assessed for collinearity. A model of CAP‐resistance was developed from variables of high interest. Guideline‐defined criteria were omitted to allow consideration of more specific measures (eg, specific antibiotic exposures as opposed to receipt of antibiotics within the preceding 90 days) during this stage. Next, guideline‐defined criteria, and subsequently variables of lesser interest, were added in an attempt to improve the model. Annual trends and plausible interactions were considered. Model selection was by Akaike's Information Criterion (AIC).16 To promote model reliability, the final model was required to lack evidence of over‐fitting in bootstrapped internal validation.17 The guideline‐defined and cohort‐developed models were compared by difference in area under receiver operating characteristic (aROC) curves. The model development process was repeated for MRSA and Pseudomonas aeruginosa endpoints. Finally, to determine if the CAP‐resistance model sufficiently predicted pathogen‐specific MDR, the CAP‐resistance model was re‐estimated for MRSA and Pseudomonas aeruginosa endpoints. Statistical analysis was performed with R version 2.10.0 (The R Foundation for Statistical Computing, Vienna, Austria).

RESULTS

The cohort was comprised of 1300 patients with HCAP. Of these, 375 (28.8% [26.4‐31.4]) met culture‐positive criteria for potential pneumonia pathogens. CAP‐resistant organisms were identified in 118 (31.5% [26.8‐36.4]) patients within 48 hours of admission. CAP‐resistant organisms included: MRSA (49.2% [40.4‐58.1]), Pseudomonas aeruginosa (29.5% [21.9‐38.1]), Enterobacteriaceae (11.4% [6.5‐18.0]), Gram‐negative non‐enterics (8.3% [4.2‐14.4]), Streptococcus pneumoniae (1.5% [0.2‐5.4]), and opportunistic organisms (eg, Mycobacterium spp) (8.3% [4.2‐14.4]). Differences in select characteristics and exposures between culture‐positive and culture‐negative admissions, as well as CAP‐resistant and CAP‐sensitive admissions, were evident (Table 1).

Of the guideline‐defined criteria, direct admission from a nursing home, prior hospitalization, and recent antibiotic exposure were associated with CAP‐resistance (Table 2). The cohort‐derived CAP‐resistance model included 6 variables. Prior MRSA colonization or infection within 90 days preceding admission was strongly predictive of CAP‐resistance. A composite variable consisting of direct admission from a nursing home or admission from the community after recent discharge from a nursing home was more predictive than direct admission from a nursing home alone. Exposure to cephalosporin antibiotics within the prior year was also predictive of CAP‐resistance. Subcategorizing cephalosporins by class or by most recent exposure in 90‐day increments did not improve the model. The remaining predictors in the model were guideline‐defined infusion therapy criterion, diabetes, and intensive care unit (ICU) admission.

Of the guideline‐defined criteria, direct admission from a nursing home was most predictive of MRSA HCAP (n = 57), followed by prior hospitalization and recent antibiotic exposure (Table 3). The cohort‐developed model of MRSA HCAP included predictors common to the CAP‐resistance model: direct admission from a nursing home or patients who were recently discharged from a nursing home, history of prior MRSA, and diabetes. Positive MRSA status within 90 days preceding admission exhibited the strongest prediction of MRSA HCAP. Exposure to anti‐pseudomonal fluoroquinolones (ciprofloxacin and levofloxacin) within the prior year was also predictive of MRSA HCAP, however, exposure to 8‐methoxy fluoroquinolone was not (crude odds ratio (OR) = 0.7 [0.3‐1.4]; final model adjusted OR = 0.6 [0.2‐1.2]). Exposure to third generation cephalosporins within the previous year was more predictive than other cephalosporin exposures, and more predictive than exposure times categorized in 90‐day increments.

Of the guideline‐defined criteria, only prior hospitalization within 90 days and admission from a nursing home were predictive of Pseudomonas aeruginosa HCAP (n = 36) (Table 4). In the cohort‐developed model of Pseudomonas aeruginosa HCAP, Pseudomonas aeruginosa was predicted by prior cephalosporin exposure within the preceding year, prior culture of Pseudomonas aeruginosa from any anatomical source within the preceding year, and chronic steroid use of 10 mg/day prednisone equivalents. Again, the model was not improved by subcategorizing cephalosporin by class or by most recent exposure time. Finally, a negative annual trend in Pseudomonas aeruginosa HCAP was evident.

The cohort‐developed model of CAP‐resistance was re‐estimated for MRSA and Pseudomonas aeruginosa endpoints. Only positive MRSA status within 90 days preceding admission was associated with both endpoints (OR = 8.7 [3.5‐22.1] for MRSA; OR = 4.3 [1.4‐12.2] for Pseudomonas aeruginosa). Direct or recent nursing home residence (OR = 2.4 [1.3‐4.6]) and diabetes (OR = 2.4 [1.3‐4.5]) were highly predictive of MRSA, but not Pseudomonas aeruginosa (OR = 1.8 [0.8‐3.9] for nursing home residence; OR = 1.3 [0.6‐2.7] for diabetes), respectively. Cephalosporin exposure preceding admission was highly predictive of Pseudomonas aeruginosa (OR = 4.0 [1.9‐9.3]), but not with MRSA (OR = 1.1 [0.6‐2.1]). In these models, all estimated odds ratios were >1.0, consistent with the cohort‐developed model of CAP‐resistance.

For each endpoint, the cohort‐developed model was more predictive than the guideline‐defined model (Table 5) (to view ROC curves see Supporting Figures 1 to 3 in the online version of the article.). The cohort‐developed model of CAP‐resistance re‐estimated for pathogen‐specific endpoints resulted in similar predictive performance. To assess performance of the cohort developed models by facility, aROC was calculated for each of the 3 larger sites separately and for the 3 smaller facilities combined due to limited counts. Site specific aROC ranged from 0.652 to 0.762 for CAP‐resistance, 0.725 to 0.815 for MRSA, and 0.719 to 0.801 for Pseudomonas aeruginosa. The cohort‐developed model of CAP‐resistance re‐estimated for pathogen‐specific endpoints resulted in similar predictive performance.

A nomogram for the cohort‐developed model of CAP‐resistance can provide the predicted probability of culturing a CAP‐resistant organism for an individual patient (Table 6). Point scores assigned to levels of variables, are summed to obtain a total score, and the total score corresponds to a predicted probability of CAP‐resistance. The prevalence of CAP‐resistance (%) from highest to lowest quartile of predicted probability was 92.9, 58.8, 32.9, and 18.5, respectively.

DISCUSSION

In this study, select ATS/IDSA guideline‐defined criteria predicted identification of CAP‐resistant organisms in patients with HCAP. Admission from a nursing home was most predictive of CAP‐resistant organisms, whereas recent hospitalization and antibiotic exposure were predictive to a lesser extent. There was weak evidence of associations between recent infusion and chronic hemodialysis criteria with MDR endpoints. Recent wound care and a composite definition of immunosuppression were not predictive of these endpoints.

The cohort‐developed model resulted in improved prediction of CAP‐resistance endpoints. Culture history, particularly history of MRSA within 90 days preceding admission, was a strong predictor of MDR endpoints. The MRSA history variable definition included cultures from all anatomical sources and nares polymerase chain reaction surveillance results, the latter increasing in 2007‐2008 due to the implementation of the VA MRSA initiative.18 This finding suggests that prior culture results should be considered when selecting empirical antimicrobial therapy, and the rapid proliferation of electronic medical records increases potential to utilize this information routinely. While the guideline‐defined nursing home admission criterion was a strong predictor of CAP‐resistance, admission from the community after recent discharge from a nursing home, in addition to direct admission from a nursing home, was also important.

Similarities in variables included in the pathogen‐specific and CAP‐resistance models reflect the importance of MRSA in defining the CAP‐resistance endpoint. Both CAP‐resistance and MRSA models included prior MRSA status, diabetes, and ICU admission, whereas cephalosporin exposure was common to the Pseudomonas aeruginosa and CAP‐resistance models. Annual trends in CAP‐resistance and MRSA recovery were not identified. The negative annual trend in Pseudomonas aeruginosa HCAP is unexplained and beyond the scope of this study. The percentage of culture‐positive admissions with Pseudomonas aeruginosa HCAP averaged 12% in 2003‐2006, but dropped to <5% in 2007‐2008. A potential explanation is that identification and isolation of patients with MRSA, as a result of the VA‐wide MRSA initiative, may have impacted Pseudomonas aeruginosa colonization by isolating patients co‐colonized with these pathogens during prior healthcare exposures. This is consistent with the observation that when the cohort‐derived CAP‐resistance model was refit with the Pseudomonas aeruginosa endpoint, recent MRSA colonization was strongly predictive of Pseudomonas aeruginosa. Despite differences between variables in pathogen‐specific and CAP‐resistant models, the CAP‐resistance model provided a similar degree of MRSA and Pseudomonas aeruginosa prediction. Finally, as a study purpose included developing best predictive models for each endpoint, and not merely identifying associations, there were other plausible models not reported.

Study strengths included use of the VISN20 Data Warehouse, which provided an integrated outpatient and inpatient medical record. This facilitated analysis of prior healthcare exposures and inpatient study endpoints. In addition, poor blood and sputum specimens and unlikely pneumonia pathogens were not included in establishing MDR endpoints. The variable set explored in regression modeling was extensive and detailed, and analysis included time and intensity‐based components of the variables. Importantly, a standardized approach to regression modeling was specified in advance, which included identification of variables with high potential for association with MDR endpoints, model selection by AIC, re‐evaluation of guideline‐defined criteria and variables of lower interest, and bootstrapped internal model validation.19

Study limitations included the use of ICD‐9 codes to establish a pneumonia diagnosis, which may lack sensitivity and specificity. However, an enhanced ICD‐9based algorithm superior to other claims‐based definitions of pneumonia was utilized.4, 20 Veterans may have received care at non‐VA facilities impacting identification of all healthcare system exposures preceding admission. Data for microbial endpoints were obtained from sterile and non‐sterile site cultures, and it was not possible to determine if the cultured organisms were truly pathogenic. While pathogen‐specific endpoints were not affected, the use of expert rules in select cases to establish CAP‐resistance may have impacted precision for this endpoint. It is also possible that refitting the cohort‐developed CAP‐resistance model for pathogen‐specific endpoints resulted in optimistic aROC due to model over‐fitting. Finally, the cohort was comprised of elderly males, and caution is warranted in extrapolating the results to other populations.

The predictive ability of the guideline‐defined criteria to identify patients with MDR pathogens has been studied. A prospective observational cohort study of 625 consecutive ICU admissions determined that the guideline‐defined criteriaprior antimicrobial treatment, nursing home residence, and prior hospitalizationwere associated with recovery of MDR colonization.21 Shorr et al., investigating a retrospective cohort of 619 patients with HCAP, reported that recent hospitalization, nursing home residence, hemodialysis, and ICU admission were associated with infections caused by CAP‐resistant organisms.22 This study did not report antimicrobial exposures. Our study complements these studies by evaluating existing HCAP guideline criteria, and identifying specific antibiotic exposure, prior culture data, comorbid illness, and immunosuppressive medications that are predictive of MDR infection.

Studies comparing the bacterial etiology of patients with pneumonia in nursing homes relative to CAP, have demonstrated mixed results in recovery of Gram‐negative MDR pathogens, but generally increased MRSA pneumonia.3 Our study suggests that a nursing home stay in the last 6 months is associated with an increased risk for MRSA, but not Pseudomonas aeruginosa, although this was limited by small sample size. Recent infusion therapy has not been previously reported to be associated with MDR pathogens in an HCAP population. In our study, this criterion was predictive of CAP‐resistance in the cohort‐developed model, but not in conjunction with other variables in the guideline‐defined model. Predictors of pathogen‐specific HCAP are limited to an aforementioned single prior study, which identified recent hospitalization, nursing home residence, and ICU admission as risk factors for MRSA HCAP.22

Many studies have investigated risks for infection with MRSA and Pseudomonas aeruginosa outside of the context of HCAP. Predictor variables in cohort‐developed pathogen‐specific models in our study are known risk factors for colonization or infection with these pathogens. For example, antecedent MRSA colonization has been noted as a strong risk factor for MRSA infection, particularly pneumonia.23, 24 Further, patients with diabetes and inhaled corticosteroid exposure are immunosuppressed and at increased risk for colonization with MRSA.25, 26 Likewise, bronchiolar colonization and corticosteroid exposures are known risk factors for pneumonia due to Pseudomonas aeruginosa.27

Many studies have identified prior antibiotic use as a risk factor for infections caused by MRSA and Pseudomonas aeruginosa. However, this criterion is excessively broad and specific antimicrobial exposures carry different magnitudes of risk. Third generation cephalosporins and anti‐pseudomonal fluoroquinolones are commonly reported antibiotics associated with risk for MRSA infection, whereas 8‐methoxy fluoroquinolones appear not to possess the same effect.2831 Likewise, cephalosporins have been reported as risk factors for MDR Pseudomonas aeruginosa infections.32

Several areas of research involving HCAP MDR risk should be investigated. First, the predictive models developed in our and other studies should be evaluated in larger, more diverse populations to establish generalizability. Second, empirical broad‐spectrum antibiotic therapy in all patients with HCAP results in overtreatment of many patients. To date, no reported models provided optimal performance for selecting empirical therapy for unstable ICU patients with HCAP, and many patients do not receive de‐escalation therapy. Thus, models to identify patients with low probability of MDR pathogens upon admission and to aid in de‐escalation are warranted. Finally, the negative trend in Pseudomonas aeruginosa HCAP requires confirmation and further study.

In conclusion, of the ATS/IDSA guideline‐defined criteria for MDR, nursing home admission, recent hospitalization, and antibiotic exposure were predictive of the recovery of CAP‐resistant organisms. Alternative models primarily based on prior culture data, specific antibiotic exposures, and immunosuppression‐related variables improved predictive performance of HCAP associated with MDR.

Healthcare associated pneumonia (HCAP) is defined as pneumonia that is present upon admission, and occurs in patients that have recently been hospitalized, reside in a nursing home, or have had other recent healthcare exposures. Practice guidelines developed by the American Thoracic Society (ATS) and the Infectious Diseases Society of America (IDSA), recommend strategies for the diagnosis and treatment of patients with HCAP.1 A premise of the guidelines is that recent healthcare exposure places patients at risk for infection due to multi‐drug resistant (MDR) pathogens such as methicillin‐resistant Staphylococcus aureus (MRSA) or Pseudomonas aeruginosa. In addition to criteria utilized to define HCAP, the guidelines state that recent immunosuppression and antibiotic exposure are risk factors for pneumonia due to MDR pathogens. In contrast to the treatment of community‐acquired pneumonia (CAP), the guidelines recommend empirical administration of antibiotics with activity against MRSA and Pseudomonas aeruginosa for all patients with HCAP.

We recently reported that antimicrobial resistance to CAP antibiotics (CAP‐resistance) was identified in one‐third of culture‐positive patients with HCAP.2 Data regarding the predictive ability of the guideline‐defined criteria specific to HCAP are limited.3 Evaluation and potential refinement of the criteria to identify patients at risk for MDR pathogens can aid in making antibiotic‐related treatment decisions.

The purposes of this study are to: 1) develop and validate a model to predict CAP‐resistance among patients with HCAP, and to compare the model's predictive performance to a model that includes traditional guideline‐defined risk factors; and 2) develop models to predict recovery of pathogen‐specific etiology (MRSA and Pseudomonas aeruginosa), and to compare the predictive performance of the pathogen‐specific and CAP‐resistance models.

METHODS

Patients with HCAP who were admitted to 6 Veterans Affairs Medical Centers (VAMC) in the northwestern United States between January 1, 2003 and December 31, 2008 were included in the retrospective cohort study. The cohort was identified utilizing medical records data extracted from the Veterans Integrated Service Network (VISN20) Data Warehouse. The Data Warehouse is a centralized open architecture relational database that houses medical and administrative records data for VISN20 patients. This research complies with all federal guidelines and VAMC policies relative to human subjects and clinical research.

Subjects were identified by the following pneumonia‐related discharge International Classification of Diseases (ICD‐9 CM) codes: 1) a primary diagnosis of 480‐483; 485‐487.0 (pneumonia); or 2) a primary diagnosis of 507.0 (pneumonitis), 518.8 (respiratory failure), or 0.38 (septicemia), and a secondary diagnosis of 480‐483; 485‐487.0.4 Eligibility required that patients received antibiotic therapy for pneumonia within 24 hours of admission, continue inpatient treatment for >24 hours, and meet any of the following guideline‐defined criteria: 1) hospitalization during the preceding 90 days; 2) admission from a nursing home; 3) outpatient or home wound care, outpatient or home infusion therapy, or chronic hemodialysis.1 In addition, patients not meeting guideline‐defined criteria, who had frequent healthcare system exposure, defined as 12 Emergency Department, Medicine, or Surgery clinic visits within 90 days of admission, were also included. Patients were excluded if they were directly transferred from another hospital, or had pneumonia‐related ICD‐9 codes but received inpatient care for pneumonia in a non‐VA hospital.

Study data included medical records for the year prior to admission for HCAP through 30 days afterwards. Data included: demographics; domicile preceding admission; healthcare utilization including diagnosis and procedure codes; inpatient medications administered, and outpatient prescription fills; vital signs; and laboratory test results, including cultures and susceptibilities.

Guideline‐defined criteria for predicting CAP‐resistance were similar to those used to identify the study cohort. Nursing home admission included patients who were directly admitted from a nursing home, skilled nursing facility, or domiciliary. Prior hospitalization 2 days within 90 days was calculated by summing the length of stay for all admissions during the preceding 90 days. Outpatient intravenous therapy, chronic hemodialysis, and wound care therapy was determined from medication administration records and relevant Current Procedural Terminology (CPT) or ICD‐9 procedure codes for care administered within 30 days. Antibiotic exposure was defined as administration of 1 dose of antibiotic during inpatient care, or fill of an outpatient prescription for 1 antibiotic dose within 90 days preceding admission. Immunosuppression was defined as: human immunodeficiency virus (HIV) diagnosis; white blood cell (WBC) count of 2500 cells/mm³ within 30 days of admission; corticosteroid ingestion during prior admission, or outpatient prescription fills for a corticosteroid with quantity sufficient to last 14 days preceding admission; or inpatient ingestion of, or outpatient prescription fills for, transplant or rheumatologic‐related immunosuppressants within 90 days preceding admission.

Additional variables assessed to predict CAP‐resistance were obtained as follows. First, modifications of guideline‐defined criteria were constructed. These included: direct nursing home admission, or recent nursing home stay preceding admission; total days of hospitalization within 90 days preceding admission; specific antibiotic exposures, including dates since last exposure preceding admission; and individual components of the immunosuppression criterion. Other cohort‐developed variables included: demographics; substance use history; chronic comorbidity determined by individual and composite measures of Charlson score; pulmonary disease history (eg, bronchiectasis); type and frequency of outpatient visits; consecutive (2) prescription fills for chronic medications of interest; clinical and surveillance culture results preceding admission; admitting ward; vital signs; and relevant hematology and chemistry labs.5

Sputum, blood, and bronchoscopy‐collected cultures obtained within 48 hours after admission were assessed to determine specimen acceptability. Poor sputum specimens were defined by Gram stain quantitative results indicating >10 epithelial cells (EPI) per low power field (LPF), or in the absence of quantitative results, semi‐quantitative results indicating 2‐4+EPI. Single positive blood cultures with results indicating likely contaminants were considered poor specimens. All bronchoscopy‐obtained specimens were considered acceptable. All cultures classified as poor specimens were excluded, and microbiology results were evaluated for the remaining specimens.2, 6 Organisms thought to represent colonization or contamination were excluded: coagulase‐negative (CN) Staphylococcus, Enterococcus sp, Bacillus sp, Proprionibacterium sp, and Candida sp. Recovery of a potential pneumonia pathogen from 1 acceptable culture constituted a culture‐positive admission.

CAP‐resistance was determined for each isolate. CAP‐resistance was defined as non‐susceptibility to non‐pseudomonal third generation cephalosporins (ceftriaxone or cefotaxime) or non‐pseudomonal 8‐methoxy fluoroquinolones (moxifloxacin, gatifloxacin), the VA preferred agents for treatment of CAP.7 There were differences between facilities in susceptibility reporting criteria; therefore, the following approach was used to determine CAP‐resistance. First, MRSA and Pseudomonas aeruginosa isolates were classified as CAP‐resistant. Second, susceptibility results were directly utilized to determine CAP‐resistance if both antibiotic results were available. Third, if only a surrogate antibiotic from a class was reported, a representative antibiotic consistent with Clinical Laboratory Standards Institute reporting criteria was utilized.8 Finally, expert rules determined CAP‐resistance for select potential pneumonia pathogens (eg, Haemophilus sp) if antibiotic susceptibility results for both cephalosporin and fluoroquinolone classes were not reported.815 Presence of 1 CAP‐resistant isolate resulted in a CAP‐resistant classification for an admission. MRSA and Pseudomonas aeruginosa endpoints were defined in a similar manner. Only the first admission for each patient was utilized in the analysis.

The probability of CAP‐resistance was predicted from guideline‐defined criteria (guideline‐defined model) with logistic regression. Next, non‐guideline variables were classified as high, medium, or low interest for association with CAP‐resistance. Variables were assessed for collinearity. A model of CAP‐resistance was developed from variables of high interest. Guideline‐defined criteria were omitted to allow consideration of more specific measures (eg, specific antibiotic exposures as opposed to receipt of antibiotics within the preceding 90 days) during this stage. Next, guideline‐defined criteria, and subsequently variables of lesser interest, were added in an attempt to improve the model. Annual trends and plausible interactions were considered. Model selection was by Akaike's Information Criterion (AIC).16 To promote model reliability, the final model was required to lack evidence of over‐fitting in bootstrapped internal validation.17 The guideline‐defined and cohort‐developed models were compared by difference in area under receiver operating characteristic (aROC) curves. The model development process was repeated for MRSA and Pseudomonas aeruginosa endpoints. Finally, to determine if the CAP‐resistance model sufficiently predicted pathogen‐specific MDR, the CAP‐resistance model was re‐estimated for MRSA and Pseudomonas aeruginosa endpoints. Statistical analysis was performed with R version 2.10.0 (The R Foundation for Statistical Computing, Vienna, Austria).

RESULTS

The cohort was comprised of 1300 patients with HCAP. Of these, 375 (28.8% [26.4‐31.4]) met culture‐positive criteria for potential pneumonia pathogens. CAP‐resistant organisms were identified in 118 (31.5% [26.8‐36.4]) patients within 48 hours of admission. CAP‐resistant organisms included: MRSA (49.2% [40.4‐58.1]), Pseudomonas aeruginosa (29.5% [21.9‐38.1]), Enterobacteriaceae (11.4% [6.5‐18.0]), Gram‐negative non‐enterics (8.3% [4.2‐14.4]), Streptococcus pneumoniae (1.5% [0.2‐5.4]), and opportunistic organisms (eg, Mycobacterium spp) (8.3% [4.2‐14.4]). Differences in select characteristics and exposures between culture‐positive and culture‐negative admissions, as well as CAP‐resistant and CAP‐sensitive admissions, were evident (Table 1).