The systematic deployment of prediction rules within health systems remains a challenge, although such decision aids have been available for decades.[1, 2] We previously developed and validated a prediction rule for 30‐day mortality in a retrospective cohort, noting that the mortality risk is associated with a number of other clinical events.[3] These relationships suggest risk strata, defined by the predicted probability of 30‐day mortality, and could trigger a number of coordinated care processes proportional to the level of risk.[4] For example, patients within the higher‐risk strata could be considered for placement into an intermediate or intensive care unit (ICU), be monitored more closely by physician and nurse team members for clinical deterioration, be seen by a physician within a few days of hospital discharge, and be considered for advance care planning discussions.[3, 4, 5, 6, 7] Patients within the lower‐risk strata might not need the same intensity of these processes routinely unless some other indication were present.

However attractive this conceptual framework may be, its realization is dependent on the willingness of clinical staff to generate predictions consistently on a substantial portion of the patient population, and on the accuracy of the predictions when the risk factors are determined with some level of uncertainty at the beginning of the hospitalization.[2, 8] Skepticism is justified, because the work involved in completing the prediction rule might be incompatible with existing workflow. A patient might not be scored if the emergency physician lacks time or if technical issues arise with the information system and computation process.[9] There is also a generic concern that the predictions will prove to be less accurate outside of the original study population.[8, 9, 10] A more specific concern for our rule is how well present on admission diagnoses can be determined during the relatively short emergency department or presurgery evaluation period. For example, a final diagnosis of heart failure might not be established until later in the hospitalization, after the results of diagnostic testing and clinical response to treatment are known. Moreover, our retrospective prediction rule requires an assessment of the presence or absence of sepsis and respiratory failure. These diagnoses appear to be susceptible to secular trends in medical record coding practices, suggesting the rule's accuracy might not be stable over time.[11]

We report the feasibility of having emergency physicians and the surgical preparation center team generate mortality predictions before an inpatient bed is assigned. We evaluate and report the accuracy of these prospective predictions.

METHODS

The study population consisted of all patients 18 years of age or less than 100 years who were admitted from the emergency department or assigned an inpatient bed following elective surgery at a tertiary, community teaching hospital in the Midwestern United States from September 1, 2012 through February 15, 2013. Although patients entering the hospital from these 2 pathways would be expected to have different levels of mortality risk, we used the original prediction rule for both because such distinctions were not made in its derivation and validation. Patients were not considered if they were admitted for childbirth or other obstetrical reasons, admitted directly from physician offices, the cardiac catheterization laboratory, hemodialysis unit, or from another hospital. The site institutional review board approved this study.

The implementation process began with presentations to the administrative and medical staff leadership on the accuracy of the retrospectively generated mortality predictions and risk of other adverse events.[3] The chief medical and nursing officers became project champions, secured internal funding for the technical components, and arranged to have 2 project comanagers available. A multidisciplinary task force endorsed the implementation details at biweekly meetings throughout the planning year. The leadership of the emergency department and surgical preparation center committed their colleagues to generate the predictions. The support of the emergency leadership was contingent on the completion of the entire prediction generating process in a very short time (within the time a physician could hold his/her breath). The chief medical officer, with the support of the leadership of the hospitalists and emergency physicians, made the administrative decision that a prediction must be generated prior to the assignment of a hospital room.

During the consensus‐building phase, a Web‐based application was developed to generate the predictions. Emergency physicians and surgical preparation staff were trained on the definitions of the risk factors (see Supporting Information, Appendix, in the online version of this article) and how to use the Web application. Three supporting databases were created. Each midnight, a past medical history database was updated, identifying those who had been discharged from the study hospital in the previous 365 days, and whether or not their diagnoses included atrial fibrillation, leukemia/lymphoma, metastatic cancer, cancer other than leukemia, lymphoma, cognitive disorder, or other neurological conditions (eg, Parkinson's, multiple sclerosis, epilepsy, coma, and stupor). Similarly, a clinical laboratory results database was created and updated real time through an HL7 (Health Level Seven, a standard data exchange format[12]) interface with the laboratory information system for the following tests performed in the preceding 30 days at a hospital‐affiliated facility: hemoglobin, platelet count, white blood count, serum troponin, blood urea nitrogen, serum albumin, serum lactate, arterial pH, arterial partial pressure of oxygen values. The third database, admission‐discharge‐transfer, was created and updated every 15 minutes to identify patients currently in the emergency room or scheduled for surgery. When a patient registration event was added to this database, the Web application created a record, retrieved all relevant data, and displayed the patient name for scoring. When the decision for hospitalization was made, the clinician selected the patient's name and reviewed the pre‐populated medical diagnoses of interest, which could be overwritten based on his/her own assessment (Figure 1A,B). The clinician then indicated (yes, no, or unknown) if the patient currently had or was being treated for each of the following: injury, heart failure, sepsis, respiratory failure, and whether or not the admitting service would be medicine (ie, nonsurgical, nonobstetrical). We considered unknown status to indicate the patient did not have the condition. When laboratory values were not available, a normal value was imputed using a previously developed algorithm.[3] Two additional questions, not used in the current prediction process, were answered to provide data for a future analysis: 1 concerning the change in the patient's condition while in the emergency department and the other concerning the presence of abnormal vital signs. The probability of 30‐day mortality was calculated via the Web application using the risk information supplied and the scoring weights (ie, parameter estimates) provided in the Appendices of our original publication.[3] Predictions were updated every minute as new laboratory values became available, and flagged with an alert if a more severe score resulted.

Figure 1

For the analyses of this study, the last prospective prediction viewed by emergency department personnel, a hospital bed manager, or surgical suite staff prior to arrival on the nursing unit is the one referenced as prospective. Once the patient had been discharged from the hospital, we generated a second mortality prediction based on previously published parameter estimates applied to risk factor data ascertained retrospectively as was done in the original article[3]; we subsequently refer to this prediction as retrospective. We will report on the group of patients who had both prospective and retrospective scores (1 patient had a prospective but not retrospective score available).

The prediction scores were made available to the clinical teams gradually during the study period. All scores were viewable by the midpoint of the study for emergency department admissions and near the end of the study for elective‐surgery patients. Only 2 changes in care processes based on level of risk were introduced during the study period. The first required initial placement of patients having a probability of dying of 0.3 or greater into an intensive or intermediate care unit unless the patient or family requested a less aggressive approach. The second occurred in the final 2 months of the study when a large multispecialty practice began routinely arranging for high‐risk patients to be seen within 3 or 7 days of hospital discharge.

RESULTS

Mortality predictions were generated on demand for 7291 out of 7777 (93.8%) eligible patients admitted from the emergency department, and for 2021 out of 2250 (89.8%) eligible elective surgical cases, for a total of 9312 predictions generated out of a possible 10,027 hospitalizations (92.9%). Table 1 displays the characteristics of the study population. The mean age was 65.2 years and 53.8% were women. The most common risk factors were atrial fibrillation (16.4%) and cancer (14.6%). Orders for a comfort care approach (rather than curative) were entered within 4 hours of admission for 32/9312 patients (0.34%), and 9/9312 (0.1%) were hospice patients on admission.

Evaluation of Prediction Accuracy

The AROC for 30‐day mortality was 0.850 (95% confidence interval [CI]: 0.833‐0.866) for prospectively collected covariates, and 0.870 (95% CI: 0.855‐0.885) for retrospectively determined risk factors. These AROCs are not substantively different from each other, demonstrating comparable prediction performance. Calibration was excellent, as indicated in Figure 2, in which the predicted level of risk lay within the 95% confidence limits of the actual 30‐day mortality for 19 out of 20 intervals of 5 percentile increments.

Figure 2

DISCUSSION

Emergency physicians and surgical preparation center nurses generated predictions by the time of hospital admission for over 90% of the target population during usual workflow, without the addition of staff or resources. The discrimination of the prospectively generated predictions was very good to excellent, with an AROC of 0.850 (95% CI: 0.833‐0.866), similar to that obtained from the retrospective version. Calibration was excellent. The prospectively calculated mortality risk was associated with a number of other events. As shown in Table 3, the differing frequency of events within the risk strata support the development of differing intensities of multidisciplinary strategies according to the level of risk.[5] Our study provides useful experience for others who anticipate generating real‐time predictions. We consider the key reasons for success to be the considerable time spent achieving consensus, the technical development of the Web application, the brief clinician time required for the scoring process, the leadership of the chief medical and nursing officers, and the requirement that a prediction be generated before assignment of a hospital room.

Our study has a number of limitations, some of which were noted in our original publication, and although still relevant, will not be repeated here for space considerations. This is a single‐site study that used a prediction rule developed by the same site, albeit on a patient population 4 to 5 years earlier. It is not known how well the specific rule might perform in other hospital populations; any such use should therefore be accompanied by independent validation studies prior to implementation. Our successful experience should motivate future validation studies. Second, because the prognoses of patients scored from the emergency department are likely to be worse than those of elective surgery patients, our rule should be recalibrated for each subgroup separately. We plan to do this in the near future, as well as consider additional risk factors. Third, the other events of interest might be predicted more accurately if rules specifically developed for each were deployed. The mortality risk by itself is unlikely to be a sufficiently accurate predictor, particularly for complications and intensive care use, for reasons outlined in our original publication.[3] However, the varying levels of events within the higher versus lower strata should be noted by the clinical team as they design their team‐based processes. A follow‐up visit with a physician within a few days of discharge could address the concurrent risk of dying as well as readmission, for example. Finally, it is too early to determine if the availability of mortality predictions from this rule will benefit patients.[2, 8, 10] During the study period, we implemented only 2 new care processes based on the level of risk. This lack of interventions allowed us to evaluate the prediction accuracy with minimal additional confounding, but at the expense of not yet knowing the clinical impact of this work. After the study period, we implemented a number of other interventions and plan on evaluating their effectiveness in the future. We are also considering an evaluation of the potential information gained by updating the predictions throughout the course of the hospitalization.[14]

In conclusion, it is feasible to have a reasonably accurate prediction of mortality risk for most adult patients at the beginning of their hospitalizations. The availability of this prognostic information provides an opportunity to develop proactive care plans for high‐ and low‐risk subsets of patients.

The systematic deployment of prediction rules within health systems remains a challenge, although such decision aids have been available for decades.[1, 2] We previously developed and validated a prediction rule for 30‐day mortality in a retrospective cohort, noting that the mortality risk is associated with a number of other clinical events.[3] These relationships suggest risk strata, defined by the predicted probability of 30‐day mortality, and could trigger a number of coordinated care processes proportional to the level of risk.[4] For example, patients within the higher‐risk strata could be considered for placement into an intermediate or intensive care unit (ICU), be monitored more closely by physician and nurse team members for clinical deterioration, be seen by a physician within a few days of hospital discharge, and be considered for advance care planning discussions.[3, 4, 5, 6, 7] Patients within the lower‐risk strata might not need the same intensity of these processes routinely unless some other indication were present.

However attractive this conceptual framework may be, its realization is dependent on the willingness of clinical staff to generate predictions consistently on a substantial portion of the patient population, and on the accuracy of the predictions when the risk factors are determined with some level of uncertainty at the beginning of the hospitalization.[2, 8] Skepticism is justified, because the work involved in completing the prediction rule might be incompatible with existing workflow. A patient might not be scored if the emergency physician lacks time or if technical issues arise with the information system and computation process.[9] There is also a generic concern that the predictions will prove to be less accurate outside of the original study population.[8, 9, 10] A more specific concern for our rule is how well present on admission diagnoses can be determined during the relatively short emergency department or presurgery evaluation period. For example, a final diagnosis of heart failure might not be established until later in the hospitalization, after the results of diagnostic testing and clinical response to treatment are known. Moreover, our retrospective prediction rule requires an assessment of the presence or absence of sepsis and respiratory failure. These diagnoses appear to be susceptible to secular trends in medical record coding practices, suggesting the rule's accuracy might not be stable over time.[11]

We report the feasibility of having emergency physicians and the surgical preparation center team generate mortality predictions before an inpatient bed is assigned. We evaluate and report the accuracy of these prospective predictions.

METHODS

The study population consisted of all patients 18 years of age or less than 100 years who were admitted from the emergency department or assigned an inpatient bed following elective surgery at a tertiary, community teaching hospital in the Midwestern United States from September 1, 2012 through February 15, 2013. Although patients entering the hospital from these 2 pathways would be expected to have different levels of mortality risk, we used the original prediction rule for both because such distinctions were not made in its derivation and validation. Patients were not considered if they were admitted for childbirth or other obstetrical reasons, admitted directly from physician offices, the cardiac catheterization laboratory, hemodialysis unit, or from another hospital. The site institutional review board approved this study.

The implementation process began with presentations to the administrative and medical staff leadership on the accuracy of the retrospectively generated mortality predictions and risk of other adverse events.[3] The chief medical and nursing officers became project champions, secured internal funding for the technical components, and arranged to have 2 project comanagers available. A multidisciplinary task force endorsed the implementation details at biweekly meetings throughout the planning year. The leadership of the emergency department and surgical preparation center committed their colleagues to generate the predictions. The support of the emergency leadership was contingent on the completion of the entire prediction generating process in a very short time (within the time a physician could hold his/her breath). The chief medical officer, with the support of the leadership of the hospitalists and emergency physicians, made the administrative decision that a prediction must be generated prior to the assignment of a hospital room.

During the consensus‐building phase, a Web‐based application was developed to generate the predictions. Emergency physicians and surgical preparation staff were trained on the definitions of the risk factors (see Supporting Information, Appendix, in the online version of this article) and how to use the Web application. Three supporting databases were created. Each midnight, a past medical history database was updated, identifying those who had been discharged from the study hospital in the previous 365 days, and whether or not their diagnoses included atrial fibrillation, leukemia/lymphoma, metastatic cancer, cancer other than leukemia, lymphoma, cognitive disorder, or other neurological conditions (eg, Parkinson's, multiple sclerosis, epilepsy, coma, and stupor). Similarly, a clinical laboratory results database was created and updated real time through an HL7 (Health Level Seven, a standard data exchange format[12]) interface with the laboratory information system for the following tests performed in the preceding 30 days at a hospital‐affiliated facility: hemoglobin, platelet count, white blood count, serum troponin, blood urea nitrogen, serum albumin, serum lactate, arterial pH, arterial partial pressure of oxygen values. The third database, admission‐discharge‐transfer, was created and updated every 15 minutes to identify patients currently in the emergency room or scheduled for surgery. When a patient registration event was added to this database, the Web application created a record, retrieved all relevant data, and displayed the patient name for scoring. When the decision for hospitalization was made, the clinician selected the patient's name and reviewed the pre‐populated medical diagnoses of interest, which could be overwritten based on his/her own assessment (Figure 1A,B). The clinician then indicated (yes, no, or unknown) if the patient currently had or was being treated for each of the following: injury, heart failure, sepsis, respiratory failure, and whether or not the admitting service would be medicine (ie, nonsurgical, nonobstetrical). We considered unknown status to indicate the patient did not have the condition. When laboratory values were not available, a normal value was imputed using a previously developed algorithm.[3] Two additional questions, not used in the current prediction process, were answered to provide data for a future analysis: 1 concerning the change in the patient's condition while in the emergency department and the other concerning the presence of abnormal vital signs. The probability of 30‐day mortality was calculated via the Web application using the risk information supplied and the scoring weights (ie, parameter estimates) provided in the Appendices of our original publication.[3] Predictions were updated every minute as new laboratory values became available, and flagged with an alert if a more severe score resulted.

Figure 1

For the analyses of this study, the last prospective prediction viewed by emergency department personnel, a hospital bed manager, or surgical suite staff prior to arrival on the nursing unit is the one referenced as prospective. Once the patient had been discharged from the hospital, we generated a second mortality prediction based on previously published parameter estimates applied to risk factor data ascertained retrospectively as was done in the original article[3]; we subsequently refer to this prediction as retrospective. We will report on the group of patients who had both prospective and retrospective scores (1 patient had a prospective but not retrospective score available).

The prediction scores were made available to the clinical teams gradually during the study period. All scores were viewable by the midpoint of the study for emergency department admissions and near the end of the study for elective‐surgery patients. Only 2 changes in care processes based on level of risk were introduced during the study period. The first required initial placement of patients having a probability of dying of 0.3 or greater into an intensive or intermediate care unit unless the patient or family requested a less aggressive approach. The second occurred in the final 2 months of the study when a large multispecialty practice began routinely arranging for high‐risk patients to be seen within 3 or 7 days of hospital discharge.

RESULTS

Mortality predictions were generated on demand for 7291 out of 7777 (93.8%) eligible patients admitted from the emergency department, and for 2021 out of 2250 (89.8%) eligible elective surgical cases, for a total of 9312 predictions generated out of a possible 10,027 hospitalizations (92.9%). Table 1 displays the characteristics of the study population. The mean age was 65.2 years and 53.8% were women. The most common risk factors were atrial fibrillation (16.4%) and cancer (14.6%). Orders for a comfort care approach (rather than curative) were entered within 4 hours of admission for 32/9312 patients (0.34%), and 9/9312 (0.1%) were hospice patients on admission.

Evaluation of Prediction Accuracy

The AROC for 30‐day mortality was 0.850 (95% confidence interval [CI]: 0.833‐0.866) for prospectively collected covariates, and 0.870 (95% CI: 0.855‐0.885) for retrospectively determined risk factors. These AROCs are not substantively different from each other, demonstrating comparable prediction performance. Calibration was excellent, as indicated in Figure 2, in which the predicted level of risk lay within the 95% confidence limits of the actual 30‐day mortality for 19 out of 20 intervals of 5 percentile increments.

Figure 2

DISCUSSION

Emergency physicians and surgical preparation center nurses generated predictions by the time of hospital admission for over 90% of the target population during usual workflow, without the addition of staff or resources. The discrimination of the prospectively generated predictions was very good to excellent, with an AROC of 0.850 (95% CI: 0.833‐0.866), similar to that obtained from the retrospective version. Calibration was excellent. The prospectively calculated mortality risk was associated with a number of other events. As shown in Table 3, the differing frequency of events within the risk strata support the development of differing intensities of multidisciplinary strategies according to the level of risk.[5] Our study provides useful experience for others who anticipate generating real‐time predictions. We consider the key reasons for success to be the considerable time spent achieving consensus, the technical development of the Web application, the brief clinician time required for the scoring process, the leadership of the chief medical and nursing officers, and the requirement that a prediction be generated before assignment of a hospital room.

Our study has a number of limitations, some of which were noted in our original publication, and although still relevant, will not be repeated here for space considerations. This is a single‐site study that used a prediction rule developed by the same site, albeit on a patient population 4 to 5 years earlier. It is not known how well the specific rule might perform in other hospital populations; any such use should therefore be accompanied by independent validation studies prior to implementation. Our successful experience should motivate future validation studies. Second, because the prognoses of patients scored from the emergency department are likely to be worse than those of elective surgery patients, our rule should be recalibrated for each subgroup separately. We plan to do this in the near future, as well as consider additional risk factors. Third, the other events of interest might be predicted more accurately if rules specifically developed for each were deployed. The mortality risk by itself is unlikely to be a sufficiently accurate predictor, particularly for complications and intensive care use, for reasons outlined in our original publication.[3] However, the varying levels of events within the higher versus lower strata should be noted by the clinical team as they design their team‐based processes. A follow‐up visit with a physician within a few days of discharge could address the concurrent risk of dying as well as readmission, for example. Finally, it is too early to determine if the availability of mortality predictions from this rule will benefit patients.[2, 8, 10] During the study period, we implemented only 2 new care processes based on the level of risk. This lack of interventions allowed us to evaluate the prediction accuracy with minimal additional confounding, but at the expense of not yet knowing the clinical impact of this work. After the study period, we implemented a number of other interventions and plan on evaluating their effectiveness in the future. We are also considering an evaluation of the potential information gained by updating the predictions throughout the course of the hospitalization.[14]

In conclusion, it is feasible to have a reasonably accurate prediction of mortality risk for most adult patients at the beginning of their hospitalizations. The availability of this prognostic information provides an opportunity to develop proactive care plans for high‐ and low‐risk subsets of patients.

The systematic deployment of prediction rules within health systems remains a challenge, although such decision aids have been available for decades.[1, 2] We previously developed and validated a prediction rule for 30‐day mortality in a retrospective cohort, noting that the mortality risk is associated with a number of other clinical events.[3] These relationships suggest risk strata, defined by the predicted probability of 30‐day mortality, and could trigger a number of coordinated care processes proportional to the level of risk.[4] For example, patients within the higher‐risk strata could be considered for placement into an intermediate or intensive care unit (ICU), be monitored more closely by physician and nurse team members for clinical deterioration, be seen by a physician within a few days of hospital discharge, and be considered for advance care planning discussions.[3, 4, 5, 6, 7] Patients within the lower‐risk strata might not need the same intensity of these processes routinely unless some other indication were present.

However attractive this conceptual framework may be, its realization is dependent on the willingness of clinical staff to generate predictions consistently on a substantial portion of the patient population, and on the accuracy of the predictions when the risk factors are determined with some level of uncertainty at the beginning of the hospitalization.[2, 8] Skepticism is justified, because the work involved in completing the prediction rule might be incompatible with existing workflow. A patient might not be scored if the emergency physician lacks time or if technical issues arise with the information system and computation process.[9] There is also a generic concern that the predictions will prove to be less accurate outside of the original study population.[8, 9, 10] A more specific concern for our rule is how well present on admission diagnoses can be determined during the relatively short emergency department or presurgery evaluation period. For example, a final diagnosis of heart failure might not be established until later in the hospitalization, after the results of diagnostic testing and clinical response to treatment are known. Moreover, our retrospective prediction rule requires an assessment of the presence or absence of sepsis and respiratory failure. These diagnoses appear to be susceptible to secular trends in medical record coding practices, suggesting the rule's accuracy might not be stable over time.[11]

We report the feasibility of having emergency physicians and the surgical preparation center team generate mortality predictions before an inpatient bed is assigned. We evaluate and report the accuracy of these prospective predictions.

METHODS

The study population consisted of all patients 18 years of age or less than 100 years who were admitted from the emergency department or assigned an inpatient bed following elective surgery at a tertiary, community teaching hospital in the Midwestern United States from September 1, 2012 through February 15, 2013. Although patients entering the hospital from these 2 pathways would be expected to have different levels of mortality risk, we used the original prediction rule for both because such distinctions were not made in its derivation and validation. Patients were not considered if they were admitted for childbirth or other obstetrical reasons, admitted directly from physician offices, the cardiac catheterization laboratory, hemodialysis unit, or from another hospital. The site institutional review board approved this study.

The implementation process began with presentations to the administrative and medical staff leadership on the accuracy of the retrospectively generated mortality predictions and risk of other adverse events.[3] The chief medical and nursing officers became project champions, secured internal funding for the technical components, and arranged to have 2 project comanagers available. A multidisciplinary task force endorsed the implementation details at biweekly meetings throughout the planning year. The leadership of the emergency department and surgical preparation center committed their colleagues to generate the predictions. The support of the emergency leadership was contingent on the completion of the entire prediction generating process in a very short time (within the time a physician could hold his/her breath). The chief medical officer, with the support of the leadership of the hospitalists and emergency physicians, made the administrative decision that a prediction must be generated prior to the assignment of a hospital room.

During the consensus‐building phase, a Web‐based application was developed to generate the predictions. Emergency physicians and surgical preparation staff were trained on the definitions of the risk factors (see Supporting Information, Appendix, in the online version of this article) and how to use the Web application. Three supporting databases were created. Each midnight, a past medical history database was updated, identifying those who had been discharged from the study hospital in the previous 365 days, and whether or not their diagnoses included atrial fibrillation, leukemia/lymphoma, metastatic cancer, cancer other than leukemia, lymphoma, cognitive disorder, or other neurological conditions (eg, Parkinson's, multiple sclerosis, epilepsy, coma, and stupor). Similarly, a clinical laboratory results database was created and updated real time through an HL7 (Health Level Seven, a standard data exchange format[12]) interface with the laboratory information system for the following tests performed in the preceding 30 days at a hospital‐affiliated facility: hemoglobin, platelet count, white blood count, serum troponin, blood urea nitrogen, serum albumin, serum lactate, arterial pH, arterial partial pressure of oxygen values. The third database, admission‐discharge‐transfer, was created and updated every 15 minutes to identify patients currently in the emergency room or scheduled for surgery. When a patient registration event was added to this database, the Web application created a record, retrieved all relevant data, and displayed the patient name for scoring. When the decision for hospitalization was made, the clinician selected the patient's name and reviewed the pre‐populated medical diagnoses of interest, which could be overwritten based on his/her own assessment (Figure 1A,B). The clinician then indicated (yes, no, or unknown) if the patient currently had or was being treated for each of the following: injury, heart failure, sepsis, respiratory failure, and whether or not the admitting service would be medicine (ie, nonsurgical, nonobstetrical). We considered unknown status to indicate the patient did not have the condition. When laboratory values were not available, a normal value was imputed using a previously developed algorithm.[3] Two additional questions, not used in the current prediction process, were answered to provide data for a future analysis: 1 concerning the change in the patient's condition while in the emergency department and the other concerning the presence of abnormal vital signs. The probability of 30‐day mortality was calculated via the Web application using the risk information supplied and the scoring weights (ie, parameter estimates) provided in the Appendices of our original publication.[3] Predictions were updated every minute as new laboratory values became available, and flagged with an alert if a more severe score resulted.

Figure 1

For the analyses of this study, the last prospective prediction viewed by emergency department personnel, a hospital bed manager, or surgical suite staff prior to arrival on the nursing unit is the one referenced as prospective. Once the patient had been discharged from the hospital, we generated a second mortality prediction based on previously published parameter estimates applied to risk factor data ascertained retrospectively as was done in the original article[3]; we subsequently refer to this prediction as retrospective. We will report on the group of patients who had both prospective and retrospective scores (1 patient had a prospective but not retrospective score available).

The prediction scores were made available to the clinical teams gradually during the study period. All scores were viewable by the midpoint of the study for emergency department admissions and near the end of the study for elective‐surgery patients. Only 2 changes in care processes based on level of risk were introduced during the study period. The first required initial placement of patients having a probability of dying of 0.3 or greater into an intensive or intermediate care unit unless the patient or family requested a less aggressive approach. The second occurred in the final 2 months of the study when a large multispecialty practice began routinely arranging for high‐risk patients to be seen within 3 or 7 days of hospital discharge.

RESULTS

Mortality predictions were generated on demand for 7291 out of 7777 (93.8%) eligible patients admitted from the emergency department, and for 2021 out of 2250 (89.8%) eligible elective surgical cases, for a total of 9312 predictions generated out of a possible 10,027 hospitalizations (92.9%). Table 1 displays the characteristics of the study population. The mean age was 65.2 years and 53.8% were women. The most common risk factors were atrial fibrillation (16.4%) and cancer (14.6%). Orders for a comfort care approach (rather than curative) were entered within 4 hours of admission for 32/9312 patients (0.34%), and 9/9312 (0.1%) were hospice patients on admission.

Evaluation of Prediction Accuracy

The AROC for 30‐day mortality was 0.850 (95% confidence interval [CI]: 0.833‐0.866) for prospectively collected covariates, and 0.870 (95% CI: 0.855‐0.885) for retrospectively determined risk factors. These AROCs are not substantively different from each other, demonstrating comparable prediction performance. Calibration was excellent, as indicated in Figure 2, in which the predicted level of risk lay within the 95% confidence limits of the actual 30‐day mortality for 19 out of 20 intervals of 5 percentile increments.

Figure 2

DISCUSSION

Emergency physicians and surgical preparation center nurses generated predictions by the time of hospital admission for over 90% of the target population during usual workflow, without the addition of staff or resources. The discrimination of the prospectively generated predictions was very good to excellent, with an AROC of 0.850 (95% CI: 0.833‐0.866), similar to that obtained from the retrospective version. Calibration was excellent. The prospectively calculated mortality risk was associated with a number of other events. As shown in Table 3, the differing frequency of events within the risk strata support the development of differing intensities of multidisciplinary strategies according to the level of risk.[5] Our study provides useful experience for others who anticipate generating real‐time predictions. We consider the key reasons for success to be the considerable time spent achieving consensus, the technical development of the Web application, the brief clinician time required for the scoring process, the leadership of the chief medical and nursing officers, and the requirement that a prediction be generated before assignment of a hospital room.

Our study has a number of limitations, some of which were noted in our original publication, and although still relevant, will not be repeated here for space considerations. This is a single‐site study that used a prediction rule developed by the same site, albeit on a patient population 4 to 5 years earlier. It is not known how well the specific rule might perform in other hospital populations; any such use should therefore be accompanied by independent validation studies prior to implementation. Our successful experience should motivate future validation studies. Second, because the prognoses of patients scored from the emergency department are likely to be worse than those of elective surgery patients, our rule should be recalibrated for each subgroup separately. We plan to do this in the near future, as well as consider additional risk factors. Third, the other events of interest might be predicted more accurately if rules specifically developed for each were deployed. The mortality risk by itself is unlikely to be a sufficiently accurate predictor, particularly for complications and intensive care use, for reasons outlined in our original publication.[3] However, the varying levels of events within the higher versus lower strata should be noted by the clinical team as they design their team‐based processes. A follow‐up visit with a physician within a few days of discharge could address the concurrent risk of dying as well as readmission, for example. Finally, it is too early to determine if the availability of mortality predictions from this rule will benefit patients.[2, 8, 10] During the study period, we implemented only 2 new care processes based on the level of risk. This lack of interventions allowed us to evaluate the prediction accuracy with minimal additional confounding, but at the expense of not yet knowing the clinical impact of this work. After the study period, we implemented a number of other interventions and plan on evaluating their effectiveness in the future. We are also considering an evaluation of the potential information gained by updating the predictions throughout the course of the hospitalization.[14]

In conclusion, it is feasible to have a reasonably accurate prediction of mortality risk for most adult patients at the beginning of their hospitalizations. The availability of this prognostic information provides an opportunity to develop proactive care plans for high‐ and low‐risk subsets of patients.

Delirium affects up to 82% of critical‐care patients and 29% to 64% of general medical patients, resulting in medical morbidity, decreased function, and mortality.[1, 2, 3, 4, 5] Delirium is associated with specific, adverse hospital outcomes, including falls, aspiration pneumonia, pressure ulcers, and restraint use.[6, 7] As a consequence of delirium, individuals may be transformed from independent and active at the time of admission to requiring skilled nursing supervision at discharge, in some cases resulting in permanent cognitive disability for the remainder of the person's life.[4, 8]

Combative behavior in hospitalized patients can be a threat to self or others, including other patients and staff. An emergency alert to staff regarding the presence of a combative patient requiring intervention is known as a behavioral code or Code Gray.[9] Guidelines addressing this particular hospital emergency typically refer to de‐escalation methods and implementation of security measures. Ideally, however, at‐risk patients would be identified prior to development into a full behavioral code. Unfortunately, the medical literature on the causes, prevention, and interventions for combative behavior requiring intervention is limited. Interventions published to date focus on patients living with severe and persistent mental illness, such as schizophrenia or bipolar disorder.[9, 10, 11, 12]

We hypothesized that delirium contributes to combative behavior in hospitalized patients, leading to the adverse outcome of behavioral codes. Delirium identification would therefore provide an opportunity for prevention and early identification of patients at risk, thereby improving safety for patients and staff. However, no studies published to date address the impact of delirium on the likelihood of a patient hospitalized in a general medical/surgical setting becoming combative and requiring intervention such as a behavioral code. The purpose of this article is to determine the strength of the association between delirium and combative behavior requiring intervention.

METHODS

This study was conducted as part of a quality improvement project, resulting in a waiver by the institutional review board of the hospital. All data with patient‐specific information were securely handled and deidentified prior to analysis. The setting is a 336‐bed, nonuniversity, teaching hospital serving adults in the Pacific Northwest, with approximately 16,000 admissions per year, and 31 critical‐care beds. Delirium prevention has been identified as an institutional priority, and we have been screening for delirium on admission and with twice‐daily Confusion Assessment Method (CAM) scores since 2010.

The study design was a retrospective case control study of hospital inpatients. Consecutive patients experiencing combative behavior requiring a specific behavioral code intervention (n=125) between January 1, 2011 to December 31, 2011 were identified using security reports and operator code logs. Five patients were excluded because the combative behavior requiring intervention occurred prior to hospital admission, in the emergency department or short‐stay observation unit. Interventions for combative behavior are institution specific. At our institution, a behavioral code is called when a patient or visitor is disruptive and exhibiting behavior that, if not controlled immediately, may result in serious injury to self or others, in line with the approach recommended by the Washington State Hospital Association.[9] When this situation arises, a staff member calls the hospital operator and an alert is paged overhead. Security staff then comes to the area to assist the clinical staff in determining the proper response.

The sample of 120 patients with behavioral codes was compared to a control group of 159 inpatients from the same year, randomly selected from all hospital discharges. For both groups, patients under the age of 18 and patients who were cared for only in the emergency department or short stay observation unit were excluded.

The presence or absence of delirium, the primary exposure of interest, was determined using a combined reference standard. First, by institutional protocol, all inpatients underwent nursing administration of the CAM[13] or the CAM for the Intensive Care Unit[14] on admission and every 12 hours thereafter. Patients with a positive CAM score within the 48 hours preceding the behavioral code event in cases, or anytime during the hospitalization in controls, were considered to have delirium. The CAM was performed as part of routine care at our institution by clinical nurses. However, when used outside of the research setting, untrained nurses using the CAM may substantially underestimate the incidence of delirium.[15, 16] Therefore, as a second reference standard, among patients who did not have delirium by the CAM criteria above, chart review was performed to identify delirium. Though chart review is also imperfect for the determination of both delirium and potential confounders, given the limitations of CAM scores in clinical rather than research settings, the use of this combined reference standard improves detection of delirium. Our chart review method is based on the previously reported work by Lakatos et al. and Inouye et al., identifying delirium from key words in the medical record demonstrating the diagnostic criteria for delirium.[6, 16] The Lakatos et al. study provided the specific key words used in our chart review.[6]

For each case and control subject, 1 of 2 experienced chart reviewers abstracted data from the electronic medical record retrospectively. The abstractors were not informed as to whether or not the patients were cases or controls, but could not be blinded as this information may be clear from the medical record. (The chart abstraction tool, with the key words mapped to the Diagnostic and Statistical Manual of Mental Disorders, 4th Edition criteria for delirium and the CAM is included in the Supporting Information, Appendix, in the online version of this article). We assessed interobserver agreement for delirium diagnosis through double review of 20% of charts. The presence or absence of potential confounders, including dementia, substance use or other psychiatric illness, and use of drugs associated with delirium specific (opiates, sedatives, anticholinergics, and antihistamines), and demographic information, including time of admission; hospital length of stay; any intensive care unit (ICU) visit; and discharge disposition was also determined from the electronic medical record.

Bivariate statistics were completed comparing patients with a behavioral code to the control group. Categorical variables were compared using ², and continuous variables were compared using the t test. Logistic regression using stepwise regression, threshold of 0.1, dependent variable of behavioral code, and independent variable of delirium was performed to determine the association of delirium with behavioral codes after adjustment for confounders. In the multivariate model, use of any medication associated with delirium (Table 1) was considered as a single binary variable, regardless of drug type or class. Agreement was assessed through the kappa statistic. All statistics were performed using Stata MP v.12 (StataCorp, College Station, TX).

RESULTS

Patients experiencing combative behavior requiring intervention through a behavioral code were significantly more likely to be male, admitted overnight, require an ICU stay during their hospitalization, and have a diagnosis of dementia or substance‐use disorder (Table 2). Patients with a behavioral code demonstrated an increased hospital length of stay (9.4 vs 4.5 days) and were significantly more likely to be discharged to a skilled nursing facility (31/120, 26% vs 16/159, 10%), or leave against medical advice (10% 12/120 vs 0%, P<0.001). Of the patients leaving against medical advice, none had evidence of delirium or dementia in the record; 92% (11/12) had International Classification of Disease9th Revision codes reflecting alcohol and/or drug abuse or dependence. Exposure to medications commonly associated with delirium was common in both groups, with differences in usage patterns for different drugs (Table 1).

Although combative behavior requiring intervention occurred throughout the stay, the majority (84/120, 70%) of behavioral codes occurred during the first 72 hours of hospital admission. Behavioral codes occurred throughout all nursing shifts, with 27% (32/120) between 7:00 _am and 3:00 _pm, 32% (39/120) between 3:00 _pm and 11:00 _pm, and 41% (49/120) between 11:00 _pm and 7:00 _am. Psychiatric consultation occurred in only 17% (20/120) of patients prior to the behavioral code.

Delirium was evident in the 48 hours preceding the behavioral code event in 50% (60/120) of cases, and was present overall in 16% of the comparison group. In the cases, delirium prior to behavioral code was identified by positive CAM scores in 23% (28/120) and by chart review in 27% (32/120). In the control subjects, delirium was identified by CAM scores in 10% (16/159) and by chart review in 10% (16/159). The chart review delirium assessment demonstrated high interobserver reliability (kappa=0.71). Among patients with behavioral codes, only 28/60 (46.7%) of delirium cases were identified by the CAM score.

The unadjusted odds ratio (OR) for having a behavioral code in the setting of delirium (within 48 hours prior to the code) was 5.1 (95% confidence interval [CI]: 2.9‐8.9, P<0.001), with the combined reference standard of positive CAM or delirium by chart review. Because each reference standard (chart review or CAM) only identified some of the delirium, the OR with only a single reference standard was lower. The risk of behavioral code in the setting of delirium when only CAM scores are considered for the diagnosis of delirium, the OR for behavioral code was 2.7 (95% CI: 1.4‐ 5.3, P=0.003), and when only chart review was used for delirium diagnosis, the OR was 2.7 (95% CI: 1.5‐5.0, P=0.001).

In the stepwise logistic regression model (using the composite reference standard of positive CAM score or delirium on chart review), the odds of having a behavioral code was 3.8 times greater in the setting of delirium (OR: 3.8, 95% CI: 2.07.3, P<0.001), after adjustment for substance abuse (OR: 5.3, 95% CI: 2.810.2, P<0.001), dementia (OR: 6.5, 95% CI: 2.616.1, P<0.001), other mental health diagnosis (OR: 3.2, 95% CI: 1.7‐6.1, P<0.001), and gender (OR male gender: 2.4, 95% CI: 1.34.5, P=0.006). Other potential confounders (age, use of delirium‐associated medications, ICU stay, time of admission) were not significant and so were not included in the final multivariate model.

DISCUSSION

In this study, we identify a 3.8‐fold increased odds of combative behavior requiring a behavioral code intervention in hospitalized patients with delirium. Although a previous association between delirium and restraint use among mechanically ventilated patients in the ICU has been published,[6] this is the first article to describe the strong, statistically significant association between combative patient behavior and delirium in the general medical/surgical acute‐care setting. This association raises the possibility that prevention, early identification, and treatment of delirium in hospitalized patients can decrease the incidence of such combative behavior, and may lead to shorter length of stay and less institutionalization after discharge.

In the multivariate model, we did identify other predictors of combative behavior requiring intervention, including substance abuse, dementia, and other psychiatric diagnoses. Our results do not support age as predictive of combative behavior after adjustment for other predictors. However, our hospital population is relatively old (mean age in controls, 63.9 years). In populations different from ours, age may still be an important predictor. We also did not identify use of medications potentially associated with delirium as a predictor of combative behavior requiring intervention, after adjustment for other predictors. We did, however, consider all medications together, and are not able to differentiate the potential predictive ability of any single drug or class. Finally, we report relatively high rates of opiate and sedative use in our sample, likely because we included short‐acting agents (ie, midazolam, fentanyl) that are commonly used for procedures and perioperative care.

Our study also highlights the challenges of accurately identifying delirium for quality improvement interventions. The CAM is a validated and widely accepted method of prospective screening for delirium, but in relatively untrained hands outside of research settings (as in our institution) does underestimate the true incidence of delirium.[15, 16, 17] Further, both CAM assessment and chart review may underestimate the incidence of hypoactive delirium. In our study, we note that the CAM scores only identified 46.7% of the behavioral code patients with delirium. Chart review also has limitations, identified by Inouye et al.,[16] particularly in the setting of comorbid dementia, high baseline delirium risk, and other comorbid conditions. Comorbidities as defined by APACHE II (Acute Physiology and Chronic Health Evaluation II) score may contribute to false positives, and poor documentation may result in false negatives. Our use of a combined reference standard of CAM assessment and chart review for delirium is supported by the fact that the incidence of delirium we report in the control group is similar to the published literature.[5]

Combative behavior in the hospital setting may be a threat to patients, staff, and visitors, and multiple state hospital associations have called for standardized responses, including the calling of behavioral codes when such circumstances arise.[9] In general, de‐escalation techniques and security measures are sufficient for patients exhibiting combative behavior. However, in cases of delirium‐associated combative behavior, clinical evaluation of root causes and both pharmacological and nonpharmacological interventions, including proactive psychiatric consultation, may also be beneficial.[18, 19, 20, 21] Many nonpharmacological interventions may need to be multicomponent in nature, as has been described previously.[18, 19, 20, 21]

We acknowledge the limitations of this article. First, we identify a strong association between combative behavior requiring intervention and delirium, but cannot prove causality. Second, though the chart reviewers where not told if reviewing cases or controls, we were not able to blind them to information on combative behavior that might have been present in the medical record. The use of unblinded reviewers could lead to overestimation of the presence of delirium in cases, and overestimation of the association between delirium and combative behavior requiring intervention. Third, chart review methods may underestimate the prevalence of dementia, which can confound the diagnosis of delirium. Finally, we defined delirium in the behavioral code cases as occurring in the 48 hours prior to the code event. However, as no such event occurred in the controls, we considered delirium as present if identified any time during the hospitalization. This could potentially lead to underestimation of the true association of delirium with combative behavior requiring intervention. It is also worth noting that many of the identified predictors for combative behavior are also predictors for delirium and may be identifying a subset of combative behavior related to agitated delirium. Overall, however, the strength of the association we report does strongly support identification and treatment of delirium in the context of combative behavior.

In this article, we identify a strong association between delirium and combative behavior requiring intervention, even after adjustment for other predictors. Understanding this association can help providers consider delirium as a potential cause of combative behavior in a medical/surgical setting, beyond behavioral issues associated with community violence, serious mental illness, progressive dementia, or substance use. Overall, therefore, delirium risk assessment, screening, prevention, and early intervention may potentially decrease combative behavior, and contribute to improving patient and staff safety in hospital settings.

Disclosures: C. Craig Blackmore, MD, reports royalties from the Springer Verlag publishing company for a textbook series on evidence‐based imaging, which is not relevant to this article. None of the authors report any conflicts of interest.

Delirium affects up to 82% of critical‐care patients and 29% to 64% of general medical patients, resulting in medical morbidity, decreased function, and mortality.[1, 2, 3, 4, 5] Delirium is associated with specific, adverse hospital outcomes, including falls, aspiration pneumonia, pressure ulcers, and restraint use.[6, 7] As a consequence of delirium, individuals may be transformed from independent and active at the time of admission to requiring skilled nursing supervision at discharge, in some cases resulting in permanent cognitive disability for the remainder of the person's life.[4, 8]

Combative behavior in hospitalized patients can be a threat to self or others, including other patients and staff. An emergency alert to staff regarding the presence of a combative patient requiring intervention is known as a behavioral code or Code Gray.[9] Guidelines addressing this particular hospital emergency typically refer to de‐escalation methods and implementation of security measures. Ideally, however, at‐risk patients would be identified prior to development into a full behavioral code. Unfortunately, the medical literature on the causes, prevention, and interventions for combative behavior requiring intervention is limited. Interventions published to date focus on patients living with severe and persistent mental illness, such as schizophrenia or bipolar disorder.[9, 10, 11, 12]

We hypothesized that delirium contributes to combative behavior in hospitalized patients, leading to the adverse outcome of behavioral codes. Delirium identification would therefore provide an opportunity for prevention and early identification of patients at risk, thereby improving safety for patients and staff. However, no studies published to date address the impact of delirium on the likelihood of a patient hospitalized in a general medical/surgical setting becoming combative and requiring intervention such as a behavioral code. The purpose of this article is to determine the strength of the association between delirium and combative behavior requiring intervention.

METHODS

This study was conducted as part of a quality improvement project, resulting in a waiver by the institutional review board of the hospital. All data with patient‐specific information were securely handled and deidentified prior to analysis. The setting is a 336‐bed, nonuniversity, teaching hospital serving adults in the Pacific Northwest, with approximately 16,000 admissions per year, and 31 critical‐care beds. Delirium prevention has been identified as an institutional priority, and we have been screening for delirium on admission and with twice‐daily Confusion Assessment Method (CAM) scores since 2010.

The study design was a retrospective case control study of hospital inpatients. Consecutive patients experiencing combative behavior requiring a specific behavioral code intervention (n=125) between January 1, 2011 to December 31, 2011 were identified using security reports and operator code logs. Five patients were excluded because the combative behavior requiring intervention occurred prior to hospital admission, in the emergency department or short‐stay observation unit. Interventions for combative behavior are institution specific. At our institution, a behavioral code is called when a patient or visitor is disruptive and exhibiting behavior that, if not controlled immediately, may result in serious injury to self or others, in line with the approach recommended by the Washington State Hospital Association.[9] When this situation arises, a staff member calls the hospital operator and an alert is paged overhead. Security staff then comes to the area to assist the clinical staff in determining the proper response.

The sample of 120 patients with behavioral codes was compared to a control group of 159 inpatients from the same year, randomly selected from all hospital discharges. For both groups, patients under the age of 18 and patients who were cared for only in the emergency department or short stay observation unit were excluded.

The presence or absence of delirium, the primary exposure of interest, was determined using a combined reference standard. First, by institutional protocol, all inpatients underwent nursing administration of the CAM[13] or the CAM for the Intensive Care Unit[14] on admission and every 12 hours thereafter. Patients with a positive CAM score within the 48 hours preceding the behavioral code event in cases, or anytime during the hospitalization in controls, were considered to have delirium. The CAM was performed as part of routine care at our institution by clinical nurses. However, when used outside of the research setting, untrained nurses using the CAM may substantially underestimate the incidence of delirium.[15, 16] Therefore, as a second reference standard, among patients who did not have delirium by the CAM criteria above, chart review was performed to identify delirium. Though chart review is also imperfect for the determination of both delirium and potential confounders, given the limitations of CAM scores in clinical rather than research settings, the use of this combined reference standard improves detection of delirium. Our chart review method is based on the previously reported work by Lakatos et al. and Inouye et al., identifying delirium from key words in the medical record demonstrating the diagnostic criteria for delirium.[6, 16] The Lakatos et al. study provided the specific key words used in our chart review.[6]

For each case and control subject, 1 of 2 experienced chart reviewers abstracted data from the electronic medical record retrospectively. The abstractors were not informed as to whether or not the patients were cases or controls, but could not be blinded as this information may be clear from the medical record. (The chart abstraction tool, with the key words mapped to the Diagnostic and Statistical Manual of Mental Disorders, 4th Edition criteria for delirium and the CAM is included in the Supporting Information, Appendix, in the online version of this article). We assessed interobserver agreement for delirium diagnosis through double review of 20% of charts. The presence or absence of potential confounders, including dementia, substance use or other psychiatric illness, and use of drugs associated with delirium specific (opiates, sedatives, anticholinergics, and antihistamines), and demographic information, including time of admission; hospital length of stay; any intensive care unit (ICU) visit; and discharge disposition was also determined from the electronic medical record.

Bivariate statistics were completed comparing patients with a behavioral code to the control group. Categorical variables were compared using ², and continuous variables were compared using the t test. Logistic regression using stepwise regression, threshold of 0.1, dependent variable of behavioral code, and independent variable of delirium was performed to determine the association of delirium with behavioral codes after adjustment for confounders. In the multivariate model, use of any medication associated with delirium (Table 1) was considered as a single binary variable, regardless of drug type or class. Agreement was assessed through the kappa statistic. All statistics were performed using Stata MP v.12 (StataCorp, College Station, TX).

RESULTS

Patients experiencing combative behavior requiring intervention through a behavioral code were significantly more likely to be male, admitted overnight, require an ICU stay during their hospitalization, and have a diagnosis of dementia or substance‐use disorder (Table 2). Patients with a behavioral code demonstrated an increased hospital length of stay (9.4 vs 4.5 days) and were significantly more likely to be discharged to a skilled nursing facility (31/120, 26% vs 16/159, 10%), or leave against medical advice (10% 12/120 vs 0%, P<0.001). Of the patients leaving against medical advice, none had evidence of delirium or dementia in the record; 92% (11/12) had International Classification of Disease9th Revision codes reflecting alcohol and/or drug abuse or dependence. Exposure to medications commonly associated with delirium was common in both groups, with differences in usage patterns for different drugs (Table 1).

Although combative behavior requiring intervention occurred throughout the stay, the majority (84/120, 70%) of behavioral codes occurred during the first 72 hours of hospital admission. Behavioral codes occurred throughout all nursing shifts, with 27% (32/120) between 7:00 _am and 3:00 _pm, 32% (39/120) between 3:00 _pm and 11:00 _pm, and 41% (49/120) between 11:00 _pm and 7:00 _am. Psychiatric consultation occurred in only 17% (20/120) of patients prior to the behavioral code.

Delirium was evident in the 48 hours preceding the behavioral code event in 50% (60/120) of cases, and was present overall in 16% of the comparison group. In the cases, delirium prior to behavioral code was identified by positive CAM scores in 23% (28/120) and by chart review in 27% (32/120). In the control subjects, delirium was identified by CAM scores in 10% (16/159) and by chart review in 10% (16/159). The chart review delirium assessment demonstrated high interobserver reliability (kappa=0.71). Among patients with behavioral codes, only 28/60 (46.7%) of delirium cases were identified by the CAM score.

The unadjusted odds ratio (OR) for having a behavioral code in the setting of delirium (within 48 hours prior to the code) was 5.1 (95% confidence interval [CI]: 2.9‐8.9, P<0.001), with the combined reference standard of positive CAM or delirium by chart review. Because each reference standard (chart review or CAM) only identified some of the delirium, the OR with only a single reference standard was lower. The risk of behavioral code in the setting of delirium when only CAM scores are considered for the diagnosis of delirium, the OR for behavioral code was 2.7 (95% CI: 1.4‐ 5.3, P=0.003), and when only chart review was used for delirium diagnosis, the OR was 2.7 (95% CI: 1.5‐5.0, P=0.001).

In the stepwise logistic regression model (using the composite reference standard of positive CAM score or delirium on chart review), the odds of having a behavioral code was 3.8 times greater in the setting of delirium (OR: 3.8, 95% CI: 2.07.3, P<0.001), after adjustment for substance abuse (OR: 5.3, 95% CI: 2.810.2, P<0.001), dementia (OR: 6.5, 95% CI: 2.616.1, P<0.001), other mental health diagnosis (OR: 3.2, 95% CI: 1.7‐6.1, P<0.001), and gender (OR male gender: 2.4, 95% CI: 1.34.5, P=0.006). Other potential confounders (age, use of delirium‐associated medications, ICU stay, time of admission) were not significant and so were not included in the final multivariate model.

DISCUSSION

In this study, we identify a 3.8‐fold increased odds of combative behavior requiring a behavioral code intervention in hospitalized patients with delirium. Although a previous association between delirium and restraint use among mechanically ventilated patients in the ICU has been published,[6] this is the first article to describe the strong, statistically significant association between combative patient behavior and delirium in the general medical/surgical acute‐care setting. This association raises the possibility that prevention, early identification, and treatment of delirium in hospitalized patients can decrease the incidence of such combative behavior, and may lead to shorter length of stay and less institutionalization after discharge.

In the multivariate model, we did identify other predictors of combative behavior requiring intervention, including substance abuse, dementia, and other psychiatric diagnoses. Our results do not support age as predictive of combative behavior after adjustment for other predictors. However, our hospital population is relatively old (mean age in controls, 63.9 years). In populations different from ours, age may still be an important predictor. We also did not identify use of medications potentially associated with delirium as a predictor of combative behavior requiring intervention, after adjustment for other predictors. We did, however, consider all medications together, and are not able to differentiate the potential predictive ability of any single drug or class. Finally, we report relatively high rates of opiate and sedative use in our sample, likely because we included short‐acting agents (ie, midazolam, fentanyl) that are commonly used for procedures and perioperative care.

Our study also highlights the challenges of accurately identifying delirium for quality improvement interventions. The CAM is a validated and widely accepted method of prospective screening for delirium, but in relatively untrained hands outside of research settings (as in our institution) does underestimate the true incidence of delirium.[15, 16, 17] Further, both CAM assessment and chart review may underestimate the incidence of hypoactive delirium. In our study, we note that the CAM scores only identified 46.7% of the behavioral code patients with delirium. Chart review also has limitations, identified by Inouye et al.,[16] particularly in the setting of comorbid dementia, high baseline delirium risk, and other comorbid conditions. Comorbidities as defined by APACHE II (Acute Physiology and Chronic Health Evaluation II) score may contribute to false positives, and poor documentation may result in false negatives. Our use of a combined reference standard of CAM assessment and chart review for delirium is supported by the fact that the incidence of delirium we report in the control group is similar to the published literature.[5]

Combative behavior in the hospital setting may be a threat to patients, staff, and visitors, and multiple state hospital associations have called for standardized responses, including the calling of behavioral codes when such circumstances arise.[9] In general, de‐escalation techniques and security measures are sufficient for patients exhibiting combative behavior. However, in cases of delirium‐associated combative behavior, clinical evaluation of root causes and both pharmacological and nonpharmacological interventions, including proactive psychiatric consultation, may also be beneficial.[18, 19, 20, 21] Many nonpharmacological interventions may need to be multicomponent in nature, as has been described previously.[18, 19, 20, 21]

We acknowledge the limitations of this article. First, we identify a strong association between combative behavior requiring intervention and delirium, but cannot prove causality. Second, though the chart reviewers where not told if reviewing cases or controls, we were not able to blind them to information on combative behavior that might have been present in the medical record. The use of unblinded reviewers could lead to overestimation of the presence of delirium in cases, and overestimation of the association between delirium and combative behavior requiring intervention. Third, chart review methods may underestimate the prevalence of dementia, which can confound the diagnosis of delirium. Finally, we defined delirium in the behavioral code cases as occurring in the 48 hours prior to the code event. However, as no such event occurred in the controls, we considered delirium as present if identified any time during the hospitalization. This could potentially lead to underestimation of the true association of delirium with combative behavior requiring intervention. It is also worth noting that many of the identified predictors for combative behavior are also predictors for delirium and may be identifying a subset of combative behavior related to agitated delirium. Overall, however, the strength of the association we report does strongly support identification and treatment of delirium in the context of combative behavior.

In this article, we identify a strong association between delirium and combative behavior requiring intervention, even after adjustment for other predictors. Understanding this association can help providers consider delirium as a potential cause of combative behavior in a medical/surgical setting, beyond behavioral issues associated with community violence, serious mental illness, progressive dementia, or substance use. Overall, therefore, delirium risk assessment, screening, prevention, and early intervention may potentially decrease combative behavior, and contribute to improving patient and staff safety in hospital settings.

Disclosures: C. Craig Blackmore, MD, reports royalties from the Springer Verlag publishing company for a textbook series on evidence‐based imaging, which is not relevant to this article. None of the authors report any conflicts of interest.

Delirium affects up to 82% of critical‐care patients and 29% to 64% of general medical patients, resulting in medical morbidity, decreased function, and mortality.[1, 2, 3, 4, 5] Delirium is associated with specific, adverse hospital outcomes, including falls, aspiration pneumonia, pressure ulcers, and restraint use.[6, 7] As a consequence of delirium, individuals may be transformed from independent and active at the time of admission to requiring skilled nursing supervision at discharge, in some cases resulting in permanent cognitive disability for the remainder of the person's life.[4, 8]

Combative behavior in hospitalized patients can be a threat to self or others, including other patients and staff. An emergency alert to staff regarding the presence of a combative patient requiring intervention is known as a behavioral code or Code Gray.[9] Guidelines addressing this particular hospital emergency typically refer to de‐escalation methods and implementation of security measures. Ideally, however, at‐risk patients would be identified prior to development into a full behavioral code. Unfortunately, the medical literature on the causes, prevention, and interventions for combative behavior requiring intervention is limited. Interventions published to date focus on patients living with severe and persistent mental illness, such as schizophrenia or bipolar disorder.[9, 10, 11, 12]

We hypothesized that delirium contributes to combative behavior in hospitalized patients, leading to the adverse outcome of behavioral codes. Delirium identification would therefore provide an opportunity for prevention and early identification of patients at risk, thereby improving safety for patients and staff. However, no studies published to date address the impact of delirium on the likelihood of a patient hospitalized in a general medical/surgical setting becoming combative and requiring intervention such as a behavioral code. The purpose of this article is to determine the strength of the association between delirium and combative behavior requiring intervention.

METHODS

This study was conducted as part of a quality improvement project, resulting in a waiver by the institutional review board of the hospital. All data with patient‐specific information were securely handled and deidentified prior to analysis. The setting is a 336‐bed, nonuniversity, teaching hospital serving adults in the Pacific Northwest, with approximately 16,000 admissions per year, and 31 critical‐care beds. Delirium prevention has been identified as an institutional priority, and we have been screening for delirium on admission and with twice‐daily Confusion Assessment Method (CAM) scores since 2010.

The study design was a retrospective case control study of hospital inpatients. Consecutive patients experiencing combative behavior requiring a specific behavioral code intervention (n=125) between January 1, 2011 to December 31, 2011 were identified using security reports and operator code logs. Five patients were excluded because the combative behavior requiring intervention occurred prior to hospital admission, in the emergency department or short‐stay observation unit. Interventions for combative behavior are institution specific. At our institution, a behavioral code is called when a patient or visitor is disruptive and exhibiting behavior that, if not controlled immediately, may result in serious injury to self or others, in line with the approach recommended by the Washington State Hospital Association.[9] When this situation arises, a staff member calls the hospital operator and an alert is paged overhead. Security staff then comes to the area to assist the clinical staff in determining the proper response.

The sample of 120 patients with behavioral codes was compared to a control group of 159 inpatients from the same year, randomly selected from all hospital discharges. For both groups, patients under the age of 18 and patients who were cared for only in the emergency department or short stay observation unit were excluded.

The presence or absence of delirium, the primary exposure of interest, was determined using a combined reference standard. First, by institutional protocol, all inpatients underwent nursing administration of the CAM[13] or the CAM for the Intensive Care Unit[14] on admission and every 12 hours thereafter. Patients with a positive CAM score within the 48 hours preceding the behavioral code event in cases, or anytime during the hospitalization in controls, were considered to have delirium. The CAM was performed as part of routine care at our institution by clinical nurses. However, when used outside of the research setting, untrained nurses using the CAM may substantially underestimate the incidence of delirium.[15, 16] Therefore, as a second reference standard, among patients who did not have delirium by the CAM criteria above, chart review was performed to identify delirium. Though chart review is also imperfect for the determination of both delirium and potential confounders, given the limitations of CAM scores in clinical rather than research settings, the use of this combined reference standard improves detection of delirium. Our chart review method is based on the previously reported work by Lakatos et al. and Inouye et al., identifying delirium from key words in the medical record demonstrating the diagnostic criteria for delirium.[6, 16] The Lakatos et al. study provided the specific key words used in our chart review.[6]

For each case and control subject, 1 of 2 experienced chart reviewers abstracted data from the electronic medical record retrospectively. The abstractors were not informed as to whether or not the patients were cases or controls, but could not be blinded as this information may be clear from the medical record. (The chart abstraction tool, with the key words mapped to the Diagnostic and Statistical Manual of Mental Disorders, 4th Edition criteria for delirium and the CAM is included in the Supporting Information, Appendix, in the online version of this article). We assessed interobserver agreement for delirium diagnosis through double review of 20% of charts. The presence or absence of potential confounders, including dementia, substance use or other psychiatric illness, and use of drugs associated with delirium specific (opiates, sedatives, anticholinergics, and antihistamines), and demographic information, including time of admission; hospital length of stay; any intensive care unit (ICU) visit; and discharge disposition was also determined from the electronic medical record.

Bivariate statistics were completed comparing patients with a behavioral code to the control group. Categorical variables were compared using ², and continuous variables were compared using the t test. Logistic regression using stepwise regression, threshold of 0.1, dependent variable of behavioral code, and independent variable of delirium was performed to determine the association of delirium with behavioral codes after adjustment for confounders. In the multivariate model, use of any medication associated with delirium (Table 1) was considered as a single binary variable, regardless of drug type or class. Agreement was assessed through the kappa statistic. All statistics were performed using Stata MP v.12 (StataCorp, College Station, TX).

RESULTS

Patients experiencing combative behavior requiring intervention through a behavioral code were significantly more likely to be male, admitted overnight, require an ICU stay during their hospitalization, and have a diagnosis of dementia or substance‐use disorder (Table 2). Patients with a behavioral code demonstrated an increased hospital length of stay (9.4 vs 4.5 days) and were significantly more likely to be discharged to a skilled nursing facility (31/120, 26% vs 16/159, 10%), or leave against medical advice (10% 12/120 vs 0%, P<0.001). Of the patients leaving against medical advice, none had evidence of delirium or dementia in the record; 92% (11/12) had International Classification of Disease9th Revision codes reflecting alcohol and/or drug abuse or dependence. Exposure to medications commonly associated with delirium was common in both groups, with differences in usage patterns for different drugs (Table 1).

Although combative behavior requiring intervention occurred throughout the stay, the majority (84/120, 70%) of behavioral codes occurred during the first 72 hours of hospital admission. Behavioral codes occurred throughout all nursing shifts, with 27% (32/120) between 7:00 _am and 3:00 _pm, 32% (39/120) between 3:00 _pm and 11:00 _pm, and 41% (49/120) between 11:00 _pm and 7:00 _am. Psychiatric consultation occurred in only 17% (20/120) of patients prior to the behavioral code.

Delirium was evident in the 48 hours preceding the behavioral code event in 50% (60/120) of cases, and was present overall in 16% of the comparison group. In the cases, delirium prior to behavioral code was identified by positive CAM scores in 23% (28/120) and by chart review in 27% (32/120). In the control subjects, delirium was identified by CAM scores in 10% (16/159) and by chart review in 10% (16/159). The chart review delirium assessment demonstrated high interobserver reliability (kappa=0.71). Among patients with behavioral codes, only 28/60 (46.7%) of delirium cases were identified by the CAM score.

The unadjusted odds ratio (OR) for having a behavioral code in the setting of delirium (within 48 hours prior to the code) was 5.1 (95% confidence interval [CI]: 2.9‐8.9, P<0.001), with the combined reference standard of positive CAM or delirium by chart review. Because each reference standard (chart review or CAM) only identified some of the delirium, the OR with only a single reference standard was lower. The risk of behavioral code in the setting of delirium when only CAM scores are considered for the diagnosis of delirium, the OR for behavioral code was 2.7 (95% CI: 1.4‐ 5.3, P=0.003), and when only chart review was used for delirium diagnosis, the OR was 2.7 (95% CI: 1.5‐5.0, P=0.001).

In the stepwise logistic regression model (using the composite reference standard of positive CAM score or delirium on chart review), the odds of having a behavioral code was 3.8 times greater in the setting of delirium (OR: 3.8, 95% CI: 2.07.3, P<0.001), after adjustment for substance abuse (OR: 5.3, 95% CI: 2.810.2, P<0.001), dementia (OR: 6.5, 95% CI: 2.616.1, P<0.001), other mental health diagnosis (OR: 3.2, 95% CI: 1.7‐6.1, P<0.001), and gender (OR male gender: 2.4, 95% CI: 1.34.5, P=0.006). Other potential confounders (age, use of delirium‐associated medications, ICU stay, time of admission) were not significant and so were not included in the final multivariate model.

DISCUSSION

In this study, we identify a 3.8‐fold increased odds of combative behavior requiring a behavioral code intervention in hospitalized patients with delirium. Although a previous association between delirium and restraint use among mechanically ventilated patients in the ICU has been published,[6] this is the first article to describe the strong, statistically significant association between combative patient behavior and delirium in the general medical/surgical acute‐care setting. This association raises the possibility that prevention, early identification, and treatment of delirium in hospitalized patients can decrease the incidence of such combative behavior, and may lead to shorter length of stay and less institutionalization after discharge.

In the multivariate model, we did identify other predictors of combative behavior requiring intervention, including substance abuse, dementia, and other psychiatric diagnoses. Our results do not support age as predictive of combative behavior after adjustment for other predictors. However, our hospital population is relatively old (mean age in controls, 63.9 years). In populations different from ours, age may still be an important predictor. We also did not identify use of medications potentially associated with delirium as a predictor of combative behavior requiring intervention, after adjustment for other predictors. We did, however, consider all medications together, and are not able to differentiate the potential predictive ability of any single drug or class. Finally, we report relatively high rates of opiate and sedative use in our sample, likely because we included short‐acting agents (ie, midazolam, fentanyl) that are commonly used for procedures and perioperative care.

Our study also highlights the challenges of accurately identifying delirium for quality improvement interventions. The CAM is a validated and widely accepted method of prospective screening for delirium, but in relatively untrained hands outside of research settings (as in our institution) does underestimate the true incidence of delirium.[15, 16, 17] Further, both CAM assessment and chart review may underestimate the incidence of hypoactive delirium. In our study, we note that the CAM scores only identified 46.7% of the behavioral code patients with delirium. Chart review also has limitations, identified by Inouye et al.,[16] particularly in the setting of comorbid dementia, high baseline delirium risk, and other comorbid conditions. Comorbidities as defined by APACHE II (Acute Physiology and Chronic Health Evaluation II) score may contribute to false positives, and poor documentation may result in false negatives. Our use of a combined reference standard of CAM assessment and chart review for delirium is supported by the fact that the incidence of delirium we report in the control group is similar to the published literature.[5]

Combative behavior in the hospital setting may be a threat to patients, staff, and visitors, and multiple state hospital associations have called for standardized responses, including the calling of behavioral codes when such circumstances arise.[9] In general, de‐escalation techniques and security measures are sufficient for patients exhibiting combative behavior. However, in cases of delirium‐associated combative behavior, clinical evaluation of root causes and both pharmacological and nonpharmacological interventions, including proactive psychiatric consultation, may also be beneficial.[18, 19, 20, 21] Many nonpharmacological interventions may need to be multicomponent in nature, as has been described previously.[18, 19, 20, 21]

We acknowledge the limitations of this article. First, we identify a strong association between combative behavior requiring intervention and delirium, but cannot prove causality. Second, though the chart reviewers where not told if reviewing cases or controls, we were not able to blind them to information on combative behavior that might have been present in the medical record. The use of unblinded reviewers could lead to overestimation of the presence of delirium in cases, and overestimation of the association between delirium and combative behavior requiring intervention. Third, chart review methods may underestimate the prevalence of dementia, which can confound the diagnosis of delirium. Finally, we defined delirium in the behavioral code cases as occurring in the 48 hours prior to the code event. However, as no such event occurred in the controls, we considered delirium as present if identified any time during the hospitalization. This could potentially lead to underestimation of the true association of delirium with combative behavior requiring intervention. It is also worth noting that many of the identified predictors for combative behavior are also predictors for delirium and may be identifying a subset of combative behavior related to agitated delirium. Overall, however, the strength of the association we report does strongly support identification and treatment of delirium in the context of combative behavior.

In this article, we identify a strong association between delirium and combative behavior requiring intervention, even after adjustment for other predictors. Understanding this association can help providers consider delirium as a potential cause of combative behavior in a medical/surgical setting, beyond behavioral issues associated with community violence, serious mental illness, progressive dementia, or substance use. Overall, therefore, delirium risk assessment, screening, prevention, and early intervention may potentially decrease combative behavior, and contribute to improving patient and staff safety in hospital settings.

Disclosures: C. Craig Blackmore, MD, reports royalties from the Springer Verlag publishing company for a textbook series on evidence‐based imaging, which is not relevant to this article. None of the authors report any conflicts of interest.