Hospital discharge can be hazardous because discontinuity and fragmentation of care increase risks to the patient. Inadequate communication has been identified as a major etiology for errors and adverse events occurring shortly after discharge.1, 2 Another potential result of a failed hospital discharge is patient dissatisfaction. Increased patient involvement in care improves health outcomes, and may improve patient satisfaction.3 To engage patients in their care, healthcare providers must collaborate with patients to coordinate care across settings.

In this study, we sought to determine what patients and their caregivers view as essential elements of a safe and high‐quality discharge process. We developed a survey with a broad range of questions related to the hospital discharge process (see Supporting Information, Appendix A, in the online version of this article). The survey included several questions derived from Project BOOST (Better Outcomes for Older adults through Safe Transitions) discharge care plans.4

METHODS

RESULTS

Survey Results

One hundred percent of patients rated the following items as essential (highest category on Likert Scale): when you need to follow‐up with primary care doctor, warning signs to call primary care doctor, and medicines to continue post‐hospitalization (Figure 1). Patients rated the following items as less important (these items were not unanimously rated as extremely important or essential): treatment you received, medicines you took pre‐hospitalization, importance of bringing all your medicines to follow‐up appointments, and given the side effect of each medication. One hundred percent of patients wanted a lot of explanation (highest category on Likert Scale) about my condition and my test results. Only 39% of patients wanted a lot of explanation about discharge medications. Sixty‐one percent wanted somewhat of an explanation about discharge medications. When asked to choose the most important piece of information, 67.5% of patients chose lifestyle changes. See Figure 1 for the relative importance of the items.

Figure 1

The majority of patients surveyed, 173 (86.5%), wanted verbal discharge instructions with or without written discharge instructions, with only 10.5% requesting only written discharge instructions (P < 0.0001). The majority of patients, 168 (84%), wanted resources to read about their medical condition, with 97 (57%) requesting brochures and 62 (36.9%) requesting Web sites. One hundred percent of patients thought that personal communication between the inpatient provider and the primary care doctor was extremely important or essential.

We identified 4 major themes in our qualitative review of the patients' and proxies' comments: verbal communication, frustration, opacity of system, and too many physicians. Participant quotes related to the 4 major themes are presented in Table 2. Many participants expressed a desire for verbal, rather than written, communication at the time of discharge with their healthcare team; patients particularly requested time for verbal communication with their physician. In the frustration theme, many patients and caregivers expressed frustration that the healthcare team was not carefully listening to them. In the theme of too many physicians, many patients expressed feeling overwhelmed by the number of different doctors involved in their care; particularly at discharge, patients did not know to whom to direct questions. Finally, as part of the opacity of system theme, patient comments included concerns regarding how information will be passed to outside doctors, and that the system of communication is not clear.

Table 2.Major Themes From Open‐Ended Comments With Patient Quotes

Verbal communication
Can we just stop and talk? Everybody is rushing in and out.
I just want my doctor to stop by before I go home and tell me what the plan is.
Sometimes I feel like no one is talking to me. All they do is give me paperwork.
I want my doctors to sit down with me before I leave the hospital and tell me exactly what I need to do so that I don't come back.
I don't want papers, I want people. I want to talk to someone and not read my problems from a sheet of paper.
Frustration
I wonder sometimes if anyone is listening to me I seem to be part of a very elaborate organization that has its own rules and regulations and will not alter its ways.
Why do I have to keep retelling my story? It gets tiring. I wish my story could just be told once.
Too many physicians
I saw lots of doctors during my time here, but I didn't see them again when I was leaving.
I see so many doctors I have no idea who is in charge and who I should direct my questions to.
I feel overwhelmed by the number of doctors I see every time I come into the hospital.
I want my main doctor to talk to me. I get so confused when I hear from more than one doctor.
I miss the days when my primary doctor came in to check on me. He knew exactly what I needed. Now, I meet new people every time I go into the hospital.
Opacity of system
I wonder if all my doctors talk to each other. Sometimes, it seems like they don't.
Who keeps track of all this information? Is there someone who will pass on what happened to me here to the outside world?

DISCUSSION

Discharge is a period of transition from hospital to home that involves a transfer in responsibility from the inpatient care team to the patient and/or caregivers and primary care physician. Ineffective communication, planning, and coordination of care can undermine patient satisfaction, increase adverse events, and contribute to more frequent hospital readmissions.

The patients we surveyed uniformly placed high value on verbal (more than written) communication about discharge care plans. Protected time during the discharge process for hospital staff to provide verbal recommendations to patients, especially about when they should return for follow‐up, warning signs to contact PCP sooner, and medications to continue after discharge, may improve patient satisfaction.

In open‐ended comments, several subjects suggested that physicians should sit down in the patient's room and provide verbal discharge instructions. Although it is well recognized that verbal communication alone has limitations and that providing patients with written instructions remains crucial, verbal reinforcement may highlight the most important instructions.

Interestingly, subjects valued information about lifestyle changes over detailed information about their medications. This may suggest that hospitalized patients are particularly receptive to information about lifestyle changes such as smoking cessation or importance of compliance with medical appointments.

Lastly, patients we surveyed value personal communication between inpatient and outpatient providers. It is plausible that this would improve transitions of care, and previous studies have suggested that direct communication between inpatient and outpatient providers occurs infrequently, with only 20% of primary care providers in 1 study reporting that they are always notified when their patient is being discharged from a hospitalist service.6

The themes that emerged from our open‐ended questions also highlight the importance of direct verbal communication with patients and careful coordination of care with outside physicians. Because patients may be unlikely to fully remember verbal instructions at discharge, providers may consider providing patients and family members with patient‐centered written materials to take home in order to reinforce important self‐care instructions. The patient comments further suggest that patients may be more satisfied, and that discharges may be smoother, if 1 or 2 physicians were always identified to the patients and their caregivers as the leaders of the care team throughout the hospital course and discharge process.

Our study had several limitations. We only surveyed patients on general medicine services, so our findings might not apply to other populations. We did not enroll participants on weekends and holidays; it is possible that this led to some bias in the enrollment of subjects. We also only surveyed patients and/or proxies who could speak and read English, and this was a fairly highly educated population, with almost half having completed 4 years of college. Finally, we relied on participant self‐report for demographic information because we did not have access to the electronic medical record. This study was conducted at 2 large academic medical centers that include resident physicians in the daily care of patients; thus, these results may not be generalizable to other settings.

Effective verbal communication between physicians, outpatient providers, patients, and their caregivers about discharge care plans might improve patients' understanding of their hospitalizations, increase their satisfaction with care, and reduce readmissions. In addition, physicians should recognize that patients value advice about lifestyle interventions that might improve their health, as part of the discharge care plan. Intervention studies are necessary to test these hypotheses in large, diverse populations.

Hospital discharge can be hazardous because discontinuity and fragmentation of care increase risks to the patient. Inadequate communication has been identified as a major etiology for errors and adverse events occurring shortly after discharge.1, 2 Another potential result of a failed hospital discharge is patient dissatisfaction. Increased patient involvement in care improves health outcomes, and may improve patient satisfaction.3 To engage patients in their care, healthcare providers must collaborate with patients to coordinate care across settings.

In this study, we sought to determine what patients and their caregivers view as essential elements of a safe and high‐quality discharge process. We developed a survey with a broad range of questions related to the hospital discharge process (see Supporting Information, Appendix A, in the online version of this article). The survey included several questions derived from Project BOOST (Better Outcomes for Older adults through Safe Transitions) discharge care plans.4

METHODS

RESULTS

Survey Results

One hundred percent of patients rated the following items as essential (highest category on Likert Scale): when you need to follow‐up with primary care doctor, warning signs to call primary care doctor, and medicines to continue post‐hospitalization (Figure 1). Patients rated the following items as less important (these items were not unanimously rated as extremely important or essential): treatment you received, medicines you took pre‐hospitalization, importance of bringing all your medicines to follow‐up appointments, and given the side effect of each medication. One hundred percent of patients wanted a lot of explanation (highest category on Likert Scale) about my condition and my test results. Only 39% of patients wanted a lot of explanation about discharge medications. Sixty‐one percent wanted somewhat of an explanation about discharge medications. When asked to choose the most important piece of information, 67.5% of patients chose lifestyle changes. See Figure 1 for the relative importance of the items.

Figure 1

The majority of patients surveyed, 173 (86.5%), wanted verbal discharge instructions with or without written discharge instructions, with only 10.5% requesting only written discharge instructions (P < 0.0001). The majority of patients, 168 (84%), wanted resources to read about their medical condition, with 97 (57%) requesting brochures and 62 (36.9%) requesting Web sites. One hundred percent of patients thought that personal communication between the inpatient provider and the primary care doctor was extremely important or essential.

We identified 4 major themes in our qualitative review of the patients' and proxies' comments: verbal communication, frustration, opacity of system, and too many physicians. Participant quotes related to the 4 major themes are presented in Table 2. Many participants expressed a desire for verbal, rather than written, communication at the time of discharge with their healthcare team; patients particularly requested time for verbal communication with their physician. In the frustration theme, many patients and caregivers expressed frustration that the healthcare team was not carefully listening to them. In the theme of too many physicians, many patients expressed feeling overwhelmed by the number of different doctors involved in their care; particularly at discharge, patients did not know to whom to direct questions. Finally, as part of the opacity of system theme, patient comments included concerns regarding how information will be passed to outside doctors, and that the system of communication is not clear.

Table 2.Major Themes From Open‐Ended Comments With Patient Quotes

Verbal communication
Can we just stop and talk? Everybody is rushing in and out.
I just want my doctor to stop by before I go home and tell me what the plan is.
Sometimes I feel like no one is talking to me. All they do is give me paperwork.
I want my doctors to sit down with me before I leave the hospital and tell me exactly what I need to do so that I don't come back.
I don't want papers, I want people. I want to talk to someone and not read my problems from a sheet of paper.
Frustration
I wonder sometimes if anyone is listening to me I seem to be part of a very elaborate organization that has its own rules and regulations and will not alter its ways.
Why do I have to keep retelling my story? It gets tiring. I wish my story could just be told once.
Too many physicians
I saw lots of doctors during my time here, but I didn't see them again when I was leaving.
I see so many doctors I have no idea who is in charge and who I should direct my questions to.
I feel overwhelmed by the number of doctors I see every time I come into the hospital.
I want my main doctor to talk to me. I get so confused when I hear from more than one doctor.
I miss the days when my primary doctor came in to check on me. He knew exactly what I needed. Now, I meet new people every time I go into the hospital.
Opacity of system
I wonder if all my doctors talk to each other. Sometimes, it seems like they don't.
Who keeps track of all this information? Is there someone who will pass on what happened to me here to the outside world?

DISCUSSION

Discharge is a period of transition from hospital to home that involves a transfer in responsibility from the inpatient care team to the patient and/or caregivers and primary care physician. Ineffective communication, planning, and coordination of care can undermine patient satisfaction, increase adverse events, and contribute to more frequent hospital readmissions.

The patients we surveyed uniformly placed high value on verbal (more than written) communication about discharge care plans. Protected time during the discharge process for hospital staff to provide verbal recommendations to patients, especially about when they should return for follow‐up, warning signs to contact PCP sooner, and medications to continue after discharge, may improve patient satisfaction.

In open‐ended comments, several subjects suggested that physicians should sit down in the patient's room and provide verbal discharge instructions. Although it is well recognized that verbal communication alone has limitations and that providing patients with written instructions remains crucial, verbal reinforcement may highlight the most important instructions.

Interestingly, subjects valued information about lifestyle changes over detailed information about their medications. This may suggest that hospitalized patients are particularly receptive to information about lifestyle changes such as smoking cessation or importance of compliance with medical appointments.

Lastly, patients we surveyed value personal communication between inpatient and outpatient providers. It is plausible that this would improve transitions of care, and previous studies have suggested that direct communication between inpatient and outpatient providers occurs infrequently, with only 20% of primary care providers in 1 study reporting that they are always notified when their patient is being discharged from a hospitalist service.6

The themes that emerged from our open‐ended questions also highlight the importance of direct verbal communication with patients and careful coordination of care with outside physicians. Because patients may be unlikely to fully remember verbal instructions at discharge, providers may consider providing patients and family members with patient‐centered written materials to take home in order to reinforce important self‐care instructions. The patient comments further suggest that patients may be more satisfied, and that discharges may be smoother, if 1 or 2 physicians were always identified to the patients and their caregivers as the leaders of the care team throughout the hospital course and discharge process.

Our study had several limitations. We only surveyed patients on general medicine services, so our findings might not apply to other populations. We did not enroll participants on weekends and holidays; it is possible that this led to some bias in the enrollment of subjects. We also only surveyed patients and/or proxies who could speak and read English, and this was a fairly highly educated population, with almost half having completed 4 years of college. Finally, we relied on participant self‐report for demographic information because we did not have access to the electronic medical record. This study was conducted at 2 large academic medical centers that include resident physicians in the daily care of patients; thus, these results may not be generalizable to other settings.

Effective verbal communication between physicians, outpatient providers, patients, and their caregivers about discharge care plans might improve patients' understanding of their hospitalizations, increase their satisfaction with care, and reduce readmissions. In addition, physicians should recognize that patients value advice about lifestyle interventions that might improve their health, as part of the discharge care plan. Intervention studies are necessary to test these hypotheses in large, diverse populations.

Hospital discharge can be hazardous because discontinuity and fragmentation of care increase risks to the patient. Inadequate communication has been identified as a major etiology for errors and adverse events occurring shortly after discharge.1, 2 Another potential result of a failed hospital discharge is patient dissatisfaction. Increased patient involvement in care improves health outcomes, and may improve patient satisfaction.3 To engage patients in their care, healthcare providers must collaborate with patients to coordinate care across settings.

In this study, we sought to determine what patients and their caregivers view as essential elements of a safe and high‐quality discharge process. We developed a survey with a broad range of questions related to the hospital discharge process (see Supporting Information, Appendix A, in the online version of this article). The survey included several questions derived from Project BOOST (Better Outcomes for Older adults through Safe Transitions) discharge care plans.4

METHODS

RESULTS

Survey Results

One hundred percent of patients rated the following items as essential (highest category on Likert Scale): when you need to follow‐up with primary care doctor, warning signs to call primary care doctor, and medicines to continue post‐hospitalization (Figure 1). Patients rated the following items as less important (these items were not unanimously rated as extremely important or essential): treatment you received, medicines you took pre‐hospitalization, importance of bringing all your medicines to follow‐up appointments, and given the side effect of each medication. One hundred percent of patients wanted a lot of explanation (highest category on Likert Scale) about my condition and my test results. Only 39% of patients wanted a lot of explanation about discharge medications. Sixty‐one percent wanted somewhat of an explanation about discharge medications. When asked to choose the most important piece of information, 67.5% of patients chose lifestyle changes. See Figure 1 for the relative importance of the items.

Figure 1

The majority of patients surveyed, 173 (86.5%), wanted verbal discharge instructions with or without written discharge instructions, with only 10.5% requesting only written discharge instructions (P < 0.0001). The majority of patients, 168 (84%), wanted resources to read about their medical condition, with 97 (57%) requesting brochures and 62 (36.9%) requesting Web sites. One hundred percent of patients thought that personal communication between the inpatient provider and the primary care doctor was extremely important or essential.

We identified 4 major themes in our qualitative review of the patients' and proxies' comments: verbal communication, frustration, opacity of system, and too many physicians. Participant quotes related to the 4 major themes are presented in Table 2. Many participants expressed a desire for verbal, rather than written, communication at the time of discharge with their healthcare team; patients particularly requested time for verbal communication with their physician. In the frustration theme, many patients and caregivers expressed frustration that the healthcare team was not carefully listening to them. In the theme of too many physicians, many patients expressed feeling overwhelmed by the number of different doctors involved in their care; particularly at discharge, patients did not know to whom to direct questions. Finally, as part of the opacity of system theme, patient comments included concerns regarding how information will be passed to outside doctors, and that the system of communication is not clear.

Table 2.Major Themes From Open‐Ended Comments With Patient Quotes

Verbal communication
Can we just stop and talk? Everybody is rushing in and out.
I just want my doctor to stop by before I go home and tell me what the plan is.
Sometimes I feel like no one is talking to me. All they do is give me paperwork.
I want my doctors to sit down with me before I leave the hospital and tell me exactly what I need to do so that I don't come back.
I don't want papers, I want people. I want to talk to someone and not read my problems from a sheet of paper.
Frustration
I wonder sometimes if anyone is listening to me I seem to be part of a very elaborate organization that has its own rules and regulations and will not alter its ways.
Why do I have to keep retelling my story? It gets tiring. I wish my story could just be told once.
Too many physicians
I saw lots of doctors during my time here, but I didn't see them again when I was leaving.
I see so many doctors I have no idea who is in charge and who I should direct my questions to.
I feel overwhelmed by the number of doctors I see every time I come into the hospital.
I want my main doctor to talk to me. I get so confused when I hear from more than one doctor.
I miss the days when my primary doctor came in to check on me. He knew exactly what I needed. Now, I meet new people every time I go into the hospital.
Opacity of system
I wonder if all my doctors talk to each other. Sometimes, it seems like they don't.
Who keeps track of all this information? Is there someone who will pass on what happened to me here to the outside world?

DISCUSSION

Discharge is a period of transition from hospital to home that involves a transfer in responsibility from the inpatient care team to the patient and/or caregivers and primary care physician. Ineffective communication, planning, and coordination of care can undermine patient satisfaction, increase adverse events, and contribute to more frequent hospital readmissions.

The patients we surveyed uniformly placed high value on verbal (more than written) communication about discharge care plans. Protected time during the discharge process for hospital staff to provide verbal recommendations to patients, especially about when they should return for follow‐up, warning signs to contact PCP sooner, and medications to continue after discharge, may improve patient satisfaction.

In open‐ended comments, several subjects suggested that physicians should sit down in the patient's room and provide verbal discharge instructions. Although it is well recognized that verbal communication alone has limitations and that providing patients with written instructions remains crucial, verbal reinforcement may highlight the most important instructions.

Interestingly, subjects valued information about lifestyle changes over detailed information about their medications. This may suggest that hospitalized patients are particularly receptive to information about lifestyle changes such as smoking cessation or importance of compliance with medical appointments.

Lastly, patients we surveyed value personal communication between inpatient and outpatient providers. It is plausible that this would improve transitions of care, and previous studies have suggested that direct communication between inpatient and outpatient providers occurs infrequently, with only 20% of primary care providers in 1 study reporting that they are always notified when their patient is being discharged from a hospitalist service.6

The themes that emerged from our open‐ended questions also highlight the importance of direct verbal communication with patients and careful coordination of care with outside physicians. Because patients may be unlikely to fully remember verbal instructions at discharge, providers may consider providing patients and family members with patient‐centered written materials to take home in order to reinforce important self‐care instructions. The patient comments further suggest that patients may be more satisfied, and that discharges may be smoother, if 1 or 2 physicians were always identified to the patients and their caregivers as the leaders of the care team throughout the hospital course and discharge process.

Our study had several limitations. We only surveyed patients on general medicine services, so our findings might not apply to other populations. We did not enroll participants on weekends and holidays; it is possible that this led to some bias in the enrollment of subjects. We also only surveyed patients and/or proxies who could speak and read English, and this was a fairly highly educated population, with almost half having completed 4 years of college. Finally, we relied on participant self‐report for demographic information because we did not have access to the electronic medical record. This study was conducted at 2 large academic medical centers that include resident physicians in the daily care of patients; thus, these results may not be generalizable to other settings.

Effective verbal communication between physicians, outpatient providers, patients, and their caregivers about discharge care plans might improve patients' understanding of their hospitalizations, increase their satisfaction with care, and reduce readmissions. In addition, physicians should recognize that patients value advice about lifestyle interventions that might improve their health, as part of the discharge care plan. Intervention studies are necessary to test these hypotheses in large, diverse populations.

Patients in general medicalsurgical wards who experience unplanned transfer to the intensive care unit (ICU) have increased mortality and morbidity.13 Using an externally validated methodology permitting assessment of illness severity and mortality risk among all hospitalized patients,4, 5 we recently documented observed‐to‐expected mortality ratios >3.0 and excess length of stay of 10 days among patients who experienced such transfers.6

It is possible to predict adverse outcomes among monitored patients (eg, patients in the ICU or undergoing continuous electronic monitoring).7, 8 However, prediction of unplanned transfers among medicalsurgical ward patients presents challenges. Data collection (vital signs and laboratory tests) is relatively infrequent. The event rate (3% of hospital admissions) is low, and the rate in narrow time periods (eg, 12 hours) is extremely low: a hospital with 4000 admissions per year might experience 1 unplanned transfer to the ICU every 3 days. Not surprisingly, performance of models suitable for predicting ward patients' need for intensive care within narrow time frames have been disappointing.9 The Modified Early Warning Score (MEWS), has a c‐statistic, or area under the receiver operator characteristic of 0.67,1012 and our own model incorporating 14 laboratory tests, but no vital signs, has excellent performance with respect to predicting inpatient mortality, but poor performance with respect to unplanned transfer.6

In this report, we describe the development and validation of a complex predictive model suitable for use with ward patients. Our objective for this work was to develop a predictive model based on clinical and physiologic data available in real time from a comprehensive electronic medical record (EMR), not a clinically intuitive, manually assigned tool. The outcome of interest was unplanned transfer from the ward to the ICU, or death on the ward in a patient who was full code. This model has been developed as part of a regional effort to decrease preventable mortality in the Northern California Kaiser Permanente Medical Care Program (KPMCP), an integrated healthcare delivery system with 22 hospitals.

MATERIALS AND METHODS

For additional details, see the Supporting Information, Appendices 112, in the online version of this article.

This project was approved by the KPMCP Institutional Board for the Protection of Human Subjects.

The Northern California KPMCP serves a total population of approximately 3.3 million members. All Northern California KPMCP hospitals and clinics employ the same information systems with a common medical record number and can track care covered by the plan but delivered elsewhere. Databases maintained by the KPMCP capture admission and discharge times, admission and discharge diagnoses and procedures (assigned by professional coders), bed histories permitting quantification of intra‐hospital transfers, inter‐hospital transfers, as well as the results of all inpatient and outpatient laboratory tests. In July 2006, the KPMCP began deployment of the EMR developed by Epic Systems Corporation (www.epic. com), which has been adapted for the KPMCP and is known as KP HealthConnect (KPHC) in its hospitals. The last of these 22 hospitals went online in March 2010.

Our setting consisted of 14 hospitals in which the KPHC inpatient EMR had been running for at least 3 months (the KPMCP Antioch, Fremont, Hayward, Manteca, Modesto, Roseville, Sacramento, Santa Clara, San Francisco, Santa Rosa, South Sacramento, South San Francisco, Santa Teresa, and Walnut Creek hospitals). We have described the general characteristics of KPMCP hospitals elsewhere.4, 6 Our initial study population consisted of all patients admitted to these hospitals who met the following criteria: hospitalization began from November 1, 2006 through December 31, 2009; initial hospitalization occurred at a Northern California KPMCP hospital (ie, for inter‐hospital transfers, the first hospital stay occurred within the KPMCP); age 18 years; hospitalization was not for childbirth; and KPHC had been operational at the hospital for at least 3 months.

RESULTS

During the study period, a total of 102,488 patients experienced 145,335 hospitalizations at the study hospitals. We removed 66 patients with 138 hospitalizations for data quality reasons, leaving us with our initial study sample of 102,422 patients whose characteristics are summarized in Table 1. Table 1, in which the unit of analysis is an individual patient, shows that patients who experienced the primary outcome were similar to those patients described in our previous report, in terms of their characteristics on admission as well as in experiencing excess morbidity and mortality.6

Figure 1shows how we developed the analysis cohort, by removing patients with a comfort‐care‐only order placed within 4 hours after admission (369 patients/744 hospitalizations) and patients who were never admitted to the ward or TCU (7,220/10,574). This left a cohort consisting of 94,833 patients who experienced 133,879 hospitalizations spanning a total of 1,079,062 shifts. We then removed shifts where: 1) a patient was not on the ward at the start of a shift, or was on the ward for <4 hours of a shift; 2) the patient had a comfort‐care order in place at the start of the shift; and 3) the patient died and was ineligible to be a case (the patient had a DNR order in place or died in the ICU). The final cohort eligible for sampling consisted of 846,907 shifts, which involved a total of 92,797 patients and 130,627 hospitalizations. There were a total of 4,036 event shifts, which included 3,224 where a patient was transferred from the ward to the ICU, 717 from the TCU to the ICU, and 95 where a patient died on the ward or TCU without a DNR order in place. We then randomly selected 39,782 comparison shifts. Thus, our final cohort for analysis included 4,036 event shifts (1,979 derivation/2,057 validation and 39,782 comparison shifts (19,509/20,273). As a secondary validation, we also applied model coefficients to the 429 event shifts excluded due to the <4‐hour length‐of‐stay requirement.

Figure 1

Table 2 compares event shifts with comparison shifts. In the 24 hours preceding ICU transfer, patients who were subsequently transferred had statistically significant, but not necessarily clinically significant, differences in terms of these variables. However, missing laboratory data were more common, ranging from 18% to 31% of all shifts (we did not incorporate laboratory tests where 35% of the shifts had missing data for that test).

After conducting multiple analyses using the derivation dataset, we developed 24 submodels, a compromise between our finding that primary‐condition‐specific models showed better performance and the fact that we had very few events among patients with certain primary conditions (eg, pericarditis/valvular heart disease), which forced us to create composite categories (eg, a category pooling patients with pericarditis, atherosclerosis, and peripheral vascular disease). Table 3 lists variables included in our final submodels.

Table 4 summarizes key results in the validation dataset. Across all diagnoses, the MEWS(re) had c‐statistic of 0.709 (95% confidence interval, 0.6970.721) in the derivation dataset and 0.698 (0.6860.710) in the validation dataset. In the validation dataset, the MEWS(re) performed best among patients with a set of gastrointestinal diagnoses (c = 0.792; 0.7260.857) and worst among patients with congestive heart failure (0.541; 0.5000.620). In contrast, across all primary conditions, the EMR‐based models had a c‐statistic of 0.845 (0.8260.863) in the derivation dataset and 0.775 (0.7530.797) in the validation dataset. In the validation dataset, the EMR‐based models also performed best among patients with a set of gastrointestinal diagnoses (0.841; 0.7830.897) and worst among patients with congestive heart failure (0.683; 0.6100.755). A negative correlation (R = 0.63) was evident between the number of event shifts in a submodel and the drop in the c‐statistic seen in the validation dataset.

We also compared model performance when our datasets were restricted to 1 randomly selected observation per patient; in these analyses, the total number of event shifts was 3,647 and the number of comparison shifts was 29,052. The c‐statistic for the MEWS(re) in the derivation dataset was 0.709 (0.6940.725); in the validation dataset, it was 0.698 (0.6920.714). The corresponding values for the EMR‐based models were 0.856 (0.8350.877) and 0.780 (0.7560.804). We also tested models in which, instead of dropping shifts with missing vital signs, we imputed missing vital signs to their normal value. The c‐statistic for the EMR‐based model with imputed vital sign values was 0.842 (0.8230.861) in the derivation dataset and 0.773 (0.7520.794) in the validation dataset. Lastly, we applied model coefficients to a dataset consisting of 4,290 randomly selected comparison shifts plus the 429 shifts excluded because of the 4‐hour length‐of‐stay criterion. The c‐statistic for this analysis was 0.756 (0.7030.809).

As a general rule, the EMR‐based models were more than twice as efficient as the MEWS(re). For example, a MEWS(re) threshold of 6 as the trigger for an alarm would identify 15% of all transfers to the ICU, with 34.4 false alarms for each transfer; in contrast, using the EMR‐based approach to identify 15% of all transfers, there were 14.5 false alarms for each transfer. Applied to the entire KPMCP Northern California Region, using the MEWS(re), a total of 52 patients per day would need to be evaluated, but only 22 per day using the EMR‐based approach. If one employed a MEWS(re) threshold of 4, this would lead to identification of 44% of all transfers, with a ratio of 69 false alarms for each transfer; using the EMR, the ratio would be 34 to 1. Across the entire KPMCP, a total of 276 patients per day (or about 19.5 a day per hospital) would need to be evaluated using the MEWS(re), but only 136 (or about 9.5 per hospital per day) using the EMR.

DISCUSSION

Using data from a large hospital cohort, we have developed a predictive model suitable for use in non‐ICU populations cared for in integrated healthcare settings with fully automated EMRs. The overall performance of our model, which incorporates acute physiology, diagnosis, and longitudinal data, is superior to the predictive ability of a model that can be assigned manually. This is not surprising, given that scoring systems such as the MEWS make an explicit tradeoff losing information found in multiple variables in exchange for ease of manual assignment. Currently, the model described in this report is being implemented in a simulated environment, a final safety test prior to piloting real‐time provision of probability estimates to clinicians and nurses. Though not yet ready for real‐time use, it is reasonable for our model to be tested using the KPHC shadow server, since evaluation in a simulated environment constitutes a critical evaluation step prior to deployment for clinical use. We also anticipate further refinement and revalidation to occur as more inpatient data become available in the KPMCP and elsewhere.

A number of limitations to our approach must be emphasized. In developing our models, we determined that, while modeling by clinical condition was important, the study outcome was rare for some primary conditions. In these diagnostic groups, which accounted for 12.5% of the event shifts and 10.6% of the comparison shifts, the c‐statistic in the validation dataset was <0.70. Since all 22 KPMCP hospitals are now online and will generate an additional 150,000 adult hospitalizations per year, we expect to be able to correct this problem prior to deployment of these models for clinical use. Having additional data will permit us to improve model discrimination and thus decrease the evaluation‐to‐detection ratio. In future iterations of these models, more experimentation with grouping of International Classification of Diseases (ICD) codes may be required. The problem of grouping ICD codes is not an easy one to resolve, in that diagnoses in the grouping must share common pathophysiology while having a grouping with a sufficient number of adverse events for stable statistical models.

Ideally, it would have been desirable to employ a more objective measure of deterioration, since the decision to transfer a patient to the ICU is discretionary. However, we have found that key data points needed to define such a measure (eg, vital signs) are not consistently charted when a patient deterioratesthis is not surprising outside the research setting, given that nurses and physicians involved in a transfer may be focusing on caring for the patient rather than immediately charting. Given the complexities of end‐of‐life‐care decision‐making, we could not employ death as the outcome of interest. A related issue is that our model does not differentiate between reasons for needing transfer to the ICU, an issue recently discussed by Bapoje et al.18

Our model does not address an important issue raised by Bapoje et al18 and Litvak, Pronovost, and others,19, 20 namely, whether a patient should have been admitted to a non‐ICU setting in the first place. Our team is currently developing a model for doing exactly this (providing decision support for triage in the emergency department), but discussion of this methodology is outside the scope of this article.

Because of resource and data limitations, our model also does not include newborns, children, women admitted for childbirth, or patients transferred from non‐KPMCP hospitals. However, the approach described here could serve as a starting point for developing models for these other populations.

The generalizability of our model must also be considered. The Northern California KPMCP is unusual in having large electronic databases that include physiologic as well as longitudinal patient data. Many hospitals cannot take advantage of all the methods described here. However, the methods we employed could be modified for use by hospital systems in countries such as Great Britain and Canada, and entities such as the Veterans Administration Hospital System in the United States. The KPMCP population, an insured population with few barriers to access, is healthier than the general population, and some population subsets are underrepresented in our cohort. Practice patterns may also vary. Nonetheless, the model described here could serve as a good starting point for future collaborative studies, and it would be possible to develop models suitable for use by stand‐alone hospitals (eg, recalibrating so that one used a Charlson comorbidity21 score based on present on‐admission codes rather than the COPS).

The need for early detection of patient deterioration has played a major role in the development of rapid response teams, as well as scores such as the MEWS. In particular, entities such as the Institute for Healthcare Improvement have advocated the use of early warning systems.22 However, having a statistically robust model to support an early warning system is only part of the solution, and a number of new challenges must then be addressed. The first is actual electronic deployment. Existing inpatient EMRs were not designed with complex calculations in mind, and we anticipate that some degradation in performance will occur when we test our models using real‐time data capture. As Bapoje et al point out, simply having an alert may be insufficient, since not all transfers are preventable.18 Early warning systems also raise ethical issues (for example, what should be done if an alert leads a clinician to confront the fact that an end‐of‐life‐care discussion needs to occur?). From a research perspective, if one were to formally test the benefits of such models, it would be critical to define outcome measures other than death (which is strongly affected by end‐of‐life‐care decisions) or ICU transfer (which is often desirable).

In conclusion, we have developed an approach for predicting impending physiologic deterioration of hospitalized adults outside the ICU. Our approach illustrates how organizations can take maximal advantage of EMRs in a manner that exceeds meaningful use specifications.23, 24 Our study highlights the possibility of using fully automated EMR data for building and applying sophisticated statistical models in settings other than the highly monitored ICU without the need for additional equipment. It also expands the universe of severity scoring to one in which probability estimates are provided in real time and throughout an entire hospitalization. Model performance will undoubtedly improve over time, as more patient data become available. Although our approach has important limitations, it is suitable for testing using real‐time data in a simulated environment. Such testing would permit identification of unanticipated problems and quantification of the degradation of model performance due to real life factors, such as delays in vital signs charting or EMR system brownouts. It could also serve as the springboard for future collaborative studies, with a broader population base, in which the EMR becomes a tool for care, not just documentation.

Patients in general medicalsurgical wards who experience unplanned transfer to the intensive care unit (ICU) have increased mortality and morbidity.13 Using an externally validated methodology permitting assessment of illness severity and mortality risk among all hospitalized patients,4, 5 we recently documented observed‐to‐expected mortality ratios >3.0 and excess length of stay of 10 days among patients who experienced such transfers.6

It is possible to predict adverse outcomes among monitored patients (eg, patients in the ICU or undergoing continuous electronic monitoring).7, 8 However, prediction of unplanned transfers among medicalsurgical ward patients presents challenges. Data collection (vital signs and laboratory tests) is relatively infrequent. The event rate (3% of hospital admissions) is low, and the rate in narrow time periods (eg, 12 hours) is extremely low: a hospital with 4000 admissions per year might experience 1 unplanned transfer to the ICU every 3 days. Not surprisingly, performance of models suitable for predicting ward patients' need for intensive care within narrow time frames have been disappointing.9 The Modified Early Warning Score (MEWS), has a c‐statistic, or area under the receiver operator characteristic of 0.67,1012 and our own model incorporating 14 laboratory tests, but no vital signs, has excellent performance with respect to predicting inpatient mortality, but poor performance with respect to unplanned transfer.6

In this report, we describe the development and validation of a complex predictive model suitable for use with ward patients. Our objective for this work was to develop a predictive model based on clinical and physiologic data available in real time from a comprehensive electronic medical record (EMR), not a clinically intuitive, manually assigned tool. The outcome of interest was unplanned transfer from the ward to the ICU, or death on the ward in a patient who was full code. This model has been developed as part of a regional effort to decrease preventable mortality in the Northern California Kaiser Permanente Medical Care Program (KPMCP), an integrated healthcare delivery system with 22 hospitals.

MATERIALS AND METHODS

For additional details, see the Supporting Information, Appendices 112, in the online version of this article.

This project was approved by the KPMCP Institutional Board for the Protection of Human Subjects.

The Northern California KPMCP serves a total population of approximately 3.3 million members. All Northern California KPMCP hospitals and clinics employ the same information systems with a common medical record number and can track care covered by the plan but delivered elsewhere. Databases maintained by the KPMCP capture admission and discharge times, admission and discharge diagnoses and procedures (assigned by professional coders), bed histories permitting quantification of intra‐hospital transfers, inter‐hospital transfers, as well as the results of all inpatient and outpatient laboratory tests. In July 2006, the KPMCP began deployment of the EMR developed by Epic Systems Corporation (www.epic. com), which has been adapted for the KPMCP and is known as KP HealthConnect (KPHC) in its hospitals. The last of these 22 hospitals went online in March 2010.

Our setting consisted of 14 hospitals in which the KPHC inpatient EMR had been running for at least 3 months (the KPMCP Antioch, Fremont, Hayward, Manteca, Modesto, Roseville, Sacramento, Santa Clara, San Francisco, Santa Rosa, South Sacramento, South San Francisco, Santa Teresa, and Walnut Creek hospitals). We have described the general characteristics of KPMCP hospitals elsewhere.4, 6 Our initial study population consisted of all patients admitted to these hospitals who met the following criteria: hospitalization began from November 1, 2006 through December 31, 2009; initial hospitalization occurred at a Northern California KPMCP hospital (ie, for inter‐hospital transfers, the first hospital stay occurred within the KPMCP); age 18 years; hospitalization was not for childbirth; and KPHC had been operational at the hospital for at least 3 months.

RESULTS

During the study period, a total of 102,488 patients experienced 145,335 hospitalizations at the study hospitals. We removed 66 patients with 138 hospitalizations for data quality reasons, leaving us with our initial study sample of 102,422 patients whose characteristics are summarized in Table 1. Table 1, in which the unit of analysis is an individual patient, shows that patients who experienced the primary outcome were similar to those patients described in our previous report, in terms of their characteristics on admission as well as in experiencing excess morbidity and mortality.6

Figure 1shows how we developed the analysis cohort, by removing patients with a comfort‐care‐only order placed within 4 hours after admission (369 patients/744 hospitalizations) and patients who were never admitted to the ward or TCU (7,220/10,574). This left a cohort consisting of 94,833 patients who experienced 133,879 hospitalizations spanning a total of 1,079,062 shifts. We then removed shifts where: 1) a patient was not on the ward at the start of a shift, or was on the ward for <4 hours of a shift; 2) the patient had a comfort‐care order in place at the start of the shift; and 3) the patient died and was ineligible to be a case (the patient had a DNR order in place or died in the ICU). The final cohort eligible for sampling consisted of 846,907 shifts, which involved a total of 92,797 patients and 130,627 hospitalizations. There were a total of 4,036 event shifts, which included 3,224 where a patient was transferred from the ward to the ICU, 717 from the TCU to the ICU, and 95 where a patient died on the ward or TCU without a DNR order in place. We then randomly selected 39,782 comparison shifts. Thus, our final cohort for analysis included 4,036 event shifts (1,979 derivation/2,057 validation and 39,782 comparison shifts (19,509/20,273). As a secondary validation, we also applied model coefficients to the 429 event shifts excluded due to the <4‐hour length‐of‐stay requirement.

Figure 1

Table 2 compares event shifts with comparison shifts. In the 24 hours preceding ICU transfer, patients who were subsequently transferred had statistically significant, but not necessarily clinically significant, differences in terms of these variables. However, missing laboratory data were more common, ranging from 18% to 31% of all shifts (we did not incorporate laboratory tests where 35% of the shifts had missing data for that test).

After conducting multiple analyses using the derivation dataset, we developed 24 submodels, a compromise between our finding that primary‐condition‐specific models showed better performance and the fact that we had very few events among patients with certain primary conditions (eg, pericarditis/valvular heart disease), which forced us to create composite categories (eg, a category pooling patients with pericarditis, atherosclerosis, and peripheral vascular disease). Table 3 lists variables included in our final submodels.

Table 4 summarizes key results in the validation dataset. Across all diagnoses, the MEWS(re) had c‐statistic of 0.709 (95% confidence interval, 0.6970.721) in the derivation dataset and 0.698 (0.6860.710) in the validation dataset. In the validation dataset, the MEWS(re) performed best among patients with a set of gastrointestinal diagnoses (c = 0.792; 0.7260.857) and worst among patients with congestive heart failure (0.541; 0.5000.620). In contrast, across all primary conditions, the EMR‐based models had a c‐statistic of 0.845 (0.8260.863) in the derivation dataset and 0.775 (0.7530.797) in the validation dataset. In the validation dataset, the EMR‐based models also performed best among patients with a set of gastrointestinal diagnoses (0.841; 0.7830.897) and worst among patients with congestive heart failure (0.683; 0.6100.755). A negative correlation (R = 0.63) was evident between the number of event shifts in a submodel and the drop in the c‐statistic seen in the validation dataset.

We also compared model performance when our datasets were restricted to 1 randomly selected observation per patient; in these analyses, the total number of event shifts was 3,647 and the number of comparison shifts was 29,052. The c‐statistic for the MEWS(re) in the derivation dataset was 0.709 (0.6940.725); in the validation dataset, it was 0.698 (0.6920.714). The corresponding values for the EMR‐based models were 0.856 (0.8350.877) and 0.780 (0.7560.804). We also tested models in which, instead of dropping shifts with missing vital signs, we imputed missing vital signs to their normal value. The c‐statistic for the EMR‐based model with imputed vital sign values was 0.842 (0.8230.861) in the derivation dataset and 0.773 (0.7520.794) in the validation dataset. Lastly, we applied model coefficients to a dataset consisting of 4,290 randomly selected comparison shifts plus the 429 shifts excluded because of the 4‐hour length‐of‐stay criterion. The c‐statistic for this analysis was 0.756 (0.7030.809).

As a general rule, the EMR‐based models were more than twice as efficient as the MEWS(re). For example, a MEWS(re) threshold of 6 as the trigger for an alarm would identify 15% of all transfers to the ICU, with 34.4 false alarms for each transfer; in contrast, using the EMR‐based approach to identify 15% of all transfers, there were 14.5 false alarms for each transfer. Applied to the entire KPMCP Northern California Region, using the MEWS(re), a total of 52 patients per day would need to be evaluated, but only 22 per day using the EMR‐based approach. If one employed a MEWS(re) threshold of 4, this would lead to identification of 44% of all transfers, with a ratio of 69 false alarms for each transfer; using the EMR, the ratio would be 34 to 1. Across the entire KPMCP, a total of 276 patients per day (or about 19.5 a day per hospital) would need to be evaluated using the MEWS(re), but only 136 (or about 9.5 per hospital per day) using the EMR.

DISCUSSION

Using data from a large hospital cohort, we have developed a predictive model suitable for use in non‐ICU populations cared for in integrated healthcare settings with fully automated EMRs. The overall performance of our model, which incorporates acute physiology, diagnosis, and longitudinal data, is superior to the predictive ability of a model that can be assigned manually. This is not surprising, given that scoring systems such as the MEWS make an explicit tradeoff losing information found in multiple variables in exchange for ease of manual assignment. Currently, the model described in this report is being implemented in a simulated environment, a final safety test prior to piloting real‐time provision of probability estimates to clinicians and nurses. Though not yet ready for real‐time use, it is reasonable for our model to be tested using the KPHC shadow server, since evaluation in a simulated environment constitutes a critical evaluation step prior to deployment for clinical use. We also anticipate further refinement and revalidation to occur as more inpatient data become available in the KPMCP and elsewhere.

A number of limitations to our approach must be emphasized. In developing our models, we determined that, while modeling by clinical condition was important, the study outcome was rare for some primary conditions. In these diagnostic groups, which accounted for 12.5% of the event shifts and 10.6% of the comparison shifts, the c‐statistic in the validation dataset was <0.70. Since all 22 KPMCP hospitals are now online and will generate an additional 150,000 adult hospitalizations per year, we expect to be able to correct this problem prior to deployment of these models for clinical use. Having additional data will permit us to improve model discrimination and thus decrease the evaluation‐to‐detection ratio. In future iterations of these models, more experimentation with grouping of International Classification of Diseases (ICD) codes may be required. The problem of grouping ICD codes is not an easy one to resolve, in that diagnoses in the grouping must share common pathophysiology while having a grouping with a sufficient number of adverse events for stable statistical models.

Ideally, it would have been desirable to employ a more objective measure of deterioration, since the decision to transfer a patient to the ICU is discretionary. However, we have found that key data points needed to define such a measure (eg, vital signs) are not consistently charted when a patient deterioratesthis is not surprising outside the research setting, given that nurses and physicians involved in a transfer may be focusing on caring for the patient rather than immediately charting. Given the complexities of end‐of‐life‐care decision‐making, we could not employ death as the outcome of interest. A related issue is that our model does not differentiate between reasons for needing transfer to the ICU, an issue recently discussed by Bapoje et al.18

Our model does not address an important issue raised by Bapoje et al18 and Litvak, Pronovost, and others,19, 20 namely, whether a patient should have been admitted to a non‐ICU setting in the first place. Our team is currently developing a model for doing exactly this (providing decision support for triage in the emergency department), but discussion of this methodology is outside the scope of this article.

Because of resource and data limitations, our model also does not include newborns, children, women admitted for childbirth, or patients transferred from non‐KPMCP hospitals. However, the approach described here could serve as a starting point for developing models for these other populations.

The generalizability of our model must also be considered. The Northern California KPMCP is unusual in having large electronic databases that include physiologic as well as longitudinal patient data. Many hospitals cannot take advantage of all the methods described here. However, the methods we employed could be modified for use by hospital systems in countries such as Great Britain and Canada, and entities such as the Veterans Administration Hospital System in the United States. The KPMCP population, an insured population with few barriers to access, is healthier than the general population, and some population subsets are underrepresented in our cohort. Practice patterns may also vary. Nonetheless, the model described here could serve as a good starting point for future collaborative studies, and it would be possible to develop models suitable for use by stand‐alone hospitals (eg, recalibrating so that one used a Charlson comorbidity21 score based on present on‐admission codes rather than the COPS).

The need for early detection of patient deterioration has played a major role in the development of rapid response teams, as well as scores such as the MEWS. In particular, entities such as the Institute for Healthcare Improvement have advocated the use of early warning systems.22 However, having a statistically robust model to support an early warning system is only part of the solution, and a number of new challenges must then be addressed. The first is actual electronic deployment. Existing inpatient EMRs were not designed with complex calculations in mind, and we anticipate that some degradation in performance will occur when we test our models using real‐time data capture. As Bapoje et al point out, simply having an alert may be insufficient, since not all transfers are preventable.18 Early warning systems also raise ethical issues (for example, what should be done if an alert leads a clinician to confront the fact that an end‐of‐life‐care discussion needs to occur?). From a research perspective, if one were to formally test the benefits of such models, it would be critical to define outcome measures other than death (which is strongly affected by end‐of‐life‐care decisions) or ICU transfer (which is often desirable).

In conclusion, we have developed an approach for predicting impending physiologic deterioration of hospitalized adults outside the ICU. Our approach illustrates how organizations can take maximal advantage of EMRs in a manner that exceeds meaningful use specifications.23, 24 Our study highlights the possibility of using fully automated EMR data for building and applying sophisticated statistical models in settings other than the highly monitored ICU without the need for additional equipment. It also expands the universe of severity scoring to one in which probability estimates are provided in real time and throughout an entire hospitalization. Model performance will undoubtedly improve over time, as more patient data become available. Although our approach has important limitations, it is suitable for testing using real‐time data in a simulated environment. Such testing would permit identification of unanticipated problems and quantification of the degradation of model performance due to real life factors, such as delays in vital signs charting or EMR system brownouts. It could also serve as the springboard for future collaborative studies, with a broader population base, in which the EMR becomes a tool for care, not just documentation.

Patients in general medicalsurgical wards who experience unplanned transfer to the intensive care unit (ICU) have increased mortality and morbidity.13 Using an externally validated methodology permitting assessment of illness severity and mortality risk among all hospitalized patients,4, 5 we recently documented observed‐to‐expected mortality ratios >3.0 and excess length of stay of 10 days among patients who experienced such transfers.6

It is possible to predict adverse outcomes among monitored patients (eg, patients in the ICU or undergoing continuous electronic monitoring).7, 8 However, prediction of unplanned transfers among medicalsurgical ward patients presents challenges. Data collection (vital signs and laboratory tests) is relatively infrequent. The event rate (3% of hospital admissions) is low, and the rate in narrow time periods (eg, 12 hours) is extremely low: a hospital with 4000 admissions per year might experience 1 unplanned transfer to the ICU every 3 days. Not surprisingly, performance of models suitable for predicting ward patients' need for intensive care within narrow time frames have been disappointing.9 The Modified Early Warning Score (MEWS), has a c‐statistic, or area under the receiver operator characteristic of 0.67,1012 and our own model incorporating 14 laboratory tests, but no vital signs, has excellent performance with respect to predicting inpatient mortality, but poor performance with respect to unplanned transfer.6

In this report, we describe the development and validation of a complex predictive model suitable for use with ward patients. Our objective for this work was to develop a predictive model based on clinical and physiologic data available in real time from a comprehensive electronic medical record (EMR), not a clinically intuitive, manually assigned tool. The outcome of interest was unplanned transfer from the ward to the ICU, or death on the ward in a patient who was full code. This model has been developed as part of a regional effort to decrease preventable mortality in the Northern California Kaiser Permanente Medical Care Program (KPMCP), an integrated healthcare delivery system with 22 hospitals.

MATERIALS AND METHODS

For additional details, see the Supporting Information, Appendices 112, in the online version of this article.

This project was approved by the KPMCP Institutional Board for the Protection of Human Subjects.

The Northern California KPMCP serves a total population of approximately 3.3 million members. All Northern California KPMCP hospitals and clinics employ the same information systems with a common medical record number and can track care covered by the plan but delivered elsewhere. Databases maintained by the KPMCP capture admission and discharge times, admission and discharge diagnoses and procedures (assigned by professional coders), bed histories permitting quantification of intra‐hospital transfers, inter‐hospital transfers, as well as the results of all inpatient and outpatient laboratory tests. In July 2006, the KPMCP began deployment of the EMR developed by Epic Systems Corporation (www.epic. com), which has been adapted for the KPMCP and is known as KP HealthConnect (KPHC) in its hospitals. The last of these 22 hospitals went online in March 2010.

Our setting consisted of 14 hospitals in which the KPHC inpatient EMR had been running for at least 3 months (the KPMCP Antioch, Fremont, Hayward, Manteca, Modesto, Roseville, Sacramento, Santa Clara, San Francisco, Santa Rosa, South Sacramento, South San Francisco, Santa Teresa, and Walnut Creek hospitals). We have described the general characteristics of KPMCP hospitals elsewhere.4, 6 Our initial study population consisted of all patients admitted to these hospitals who met the following criteria: hospitalization began from November 1, 2006 through December 31, 2009; initial hospitalization occurred at a Northern California KPMCP hospital (ie, for inter‐hospital transfers, the first hospital stay occurred within the KPMCP); age 18 years; hospitalization was not for childbirth; and KPHC had been operational at the hospital for at least 3 months.

RESULTS

During the study period, a total of 102,488 patients experienced 145,335 hospitalizations at the study hospitals. We removed 66 patients with 138 hospitalizations for data quality reasons, leaving us with our initial study sample of 102,422 patients whose characteristics are summarized in Table 1. Table 1, in which the unit of analysis is an individual patient, shows that patients who experienced the primary outcome were similar to those patients described in our previous report, in terms of their characteristics on admission as well as in experiencing excess morbidity and mortality.6

Figure 1shows how we developed the analysis cohort, by removing patients with a comfort‐care‐only order placed within 4 hours after admission (369 patients/744 hospitalizations) and patients who were never admitted to the ward or TCU (7,220/10,574). This left a cohort consisting of 94,833 patients who experienced 133,879 hospitalizations spanning a total of 1,079,062 shifts. We then removed shifts where: 1) a patient was not on the ward at the start of a shift, or was on the ward for <4 hours of a shift; 2) the patient had a comfort‐care order in place at the start of the shift; and 3) the patient died and was ineligible to be a case (the patient had a DNR order in place or died in the ICU). The final cohort eligible for sampling consisted of 846,907 shifts, which involved a total of 92,797 patients and 130,627 hospitalizations. There were a total of 4,036 event shifts, which included 3,224 where a patient was transferred from the ward to the ICU, 717 from the TCU to the ICU, and 95 where a patient died on the ward or TCU without a DNR order in place. We then randomly selected 39,782 comparison shifts. Thus, our final cohort for analysis included 4,036 event shifts (1,979 derivation/2,057 validation and 39,782 comparison shifts (19,509/20,273). As a secondary validation, we also applied model coefficients to the 429 event shifts excluded due to the <4‐hour length‐of‐stay requirement.

Figure 1

Table 2 compares event shifts with comparison shifts. In the 24 hours preceding ICU transfer, patients who were subsequently transferred had statistically significant, but not necessarily clinically significant, differences in terms of these variables. However, missing laboratory data were more common, ranging from 18% to 31% of all shifts (we did not incorporate laboratory tests where 35% of the shifts had missing data for that test).

After conducting multiple analyses using the derivation dataset, we developed 24 submodels, a compromise between our finding that primary‐condition‐specific models showed better performance and the fact that we had very few events among patients with certain primary conditions (eg, pericarditis/valvular heart disease), which forced us to create composite categories (eg, a category pooling patients with pericarditis, atherosclerosis, and peripheral vascular disease). Table 3 lists variables included in our final submodels.

Table 4 summarizes key results in the validation dataset. Across all diagnoses, the MEWS(re) had c‐statistic of 0.709 (95% confidence interval, 0.6970.721) in the derivation dataset and 0.698 (0.6860.710) in the validation dataset. In the validation dataset, the MEWS(re) performed best among patients with a set of gastrointestinal diagnoses (c = 0.792; 0.7260.857) and worst among patients with congestive heart failure (0.541; 0.5000.620). In contrast, across all primary conditions, the EMR‐based models had a c‐statistic of 0.845 (0.8260.863) in the derivation dataset and 0.775 (0.7530.797) in the validation dataset. In the validation dataset, the EMR‐based models also performed best among patients with a set of gastrointestinal diagnoses (0.841; 0.7830.897) and worst among patients with congestive heart failure (0.683; 0.6100.755). A negative correlation (R = 0.63) was evident between the number of event shifts in a submodel and the drop in the c‐statistic seen in the validation dataset.

We also compared model performance when our datasets were restricted to 1 randomly selected observation per patient; in these analyses, the total number of event shifts was 3,647 and the number of comparison shifts was 29,052. The c‐statistic for the MEWS(re) in the derivation dataset was 0.709 (0.6940.725); in the validation dataset, it was 0.698 (0.6920.714). The corresponding values for the EMR‐based models were 0.856 (0.8350.877) and 0.780 (0.7560.804). We also tested models in which, instead of dropping shifts with missing vital signs, we imputed missing vital signs to their normal value. The c‐statistic for the EMR‐based model with imputed vital sign values was 0.842 (0.8230.861) in the derivation dataset and 0.773 (0.7520.794) in the validation dataset. Lastly, we applied model coefficients to a dataset consisting of 4,290 randomly selected comparison shifts plus the 429 shifts excluded because of the 4‐hour length‐of‐stay criterion. The c‐statistic for this analysis was 0.756 (0.7030.809).

As a general rule, the EMR‐based models were more than twice as efficient as the MEWS(re). For example, a MEWS(re) threshold of 6 as the trigger for an alarm would identify 15% of all transfers to the ICU, with 34.4 false alarms for each transfer; in contrast, using the EMR‐based approach to identify 15% of all transfers, there were 14.5 false alarms for each transfer. Applied to the entire KPMCP Northern California Region, using the MEWS(re), a total of 52 patients per day would need to be evaluated, but only 22 per day using the EMR‐based approach. If one employed a MEWS(re) threshold of 4, this would lead to identification of 44% of all transfers, with a ratio of 69 false alarms for each transfer; using the EMR, the ratio would be 34 to 1. Across the entire KPMCP, a total of 276 patients per day (or about 19.5 a day per hospital) would need to be evaluated using the MEWS(re), but only 136 (or about 9.5 per hospital per day) using the EMR.

DISCUSSION

Using data from a large hospital cohort, we have developed a predictive model suitable for use in non‐ICU populations cared for in integrated healthcare settings with fully automated EMRs. The overall performance of our model, which incorporates acute physiology, diagnosis, and longitudinal data, is superior to the predictive ability of a model that can be assigned manually. This is not surprising, given that scoring systems such as the MEWS make an explicit tradeoff losing information found in multiple variables in exchange for ease of manual assignment. Currently, the model described in this report is being implemented in a simulated environment, a final safety test prior to piloting real‐time provision of probability estimates to clinicians and nurses. Though not yet ready for real‐time use, it is reasonable for our model to be tested using the KPHC shadow server, since evaluation in a simulated environment constitutes a critical evaluation step prior to deployment for clinical use. We also anticipate further refinement and revalidation to occur as more inpatient data become available in the KPMCP and elsewhere.

A number of limitations to our approach must be emphasized. In developing our models, we determined that, while modeling by clinical condition was important, the study outcome was rare for some primary conditions. In these diagnostic groups, which accounted for 12.5% of the event shifts and 10.6% of the comparison shifts, the c‐statistic in the validation dataset was <0.70. Since all 22 KPMCP hospitals are now online and will generate an additional 150,000 adult hospitalizations per year, we expect to be able to correct this problem prior to deployment of these models for clinical use. Having additional data will permit us to improve model discrimination and thus decrease the evaluation‐to‐detection ratio. In future iterations of these models, more experimentation with grouping of International Classification of Diseases (ICD) codes may be required. The problem of grouping ICD codes is not an easy one to resolve, in that diagnoses in the grouping must share common pathophysiology while having a grouping with a sufficient number of adverse events for stable statistical models.

Ideally, it would have been desirable to employ a more objective measure of deterioration, since the decision to transfer a patient to the ICU is discretionary. However, we have found that key data points needed to define such a measure (eg, vital signs) are not consistently charted when a patient deterioratesthis is not surprising outside the research setting, given that nurses and physicians involved in a transfer may be focusing on caring for the patient rather than immediately charting. Given the complexities of end‐of‐life‐care decision‐making, we could not employ death as the outcome of interest. A related issue is that our model does not differentiate between reasons for needing transfer to the ICU, an issue recently discussed by Bapoje et al.18

Our model does not address an important issue raised by Bapoje et al18 and Litvak, Pronovost, and others,19, 20 namely, whether a patient should have been admitted to a non‐ICU setting in the first place. Our team is currently developing a model for doing exactly this (providing decision support for triage in the emergency department), but discussion of this methodology is outside the scope of this article.

Because of resource and data limitations, our model also does not include newborns, children, women admitted for childbirth, or patients transferred from non‐KPMCP hospitals. However, the approach described here could serve as a starting point for developing models for these other populations.

The generalizability of our model must also be considered. The Northern California KPMCP is unusual in having large electronic databases that include physiologic as well as longitudinal patient data. Many hospitals cannot take advantage of all the methods described here. However, the methods we employed could be modified for use by hospital systems in countries such as Great Britain and Canada, and entities such as the Veterans Administration Hospital System in the United States. The KPMCP population, an insured population with few barriers to access, is healthier than the general population, and some population subsets are underrepresented in our cohort. Practice patterns may also vary. Nonetheless, the model described here could serve as a good starting point for future collaborative studies, and it would be possible to develop models suitable for use by stand‐alone hospitals (eg, recalibrating so that one used a Charlson comorbidity21 score based on present on‐admission codes rather than the COPS).

The need for early detection of patient deterioration has played a major role in the development of rapid response teams, as well as scores such as the MEWS. In particular, entities such as the Institute for Healthcare Improvement have advocated the use of early warning systems.22 However, having a statistically robust model to support an early warning system is only part of the solution, and a number of new challenges must then be addressed. The first is actual electronic deployment. Existing inpatient EMRs were not designed with complex calculations in mind, and we anticipate that some degradation in performance will occur when we test our models using real‐time data capture. As Bapoje et al point out, simply having an alert may be insufficient, since not all transfers are preventable.18 Early warning systems also raise ethical issues (for example, what should be done if an alert leads a clinician to confront the fact that an end‐of‐life‐care discussion needs to occur?). From a research perspective, if one were to formally test the benefits of such models, it would be critical to define outcome measures other than death (which is strongly affected by end‐of‐life‐care decisions) or ICU transfer (which is often desirable).

In conclusion, we have developed an approach for predicting impending physiologic deterioration of hospitalized adults outside the ICU. Our approach illustrates how organizations can take maximal advantage of EMRs in a manner that exceeds meaningful use specifications.23, 24 Our study highlights the possibility of using fully automated EMR data for building and applying sophisticated statistical models in settings other than the highly monitored ICU without the need for additional equipment. It also expands the universe of severity scoring to one in which probability estimates are provided in real time and throughout an entire hospitalization. Model performance will undoubtedly improve over time, as more patient data become available. Although our approach has important limitations, it is suitable for testing using real‐time data in a simulated environment. Such testing would permit identification of unanticipated problems and quantification of the degradation of model performance due to real life factors, such as delays in vital signs charting or EMR system brownouts. It could also serve as the springboard for future collaborative studies, with a broader population base, in which the EMR becomes a tool for care, not just documentation.