Hospital Medicine

Return visit (RV) rate is a quality measure commonly used in the emergency department (ED) setting. This metric may represent suboptimal care at the index ED visit.^1-5 Although patient- and visit-level factors affecting ED RVs have been evaluated,^1,3,4,6-9 hospital-level factors and factors of a hospital’s patient population that may play roles in ED RV rates have not been examined. Identifying the factors associated with increased RVs may allow resources to be designated to areas that improve emergent care for children.¹⁰

Hospital readmission rates are a closely followed quality measure and are linked to reimbursement by the federal government, but a recent study found the influence a hospital can have on this marker may be mitigated by the impact of the social determinates of health (SDHs) of the hospital’s patient population.¹¹ That study and others have prompted an ongoing debate about adjusting quality measures for SDHs.^12,13 A clearer understanding of these interactions may permit us to focus on factors that can truly lead to improvement in care instead of penalizing practitioners or hospitals that provide care to those most in need.

Prior work has identified several SDHs associated with higher ED RV rates in patient- or visit-level analyses.^3,11,14 We conducted a study of hospital-level characteristics and characteristics of a hospital’s patient population to identify potentially mutable factors associated with increased ED RV rates that, once recognized, may allow for improvement in this quality measure.

This study was not considered human subjects research in accordance with Common Rule 45 CFR§46.104(f) and was evaluated by the Ann and Robert H. Lurie Children’s Hospital and Northwestern University Feinberg School of Medicine Institutional Review Boards and deemed exempt from review.

Study Population and Protocol

Our study had 2 data sources (to be described in detail): the Pediatric Health Information System (PHIS) and a survey of ED medical directors of the hospitals represented within PHIS. Hospitals were eligible for inclusion in the study if their data (1) met PHIS quality control standards for ED patient visits as determined by internal data assurance processes incorporated in PHIS,^3,14,15 (2) included data only from an identifiable single main ED, and (3) completed the ED medical director’s survey.

PHIS Database

PHIS, an administrative database managed by Truven Health Analytics, includes data from ED, ambulatory surgery, observation, and inpatient encounters across Children’s Hospital Association member children’s hospitals in North America. Data are subjected to validity checks before being included in the database.¹⁶ PHIS assigns unique patient identifiers to track individual patient visits within participating institutions over time.

Hospitals were described by percentages of ED patients in several groups: age (<1, 1-4, 5-9, 10-14, and 15-18 years)¹⁷; sex; race/ethnicity; insurance type (commercial, government, other); ED International Classification of Diseases, Ninth Edition (ICD-9) diagnosis code–based severity classification system score (1-2, low severity; 3-5, high severity)¹⁸; complex chronic condition presence at ED visits in prior year^14,19-21; home postal (Zip) code median household income from 2010 US Census data compared with Federal Poverty Level (<1.5, 1.5-2, 2-3, and >3 × FPL)¹⁷; and primary care physician (PCP) density in Federal Health Service Area of patient’s home address as reported by Dartmouth Atlas of Health Care modeled by quartiles.²² Density of PCPs—general pediatricians, family practitioners, general practitioners, and general internists—is calculated as number of PCPs per 100,000 residents. We used PCP density to account for potential care provided by any of the PCPs mentioned. We also assessed, at hospital level, index visit arrival time (8:01 am to 4:00 pm; 4:01 pm to 12:00 am; 12:01 am to 8:00 am) and index visit season.²³

ED Medical Director Survey

A web-based survey was constructed in an iterative process based on literature review and expert opinion to assess hospital-level factors that may impact ED RV rates.^3,7,24-26 The survey was piloted at 3 institutions to refine its structure and content.

The survey included 15 close-ended or multiple-choice questions on ED environment and operations and 2 open-ended questions, “What is the largest barrier to reducing the number of return visits within 72 hours of discharge from a previous ED visit?” and “In your opinion, what is the best way of reducing the number of the return visits within 72 hours of previous ED visit ?” (questionnaire in Supplemental material). Hospital characteristics from the survey included total clinical time allotment, or full-time equivalent (FTE), among all physicians, pediatric emergency medicine (PEM) fellowship-trained physicians, and all other (non-PEM) physicians. The data were standardized across sites by calculating FTE-per-10,000-visits values for each hospital; median duration of ED visit for admitted and discharged patients; median time from arrival to ED physician evaluation; rate of leaving without being seen; discharge educational material authorship and age specificity; follow-up visit scheduling procedure; and percentage of ED patients for whom English was a second language.

Responses to the 2 open-ended questions were independently categorized by Drs. Pittsenbarger and Alpern. Responses could be placed in more than 1 category if multiple answers to the question were included in the response. Categorizations were compared for consistency, and any inconsistencies were resolved by the consensus of the study investigators.

Outcome Measures From PHIS Database

All ED visits within a 12-month period (July 1, 2013–June 30, 2014) by patients younger than 18 years at time of index ED visit were eligible for inclusion in the study. An index visit was defined as any ED visit without another ED visit within the preceding 72 hours. The 72-hour time frame was used because it is the most widely studied time frame for ED RVs.⁵ Index ED visits that led to admission, observation status, death, or transfer were excluded.

The 2 primary outcomes of interest were (1) RVs within 72 hours of index ED visit discharge and (2) RVs within 72 hours that resulted in hospital admission or observation status at the next ED visit (RVA).^7,9,27-30 For patients with multiple ED revisits within 72 hours, only the first was assessed. There was a 72-hour minimum between index visits for the same patient.

Statistical Analyses

To determine hospital groups based on RV and RVA rates, we adjusted RV and RVA rates using generalized linear mixed-effects models, controlling for clustering and allowing for correlated data (within hospitals), nonconstant variability (across hospitals), and non-normally distributed data, as we did in a study of patient-level factors associated with ED RV and RVA.³ For each calculated rate (RV, RVA), the hospitals were then classified into 3 groups based on whether the hospital’s adjusted RV and RVA rates were outside 2 SDs from the mean, below the 5th or above the 95th percentile, or within that range. These groups were labeled lowest outliers, highest outliers, and average-performing hospitals.

After the groups of hospitals were determined, we returned to using unadjusted data to statistically analyze them. We summarized continuous variables using minimum and maximum values, medians, and interquartile ranges (IQRs). We present categorical variables using counts and percentages. To identify hospital characteristics with the most potential to gain from improvement, we also analyzed associations using 2 collapsed groups: hospitals with RV (or RVA) rates included in the average-performing and lowest outlier groups and hospitals within the highest outlier group. Hospital characteristics and hospital’s patient population characteristics from the surveys are summarized based on RV and RVA rate groups. Differences in distributions among continuous variables were assessed by Kruskal-Wallis 1-way analysis of variance. Chi-square tests were used to evaluate differences in proportions among categorical variables. All statistical analyses were performed with SAS Version 9.4 (SAS Institute); 2-sided P < 0.05 was considered statistically significant.

Return Visit Rates and Hospital ED Site Population Characteristics

Twenty-four of 35 (68%) eligible hospitals that met PHIS quality control standards for ED patient visits responded to the ED medical director survey. The included hospitals that both met quality control standards and completed the survey had a total of 1,456,377 patient visits during the study period. Individual sites had annual volumes ranging from 26,627 to 96,637 ED encounters. The mean RV rate across the institutions was 3.7% (range, 3.0%-4.8%), and the mean RVA rate across the hospitals was 0.7% (range, 0.5%-1.1%) (Figure).

There were 5 hospitals with RV rates less than 2 SDs of the mean rate, placing them in the lowest outlier group for RV; 13 hospitals with RV rates within 2 SDs of the mean RV rate, placing them in the average-performing group; and 6 hospitals with RV rates above 2 SDs of the mean, placing them in the highest outlier group. Table 1 lists the hospital ED site population characteristics among the 3 RV rate groups. Hospitals in the highest outlier group served populations with higher proportions of patients with insurance from a government payer, lower proportions of patients covered by a commercial insurance plan, and higher proportion of patients with lower median household incomes.

RV Rates and Hospital-Level Factors Survey Characteristics

Table 2 lists the ED medical director survey hospital-level data among the 3 RV rate groups. There were fewer FTEs by PEM fellowship-trained physicians per 10,000 patient visits at sites with higher RV rates (Table 2). Hospital-level characteristics assessed by the survey were not associated with RVA rates (Supplemental Table 2).

Evaluating characteristics of hospitals with the most potential to gain from improvement, hospitals with the highest RV rates (highest outlier group), compared with hospitals in the lowest outlier and average-performing groups collapsed together, persisted in having fewer PEM fellowship-trained physician FTEs per patient visit (Table 3). A similar collapsed analysis of RVA rates demonstrated that hospitals in the highest outlier group had longer-wait-to-physician time (81 minutes; IQR, 51-105 minutes) compared with hospitals in the other 2 groups (30 minutes; IQR, 19-42.5 minutes) (Table 3).

Other studies have identified patient- and visit-level characteristics associated with higher ED RV and RVA rates.^3,8,9,31 However, as our goal was to identify possible modifiable institutional features, our study examined factors at hospital and population-served levels (instead of patient or visit level) that may impact ED RV and RVA rates. Interestingly, our sample of tertiary-care pediatric center EDs provided evidence of variability in RV and RVA rates. We identified factors associated with RV rates related to the SDHs of the populations served by the ED, which suggests these factors are not modifiable at an institution level. In addition, we found that the increased availability of PEM providers per patient visit correlated with fewer ED RVs.

Hospitals serving ED populations with more government-insured and fewer commercially insured patients had higher rates of return to the ED. Similarly, hospitals with larger proportions of patients from areas with lower median household incomes had higher RV rates. These factors may indicate that patients with limited resources may have more frequent ED RVs,^3,6,32,33 and hospitals that serve them have higher ED RV rates. Our findings complement those of a recent study by Sills et al.,¹¹ who evaluated hospital readmissions and proposed risk adjustment for performance reimbursement. This study found that hospital population-level race, ethnicity, insurance status, and household income were predictors of hospital readmission after discharge.

Of note, our data did not identify similar site-level attributes related to the population served that correlated with RVA rates. We postulate that the need for admission on RV may indicate an inherent clinical urgency or medical need associated with the return to the ED, whereas RV without admission may be related more to patient- or population-level sociodemographic factors than to quality of care and clinical course, which influence ED utilization.^1,3,30 EDs treating higher proportions of patients of minority race or ethnicity, those with fewer financial resources, and those in more need of government health insurance are at higher risk for ED revisits.

We observed that increased PEM fellowship-trained physician staffing was associated with decreased RV rates. The availability of specialty-trained physicians in PEM may allow a larger proportion of patients treated by physicians with honed clinical skills for the patient population. Data from a single pediatric center showed PEM fellowship-trained physicians had admission rates lower than those of their counterparts without subspecialty fellowship training.³⁴ The lower RV rate for this group in our study is especially interesting in light of previously reported lower admission rates at index visit in PEM trained physicians. With lower index admission rates, it may have been assumed that visits associated with PEM trained physician care would have an increased (rather than decreased) chance of RV. In addition, we noted the increased RVA rates were associated with longer waits to see a physician. These measures may indicate the effect of institutional access to robust resources (the ability to hire and support more specialty-trained physicians). These novel findings warrant further evaluation, particularly as our sample included only pediatric centers.

Our survey data demonstrated the impact that access to care has on ED RV rates. The ED medical directors indicated that limited access to outpatient appointments with PCPs and specialists was an important factor increasing ED RVs and a potential avenue for interventions. As the 2 open-ended questions addressed barriers and potential solutions, it is interesting that the respondents cited access to care and discharge instructions as the largest barriers and identified innovations in access to care and discharge education as important potential remedies.

This study demonstrated that, at the hospital level, ED RV quality measures are influenced by complex and varied SDHs that primarily reflect the characteristics of the patient populations served. Prior work has similarly highlighted the importance of gaining a rigorous understanding of other quality measures before widespread use, reporting, and dissemination of results.^11,35-38 With this in mind, as quality measures are developed and implemented, care should be taken to ensure they accurately and appropriately reflect the quality of care provided to the patient and are not more representative of other factors not directly within institutional control. These findings call into question the usefulness of ED RVs as a quality measure for comparing institutions.

Study Limitations

This study had several limitations. The PHIS dataset tracks only patients within each institution and does not include RVs to other EDs, which may account for a proportion of RVs.³⁹ Our survey response rate was 68% among medical directors, excluding 11 hospitals from analysis, which decreased the study’s power to detect differences that may be present between groups. In addition, the generalizability of our findings may be limited to tertiary-care children’s hospitals, as the PHIS dataset included only these types of healthcare facilities. We also included data only from the sites’ main EDs, and therefore cannot know if our results are applicable to satellite EDs. ED staffing of PEM physicians was analyzed using FTEs. However, number of clinical hours in 1 FTE may vary among sites, leading to imprecision in this hospital characteristic.

Hospitals with the highest RV rates served populations with a larger proportion of patients with government insurance and lower household income, and these hospitals had fewer PEM trained physicians. Variation in RV rates among hospitals may be indicative of the SDHs of their unique patient populations. ED revisit rates should be used cautiously in determining the quality of care of hospitals providing care to differing populations.

Disclosure

Nothing to report.

Return visit (RV) rate is a quality measure commonly used in the emergency department (ED) setting. This metric may represent suboptimal care at the index ED visit.^1-5 Although patient- and visit-level factors affecting ED RVs have been evaluated,^1,3,4,6-9 hospital-level factors and factors of a hospital’s patient population that may play roles in ED RV rates have not been examined. Identifying the factors associated with increased RVs may allow resources to be designated to areas that improve emergent care for children.¹⁰

Hospital readmission rates are a closely followed quality measure and are linked to reimbursement by the federal government, but a recent study found the influence a hospital can have on this marker may be mitigated by the impact of the social determinates of health (SDHs) of the hospital’s patient population.¹¹ That study and others have prompted an ongoing debate about adjusting quality measures for SDHs.^12,13 A clearer understanding of these interactions may permit us to focus on factors that can truly lead to improvement in care instead of penalizing practitioners or hospitals that provide care to those most in need.

Prior work has identified several SDHs associated with higher ED RV rates in patient- or visit-level analyses.^3,11,14 We conducted a study of hospital-level characteristics and characteristics of a hospital’s patient population to identify potentially mutable factors associated with increased ED RV rates that, once recognized, may allow for improvement in this quality measure.

This study was not considered human subjects research in accordance with Common Rule 45 CFR§46.104(f) and was evaluated by the Ann and Robert H. Lurie Children’s Hospital and Northwestern University Feinberg School of Medicine Institutional Review Boards and deemed exempt from review.

Study Population and Protocol

Our study had 2 data sources (to be described in detail): the Pediatric Health Information System (PHIS) and a survey of ED medical directors of the hospitals represented within PHIS. Hospitals were eligible for inclusion in the study if their data (1) met PHIS quality control standards for ED patient visits as determined by internal data assurance processes incorporated in PHIS,^3,14,15 (2) included data only from an identifiable single main ED, and (3) completed the ED medical director’s survey.

PHIS Database

PHIS, an administrative database managed by Truven Health Analytics, includes data from ED, ambulatory surgery, observation, and inpatient encounters across Children’s Hospital Association member children’s hospitals in North America. Data are subjected to validity checks before being included in the database.¹⁶ PHIS assigns unique patient identifiers to track individual patient visits within participating institutions over time.

Hospitals were described by percentages of ED patients in several groups: age (<1, 1-4, 5-9, 10-14, and 15-18 years)¹⁷; sex; race/ethnicity; insurance type (commercial, government, other); ED International Classification of Diseases, Ninth Edition (ICD-9) diagnosis code–based severity classification system score (1-2, low severity; 3-5, high severity)¹⁸; complex chronic condition presence at ED visits in prior year^14,19-21; home postal (Zip) code median household income from 2010 US Census data compared with Federal Poverty Level (<1.5, 1.5-2, 2-3, and >3 × FPL)¹⁷; and primary care physician (PCP) density in Federal Health Service Area of patient’s home address as reported by Dartmouth Atlas of Health Care modeled by quartiles.²² Density of PCPs—general pediatricians, family practitioners, general practitioners, and general internists—is calculated as number of PCPs per 100,000 residents. We used PCP density to account for potential care provided by any of the PCPs mentioned. We also assessed, at hospital level, index visit arrival time (8:01 am to 4:00 pm; 4:01 pm to 12:00 am; 12:01 am to 8:00 am) and index visit season.²³

ED Medical Director Survey

A web-based survey was constructed in an iterative process based on literature review and expert opinion to assess hospital-level factors that may impact ED RV rates.^3,7,24-26 The survey was piloted at 3 institutions to refine its structure and content.

The survey included 15 close-ended or multiple-choice questions on ED environment and operations and 2 open-ended questions, “What is the largest barrier to reducing the number of return visits within 72 hours of discharge from a previous ED visit?” and “In your opinion, what is the best way of reducing the number of the return visits within 72 hours of previous ED visit ?” (questionnaire in Supplemental material). Hospital characteristics from the survey included total clinical time allotment, or full-time equivalent (FTE), among all physicians, pediatric emergency medicine (PEM) fellowship-trained physicians, and all other (non-PEM) physicians. The data were standardized across sites by calculating FTE-per-10,000-visits values for each hospital; median duration of ED visit for admitted and discharged patients; median time from arrival to ED physician evaluation; rate of leaving without being seen; discharge educational material authorship and age specificity; follow-up visit scheduling procedure; and percentage of ED patients for whom English was a second language.

Responses to the 2 open-ended questions were independently categorized by Drs. Pittsenbarger and Alpern. Responses could be placed in more than 1 category if multiple answers to the question were included in the response. Categorizations were compared for consistency, and any inconsistencies were resolved by the consensus of the study investigators.

Outcome Measures From PHIS Database

All ED visits within a 12-month period (July 1, 2013–June 30, 2014) by patients younger than 18 years at time of index ED visit were eligible for inclusion in the study. An index visit was defined as any ED visit without another ED visit within the preceding 72 hours. The 72-hour time frame was used because it is the most widely studied time frame for ED RVs.⁵ Index ED visits that led to admission, observation status, death, or transfer were excluded.

The 2 primary outcomes of interest were (1) RVs within 72 hours of index ED visit discharge and (2) RVs within 72 hours that resulted in hospital admission or observation status at the next ED visit (RVA).^7,9,27-30 For patients with multiple ED revisits within 72 hours, only the first was assessed. There was a 72-hour minimum between index visits for the same patient.

Statistical Analyses

To determine hospital groups based on RV and RVA rates, we adjusted RV and RVA rates using generalized linear mixed-effects models, controlling for clustering and allowing for correlated data (within hospitals), nonconstant variability (across hospitals), and non-normally distributed data, as we did in a study of patient-level factors associated with ED RV and RVA.³ For each calculated rate (RV, RVA), the hospitals were then classified into 3 groups based on whether the hospital’s adjusted RV and RVA rates were outside 2 SDs from the mean, below the 5th or above the 95th percentile, or within that range. These groups were labeled lowest outliers, highest outliers, and average-performing hospitals.

After the groups of hospitals were determined, we returned to using unadjusted data to statistically analyze them. We summarized continuous variables using minimum and maximum values, medians, and interquartile ranges (IQRs). We present categorical variables using counts and percentages. To identify hospital characteristics with the most potential to gain from improvement, we also analyzed associations using 2 collapsed groups: hospitals with RV (or RVA) rates included in the average-performing and lowest outlier groups and hospitals within the highest outlier group. Hospital characteristics and hospital’s patient population characteristics from the surveys are summarized based on RV and RVA rate groups. Differences in distributions among continuous variables were assessed by Kruskal-Wallis 1-way analysis of variance. Chi-square tests were used to evaluate differences in proportions among categorical variables. All statistical analyses were performed with SAS Version 9.4 (SAS Institute); 2-sided P < 0.05 was considered statistically significant.

Return Visit Rates and Hospital ED Site Population Characteristics

Twenty-four of 35 (68%) eligible hospitals that met PHIS quality control standards for ED patient visits responded to the ED medical director survey. The included hospitals that both met quality control standards and completed the survey had a total of 1,456,377 patient visits during the study period. Individual sites had annual volumes ranging from 26,627 to 96,637 ED encounters. The mean RV rate across the institutions was 3.7% (range, 3.0%-4.8%), and the mean RVA rate across the hospitals was 0.7% (range, 0.5%-1.1%) (Figure).

There were 5 hospitals with RV rates less than 2 SDs of the mean rate, placing them in the lowest outlier group for RV; 13 hospitals with RV rates within 2 SDs of the mean RV rate, placing them in the average-performing group; and 6 hospitals with RV rates above 2 SDs of the mean, placing them in the highest outlier group. Table 1 lists the hospital ED site population characteristics among the 3 RV rate groups. Hospitals in the highest outlier group served populations with higher proportions of patients with insurance from a government payer, lower proportions of patients covered by a commercial insurance plan, and higher proportion of patients with lower median household incomes.

RV Rates and Hospital-Level Factors Survey Characteristics

Table 2 lists the ED medical director survey hospital-level data among the 3 RV rate groups. There were fewer FTEs by PEM fellowship-trained physicians per 10,000 patient visits at sites with higher RV rates (Table 2). Hospital-level characteristics assessed by the survey were not associated with RVA rates (Supplemental Table 2).

Evaluating characteristics of hospitals with the most potential to gain from improvement, hospitals with the highest RV rates (highest outlier group), compared with hospitals in the lowest outlier and average-performing groups collapsed together, persisted in having fewer PEM fellowship-trained physician FTEs per patient visit (Table 3). A similar collapsed analysis of RVA rates demonstrated that hospitals in the highest outlier group had longer-wait-to-physician time (81 minutes; IQR, 51-105 minutes) compared with hospitals in the other 2 groups (30 minutes; IQR, 19-42.5 minutes) (Table 3).

Other studies have identified patient- and visit-level characteristics associated with higher ED RV and RVA rates.^3,8,9,31 However, as our goal was to identify possible modifiable institutional features, our study examined factors at hospital and population-served levels (instead of patient or visit level) that may impact ED RV and RVA rates. Interestingly, our sample of tertiary-care pediatric center EDs provided evidence of variability in RV and RVA rates. We identified factors associated with RV rates related to the SDHs of the populations served by the ED, which suggests these factors are not modifiable at an institution level. In addition, we found that the increased availability of PEM providers per patient visit correlated with fewer ED RVs.

Hospitals serving ED populations with more government-insured and fewer commercially insured patients had higher rates of return to the ED. Similarly, hospitals with larger proportions of patients from areas with lower median household incomes had higher RV rates. These factors may indicate that patients with limited resources may have more frequent ED RVs,^3,6,32,33 and hospitals that serve them have higher ED RV rates. Our findings complement those of a recent study by Sills et al.,¹¹ who evaluated hospital readmissions and proposed risk adjustment for performance reimbursement. This study found that hospital population-level race, ethnicity, insurance status, and household income were predictors of hospital readmission after discharge.

Of note, our data did not identify similar site-level attributes related to the population served that correlated with RVA rates. We postulate that the need for admission on RV may indicate an inherent clinical urgency or medical need associated with the return to the ED, whereas RV without admission may be related more to patient- or population-level sociodemographic factors than to quality of care and clinical course, which influence ED utilization.^1,3,30 EDs treating higher proportions of patients of minority race or ethnicity, those with fewer financial resources, and those in more need of government health insurance are at higher risk for ED revisits.

We observed that increased PEM fellowship-trained physician staffing was associated with decreased RV rates. The availability of specialty-trained physicians in PEM may allow a larger proportion of patients treated by physicians with honed clinical skills for the patient population. Data from a single pediatric center showed PEM fellowship-trained physicians had admission rates lower than those of their counterparts without subspecialty fellowship training.³⁴ The lower RV rate for this group in our study is especially interesting in light of previously reported lower admission rates at index visit in PEM trained physicians. With lower index admission rates, it may have been assumed that visits associated with PEM trained physician care would have an increased (rather than decreased) chance of RV. In addition, we noted the increased RVA rates were associated with longer waits to see a physician. These measures may indicate the effect of institutional access to robust resources (the ability to hire and support more specialty-trained physicians). These novel findings warrant further evaluation, particularly as our sample included only pediatric centers.

Our survey data demonstrated the impact that access to care has on ED RV rates. The ED medical directors indicated that limited access to outpatient appointments with PCPs and specialists was an important factor increasing ED RVs and a potential avenue for interventions. As the 2 open-ended questions addressed barriers and potential solutions, it is interesting that the respondents cited access to care and discharge instructions as the largest barriers and identified innovations in access to care and discharge education as important potential remedies.

This study demonstrated that, at the hospital level, ED RV quality measures are influenced by complex and varied SDHs that primarily reflect the characteristics of the patient populations served. Prior work has similarly highlighted the importance of gaining a rigorous understanding of other quality measures before widespread use, reporting, and dissemination of results.^11,35-38 With this in mind, as quality measures are developed and implemented, care should be taken to ensure they accurately and appropriately reflect the quality of care provided to the patient and are not more representative of other factors not directly within institutional control. These findings call into question the usefulness of ED RVs as a quality measure for comparing institutions.

Study Limitations

This study had several limitations. The PHIS dataset tracks only patients within each institution and does not include RVs to other EDs, which may account for a proportion of RVs.³⁹ Our survey response rate was 68% among medical directors, excluding 11 hospitals from analysis, which decreased the study’s power to detect differences that may be present between groups. In addition, the generalizability of our findings may be limited to tertiary-care children’s hospitals, as the PHIS dataset included only these types of healthcare facilities. We also included data only from the sites’ main EDs, and therefore cannot know if our results are applicable to satellite EDs. ED staffing of PEM physicians was analyzed using FTEs. However, number of clinical hours in 1 FTE may vary among sites, leading to imprecision in this hospital characteristic.

Hospitals with the highest RV rates served populations with a larger proportion of patients with government insurance and lower household income, and these hospitals had fewer PEM trained physicians. Variation in RV rates among hospitals may be indicative of the SDHs of their unique patient populations. ED revisit rates should be used cautiously in determining the quality of care of hospitals providing care to differing populations.

Disclosure

Nothing to report.

Return visit (RV) rate is a quality measure commonly used in the emergency department (ED) setting. This metric may represent suboptimal care at the index ED visit.^1-5 Although patient- and visit-level factors affecting ED RVs have been evaluated,^1,3,4,6-9 hospital-level factors and factors of a hospital’s patient population that may play roles in ED RV rates have not been examined. Identifying the factors associated with increased RVs may allow resources to be designated to areas that improve emergent care for children.¹⁰

Hospital readmission rates are a closely followed quality measure and are linked to reimbursement by the federal government, but a recent study found the influence a hospital can have on this marker may be mitigated by the impact of the social determinates of health (SDHs) of the hospital’s patient population.¹¹ That study and others have prompted an ongoing debate about adjusting quality measures for SDHs.^12,13 A clearer understanding of these interactions may permit us to focus on factors that can truly lead to improvement in care instead of penalizing practitioners or hospitals that provide care to those most in need.

Prior work has identified several SDHs associated with higher ED RV rates in patient- or visit-level analyses.^3,11,14 We conducted a study of hospital-level characteristics and characteristics of a hospital’s patient population to identify potentially mutable factors associated with increased ED RV rates that, once recognized, may allow for improvement in this quality measure.

This study was not considered human subjects research in accordance with Common Rule 45 CFR§46.104(f) and was evaluated by the Ann and Robert H. Lurie Children’s Hospital and Northwestern University Feinberg School of Medicine Institutional Review Boards and deemed exempt from review.

Study Population and Protocol

Our study had 2 data sources (to be described in detail): the Pediatric Health Information System (PHIS) and a survey of ED medical directors of the hospitals represented within PHIS. Hospitals were eligible for inclusion in the study if their data (1) met PHIS quality control standards for ED patient visits as determined by internal data assurance processes incorporated in PHIS,^3,14,15 (2) included data only from an identifiable single main ED, and (3) completed the ED medical director’s survey.

PHIS Database

PHIS, an administrative database managed by Truven Health Analytics, includes data from ED, ambulatory surgery, observation, and inpatient encounters across Children’s Hospital Association member children’s hospitals in North America. Data are subjected to validity checks before being included in the database.¹⁶ PHIS assigns unique patient identifiers to track individual patient visits within participating institutions over time.

Hospitals were described by percentages of ED patients in several groups: age (<1, 1-4, 5-9, 10-14, and 15-18 years)¹⁷; sex; race/ethnicity; insurance type (commercial, government, other); ED International Classification of Diseases, Ninth Edition (ICD-9) diagnosis code–based severity classification system score (1-2, low severity; 3-5, high severity)¹⁸; complex chronic condition presence at ED visits in prior year^14,19-21; home postal (Zip) code median household income from 2010 US Census data compared with Federal Poverty Level (<1.5, 1.5-2, 2-3, and >3 × FPL)¹⁷; and primary care physician (PCP) density in Federal Health Service Area of patient’s home address as reported by Dartmouth Atlas of Health Care modeled by quartiles.²² Density of PCPs—general pediatricians, family practitioners, general practitioners, and general internists—is calculated as number of PCPs per 100,000 residents. We used PCP density to account for potential care provided by any of the PCPs mentioned. We also assessed, at hospital level, index visit arrival time (8:01 am to 4:00 pm; 4:01 pm to 12:00 am; 12:01 am to 8:00 am) and index visit season.²³

ED Medical Director Survey

A web-based survey was constructed in an iterative process based on literature review and expert opinion to assess hospital-level factors that may impact ED RV rates.^3,7,24-26 The survey was piloted at 3 institutions to refine its structure and content.

The survey included 15 close-ended or multiple-choice questions on ED environment and operations and 2 open-ended questions, “What is the largest barrier to reducing the number of return visits within 72 hours of discharge from a previous ED visit?” and “In your opinion, what is the best way of reducing the number of the return visits within 72 hours of previous ED visit ?” (questionnaire in Supplemental material). Hospital characteristics from the survey included total clinical time allotment, or full-time equivalent (FTE), among all physicians, pediatric emergency medicine (PEM) fellowship-trained physicians, and all other (non-PEM) physicians. The data were standardized across sites by calculating FTE-per-10,000-visits values for each hospital; median duration of ED visit for admitted and discharged patients; median time from arrival to ED physician evaluation; rate of leaving without being seen; discharge educational material authorship and age specificity; follow-up visit scheduling procedure; and percentage of ED patients for whom English was a second language.

Responses to the 2 open-ended questions were independently categorized by Drs. Pittsenbarger and Alpern. Responses could be placed in more than 1 category if multiple answers to the question were included in the response. Categorizations were compared for consistency, and any inconsistencies were resolved by the consensus of the study investigators.

Outcome Measures From PHIS Database

All ED visits within a 12-month period (July 1, 2013–June 30, 2014) by patients younger than 18 years at time of index ED visit were eligible for inclusion in the study. An index visit was defined as any ED visit without another ED visit within the preceding 72 hours. The 72-hour time frame was used because it is the most widely studied time frame for ED RVs.⁵ Index ED visits that led to admission, observation status, death, or transfer were excluded.

The 2 primary outcomes of interest were (1) RVs within 72 hours of index ED visit discharge and (2) RVs within 72 hours that resulted in hospital admission or observation status at the next ED visit (RVA).^7,9,27-30 For patients with multiple ED revisits within 72 hours, only the first was assessed. There was a 72-hour minimum between index visits for the same patient.

Statistical Analyses

To determine hospital groups based on RV and RVA rates, we adjusted RV and RVA rates using generalized linear mixed-effects models, controlling for clustering and allowing for correlated data (within hospitals), nonconstant variability (across hospitals), and non-normally distributed data, as we did in a study of patient-level factors associated with ED RV and RVA.³ For each calculated rate (RV, RVA), the hospitals were then classified into 3 groups based on whether the hospital’s adjusted RV and RVA rates were outside 2 SDs from the mean, below the 5th or above the 95th percentile, or within that range. These groups were labeled lowest outliers, highest outliers, and average-performing hospitals.

After the groups of hospitals were determined, we returned to using unadjusted data to statistically analyze them. We summarized continuous variables using minimum and maximum values, medians, and interquartile ranges (IQRs). We present categorical variables using counts and percentages. To identify hospital characteristics with the most potential to gain from improvement, we also analyzed associations using 2 collapsed groups: hospitals with RV (or RVA) rates included in the average-performing and lowest outlier groups and hospitals within the highest outlier group. Hospital characteristics and hospital’s patient population characteristics from the surveys are summarized based on RV and RVA rate groups. Differences in distributions among continuous variables were assessed by Kruskal-Wallis 1-way analysis of variance. Chi-square tests were used to evaluate differences in proportions among categorical variables. All statistical analyses were performed with SAS Version 9.4 (SAS Institute); 2-sided P < 0.05 was considered statistically significant.

Return Visit Rates and Hospital ED Site Population Characteristics

Twenty-four of 35 (68%) eligible hospitals that met PHIS quality control standards for ED patient visits responded to the ED medical director survey. The included hospitals that both met quality control standards and completed the survey had a total of 1,456,377 patient visits during the study period. Individual sites had annual volumes ranging from 26,627 to 96,637 ED encounters. The mean RV rate across the institutions was 3.7% (range, 3.0%-4.8%), and the mean RVA rate across the hospitals was 0.7% (range, 0.5%-1.1%) (Figure).

There were 5 hospitals with RV rates less than 2 SDs of the mean rate, placing them in the lowest outlier group for RV; 13 hospitals with RV rates within 2 SDs of the mean RV rate, placing them in the average-performing group; and 6 hospitals with RV rates above 2 SDs of the mean, placing them in the highest outlier group. Table 1 lists the hospital ED site population characteristics among the 3 RV rate groups. Hospitals in the highest outlier group served populations with higher proportions of patients with insurance from a government payer, lower proportions of patients covered by a commercial insurance plan, and higher proportion of patients with lower median household incomes.

RV Rates and Hospital-Level Factors Survey Characteristics

Table 2 lists the ED medical director survey hospital-level data among the 3 RV rate groups. There were fewer FTEs by PEM fellowship-trained physicians per 10,000 patient visits at sites with higher RV rates (Table 2). Hospital-level characteristics assessed by the survey were not associated with RVA rates (Supplemental Table 2).

Evaluating characteristics of hospitals with the most potential to gain from improvement, hospitals with the highest RV rates (highest outlier group), compared with hospitals in the lowest outlier and average-performing groups collapsed together, persisted in having fewer PEM fellowship-trained physician FTEs per patient visit (Table 3). A similar collapsed analysis of RVA rates demonstrated that hospitals in the highest outlier group had longer-wait-to-physician time (81 minutes; IQR, 51-105 minutes) compared with hospitals in the other 2 groups (30 minutes; IQR, 19-42.5 minutes) (Table 3).

Other studies have identified patient- and visit-level characteristics associated with higher ED RV and RVA rates.^3,8,9,31 However, as our goal was to identify possible modifiable institutional features, our study examined factors at hospital and population-served levels (instead of patient or visit level) that may impact ED RV and RVA rates. Interestingly, our sample of tertiary-care pediatric center EDs provided evidence of variability in RV and RVA rates. We identified factors associated with RV rates related to the SDHs of the populations served by the ED, which suggests these factors are not modifiable at an institution level. In addition, we found that the increased availability of PEM providers per patient visit correlated with fewer ED RVs.

Hospitals serving ED populations with more government-insured and fewer commercially insured patients had higher rates of return to the ED. Similarly, hospitals with larger proportions of patients from areas with lower median household incomes had higher RV rates. These factors may indicate that patients with limited resources may have more frequent ED RVs,^3,6,32,33 and hospitals that serve them have higher ED RV rates. Our findings complement those of a recent study by Sills et al.,¹¹ who evaluated hospital readmissions and proposed risk adjustment for performance reimbursement. This study found that hospital population-level race, ethnicity, insurance status, and household income were predictors of hospital readmission after discharge.

Of note, our data did not identify similar site-level attributes related to the population served that correlated with RVA rates. We postulate that the need for admission on RV may indicate an inherent clinical urgency or medical need associated with the return to the ED, whereas RV without admission may be related more to patient- or population-level sociodemographic factors than to quality of care and clinical course, which influence ED utilization.^1,3,30 EDs treating higher proportions of patients of minority race or ethnicity, those with fewer financial resources, and those in more need of government health insurance are at higher risk for ED revisits.

We observed that increased PEM fellowship-trained physician staffing was associated with decreased RV rates. The availability of specialty-trained physicians in PEM may allow a larger proportion of patients treated by physicians with honed clinical skills for the patient population. Data from a single pediatric center showed PEM fellowship-trained physicians had admission rates lower than those of their counterparts without subspecialty fellowship training.³⁴ The lower RV rate for this group in our study is especially interesting in light of previously reported lower admission rates at index visit in PEM trained physicians. With lower index admission rates, it may have been assumed that visits associated with PEM trained physician care would have an increased (rather than decreased) chance of RV. In addition, we noted the increased RVA rates were associated with longer waits to see a physician. These measures may indicate the effect of institutional access to robust resources (the ability to hire and support more specialty-trained physicians). These novel findings warrant further evaluation, particularly as our sample included only pediatric centers.

Our survey data demonstrated the impact that access to care has on ED RV rates. The ED medical directors indicated that limited access to outpatient appointments with PCPs and specialists was an important factor increasing ED RVs and a potential avenue for interventions. As the 2 open-ended questions addressed barriers and potential solutions, it is interesting that the respondents cited access to care and discharge instructions as the largest barriers and identified innovations in access to care and discharge education as important potential remedies.

This study demonstrated that, at the hospital level, ED RV quality measures are influenced by complex and varied SDHs that primarily reflect the characteristics of the patient populations served. Prior work has similarly highlighted the importance of gaining a rigorous understanding of other quality measures before widespread use, reporting, and dissemination of results.^11,35-38 With this in mind, as quality measures are developed and implemented, care should be taken to ensure they accurately and appropriately reflect the quality of care provided to the patient and are not more representative of other factors not directly within institutional control. These findings call into question the usefulness of ED RVs as a quality measure for comparing institutions.

Study Limitations

This study had several limitations. The PHIS dataset tracks only patients within each institution and does not include RVs to other EDs, which may account for a proportion of RVs.³⁹ Our survey response rate was 68% among medical directors, excluding 11 hospitals from analysis, which decreased the study’s power to detect differences that may be present between groups. In addition, the generalizability of our findings may be limited to tertiary-care children’s hospitals, as the PHIS dataset included only these types of healthcare facilities. We also included data only from the sites’ main EDs, and therefore cannot know if our results are applicable to satellite EDs. ED staffing of PEM physicians was analyzed using FTEs. However, number of clinical hours in 1 FTE may vary among sites, leading to imprecision in this hospital characteristic.

Hospitals with the highest RV rates served populations with a larger proportion of patients with government insurance and lower household income, and these hospitals had fewer PEM trained physicians. Variation in RV rates among hospitals may be indicative of the SDHs of their unique patient populations. ED revisit rates should be used cautiously in determining the quality of care of hospitals providing care to differing populations.

Disclosure

Nothing to report.

It has been theorized that the physiologic stress that hospitalized patients experience accounts for their transient vulnerability after discharge, or posthospital syndrome.¹ Their acute illness and life-habit changes during hospitalization result in continued impairment of physiologic systems after discharge, and this impairment might leave them more susceptible to new health threats.¹ However, the theory that the stress experienced after a hospitalization might be associated with readmission has never been investigated.

Four biomarkers of the hypothalamic-pituitary-adrenal (HPA) axis may help quantify posthospitalization stress: (1) midregional pro-adrenomedullin (ADM), a precursor reflecting adrenomedullin activity²; (2) copeptin (the C-terminal part of prepro-vasopressin, produced by the hypothalamus in response to stress^3,4), the level of which closely correlates to the vasopressin level but is more stable and lacks circadian rhythm fluctuations^5-7; (3) cortisol, released by the adrenal cortex in response to stress; and (4) prolactin, an indicator of HPA axis activity. These 4 stress biomarkers have been related to the severity, complications, or mortality of several diseases.^3,5,8-17 Besides explaining the hypothetical association between posthospitalization stress and readmission and death, these biomarkers might be valuable in predicting which patients are at higher risk for readmission. Indeed, many prediction models have been developed to identify those patients, but most of these models underperform, target only very specific populations, or have not been externally validated.¹⁸

We hypothesized that the hospitalization stress measured by biomarkers is associated with readmission or death after discharge. In a prospective cohort study, we evaluated the association between 4 stress biomarkers (ADM, copeptin, cortisol, prolactin) and 30-day unplanned readmissions and deaths after an acute-care medical hospitalization, and assessed their additive value to validated readmission prediction scores.

Study Design and Population

Our prospective cohort study included all consecutive patients aged ≥50 years and admitted to the department of general internal medicine at Fribourg Cantonal Hospital in Switzerland between April 8, 2013 and September 23, 2013. Exclusion criteria were discharge on day of admission; death before discharge; discharge to another division, another acute-care hospital, a rehabilitation clinic, or a palliative-care clinic; and refusal or inability to give informed consent. In this hypothesis-generating observational study, we collected data on a convenience sample of patients and did not calculate sample size before data collection. The study was approved by the local ethics committee, and all patients gave informed consent.

Outcomes

The primary outcome was the composite of first unplanned readmission (to any division of any acute-care hospital) or death within 30 days after discharge from index admission. We also included deaths that occurred after discharge, hypothesizing that patients who died may have been readmitted had they lived. The secondary outcome was the same as the primary, but the period was 90 days. Planned readmission was defined as scheduled hospitalization for nonemergent treatment (eg, chemotherapy) or investigation (eg, elective coronarography). All patients were called 6 months after discharge, and readmissions and deaths recorded. If a patient could not be reached directly, we called his or her next of kin, primary care physician, or nursing home, depending on availability. Furthermore, we checked electronic health records for any readmission or death recorded within the Fribourg hospital network, which includes all 3 acute-care hospitals (Fribourg, Riaz, Tavel) in the same canton (state).

Independent Variables

Stress biomarkers. We measured serum levels of 4 stress biomarkers (ADM, copeptin, cortisol, prolactin) at 8 am on an empty stomach on both day of admission and day of discharge. For a patient whose discharge decision was made after 8 hours for the same day, a blood sample was collected as soon as discharge was planned.

Clinical data. Collected data included demographics, history of hospitalization within 6 months before index admission, hospitalization diagnosis, and Charlson Comorbidity Index (CCI), which includes a list of medical conditions that are assigned a number of 1, 2, 3, or 6 points, according to their severity, and which has been associated with mortality.¹⁹

Causes of Admission, Unplanned Readmission, and Death

Causes of index admission, unplanned readmission, and death were obtained from medical records. We used our consensus opinion and a previous analysis²⁰ to classify these causes by body system, and added 2 categories, cancer and infection (both associated with readmission²⁰). The resulting 9 categories were (1) cancer, (2) respiratory disorder, (3) infectious disorder, (4) neurologic disorder (including dementia, psychiatric disorder, alcohol disorder, and intoxication), (5) gastrointestinal disorder, (6) osteoarticular disorder, (7) renal disorder, (8) cardiovascular disorder (including ischemic disease and heart failure), and (9) other.

Additional Performance With Existing Predictive Models

To better define the explanatory power of biomarkers to predict our outcome, we assessed the performance improvement of 2 validated readmission prediction scores by adding the stress biomarkers. As large effect sizes from additional predictors are needed to increase the power discrimination of a model, a significant performance improvement would further support the biomarkers’ important explanatory power. The 2 prediction scores tested were the LACE index (Length of stay, Admission Acuity, CCI, number of Emergency department visits within preceding 6 months²¹) and the HOSPITAL score (Hemoglobin level at discharge, discharge from Oncology service, Sodium level at discharge, any Procedure performed during index hospitalization, Index admission Type, number of Admissions within preceding 12 months, Length of stay). As we did not have an oncology service, we replaced “discharge from oncology service” with “active diagnosis of cancer.” “Length of stay” was tailored to the median in Switzerland (8 days instead of 5 days; Supplement Table 1).^22,23

Data Analysis

Continuous variables were presented as medians with interquartile ranges (IQRs) because of their non-normal distribution, and categorical variables were presented as frequencies and percentages. We compared medians using the nonparametric K-sample test on the equality of medians, and compared frequencies using the Pearson χ² test. The discriminatory power of each biomarker in predicting readmission and death was calculated with the area under the receiver operating characteristic (ROC) curve (AUROC), using serum levels at discharge to better reflect the postdischarge period. Cutoff levels were selected by taking the best compromise between sensitivity and specificity according to the ROC curves (point nearest top left corner).²⁴

Univariate logistic regression analysis was used to test the prediction of 30-day and 90-day unplanned readmission or death by each biomarker. We built 2 different multivariate models: one adjusting for age and LACE index points²¹ and the other adjusting for age and HOSPITAL score.^22,23

To explore any association between reduction of stress during hospitalization and postdischarge outcome, we additionally calculated for each biomarker the difference between admission and discharge serum levels and assessed its association with readmission or death by logistic regression analysis. Because of the modification of cortisol serum levels during corticosteroid therapy, we excluded patients who underwent systemic corticosteroid therapy before or during hospitalization for the cortisol analysis (n = 105/346). Patients with a missing biomarker level were excluded from the respective analyses: discharge (ADM, 28 patients; copeptin, 27; cortisol, 24; prolactin, 24) and admission (ADM, 12 patients; copeptin, 15; cortisol, 8; prolactin, 8).

To assess an additional value of the biomarkers to prediction scores, we assessed the accuracy of the HOSPITAL score and LACE index in their original versions^21,22 and after adding each biomarker. We used AUROC to assess the discriminatory power and used the method of DeLong et al.²⁵ to compare results with and without adding each biomarker. Calibration was evaluated by comparing Hosmer-Lemeshow goodness-of-fit tests (P > 0.05 indicates good fit). Risk reclassification was assessed by Net Reclassification Improvement (NRI),²⁶ quantifying how appropriately a new model reclassifies patients, compared with an old model. Basically, patients without outcome are assigned +1 if correctly reclassified to a lower risk category or –1 if incorrectly reclassified to a higher risk category. NRI_nonevent is the sum of all points/numbers of patients. Conversely, patients with outcome are assigned +1 if correctly reclassified to a higher risk category or –1 if incorrectly reclassified to a lower risk category. NRI_event is the sum of all points/numbers of patients. NRI_overall is the sum of NRI_event and NRI_nonevent ranging from –2 to 2, with a positive value indicating better classification with the new model.

Two-sided P < 0.05 was used for statistical significance. All statistical analyses were performed with Stata Release 13.0 (StataCorp).

Among the 530 patients admitted to the ward, 184 were excluded (120 meeting exclusion criteria, 64 unable to give consent, Figure). Among the 346 patients included, 11.6% (n = 40) had a 30-day unplanned readmission or death (37 were readmitted, 2 died during readmission, 3 died without readmission). Within 90 days, 26.6% (n = 92) had a readmission or death (84 were readmitted, 10 died during or after readmission, 8 died without readmission).

Clinical Characteristics

Table 1 lists the patients’ baseline characteristics. Median age was 73 years (IQR, 64-82 years). Of the 346 patients included, 172 (49.7%) were men. Median CCI was 7 (IQR, 5-9); according to this index, 310 patients (89.6%) had at least 2 comorbidities. Median length of stay was 7 days (IQR, 4-12 days).

Primary Diagnoses of Admission, Unplanned Readmission, and Death

The 3 main causes of index admission were cardiovascular disorder (n = 92), infectious disorder (n = 70), and neurologic disorder (n = 66). Table 2 lists the causes of readmissions and deaths. A same-diagnosis category between index admission and readmission was found in 17 (45.9%) of the 37 readmitted patients and in 3 (60%) of the 5 patients who died.

Biomarkers and 30-Day Unplanned Readmission or Death

AUROC was 0.53 (95% confidence interval [CI], 0.43-0.63) for ADM, 0.60 (95% CI, 0.50-0.70) for copeptin, 0.59 (95% CI, 0.44-0.73) for cortisol, and 0.56 (95% CI, 0.45-0.66) for prolactin. The difference between admission and discharge levels was not associated with unplanned readmission or death for any of the biomarkers (Supplemental Table 2).

ADM and readmission or death. Median ADM level was not different between patients with and without readmission or death (1.0 nmol/L in each case; P = 1.00). The best cutoff level for ADM was 2 nmol/L (sensitivity, 16.7%; specificity, 91.8%). At this level, ADM was associated with a nonstatistically significant 130% increased odds of 30-day readmission or death (P = 0.09; Table 3, Supplemental Table 3). Conversely, the association with the 90-day outcome was significant (P = 0.02; Table 3, Supplemental Table 4).

Copeptin and readmission or death. Patients with 30-day readmission or death had a higher median copeptin level at discharge than patients without (10.4 pmol/L vs 7.3 pmol/L; P = 0.03). At a copeptin level higher than 9 pmol/L (to convert to pg/mL, divide by 0.249; sensitivity, 66.7%; specificity, 59.7%), both 30-day readmission or death (adjusted odds ratio [OR], 2.69; 95% CI, 1.29-5.64; P = 0.009) and 90-day readmission or death (adjusted OR, 2.76; 95% CI, 1.56-4.88; P < 0.001) were nearly 3 times as likely (Table 3, Supplemental Tables 3 and 4).

Cortisol and readmission or death. Median cortisol was not statistically different between patients with and without the primary outcome (431 nmol/L vs 465 nmol/L; P = 0.72). At a cortisol level higher than 590 nmol/L (to convert to μg/dL, divide by 27.59; sensitivity, 54.6%; specificity, 76.4%), 30-day outcome was more than 3 times as likely (adjusted OR, 3.43; 95% CI, 1.36-8.65; P = 0.009; Table 3, Supplemental Table 3). At 90 days, only the model that adjusted for age and LACE index points remained statistically significant (P = 0.02; Table 3, Supplemental Table 4).

Prolactin and readmission or death. Median prolactin was not statistically different between patients with and without the primary outcome (15.1 μg/L vs 14.1 μg/L; P = 0.24). The best cutoff level for prolactin was 23 μg/L (to convert to mIU/L, divide by 0.05; sensitivity, 27.8%; specificity, 82.9%). Prolactin was associated with a nonstatistically significant increased odds of 30-day (P = 0.16) and 90-day (P = 0.24) readmission or death (Table 3, Supplemental Tables 3 and 4).

Additive Value of Biomarkers to HOSPITAL Score and LACE Index

The AUROC for the original HOSPITAL score, 0.70 (95% CI, 0.60-0.80), nonsignificantly increased to 0.76 after adding the biomarkers (P > 0.14). For the LACE index, AUROC was 0.59 (95% CI, 0.49-0.68), with a significant 0.10 increase with cortisol (P = 0.04) and a near significant increase with copeptin (P = 0.08). Calibration remained almost unchanged after adding the biomarkers to both models (Supplemental Table 5). NRI_overall was positive for all biomarkers, with statistical significance for copeptin added to the HOSPITAL score (0.47; 95% CI, 0.13-0.79) and for cortisol added to the LACE index (0.62; 95% CI, 0.15-1.06).

In this prospective cohort study, 30-day and 90-day unplanned readmission or death was nearly 3 times as likely for patients with high copeptin levels on discharge from an acute-care medical hospitalization, and 30-day readmission or death was more than 3 times as likely for patients with high cortisol levels. High ADM and prolactin levels were not consistently associated with readmission or death. Adding such biomarkers to readmission prediction models improved their performance.

These findings support the theory of posthospital syndrome,¹ which describes a period of vulnerability with increased stress after discharge from an acute-care hospitalization, and which may be associated with adverse outcome. The hormones cortisol and copeptin are strongly related to the stress response in humans.^4,5 As copeptin level has been associated with adverse prognosis for several disorders affecting a wide range of physiologic systems,^3,5,15,27 it may be a valuable biomarker of a stressful condition, even independent of the system affected by the acute illness, and its use may be widely generalizable, in contrast to predictive factors identified in other studies.^18,28,29

Although cortisol was independently associated with 30-day readmission or death, and may be an interesting biomarker and less expensive than copeptin, its measurement is limited in patients treated with systemic corticosteroids. Compared with cortisol, copeptin does not undergo diurnal variation, is less affected by corticosteroid therapy, and mirrors stress levels better.^5,7,30,31 Our results showed that, contrary to cortisol, copeptin was also associated with longer term outcome. High ADM level was associated with readmission or death at 90 days only; lack of a significant association at 30 days might be attributable to a lack of power (fewer outcomes at 30 days). Conversely, prolactin level was consistently not associated with outcome. Prolactin may be affected by many drugs that act on the dopaminergic system (eg, domperidone), and therefore its levels may be more difficult to interpret.

Levels of biomarkers were similar to those measured in patients without previously studied conditions (eg, myocardial infarction).^{5,8,10,13,14,16,17} In most previous analyses, levels were measured during a stressful event, whereas we measured them at discharge. Therefore, these biomarkers may constitute sensitive markers of remaining stress at discharge.

Our finding that copeptin level was independently associated with readmission or death supports its relevance as a possible simple measure of the risk of adverse postdischarge outcome, independently of disease type and independently of known predictors. Stress biomarkers may therefore be valuable predictors of which patients are at high risk. All these biomarkers can be measured within 30 minutes, extending their use beyond everyday practice, except for the possible need of an extra blood draw.

The most accurate and validated models are the HOSPITAL score (AUROC range, 0.68-0.77^23,32-36) and the LACE index (AUROC range, 0.56-0.68^23,34,35,37). Adding biomarkers to these models improved overall performance (up to 0.10 increase in AUROC), which is remarkable given that, once a particular level of discriminatory power is reached, extremely large effect sizes from additional markers are needed to increase AUROC.²⁶ Incremental improvement is objectively supported by positive NRI. Our results suggest biomarkers added to prediction models may improve identification of high-risk patients.

We found that less than 50% of the primary diagnoses belonged to the same diagnosis category at readmission and at index admission. This result is in line with previous findings that readmissions were related to the primary diagnosis at index admission in only 22% to 46% of cases,^20,38 and supports our study hypothesis that readmission is related to underlying stress factors often independent of the underlying illness.¹

Study Limitations and Strengths

Our study had some limitations. First, it was a single-center study with a limited sample size. However, we found significant results within the sample. Second, we could not adjust for drugs that were acting on the dopaminergic system and might have affected prolactin levels. However, such interactions would limit the use of this biomarker in clinical practice anyway. Third, we used specific cutoffs, which might decrease analytical power, in comparison with continuous analyses. However, we followed a recognized method²⁴ and found a significant association even with categorized levels. Furthermore, the distribution of biomarkers could not be normalized by logarithmic transformation, and cutoff values have the advantage of being integrable into score point systems (eg, HOSPITAL score, LACE index). Fourth, although in 2 models we found consistent associations with several potential confounders, we could not exclude residual confounding. Fifth, this study was not powered to assess the biomarkers’ predictive value for readmission and death, which might explain the lack of significant differences between AUROC with and without the biomarkers. For all these reasons, this study should be considered hypothesis-generating.

Our study also had its strengths. First, to our knowledge, this is the first study of the association between stress biomarkers at discharge and unplanned readmission or death. Second, the quality of our data was high, with a low percentage of missing biomarker levels. Third, we excluded planned readmissions. Fourth, we used an unselected medical patient population, which had the noteworthy advantage of widening the generalizability of results.

In this prospective cohort study, high copeptin and cortisol levels at discharge were significantly associated with increased odds, ranging from 2-fold to more than 3-fold, of unplanned readmission or death within 30 days after discharge from an internal medicine ward. This finding supports the theory that a physiologic stress that patients experience during hospitalization makes them more susceptible to new health threats (posthospital syndrome). These biomarkers, copeptin in particular, may help us better identify patients at high risk of early unplanned readmission or death.

Acknowledgment

Biomarker measurement was funded by the research fund of the Department of General Internal Medicine, Fribourg Cantonal Hospital, Fribourg, Switzerland.

Disclosure

Nothing to report.

It has been theorized that the physiologic stress that hospitalized patients experience accounts for their transient vulnerability after discharge, or posthospital syndrome.¹ Their acute illness and life-habit changes during hospitalization result in continued impairment of physiologic systems after discharge, and this impairment might leave them more susceptible to new health threats.¹ However, the theory that the stress experienced after a hospitalization might be associated with readmission has never been investigated.

Four biomarkers of the hypothalamic-pituitary-adrenal (HPA) axis may help quantify posthospitalization stress: (1) midregional pro-adrenomedullin (ADM), a precursor reflecting adrenomedullin activity²; (2) copeptin (the C-terminal part of prepro-vasopressin, produced by the hypothalamus in response to stress^3,4), the level of which closely correlates to the vasopressin level but is more stable and lacks circadian rhythm fluctuations^5-7; (3) cortisol, released by the adrenal cortex in response to stress; and (4) prolactin, an indicator of HPA axis activity. These 4 stress biomarkers have been related to the severity, complications, or mortality of several diseases.^3,5,8-17 Besides explaining the hypothetical association between posthospitalization stress and readmission and death, these biomarkers might be valuable in predicting which patients are at higher risk for readmission. Indeed, many prediction models have been developed to identify those patients, but most of these models underperform, target only very specific populations, or have not been externally validated.¹⁸

We hypothesized that the hospitalization stress measured by biomarkers is associated with readmission or death after discharge. In a prospective cohort study, we evaluated the association between 4 stress biomarkers (ADM, copeptin, cortisol, prolactin) and 30-day unplanned readmissions and deaths after an acute-care medical hospitalization, and assessed their additive value to validated readmission prediction scores.

Study Design and Population

Our prospective cohort study included all consecutive patients aged ≥50 years and admitted to the department of general internal medicine at Fribourg Cantonal Hospital in Switzerland between April 8, 2013 and September 23, 2013. Exclusion criteria were discharge on day of admission; death before discharge; discharge to another division, another acute-care hospital, a rehabilitation clinic, or a palliative-care clinic; and refusal or inability to give informed consent. In this hypothesis-generating observational study, we collected data on a convenience sample of patients and did not calculate sample size before data collection. The study was approved by the local ethics committee, and all patients gave informed consent.

Outcomes

The primary outcome was the composite of first unplanned readmission (to any division of any acute-care hospital) or death within 30 days after discharge from index admission. We also included deaths that occurred after discharge, hypothesizing that patients who died may have been readmitted had they lived. The secondary outcome was the same as the primary, but the period was 90 days. Planned readmission was defined as scheduled hospitalization for nonemergent treatment (eg, chemotherapy) or investigation (eg, elective coronarography). All patients were called 6 months after discharge, and readmissions and deaths recorded. If a patient could not be reached directly, we called his or her next of kin, primary care physician, or nursing home, depending on availability. Furthermore, we checked electronic health records for any readmission or death recorded within the Fribourg hospital network, which includes all 3 acute-care hospitals (Fribourg, Riaz, Tavel) in the same canton (state).

Independent Variables

Stress biomarkers. We measured serum levels of 4 stress biomarkers (ADM, copeptin, cortisol, prolactin) at 8 am on an empty stomach on both day of admission and day of discharge. For a patient whose discharge decision was made after 8 hours for the same day, a blood sample was collected as soon as discharge was planned.

Clinical data. Collected data included demographics, history of hospitalization within 6 months before index admission, hospitalization diagnosis, and Charlson Comorbidity Index (CCI), which includes a list of medical conditions that are assigned a number of 1, 2, 3, or 6 points, according to their severity, and which has been associated with mortality.¹⁹

Causes of Admission, Unplanned Readmission, and Death

Causes of index admission, unplanned readmission, and death were obtained from medical records. We used our consensus opinion and a previous analysis²⁰ to classify these causes by body system, and added 2 categories, cancer and infection (both associated with readmission²⁰). The resulting 9 categories were (1) cancer, (2) respiratory disorder, (3) infectious disorder, (4) neurologic disorder (including dementia, psychiatric disorder, alcohol disorder, and intoxication), (5) gastrointestinal disorder, (6) osteoarticular disorder, (7) renal disorder, (8) cardiovascular disorder (including ischemic disease and heart failure), and (9) other.

Additional Performance With Existing Predictive Models

To better define the explanatory power of biomarkers to predict our outcome, we assessed the performance improvement of 2 validated readmission prediction scores by adding the stress biomarkers. As large effect sizes from additional predictors are needed to increase the power discrimination of a model, a significant performance improvement would further support the biomarkers’ important explanatory power. The 2 prediction scores tested were the LACE index (Length of stay, Admission Acuity, CCI, number of Emergency department visits within preceding 6 months²¹) and the HOSPITAL score (Hemoglobin level at discharge, discharge from Oncology service, Sodium level at discharge, any Procedure performed during index hospitalization, Index admission Type, number of Admissions within preceding 12 months, Length of stay). As we did not have an oncology service, we replaced “discharge from oncology service” with “active diagnosis of cancer.” “Length of stay” was tailored to the median in Switzerland (8 days instead of 5 days; Supplement Table 1).^22,23

Data Analysis

Continuous variables were presented as medians with interquartile ranges (IQRs) because of their non-normal distribution, and categorical variables were presented as frequencies and percentages. We compared medians using the nonparametric K-sample test on the equality of medians, and compared frequencies using the Pearson χ² test. The discriminatory power of each biomarker in predicting readmission and death was calculated with the area under the receiver operating characteristic (ROC) curve (AUROC), using serum levels at discharge to better reflect the postdischarge period. Cutoff levels were selected by taking the best compromise between sensitivity and specificity according to the ROC curves (point nearest top left corner).²⁴

Univariate logistic regression analysis was used to test the prediction of 30-day and 90-day unplanned readmission or death by each biomarker. We built 2 different multivariate models: one adjusting for age and LACE index points²¹ and the other adjusting for age and HOSPITAL score.^22,23

To explore any association between reduction of stress during hospitalization and postdischarge outcome, we additionally calculated for each biomarker the difference between admission and discharge serum levels and assessed its association with readmission or death by logistic regression analysis. Because of the modification of cortisol serum levels during corticosteroid therapy, we excluded patients who underwent systemic corticosteroid therapy before or during hospitalization for the cortisol analysis (n = 105/346). Patients with a missing biomarker level were excluded from the respective analyses: discharge (ADM, 28 patients; copeptin, 27; cortisol, 24; prolactin, 24) and admission (ADM, 12 patients; copeptin, 15; cortisol, 8; prolactin, 8).

To assess an additional value of the biomarkers to prediction scores, we assessed the accuracy of the HOSPITAL score and LACE index in their original versions^21,22 and after adding each biomarker. We used AUROC to assess the discriminatory power and used the method of DeLong et al.²⁵ to compare results with and without adding each biomarker. Calibration was evaluated by comparing Hosmer-Lemeshow goodness-of-fit tests (P > 0.05 indicates good fit). Risk reclassification was assessed by Net Reclassification Improvement (NRI),²⁶ quantifying how appropriately a new model reclassifies patients, compared with an old model. Basically, patients without outcome are assigned +1 if correctly reclassified to a lower risk category or –1 if incorrectly reclassified to a higher risk category. NRI_nonevent is the sum of all points/numbers of patients. Conversely, patients with outcome are assigned +1 if correctly reclassified to a higher risk category or –1 if incorrectly reclassified to a lower risk category. NRI_event is the sum of all points/numbers of patients. NRI_overall is the sum of NRI_event and NRI_nonevent ranging from –2 to 2, with a positive value indicating better classification with the new model.

Two-sided P < 0.05 was used for statistical significance. All statistical analyses were performed with Stata Release 13.0 (StataCorp).

Among the 530 patients admitted to the ward, 184 were excluded (120 meeting exclusion criteria, 64 unable to give consent, Figure). Among the 346 patients included, 11.6% (n = 40) had a 30-day unplanned readmission or death (37 were readmitted, 2 died during readmission, 3 died without readmission). Within 90 days, 26.6% (n = 92) had a readmission or death (84 were readmitted, 10 died during or after readmission, 8 died without readmission).

Clinical Characteristics

Table 1 lists the patients’ baseline characteristics. Median age was 73 years (IQR, 64-82 years). Of the 346 patients included, 172 (49.7%) were men. Median CCI was 7 (IQR, 5-9); according to this index, 310 patients (89.6%) had at least 2 comorbidities. Median length of stay was 7 days (IQR, 4-12 days).

Primary Diagnoses of Admission, Unplanned Readmission, and Death

The 3 main causes of index admission were cardiovascular disorder (n = 92), infectious disorder (n = 70), and neurologic disorder (n = 66). Table 2 lists the causes of readmissions and deaths. A same-diagnosis category between index admission and readmission was found in 17 (45.9%) of the 37 readmitted patients and in 3 (60%) of the 5 patients who died.

Biomarkers and 30-Day Unplanned Readmission or Death

AUROC was 0.53 (95% confidence interval [CI], 0.43-0.63) for ADM, 0.60 (95% CI, 0.50-0.70) for copeptin, 0.59 (95% CI, 0.44-0.73) for cortisol, and 0.56 (95% CI, 0.45-0.66) for prolactin. The difference between admission and discharge levels was not associated with unplanned readmission or death for any of the biomarkers (Supplemental Table 2).

ADM and readmission or death. Median ADM level was not different between patients with and without readmission or death (1.0 nmol/L in each case; P = 1.00). The best cutoff level for ADM was 2 nmol/L (sensitivity, 16.7%; specificity, 91.8%). At this level, ADM was associated with a nonstatistically significant 130% increased odds of 30-day readmission or death (P = 0.09; Table 3, Supplemental Table 3). Conversely, the association with the 90-day outcome was significant (P = 0.02; Table 3, Supplemental Table 4).

Copeptin and readmission or death. Patients with 30-day readmission or death had a higher median copeptin level at discharge than patients without (10.4 pmol/L vs 7.3 pmol/L; P = 0.03). At a copeptin level higher than 9 pmol/L (to convert to pg/mL, divide by 0.249; sensitivity, 66.7%; specificity, 59.7%), both 30-day readmission or death (adjusted odds ratio [OR], 2.69; 95% CI, 1.29-5.64; P = 0.009) and 90-day readmission or death (adjusted OR, 2.76; 95% CI, 1.56-4.88; P < 0.001) were nearly 3 times as likely (Table 3, Supplemental Tables 3 and 4).

Cortisol and readmission or death. Median cortisol was not statistically different between patients with and without the primary outcome (431 nmol/L vs 465 nmol/L; P = 0.72). At a cortisol level higher than 590 nmol/L (to convert to μg/dL, divide by 27.59; sensitivity, 54.6%; specificity, 76.4%), 30-day outcome was more than 3 times as likely (adjusted OR, 3.43; 95% CI, 1.36-8.65; P = 0.009; Table 3, Supplemental Table 3). At 90 days, only the model that adjusted for age and LACE index points remained statistically significant (P = 0.02; Table 3, Supplemental Table 4).

Prolactin and readmission or death. Median prolactin was not statistically different between patients with and without the primary outcome (15.1 μg/L vs 14.1 μg/L; P = 0.24). The best cutoff level for prolactin was 23 μg/L (to convert to mIU/L, divide by 0.05; sensitivity, 27.8%; specificity, 82.9%). Prolactin was associated with a nonstatistically significant increased odds of 30-day (P = 0.16) and 90-day (P = 0.24) readmission or death (Table 3, Supplemental Tables 3 and 4).

Additive Value of Biomarkers to HOSPITAL Score and LACE Index

The AUROC for the original HOSPITAL score, 0.70 (95% CI, 0.60-0.80), nonsignificantly increased to 0.76 after adding the biomarkers (P > 0.14). For the LACE index, AUROC was 0.59 (95% CI, 0.49-0.68), with a significant 0.10 increase with cortisol (P = 0.04) and a near significant increase with copeptin (P = 0.08). Calibration remained almost unchanged after adding the biomarkers to both models (Supplemental Table 5). NRI_overall was positive for all biomarkers, with statistical significance for copeptin added to the HOSPITAL score (0.47; 95% CI, 0.13-0.79) and for cortisol added to the LACE index (0.62; 95% CI, 0.15-1.06).

In this prospective cohort study, 30-day and 90-day unplanned readmission or death was nearly 3 times as likely for patients with high copeptin levels on discharge from an acute-care medical hospitalization, and 30-day readmission or death was more than 3 times as likely for patients with high cortisol levels. High ADM and prolactin levels were not consistently associated with readmission or death. Adding such biomarkers to readmission prediction models improved their performance.

These findings support the theory of posthospital syndrome,¹ which describes a period of vulnerability with increased stress after discharge from an acute-care hospitalization, and which may be associated with adverse outcome. The hormones cortisol and copeptin are strongly related to the stress response in humans.^4,5 As copeptin level has been associated with adverse prognosis for several disorders affecting a wide range of physiologic systems,^3,5,15,27 it may be a valuable biomarker of a stressful condition, even independent of the system affected by the acute illness, and its use may be widely generalizable, in contrast to predictive factors identified in other studies.^18,28,29

Although cortisol was independently associated with 30-day readmission or death, and may be an interesting biomarker and less expensive than copeptin, its measurement is limited in patients treated with systemic corticosteroids. Compared with cortisol, copeptin does not undergo diurnal variation, is less affected by corticosteroid therapy, and mirrors stress levels better.^5,7,30,31 Our results showed that, contrary to cortisol, copeptin was also associated with longer term outcome. High ADM level was associated with readmission or death at 90 days only; lack of a significant association at 30 days might be attributable to a lack of power (fewer outcomes at 30 days). Conversely, prolactin level was consistently not associated with outcome. Prolactin may be affected by many drugs that act on the dopaminergic system (eg, domperidone), and therefore its levels may be more difficult to interpret.

Levels of biomarkers were similar to those measured in patients without previously studied conditions (eg, myocardial infarction).^{5,8,10,13,14,16,17} In most previous analyses, levels were measured during a stressful event, whereas we measured them at discharge. Therefore, these biomarkers may constitute sensitive markers of remaining stress at discharge.

Our finding that copeptin level was independently associated with readmission or death supports its relevance as a possible simple measure of the risk of adverse postdischarge outcome, independently of disease type and independently of known predictors. Stress biomarkers may therefore be valuable predictors of which patients are at high risk. All these biomarkers can be measured within 30 minutes, extending their use beyond everyday practice, except for the possible need of an extra blood draw.

The most accurate and validated models are the HOSPITAL score (AUROC range, 0.68-0.77^23,32-36) and the LACE index (AUROC range, 0.56-0.68^23,34,35,37). Adding biomarkers to these models improved overall performance (up to 0.10 increase in AUROC), which is remarkable given that, once a particular level of discriminatory power is reached, extremely large effect sizes from additional markers are needed to increase AUROC.²⁶ Incremental improvement is objectively supported by positive NRI. Our results suggest biomarkers added to prediction models may improve identification of high-risk patients.

We found that less than 50% of the primary diagnoses belonged to the same diagnosis category at readmission and at index admission. This result is in line with previous findings that readmissions were related to the primary diagnosis at index admission in only 22% to 46% of cases,^20,38 and supports our study hypothesis that readmission is related to underlying stress factors often independent of the underlying illness.¹

Study Limitations and Strengths

Our study had some limitations. First, it was a single-center study with a limited sample size. However, we found significant results within the sample. Second, we could not adjust for drugs that were acting on the dopaminergic system and might have affected prolactin levels. However, such interactions would limit the use of this biomarker in clinical practice anyway. Third, we used specific cutoffs, which might decrease analytical power, in comparison with continuous analyses. However, we followed a recognized method²⁴ and found a significant association even with categorized levels. Furthermore, the distribution of biomarkers could not be normalized by logarithmic transformation, and cutoff values have the advantage of being integrable into score point systems (eg, HOSPITAL score, LACE index). Fourth, although in 2 models we found consistent associations with several potential confounders, we could not exclude residual confounding. Fifth, this study was not powered to assess the biomarkers’ predictive value for readmission and death, which might explain the lack of significant differences between AUROC with and without the biomarkers. For all these reasons, this study should be considered hypothesis-generating.

Our study also had its strengths. First, to our knowledge, this is the first study of the association between stress biomarkers at discharge and unplanned readmission or death. Second, the quality of our data was high, with a low percentage of missing biomarker levels. Third, we excluded planned readmissions. Fourth, we used an unselected medical patient population, which had the noteworthy advantage of widening the generalizability of results.

In this prospective cohort study, high copeptin and cortisol levels at discharge were significantly associated with increased odds, ranging from 2-fold to more than 3-fold, of unplanned readmission or death within 30 days after discharge from an internal medicine ward. This finding supports the theory that a physiologic stress that patients experience during hospitalization makes them more susceptible to new health threats (posthospital syndrome). These biomarkers, copeptin in particular, may help us better identify patients at high risk of early unplanned readmission or death.

Acknowledgment

Biomarker measurement was funded by the research fund of the Department of General Internal Medicine, Fribourg Cantonal Hospital, Fribourg, Switzerland.

Disclosure

Nothing to report.

It has been theorized that the physiologic stress that hospitalized patients experience accounts for their transient vulnerability after discharge, or posthospital syndrome.¹ Their acute illness and life-habit changes during hospitalization result in continued impairment of physiologic systems after discharge, and this impairment might leave them more susceptible to new health threats.¹ However, the theory that the stress experienced after a hospitalization might be associated with readmission has never been investigated.

Four biomarkers of the hypothalamic-pituitary-adrenal (HPA) axis may help quantify posthospitalization stress: (1) midregional pro-adrenomedullin (ADM), a precursor reflecting adrenomedullin activity²; (2) copeptin (the C-terminal part of prepro-vasopressin, produced by the hypothalamus in response to stress^3,4), the level of which closely correlates to the vasopressin level but is more stable and lacks circadian rhythm fluctuations^5-7; (3) cortisol, released by the adrenal cortex in response to stress; and (4) prolactin, an indicator of HPA axis activity. These 4 stress biomarkers have been related to the severity, complications, or mortality of several diseases.^3,5,8-17 Besides explaining the hypothetical association between posthospitalization stress and readmission and death, these biomarkers might be valuable in predicting which patients are at higher risk for readmission. Indeed, many prediction models have been developed to identify those patients, but most of these models underperform, target only very specific populations, or have not been externally validated.¹⁸

We hypothesized that the hospitalization stress measured by biomarkers is associated with readmission or death after discharge. In a prospective cohort study, we evaluated the association between 4 stress biomarkers (ADM, copeptin, cortisol, prolactin) and 30-day unplanned readmissions and deaths after an acute-care medical hospitalization, and assessed their additive value to validated readmission prediction scores.

Study Design and Population

Our prospective cohort study included all consecutive patients aged ≥50 years and admitted to the department of general internal medicine at Fribourg Cantonal Hospital in Switzerland between April 8, 2013 and September 23, 2013. Exclusion criteria were discharge on day of admission; death before discharge; discharge to another division, another acute-care hospital, a rehabilitation clinic, or a palliative-care clinic; and refusal or inability to give informed consent. In this hypothesis-generating observational study, we collected data on a convenience sample of patients and did not calculate sample size before data collection. The study was approved by the local ethics committee, and all patients gave informed consent.

Outcomes

The primary outcome was the composite of first unplanned readmission (to any division of any acute-care hospital) or death within 30 days after discharge from index admission. We also included deaths that occurred after discharge, hypothesizing that patients who died may have been readmitted had they lived. The secondary outcome was the same as the primary, but the period was 90 days. Planned readmission was defined as scheduled hospitalization for nonemergent treatment (eg, chemotherapy) or investigation (eg, elective coronarography). All patients were called 6 months after discharge, and readmissions and deaths recorded. If a patient could not be reached directly, we called his or her next of kin, primary care physician, or nursing home, depending on availability. Furthermore, we checked electronic health records for any readmission or death recorded within the Fribourg hospital network, which includes all 3 acute-care hospitals (Fribourg, Riaz, Tavel) in the same canton (state).

Independent Variables

Stress biomarkers. We measured serum levels of 4 stress biomarkers (ADM, copeptin, cortisol, prolactin) at 8 am on an empty stomach on both day of admission and day of discharge. For a patient whose discharge decision was made after 8 hours for the same day, a blood sample was collected as soon as discharge was planned.

Clinical data. Collected data included demographics, history of hospitalization within 6 months before index admission, hospitalization diagnosis, and Charlson Comorbidity Index (CCI), which includes a list of medical conditions that are assigned a number of 1, 2, 3, or 6 points, according to their severity, and which has been associated with mortality.¹⁹

Causes of Admission, Unplanned Readmission, and Death

Causes of index admission, unplanned readmission, and death were obtained from medical records. We used our consensus opinion and a previous analysis²⁰ to classify these causes by body system, and added 2 categories, cancer and infection (both associated with readmission²⁰). The resulting 9 categories were (1) cancer, (2) respiratory disorder, (3) infectious disorder, (4) neurologic disorder (including dementia, psychiatric disorder, alcohol disorder, and intoxication), (5) gastrointestinal disorder, (6) osteoarticular disorder, (7) renal disorder, (8) cardiovascular disorder (including ischemic disease and heart failure), and (9) other.

Additional Performance With Existing Predictive Models

To better define the explanatory power of biomarkers to predict our outcome, we assessed the performance improvement of 2 validated readmission prediction scores by adding the stress biomarkers. As large effect sizes from additional predictors are needed to increase the power discrimination of a model, a significant performance improvement would further support the biomarkers’ important explanatory power. The 2 prediction scores tested were the LACE index (Length of stay, Admission Acuity, CCI, number of Emergency department visits within preceding 6 months²¹) and the HOSPITAL score (Hemoglobin level at discharge, discharge from Oncology service, Sodium level at discharge, any Procedure performed during index hospitalization, Index admission Type, number of Admissions within preceding 12 months, Length of stay). As we did not have an oncology service, we replaced “discharge from oncology service” with “active diagnosis of cancer.” “Length of stay” was tailored to the median in Switzerland (8 days instead of 5 days; Supplement Table 1).^22,23

Data Analysis

Continuous variables were presented as medians with interquartile ranges (IQRs) because of their non-normal distribution, and categorical variables were presented as frequencies and percentages. We compared medians using the nonparametric K-sample test on the equality of medians, and compared frequencies using the Pearson χ² test. The discriminatory power of each biomarker in predicting readmission and death was calculated with the area under the receiver operating characteristic (ROC) curve (AUROC), using serum levels at discharge to better reflect the postdischarge period. Cutoff levels were selected by taking the best compromise between sensitivity and specificity according to the ROC curves (point nearest top left corner).²⁴

Univariate logistic regression analysis was used to test the prediction of 30-day and 90-day unplanned readmission or death by each biomarker. We built 2 different multivariate models: one adjusting for age and LACE index points²¹ and the other adjusting for age and HOSPITAL score.^22,23

To explore any association between reduction of stress during hospitalization and postdischarge outcome, we additionally calculated for each biomarker the difference between admission and discharge serum levels and assessed its association with readmission or death by logistic regression analysis. Because of the modification of cortisol serum levels during corticosteroid therapy, we excluded patients who underwent systemic corticosteroid therapy before or during hospitalization for the cortisol analysis (n = 105/346). Patients with a missing biomarker level were excluded from the respective analyses: discharge (ADM, 28 patients; copeptin, 27; cortisol, 24; prolactin, 24) and admission (ADM, 12 patients; copeptin, 15; cortisol, 8; prolactin, 8).

To assess an additional value of the biomarkers to prediction scores, we assessed the accuracy of the HOSPITAL score and LACE index in their original versions^21,22 and after adding each biomarker. We used AUROC to assess the discriminatory power and used the method of DeLong et al.²⁵ to compare results with and without adding each biomarker. Calibration was evaluated by comparing Hosmer-Lemeshow goodness-of-fit tests (P > 0.05 indicates good fit). Risk reclassification was assessed by Net Reclassification Improvement (NRI),²⁶ quantifying how appropriately a new model reclassifies patients, compared with an old model. Basically, patients without outcome are assigned +1 if correctly reclassified to a lower risk category or –1 if incorrectly reclassified to a higher risk category. NRI_nonevent is the sum of all points/numbers of patients. Conversely, patients with outcome are assigned +1 if correctly reclassified to a higher risk category or –1 if incorrectly reclassified to a lower risk category. NRI_event is the sum of all points/numbers of patients. NRI_overall is the sum of NRI_event and NRI_nonevent ranging from –2 to 2, with a positive value indicating better classification with the new model.

Two-sided P < 0.05 was used for statistical significance. All statistical analyses were performed with Stata Release 13.0 (StataCorp).

Among the 530 patients admitted to the ward, 184 were excluded (120 meeting exclusion criteria, 64 unable to give consent, Figure). Among the 346 patients included, 11.6% (n = 40) had a 30-day unplanned readmission or death (37 were readmitted, 2 died during readmission, 3 died without readmission). Within 90 days, 26.6% (n = 92) had a readmission or death (84 were readmitted, 10 died during or after readmission, 8 died without readmission).

Clinical Characteristics

Table 1 lists the patients’ baseline characteristics. Median age was 73 years (IQR, 64-82 years). Of the 346 patients included, 172 (49.7%) were men. Median CCI was 7 (IQR, 5-9); according to this index, 310 patients (89.6%) had at least 2 comorbidities. Median length of stay was 7 days (IQR, 4-12 days).

Primary Diagnoses of Admission, Unplanned Readmission, and Death

The 3 main causes of index admission were cardiovascular disorder (n = 92), infectious disorder (n = 70), and neurologic disorder (n = 66). Table 2 lists the causes of readmissions and deaths. A same-diagnosis category between index admission and readmission was found in 17 (45.9%) of the 37 readmitted patients and in 3 (60%) of the 5 patients who died.

Biomarkers and 30-Day Unplanned Readmission or Death

AUROC was 0.53 (95% confidence interval [CI], 0.43-0.63) for ADM, 0.60 (95% CI, 0.50-0.70) for copeptin, 0.59 (95% CI, 0.44-0.73) for cortisol, and 0.56 (95% CI, 0.45-0.66) for prolactin. The difference between admission and discharge levels was not associated with unplanned readmission or death for any of the biomarkers (Supplemental Table 2).

ADM and readmission or death. Median ADM level was not different between patients with and without readmission or death (1.0 nmol/L in each case; P = 1.00). The best cutoff level for ADM was 2 nmol/L (sensitivity, 16.7%; specificity, 91.8%). At this level, ADM was associated with a nonstatistically significant 130% increased odds of 30-day readmission or death (P = 0.09; Table 3, Supplemental Table 3). Conversely, the association with the 90-day outcome was significant (P = 0.02; Table 3, Supplemental Table 4).

Copeptin and readmission or death. Patients with 30-day readmission or death had a higher median copeptin level at discharge than patients without (10.4 pmol/L vs 7.3 pmol/L; P = 0.03). At a copeptin level higher than 9 pmol/L (to convert to pg/mL, divide by 0.249; sensitivity, 66.7%; specificity, 59.7%), both 30-day readmission or death (adjusted odds ratio [OR], 2.69; 95% CI, 1.29-5.64; P = 0.009) and 90-day readmission or death (adjusted OR, 2.76; 95% CI, 1.56-4.88; P < 0.001) were nearly 3 times as likely (Table 3, Supplemental Tables 3 and 4).

Cortisol and readmission or death. Median cortisol was not statistically different between patients with and without the primary outcome (431 nmol/L vs 465 nmol/L; P = 0.72). At a cortisol level higher than 590 nmol/L (to convert to μg/dL, divide by 27.59; sensitivity, 54.6%; specificity, 76.4%), 30-day outcome was more than 3 times as likely (adjusted OR, 3.43; 95% CI, 1.36-8.65; P = 0.009; Table 3, Supplemental Table 3). At 90 days, only the model that adjusted for age and LACE index points remained statistically significant (P = 0.02; Table 3, Supplemental Table 4).

Prolactin and readmission or death. Median prolactin was not statistically different between patients with and without the primary outcome (15.1 μg/L vs 14.1 μg/L; P = 0.24). The best cutoff level for prolactin was 23 μg/L (to convert to mIU/L, divide by 0.05; sensitivity, 27.8%; specificity, 82.9%). Prolactin was associated with a nonstatistically significant increased odds of 30-day (P = 0.16) and 90-day (P = 0.24) readmission or death (Table 3, Supplemental Tables 3 and 4).

Additive Value of Biomarkers to HOSPITAL Score and LACE Index

The AUROC for the original HOSPITAL score, 0.70 (95% CI, 0.60-0.80), nonsignificantly increased to 0.76 after adding the biomarkers (P > 0.14). For the LACE index, AUROC was 0.59 (95% CI, 0.49-0.68), with a significant 0.10 increase with cortisol (P = 0.04) and a near significant increase with copeptin (P = 0.08). Calibration remained almost unchanged after adding the biomarkers to both models (Supplemental Table 5). NRI_overall was positive for all biomarkers, with statistical significance for copeptin added to the HOSPITAL score (0.47; 95% CI, 0.13-0.79) and for cortisol added to the LACE index (0.62; 95% CI, 0.15-1.06).

In this prospective cohort study, 30-day and 90-day unplanned readmission or death was nearly 3 times as likely for patients with high copeptin levels on discharge from an acute-care medical hospitalization, and 30-day readmission or death was more than 3 times as likely for patients with high cortisol levels. High ADM and prolactin levels were not consistently associated with readmission or death. Adding such biomarkers to readmission prediction models improved their performance.

These findings support the theory of posthospital syndrome,¹ which describes a period of vulnerability with increased stress after discharge from an acute-care hospitalization, and which may be associated with adverse outcome. The hormones cortisol and copeptin are strongly related to the stress response in humans.^4,5 As copeptin level has been associated with adverse prognosis for several disorders affecting a wide range of physiologic systems,^3,5,15,27 it may be a valuable biomarker of a stressful condition, even independent of the system affected by the acute illness, and its use may be widely generalizable, in contrast to predictive factors identified in other studies.^18,28,29

Although cortisol was independently associated with 30-day readmission or death, and may be an interesting biomarker and less expensive than copeptin, its measurement is limited in patients treated with systemic corticosteroids. Compared with cortisol, copeptin does not undergo diurnal variation, is less affected by corticosteroid therapy, and mirrors stress levels better.^5,7,30,31 Our results showed that, contrary to cortisol, copeptin was also associated with longer term outcome. High ADM level was associated with readmission or death at 90 days only; lack of a significant association at 30 days might be attributable to a lack of power (fewer outcomes at 30 days). Conversely, prolactin level was consistently not associated with outcome. Prolactin may be affected by many drugs that act on the dopaminergic system (eg, domperidone), and therefore its levels may be more difficult to interpret.

Levels of biomarkers were similar to those measured in patients without previously studied conditions (eg, myocardial infarction).^{5,8,10,13,14,16,17} In most previous analyses, levels were measured during a stressful event, whereas we measured them at discharge. Therefore, these biomarkers may constitute sensitive markers of remaining stress at discharge.

Our finding that copeptin level was independently associated with readmission or death supports its relevance as a possible simple measure of the risk of adverse postdischarge outcome, independently of disease type and independently of known predictors. Stress biomarkers may therefore be valuable predictors of which patients are at high risk. All these biomarkers can be measured within 30 minutes, extending their use beyond everyday practice, except for the possible need of an extra blood draw.

The most accurate and validated models are the HOSPITAL score (AUROC range, 0.68-0.77^23,32-36) and the LACE index (AUROC range, 0.56-0.68^23,34,35,37). Adding biomarkers to these models improved overall performance (up to 0.10 increase in AUROC), which is remarkable given that, once a particular level of discriminatory power is reached, extremely large effect sizes from additional markers are needed to increase AUROC.²⁶ Incremental improvement is objectively supported by positive NRI. Our results suggest biomarkers added to prediction models may improve identification of high-risk patients.

We found that less than 50% of the primary diagnoses belonged to the same diagnosis category at readmission and at index admission. This result is in line with previous findings that readmissions were related to the primary diagnosis at index admission in only 22% to 46% of cases,^20,38 and supports our study hypothesis that readmission is related to underlying stress factors often independent of the underlying illness.¹

Study Limitations and Strengths

Our study had some limitations. First, it was a single-center study with a limited sample size. However, we found significant results within the sample. Second, we could not adjust for drugs that were acting on the dopaminergic system and might have affected prolactin levels. However, such interactions would limit the use of this biomarker in clinical practice anyway. Third, we used specific cutoffs, which might decrease analytical power, in comparison with continuous analyses. However, we followed a recognized method²⁴ and found a significant association even with categorized levels. Furthermore, the distribution of biomarkers could not be normalized by logarithmic transformation, and cutoff values have the advantage of being integrable into score point systems (eg, HOSPITAL score, LACE index). Fourth, although in 2 models we found consistent associations with several potential confounders, we could not exclude residual confounding. Fifth, this study was not powered to assess the biomarkers’ predictive value for readmission and death, which might explain the lack of significant differences between AUROC with and without the biomarkers. For all these reasons, this study should be considered hypothesis-generating.

Our study also had its strengths. First, to our knowledge, this is the first study of the association between stress biomarkers at discharge and unplanned readmission or death. Second, the quality of our data was high, with a low percentage of missing biomarker levels. Third, we excluded planned readmissions. Fourth, we used an unselected medical patient population, which had the noteworthy advantage of widening the generalizability of results.

In this prospective cohort study, high copeptin and cortisol levels at discharge were significantly associated with increased odds, ranging from 2-fold to more than 3-fold, of unplanned readmission or death within 30 days after discharge from an internal medicine ward. This finding supports the theory that a physiologic stress that patients experience during hospitalization makes them more susceptible to new health threats (posthospital syndrome). These biomarkers, copeptin in particular, may help us better identify patients at high risk of early unplanned readmission or death.

Acknowledgment

Biomarker measurement was funded by the research fund of the Department of General Internal Medicine, Fribourg Cantonal Hospital, Fribourg, Switzerland.

Disclosure

Nothing to report.

Communication among healthcare professionals is essential for high-quality patient care. However, communication is difficult in hospitals because of heavy workloads, rapidly evolving plans of care, and geographic dispersion of team members. When hospital-based professionals are not in the same place at the same time, they rely on technology to communicate. Pagers have historically been used to support communication in hospitals, but are limited in their capabilities. Several recent small studies have shown that some physicians have started using standard text messaging on smartphones for patient care–related (PCR) messages.^1-3 Although potentially enhancing clinician efficiency, use of standard text messaging for PCR messages raises concern about security risks related to transmission of protected health information. Addressing these concerns are emerging secure mobile messaging applications designed for PCR communication. Although recent studies suggest these applications are well received by users, the adoption rate is largely unknown.^4,5

We conducted a study to see if there was a shift in use of hospital-based communication technologies under way. We surveyed a national sample of hospital-based clinicians to characterize current use of communication technologies, assess potential risks and perceptions related to use of standard text messaging for PCR messages, and characterize the adoption of secure mobile messaging applications designed for PCR communication.

Study Design

The study was a cross-sectional survey of hospitalists—physicians and advanced practice providers whose primary professional focus is care of hospitalized patients. We studied hospitalists because of their role in coordinating care for complex medical patients and because prior studies identified communication as a major component of their work.^6,7 The Northwestern University Institutional Review Board deemed this study exempt.

Survey Instrument

Four investigators (Drs. O’Leary, Liebovitz, Wu, and Reddy) with expertise in interprofessional communication and information technology created a draft survey based in part on results of prior studies assessing clinicians’ use of smartphones and standard text messaging for PCR communication.^1,3 In the first section of the survey, we asked respondents which technologies were provided by their organization and which technologies they used for PCR communication. In the second section, we asked respondents about their use and perceptions of standard text messaging for PCR communication. In the third section, we asked about implementation and adoption of secure mobile messaging applications at their hospital. In the fourth and final section, we asked for demographic information.

We randomly selected 8 attendees of the 2015 Midwest Hospital Medicine Conference and invited them to participate in a focus group that would review a paper version of the draft survey and recommend revisions. Using the group’s feedback, we revised the ordinal response scale for questions related to standard text messaging and made other minor edits. We then created an Internet-based version of the survey and pilot-tested it with 8 hospitalists from 4 diverse hospitalist groups within the Northwestern Medicine Health System. We made additional minor edits based on pilot-test feedback.

Sampling Strategy

We used the largest hospitalist database maintained by the Society of Hospital Medicine (SHM). This database includes information on more than 28,000 individuals, representing SHM members and nonmembers who had participated in organizational events. In addition to clinically active hospitalists, the database includes non-hospitalists and clinically inactive hospitalists. We used this database to try to capture the largest possible number of potentially eligible hospitalists.

Survey Administration

We administered the survey in collaboration with SHM staff. E-mails that included a link to the survey on the Survey Monkey website were sent by SHM staff to individuals within the database. These e-mails were sent through Real Magnet, an e-mail marketing platform⁸ that allowed the SHM staff to determine the number of individuals who received and opened the e-mail and the number who clicked on the survey link. To try to promote participation, we offered respondents the chance to enter a lottery to win one of four $50 gift certificates. The initial e-mail was sent in April 2016, a reminder in May 2016, and a final reminder in July 2016.

Data Analysis

We calculated descriptive statistics of participants’ demographic characteristics. We estimated nonresponse bias by comparing demographic characteristics across waves of respondents using analysis of variance, t tests, and χ² tests. This method is based on the finding that characteristics of late respondents often resemble those of nonrespondents.⁹ We collapsed response categories for communication technologies to simplify interpretation. For example, numeric pagers, alphanumeric pagers, and 2-way pagers were collapsed into a pagers category. We used t tests and χ² tests to assess for associations between receipt of standard text messages for PCR communication and respondents’ age, sex, race, professional type, hospital size, practice location, and hospital teaching status. Similarly, we used t tests and χ² tests to explore associations between implementation of secure mobile messaging application and respondents’ age, sex, race, professional type, hospital size, practice location, and hospital teaching status. All statistical analyses were performed with Stata Release 11.2 (StataCorp).

Participant Characteristics

Overall, the survey link was sent to 28,870 e-mail addresses. Addresses for which e-mails were undeliverable or for which the e-mail was never opened were excluded, yielding a total of 5,786 eligible respondents in the sample. After rejecting 42 clinically inactive individuals, 70 individuals who responded to only the initial item, and 27 duplicates, a total of 620 participant surveys were included in the final analysis. The adjusted response rate was 11.0%.

As shown in Table 1, mean (SD) respondent age was 42.9 (10.0) years, nearly half of the respondents were female, nearly a third were of nonwhite race, an overwhelming majority were physicians, and workplaces were in a variety of hospital settings. The sample size used to calculate demographic characteristics varied from 538 to 549 because of missing data for these items. We found no significant differences in demographic characteristics of respondents across the 3 survey waves, suggesting a lack of survey response bias (Supplemental Table).

Provision and Use of Communication Technologies for PCR Communication

Pagers were provided to the majority of respondents by their hospitals (79.8%, 495/620). Other devices were provided much less frequently, with 21.0% (130/620) reporting their organization provided a smartphone, 20.2% (125/620) a mobile phone, and 4.4% (27/620) a hands-free communication device. Organizations provided no device to 8.2% (51/620) of respondents and an “other” device to 5.5% (34/620).

An overwhelming majority used multiple technologies to receive PCR communication, with 17.7% (110/620) of respondents indicating use of 2 technologies, 22.7% (141/620) use of 3 technologies, and 49.4% (306/620) use of more than 3 technologies. The distribution of the most common ways respondents received PCR communication is shown in Table 2. Pagers were the most common form of technology, with 49.0% (304/620) indicating this was the primary way they received PCR communication. Being called on a mobile phone provided by the organization was the second most common form of receiving PCR communication (11.0%, 68/620), followed by standard text messaging (9.5%, 59/620) and mobile secure messaging using an application approved by the organization (9.0%, 56/620).

Participants’ Experiences With Standard Text Messaging for PCR Communication

Participants’ experiences with standard text messaging for PCR communication are summarized in Table 3. Overall, 65.1% (369/ 567) of respondents reported receiving standard text messages for PCR communication at least once per week when on clinical duty, and 52.9% (300/567) received standard text messages at least once per day.

Overall, 21.5% (122/567) of respondents received standard text messages that included individually identifiable information at least once per day, and 41.3% (234/567) received messages that included some identifiable information (eg, patient initials, room number) at least once per day. About one-fifth of respondents (21.0%, 119/567) indicated receiving standard text messages for urgent clinical issues at least once per day. Receipt of standard text messages for a patient for whom the respondent was no longer providing care, delays in receipt of messages, messages missed because smartphones were set to vibrate, and receipt of messages when not on clinical duty occurred, but less frequently. We found no significant associations between receipt of PCR standard text messages once or more per day and respondents’ age, sex, race, professional type, hospital size, or hospital teaching status. A higher percentage of respondents in the South (63.2%, 96/152) and West (57.9%, 70/121) reported receipt of at least 1 PCR standard text message per day, compared with respondents in the Northeast (51.9%, 54/104), Midwest (45.2%, 61/135), and other (25.0%, 4/16) (P = 0.003).

Senders of PCR standard text messages. Of respondents who received standard text messages for PCR communication at least once per week, a majority reported receiving messages from physicians in the same specialty (88.6%, 327/369) and from physicians in other specialties (71.3%, 263/369). A minority of respondents reported receiving messages from nurses (35.0%, 129/369), social workers (30.6%, 113/369), and pharmacists (27.9%, 103/369).

Perceptions among users. Of respondents who received standard text messages for PCR communication at least once per week, an overwhelming majority agreed or strongly agreed that use of standard text messaging allowed them to provide better care (81.7%, 295/361) and made them more efficient (87.3%, 315/361). A majority also agreed or strongly agreed that standard text messaging posed a risk to the privacy and confidentiality of patient information (56.4%, 203/360), and nearly a third indicated that standard text messaging posed a risk to the timely receipt of messages by the correct individual (27.6%, 100/362). Overall, a large majority agreed or strongly agreed that the benefits of using standard text messaging for PCR communication outweighed the risks (85.0%, 306/360).

Adoption of Organization-Approved Secure Mobile Messaging Applications

Participants’ reported adoption of organization-approved secure mobile messaging applications is shown in the Figure. About one-fourth (26.6%, 146/549) of respondents reported that their organization had implemented a secure messaging application and that some clinicians were using it, whereas relatively few (7.3%, 40/549) reported that their organization had implemented an application that was being used by most clinicians. A substantial portion of respondents (21.3%, 117/549) were not sure whether their organization was planning to implement a secure mobile messaging application for PCR communication. We found no significant associations between partial or nearly full implementation of a secure mobile messaging application and respondents’ age, sex, race, professional type, hospital size, or practice location. A lower percentage of respondents in major teaching hospitals (28.0%, 67/239) reported partial or nearly full implementation of a secure mobile messaging application, compared with respondents from teaching hospitals (39.6%, 74/187) and nonteaching hospitals (39.2%, 40/102) (P = 0.02).

We found that pagers were the technology most commonly used by hospital-based clinicians, but also that a majority have used standard text messaging for PCR communication, and that relatively few hospitals had fully implemented secure mobile messaging applications. Our findings reveal a wide range of technology use and suggest an evolution to support communication among healthcare professionals.

The persistence of pagers as the technology most commonly provided by hospitals and used by clinicians for communication is noteworthy in that pagers are limited in their capabilities, typically not allowing a response to the message sender or the ability to forward a message, and often not allowing the ability to send messages to multiple recipients. The continued heavy use of pagers may be explained by their relatively low cost, especially compared with investment in new technologies, and reliable receipt of messages, even in areas with no cell phone service or WiFi signal. Furthermore, hospitals’ providing pagers allows for oversight, directory creation, and the potential for integration into other information systems. In 2 recent studies, inpatient paging communication was evaluated in depth. Carlile et al.¹⁰ found that the majority of pages requested a response, requiring an interruption in physician workflow to initiate a callback. Kummerow Broman et al.¹¹ similarly found that a majority of pages requested a callback; they also found a high volume of nonurgent messages. With pager use, a high volume of messages, many of which require a response but are nonurgent, makes for a highly interruptive workflow.

That more than half of our hospital-based clinicians received standard text messages for PCR communication once or more per day is consistent with other, smaller studies. Kuhlmann et al.¹ surveyed 97 pediatric hospitalists and found that a majority sent and received work-related text messages. Prochaska et al.² surveyed 131 residents and found that standard text messaging was the communication method preferred by the majority of residents. Similar to these studies, our study found that receipt of standard text messages that included protected health information was fairly common. However, we identified additional risks related to standard text messaging. One-fifth of our respondents received standard text messages for urgent clinical issues once or more per day, and many respondents reported occasional receipt of messages regarding a patient for whom they were no longer providing care and receipt of messages when not on clinical duty. The usual inability to automate forwarding of standard text messages to another clinician creates the potential for clinically important messages to be delayed or missed. These risks have not been reported in the literature, and we think healthcare systems may not be fully aware of them. Our findings suggest that many clinicians have migrated from pagers to standard text messaging for the enhanced efficiency, and they perceive that the benefit of improved efficiency outweighs the risks to protected health information and the delay in receipt of clinically important messages by the correct individual.

Secure mobile messaging applications seem to address the limitations of both pagers and standard text messaging. Secure mobile messaging applications typically allow message response, message forwarding, multiple recipients, directory creation, the potential to create escalation schemes for nonresponse, and integration with other information systems, including electronic health records. Although several hospitals have developed their own systems,^4,12,13 most hospitals likely will purchase a vendor-based system. We found that a minority of hospitals had implemented a secure messaging application, and even fewer had most of their clinicians using it. Although little research has been conducted on these applications, studies suggest they are well received by users.^4,5 Given that paging communication studies have found a large portion of pages are sent by nurses and other non-physician team members, secure mobile messaging applications should allow for direct message exchange with all professionals caring for a patient.^10,11 Furthermore, hospitals will need to ensure adequate cell phone and WiFi signal strength throughout their facilities to ensure reliable and timely delivery of messages.

Our study had several limitations. We used a large database to conduct a national survey but had a low response rate and some drop-off of responses within surveys. Our sample reflected respondent diversity, and our analyses of demographic characteristics found no significant differences across survey response waves. Unfortunately, we did not have nonrespondents’ characteristics and therefore could not compare them with respondents’. It is possible that nonrespondents may have had different practices related to use of communication technology, especially in light of the fact that the survey was conducted by e-mail. However, given our finding that use of standard text messaging was comparable to that in other studies,^1,2 and given the similarity of respondents’ characteristics across response waves, our findings likely were not affected by nonresponse bias.⁹ Last, we used a survey that had not been validated. However, this survey was created by experts in interprofessional collaboration and information technology, was informed by prior studies, and was iteratively refined during pretesting and pilot testing.

Pagers remain the technology most commonly used by hospital-based clinicians, but a majority also use standard text messaging for PCR communication, and relatively few hospitals have fully implemented secure mobile messaging applications. The wide range of technologies used suggests an evolution of methods to support communication among healthcare professionals. An optimized system will improve communication efficiency while ensuring the security of their patients’ information and the timely receipt of that information by the intended clinician.

Acknowledgment

The authors thank the Society of Hospital Medicine and the society staff who helped administer the survey, especially Mr. Ethan Gray.

Disclosure

Nothing to report.

Communication among healthcare professionals is essential for high-quality patient care. However, communication is difficult in hospitals because of heavy workloads, rapidly evolving plans of care, and geographic dispersion of team members. When hospital-based professionals are not in the same place at the same time, they rely on technology to communicate. Pagers have historically been used to support communication in hospitals, but are limited in their capabilities. Several recent small studies have shown that some physicians have started using standard text messaging on smartphones for patient care–related (PCR) messages.^1-3 Although potentially enhancing clinician efficiency, use of standard text messaging for PCR messages raises concern about security risks related to transmission of protected health information. Addressing these concerns are emerging secure mobile messaging applications designed for PCR communication. Although recent studies suggest these applications are well received by users, the adoption rate is largely unknown.^4,5

We conducted a study to see if there was a shift in use of hospital-based communication technologies under way. We surveyed a national sample of hospital-based clinicians to characterize current use of communication technologies, assess potential risks and perceptions related to use of standard text messaging for PCR messages, and characterize the adoption of secure mobile messaging applications designed for PCR communication.

Study Design

The study was a cross-sectional survey of hospitalists—physicians and advanced practice providers whose primary professional focus is care of hospitalized patients. We studied hospitalists because of their role in coordinating care for complex medical patients and because prior studies identified communication as a major component of their work.^6,7 The Northwestern University Institutional Review Board deemed this study exempt.

Survey Instrument

Four investigators (Drs. O’Leary, Liebovitz, Wu, and Reddy) with expertise in interprofessional communication and information technology created a draft survey based in part on results of prior studies assessing clinicians’ use of smartphones and standard text messaging for PCR communication.^1,3 In the first section of the survey, we asked respondents which technologies were provided by their organization and which technologies they used for PCR communication. In the second section, we asked respondents about their use and perceptions of standard text messaging for PCR communication. In the third section, we asked about implementation and adoption of secure mobile messaging applications at their hospital. In the fourth and final section, we asked for demographic information.

We randomly selected 8 attendees of the 2015 Midwest Hospital Medicine Conference and invited them to participate in a focus group that would review a paper version of the draft survey and recommend revisions. Using the group’s feedback, we revised the ordinal response scale for questions related to standard text messaging and made other minor edits. We then created an Internet-based version of the survey and pilot-tested it with 8 hospitalists from 4 diverse hospitalist groups within the Northwestern Medicine Health System. We made additional minor edits based on pilot-test feedback.

Sampling Strategy

We used the largest hospitalist database maintained by the Society of Hospital Medicine (SHM). This database includes information on more than 28,000 individuals, representing SHM members and nonmembers who had participated in organizational events. In addition to clinically active hospitalists, the database includes non-hospitalists and clinically inactive hospitalists. We used this database to try to capture the largest possible number of potentially eligible hospitalists.

Survey Administration

We administered the survey in collaboration with SHM staff. E-mails that included a link to the survey on the Survey Monkey website were sent by SHM staff to individuals within the database. These e-mails were sent through Real Magnet, an e-mail marketing platform⁸ that allowed the SHM staff to determine the number of individuals who received and opened the e-mail and the number who clicked on the survey link. To try to promote participation, we offered respondents the chance to enter a lottery to win one of four $50 gift certificates. The initial e-mail was sent in April 2016, a reminder in May 2016, and a final reminder in July 2016.

Data Analysis

We calculated descriptive statistics of participants’ demographic characteristics. We estimated nonresponse bias by comparing demographic characteristics across waves of respondents using analysis of variance, t tests, and χ² tests. This method is based on the finding that characteristics of late respondents often resemble those of nonrespondents.⁹ We collapsed response categories for communication technologies to simplify interpretation. For example, numeric pagers, alphanumeric pagers, and 2-way pagers were collapsed into a pagers category. We used t tests and χ² tests to assess for associations between receipt of standard text messages for PCR communication and respondents’ age, sex, race, professional type, hospital size, practice location, and hospital teaching status. Similarly, we used t tests and χ² tests to explore associations between implementation of secure mobile messaging application and respondents’ age, sex, race, professional type, hospital size, practice location, and hospital teaching status. All statistical analyses were performed with Stata Release 11.2 (StataCorp).

Participant Characteristics

Overall, the survey link was sent to 28,870 e-mail addresses. Addresses for which e-mails were undeliverable or for which the e-mail was never opened were excluded, yielding a total of 5,786 eligible respondents in the sample. After rejecting 42 clinically inactive individuals, 70 individuals who responded to only the initial item, and 27 duplicates, a total of 620 participant surveys were included in the final analysis. The adjusted response rate was 11.0%.

As shown in Table 1, mean (SD) respondent age was 42.9 (10.0) years, nearly half of the respondents were female, nearly a third were of nonwhite race, an overwhelming majority were physicians, and workplaces were in a variety of hospital settings. The sample size used to calculate demographic characteristics varied from 538 to 549 because of missing data for these items. We found no significant differences in demographic characteristics of respondents across the 3 survey waves, suggesting a lack of survey response bias (Supplemental Table).

Provision and Use of Communication Technologies for PCR Communication

Pagers were provided to the majority of respondents by their hospitals (79.8%, 495/620). Other devices were provided much less frequently, with 21.0% (130/620) reporting their organization provided a smartphone, 20.2% (125/620) a mobile phone, and 4.4% (27/620) a hands-free communication device. Organizations provided no device to 8.2% (51/620) of respondents and an “other” device to 5.5% (34/620).

An overwhelming majority used multiple technologies to receive PCR communication, with 17.7% (110/620) of respondents indicating use of 2 technologies, 22.7% (141/620) use of 3 technologies, and 49.4% (306/620) use of more than 3 technologies. The distribution of the most common ways respondents received PCR communication is shown in Table 2. Pagers were the most common form of technology, with 49.0% (304/620) indicating this was the primary way they received PCR communication. Being called on a mobile phone provided by the organization was the second most common form of receiving PCR communication (11.0%, 68/620), followed by standard text messaging (9.5%, 59/620) and mobile secure messaging using an application approved by the organization (9.0%, 56/620).

Participants’ Experiences With Standard Text Messaging for PCR Communication

Participants’ experiences with standard text messaging for PCR communication are summarized in Table 3. Overall, 65.1% (369/ 567) of respondents reported receiving standard text messages for PCR communication at least once per week when on clinical duty, and 52.9% (300/567) received standard text messages at least once per day.

Overall, 21.5% (122/567) of respondents received standard text messages that included individually identifiable information at least once per day, and 41.3% (234/567) received messages that included some identifiable information (eg, patient initials, room number) at least once per day. About one-fifth of respondents (21.0%, 119/567) indicated receiving standard text messages for urgent clinical issues at least once per day. Receipt of standard text messages for a patient for whom the respondent was no longer providing care, delays in receipt of messages, messages missed because smartphones were set to vibrate, and receipt of messages when not on clinical duty occurred, but less frequently. We found no significant associations between receipt of PCR standard text messages once or more per day and respondents’ age, sex, race, professional type, hospital size, or hospital teaching status. A higher percentage of respondents in the South (63.2%, 96/152) and West (57.9%, 70/121) reported receipt of at least 1 PCR standard text message per day, compared with respondents in the Northeast (51.9%, 54/104), Midwest (45.2%, 61/135), and other (25.0%, 4/16) (P = 0.003).

Senders of PCR standard text messages. Of respondents who received standard text messages for PCR communication at least once per week, a majority reported receiving messages from physicians in the same specialty (88.6%, 327/369) and from physicians in other specialties (71.3%, 263/369). A minority of respondents reported receiving messages from nurses (35.0%, 129/369), social workers (30.6%, 113/369), and pharmacists (27.9%, 103/369).

Perceptions among users. Of respondents who received standard text messages for PCR communication at least once per week, an overwhelming majority agreed or strongly agreed that use of standard text messaging allowed them to provide better care (81.7%, 295/361) and made them more efficient (87.3%, 315/361). A majority also agreed or strongly agreed that standard text messaging posed a risk to the privacy and confidentiality of patient information (56.4%, 203/360), and nearly a third indicated that standard text messaging posed a risk to the timely receipt of messages by the correct individual (27.6%, 100/362). Overall, a large majority agreed or strongly agreed that the benefits of using standard text messaging for PCR communication outweighed the risks (85.0%, 306/360).

Adoption of Organization-Approved Secure Mobile Messaging Applications

Participants’ reported adoption of organization-approved secure mobile messaging applications is shown in the Figure. About one-fourth (26.6%, 146/549) of respondents reported that their organization had implemented a secure messaging application and that some clinicians were using it, whereas relatively few (7.3%, 40/549) reported that their organization had implemented an application that was being used by most clinicians. A substantial portion of respondents (21.3%, 117/549) were not sure whether their organization was planning to implement a secure mobile messaging application for PCR communication. We found no significant associations between partial or nearly full implementation of a secure mobile messaging application and respondents’ age, sex, race, professional type, hospital size, or practice location. A lower percentage of respondents in major teaching hospitals (28.0%, 67/239) reported partial or nearly full implementation of a secure mobile messaging application, compared with respondents from teaching hospitals (39.6%, 74/187) and nonteaching hospitals (39.2%, 40/102) (P = 0.02).

We found that pagers were the technology most commonly used by hospital-based clinicians, but also that a majority have used standard text messaging for PCR communication, and that relatively few hospitals had fully implemented secure mobile messaging applications. Our findings reveal a wide range of technology use and suggest an evolution to support communication among healthcare professionals.

The persistence of pagers as the technology most commonly provided by hospitals and used by clinicians for communication is noteworthy in that pagers are limited in their capabilities, typically not allowing a response to the message sender or the ability to forward a message, and often not allowing the ability to send messages to multiple recipients. The continued heavy use of pagers may be explained by their relatively low cost, especially compared with investment in new technologies, and reliable receipt of messages, even in areas with no cell phone service or WiFi signal. Furthermore, hospitals’ providing pagers allows for oversight, directory creation, and the potential for integration into other information systems. In 2 recent studies, inpatient paging communication was evaluated in depth. Carlile et al.¹⁰ found that the majority of pages requested a response, requiring an interruption in physician workflow to initiate a callback. Kummerow Broman et al.¹¹ similarly found that a majority of pages requested a callback; they also found a high volume of nonurgent messages. With pager use, a high volume of messages, many of which require a response but are nonurgent, makes for a highly interruptive workflow.

That more than half of our hospital-based clinicians received standard text messages for PCR communication once or more per day is consistent with other, smaller studies. Kuhlmann et al.¹ surveyed 97 pediatric hospitalists and found that a majority sent and received work-related text messages. Prochaska et al.² surveyed 131 residents and found that standard text messaging was the communication method preferred by the majority of residents. Similar to these studies, our study found that receipt of standard text messages that included protected health information was fairly common. However, we identified additional risks related to standard text messaging. One-fifth of our respondents received standard text messages for urgent clinical issues once or more per day, and many respondents reported occasional receipt of messages regarding a patient for whom they were no longer providing care and receipt of messages when not on clinical duty. The usual inability to automate forwarding of standard text messages to another clinician creates the potential for clinically important messages to be delayed or missed. These risks have not been reported in the literature, and we think healthcare systems may not be fully aware of them. Our findings suggest that many clinicians have migrated from pagers to standard text messaging for the enhanced efficiency, and they perceive that the benefit of improved efficiency outweighs the risks to protected health information and the delay in receipt of clinically important messages by the correct individual.

Secure mobile messaging applications seem to address the limitations of both pagers and standard text messaging. Secure mobile messaging applications typically allow message response, message forwarding, multiple recipients, directory creation, the potential to create escalation schemes for nonresponse, and integration with other information systems, including electronic health records. Although several hospitals have developed their own systems,^4,12,13 most hospitals likely will purchase a vendor-based system. We found that a minority of hospitals had implemented a secure messaging application, and even fewer had most of their clinicians using it. Although little research has been conducted on these applications, studies suggest they are well received by users.^4,5 Given that paging communication studies have found a large portion of pages are sent by nurses and other non-physician team members, secure mobile messaging applications should allow for direct message exchange with all professionals caring for a patient.^10,11 Furthermore, hospitals will need to ensure adequate cell phone and WiFi signal strength throughout their facilities to ensure reliable and timely delivery of messages.

Our study had several limitations. We used a large database to conduct a national survey but had a low response rate and some drop-off of responses within surveys. Our sample reflected respondent diversity, and our analyses of demographic characteristics found no significant differences across survey response waves. Unfortunately, we did not have nonrespondents’ characteristics and therefore could not compare them with respondents’. It is possible that nonrespondents may have had different practices related to use of communication technology, especially in light of the fact that the survey was conducted by e-mail. However, given our finding that use of standard text messaging was comparable to that in other studies,^1,2 and given the similarity of respondents’ characteristics across response waves, our findings likely were not affected by nonresponse bias.⁹ Last, we used a survey that had not been validated. However, this survey was created by experts in interprofessional collaboration and information technology, was informed by prior studies, and was iteratively refined during pretesting and pilot testing.

Pagers remain the technology most commonly used by hospital-based clinicians, but a majority also use standard text messaging for PCR communication, and relatively few hospitals have fully implemented secure mobile messaging applications. The wide range of technologies used suggests an evolution of methods to support communication among healthcare professionals. An optimized system will improve communication efficiency while ensuring the security of their patients’ information and the timely receipt of that information by the intended clinician.

Acknowledgment

The authors thank the Society of Hospital Medicine and the society staff who helped administer the survey, especially Mr. Ethan Gray.

Disclosure

Nothing to report.

Communication among healthcare professionals is essential for high-quality patient care. However, communication is difficult in hospitals because of heavy workloads, rapidly evolving plans of care, and geographic dispersion of team members. When hospital-based professionals are not in the same place at the same time, they rely on technology to communicate. Pagers have historically been used to support communication in hospitals, but are limited in their capabilities. Several recent small studies have shown that some physicians have started using standard text messaging on smartphones for patient care–related (PCR) messages.^1-3 Although potentially enhancing clinician efficiency, use of standard text messaging for PCR messages raises concern about security risks related to transmission of protected health information. Addressing these concerns are emerging secure mobile messaging applications designed for PCR communication. Although recent studies suggest these applications are well received by users, the adoption rate is largely unknown.^4,5

We conducted a study to see if there was a shift in use of hospital-based communication technologies under way. We surveyed a national sample of hospital-based clinicians to characterize current use of communication technologies, assess potential risks and perceptions related to use of standard text messaging for PCR messages, and characterize the adoption of secure mobile messaging applications designed for PCR communication.

Study Design

The study was a cross-sectional survey of hospitalists—physicians and advanced practice providers whose primary professional focus is care of hospitalized patients. We studied hospitalists because of their role in coordinating care for complex medical patients and because prior studies identified communication as a major component of their work.^6,7 The Northwestern University Institutional Review Board deemed this study exempt.

Survey Instrument

Four investigators (Drs. O’Leary, Liebovitz, Wu, and Reddy) with expertise in interprofessional communication and information technology created a draft survey based in part on results of prior studies assessing clinicians’ use of smartphones and standard text messaging for PCR communication.^1,3 In the first section of the survey, we asked respondents which technologies were provided by their organization and which technologies they used for PCR communication. In the second section, we asked respondents about their use and perceptions of standard text messaging for PCR communication. In the third section, we asked about implementation and adoption of secure mobile messaging applications at their hospital. In the fourth and final section, we asked for demographic information.

We randomly selected 8 attendees of the 2015 Midwest Hospital Medicine Conference and invited them to participate in a focus group that would review a paper version of the draft survey and recommend revisions. Using the group’s feedback, we revised the ordinal response scale for questions related to standard text messaging and made other minor edits. We then created an Internet-based version of the survey and pilot-tested it with 8 hospitalists from 4 diverse hospitalist groups within the Northwestern Medicine Health System. We made additional minor edits based on pilot-test feedback.

Sampling Strategy

We used the largest hospitalist database maintained by the Society of Hospital Medicine (SHM). This database includes information on more than 28,000 individuals, representing SHM members and nonmembers who had participated in organizational events. In addition to clinically active hospitalists, the database includes non-hospitalists and clinically inactive hospitalists. We used this database to try to capture the largest possible number of potentially eligible hospitalists.

Survey Administration

We administered the survey in collaboration with SHM staff. E-mails that included a link to the survey on the Survey Monkey website were sent by SHM staff to individuals within the database. These e-mails were sent through Real Magnet, an e-mail marketing platform⁸ that allowed the SHM staff to determine the number of individuals who received and opened the e-mail and the number who clicked on the survey link. To try to promote participation, we offered respondents the chance to enter a lottery to win one of four $50 gift certificates. The initial e-mail was sent in April 2016, a reminder in May 2016, and a final reminder in July 2016.

Data Analysis

We calculated descriptive statistics of participants’ demographic characteristics. We estimated nonresponse bias by comparing demographic characteristics across waves of respondents using analysis of variance, t tests, and χ² tests. This method is based on the finding that characteristics of late respondents often resemble those of nonrespondents.⁹ We collapsed response categories for communication technologies to simplify interpretation. For example, numeric pagers, alphanumeric pagers, and 2-way pagers were collapsed into a pagers category. We used t tests and χ² tests to assess for associations between receipt of standard text messages for PCR communication and respondents’ age, sex, race, professional type, hospital size, practice location, and hospital teaching status. Similarly, we used t tests and χ² tests to explore associations between implementation of secure mobile messaging application and respondents’ age, sex, race, professional type, hospital size, practice location, and hospital teaching status. All statistical analyses were performed with Stata Release 11.2 (StataCorp).

Participant Characteristics

Overall, the survey link was sent to 28,870 e-mail addresses. Addresses for which e-mails were undeliverable or for which the e-mail was never opened were excluded, yielding a total of 5,786 eligible respondents in the sample. After rejecting 42 clinically inactive individuals, 70 individuals who responded to only the initial item, and 27 duplicates, a total of 620 participant surveys were included in the final analysis. The adjusted response rate was 11.0%.

As shown in Table 1, mean (SD) respondent age was 42.9 (10.0) years, nearly half of the respondents were female, nearly a third were of nonwhite race, an overwhelming majority were physicians, and workplaces were in a variety of hospital settings. The sample size used to calculate demographic characteristics varied from 538 to 549 because of missing data for these items. We found no significant differences in demographic characteristics of respondents across the 3 survey waves, suggesting a lack of survey response bias (Supplemental Table).

Provision and Use of Communication Technologies for PCR Communication

Pagers were provided to the majority of respondents by their hospitals (79.8%, 495/620). Other devices were provided much less frequently, with 21.0% (130/620) reporting their organization provided a smartphone, 20.2% (125/620) a mobile phone, and 4.4% (27/620) a hands-free communication device. Organizations provided no device to 8.2% (51/620) of respondents and an “other” device to 5.5% (34/620).

An overwhelming majority used multiple technologies to receive PCR communication, with 17.7% (110/620) of respondents indicating use of 2 technologies, 22.7% (141/620) use of 3 technologies, and 49.4% (306/620) use of more than 3 technologies. The distribution of the most common ways respondents received PCR communication is shown in Table 2. Pagers were the most common form of technology, with 49.0% (304/620) indicating this was the primary way they received PCR communication. Being called on a mobile phone provided by the organization was the second most common form of receiving PCR communication (11.0%, 68/620), followed by standard text messaging (9.5%, 59/620) and mobile secure messaging using an application approved by the organization (9.0%, 56/620).

Participants’ Experiences With Standard Text Messaging for PCR Communication

Participants’ experiences with standard text messaging for PCR communication are summarized in Table 3. Overall, 65.1% (369/ 567) of respondents reported receiving standard text messages for PCR communication at least once per week when on clinical duty, and 52.9% (300/567) received standard text messages at least once per day.

Overall, 21.5% (122/567) of respondents received standard text messages that included individually identifiable information at least once per day, and 41.3% (234/567) received messages that included some identifiable information (eg, patient initials, room number) at least once per day. About one-fifth of respondents (21.0%, 119/567) indicated receiving standard text messages for urgent clinical issues at least once per day. Receipt of standard text messages for a patient for whom the respondent was no longer providing care, delays in receipt of messages, messages missed because smartphones were set to vibrate, and receipt of messages when not on clinical duty occurred, but less frequently. We found no significant associations between receipt of PCR standard text messages once or more per day and respondents’ age, sex, race, professional type, hospital size, or hospital teaching status. A higher percentage of respondents in the South (63.2%, 96/152) and West (57.9%, 70/121) reported receipt of at least 1 PCR standard text message per day, compared with respondents in the Northeast (51.9%, 54/104), Midwest (45.2%, 61/135), and other (25.0%, 4/16) (P = 0.003).

Senders of PCR standard text messages. Of respondents who received standard text messages for PCR communication at least once per week, a majority reported receiving messages from physicians in the same specialty (88.6%, 327/369) and from physicians in other specialties (71.3%, 263/369). A minority of respondents reported receiving messages from nurses (35.0%, 129/369), social workers (30.6%, 113/369), and pharmacists (27.9%, 103/369).

Perceptions among users. Of respondents who received standard text messages for PCR communication at least once per week, an overwhelming majority agreed or strongly agreed that use of standard text messaging allowed them to provide better care (81.7%, 295/361) and made them more efficient (87.3%, 315/361). A majority also agreed or strongly agreed that standard text messaging posed a risk to the privacy and confidentiality of patient information (56.4%, 203/360), and nearly a third indicated that standard text messaging posed a risk to the timely receipt of messages by the correct individual (27.6%, 100/362). Overall, a large majority agreed or strongly agreed that the benefits of using standard text messaging for PCR communication outweighed the risks (85.0%, 306/360).

Adoption of Organization-Approved Secure Mobile Messaging Applications

Participants’ reported adoption of organization-approved secure mobile messaging applications is shown in the Figure. About one-fourth (26.6%, 146/549) of respondents reported that their organization had implemented a secure messaging application and that some clinicians were using it, whereas relatively few (7.3%, 40/549) reported that their organization had implemented an application that was being used by most clinicians. A substantial portion of respondents (21.3%, 117/549) were not sure whether their organization was planning to implement a secure mobile messaging application for PCR communication. We found no significant associations between partial or nearly full implementation of a secure mobile messaging application and respondents’ age, sex, race, professional type, hospital size, or practice location. A lower percentage of respondents in major teaching hospitals (28.0%, 67/239) reported partial or nearly full implementation of a secure mobile messaging application, compared with respondents from teaching hospitals (39.6%, 74/187) and nonteaching hospitals (39.2%, 40/102) (P = 0.02).

We found that pagers were the technology most commonly used by hospital-based clinicians, but also that a majority have used standard text messaging for PCR communication, and that relatively few hospitals had fully implemented secure mobile messaging applications. Our findings reveal a wide range of technology use and suggest an evolution to support communication among healthcare professionals.

The persistence of pagers as the technology most commonly provided by hospitals and used by clinicians for communication is noteworthy in that pagers are limited in their capabilities, typically not allowing a response to the message sender or the ability to forward a message, and often not allowing the ability to send messages to multiple recipients. The continued heavy use of pagers may be explained by their relatively low cost, especially compared with investment in new technologies, and reliable receipt of messages, even in areas with no cell phone service or WiFi signal. Furthermore, hospitals’ providing pagers allows for oversight, directory creation, and the potential for integration into other information systems. In 2 recent studies, inpatient paging communication was evaluated in depth. Carlile et al.¹⁰ found that the majority of pages requested a response, requiring an interruption in physician workflow to initiate a callback. Kummerow Broman et al.¹¹ similarly found that a majority of pages requested a callback; they also found a high volume of nonurgent messages. With pager use, a high volume of messages, many of which require a response but are nonurgent, makes for a highly interruptive workflow.

That more than half of our hospital-based clinicians received standard text messages for PCR communication once or more per day is consistent with other, smaller studies. Kuhlmann et al.¹ surveyed 97 pediatric hospitalists and found that a majority sent and received work-related text messages. Prochaska et al.² surveyed 131 residents and found that standard text messaging was the communication method preferred by the majority of residents. Similar to these studies, our study found that receipt of standard text messages that included protected health information was fairly common. However, we identified additional risks related to standard text messaging. One-fifth of our respondents received standard text messages for urgent clinical issues once or more per day, and many respondents reported occasional receipt of messages regarding a patient for whom they were no longer providing care and receipt of messages when not on clinical duty. The usual inability to automate forwarding of standard text messages to another clinician creates the potential for clinically important messages to be delayed or missed. These risks have not been reported in the literature, and we think healthcare systems may not be fully aware of them. Our findings suggest that many clinicians have migrated from pagers to standard text messaging for the enhanced efficiency, and they perceive that the benefit of improved efficiency outweighs the risks to protected health information and the delay in receipt of clinically important messages by the correct individual.

Secure mobile messaging applications seem to address the limitations of both pagers and standard text messaging. Secure mobile messaging applications typically allow message response, message forwarding, multiple recipients, directory creation, the potential to create escalation schemes for nonresponse, and integration with other information systems, including electronic health records. Although several hospitals have developed their own systems,^4,12,13 most hospitals likely will purchase a vendor-based system. We found that a minority of hospitals had implemented a secure messaging application, and even fewer had most of their clinicians using it. Although little research has been conducted on these applications, studies suggest they are well received by users.^4,5 Given that paging communication studies have found a large portion of pages are sent by nurses and other non-physician team members, secure mobile messaging applications should allow for direct message exchange with all professionals caring for a patient.^10,11 Furthermore, hospitals will need to ensure adequate cell phone and WiFi signal strength throughout their facilities to ensure reliable and timely delivery of messages.

Our study had several limitations. We used a large database to conduct a national survey but had a low response rate and some drop-off of responses within surveys. Our sample reflected respondent diversity, and our analyses of demographic characteristics found no significant differences across survey response waves. Unfortunately, we did not have nonrespondents’ characteristics and therefore could not compare them with respondents’. It is possible that nonrespondents may have had different practices related to use of communication technology, especially in light of the fact that the survey was conducted by e-mail. However, given our finding that use of standard text messaging was comparable to that in other studies,^1,2 and given the similarity of respondents’ characteristics across response waves, our findings likely were not affected by nonresponse bias.⁹ Last, we used a survey that had not been validated. However, this survey was created by experts in interprofessional collaboration and information technology, was informed by prior studies, and was iteratively refined during pretesting and pilot testing.

Pagers remain the technology most commonly used by hospital-based clinicians, but a majority also use standard text messaging for PCR communication, and relatively few hospitals have fully implemented secure mobile messaging applications. The wide range of technologies used suggests an evolution of methods to support communication among healthcare professionals. An optimized system will improve communication efficiency while ensuring the security of their patients’ information and the timely receipt of that information by the intended clinician.

Acknowledgment

The authors thank the Society of Hospital Medicine and the society staff who helped administer the survey, especially Mr. Ethan Gray.

Disclosure

Nothing to report.

In 2014, the United States spent $3 trillion on healthcare; hospitalization consumed 32% of these expenditures.¹ Today, Medicare patients account for over 50% of hospital days and over 30% of all hospital discharges in the United States.² Despite this staggering financial burden, hospitalization of older adults often results in poor patient outcomes.^3-6 The exponential growth of the hospitalist movement, from 350 hospitalists nationwide in 1995 to over 44,000 in 2014, has become the key strategy for providing care to hospitalized geriatric patients.^7-10 Most of these hospitalists have not received geriatric training.^11-15

There is growing evidence that a geriatric approach, emphasizing multidisciplinary management of the complex needs of older patients, leads to improved outcomes. Geriatric Evaluation and Management Units (GEMUs), such as Acute Care for Elderly (ACE) models, have demonstrated significant decreases in functional decline, institutionalization, and death in randomized controlled trials.^16,17 Multidisciplinary, nonunit based efforts, such as the mobile acute care of elderly (MACE), proactive consultation models (Sennour/Counsell), and the Hospital Elder Life Program (HELP), have demonstrated success in preventing adverse events and decreasing length of stay (LOS).^17-20

However, these models have not been systematically implemented due to challenges in generalizability and replicability in diverse settings. To address this concern, an alternative approach must be developed to widely “generalize” geriatric expertise throughout hospitals, regardless of their location, size, and resources. This initiative will require systematic integration of evidence-based decision support tools for the standardization of clinical management in hospitalized older adults.²¹

The 1998 Assessing Care of Vulnerable Elders (ACOVE) project developed a standardized tool to measure and evaluate the quality of care by using a comprehensive set of quality indicators (QIs) to improve the care of “vulnerable elders” (VEs) at a high risk for functional and cognitive decline and death.^22-24 The latest systematic review concludes that, although many studies have used ACOVE as an assessment tool of quality, there has been a dearth of studies investigating the ACOVE QIs as an intervention to improve patient care.²⁵

Our study investigated the role of ACOVE as an intervention by using the QIs as a standardized checklist in the acute care setting. We selected the 4 most commonly encountered QIs in the hospital setting, namely venous thrombosis prophylaxis (VTE), indwelling bladder catheter, mobilization, and delirium evaluation, in order to test the feasibility and impact of systematically implementing these ACOVE QIs as a therapeutic intervention for all hospitalized older adults.

This study (IRB #13-644B) was conducted using a prospective intervention with a nonequivalent control group design comprised of retrospective chart data from May 1, 2014, to June 30, 2015. Process and outcome variables were extracted from electronic medical records ([EMR], Sunrise Clinical Manager [SCM]) of 2,396 patients, with 530 patients in the intervention unit and 1,866 on the control units, at a large academic tertiary center operating in the greater New York metropolitan area. Our study investigated the role of ACOVE as an intervention to improve patient care by using selected QIs as a standardized checklist tool in the acute care setting. Of the original 30 hospital-specific QIs, our study focused on the care of older adults admitted to the medicine service.²⁶ We selected commonly encountered QIs, with the objective of testing the feasibility and impact of implementing the ACOVE QIs as an intervention to improve care of hospitalized older adults. This intervention consisted of applying the checklist tool, constructed with 4 selected ACOVE QIs and administered daily during interdisciplinary rounds, namely: 2 general “medical” indicators, VTE prophylaxis and indwelling bladder catheters, and 2 “geriatric”-focused indicators, mobilization and delirium evaluation.

Subject matter experts (hospitalists, geriatricians, researchers, administrators, and nurses) reviewed the ACOVE QIs and agreed upon the adaptation of the QIs from a quality measure assessment into a feasible and acceptable intervention checklist tool (Table 1). The checklist was reviewed during daily interdisciplinary rounds for all patients 75 years and older. While ACOVE defined vulnerable elders by using the Vulnerable Elder Screen (VES), we wanted to apply this intervention more broadly to all hospitalized older adults who are most at risk for poor outcomes.²⁷ Patients admitted to the intensive care unit, inpatient psychiatry, inpatient leukemia/lymphoma, and surgical services were excluded.

Daily interdisciplinary rounds are held on every one of the five 40-bed medical units; they last approximately 1 hour, and consist of a lead hospitalist, nurse manager, nurse practitioners, case managers, and the nursing staff. During interdisciplinary rounds, nurses present the case to the team members who then discuss the care plan. These 5 medical units did not differ in terms of patient characteristics or staffing patterns; the intervention unit was chosen simply for logistical reasons, in that the principal investigator (PI) had been assigned to this unit prior to study start-up.

Prior to the intervention, LS held an education session for staff on the intervention unit staff (who participated on interdisciplinary rounds) to explain the concept of the ACOVE QI initiative and describe the four QIs selected for the study. Three subsequent educational sessions were held during the first week of the intervention, with new incoming staff receiving a brief individual educational session. The staff demonstrated significant knowledge improvement after session completion (pre/post mean score 70.6% vs 90.0%; P < .0001).

The Clinical Information System for the Health System EMR, The Eclipsys SCM, has alerts with different levels of severity from “soft” (user must acknowledge a recommendation) to “hard” (requires an action in order to proceed).

To measure compliance of the quality indicators, we collected the following variables:

QI 1: VTE prophylaxis

Through SCM, we collected type of VTE prophylaxis ordered (pharmacologic and/or mechanical) as well as start and stop dates for all agents. International normalized ratio levels were checked for patients receiving warfarin. Days of compliance were calculated.

QI 2: Indwelling Bladder Catheters

SCM data were collected on catheter entry and discontinuation dates, the presence of an indication, and order renewal for bladder catheter at least every 3 days.

QI 3: Mobilization

Ambulation status prior to admission was extracted from nursing documentation completed on admission to the medical ward. Patients documented as bedfast were categorized as nonambulatory prior to admission. Nursing documentation of activity level and amount of feet ambulated per nursing shift were collected. In addition, hospital day of physical therapy (PT) order and hospital days with PT performed were charted. Compliance with QI 3 in patients documented as ambulatory prior to hospital admission was recorded as present if there was a PT order within 48 hours of admission.

QI 4: Delirium Evaluation

During daily rounds, the hospitalist (PI) questioned nurses about delirium evaluation, using the first feature of the Confusion Assessment Method (CAM) as well as the “single question in delirium,” namely, “Is there evidence of an acute change in mental status from the patient’s baseline?” and “Do you think [name of patient] has been more confused lately?”^28,29 Because EMR does not contain a specified field for delirium screening and documentation, and patients are not routinely included in rounds, documentation with QI 4 was recorded using the “key words” method as described in the work by Puelle et al.³⁰ To extract SCM key words, nursing documentation of the “cognitive/perceptual/neurological exam” section of the EMR on admission and on all subsequent documentation (once per shift) was retrieved to identify acute changes in mental status (eg, “altered mental status, delirium/delirious, alert and oriented X 3, confused/confusion, disoriented, lethargy/lethargic”).³⁰ In addition, nurses were asked to activate an SCM parameter, “Acute Confusion” SCM parameter, in the nursing documentation section, which includes potential risk factors for confusion.

In addition to QI compliance, we collected LOS, discharge disposition, and 30-day readmission data.

Generalized linear mixed models (GLMM) for binary clustered (ie, hierarchical) data were used to estimate compliance rates (ie, nurse adherence) for each group (intervention group or control group) in the postintervention period, along with their corresponding 95% confidence intervals. GLMM was used to account for the hierarchical structure of the data: nursing units within a hospital. In order to calculate the Charlson Comorbidity Index, we extracted past medical history from the EMR.³¹

Subjects (N = 2,396) were included in the comparison of the intervention group vs control group for each of the following 4 ACOVE QI compliance measures: DVT, mobilization, bladder catheter, and delirium.

Of the 2,396 patient admissions, 530 were in the intervention unit and 1,866 were in the control unit. In the intervention group, the average age was 84.65 years, 75.58% were white and 47.21% were married. There was no difference in patient demographics between groups (Table 2).

QI 1: VTE Prophylaxis

Compliance with VTE prophylaxis was met in 78.3% of the intervention subjects and 76.5% of the controls (P < .4371) (Table 3). Of note, the rate of VTE prophylaxis was 57% in the intervention vs 39% in the control group (P < .0056), in the 554 patients for whom compliance was not met. Mechanical prophylaxis was used in 35.6% of intervention subjects vs 30.6 in the control (P = .048). Patients who received no form of prophylaxis were 0.5% in the intervention and 3% in the control (P = .027).

QI 2: Indwelling Bladder Catheters

Out of 2,396 subjects, 406 had an indwelling bladder catheter (16.9%). Compliance with the catheter was met in 72.2% of the intervention group vs 54.4% in the control group (P = .1061). An indication for indwelling bladder catheters was documented in 100% of the subjects. The average number of catheter days was 5.16 in the intervention vs 5.88 in the control (P < .2284). There was statistical significance in catheter compliance in the longer stay (>15 days) subjects, decreasing to 23.32% in the control group while staying constant in the intervention group 71.5% (P = .0006).

QI 3: Mobilization

Of the 2,396 patients, 1,991 (83.1%) were reported as ambulatory prior to admission. In the intervention vs control group, 74 (14%) vs 297 (15.7%), respectively, were nonambulatory. Overall compliance with Q3 was 62.9% in the intervention vs 48.2% in the control (P < .0001). More specifically, the average time to PT order in the intervention group was 1.83 days vs 2.22 days in the control group (P < .0051) and the time to PT evaluation was 2.14 days vs 2.42 days, respectively (P < .0108). In the intervention group, 84 patients (15.8%) did not have a PT consult vs 511 (27%) in the control group (P < .0001). The average times per subject in which the nurses documented the approximate number of feet ambulated was 6.48 in the intervention group vs 0.11 in the control group.

QI 4: Delirium Evaluation

In terms of nursing documentation indicating the presence of an acute confusional state, the intervention group had 148 out of 530 nursing notes (27.9%) vs 405 out of 1,866 in the control group (21.7%; P = .0027). However, utilization of the “acute confusion” parameter with documentation of a risk factor did not differ between the groups (5.8% in the intervention group vs 5.6% in the control group, P < .94).

LOS, Discharge Disposition, and 30-Day Readmissions

LOS did not differ between intervention and control groups (6.37 days vs 6.27 days, respectively), with a median of 5 days (P = .877). Discharge disposition in the 2 groups included the following: home/home with services (71.32% vs 68.7%), skilled nursing facility/assisted living/long-term care (24.34 versus 25.83), inpatient hospice/home hospice (2.64 vs 2.25), and expired (1.13 vs 1.77; P < .3282). In addition, 30-day readmissions did not differ (21% vs 20%, respectively, P = .41).

Our goal was to explore an evidence-based, standardized approach to improve the care of hospitalized older adults. This approach leverages existing automated EMR alert functions with an additional level of decision support for VEs, integrated into daily multidisciplinary rounds. The use of a daily checklist-based tool offers a cost-effective and practical pathway to distribute the burden of compliance responsibility amongst team members.

As we anticipated and similar to study findings in hospitalized medicine, geriatric trauma, and primary care, compliance with general care QIs was better than geriatric-focused QIs.^27,32 Wenger et al³³ demonstrated significant improvements with screening for falls and incontinence; however, screening for cognitive impairment did not improve in the outpatient setting by imbedding ACOVE QIs into routine physician practice.

Increased compliance with VTE prophylaxis and indwelling bladder catheters may be explained by national financial incentives for widespread implementation of EMR alert systems. Conversely, mobilization, delirium assessment, and management in hospitalized older adults don’t benefit from similar incentives.

VTE Prophylaxis

The American College of Chest Physicians (ACCP) supports the use of VTE prophylaxis, especially in hospitalized older adults with decreased mobility.³⁴ While greater adoption of EMR has already increased adherence, our intervention resulted in an even higher rate of compliance with the use of pharmacologic VTE prophylaxis.³⁵ In the future, validated scores for risk of thrombosis and bleeding may be integrated into our QI-based checklist.

Indwelling Bladder Catheters

The potential harms of catheters have been described for over 50 years, yet remain frequently used.^36,37 Previous studies have shown success in decreasing catheter days with computer-based and multidisciplinary protocols.^36-39

Our health system’s EMR has built-in “soft” and “hard” alerts for indwelling bladder catheters, so we did not expect intervention-associated changes in compliance.

Mobilization

Hospitalization in older adults frequently results in functional decline.^4,5,40 In response, the mobilization QI recommends an ambulation plan within 48 hours for those patients who were ambulatory prior to admission; it does not specifically define the components of the plan.²⁶ There are several multicomponent interventions that have demonstrated improvement in functional decline, yet they require skilled providers.^41,42 Our intervention implemented specific ambulation plan components: daily ambulation and documentation reminders and early PT evaluation.

While functional status measures have existed for decades, most are primarily geared to assess community-residing individuals and not designed to measure changes in function during hospitalization.^43,44 Furthermore, performance-based hospital measures are difficult to integrate into the daily nursing workflow as they are time consuming.^45,46 In practice, nurses routinely use free text to document functional status in the hospital setting, rendering comparative analysis problematic. Yet, we demonstrated that nurses were more engaged in reporting mobilization (increased documentation of ambulation distance and a decrease in time to PT). Future research should focus on the development of a standardized tool, integrated into the EMR, to accurately measure function in the acute care setting.

Delirium Evaluation

Delirium evaluation remains one of the most difficult clinical challenges for healthcare providers in hospitalized individuals, and our study reiterated these concerns. Previous research has consistently demonstrated that the diagnosis of delirium is missed by up to 75% of clinicians.^47,48 Indeed, our study, which exclusively examined nursing documentation of the delirium evaluation QI, found that both groups showed strikingly low compliance rates. This may have been due to the fact that we only evaluated nursing documentation of suspected or definite diagnosis of delirium and a documented attempt to attribute the altered mental state to a potential etiology.³¹ By utilizing the concept of “key words,” as developed by Puelle et al.³⁰, we were able to demonstrate a statistically significant improvement in nursing delirium documentation in the intervention group. This result should be interpreted with caution, as this approach is not validated. Furthermore, our operational definition of delirium compliance (ie, nurse documentation of delirium, requiring the launching of a separate parameter) may have been simply too cumbersome to readily integrate into the daily workflow. Future research should study the efficacy of a sensitive EMR-integrated screening tool that facilitates recognition, by all team members, of acute changes in cognition.

Although a number of QI improved for the intervention group, acute care utilization measures such as LOS, discharge disposition, and 30-day readmissions did not differ between groups. It may well be that improving quality for this very frail, vulnerable population may simply not result in decreased utilization. Our ability to further decrease LOS and readmission rates may be limited due to restriction of range in this complex patient population (eg, median LOS value of 5 days).

Limitations

Although our study had a large sample size, data were only collected from a single-center and thus require further exploration in different settings to ensure generalizability. In addition, QI observance was based on the medical record, which was problematic for some indicators, notably delirium identification. While prior literature highlights the difficulty in identifying delirium, especially during clinical practice without specialized training, our compliance was strikingly low.⁴⁷ While validated measures such as CAM may have been included as part of the assessment, there is currently no EMR documentation of such measures and therefore, these data could not be obtained.

In summary, our study demonstrates the successful integration of the established ACOVE QIs as an intervention, rather than as an assessment method, for improving care of hospitalized older patients. By utilizing a checklist-based tool at the bedside allows the multidisciplinary team to implement evidence-based practices with the ultimate goal of standardizing care, not only for VEs, but potentially for other high-risk populations with multimorbidity.⁴⁹ This innovative approach provides a much-needed direction to healthcare providers in the ever increasing stressful conditions of today’s acute care environment and for the ultimate benefit and safety of our older patients.

Disclosure

The authors declare no conflicts of interest. This study was supported by New York State Empire Clinical Research Investigators Program (ECRIP). The sponsor had no role in the conception, study design, data collection, data analysis, interpretation of data, manuscript preparation, or the decision to submit the manuscript for publication.

In 2014, the United States spent $3 trillion on healthcare; hospitalization consumed 32% of these expenditures.¹ Today, Medicare patients account for over 50% of hospital days and over 30% of all hospital discharges in the United States.² Despite this staggering financial burden, hospitalization of older adults often results in poor patient outcomes.^3-6 The exponential growth of the hospitalist movement, from 350 hospitalists nationwide in 1995 to over 44,000 in 2014, has become the key strategy for providing care to hospitalized geriatric patients.^7-10 Most of these hospitalists have not received geriatric training.^11-15

There is growing evidence that a geriatric approach, emphasizing multidisciplinary management of the complex needs of older patients, leads to improved outcomes. Geriatric Evaluation and Management Units (GEMUs), such as Acute Care for Elderly (ACE) models, have demonstrated significant decreases in functional decline, institutionalization, and death in randomized controlled trials.^16,17 Multidisciplinary, nonunit based efforts, such as the mobile acute care of elderly (MACE), proactive consultation models (Sennour/Counsell), and the Hospital Elder Life Program (HELP), have demonstrated success in preventing adverse events and decreasing length of stay (LOS).^17-20

However, these models have not been systematically implemented due to challenges in generalizability and replicability in diverse settings. To address this concern, an alternative approach must be developed to widely “generalize” geriatric expertise throughout hospitals, regardless of their location, size, and resources. This initiative will require systematic integration of evidence-based decision support tools for the standardization of clinical management in hospitalized older adults.²¹

The 1998 Assessing Care of Vulnerable Elders (ACOVE) project developed a standardized tool to measure and evaluate the quality of care by using a comprehensive set of quality indicators (QIs) to improve the care of “vulnerable elders” (VEs) at a high risk for functional and cognitive decline and death.^22-24 The latest systematic review concludes that, although many studies have used ACOVE as an assessment tool of quality, there has been a dearth of studies investigating the ACOVE QIs as an intervention to improve patient care.²⁵

Our study investigated the role of ACOVE as an intervention by using the QIs as a standardized checklist in the acute care setting. We selected the 4 most commonly encountered QIs in the hospital setting, namely venous thrombosis prophylaxis (VTE), indwelling bladder catheter, mobilization, and delirium evaluation, in order to test the feasibility and impact of systematically implementing these ACOVE QIs as a therapeutic intervention for all hospitalized older adults.

This study (IRB #13-644B) was conducted using a prospective intervention with a nonequivalent control group design comprised of retrospective chart data from May 1, 2014, to June 30, 2015. Process and outcome variables were extracted from electronic medical records ([EMR], Sunrise Clinical Manager [SCM]) of 2,396 patients, with 530 patients in the intervention unit and 1,866 on the control units, at a large academic tertiary center operating in the greater New York metropolitan area. Our study investigated the role of ACOVE as an intervention to improve patient care by using selected QIs as a standardized checklist tool in the acute care setting. Of the original 30 hospital-specific QIs, our study focused on the care of older adults admitted to the medicine service.²⁶ We selected commonly encountered QIs, with the objective of testing the feasibility and impact of implementing the ACOVE QIs as an intervention to improve care of hospitalized older adults. This intervention consisted of applying the checklist tool, constructed with 4 selected ACOVE QIs and administered daily during interdisciplinary rounds, namely: 2 general “medical” indicators, VTE prophylaxis and indwelling bladder catheters, and 2 “geriatric”-focused indicators, mobilization and delirium evaluation.

Subject matter experts (hospitalists, geriatricians, researchers, administrators, and nurses) reviewed the ACOVE QIs and agreed upon the adaptation of the QIs from a quality measure assessment into a feasible and acceptable intervention checklist tool (Table 1). The checklist was reviewed during daily interdisciplinary rounds for all patients 75 years and older. While ACOVE defined vulnerable elders by using the Vulnerable Elder Screen (VES), we wanted to apply this intervention more broadly to all hospitalized older adults who are most at risk for poor outcomes.²⁷ Patients admitted to the intensive care unit, inpatient psychiatry, inpatient leukemia/lymphoma, and surgical services were excluded.

Daily interdisciplinary rounds are held on every one of the five 40-bed medical units; they last approximately 1 hour, and consist of a lead hospitalist, nurse manager, nurse practitioners, case managers, and the nursing staff. During interdisciplinary rounds, nurses present the case to the team members who then discuss the care plan. These 5 medical units did not differ in terms of patient characteristics or staffing patterns; the intervention unit was chosen simply for logistical reasons, in that the principal investigator (PI) had been assigned to this unit prior to study start-up.

Prior to the intervention, LS held an education session for staff on the intervention unit staff (who participated on interdisciplinary rounds) to explain the concept of the ACOVE QI initiative and describe the four QIs selected for the study. Three subsequent educational sessions were held during the first week of the intervention, with new incoming staff receiving a brief individual educational session. The staff demonstrated significant knowledge improvement after session completion (pre/post mean score 70.6% vs 90.0%; P < .0001).

The Clinical Information System for the Health System EMR, The Eclipsys SCM, has alerts with different levels of severity from “soft” (user must acknowledge a recommendation) to “hard” (requires an action in order to proceed).

To measure compliance of the quality indicators, we collected the following variables:

QI 1: VTE prophylaxis

Through SCM, we collected type of VTE prophylaxis ordered (pharmacologic and/or mechanical) as well as start and stop dates for all agents. International normalized ratio levels were checked for patients receiving warfarin. Days of compliance were calculated.

QI 2: Indwelling Bladder Catheters

SCM data were collected on catheter entry and discontinuation dates, the presence of an indication, and order renewal for bladder catheter at least every 3 days.

QI 3: Mobilization

Ambulation status prior to admission was extracted from nursing documentation completed on admission to the medical ward. Patients documented as bedfast were categorized as nonambulatory prior to admission. Nursing documentation of activity level and amount of feet ambulated per nursing shift were collected. In addition, hospital day of physical therapy (PT) order and hospital days with PT performed were charted. Compliance with QI 3 in patients documented as ambulatory prior to hospital admission was recorded as present if there was a PT order within 48 hours of admission.

QI 4: Delirium Evaluation

During daily rounds, the hospitalist (PI) questioned nurses about delirium evaluation, using the first feature of the Confusion Assessment Method (CAM) as well as the “single question in delirium,” namely, “Is there evidence of an acute change in mental status from the patient’s baseline?” and “Do you think [name of patient] has been more confused lately?”^28,29 Because EMR does not contain a specified field for delirium screening and documentation, and patients are not routinely included in rounds, documentation with QI 4 was recorded using the “key words” method as described in the work by Puelle et al.³⁰ To extract SCM key words, nursing documentation of the “cognitive/perceptual/neurological exam” section of the EMR on admission and on all subsequent documentation (once per shift) was retrieved to identify acute changes in mental status (eg, “altered mental status, delirium/delirious, alert and oriented X 3, confused/confusion, disoriented, lethargy/lethargic”).³⁰ In addition, nurses were asked to activate an SCM parameter, “Acute Confusion” SCM parameter, in the nursing documentation section, which includes potential risk factors for confusion.

In addition to QI compliance, we collected LOS, discharge disposition, and 30-day readmission data.

Generalized linear mixed models (GLMM) for binary clustered (ie, hierarchical) data were used to estimate compliance rates (ie, nurse adherence) for each group (intervention group or control group) in the postintervention period, along with their corresponding 95% confidence intervals. GLMM was used to account for the hierarchical structure of the data: nursing units within a hospital. In order to calculate the Charlson Comorbidity Index, we extracted past medical history from the EMR.³¹

Subjects (N = 2,396) were included in the comparison of the intervention group vs control group for each of the following 4 ACOVE QI compliance measures: DVT, mobilization, bladder catheter, and delirium.

Of the 2,396 patient admissions, 530 were in the intervention unit and 1,866 were in the control unit. In the intervention group, the average age was 84.65 years, 75.58% were white and 47.21% were married. There was no difference in patient demographics between groups (Table 2).

QI 1: VTE Prophylaxis

Compliance with VTE prophylaxis was met in 78.3% of the intervention subjects and 76.5% of the controls (P < .4371) (Table 3). Of note, the rate of VTE prophylaxis was 57% in the intervention vs 39% in the control group (P < .0056), in the 554 patients for whom compliance was not met. Mechanical prophylaxis was used in 35.6% of intervention subjects vs 30.6 in the control (P = .048). Patients who received no form of prophylaxis were 0.5% in the intervention and 3% in the control (P = .027).

QI 2: Indwelling Bladder Catheters

Out of 2,396 subjects, 406 had an indwelling bladder catheter (16.9%). Compliance with the catheter was met in 72.2% of the intervention group vs 54.4% in the control group (P = .1061). An indication for indwelling bladder catheters was documented in 100% of the subjects. The average number of catheter days was 5.16 in the intervention vs 5.88 in the control (P < .2284). There was statistical significance in catheter compliance in the longer stay (>15 days) subjects, decreasing to 23.32% in the control group while staying constant in the intervention group 71.5% (P = .0006).

QI 3: Mobilization

Of the 2,396 patients, 1,991 (83.1%) were reported as ambulatory prior to admission. In the intervention vs control group, 74 (14%) vs 297 (15.7%), respectively, were nonambulatory. Overall compliance with Q3 was 62.9% in the intervention vs 48.2% in the control (P < .0001). More specifically, the average time to PT order in the intervention group was 1.83 days vs 2.22 days in the control group (P < .0051) and the time to PT evaluation was 2.14 days vs 2.42 days, respectively (P < .0108). In the intervention group, 84 patients (15.8%) did not have a PT consult vs 511 (27%) in the control group (P < .0001). The average times per subject in which the nurses documented the approximate number of feet ambulated was 6.48 in the intervention group vs 0.11 in the control group.

QI 4: Delirium Evaluation

In terms of nursing documentation indicating the presence of an acute confusional state, the intervention group had 148 out of 530 nursing notes (27.9%) vs 405 out of 1,866 in the control group (21.7%; P = .0027). However, utilization of the “acute confusion” parameter with documentation of a risk factor did not differ between the groups (5.8% in the intervention group vs 5.6% in the control group, P < .94).

LOS, Discharge Disposition, and 30-Day Readmissions

LOS did not differ between intervention and control groups (6.37 days vs 6.27 days, respectively), with a median of 5 days (P = .877). Discharge disposition in the 2 groups included the following: home/home with services (71.32% vs 68.7%), skilled nursing facility/assisted living/long-term care (24.34 versus 25.83), inpatient hospice/home hospice (2.64 vs 2.25), and expired (1.13 vs 1.77; P < .3282). In addition, 30-day readmissions did not differ (21% vs 20%, respectively, P = .41).

Our goal was to explore an evidence-based, standardized approach to improve the care of hospitalized older adults. This approach leverages existing automated EMR alert functions with an additional level of decision support for VEs, integrated into daily multidisciplinary rounds. The use of a daily checklist-based tool offers a cost-effective and practical pathway to distribute the burden of compliance responsibility amongst team members.

As we anticipated and similar to study findings in hospitalized medicine, geriatric trauma, and primary care, compliance with general care QIs was better than geriatric-focused QIs.^27,32 Wenger et al³³ demonstrated significant improvements with screening for falls and incontinence; however, screening for cognitive impairment did not improve in the outpatient setting by imbedding ACOVE QIs into routine physician practice.

Increased compliance with VTE prophylaxis and indwelling bladder catheters may be explained by national financial incentives for widespread implementation of EMR alert systems. Conversely, mobilization, delirium assessment, and management in hospitalized older adults don’t benefit from similar incentives.

VTE Prophylaxis

The American College of Chest Physicians (ACCP) supports the use of VTE prophylaxis, especially in hospitalized older adults with decreased mobility.³⁴ While greater adoption of EMR has already increased adherence, our intervention resulted in an even higher rate of compliance with the use of pharmacologic VTE prophylaxis.³⁵ In the future, validated scores for risk of thrombosis and bleeding may be integrated into our QI-based checklist.

Indwelling Bladder Catheters

The potential harms of catheters have been described for over 50 years, yet remain frequently used.^36,37 Previous studies have shown success in decreasing catheter days with computer-based and multidisciplinary protocols.^36-39

Our health system’s EMR has built-in “soft” and “hard” alerts for indwelling bladder catheters, so we did not expect intervention-associated changes in compliance.

Mobilization

Hospitalization in older adults frequently results in functional decline.^4,5,40 In response, the mobilization QI recommends an ambulation plan within 48 hours for those patients who were ambulatory prior to admission; it does not specifically define the components of the plan.²⁶ There are several multicomponent interventions that have demonstrated improvement in functional decline, yet they require skilled providers.^41,42 Our intervention implemented specific ambulation plan components: daily ambulation and documentation reminders and early PT evaluation.

While functional status measures have existed for decades, most are primarily geared to assess community-residing individuals and not designed to measure changes in function during hospitalization.^43,44 Furthermore, performance-based hospital measures are difficult to integrate into the daily nursing workflow as they are time consuming.^45,46 In practice, nurses routinely use free text to document functional status in the hospital setting, rendering comparative analysis problematic. Yet, we demonstrated that nurses were more engaged in reporting mobilization (increased documentation of ambulation distance and a decrease in time to PT). Future research should focus on the development of a standardized tool, integrated into the EMR, to accurately measure function in the acute care setting.

Delirium Evaluation

Delirium evaluation remains one of the most difficult clinical challenges for healthcare providers in hospitalized individuals, and our study reiterated these concerns. Previous research has consistently demonstrated that the diagnosis of delirium is missed by up to 75% of clinicians.^47,48 Indeed, our study, which exclusively examined nursing documentation of the delirium evaluation QI, found that both groups showed strikingly low compliance rates. This may have been due to the fact that we only evaluated nursing documentation of suspected or definite diagnosis of delirium and a documented attempt to attribute the altered mental state to a potential etiology.³¹ By utilizing the concept of “key words,” as developed by Puelle et al.³⁰, we were able to demonstrate a statistically significant improvement in nursing delirium documentation in the intervention group. This result should be interpreted with caution, as this approach is not validated. Furthermore, our operational definition of delirium compliance (ie, nurse documentation of delirium, requiring the launching of a separate parameter) may have been simply too cumbersome to readily integrate into the daily workflow. Future research should study the efficacy of a sensitive EMR-integrated screening tool that facilitates recognition, by all team members, of acute changes in cognition.

Although a number of QI improved for the intervention group, acute care utilization measures such as LOS, discharge disposition, and 30-day readmissions did not differ between groups. It may well be that improving quality for this very frail, vulnerable population may simply not result in decreased utilization. Our ability to further decrease LOS and readmission rates may be limited due to restriction of range in this complex patient population (eg, median LOS value of 5 days).

Limitations

Although our study had a large sample size, data were only collected from a single-center and thus require further exploration in different settings to ensure generalizability. In addition, QI observance was based on the medical record, which was problematic for some indicators, notably delirium identification. While prior literature highlights the difficulty in identifying delirium, especially during clinical practice without specialized training, our compliance was strikingly low.⁴⁷ While validated measures such as CAM may have been included as part of the assessment, there is currently no EMR documentation of such measures and therefore, these data could not be obtained.

In summary, our study demonstrates the successful integration of the established ACOVE QIs as an intervention, rather than as an assessment method, for improving care of hospitalized older patients. By utilizing a checklist-based tool at the bedside allows the multidisciplinary team to implement evidence-based practices with the ultimate goal of standardizing care, not only for VEs, but potentially for other high-risk populations with multimorbidity.⁴⁹ This innovative approach provides a much-needed direction to healthcare providers in the ever increasing stressful conditions of today’s acute care environment and for the ultimate benefit and safety of our older patients.

Disclosure

The authors declare no conflicts of interest. This study was supported by New York State Empire Clinical Research Investigators Program (ECRIP). The sponsor had no role in the conception, study design, data collection, data analysis, interpretation of data, manuscript preparation, or the decision to submit the manuscript for publication.

In 2014, the United States spent $3 trillion on healthcare; hospitalization consumed 32% of these expenditures.¹ Today, Medicare patients account for over 50% of hospital days and over 30% of all hospital discharges in the United States.² Despite this staggering financial burden, hospitalization of older adults often results in poor patient outcomes.^3-6 The exponential growth of the hospitalist movement, from 350 hospitalists nationwide in 1995 to over 44,000 in 2014, has become the key strategy for providing care to hospitalized geriatric patients.^7-10 Most of these hospitalists have not received geriatric training.^11-15

There is growing evidence that a geriatric approach, emphasizing multidisciplinary management of the complex needs of older patients, leads to improved outcomes. Geriatric Evaluation and Management Units (GEMUs), such as Acute Care for Elderly (ACE) models, have demonstrated significant decreases in functional decline, institutionalization, and death in randomized controlled trials.^16,17 Multidisciplinary, nonunit based efforts, such as the mobile acute care of elderly (MACE), proactive consultation models (Sennour/Counsell), and the Hospital Elder Life Program (HELP), have demonstrated success in preventing adverse events and decreasing length of stay (LOS).^17-20

However, these models have not been systematically implemented due to challenges in generalizability and replicability in diverse settings. To address this concern, an alternative approach must be developed to widely “generalize” geriatric expertise throughout hospitals, regardless of their location, size, and resources. This initiative will require systematic integration of evidence-based decision support tools for the standardization of clinical management in hospitalized older adults.²¹

The 1998 Assessing Care of Vulnerable Elders (ACOVE) project developed a standardized tool to measure and evaluate the quality of care by using a comprehensive set of quality indicators (QIs) to improve the care of “vulnerable elders” (VEs) at a high risk for functional and cognitive decline and death.^22-24 The latest systematic review concludes that, although many studies have used ACOVE as an assessment tool of quality, there has been a dearth of studies investigating the ACOVE QIs as an intervention to improve patient care.²⁵

Our study investigated the role of ACOVE as an intervention by using the QIs as a standardized checklist in the acute care setting. We selected the 4 most commonly encountered QIs in the hospital setting, namely venous thrombosis prophylaxis (VTE), indwelling bladder catheter, mobilization, and delirium evaluation, in order to test the feasibility and impact of systematically implementing these ACOVE QIs as a therapeutic intervention for all hospitalized older adults.

This study (IRB #13-644B) was conducted using a prospective intervention with a nonequivalent control group design comprised of retrospective chart data from May 1, 2014, to June 30, 2015. Process and outcome variables were extracted from electronic medical records ([EMR], Sunrise Clinical Manager [SCM]) of 2,396 patients, with 530 patients in the intervention unit and 1,866 on the control units, at a large academic tertiary center operating in the greater New York metropolitan area. Our study investigated the role of ACOVE as an intervention to improve patient care by using selected QIs as a standardized checklist tool in the acute care setting. Of the original 30 hospital-specific QIs, our study focused on the care of older adults admitted to the medicine service.²⁶ We selected commonly encountered QIs, with the objective of testing the feasibility and impact of implementing the ACOVE QIs as an intervention to improve care of hospitalized older adults. This intervention consisted of applying the checklist tool, constructed with 4 selected ACOVE QIs and administered daily during interdisciplinary rounds, namely: 2 general “medical” indicators, VTE prophylaxis and indwelling bladder catheters, and 2 “geriatric”-focused indicators, mobilization and delirium evaluation.

Subject matter experts (hospitalists, geriatricians, researchers, administrators, and nurses) reviewed the ACOVE QIs and agreed upon the adaptation of the QIs from a quality measure assessment into a feasible and acceptable intervention checklist tool (Table 1). The checklist was reviewed during daily interdisciplinary rounds for all patients 75 years and older. While ACOVE defined vulnerable elders by using the Vulnerable Elder Screen (VES), we wanted to apply this intervention more broadly to all hospitalized older adults who are most at risk for poor outcomes.²⁷ Patients admitted to the intensive care unit, inpatient psychiatry, inpatient leukemia/lymphoma, and surgical services were excluded.

Daily interdisciplinary rounds are held on every one of the five 40-bed medical units; they last approximately 1 hour, and consist of a lead hospitalist, nurse manager, nurse practitioners, case managers, and the nursing staff. During interdisciplinary rounds, nurses present the case to the team members who then discuss the care plan. These 5 medical units did not differ in terms of patient characteristics or staffing patterns; the intervention unit was chosen simply for logistical reasons, in that the principal investigator (PI) had been assigned to this unit prior to study start-up.

Prior to the intervention, LS held an education session for staff on the intervention unit staff (who participated on interdisciplinary rounds) to explain the concept of the ACOVE QI initiative and describe the four QIs selected for the study. Three subsequent educational sessions were held during the first week of the intervention, with new incoming staff receiving a brief individual educational session. The staff demonstrated significant knowledge improvement after session completion (pre/post mean score 70.6% vs 90.0%; P < .0001).

The Clinical Information System for the Health System EMR, The Eclipsys SCM, has alerts with different levels of severity from “soft” (user must acknowledge a recommendation) to “hard” (requires an action in order to proceed).

To measure compliance of the quality indicators, we collected the following variables:

QI 1: VTE prophylaxis

Through SCM, we collected type of VTE prophylaxis ordered (pharmacologic and/or mechanical) as well as start and stop dates for all agents. International normalized ratio levels were checked for patients receiving warfarin. Days of compliance were calculated.

QI 2: Indwelling Bladder Catheters

SCM data were collected on catheter entry and discontinuation dates, the presence of an indication, and order renewal for bladder catheter at least every 3 days.

QI 3: Mobilization

Ambulation status prior to admission was extracted from nursing documentation completed on admission to the medical ward. Patients documented as bedfast were categorized as nonambulatory prior to admission. Nursing documentation of activity level and amount of feet ambulated per nursing shift were collected. In addition, hospital day of physical therapy (PT) order and hospital days with PT performed were charted. Compliance with QI 3 in patients documented as ambulatory prior to hospital admission was recorded as present if there was a PT order within 48 hours of admission.

QI 4: Delirium Evaluation

During daily rounds, the hospitalist (PI) questioned nurses about delirium evaluation, using the first feature of the Confusion Assessment Method (CAM) as well as the “single question in delirium,” namely, “Is there evidence of an acute change in mental status from the patient’s baseline?” and “Do you think [name of patient] has been more confused lately?”^28,29 Because EMR does not contain a specified field for delirium screening and documentation, and patients are not routinely included in rounds, documentation with QI 4 was recorded using the “key words” method as described in the work by Puelle et al.³⁰ To extract SCM key words, nursing documentation of the “cognitive/perceptual/neurological exam” section of the EMR on admission and on all subsequent documentation (once per shift) was retrieved to identify acute changes in mental status (eg, “altered mental status, delirium/delirious, alert and oriented X 3, confused/confusion, disoriented, lethargy/lethargic”).³⁰ In addition, nurses were asked to activate an SCM parameter, “Acute Confusion” SCM parameter, in the nursing documentation section, which includes potential risk factors for confusion.

In addition to QI compliance, we collected LOS, discharge disposition, and 30-day readmission data.

Generalized linear mixed models (GLMM) for binary clustered (ie, hierarchical) data were used to estimate compliance rates (ie, nurse adherence) for each group (intervention group or control group) in the postintervention period, along with their corresponding 95% confidence intervals. GLMM was used to account for the hierarchical structure of the data: nursing units within a hospital. In order to calculate the Charlson Comorbidity Index, we extracted past medical history from the EMR.³¹

Subjects (N = 2,396) were included in the comparison of the intervention group vs control group for each of the following 4 ACOVE QI compliance measures: DVT, mobilization, bladder catheter, and delirium.

Of the 2,396 patient admissions, 530 were in the intervention unit and 1,866 were in the control unit. In the intervention group, the average age was 84.65 years, 75.58% were white and 47.21% were married. There was no difference in patient demographics between groups (Table 2).

QI 1: VTE Prophylaxis

Compliance with VTE prophylaxis was met in 78.3% of the intervention subjects and 76.5% of the controls (P < .4371) (Table 3). Of note, the rate of VTE prophylaxis was 57% in the intervention vs 39% in the control group (P < .0056), in the 554 patients for whom compliance was not met. Mechanical prophylaxis was used in 35.6% of intervention subjects vs 30.6 in the control (P = .048). Patients who received no form of prophylaxis were 0.5% in the intervention and 3% in the control (P = .027).

QI 2: Indwelling Bladder Catheters

Out of 2,396 subjects, 406 had an indwelling bladder catheter (16.9%). Compliance with the catheter was met in 72.2% of the intervention group vs 54.4% in the control group (P = .1061). An indication for indwelling bladder catheters was documented in 100% of the subjects. The average number of catheter days was 5.16 in the intervention vs 5.88 in the control (P < .2284). There was statistical significance in catheter compliance in the longer stay (>15 days) subjects, decreasing to 23.32% in the control group while staying constant in the intervention group 71.5% (P = .0006).

QI 3: Mobilization

Of the 2,396 patients, 1,991 (83.1%) were reported as ambulatory prior to admission. In the intervention vs control group, 74 (14%) vs 297 (15.7%), respectively, were nonambulatory. Overall compliance with Q3 was 62.9% in the intervention vs 48.2% in the control (P < .0001). More specifically, the average time to PT order in the intervention group was 1.83 days vs 2.22 days in the control group (P < .0051) and the time to PT evaluation was 2.14 days vs 2.42 days, respectively (P < .0108). In the intervention group, 84 patients (15.8%) did not have a PT consult vs 511 (27%) in the control group (P < .0001). The average times per subject in which the nurses documented the approximate number of feet ambulated was 6.48 in the intervention group vs 0.11 in the control group.

QI 4: Delirium Evaluation

In terms of nursing documentation indicating the presence of an acute confusional state, the intervention group had 148 out of 530 nursing notes (27.9%) vs 405 out of 1,866 in the control group (21.7%; P = .0027). However, utilization of the “acute confusion” parameter with documentation of a risk factor did not differ between the groups (5.8% in the intervention group vs 5.6% in the control group, P < .94).

LOS, Discharge Disposition, and 30-Day Readmissions

LOS did not differ between intervention and control groups (6.37 days vs 6.27 days, respectively), with a median of 5 days (P = .877). Discharge disposition in the 2 groups included the following: home/home with services (71.32% vs 68.7%), skilled nursing facility/assisted living/long-term care (24.34 versus 25.83), inpatient hospice/home hospice (2.64 vs 2.25), and expired (1.13 vs 1.77; P < .3282). In addition, 30-day readmissions did not differ (21% vs 20%, respectively, P = .41).

Our goal was to explore an evidence-based, standardized approach to improve the care of hospitalized older adults. This approach leverages existing automated EMR alert functions with an additional level of decision support for VEs, integrated into daily multidisciplinary rounds. The use of a daily checklist-based tool offers a cost-effective and practical pathway to distribute the burden of compliance responsibility amongst team members.

As we anticipated and similar to study findings in hospitalized medicine, geriatric trauma, and primary care, compliance with general care QIs was better than geriatric-focused QIs.^27,32 Wenger et al³³ demonstrated significant improvements with screening for falls and incontinence; however, screening for cognitive impairment did not improve in the outpatient setting by imbedding ACOVE QIs into routine physician practice.

Increased compliance with VTE prophylaxis and indwelling bladder catheters may be explained by national financial incentives for widespread implementation of EMR alert systems. Conversely, mobilization, delirium assessment, and management in hospitalized older adults don’t benefit from similar incentives.

VTE Prophylaxis

The American College of Chest Physicians (ACCP) supports the use of VTE prophylaxis, especially in hospitalized older adults with decreased mobility.³⁴ While greater adoption of EMR has already increased adherence, our intervention resulted in an even higher rate of compliance with the use of pharmacologic VTE prophylaxis.³⁵ In the future, validated scores for risk of thrombosis and bleeding may be integrated into our QI-based checklist.

Indwelling Bladder Catheters

The potential harms of catheters have been described for over 50 years, yet remain frequently used.^36,37 Previous studies have shown success in decreasing catheter days with computer-based and multidisciplinary protocols.^36-39

Our health system’s EMR has built-in “soft” and “hard” alerts for indwelling bladder catheters, so we did not expect intervention-associated changes in compliance.

Mobilization

Hospitalization in older adults frequently results in functional decline.^4,5,40 In response, the mobilization QI recommends an ambulation plan within 48 hours for those patients who were ambulatory prior to admission; it does not specifically define the components of the plan.²⁶ There are several multicomponent interventions that have demonstrated improvement in functional decline, yet they require skilled providers.^41,42 Our intervention implemented specific ambulation plan components: daily ambulation and documentation reminders and early PT evaluation.

While functional status measures have existed for decades, most are primarily geared to assess community-residing individuals and not designed to measure changes in function during hospitalization.^43,44 Furthermore, performance-based hospital measures are difficult to integrate into the daily nursing workflow as they are time consuming.^45,46 In practice, nurses routinely use free text to document functional status in the hospital setting, rendering comparative analysis problematic. Yet, we demonstrated that nurses were more engaged in reporting mobilization (increased documentation of ambulation distance and a decrease in time to PT). Future research should focus on the development of a standardized tool, integrated into the EMR, to accurately measure function in the acute care setting.

Delirium Evaluation

Delirium evaluation remains one of the most difficult clinical challenges for healthcare providers in hospitalized individuals, and our study reiterated these concerns. Previous research has consistently demonstrated that the diagnosis of delirium is missed by up to 75% of clinicians.^47,48 Indeed, our study, which exclusively examined nursing documentation of the delirium evaluation QI, found that both groups showed strikingly low compliance rates. This may have been due to the fact that we only evaluated nursing documentation of suspected or definite diagnosis of delirium and a documented attempt to attribute the altered mental state to a potential etiology.³¹ By utilizing the concept of “key words,” as developed by Puelle et al.³⁰, we were able to demonstrate a statistically significant improvement in nursing delirium documentation in the intervention group. This result should be interpreted with caution, as this approach is not validated. Furthermore, our operational definition of delirium compliance (ie, nurse documentation of delirium, requiring the launching of a separate parameter) may have been simply too cumbersome to readily integrate into the daily workflow. Future research should study the efficacy of a sensitive EMR-integrated screening tool that facilitates recognition, by all team members, of acute changes in cognition.

Although a number of QI improved for the intervention group, acute care utilization measures such as LOS, discharge disposition, and 30-day readmissions did not differ between groups. It may well be that improving quality for this very frail, vulnerable population may simply not result in decreased utilization. Our ability to further decrease LOS and readmission rates may be limited due to restriction of range in this complex patient population (eg, median LOS value of 5 days).

Limitations

Although our study had a large sample size, data were only collected from a single-center and thus require further exploration in different settings to ensure generalizability. In addition, QI observance was based on the medical record, which was problematic for some indicators, notably delirium identification. While prior literature highlights the difficulty in identifying delirium, especially during clinical practice without specialized training, our compliance was strikingly low.⁴⁷ While validated measures such as CAM may have been included as part of the assessment, there is currently no EMR documentation of such measures and therefore, these data could not be obtained.

In summary, our study demonstrates the successful integration of the established ACOVE QIs as an intervention, rather than as an assessment method, for improving care of hospitalized older patients. By utilizing a checklist-based tool at the bedside allows the multidisciplinary team to implement evidence-based practices with the ultimate goal of standardizing care, not only for VEs, but potentially for other high-risk populations with multimorbidity.⁴⁹ This innovative approach provides a much-needed direction to healthcare providers in the ever increasing stressful conditions of today’s acute care environment and for the ultimate benefit and safety of our older patients.

Disclosure

The authors declare no conflicts of interest. This study was supported by New York State Empire Clinical Research Investigators Program (ECRIP). The sponsor had no role in the conception, study design, data collection, data analysis, interpretation of data, manuscript preparation, or the decision to submit the manuscript for publication.

Fever in hospitalized patients is a nonspecific finding with many potential causes. Blood cultures (BC) are commonly obtained prior to commencing parenteral antibiotics in febrile patients. However, as many as 35% to 50% of positive BCs represent a contamination with organisms inoculated from the skin into culture bottles at the time of sample collection.^1-3 Such results represent false-positive BCs that can lead to unnecessary investigations and treatment.

Recently, Coburn et al. reviewed the severity of chills (graded on an ordinal scale) as the most useful predictor of true bacteremia (positive likelihood ratio [LR], 4.7; 95% confidence interval [CI], 3.0–7.2),^4-6 and the lack of the systemic inflammatory response syndrome (SIRS) criteria as the best negative indicator of true bacteremia with a negative LR of 0.09 (95% CI, 0.03-0.3).^6,7 We have also previously reported normal food consumption as a negative indicator of true bacteremia, with a 98.3% negative predictive value.⁸ Henderson’s Basic Principles of Nursing Care emphasizes the importance of evaluating whether a patient can eat and drink adequately,⁹ and the evaluation of a patient’s food consumption is a routine nursing staff practice, which is treated as vital sign in Japan, in contrast to nursing practices in the United States.

However, these data were the result of a single-center retrospective study using the nursing staff’s assessment of food consumption, and they cannot be generalized to larger patient populations. Therefore, the aim of this prospective, multicenter study was to measure the accuracy of food consumption and shaking chills as predictive factors for true bacteremia.

Study Design

This was a prospective multicenter observational study (UMIN ID: R000013768) involving 3 hospitals in Tokyo, Japan, that enrolled consecutive patients who had BCs obtained. This study was approved by the ethical committee at Juntendo University Nerima Hospital and each of the participating centers, and the study was conducted in accordance with the Declaration of Helsinki 1971, as revised in 1983. We evaluated 2,792 consecutive hospitalized patients (mean age, 68.9 ± 17.1 years; 55.3% men) who had BCs obtained between April 2013 and August 2014, inclusive. The indication for BC acquisition was at the discretion of the treating physician. The study protocol and the indication for BCs are described in detail elsewhere.⁸ We excluded patients with anorexia-inducing conditions such as gastrointestinal disease, including gastrointestinal bleeding, enterocolitis, gastric ulceration, peritonitis, appendicitis, cholangitis, pancreatitis, diverticulitis, and ischemic colitis. We also excluded patients receiving chemotherapy for malignancy. In this study, true bacteremia was defined as identical organisms isolated from 2 sets of blood cultures (a set refers to one aerobic bottle and one anaerobic bottle). Moreover, even if only one set of blood cultures was acquired, when the identified pathogen could account for the clinical presentation, we also defined this as true bacteremia. Briefly, contaminants were defined as organisms common to skin flora, including Bacillus species, coagulase-negative Staphylococcus, Corynebacterium species, and Micrococcus species, without isolation of an identical organism with the same antibiotic susceptibilities from another potentially infected site in a patient with incompatible clinical features and no risk factors for infection with the isolated organism. Single BCs that were positive for organisms that were unlikely to explain the patient’s symptoms were also considered as contaminants. Patients with contaminated BCs were excluded from the analyses.

Structure of Reliability Study Procedures

Nurses in the 3 different hospitals performed daily independent food consumption ratings during each patient’s stay. Interrater reliability assessments were conducted in the morning or afternoon, and none of the raters had access to the other nurses’ scores at any time. The study nurses performed simultaneous ratings during these assessments (one interacted with and rated the patient while the other observed and rated the same patient).

Prediction Variables of True Bacteremia


1. Food consumption. Assessment of food consumption has been previously described in detail.⁸ Briefly, we characterized the patients’ oral intake based on the meal taken immediately prior to the BCs. For example, if a fever developed at 2 pm, lunch consumption was evaluated. If a fever developed at 2 am, dinner consumption was evaluated. We categorized the patients into 3 groups: low food consumption (<50% consumed), moderate food consumption (>50% to <80% consumed), and high food consumption (>80% consumed). To simplify our prediction rule, we subsequently divided food consumption into just 2 groups: high food consumption, referred to as the “normal food consumption group,” and the combination of low and moderate food consumption, referred to as the “poor food consumption group.”

2. Chills. As done previously, the physician evaluated the patient for a history of chills at the time of BCs and classified the patients into 1 of 4 grades^4,5: “no chills,” the absence of any chills; “mild chills,” feeling cold, equivalent to needing an outer jacket; “moderate chills,” feeling very cold, equivalent to needing a thick blanket; and “shaking chills,” feeling extremely cold with rigors and generalized bodily shaking, even under a thick blanket. To distinguish between those patients who had shaking chills and those who did not, we divided the patients into 2 groups: the “shaking chills group” and the combination of none, mild, and moderate chills, referred to as the “negative shaking chills group.”

3. Other predictive variables. We considered the following additional predictive variables: age, gender, axillary body temperature (BT), heart rate (HR), systolic blood pressure (SBP), respiratory rate (RR), white blood cell count (WBC), and serum C-reactive protein level (CRP). These predictive variables were obtained immediately prior to the BCs. We defined SIRS based on standard criteria (HR >90 beats/m, RR >20/m, BT <36°C or >38°C, and a WBC <4 × 10³ WBC/μL or >12 × 10³ WBC/μL). Patients were subcategorized by age into 2 groups (≤69 years and >70 years). CRP levels were dichotomized as >10.0 mg/dL or ≤10.0 mg/dL. We reviewed the patients’ charts to determine whether they had received antibiotics. In the case of walk-in patients, we interviewed the patients regarding whether they had visited a clinic; if they had, they were questioned as to whether any antibiotic therapy had been prescribed.

Statistical Analysis

Continuous variables are presented as the mean with the associated standard deviation (SD). All potential variables predictive of true bacteremia are shown in Table 1. The variables were dichotomized by clinically meaningful thresholds and used as potential risk-adjusted variables. We calculated the sensitivity and specificity and positive and negative predictive value for each criterion. Multiple logistic regression analysis was used to select components that were significantly associated with true bacteremia (the level of statistical significance determined with maximum likelihood methods was set at P < .05). To visualize and quantify other aspects in the prediction of true bacteremia, a recursive partitioning analysis (RPA) was used to make a decision tree model for true bacteremia. This nonparametric regression method produces a classification tree following a series of nonsequential top-down binary splits. The tree-building process starts by considering a set of predictive variables and selects the variable that produces 2 subsets of participants with the greatest purity. Two factors are considered when splitting a node into its daughter nodes: the goodness of the split and the amount of impurity in the daughter nodes. The splitting process is repeated until further partitioning is no longer possible and the terminal nodes have been reached. Details on this method are discussed in Monte Carlo Calibration of Distributions of Partition Statistics (www.jmp.com).

Probability was considered significant at a value of P < .05. All statistical tests were 2-tailed. Statistical analyses were conducted by a physician (KI) and an independent statistician (JM) with the use of the SPSS^® v.16.0 software package (SPSS Inc., Chicago, IL) and JMP^® version 8.0.2 (SAS Institute, Cary, NC).

Patients Characteristics

Two thousand seven hundred and ninety-two patients met the inclusion criteria for our study, from which 849 were excluded (see Figure 1 for flow diagram). Among the remaining 1,943 patients, there were 317 patients with positive BCs, of which 221 patients (69.7%) were considered to have true-positive BCs and 96 (30.3%) were considered to have contaminated BCs. After excluding these 96 patients, 221 patients with true bacteremia (true bacteremic group) were compared with 1,626 nonbacteremic patients (nonbacteremic group; Figure 1). The baseline characteristics of the subjects are shown in Table 1. The mean BT was 38.4 ± 1.2°C in the true bacteremic group and 37.9 ± 1.0°C in the nonbacteremic group. The mean serum CRP level was 11.6 ± 9.6 mg/dL in the true bacteremic group and 7.3 ± 6.9 mg/dL in the nonbacteremic group. In the true bacteremic group, there were 6 afebrile patients, and 27 patients without leukocytosis. The pathogens identified from the true-positive BCs were Escherichia coli (n = 59, 26.7%), including extended-spectrum beta-lactamase producing species, Staphylococcus aureus (n = 36, 16.3%), including methicillin-resistant Staphylococcus aureus, and Klebsiella pneumoniae (n = 22, 10.0%; Supplemental Table 1).

The underlying clinical diagnoses in the true bacteremic group included urinary tract infection (UTI), pneumonia, abscess, catheter-related bloodstream infection (CRBSI), cellulitis, osteomyelitis, infective endocarditis (IE), chorioamnionitis, iatrogenic infection at hemodialysis puncture sites, bacterial meningitis, septic arthritis, and infection of unknown cause (Supplemental Table 2).

Interrater Reliability Testing of Food Consumption

Patients were evaluated during their hospital stays. The interrater reliability of the evaluation of food consumption was very high across all participating hospitals (Supplemental Table 3). To assess the reliability of the evaluations of food consumption, patients (separate from this main study) were selected randomly and evaluated independently by 2 nurses in 3 different hospitals. The kappa scores of agreement between the nurses at the 3 different hospitals were 0.83 (95% CI, 0.63-0.88), 0.90 (95% CI, 0.80-0.99), and 0.80 (95% CI, 0.67-0.99), respectively. The interrater reliability of food consumption evaluation by the nurses was very high at all participating hospitals.

Food Consumption

The low, moderate, and high food consumption groups consisted of 964 (52.1%), 306 (16.6%), and 577 (31.2%) patients, respectively (Table 1). Of these, 174 (18.0%), 33 (10.8%), and 14 (2.4%) patients, respectively, had true bacteremia. The presence of poor food consumption had a sensitivity of 93.7% (95% CI, 89.4%-97.9%), specificity of 34.6% (95% CI, 33.0%-36.2%), and a positive LR of 1.43 (95% CI, 1.37-1.50) for predicting true bacteremia. Conversely, the absence of poor food consumption (ie, normal food consumption) had a negative LR of 0.18 (95% CI, 0.17-0.19).

Chills

The no, mild, moderate, and shaking chills groups consisted of 1,514 (82.0%), 148 (8.0%), 53 (2.9%), and 132 (7.1%) patients, respectively (Table 1). Of these, 136 (9.0%), 25 (16.9%), 8 (15.1%), and 52 (39.4%) patients, respectively, had true bacteremia. The presence of shaking chills had a sensitivity of 23.5% (95% CI, 22.5%-24.6%), a specificity of 95.1% (95% CI, 90.7%-99.4%), and a positive LR of 4.78 (95% CI, 4.56–5.00) for predicting true bacteremia. Conversely, the absence of shaking chills had a negative LR of 0.80 (95% CI, 0.77-0.84).

Prediction Model for True Bacteremia

The components identified as significantly related to true bacteremia by multiple logistic regression analysis are indicated in Table 2. The significant predictors of true bacteremia were shaking chills (odds ratio [OR], 5.6; 95% CI, 3.6-8.6; P < .01), SBP <90 mmHg (OR, 3.1; 95% CI, 1.6-5.7; P < 01), CRP levels >10.0 mg/dL (OR, 2.2; 95% CI, 1.6-3.1; P < .01), BT <36°C or >38°C (OR, 1.8; 95% CI, 1.3-2.6; P < .01), WBC <4 × 10³/μL or >12 × 10³/μL (OR, 1.6; 95% CI, 1.2-2.3; P = .003), HR >90 bpm (OR, 1.5; 95% CI, 1.1-2.1; P = .021), and female (OR, 1.4; 95% CI, 1.0-1.9; P = .036). An RPA to create an ideal prediction model for patients with true bacteremia or nonbacteremia is shown in Figure 2. The original group consisted of 1,847 patients, including 221 patients with true bacteremia. The pretest probability of true bacteremia was 2.4% (14/577) for those with normal food consumption (Group 1) and 2.4% (13/552) for those with both normal food consumption and the absence of shaking chills (Group 2). Conversely, the pretest probability of true bacteremia was 16.3% (207/1270) for those with poor food consumption and 47.7% (51/107) for those with both poor food consumption and shaking chills. The patients with true bacteremia with normal food consumption and without shaking chills consisted of 4 cases of CRBSI and UTI, 2 cases of osteomyelitis, 1 case of IE, 1 case of chorioamnionitis, and 1 case for which the focus was unknown (Supplemental Table 4).

In this observational study, we evaluated if a simple algorithm using food consumption and shaking chills was useful for assessing whether a patient had true bacteremia. A 2-item screening checklist (nursing assessment of food consumption and shaking chills) had excellent statistical properties as a brief screening instrument for true bacteremia.

We have prospectively validated that food consumption, as assessed by nurses, is a reliable predictor of true bacteremia.⁸ A previous single-center retrospective study showed similar findings, but these could not be generalized across all institutions because of the limited nature of the study. In this multicenter study, we used 2 statistical methods to reduce selection bias. First, we performed a kappa analysis across the hospitals to evaluate the interrater reliability of the evaluation of food consumption. Second, we used an RPA (Figure 2), also known as a decision tree model. RPA is a step-by-step process by which a decision tree is constructed by either splitting or not splitting each node on the tree into 2 daughter nodes.¹⁰ By using this method, we successfully generated an ideal approach to predict true bacteremia using food consumption and shaking chills. After adjusting for food consumption and shaking chills, groups 1 to 2 had sequentially decreasing diagnoses of true bacteremia, varying from 221 patients to only 13 patients.

Appetite is influenced by many factors that are integrated by the brain, most importantly within the hypothalamus. Signals that impinge on the hypothalamic center include neural afferents, hormones, cytokines, and metabolites.¹¹ These factors elicit “sickness behavior,” which includes a decrease in food-motivated behavior.¹² Furthermore, exposure to pathogenic bacteria increases serotonin, which has been shown to decrease metabolism in amphid neurons by transcriptional and post-transcriptional mechanisms.¹³ Therefore, nonbacteremic patients retain their appetites. Shaking chills are a well-known predictor of true bacteremia.^4,5 Several cytokines, including tumor necrosis factor-alpha and interleukins 6 and 10, may be related to shaking chills.¹⁴ Coburn et al. reviewed that shaking chills appear to be useful for identifying true bacteremia (positive LR, 4.7; 95% CI, 3.0-7.2),^5,6 similar to our study. In our study, the pretest probability of true bacteremia was the same whether shaking chills was included or not (ie, 2.4% for normal food consumption and 2.4% for normal food consumption plus absence of shaking chills). This would seem to imply that the assessment of shaking chills does not appear to add anything over food assessment alone when trying to rule out bacteremia. Rather, shaking chills seem more important for ruling in bacteremia rather than ruling it out. Moreover, the recent retrospective study revealed that age >60 years (OR = 2.75, 95% CI, 1.23-6.48, P = .015), female sex (OR = 2.21, 95% CI, 1.07- 4.67, P = .038), heart rate >90 bpm (OR = 5.18, 95% CI, 2.25-12.48, P < .001) and neutrophil percentage >80% (OR = 3.61, 95% CI, 1.71- 8.00, P = .001) were independent risk factors for true bacteremia.¹⁵ Conversely, the lack of the SIRS criteria was reported as the best negative indicator of true bacteremia with a negative LR of 0.09 (95% CI, 0.03-0.3).^6,7 However, the evaluation of SIRS criteria requires the acquisition of laboratory data. To our knowledge, no previous prospective studies have evaluated food consumption in terms of a risk prediction for true bacteremia. This extremely simple model can enable a physician to make a rapid bedside estimation of the risk of true bacteremia.

The strengths of this study include its relatively large sample size, multicenter design, uniformity of data collection across sites, and completeness of data collection from study participants. All of these factors allowed for a robust analysis.

However, there are several limitations of this study. First, the physicians or nurses asked the patients about the presence of shaking chills when they obtained the BCs. It may be difficult for patients, especially elderly patients, to provide this information promptly and accurately. Some patients did not call the nurse when they had shaking chills, and the chills were not witnessed by a healthcare provider. However, we used a more specific definition for shaking chills: a feeling of being extremely cold with rigors and generalized bodily shaking, even under a thick blanket. Second, this algorithm is not applicable to patients with immunosuppressed states because none of the hospitals involved in this study perform bone marrow or organ transplantation. Third, although we included patients with dementia in our cohort, we did not specifically evaluate performance of the algorithm in patients with this medical condition. It is possible that the algorithm would not perform well in this subset of patients owing to their unreliable appetite and food intake. Fourth, some medications may affect appetite, leading to reduced food consumption. Although we have not considered the details of medications in this study, we found that the pretest probability of true bacteremia was low for those patients with normal food consumption regardless of whether the medication affected their appetites or not. However, the question of whether medications truly affect patients’ appetites concurrently with bacteremia would need to be specifically addressed in a future study.

CONCLUSION

In conclusion, we have established a simple algorithm to identify patients with suspected true bacteremia who require the acquisition of blood cultures. This extremely simple model can enable physicians to make a rapid bedside estimation of the risk of true bacteremia.

Acknowledgment

The authors thank Drs. H. Honda and S. Saint, and Ms. A. Okada for their helpful discussions with regard to this study; Ms. M. Takigawa for the collection of data; and Ms. T. Oguri for providing infectious disease consultation on the pathogenicity of the identified organisms.

Disclosure

This work was supported by JSPS KAKENHI Grant Number 15K19294 (to TK) and 20590840 (to KI) from the Japan Society for the Promotion of Science. The authors report no potential conflicts of interest relevant to this article.

Fever in hospitalized patients is a nonspecific finding with many potential causes. Blood cultures (BC) are commonly obtained prior to commencing parenteral antibiotics in febrile patients. However, as many as 35% to 50% of positive BCs represent a contamination with organisms inoculated from the skin into culture bottles at the time of sample collection.^1-3 Such results represent false-positive BCs that can lead to unnecessary investigations and treatment.

Recently, Coburn et al. reviewed the severity of chills (graded on an ordinal scale) as the most useful predictor of true bacteremia (positive likelihood ratio [LR], 4.7; 95% confidence interval [CI], 3.0–7.2),^4-6 and the lack of the systemic inflammatory response syndrome (SIRS) criteria as the best negative indicator of true bacteremia with a negative LR of 0.09 (95% CI, 0.03-0.3).^6,7 We have also previously reported normal food consumption as a negative indicator of true bacteremia, with a 98.3% negative predictive value.⁸ Henderson’s Basic Principles of Nursing Care emphasizes the importance of evaluating whether a patient can eat and drink adequately,⁹ and the evaluation of a patient’s food consumption is a routine nursing staff practice, which is treated as vital sign in Japan, in contrast to nursing practices in the United States.

However, these data were the result of a single-center retrospective study using the nursing staff’s assessment of food consumption, and they cannot be generalized to larger patient populations. Therefore, the aim of this prospective, multicenter study was to measure the accuracy of food consumption and shaking chills as predictive factors for true bacteremia.

Study Design

This was a prospective multicenter observational study (UMIN ID: R000013768) involving 3 hospitals in Tokyo, Japan, that enrolled consecutive patients who had BCs obtained. This study was approved by the ethical committee at Juntendo University Nerima Hospital and each of the participating centers, and the study was conducted in accordance with the Declaration of Helsinki 1971, as revised in 1983. We evaluated 2,792 consecutive hospitalized patients (mean age, 68.9 ± 17.1 years; 55.3% men) who had BCs obtained between April 2013 and August 2014, inclusive. The indication for BC acquisition was at the discretion of the treating physician. The study protocol and the indication for BCs are described in detail elsewhere.⁸ We excluded patients with anorexia-inducing conditions such as gastrointestinal disease, including gastrointestinal bleeding, enterocolitis, gastric ulceration, peritonitis, appendicitis, cholangitis, pancreatitis, diverticulitis, and ischemic colitis. We also excluded patients receiving chemotherapy for malignancy. In this study, true bacteremia was defined as identical organisms isolated from 2 sets of blood cultures (a set refers to one aerobic bottle and one anaerobic bottle). Moreover, even if only one set of blood cultures was acquired, when the identified pathogen could account for the clinical presentation, we also defined this as true bacteremia. Briefly, contaminants were defined as organisms common to skin flora, including Bacillus species, coagulase-negative Staphylococcus, Corynebacterium species, and Micrococcus species, without isolation of an identical organism with the same antibiotic susceptibilities from another potentially infected site in a patient with incompatible clinical features and no risk factors for infection with the isolated organism. Single BCs that were positive for organisms that were unlikely to explain the patient’s symptoms were also considered as contaminants. Patients with contaminated BCs were excluded from the analyses.

Structure of Reliability Study Procedures

Nurses in the 3 different hospitals performed daily independent food consumption ratings during each patient’s stay. Interrater reliability assessments were conducted in the morning or afternoon, and none of the raters had access to the other nurses’ scores at any time. The study nurses performed simultaneous ratings during these assessments (one interacted with and rated the patient while the other observed and rated the same patient).

Prediction Variables of True Bacteremia


1. Food consumption. Assessment of food consumption has been previously described in detail.⁸ Briefly, we characterized the patients’ oral intake based on the meal taken immediately prior to the BCs. For example, if a fever developed at 2 pm, lunch consumption was evaluated. If a fever developed at 2 am, dinner consumption was evaluated. We categorized the patients into 3 groups: low food consumption (<50% consumed), moderate food consumption (>50% to <80% consumed), and high food consumption (>80% consumed). To simplify our prediction rule, we subsequently divided food consumption into just 2 groups: high food consumption, referred to as the “normal food consumption group,” and the combination of low and moderate food consumption, referred to as the “poor food consumption group.”

2. Chills. As done previously, the physician evaluated the patient for a history of chills at the time of BCs and classified the patients into 1 of 4 grades^4,5: “no chills,” the absence of any chills; “mild chills,” feeling cold, equivalent to needing an outer jacket; “moderate chills,” feeling very cold, equivalent to needing a thick blanket; and “shaking chills,” feeling extremely cold with rigors and generalized bodily shaking, even under a thick blanket. To distinguish between those patients who had shaking chills and those who did not, we divided the patients into 2 groups: the “shaking chills group” and the combination of none, mild, and moderate chills, referred to as the “negative shaking chills group.”

3. Other predictive variables. We considered the following additional predictive variables: age, gender, axillary body temperature (BT), heart rate (HR), systolic blood pressure (SBP), respiratory rate (RR), white blood cell count (WBC), and serum C-reactive protein level (CRP). These predictive variables were obtained immediately prior to the BCs. We defined SIRS based on standard criteria (HR >90 beats/m, RR >20/m, BT <36°C or >38°C, and a WBC <4 × 10³ WBC/μL or >12 × 10³ WBC/μL). Patients were subcategorized by age into 2 groups (≤69 years and >70 years). CRP levels were dichotomized as >10.0 mg/dL or ≤10.0 mg/dL. We reviewed the patients’ charts to determine whether they had received antibiotics. In the case of walk-in patients, we interviewed the patients regarding whether they had visited a clinic; if they had, they were questioned as to whether any antibiotic therapy had been prescribed.

Statistical Analysis

Continuous variables are presented as the mean with the associated standard deviation (SD). All potential variables predictive of true bacteremia are shown in Table 1. The variables were dichotomized by clinically meaningful thresholds and used as potential risk-adjusted variables. We calculated the sensitivity and specificity and positive and negative predictive value for each criterion. Multiple logistic regression analysis was used to select components that were significantly associated with true bacteremia (the level of statistical significance determined with maximum likelihood methods was set at P < .05). To visualize and quantify other aspects in the prediction of true bacteremia, a recursive partitioning analysis (RPA) was used to make a decision tree model for true bacteremia. This nonparametric regression method produces a classification tree following a series of nonsequential top-down binary splits. The tree-building process starts by considering a set of predictive variables and selects the variable that produces 2 subsets of participants with the greatest purity. Two factors are considered when splitting a node into its daughter nodes: the goodness of the split and the amount of impurity in the daughter nodes. The splitting process is repeated until further partitioning is no longer possible and the terminal nodes have been reached. Details on this method are discussed in Monte Carlo Calibration of Distributions of Partition Statistics (www.jmp.com).

Probability was considered significant at a value of P < .05. All statistical tests were 2-tailed. Statistical analyses were conducted by a physician (KI) and an independent statistician (JM) with the use of the SPSS^® v.16.0 software package (SPSS Inc., Chicago, IL) and JMP^® version 8.0.2 (SAS Institute, Cary, NC).

Patients Characteristics

Two thousand seven hundred and ninety-two patients met the inclusion criteria for our study, from which 849 were excluded (see Figure 1 for flow diagram). Among the remaining 1,943 patients, there were 317 patients with positive BCs, of which 221 patients (69.7%) were considered to have true-positive BCs and 96 (30.3%) were considered to have contaminated BCs. After excluding these 96 patients, 221 patients with true bacteremia (true bacteremic group) were compared with 1,626 nonbacteremic patients (nonbacteremic group; Figure 1). The baseline characteristics of the subjects are shown in Table 1. The mean BT was 38.4 ± 1.2°C in the true bacteremic group and 37.9 ± 1.0°C in the nonbacteremic group. The mean serum CRP level was 11.6 ± 9.6 mg/dL in the true bacteremic group and 7.3 ± 6.9 mg/dL in the nonbacteremic group. In the true bacteremic group, there were 6 afebrile patients, and 27 patients without leukocytosis. The pathogens identified from the true-positive BCs were Escherichia coli (n = 59, 26.7%), including extended-spectrum beta-lactamase producing species, Staphylococcus aureus (n = 36, 16.3%), including methicillin-resistant Staphylococcus aureus, and Klebsiella pneumoniae (n = 22, 10.0%; Supplemental Table 1).

The underlying clinical diagnoses in the true bacteremic group included urinary tract infection (UTI), pneumonia, abscess, catheter-related bloodstream infection (CRBSI), cellulitis, osteomyelitis, infective endocarditis (IE), chorioamnionitis, iatrogenic infection at hemodialysis puncture sites, bacterial meningitis, septic arthritis, and infection of unknown cause (Supplemental Table 2).

Interrater Reliability Testing of Food Consumption

Patients were evaluated during their hospital stays. The interrater reliability of the evaluation of food consumption was very high across all participating hospitals (Supplemental Table 3). To assess the reliability of the evaluations of food consumption, patients (separate from this main study) were selected randomly and evaluated independently by 2 nurses in 3 different hospitals. The kappa scores of agreement between the nurses at the 3 different hospitals were 0.83 (95% CI, 0.63-0.88), 0.90 (95% CI, 0.80-0.99), and 0.80 (95% CI, 0.67-0.99), respectively. The interrater reliability of food consumption evaluation by the nurses was very high at all participating hospitals.

Food Consumption

The low, moderate, and high food consumption groups consisted of 964 (52.1%), 306 (16.6%), and 577 (31.2%) patients, respectively (Table 1). Of these, 174 (18.0%), 33 (10.8%), and 14 (2.4%) patients, respectively, had true bacteremia. The presence of poor food consumption had a sensitivity of 93.7% (95% CI, 89.4%-97.9%), specificity of 34.6% (95% CI, 33.0%-36.2%), and a positive LR of 1.43 (95% CI, 1.37-1.50) for predicting true bacteremia. Conversely, the absence of poor food consumption (ie, normal food consumption) had a negative LR of 0.18 (95% CI, 0.17-0.19).

Chills

The no, mild, moderate, and shaking chills groups consisted of 1,514 (82.0%), 148 (8.0%), 53 (2.9%), and 132 (7.1%) patients, respectively (Table 1). Of these, 136 (9.0%), 25 (16.9%), 8 (15.1%), and 52 (39.4%) patients, respectively, had true bacteremia. The presence of shaking chills had a sensitivity of 23.5% (95% CI, 22.5%-24.6%), a specificity of 95.1% (95% CI, 90.7%-99.4%), and a positive LR of 4.78 (95% CI, 4.56–5.00) for predicting true bacteremia. Conversely, the absence of shaking chills had a negative LR of 0.80 (95% CI, 0.77-0.84).

Prediction Model for True Bacteremia

The components identified as significantly related to true bacteremia by multiple logistic regression analysis are indicated in Table 2. The significant predictors of true bacteremia were shaking chills (odds ratio [OR], 5.6; 95% CI, 3.6-8.6; P < .01), SBP <90 mmHg (OR, 3.1; 95% CI, 1.6-5.7; P < 01), CRP levels >10.0 mg/dL (OR, 2.2; 95% CI, 1.6-3.1; P < .01), BT <36°C or >38°C (OR, 1.8; 95% CI, 1.3-2.6; P < .01), WBC <4 × 10³/μL or >12 × 10³/μL (OR, 1.6; 95% CI, 1.2-2.3; P = .003), HR >90 bpm (OR, 1.5; 95% CI, 1.1-2.1; P = .021), and female (OR, 1.4; 95% CI, 1.0-1.9; P = .036). An RPA to create an ideal prediction model for patients with true bacteremia or nonbacteremia is shown in Figure 2. The original group consisted of 1,847 patients, including 221 patients with true bacteremia. The pretest probability of true bacteremia was 2.4% (14/577) for those with normal food consumption (Group 1) and 2.4% (13/552) for those with both normal food consumption and the absence of shaking chills (Group 2). Conversely, the pretest probability of true bacteremia was 16.3% (207/1270) for those with poor food consumption and 47.7% (51/107) for those with both poor food consumption and shaking chills. The patients with true bacteremia with normal food consumption and without shaking chills consisted of 4 cases of CRBSI and UTI, 2 cases of osteomyelitis, 1 case of IE, 1 case of chorioamnionitis, and 1 case for which the focus was unknown (Supplemental Table 4).

In this observational study, we evaluated if a simple algorithm using food consumption and shaking chills was useful for assessing whether a patient had true bacteremia. A 2-item screening checklist (nursing assessment of food consumption and shaking chills) had excellent statistical properties as a brief screening instrument for true bacteremia.

We have prospectively validated that food consumption, as assessed by nurses, is a reliable predictor of true bacteremia.⁸ A previous single-center retrospective study showed similar findings, but these could not be generalized across all institutions because of the limited nature of the study. In this multicenter study, we used 2 statistical methods to reduce selection bias. First, we performed a kappa analysis across the hospitals to evaluate the interrater reliability of the evaluation of food consumption. Second, we used an RPA (Figure 2), also known as a decision tree model. RPA is a step-by-step process by which a decision tree is constructed by either splitting or not splitting each node on the tree into 2 daughter nodes.¹⁰ By using this method, we successfully generated an ideal approach to predict true bacteremia using food consumption and shaking chills. After adjusting for food consumption and shaking chills, groups 1 to 2 had sequentially decreasing diagnoses of true bacteremia, varying from 221 patients to only 13 patients.

Appetite is influenced by many factors that are integrated by the brain, most importantly within the hypothalamus. Signals that impinge on the hypothalamic center include neural afferents, hormones, cytokines, and metabolites.¹¹ These factors elicit “sickness behavior,” which includes a decrease in food-motivated behavior.¹² Furthermore, exposure to pathogenic bacteria increases serotonin, which has been shown to decrease metabolism in amphid neurons by transcriptional and post-transcriptional mechanisms.¹³ Therefore, nonbacteremic patients retain their appetites. Shaking chills are a well-known predictor of true bacteremia.^4,5 Several cytokines, including tumor necrosis factor-alpha and interleukins 6 and 10, may be related to shaking chills.¹⁴ Coburn et al. reviewed that shaking chills appear to be useful for identifying true bacteremia (positive LR, 4.7; 95% CI, 3.0-7.2),^5,6 similar to our study. In our study, the pretest probability of true bacteremia was the same whether shaking chills was included or not (ie, 2.4% for normal food consumption and 2.4% for normal food consumption plus absence of shaking chills). This would seem to imply that the assessment of shaking chills does not appear to add anything over food assessment alone when trying to rule out bacteremia. Rather, shaking chills seem more important for ruling in bacteremia rather than ruling it out. Moreover, the recent retrospective study revealed that age >60 years (OR = 2.75, 95% CI, 1.23-6.48, P = .015), female sex (OR = 2.21, 95% CI, 1.07- 4.67, P = .038), heart rate >90 bpm (OR = 5.18, 95% CI, 2.25-12.48, P < .001) and neutrophil percentage >80% (OR = 3.61, 95% CI, 1.71- 8.00, P = .001) were independent risk factors for true bacteremia.¹⁵ Conversely, the lack of the SIRS criteria was reported as the best negative indicator of true bacteremia with a negative LR of 0.09 (95% CI, 0.03-0.3).^6,7 However, the evaluation of SIRS criteria requires the acquisition of laboratory data. To our knowledge, no previous prospective studies have evaluated food consumption in terms of a risk prediction for true bacteremia. This extremely simple model can enable a physician to make a rapid bedside estimation of the risk of true bacteremia.

The strengths of this study include its relatively large sample size, multicenter design, uniformity of data collection across sites, and completeness of data collection from study participants. All of these factors allowed for a robust analysis.

However, there are several limitations of this study. First, the physicians or nurses asked the patients about the presence of shaking chills when they obtained the BCs. It may be difficult for patients, especially elderly patients, to provide this information promptly and accurately. Some patients did not call the nurse when they had shaking chills, and the chills were not witnessed by a healthcare provider. However, we used a more specific definition for shaking chills: a feeling of being extremely cold with rigors and generalized bodily shaking, even under a thick blanket. Second, this algorithm is not applicable to patients with immunosuppressed states because none of the hospitals involved in this study perform bone marrow or organ transplantation. Third, although we included patients with dementia in our cohort, we did not specifically evaluate performance of the algorithm in patients with this medical condition. It is possible that the algorithm would not perform well in this subset of patients owing to their unreliable appetite and food intake. Fourth, some medications may affect appetite, leading to reduced food consumption. Although we have not considered the details of medications in this study, we found that the pretest probability of true bacteremia was low for those patients with normal food consumption regardless of whether the medication affected their appetites or not. However, the question of whether medications truly affect patients’ appetites concurrently with bacteremia would need to be specifically addressed in a future study.

CONCLUSION

In conclusion, we have established a simple algorithm to identify patients with suspected true bacteremia who require the acquisition of blood cultures. This extremely simple model can enable physicians to make a rapid bedside estimation of the risk of true bacteremia.

Acknowledgment

The authors thank Drs. H. Honda and S. Saint, and Ms. A. Okada for their helpful discussions with regard to this study; Ms. M. Takigawa for the collection of data; and Ms. T. Oguri for providing infectious disease consultation on the pathogenicity of the identified organisms.

Disclosure

This work was supported by JSPS KAKENHI Grant Number 15K19294 (to TK) and 20590840 (to KI) from the Japan Society for the Promotion of Science. The authors report no potential conflicts of interest relevant to this article.

Fever in hospitalized patients is a nonspecific finding with many potential causes. Blood cultures (BC) are commonly obtained prior to commencing parenteral antibiotics in febrile patients. However, as many as 35% to 50% of positive BCs represent a contamination with organisms inoculated from the skin into culture bottles at the time of sample collection.^1-3 Such results represent false-positive BCs that can lead to unnecessary investigations and treatment.

Recently, Coburn et al. reviewed the severity of chills (graded on an ordinal scale) as the most useful predictor of true bacteremia (positive likelihood ratio [LR], 4.7; 95% confidence interval [CI], 3.0–7.2),^4-6 and the lack of the systemic inflammatory response syndrome (SIRS) criteria as the best negative indicator of true bacteremia with a negative LR of 0.09 (95% CI, 0.03-0.3).^6,7 We have also previously reported normal food consumption as a negative indicator of true bacteremia, with a 98.3% negative predictive value.⁸ Henderson’s Basic Principles of Nursing Care emphasizes the importance of evaluating whether a patient can eat and drink adequately,⁹ and the evaluation of a patient’s food consumption is a routine nursing staff practice, which is treated as vital sign in Japan, in contrast to nursing practices in the United States.

However, these data were the result of a single-center retrospective study using the nursing staff’s assessment of food consumption, and they cannot be generalized to larger patient populations. Therefore, the aim of this prospective, multicenter study was to measure the accuracy of food consumption and shaking chills as predictive factors for true bacteremia.

Study Design

This was a prospective multicenter observational study (UMIN ID: R000013768) involving 3 hospitals in Tokyo, Japan, that enrolled consecutive patients who had BCs obtained. This study was approved by the ethical committee at Juntendo University Nerima Hospital and each of the participating centers, and the study was conducted in accordance with the Declaration of Helsinki 1971, as revised in 1983. We evaluated 2,792 consecutive hospitalized patients (mean age, 68.9 ± 17.1 years; 55.3% men) who had BCs obtained between April 2013 and August 2014, inclusive. The indication for BC acquisition was at the discretion of the treating physician. The study protocol and the indication for BCs are described in detail elsewhere.⁸ We excluded patients with anorexia-inducing conditions such as gastrointestinal disease, including gastrointestinal bleeding, enterocolitis, gastric ulceration, peritonitis, appendicitis, cholangitis, pancreatitis, diverticulitis, and ischemic colitis. We also excluded patients receiving chemotherapy for malignancy. In this study, true bacteremia was defined as identical organisms isolated from 2 sets of blood cultures (a set refers to one aerobic bottle and one anaerobic bottle). Moreover, even if only one set of blood cultures was acquired, when the identified pathogen could account for the clinical presentation, we also defined this as true bacteremia. Briefly, contaminants were defined as organisms common to skin flora, including Bacillus species, coagulase-negative Staphylococcus, Corynebacterium species, and Micrococcus species, without isolation of an identical organism with the same antibiotic susceptibilities from another potentially infected site in a patient with incompatible clinical features and no risk factors for infection with the isolated organism. Single BCs that were positive for organisms that were unlikely to explain the patient’s symptoms were also considered as contaminants. Patients with contaminated BCs were excluded from the analyses.

Structure of Reliability Study Procedures

Nurses in the 3 different hospitals performed daily independent food consumption ratings during each patient’s stay. Interrater reliability assessments were conducted in the morning or afternoon, and none of the raters had access to the other nurses’ scores at any time. The study nurses performed simultaneous ratings during these assessments (one interacted with and rated the patient while the other observed and rated the same patient).

Prediction Variables of True Bacteremia


1. Food consumption. Assessment of food consumption has been previously described in detail.⁸ Briefly, we characterized the patients’ oral intake based on the meal taken immediately prior to the BCs. For example, if a fever developed at 2 pm, lunch consumption was evaluated. If a fever developed at 2 am, dinner consumption was evaluated. We categorized the patients into 3 groups: low food consumption (<50% consumed), moderate food consumption (>50% to <80% consumed), and high food consumption (>80% consumed). To simplify our prediction rule, we subsequently divided food consumption into just 2 groups: high food consumption, referred to as the “normal food consumption group,” and the combination of low and moderate food consumption, referred to as the “poor food consumption group.”

2. Chills. As done previously, the physician evaluated the patient for a history of chills at the time of BCs and classified the patients into 1 of 4 grades^4,5: “no chills,” the absence of any chills; “mild chills,” feeling cold, equivalent to needing an outer jacket; “moderate chills,” feeling very cold, equivalent to needing a thick blanket; and “shaking chills,” feeling extremely cold with rigors and generalized bodily shaking, even under a thick blanket. To distinguish between those patients who had shaking chills and those who did not, we divided the patients into 2 groups: the “shaking chills group” and the combination of none, mild, and moderate chills, referred to as the “negative shaking chills group.”

3. Other predictive variables. We considered the following additional predictive variables: age, gender, axillary body temperature (BT), heart rate (HR), systolic blood pressure (SBP), respiratory rate (RR), white blood cell count (WBC), and serum C-reactive protein level (CRP). These predictive variables were obtained immediately prior to the BCs. We defined SIRS based on standard criteria (HR >90 beats/m, RR >20/m, BT <36°C or >38°C, and a WBC <4 × 10³ WBC/μL or >12 × 10³ WBC/μL). Patients were subcategorized by age into 2 groups (≤69 years and >70 years). CRP levels were dichotomized as >10.0 mg/dL or ≤10.0 mg/dL. We reviewed the patients’ charts to determine whether they had received antibiotics. In the case of walk-in patients, we interviewed the patients regarding whether they had visited a clinic; if they had, they were questioned as to whether any antibiotic therapy had been prescribed.

Statistical Analysis

Continuous variables are presented as the mean with the associated standard deviation (SD). All potential variables predictive of true bacteremia are shown in Table 1. The variables were dichotomized by clinically meaningful thresholds and used as potential risk-adjusted variables. We calculated the sensitivity and specificity and positive and negative predictive value for each criterion. Multiple logistic regression analysis was used to select components that were significantly associated with true bacteremia (the level of statistical significance determined with maximum likelihood methods was set at P < .05). To visualize and quantify other aspects in the prediction of true bacteremia, a recursive partitioning analysis (RPA) was used to make a decision tree model for true bacteremia. This nonparametric regression method produces a classification tree following a series of nonsequential top-down binary splits. The tree-building process starts by considering a set of predictive variables and selects the variable that produces 2 subsets of participants with the greatest purity. Two factors are considered when splitting a node into its daughter nodes: the goodness of the split and the amount of impurity in the daughter nodes. The splitting process is repeated until further partitioning is no longer possible and the terminal nodes have been reached. Details on this method are discussed in Monte Carlo Calibration of Distributions of Partition Statistics (www.jmp.com).

Probability was considered significant at a value of P < .05. All statistical tests were 2-tailed. Statistical analyses were conducted by a physician (KI) and an independent statistician (JM) with the use of the SPSS^® v.16.0 software package (SPSS Inc., Chicago, IL) and JMP^® version 8.0.2 (SAS Institute, Cary, NC).

Patients Characteristics

Two thousand seven hundred and ninety-two patients met the inclusion criteria for our study, from which 849 were excluded (see Figure 1 for flow diagram). Among the remaining 1,943 patients, there were 317 patients with positive BCs, of which 221 patients (69.7%) were considered to have true-positive BCs and 96 (30.3%) were considered to have contaminated BCs. After excluding these 96 patients, 221 patients with true bacteremia (true bacteremic group) were compared with 1,626 nonbacteremic patients (nonbacteremic group; Figure 1). The baseline characteristics of the subjects are shown in Table 1. The mean BT was 38.4 ± 1.2°C in the true bacteremic group and 37.9 ± 1.0°C in the nonbacteremic group. The mean serum CRP level was 11.6 ± 9.6 mg/dL in the true bacteremic group and 7.3 ± 6.9 mg/dL in the nonbacteremic group. In the true bacteremic group, there were 6 afebrile patients, and 27 patients without leukocytosis. The pathogens identified from the true-positive BCs were Escherichia coli (n = 59, 26.7%), including extended-spectrum beta-lactamase producing species, Staphylococcus aureus (n = 36, 16.3%), including methicillin-resistant Staphylococcus aureus, and Klebsiella pneumoniae (n = 22, 10.0%; Supplemental Table 1).

The underlying clinical diagnoses in the true bacteremic group included urinary tract infection (UTI), pneumonia, abscess, catheter-related bloodstream infection (CRBSI), cellulitis, osteomyelitis, infective endocarditis (IE), chorioamnionitis, iatrogenic infection at hemodialysis puncture sites, bacterial meningitis, septic arthritis, and infection of unknown cause (Supplemental Table 2).

Interrater Reliability Testing of Food Consumption

Patients were evaluated during their hospital stays. The interrater reliability of the evaluation of food consumption was very high across all participating hospitals (Supplemental Table 3). To assess the reliability of the evaluations of food consumption, patients (separate from this main study) were selected randomly and evaluated independently by 2 nurses in 3 different hospitals. The kappa scores of agreement between the nurses at the 3 different hospitals were 0.83 (95% CI, 0.63-0.88), 0.90 (95% CI, 0.80-0.99), and 0.80 (95% CI, 0.67-0.99), respectively. The interrater reliability of food consumption evaluation by the nurses was very high at all participating hospitals.

Food Consumption

The low, moderate, and high food consumption groups consisted of 964 (52.1%), 306 (16.6%), and 577 (31.2%) patients, respectively (Table 1). Of these, 174 (18.0%), 33 (10.8%), and 14 (2.4%) patients, respectively, had true bacteremia. The presence of poor food consumption had a sensitivity of 93.7% (95% CI, 89.4%-97.9%), specificity of 34.6% (95% CI, 33.0%-36.2%), and a positive LR of 1.43 (95% CI, 1.37-1.50) for predicting true bacteremia. Conversely, the absence of poor food consumption (ie, normal food consumption) had a negative LR of 0.18 (95% CI, 0.17-0.19).

Chills

The no, mild, moderate, and shaking chills groups consisted of 1,514 (82.0%), 148 (8.0%), 53 (2.9%), and 132 (7.1%) patients, respectively (Table 1). Of these, 136 (9.0%), 25 (16.9%), 8 (15.1%), and 52 (39.4%) patients, respectively, had true bacteremia. The presence of shaking chills had a sensitivity of 23.5% (95% CI, 22.5%-24.6%), a specificity of 95.1% (95% CI, 90.7%-99.4%), and a positive LR of 4.78 (95% CI, 4.56–5.00) for predicting true bacteremia. Conversely, the absence of shaking chills had a negative LR of 0.80 (95% CI, 0.77-0.84).

Prediction Model for True Bacteremia

The components identified as significantly related to true bacteremia by multiple logistic regression analysis are indicated in Table 2. The significant predictors of true bacteremia were shaking chills (odds ratio [OR], 5.6; 95% CI, 3.6-8.6; P < .01), SBP <90 mmHg (OR, 3.1; 95% CI, 1.6-5.7; P < 01), CRP levels >10.0 mg/dL (OR, 2.2; 95% CI, 1.6-3.1; P < .01), BT <36°C or >38°C (OR, 1.8; 95% CI, 1.3-2.6; P < .01), WBC <4 × 10³/μL or >12 × 10³/μL (OR, 1.6; 95% CI, 1.2-2.3; P = .003), HR >90 bpm (OR, 1.5; 95% CI, 1.1-2.1; P = .021), and female (OR, 1.4; 95% CI, 1.0-1.9; P = .036). An RPA to create an ideal prediction model for patients with true bacteremia or nonbacteremia is shown in Figure 2. The original group consisted of 1,847 patients, including 221 patients with true bacteremia. The pretest probability of true bacteremia was 2.4% (14/577) for those with normal food consumption (Group 1) and 2.4% (13/552) for those with both normal food consumption and the absence of shaking chills (Group 2). Conversely, the pretest probability of true bacteremia was 16.3% (207/1270) for those with poor food consumption and 47.7% (51/107) for those with both poor food consumption and shaking chills. The patients with true bacteremia with normal food consumption and without shaking chills consisted of 4 cases of CRBSI and UTI, 2 cases of osteomyelitis, 1 case of IE, 1 case of chorioamnionitis, and 1 case for which the focus was unknown (Supplemental Table 4).

In this observational study, we evaluated if a simple algorithm using food consumption and shaking chills was useful for assessing whether a patient had true bacteremia. A 2-item screening checklist (nursing assessment of food consumption and shaking chills) had excellent statistical properties as a brief screening instrument for true bacteremia.

We have prospectively validated that food consumption, as assessed by nurses, is a reliable predictor of true bacteremia.⁸ A previous single-center retrospective study showed similar findings, but these could not be generalized across all institutions because of the limited nature of the study. In this multicenter study, we used 2 statistical methods to reduce selection bias. First, we performed a kappa analysis across the hospitals to evaluate the interrater reliability of the evaluation of food consumption. Second, we used an RPA (Figure 2), also known as a decision tree model. RPA is a step-by-step process by which a decision tree is constructed by either splitting or not splitting each node on the tree into 2 daughter nodes.¹⁰ By using this method, we successfully generated an ideal approach to predict true bacteremia using food consumption and shaking chills. After adjusting for food consumption and shaking chills, groups 1 to 2 had sequentially decreasing diagnoses of true bacteremia, varying from 221 patients to only 13 patients.

Appetite is influenced by many factors that are integrated by the brain, most importantly within the hypothalamus. Signals that impinge on the hypothalamic center include neural afferents, hormones, cytokines, and metabolites.¹¹ These factors elicit “sickness behavior,” which includes a decrease in food-motivated behavior.¹² Furthermore, exposure to pathogenic bacteria increases serotonin, which has been shown to decrease metabolism in amphid neurons by transcriptional and post-transcriptional mechanisms.¹³ Therefore, nonbacteremic patients retain their appetites. Shaking chills are a well-known predictor of true bacteremia.^4,5 Several cytokines, including tumor necrosis factor-alpha and interleukins 6 and 10, may be related to shaking chills.¹⁴ Coburn et al. reviewed that shaking chills appear to be useful for identifying true bacteremia (positive LR, 4.7; 95% CI, 3.0-7.2),^5,6 similar to our study. In our study, the pretest probability of true bacteremia was the same whether shaking chills was included or not (ie, 2.4% for normal food consumption and 2.4% for normal food consumption plus absence of shaking chills). This would seem to imply that the assessment of shaking chills does not appear to add anything over food assessment alone when trying to rule out bacteremia. Rather, shaking chills seem more important for ruling in bacteremia rather than ruling it out. Moreover, the recent retrospective study revealed that age >60 years (OR = 2.75, 95% CI, 1.23-6.48, P = .015), female sex (OR = 2.21, 95% CI, 1.07- 4.67, P = .038), heart rate >90 bpm (OR = 5.18, 95% CI, 2.25-12.48, P < .001) and neutrophil percentage >80% (OR = 3.61, 95% CI, 1.71- 8.00, P = .001) were independent risk factors for true bacteremia.¹⁵ Conversely, the lack of the SIRS criteria was reported as the best negative indicator of true bacteremia with a negative LR of 0.09 (95% CI, 0.03-0.3).^6,7 However, the evaluation of SIRS criteria requires the acquisition of laboratory data. To our knowledge, no previous prospective studies have evaluated food consumption in terms of a risk prediction for true bacteremia. This extremely simple model can enable a physician to make a rapid bedside estimation of the risk of true bacteremia.

The strengths of this study include its relatively large sample size, multicenter design, uniformity of data collection across sites, and completeness of data collection from study participants. All of these factors allowed for a robust analysis.

However, there are several limitations of this study. First, the physicians or nurses asked the patients about the presence of shaking chills when they obtained the BCs. It may be difficult for patients, especially elderly patients, to provide this information promptly and accurately. Some patients did not call the nurse when they had shaking chills, and the chills were not witnessed by a healthcare provider. However, we used a more specific definition for shaking chills: a feeling of being extremely cold with rigors and generalized bodily shaking, even under a thick blanket. Second, this algorithm is not applicable to patients with immunosuppressed states because none of the hospitals involved in this study perform bone marrow or organ transplantation. Third, although we included patients with dementia in our cohort, we did not specifically evaluate performance of the algorithm in patients with this medical condition. It is possible that the algorithm would not perform well in this subset of patients owing to their unreliable appetite and food intake. Fourth, some medications may affect appetite, leading to reduced food consumption. Although we have not considered the details of medications in this study, we found that the pretest probability of true bacteremia was low for those patients with normal food consumption regardless of whether the medication affected their appetites or not. However, the question of whether medications truly affect patients’ appetites concurrently with bacteremia would need to be specifically addressed in a future study.

CONCLUSION

In conclusion, we have established a simple algorithm to identify patients with suspected true bacteremia who require the acquisition of blood cultures. This extremely simple model can enable physicians to make a rapid bedside estimation of the risk of true bacteremia.

Acknowledgment

The authors thank Drs. H. Honda and S. Saint, and Ms. A. Okada for their helpful discussions with regard to this study; Ms. M. Takigawa for the collection of data; and Ms. T. Oguri for providing infectious disease consultation on the pathogenicity of the identified organisms.

Disclosure

This work was supported by JSPS KAKENHI Grant Number 15K19294 (to TK) and 20590840 (to KI) from the Japan Society for the Promotion of Science. The authors report no potential conflicts of interest relevant to this article.

Implantable cardioverter-defibrillators (ICDs) offer lifesaving therapies to many patients and have been implanted in hundreds of thousands of patients.¹ The population of patients with ICDs is growing rapidly, and the national ICD Registry reports over 12,000 devices are implanted monthly.² This population includes patients with congenital heart disease, ischemic cardiomyopathy, and idiopathic arrhythmias. If these patients experience ventricular tachycardia or fibrillation, an ICD attempts to restore sinus rhythm and prevent death. While a shock from an ICD may be lifesaving, it can be a traumatic and startling experience for the patient and perhaps distressful for families to witness.^3,4

Although ICDs are intended to save lives, they do not slow the progress of the patient’s underlying cardiac and noncardiac comorbidities. All these patients will eventually die, whether from their cardiac disease or another condition. The literature includes many anecdotes about patients shocked multiple times by their defibrillator while actively dying.⁴ These situations could be prevented with preemptive ICD deactivation. (ICDs can function not only as cardioverters and defibrillators, as implied by their name, but also as pacemakers. “Deactivation” as used in this paper refers only to disabling the tachycardia therapies. No distinction was made between defibrillation with a shock and anti-tachycardia pacing.) Therefore, research on ICD deactivation has emphasized patients who are acutely terminally ill, while less emphasis has been placed on patients who are not actively dying.^4–8

Patients may, for a variety of reasons, request a do-not-resuscitate/do-not-intubate (DNR/DNI) order as their code status. However, it is not necessarily clear what a DNR/DNI order implies for ICD management. One survey of attending physicians found that 19% of respondents felt a DNR/DNI order was equivalent to requesting ICD deactivation.⁹ On the other hand, patients are split on whether they would want their device deactivated while in hospice or even at the very end of life.⁶ Heart Rhythm Society (HRS) guidelines favor a nuanced approach to ICD deactivation in DNR/DNI patients that emphasizes the individual patient’s comorbidities and goals.¹⁰ A patient’s individual circumstances might justify a choice to be DNR/DNI without deactivating the ICD. Decision-making in these circumstances requires a careful conversation between the patient and clinician. It is important to identify barriers that might prevent optimal shared decision-making.

Clinicians have been surveyed on ICD management at the end of life, but these studies have generally focused on attending physicians.^5,9,11 However, physician trainees (ie, residents and fellows) as well as advanced practice providers (ie, physician assistants and nurse practitioners) are responsible for much of the clinical care provided to hospitalized patients. In particular, they are often the clinicians to discuss code status with patients. Different specialties (eg, cardiology, general medicine, and geriatrics) manage different sets of patients, which might affect clinicians’ opinions on ICD management. We therefore designed a survey to assess clinician attitudes and beliefs regarding ICD deactivation, including in non-terminally ill patients, and to evaluate for differences according to training level and specialty.

Case-based and Likert-scale questions were considered for this survey, with the latter being chosen for ease of completion by respondents. An online survey tool (SurveyMonkey; San Mateo, CA) was used for data collection; no identifying data were collected. E-mail invitations to participate were sent to a combination of mailing lists and individual addresses for residents, fellows, advanced practice providers, and attending physicians in general internal medicine, cardiology, electrophysiology, and geriatrics. The survey remained open for 2.5 weeks. It was conducted 5 months into the academic year, thus trainees were well-established in their current roles. Two $25 gift cards were offered to respondents who entered their e-mail into a drawing; responses were not tied to e-mail addresses. Approval for the study was obtained from the University of Michigan Institutional Review Board.

The survey posed 12 questions assessing general attitudes about ICDs as well as individual beliefs and behaviors relating to ICD deactivation. Answers were on a Likert scale of 1 to 5 with 1 representing “strongly disagree” and 5 representing “strongly agree.” A score of 3 indicated “unsure or neutral.” The first 3 questions appeared together on the first page and were prefaced with “Please respond to the following statements about ICD shocks.” The next 9 were likewise grouped on the next page and were prefaced with “Please respond to the following statements about ICD deactivation.” All 12 questions are shown in Figures 1 and 2. Respondents could easily return to previous questions and change their answers. The survey ended with a third page showing 3 multiple choice demographic questions. The demographic questions were about clinical role (first-, second-, third-, or fourth-year resident, fellow, advanced practice provider, and attending), specialty, and number of ICD deactivations the respondent had been directly involved in (0, 1 to 5, 5 to 10, and more than 10). Specialty options were internal medicine resident, inpatient general medicine, outpatient general medicine, cardiology, electrophysiology, and geriatrics.

Likert scale answers of “agree” or “strongly agree” were grouped together as an affirmative response, while all other answers were grouped together as a nonaffirmative response. For analysis, residents were grouped together and their responses compared with attending physicians as a group. Additional analysis was done comparing attending physicians stratified by clinical specialty. Given the small number of responses from attending electrophysiologists, they were grouped with attending cardiologists for analysis. Due to the limited number of fellows and advanced practice providers who responded, further evaluation of these groups was not performed. Finally, the number of ICD deactivations respondents had been involved in was stratified by training level. All comparisons were performed using the two-tailed Pearson’s chi-squared test.

A total of 170 responses were collected from 508 individuals on the e-mail lists. Two responses were from registered nurses who were not part of the intended study sample and 7 responses were incomplete, having only answered the first 3 questions. These 9 responses were excluded from further analysis, yielding an overall response rate of 32%. The demographics of the remaining 161 respondents are shown in Table 1. Figure 1 shows overall responses to each question.

When comparing residents to attending physicians, there were no statistically significant between-differences except on questions 5 and 6. Specifically, residents were less comfortable than attending physicians discussing ICD deactivation and did so with less regularity (P < .001 and P = .018, respectively; Figure 2). Comfort levels improved markedly with experience: 29.2% of interns expressed comfort asking about ICD deactivation as compared with 60.7% of third- and-fourth year residents and 78.8% of attending physicians. Furthermore, comfort level seemed to parallel the regularity with which respondents asked about ICD deactivation: 4.2% of interns routinely asked about ICD deactivation as compared with 21.4% of third- and fourth-year residents and 34.8% of attending physicians.

Overall, our major findings were (1) residents, who provide much of the clinical care in a teaching hospital, are remarkably uncomfortable discussing ICD deactivation, (2) general internists and residents ask about ICD deactivation infrequently compared to geriatricians and cardiologists, and (3) about one fifth of our respondents believe ICD deactivation is automatically part of a DNR/DNI order.

Although the majority of respondents did not routinely address ICD deactivation in conjunction with code status, there was significant variability among subgroups. For example, 83.3% of geriatricians routinely discussed ICD deactivation as part of code status compared with only 4% of first-year residents and 10.5% of inpatient general internists. This finding is interesting because 90.7% of all respondents believed that discussions of code status should address preferences on ICD deactivation. This apparent discrepancy could be explained by the relatively small number of patients admitted to the hospital who have both an ICD and a request to be DNR/DNI. Residents and inpatient general internists see a very broad spectrum of patients; ICD deactivation is frequently irrelevant in the cases these physicians manage. The subset of patients seen in consultation by cardiologists and geriatricians, by contrast, is expected to include a larger proportion of patients with ICDs. Therefore, discussing ICD deactivation will be more relevant to their daily practice. Fear of alienating patients was not a reason for our findings, as our respondents did not express concern that recommending ICD deactivation would harm the patient-clinician relationship.

There are several possible reasons that residents, particularly interns, are uncomfortable discussing ICD deactivation. A lack of exposure to ICD deactivation is probably a key contributor. Over half of interns had never been involved in any ICD deactivations. Residents and hospitalists may also feel as if they are overstepping their boundaries to discuss deactivating ICD therapies. Their feelings may not be misplaced, as one survey of ICD patients found that over 75% thought responsibility for discussing ICD deactivation, at least at the end of life, rests with cardiologists or electrophysiologists.⁶

The HRS guidelines call for individualized decisions regarding ICD deactivation, even if a patient is DNR/DNI. However, our respondents frequently felt a standardized approach was indicated, with 21% believing that a DNR/DNI order included ICD deactivation. Additionally, 28% agreed that even non-terminally ill DNR/DNI patients should have their device deactivated. This is relevant because it is the role of clinicians to engage in shared decision-making with their patients. If the clinician holds the fixed belief that a DNR/DNI order, regardless of the precise clinical scenario, should include ICD deactivation, they may pressure a patient to have their device deactivated even if it could still benefit them.

In 2009, Kelley et al published results of a survey on ICD deactivation at the end of life.⁹ They contacted 4,876 attending physicians in cardiology, electrophysiology, geriatrics, and general medicine, receiving 558 responses. The survey included Likert-scale questions assessing attitudes and knowledge about ICD functionality. Demographic information was also collected, including how many patients in their practice had ICDs and how often they had previously discussed ICD deactivation.

There are some interesting comparisons between Kelley et al’s findings and ours, although we included trainees and the precise wording of questions was different. The specific questions used by Kelley et al to ask whether ICD shocks were painful or distressing and to ask if ICD deactivation is part of a DNR order were: “The shock from an ICD is very painful for most patients.” “The shock of an ICD at the end of life is distressing to a patient and their loved ones.” “A DNR order is equivalent to deactivation of an ICD.”

Only 47% of general internists in the Kelley et al survey thought that ICD shocks were painful, compared with 83% of electrophysiologists. In addition, 65% of general internists and 85% of electrophysiologists viewed shocks at the end of life to be distressing to patients and families. By contrast, our respondents were nearly unanimous in believing shocks to be painful and distressing. This discrepancy may be due to the growing prevalence of ICDs over the past several years as well as the growing body of literature on unnecessary shocks at the end of life. In line with our study, 19% of their respondents believed a DNR order was equivalent to ICD deactivation.⁹

Taken together, our findings indicate that additional education for clinicians of all levels could be helpful. Didactic lessons cannot replace experience, and it is important for residents to be exposed to discussions of ICD deactivation. However, lessons about ICD therapies and practical examples of how to broach the topic of deactivation could be beneficial, especially for interns whose responsibility includes discussions of code status. Within the context of an internal medicine residency, the fundamentals of ICD functionality could be covered during rotations on cardiology or palliative care services. Additionally, the recommendations of the HRS for device management can be covered in didactic sessions. Similar opportunities could be built into continuing medical education for practicing physicians and the training of advanced practice providers.

There are limitations to this survey, most notably the fact that it was restricted to a single academic medical center, the patient population and practices of which may not be generalizable to medical practice at large. Selection bias is also a distinct possibility given the 32% overall response rate; those who responded may feel more strongly about the survey topic. Our study subgroups may have interpreted questions differently because of their particular area of clinical practice. The small sample size also precluded an effective analysis of fellows and advanced practice practitioners due to lack of power. A major strength of this survey was the inclusion of a large number of residents upon whom the majority of inpatient contact rests. Future work could include expanding the survey to multiple medical centers, which would enhance generalizability and improve the ability to recruit sufficient fellows and advanced practice providers.

In summary, we conducted a single-center survey of residents, fellows, advanced practice providers, and attending physicians on their attitudes and beliefs about ICD deactivation in DNR/DNI patients. Residents are particularly uncomfortable discussing ICD deactivation with patients, which is an important finding because of their crucial role in providing patient care. Additionally, residents and hospitalists do not broach the topic of deactivation regularly, especially when compared to geriatricians and cardiologists. Despite HRS guidelines to the contrary, a fifth of our respondents believed that DNR/DNI orders include ICD deactivation. Overall, ICD deactivation in DNR/DNI patients is a topic that needs further attention in clinical education so that patients receive care that respects their individual wishes.

Disclosure

Nothing to report.

Implantable cardioverter-defibrillators (ICDs) offer lifesaving therapies to many patients and have been implanted in hundreds of thousands of patients.¹ The population of patients with ICDs is growing rapidly, and the national ICD Registry reports over 12,000 devices are implanted monthly.² This population includes patients with congenital heart disease, ischemic cardiomyopathy, and idiopathic arrhythmias. If these patients experience ventricular tachycardia or fibrillation, an ICD attempts to restore sinus rhythm and prevent death. While a shock from an ICD may be lifesaving, it can be a traumatic and startling experience for the patient and perhaps distressful for families to witness.^3,4

Although ICDs are intended to save lives, they do not slow the progress of the patient’s underlying cardiac and noncardiac comorbidities. All these patients will eventually die, whether from their cardiac disease or another condition. The literature includes many anecdotes about patients shocked multiple times by their defibrillator while actively dying.⁴ These situations could be prevented with preemptive ICD deactivation. (ICDs can function not only as cardioverters and defibrillators, as implied by their name, but also as pacemakers. “Deactivation” as used in this paper refers only to disabling the tachycardia therapies. No distinction was made between defibrillation with a shock and anti-tachycardia pacing.) Therefore, research on ICD deactivation has emphasized patients who are acutely terminally ill, while less emphasis has been placed on patients who are not actively dying.^4–8

Patients may, for a variety of reasons, request a do-not-resuscitate/do-not-intubate (DNR/DNI) order as their code status. However, it is not necessarily clear what a DNR/DNI order implies for ICD management. One survey of attending physicians found that 19% of respondents felt a DNR/DNI order was equivalent to requesting ICD deactivation.⁹ On the other hand, patients are split on whether they would want their device deactivated while in hospice or even at the very end of life.⁶ Heart Rhythm Society (HRS) guidelines favor a nuanced approach to ICD deactivation in DNR/DNI patients that emphasizes the individual patient’s comorbidities and goals.¹⁰ A patient’s individual circumstances might justify a choice to be DNR/DNI without deactivating the ICD. Decision-making in these circumstances requires a careful conversation between the patient and clinician. It is important to identify barriers that might prevent optimal shared decision-making.

Clinicians have been surveyed on ICD management at the end of life, but these studies have generally focused on attending physicians.^5,9,11 However, physician trainees (ie, residents and fellows) as well as advanced practice providers (ie, physician assistants and nurse practitioners) are responsible for much of the clinical care provided to hospitalized patients. In particular, they are often the clinicians to discuss code status with patients. Different specialties (eg, cardiology, general medicine, and geriatrics) manage different sets of patients, which might affect clinicians’ opinions on ICD management. We therefore designed a survey to assess clinician attitudes and beliefs regarding ICD deactivation, including in non-terminally ill patients, and to evaluate for differences according to training level and specialty.

Case-based and Likert-scale questions were considered for this survey, with the latter being chosen for ease of completion by respondents. An online survey tool (SurveyMonkey; San Mateo, CA) was used for data collection; no identifying data were collected. E-mail invitations to participate were sent to a combination of mailing lists and individual addresses for residents, fellows, advanced practice providers, and attending physicians in general internal medicine, cardiology, electrophysiology, and geriatrics. The survey remained open for 2.5 weeks. It was conducted 5 months into the academic year, thus trainees were well-established in their current roles. Two $25 gift cards were offered to respondents who entered their e-mail into a drawing; responses were not tied to e-mail addresses. Approval for the study was obtained from the University of Michigan Institutional Review Board.

The survey posed 12 questions assessing general attitudes about ICDs as well as individual beliefs and behaviors relating to ICD deactivation. Answers were on a Likert scale of 1 to 5 with 1 representing “strongly disagree” and 5 representing “strongly agree.” A score of 3 indicated “unsure or neutral.” The first 3 questions appeared together on the first page and were prefaced with “Please respond to the following statements about ICD shocks.” The next 9 were likewise grouped on the next page and were prefaced with “Please respond to the following statements about ICD deactivation.” All 12 questions are shown in Figures 1 and 2. Respondents could easily return to previous questions and change their answers. The survey ended with a third page showing 3 multiple choice demographic questions. The demographic questions were about clinical role (first-, second-, third-, or fourth-year resident, fellow, advanced practice provider, and attending), specialty, and number of ICD deactivations the respondent had been directly involved in (0, 1 to 5, 5 to 10, and more than 10). Specialty options were internal medicine resident, inpatient general medicine, outpatient general medicine, cardiology, electrophysiology, and geriatrics.

Likert scale answers of “agree” or “strongly agree” were grouped together as an affirmative response, while all other answers were grouped together as a nonaffirmative response. For analysis, residents were grouped together and their responses compared with attending physicians as a group. Additional analysis was done comparing attending physicians stratified by clinical specialty. Given the small number of responses from attending electrophysiologists, they were grouped with attending cardiologists for analysis. Due to the limited number of fellows and advanced practice providers who responded, further evaluation of these groups was not performed. Finally, the number of ICD deactivations respondents had been involved in was stratified by training level. All comparisons were performed using the two-tailed Pearson’s chi-squared test.

A total of 170 responses were collected from 508 individuals on the e-mail lists. Two responses were from registered nurses who were not part of the intended study sample and 7 responses were incomplete, having only answered the first 3 questions. These 9 responses were excluded from further analysis, yielding an overall response rate of 32%. The demographics of the remaining 161 respondents are shown in Table 1. Figure 1 shows overall responses to each question.

When comparing residents to attending physicians, there were no statistically significant between-differences except on questions 5 and 6. Specifically, residents were less comfortable than attending physicians discussing ICD deactivation and did so with less regularity (P < .001 and P = .018, respectively; Figure 2). Comfort levels improved markedly with experience: 29.2% of interns expressed comfort asking about ICD deactivation as compared with 60.7% of third- and-fourth year residents and 78.8% of attending physicians. Furthermore, comfort level seemed to parallel the regularity with which respondents asked about ICD deactivation: 4.2% of interns routinely asked about ICD deactivation as compared with 21.4% of third- and fourth-year residents and 34.8% of attending physicians.

Overall, our major findings were (1) residents, who provide much of the clinical care in a teaching hospital, are remarkably uncomfortable discussing ICD deactivation, (2) general internists and residents ask about ICD deactivation infrequently compared to geriatricians and cardiologists, and (3) about one fifth of our respondents believe ICD deactivation is automatically part of a DNR/DNI order.

Although the majority of respondents did not routinely address ICD deactivation in conjunction with code status, there was significant variability among subgroups. For example, 83.3% of geriatricians routinely discussed ICD deactivation as part of code status compared with only 4% of first-year residents and 10.5% of inpatient general internists. This finding is interesting because 90.7% of all respondents believed that discussions of code status should address preferences on ICD deactivation. This apparent discrepancy could be explained by the relatively small number of patients admitted to the hospital who have both an ICD and a request to be DNR/DNI. Residents and inpatient general internists see a very broad spectrum of patients; ICD deactivation is frequently irrelevant in the cases these physicians manage. The subset of patients seen in consultation by cardiologists and geriatricians, by contrast, is expected to include a larger proportion of patients with ICDs. Therefore, discussing ICD deactivation will be more relevant to their daily practice. Fear of alienating patients was not a reason for our findings, as our respondents did not express concern that recommending ICD deactivation would harm the patient-clinician relationship.

There are several possible reasons that residents, particularly interns, are uncomfortable discussing ICD deactivation. A lack of exposure to ICD deactivation is probably a key contributor. Over half of interns had never been involved in any ICD deactivations. Residents and hospitalists may also feel as if they are overstepping their boundaries to discuss deactivating ICD therapies. Their feelings may not be misplaced, as one survey of ICD patients found that over 75% thought responsibility for discussing ICD deactivation, at least at the end of life, rests with cardiologists or electrophysiologists.⁶

The HRS guidelines call for individualized decisions regarding ICD deactivation, even if a patient is DNR/DNI. However, our respondents frequently felt a standardized approach was indicated, with 21% believing that a DNR/DNI order included ICD deactivation. Additionally, 28% agreed that even non-terminally ill DNR/DNI patients should have their device deactivated. This is relevant because it is the role of clinicians to engage in shared decision-making with their patients. If the clinician holds the fixed belief that a DNR/DNI order, regardless of the precise clinical scenario, should include ICD deactivation, they may pressure a patient to have their device deactivated even if it could still benefit them.

In 2009, Kelley et al published results of a survey on ICD deactivation at the end of life.⁹ They contacted 4,876 attending physicians in cardiology, electrophysiology, geriatrics, and general medicine, receiving 558 responses. The survey included Likert-scale questions assessing attitudes and knowledge about ICD functionality. Demographic information was also collected, including how many patients in their practice had ICDs and how often they had previously discussed ICD deactivation.

There are some interesting comparisons between Kelley et al’s findings and ours, although we included trainees and the precise wording of questions was different. The specific questions used by Kelley et al to ask whether ICD shocks were painful or distressing and to ask if ICD deactivation is part of a DNR order were: “The shock from an ICD is very painful for most patients.” “The shock of an ICD at the end of life is distressing to a patient and their loved ones.” “A DNR order is equivalent to deactivation of an ICD.”

Only 47% of general internists in the Kelley et al survey thought that ICD shocks were painful, compared with 83% of electrophysiologists. In addition, 65% of general internists and 85% of electrophysiologists viewed shocks at the end of life to be distressing to patients and families. By contrast, our respondents were nearly unanimous in believing shocks to be painful and distressing. This discrepancy may be due to the growing prevalence of ICDs over the past several years as well as the growing body of literature on unnecessary shocks at the end of life. In line with our study, 19% of their respondents believed a DNR order was equivalent to ICD deactivation.⁹

Taken together, our findings indicate that additional education for clinicians of all levels could be helpful. Didactic lessons cannot replace experience, and it is important for residents to be exposed to discussions of ICD deactivation. However, lessons about ICD therapies and practical examples of how to broach the topic of deactivation could be beneficial, especially for interns whose responsibility includes discussions of code status. Within the context of an internal medicine residency, the fundamentals of ICD functionality could be covered during rotations on cardiology or palliative care services. Additionally, the recommendations of the HRS for device management can be covered in didactic sessions. Similar opportunities could be built into continuing medical education for practicing physicians and the training of advanced practice providers.

There are limitations to this survey, most notably the fact that it was restricted to a single academic medical center, the patient population and practices of which may not be generalizable to medical practice at large. Selection bias is also a distinct possibility given the 32% overall response rate; those who responded may feel more strongly about the survey topic. Our study subgroups may have interpreted questions differently because of their particular area of clinical practice. The small sample size also precluded an effective analysis of fellows and advanced practice practitioners due to lack of power. A major strength of this survey was the inclusion of a large number of residents upon whom the majority of inpatient contact rests. Future work could include expanding the survey to multiple medical centers, which would enhance generalizability and improve the ability to recruit sufficient fellows and advanced practice providers.

In summary, we conducted a single-center survey of residents, fellows, advanced practice providers, and attending physicians on their attitudes and beliefs about ICD deactivation in DNR/DNI patients. Residents are particularly uncomfortable discussing ICD deactivation with patients, which is an important finding because of their crucial role in providing patient care. Additionally, residents and hospitalists do not broach the topic of deactivation regularly, especially when compared to geriatricians and cardiologists. Despite HRS guidelines to the contrary, a fifth of our respondents believed that DNR/DNI orders include ICD deactivation. Overall, ICD deactivation in DNR/DNI patients is a topic that needs further attention in clinical education so that patients receive care that respects their individual wishes.

Disclosure

Nothing to report.

Implantable cardioverter-defibrillators (ICDs) offer lifesaving therapies to many patients and have been implanted in hundreds of thousands of patients.¹ The population of patients with ICDs is growing rapidly, and the national ICD Registry reports over 12,000 devices are implanted monthly.² This population includes patients with congenital heart disease, ischemic cardiomyopathy, and idiopathic arrhythmias. If these patients experience ventricular tachycardia or fibrillation, an ICD attempts to restore sinus rhythm and prevent death. While a shock from an ICD may be lifesaving, it can be a traumatic and startling experience for the patient and perhaps distressful for families to witness.^3,4

Although ICDs are intended to save lives, they do not slow the progress of the patient’s underlying cardiac and noncardiac comorbidities. All these patients will eventually die, whether from their cardiac disease or another condition. The literature includes many anecdotes about patients shocked multiple times by their defibrillator while actively dying.⁴ These situations could be prevented with preemptive ICD deactivation. (ICDs can function not only as cardioverters and defibrillators, as implied by their name, but also as pacemakers. “Deactivation” as used in this paper refers only to disabling the tachycardia therapies. No distinction was made between defibrillation with a shock and anti-tachycardia pacing.) Therefore, research on ICD deactivation has emphasized patients who are acutely terminally ill, while less emphasis has been placed on patients who are not actively dying.^4–8

Patients may, for a variety of reasons, request a do-not-resuscitate/do-not-intubate (DNR/DNI) order as their code status. However, it is not necessarily clear what a DNR/DNI order implies for ICD management. One survey of attending physicians found that 19% of respondents felt a DNR/DNI order was equivalent to requesting ICD deactivation.⁹ On the other hand, patients are split on whether they would want their device deactivated while in hospice or even at the very end of life.⁶ Heart Rhythm Society (HRS) guidelines favor a nuanced approach to ICD deactivation in DNR/DNI patients that emphasizes the individual patient’s comorbidities and goals.¹⁰ A patient’s individual circumstances might justify a choice to be DNR/DNI without deactivating the ICD. Decision-making in these circumstances requires a careful conversation between the patient and clinician. It is important to identify barriers that might prevent optimal shared decision-making.

Clinicians have been surveyed on ICD management at the end of life, but these studies have generally focused on attending physicians.^5,9,11 However, physician trainees (ie, residents and fellows) as well as advanced practice providers (ie, physician assistants and nurse practitioners) are responsible for much of the clinical care provided to hospitalized patients. In particular, they are often the clinicians to discuss code status with patients. Different specialties (eg, cardiology, general medicine, and geriatrics) manage different sets of patients, which might affect clinicians’ opinions on ICD management. We therefore designed a survey to assess clinician attitudes and beliefs regarding ICD deactivation, including in non-terminally ill patients, and to evaluate for differences according to training level and specialty.

Case-based and Likert-scale questions were considered for this survey, with the latter being chosen for ease of completion by respondents. An online survey tool (SurveyMonkey; San Mateo, CA) was used for data collection; no identifying data were collected. E-mail invitations to participate were sent to a combination of mailing lists and individual addresses for residents, fellows, advanced practice providers, and attending physicians in general internal medicine, cardiology, electrophysiology, and geriatrics. The survey remained open for 2.5 weeks. It was conducted 5 months into the academic year, thus trainees were well-established in their current roles. Two $25 gift cards were offered to respondents who entered their e-mail into a drawing; responses were not tied to e-mail addresses. Approval for the study was obtained from the University of Michigan Institutional Review Board.

The survey posed 12 questions assessing general attitudes about ICDs as well as individual beliefs and behaviors relating to ICD deactivation. Answers were on a Likert scale of 1 to 5 with 1 representing “strongly disagree” and 5 representing “strongly agree.” A score of 3 indicated “unsure or neutral.” The first 3 questions appeared together on the first page and were prefaced with “Please respond to the following statements about ICD shocks.” The next 9 were likewise grouped on the next page and were prefaced with “Please respond to the following statements about ICD deactivation.” All 12 questions are shown in Figures 1 and 2. Respondents could easily return to previous questions and change their answers. The survey ended with a third page showing 3 multiple choice demographic questions. The demographic questions were about clinical role (first-, second-, third-, or fourth-year resident, fellow, advanced practice provider, and attending), specialty, and number of ICD deactivations the respondent had been directly involved in (0, 1 to 5, 5 to 10, and more than 10). Specialty options were internal medicine resident, inpatient general medicine, outpatient general medicine, cardiology, electrophysiology, and geriatrics.

Likert scale answers of “agree” or “strongly agree” were grouped together as an affirmative response, while all other answers were grouped together as a nonaffirmative response. For analysis, residents were grouped together and their responses compared with attending physicians as a group. Additional analysis was done comparing attending physicians stratified by clinical specialty. Given the small number of responses from attending electrophysiologists, they were grouped with attending cardiologists for analysis. Due to the limited number of fellows and advanced practice providers who responded, further evaluation of these groups was not performed. Finally, the number of ICD deactivations respondents had been involved in was stratified by training level. All comparisons were performed using the two-tailed Pearson’s chi-squared test.

A total of 170 responses were collected from 508 individuals on the e-mail lists. Two responses were from registered nurses who were not part of the intended study sample and 7 responses were incomplete, having only answered the first 3 questions. These 9 responses were excluded from further analysis, yielding an overall response rate of 32%. The demographics of the remaining 161 respondents are shown in Table 1. Figure 1 shows overall responses to each question.

When comparing residents to attending physicians, there were no statistically significant between-differences except on questions 5 and 6. Specifically, residents were less comfortable than attending physicians discussing ICD deactivation and did so with less regularity (P < .001 and P = .018, respectively; Figure 2). Comfort levels improved markedly with experience: 29.2% of interns expressed comfort asking about ICD deactivation as compared with 60.7% of third- and-fourth year residents and 78.8% of attending physicians. Furthermore, comfort level seemed to parallel the regularity with which respondents asked about ICD deactivation: 4.2% of interns routinely asked about ICD deactivation as compared with 21.4% of third- and fourth-year residents and 34.8% of attending physicians.

Overall, our major findings were (1) residents, who provide much of the clinical care in a teaching hospital, are remarkably uncomfortable discussing ICD deactivation, (2) general internists and residents ask about ICD deactivation infrequently compared to geriatricians and cardiologists, and (3) about one fifth of our respondents believe ICD deactivation is automatically part of a DNR/DNI order.

Although the majority of respondents did not routinely address ICD deactivation in conjunction with code status, there was significant variability among subgroups. For example, 83.3% of geriatricians routinely discussed ICD deactivation as part of code status compared with only 4% of first-year residents and 10.5% of inpatient general internists. This finding is interesting because 90.7% of all respondents believed that discussions of code status should address preferences on ICD deactivation. This apparent discrepancy could be explained by the relatively small number of patients admitted to the hospital who have both an ICD and a request to be DNR/DNI. Residents and inpatient general internists see a very broad spectrum of patients; ICD deactivation is frequently irrelevant in the cases these physicians manage. The subset of patients seen in consultation by cardiologists and geriatricians, by contrast, is expected to include a larger proportion of patients with ICDs. Therefore, discussing ICD deactivation will be more relevant to their daily practice. Fear of alienating patients was not a reason for our findings, as our respondents did not express concern that recommending ICD deactivation would harm the patient-clinician relationship.

There are several possible reasons that residents, particularly interns, are uncomfortable discussing ICD deactivation. A lack of exposure to ICD deactivation is probably a key contributor. Over half of interns had never been involved in any ICD deactivations. Residents and hospitalists may also feel as if they are overstepping their boundaries to discuss deactivating ICD therapies. Their feelings may not be misplaced, as one survey of ICD patients found that over 75% thought responsibility for discussing ICD deactivation, at least at the end of life, rests with cardiologists or electrophysiologists.⁶

The HRS guidelines call for individualized decisions regarding ICD deactivation, even if a patient is DNR/DNI. However, our respondents frequently felt a standardized approach was indicated, with 21% believing that a DNR/DNI order included ICD deactivation. Additionally, 28% agreed that even non-terminally ill DNR/DNI patients should have their device deactivated. This is relevant because it is the role of clinicians to engage in shared decision-making with their patients. If the clinician holds the fixed belief that a DNR/DNI order, regardless of the precise clinical scenario, should include ICD deactivation, they may pressure a patient to have their device deactivated even if it could still benefit them.

In 2009, Kelley et al published results of a survey on ICD deactivation at the end of life.⁹ They contacted 4,876 attending physicians in cardiology, electrophysiology, geriatrics, and general medicine, receiving 558 responses. The survey included Likert-scale questions assessing attitudes and knowledge about ICD functionality. Demographic information was also collected, including how many patients in their practice had ICDs and how often they had previously discussed ICD deactivation.

There are some interesting comparisons between Kelley et al’s findings and ours, although we included trainees and the precise wording of questions was different. The specific questions used by Kelley et al to ask whether ICD shocks were painful or distressing and to ask if ICD deactivation is part of a DNR order were: “The shock from an ICD is very painful for most patients.” “The shock of an ICD at the end of life is distressing to a patient and their loved ones.” “A DNR order is equivalent to deactivation of an ICD.”

Only 47% of general internists in the Kelley et al survey thought that ICD shocks were painful, compared with 83% of electrophysiologists. In addition, 65% of general internists and 85% of electrophysiologists viewed shocks at the end of life to be distressing to patients and families. By contrast, our respondents were nearly unanimous in believing shocks to be painful and distressing. This discrepancy may be due to the growing prevalence of ICDs over the past several years as well as the growing body of literature on unnecessary shocks at the end of life. In line with our study, 19% of their respondents believed a DNR order was equivalent to ICD deactivation.⁹

Taken together, our findings indicate that additional education for clinicians of all levels could be helpful. Didactic lessons cannot replace experience, and it is important for residents to be exposed to discussions of ICD deactivation. However, lessons about ICD therapies and practical examples of how to broach the topic of deactivation could be beneficial, especially for interns whose responsibility includes discussions of code status. Within the context of an internal medicine residency, the fundamentals of ICD functionality could be covered during rotations on cardiology or palliative care services. Additionally, the recommendations of the HRS for device management can be covered in didactic sessions. Similar opportunities could be built into continuing medical education for practicing physicians and the training of advanced practice providers.

There are limitations to this survey, most notably the fact that it was restricted to a single academic medical center, the patient population and practices of which may not be generalizable to medical practice at large. Selection bias is also a distinct possibility given the 32% overall response rate; those who responded may feel more strongly about the survey topic. Our study subgroups may have interpreted questions differently because of their particular area of clinical practice. The small sample size also precluded an effective analysis of fellows and advanced practice practitioners due to lack of power. A major strength of this survey was the inclusion of a large number of residents upon whom the majority of inpatient contact rests. Future work could include expanding the survey to multiple medical centers, which would enhance generalizability and improve the ability to recruit sufficient fellows and advanced practice providers.

In summary, we conducted a single-center survey of residents, fellows, advanced practice providers, and attending physicians on their attitudes and beliefs about ICD deactivation in DNR/DNI patients. Residents are particularly uncomfortable discussing ICD deactivation with patients, which is an important finding because of their crucial role in providing patient care. Additionally, residents and hospitalists do not broach the topic of deactivation regularly, especially when compared to geriatricians and cardiologists. Despite HRS guidelines to the contrary, a fifth of our respondents believed that DNR/DNI orders include ICD deactivation. Overall, ICD deactivation in DNR/DNI patients is a topic that needs further attention in clinical education so that patients receive care that respects their individual wishes.

Disclosure

Nothing to report.

In recent years, the American Board of Internal Medicine (ABIM) Foundation’s Choosing Wisely™ campaign has advanced the dialogue on cost-consciousness by identifying potential examples of overuse in clinical practice.¹ Eliminating low-value care can decrease costs, improve quality, and potentially decrease patient harm.² In fact, there is growing consensus among health leaders and educators on the need for a physician workforce that is conscious of high-value care.^3,4 The Institute of Medicine has issued a call-to-action for graduate medical education (GME) to emphasize value-based care,⁵ and the Accreditation Council for Graduate Medical Education has outlined expectations that residents receive formal and experiential training on overuse as a part of its Clinical Learning Environment Review.⁶

However, recent reports highlight a lack of emphasis on value-based care in medical education.⁷ For example, few residency program directors believe that residents are prepared to incorporate value and cost into their medical decisions.⁸ In 2012, only 15% of medicine residencies reported having formal curricula addressing value, although many were developing one.⁸ Of the curricula reported, most were didactic in nature and did not include an assessment component.⁸

Experiential learning through simulation is one promising method to teach clinicians-in-training to practice value-based care. Simulation-based training promotes situational awareness (defined as being cognizant of one’s working environment), a concept that is crucial for recognizing both low-value and unsafe care.^9,10 Simulated training exercises are often included in GME orientation “boot-camps,” which have typically addressed safety.¹¹ The incorporation of value into existing GME boot-camp exercises could provide a promising model for the addition of value-based training to GME.

At the University of Chicago, we had successfully implemented the “Room of Horrors,” a simulation for entering interns to promote the detection of patient safety hazards.¹¹ Here, we describe a modification to this simulation to embed low-value hazards in addition to traditional patient safety hazards. The aim of this study is to assess the entering interns’ recognition of low-value care and their ability to recognize unsafe care in a simulation designed to promote situational awareness.

Setting and Participants

The simulation was conducted during GME orientation at a large, urban academic medical institution. One hundred and twenty-five entering postgraduate year one (PGY1) interns participated in the simulation, which was a required component of a multiday orientation “boot-camp” experience. All eligible interns participated in the simulation, representing 13 specialty programs and 60 medical schools. Interns entering into pathology were excluded because of infrequent patient contact. Participating interns were divided into 7 specialty groups for analysis in order to preserve the anonymity of interns in smaller residency programs (surgical subspecialties combined with general surgery, medicine-pediatrics grouped with internal medicine). The University of Chicago Institutional Review Board deemed this study exempt from review.

Program Description

A simulation of an inpatient hospital room, known as the “Room of Horrors,” was constructed in collaboration with the University of Chicago Simulation Center and adapted from a previous version of the exercise.¹¹ The simulation consisted of a mock door chart highlighting the patient had been admitted for diarrhea (Clostridium difficile positive) following a recent hospitalization for pneumonia. A clinical scenario was constructed by using a patient mannequin and an accompanying door chart that listed information on the patient’s hospital course, allergies, and medications. In addition to the 8 patient safety hazards utilized in the prior version, our team selected 4 low-value hazards to be included in the simulation.

The 8 safety hazards have been detailed in a prior study and were previously selected from Medicare’s Hospital-Acquired Conditions (HAC) Reduction Program and Agency for Healthcare Research and Quality (AHRQ) Patient Safety Indicators.^11-13 Each of the hazards was represented either physically in the simulation room and/or was indicated on the patient’s chart. For example, the latex allergy hazard was represented by latex gloves at the bedside despite an allergy indicated on the patient’s chart and wristband. A complete list of the 8 safety hazards and their representations in the simulation is shown in Table 1.

The Choosing Wisely™ lists were reviewed to identify low-value hazards for addition to the simulation.¹⁴ Our team selected 3 low-value hazards from the Society of Hospital Medicine (SHM) list,¹⁵ including (1) arbitrary blood transfusion despite the patient’s stable hemoglobin level of 8.0 g/dL and absence of cardiac symptoms,¹⁶ (2) addition of a proton pump inhibitor (PPI) for stress ulcer prophylaxis in a patient without high risk for gastrointestinal (GI) complications who was not on a PPI prior to admission, and (3) placement of a urinary catheter without medical indication. We had originally selected continuous telemetry monitoring as a fourth hazard from the SHM list, but were unable to operationalize, as it was difficult to simulate continuous telemetry on a mannequin. Because many inpatients are older than 65 years, we reviewed the American Geriatrics Society list¹⁷ and selected our fourth low-value hazard: (4) unnecessary use of physical restraints to manage behavioral symptoms in a hospitalized patient with delirium. Several of these hazards were also quality and safety priorities at our institution, including the overuse of urinary catheters, physical restraints, and blood transfusions. All 4 low-value hazards were referenced in the patient’s door chart, and 3 were also physically represented in the room via presence of a hanging unit of blood, Foley catheter, and upper-arm restraints (Table 1). See Appendix for a photograph of the simulation setup.

Each intern was allowed 10 minutes inside the simulation room. During this time, they were instructed to read the 1-page door chart, inspect the simulation room, and write down as many potential low-value and safety hazards as they could identify on a free-response form (see Appendix). Upon exiting the room, they were allotted 5 additional minutes to complete their free-response answers and provide written feedback on the simulation. The simulation was conducted in 3 simulated hospital rooms over the course of 2 days, and the correct answers were provided via e-mail after all interns had completed the exercise.

To assess prior training and safety knowledge, interns were asked to complete a 3-question preassessment on a ScanTron^TM (Tustin, CA) form. The preassessment asked interns whether they had received training on hospital safety during medical school (yes, no, or unsure), if they were satisfied with the hospital safety training they received during medical school (strongly disagree to strongly agree on a Likert scale), and if they were confident in their ability to identify potential hazards in a hospital setting (strongly disagree to strongly agree). Interns were also given the opportunity to provide feedback on the simulation experience on the ScanTron^TM (Tustin, CA) form.

One month after participating in the simulation, interns were asked to complete an online follow-up survey on MedHub^TM (Ann Arbor, MI), which included 2 Likert-scale questions (strongly disagree to strongly agree) assessing the simulation’s impact on their experience mitigating hospital hazards during the first month of internship.

Data Analysis

Interns’ free-response answers were manually coded, and descriptive statistics were used to summarize the mean percent correct for each hazard. A paired t test was used to compare intern identification of low-value vs safety hazards. T tests were used to compare hazard identification for interns entering highly procedural-intensive specialties (ie, surgical specialties, emergency medicine, anesthesia, obstetrics/gynecology) and those entering less procedural-intensive specialties (ie, internal medicine, pediatrics, psychiatry), as well as among those graduating from “Top 30” medical schools (based on US News & World Report Medical School Rankings¹⁸) and our own institution. One-way analysis of variance (ANOVA) calculations were used to test for differences in hazard identification based on interns’ prior hospital safety training, with interns who rated their satisfaction with prior training or confidence in identifying hazards as a “4” or a “5” considered “satisfied” and “confident,” respectively. Responses to the MedHub^TM (Ann Arbor, MI) survey were dichotomized with “strongly agree” and “agree” considered positive responses. Statistical significance was defined at P < .05. All data analysis was conducted using Stata 14^TM software (College Station, TX).

Intern Characteristics

One hundred twenty-five entering PGY1 interns participated in the simulation, representing 60 medical schools and 7 different specialty groups (Table 2). Thirty-five percent (44/125) were graduates from “Top 30” medical schools, and 8.8% (11/125) graduated from our own institution. Seventy-four percent (89/121) had received prior hospital safety training during medical school, and 62.9% (56/89) were satisfied with their training. A majority of interns (64.2%, 79/123) felt confident in their ability to identify potential hazards in a hospital setting, although confidence was much higher among those with prior safety training (71.9%, 64/89) compared to those without prior training or who were unsure about their training (40.6%, 13/32; P = .09, t test).

Identification of Hazards

The mean percentage of hazards correctly identified by interns during the simulation was 50.4% (standard deviation [SD] 11.8%), with a normal distribution (Figure 1). Interns identified a significantly lower percentage of low-value hazards than safety hazards in the simulation (mean 19.2% [SD 18.6%] vs 66.0% [SD 16.0%], respectively; P < .001, paired t test). Interns also identified significantly more room-based errors than chart-based errors (mean 58.6% [SD 13.4%] vs 9.6% [SD 19.8%], respectively; P < .001, paired t test). The 3 most commonly identified hazards were unavailability of hand hygiene (120/125, 96.0%), presence of latex gloves despite the patient’s allergy (111/125, 88.8%), and fall risk due to the lowered bed rail (107/125, 85.6%). More than half of interns identified the incorrect name on the patient’s wristband and IV bag (91/125, 72.8%), a lack of isolation precautions (90/125, 72.0%), administration of penicillin despite the patient’s allergy (67/125, 53.6%), and unnecessary restraints (64/125, 51.2%). Less than half of interns identified the wrong medication being administered (50/125, 40.0%), unnecessary Foley catheter (25/125, 20.0%), and absence of venous thromboembolism (VTE) prophylaxis (24/125, 19.2%). Few interns identified the unnecessary blood transfusion (7/125, 5.6%), and no one identified the unnecessary stress ulcer prophylaxis (0/125, 0.0%; Figure 2).

Predictors of Hazard Identification

Interns who self-reported as confident in their ability to identify hazards were not any more likely to correctly identify hazards than those who were not confident (50.9% overall hazard identification vs 49.6%, respectively; P = .56, t test). Interns entering into less procedural-intensive specialties identified significantly more safety hazards than those entering highly procedural-intensive specialties (mean 69.1% [SD 16.9%] vs 61.8% [SD 13.7%], respectively; P = .01, t test). However, there was no statistically significant difference in their identification of low-value hazards (mean 19.8% [SD 18.3%] for less procedural-intensive vs 18.4% [SD 19.1%] for highly procedural-intensive; P = .68, t test). There was no statistically significant difference in hazard identification among graduates of “Top 30” medical schools or graduates of our own institution. Prior hospital safety training had no significant impact on interns’ ability to identify safety or low-value hazards. Overall, interns who were satisfied with their prior training identified a mean of 51.8% of hazards present (SD 11.8%), interns who were not satisfied with their prior training identified 51.5% (SD 12.7%), interns with no prior training identified 48.7% (SD 11.7%), and interns who were unsure about their prior training identified 47.4% (SD 11.5%) [F(3,117) = .79; P = .51, ANOVA]. There was also no significant association between prior training and the identification of any one of the 12 specific hazards (chi-square tests, all P values > .1).

Intern Feedback and Follow-Up Survey

Debriefing revealed that most interns passively assumed the patient’s chart was correct and did not think they should question the patient’s current care regimen. For example, many interns commented that they did not think to consider the patient’s blood transfusion as unnecessary, even though they were aware of the recommended hemoglobin cutoffs for stable patients.

Interns also provided formal feedback on the simulation through open-ended comments on their ScanTron^TM (Tustin, CA) form. For example, one intern wrote that they would “inherently approach every patient room ‘looking’ for safety issues, probably directly because of this exercise.” Another commented that the simulation was “more difficult than I expected, but very necessary to facilitate discussion and learning.” One intern wrote that “I wish I had done this earlier in my career.”

Ninety-six percent of participating interns (120/125) completed an online follow-up survey 1 month after beginning internship. In the survey, 68.9% (82/119) of interns indicated they were more aware of how to identify potential hazards facing hospitalized patients as a result of the simulation. Furthermore, 52.1% (62/119) of interns had taken action during internship to reduce a potential hazard that was present in the simulation.

While many GME orientations include simulation and safety training, this study is the first of its kind to incorporate low-value care from Choosing Wisely™ recommendations into simulated training. It is concerning that interns identified significantly fewer low-value hazards than safety hazards in the simulation. In some cases, no interns identified the low-value hazard. For example, while almost all interns identified the hand hygiene hazard, not one could identify the unnecessary stress ulcer prophylaxis. Furthermore, interns who self-reported as confident in their ability to identify hazards did not perform any better in the simulation. Interns entering less procedural-intensive specialties identified more safety hazards overall.

The simulation was well received by interns. Many commented that the experience was engaging, challenging, and effective in cultivating situational awareness towards low-value care. Our follow-up survey demonstrated the majority of interns reported taking action during their first month of internship to reduce a hazard included in the simulation. Most interns also reported a greater awareness of how to identify hospital hazards as a result of the simulation. These findings suggest that a brief simulation-based experience has the potential to create a lasting retention of situational awareness and behavior change.

It is worth exploring why interns identified significantly fewer low-value hazards than safety hazards in the simulation. One hypothesis is that interns were less attuned to low-value hazards, which may reflect a lacking emphasis on value-based care in undergraduate medical education (UME). It is especially concerning that so few interns identified the catheter-associated urinary tract infection (CAUTI) risk, as interns are primarily responsible for recognizing and removing an unnecessary catheter. Although the risks of low-value care should be apparent to most trainees, the process of recognizing and deliberately stopping or avoiding low-value care can be challenging for young clinicians.¹⁹ To promote value-based thinking among entering residents, UME programs should teach students to question the utility of the interventions their patients are receiving. One promising framework for doing so is the Subjective, Objective, Assessment, Plan- (SOAP)-V, in which a V for “Value” is added to the traditional SOAP note.²⁰ SOAP-V notes serve as a cognitive forcing function that requires students to pause and assess the value and cost-consciousness of their patients’ care.²⁰

The results from the “Room of Horrors” simulation can also guide health leaders and educators in identifying institutional areas of focus towards providing high-value and safe care. For example, at the University of Chicago we launched an initiative to improve the inappropriate use of urinary catheters after learning that few of our incoming interns recognized this during the simulation. Institutions could use this model to raise awareness of initiatives and redirect resources from areas that trainees perform well in (eg, hand hygiene) to areas that need improvement (eg, recognition of low-value care). Given the simulation’s low cost and minimal material requirements, it could be easily integrated into existing training programs with the support of an institution’s simulation center.

This study’s limitations include its conduction at single-institution, although the participants represented graduates of 60 different institutions. Furthermore, while the 12 hazards included in the simulation represent patient safety and value initiatives from a wide array of medical societies, they were not intended to be comprehensive and were not tailored to specific specialties. The simulation included only 4 low-value hazards, and future iterations of this exercise should aim to include an equal number of safety and low-value hazards. Furthermore, the evaluation of interns’ prior hospital safety training relied on self-reporting, and the specific context and content of each interns’ training was not examined. Finally, at this point we are unable to provide objective longitudinal data assessing the simulation’s impact on clinical practice and patient outcomes. Subsequent work will assess the sustained impact of the simulation by correlating with institutional data on measurable occurrences of low-value care.

In conclusion, interns identified significantly fewer low-value hazards than safety hazards in an inpatient simulation designed to promote situational awareness. Our results suggest that interns are on the lookout for errors of omission (eg, absence of hand hygiene, absence of isolation precautions) but are often blinded to errors of commission, such that when patients are started on therapies there is an assumption that the therapies are correct and necessary (eg, blood transfusions, stress ulcer prophylaxis). These findings suggest poor awareness of low-value care among incoming interns and highlight the need for additional training in both UME and GME to place a greater emphasis on preventing low-value care.

Disclosure

Dr. Arora is a member of the American Board of Medicine Board of Directors and has received grant funding from ABIM Foundation via Costs of Care for the Teaching Value Choosing Wisely™ Challenge. Dr. Farnan, Dr. Arora, and Ms. Hirsch receive grant funds from Accreditation Council of Graduate Medical Education as part of the Pursuing Excellence Initiative. Dr. Arora and Dr. Farnan also receive grant funds from the American Medical Association Accelerating Change in Medical Education initiative. Kathleen Wiest and Lukas Matern were funded through matching funds of the Pritzker Summer Research Program for NIA T35AG029795.

In recent years, the American Board of Internal Medicine (ABIM) Foundation’s Choosing Wisely™ campaign has advanced the dialogue on cost-consciousness by identifying potential examples of overuse in clinical practice.¹ Eliminating low-value care can decrease costs, improve quality, and potentially decrease patient harm.² In fact, there is growing consensus among health leaders and educators on the need for a physician workforce that is conscious of high-value care.^3,4 The Institute of Medicine has issued a call-to-action for graduate medical education (GME) to emphasize value-based care,⁵ and the Accreditation Council for Graduate Medical Education has outlined expectations that residents receive formal and experiential training on overuse as a part of its Clinical Learning Environment Review.⁶

However, recent reports highlight a lack of emphasis on value-based care in medical education.⁷ For example, few residency program directors believe that residents are prepared to incorporate value and cost into their medical decisions.⁸ In 2012, only 15% of medicine residencies reported having formal curricula addressing value, although many were developing one.⁸ Of the curricula reported, most were didactic in nature and did not include an assessment component.⁸

Experiential learning through simulation is one promising method to teach clinicians-in-training to practice value-based care. Simulation-based training promotes situational awareness (defined as being cognizant of one’s working environment), a concept that is crucial for recognizing both low-value and unsafe care.^9,10 Simulated training exercises are often included in GME orientation “boot-camps,” which have typically addressed safety.¹¹ The incorporation of value into existing GME boot-camp exercises could provide a promising model for the addition of value-based training to GME.

At the University of Chicago, we had successfully implemented the “Room of Horrors,” a simulation for entering interns to promote the detection of patient safety hazards.¹¹ Here, we describe a modification to this simulation to embed low-value hazards in addition to traditional patient safety hazards. The aim of this study is to assess the entering interns’ recognition of low-value care and their ability to recognize unsafe care in a simulation designed to promote situational awareness.

Setting and Participants

The simulation was conducted during GME orientation at a large, urban academic medical institution. One hundred and twenty-five entering postgraduate year one (PGY1) interns participated in the simulation, which was a required component of a multiday orientation “boot-camp” experience. All eligible interns participated in the simulation, representing 13 specialty programs and 60 medical schools. Interns entering into pathology were excluded because of infrequent patient contact. Participating interns were divided into 7 specialty groups for analysis in order to preserve the anonymity of interns in smaller residency programs (surgical subspecialties combined with general surgery, medicine-pediatrics grouped with internal medicine). The University of Chicago Institutional Review Board deemed this study exempt from review.

Program Description

A simulation of an inpatient hospital room, known as the “Room of Horrors,” was constructed in collaboration with the University of Chicago Simulation Center and adapted from a previous version of the exercise.¹¹ The simulation consisted of a mock door chart highlighting the patient had been admitted for diarrhea (Clostridium difficile positive) following a recent hospitalization for pneumonia. A clinical scenario was constructed by using a patient mannequin and an accompanying door chart that listed information on the patient’s hospital course, allergies, and medications. In addition to the 8 patient safety hazards utilized in the prior version, our team selected 4 low-value hazards to be included in the simulation.

The 8 safety hazards have been detailed in a prior study and were previously selected from Medicare’s Hospital-Acquired Conditions (HAC) Reduction Program and Agency for Healthcare Research and Quality (AHRQ) Patient Safety Indicators.^11-13 Each of the hazards was represented either physically in the simulation room and/or was indicated on the patient’s chart. For example, the latex allergy hazard was represented by latex gloves at the bedside despite an allergy indicated on the patient’s chart and wristband. A complete list of the 8 safety hazards and their representations in the simulation is shown in Table 1.

The Choosing Wisely™ lists were reviewed to identify low-value hazards for addition to the simulation.¹⁴ Our team selected 3 low-value hazards from the Society of Hospital Medicine (SHM) list,¹⁵ including (1) arbitrary blood transfusion despite the patient’s stable hemoglobin level of 8.0 g/dL and absence of cardiac symptoms,¹⁶ (2) addition of a proton pump inhibitor (PPI) for stress ulcer prophylaxis in a patient without high risk for gastrointestinal (GI) complications who was not on a PPI prior to admission, and (3) placement of a urinary catheter without medical indication. We had originally selected continuous telemetry monitoring as a fourth hazard from the SHM list, but were unable to operationalize, as it was difficult to simulate continuous telemetry on a mannequin. Because many inpatients are older than 65 years, we reviewed the American Geriatrics Society list¹⁷ and selected our fourth low-value hazard: (4) unnecessary use of physical restraints to manage behavioral symptoms in a hospitalized patient with delirium. Several of these hazards were also quality and safety priorities at our institution, including the overuse of urinary catheters, physical restraints, and blood transfusions. All 4 low-value hazards were referenced in the patient’s door chart, and 3 were also physically represented in the room via presence of a hanging unit of blood, Foley catheter, and upper-arm restraints (Table 1). See Appendix for a photograph of the simulation setup.

Each intern was allowed 10 minutes inside the simulation room. During this time, they were instructed to read the 1-page door chart, inspect the simulation room, and write down as many potential low-value and safety hazards as they could identify on a free-response form (see Appendix). Upon exiting the room, they were allotted 5 additional minutes to complete their free-response answers and provide written feedback on the simulation. The simulation was conducted in 3 simulated hospital rooms over the course of 2 days, and the correct answers were provided via e-mail after all interns had completed the exercise.

To assess prior training and safety knowledge, interns were asked to complete a 3-question preassessment on a ScanTron^TM (Tustin, CA) form. The preassessment asked interns whether they had received training on hospital safety during medical school (yes, no, or unsure), if they were satisfied with the hospital safety training they received during medical school (strongly disagree to strongly agree on a Likert scale), and if they were confident in their ability to identify potential hazards in a hospital setting (strongly disagree to strongly agree). Interns were also given the opportunity to provide feedback on the simulation experience on the ScanTron^TM (Tustin, CA) form.

One month after participating in the simulation, interns were asked to complete an online follow-up survey on MedHub^TM (Ann Arbor, MI), which included 2 Likert-scale questions (strongly disagree to strongly agree) assessing the simulation’s impact on their experience mitigating hospital hazards during the first month of internship.

Data Analysis

Interns’ free-response answers were manually coded, and descriptive statistics were used to summarize the mean percent correct for each hazard. A paired t test was used to compare intern identification of low-value vs safety hazards. T tests were used to compare hazard identification for interns entering highly procedural-intensive specialties (ie, surgical specialties, emergency medicine, anesthesia, obstetrics/gynecology) and those entering less procedural-intensive specialties (ie, internal medicine, pediatrics, psychiatry), as well as among those graduating from “Top 30” medical schools (based on US News & World Report Medical School Rankings¹⁸) and our own institution. One-way analysis of variance (ANOVA) calculations were used to test for differences in hazard identification based on interns’ prior hospital safety training, with interns who rated their satisfaction with prior training or confidence in identifying hazards as a “4” or a “5” considered “satisfied” and “confident,” respectively. Responses to the MedHub^TM (Ann Arbor, MI) survey were dichotomized with “strongly agree” and “agree” considered positive responses. Statistical significance was defined at P < .05. All data analysis was conducted using Stata 14^TM software (College Station, TX).

Intern Characteristics

One hundred twenty-five entering PGY1 interns participated in the simulation, representing 60 medical schools and 7 different specialty groups (Table 2). Thirty-five percent (44/125) were graduates from “Top 30” medical schools, and 8.8% (11/125) graduated from our own institution. Seventy-four percent (89/121) had received prior hospital safety training during medical school, and 62.9% (56/89) were satisfied with their training. A majority of interns (64.2%, 79/123) felt confident in their ability to identify potential hazards in a hospital setting, although confidence was much higher among those with prior safety training (71.9%, 64/89) compared to those without prior training or who were unsure about their training (40.6%, 13/32; P = .09, t test).

Identification of Hazards

The mean percentage of hazards correctly identified by interns during the simulation was 50.4% (standard deviation [SD] 11.8%), with a normal distribution (Figure 1). Interns identified a significantly lower percentage of low-value hazards than safety hazards in the simulation (mean 19.2% [SD 18.6%] vs 66.0% [SD 16.0%], respectively; P < .001, paired t test). Interns also identified significantly more room-based errors than chart-based errors (mean 58.6% [SD 13.4%] vs 9.6% [SD 19.8%], respectively; P < .001, paired t test). The 3 most commonly identified hazards were unavailability of hand hygiene (120/125, 96.0%), presence of latex gloves despite the patient’s allergy (111/125, 88.8%), and fall risk due to the lowered bed rail (107/125, 85.6%). More than half of interns identified the incorrect name on the patient’s wristband and IV bag (91/125, 72.8%), a lack of isolation precautions (90/125, 72.0%), administration of penicillin despite the patient’s allergy (67/125, 53.6%), and unnecessary restraints (64/125, 51.2%). Less than half of interns identified the wrong medication being administered (50/125, 40.0%), unnecessary Foley catheter (25/125, 20.0%), and absence of venous thromboembolism (VTE) prophylaxis (24/125, 19.2%). Few interns identified the unnecessary blood transfusion (7/125, 5.6%), and no one identified the unnecessary stress ulcer prophylaxis (0/125, 0.0%; Figure 2).

Predictors of Hazard Identification

Interns who self-reported as confident in their ability to identify hazards were not any more likely to correctly identify hazards than those who were not confident (50.9% overall hazard identification vs 49.6%, respectively; P = .56, t test). Interns entering into less procedural-intensive specialties identified significantly more safety hazards than those entering highly procedural-intensive specialties (mean 69.1% [SD 16.9%] vs 61.8% [SD 13.7%], respectively; P = .01, t test). However, there was no statistically significant difference in their identification of low-value hazards (mean 19.8% [SD 18.3%] for less procedural-intensive vs 18.4% [SD 19.1%] for highly procedural-intensive; P = .68, t test). There was no statistically significant difference in hazard identification among graduates of “Top 30” medical schools or graduates of our own institution. Prior hospital safety training had no significant impact on interns’ ability to identify safety or low-value hazards. Overall, interns who were satisfied with their prior training identified a mean of 51.8% of hazards present (SD 11.8%), interns who were not satisfied with their prior training identified 51.5% (SD 12.7%), interns with no prior training identified 48.7% (SD 11.7%), and interns who were unsure about their prior training identified 47.4% (SD 11.5%) [F(3,117) = .79; P = .51, ANOVA]. There was also no significant association between prior training and the identification of any one of the 12 specific hazards (chi-square tests, all P values > .1).

Intern Feedback and Follow-Up Survey

Debriefing revealed that most interns passively assumed the patient’s chart was correct and did not think they should question the patient’s current care regimen. For example, many interns commented that they did not think to consider the patient’s blood transfusion as unnecessary, even though they were aware of the recommended hemoglobin cutoffs for stable patients.

Interns also provided formal feedback on the simulation through open-ended comments on their ScanTron^TM (Tustin, CA) form. For example, one intern wrote that they would “inherently approach every patient room ‘looking’ for safety issues, probably directly because of this exercise.” Another commented that the simulation was “more difficult than I expected, but very necessary to facilitate discussion and learning.” One intern wrote that “I wish I had done this earlier in my career.”

Ninety-six percent of participating interns (120/125) completed an online follow-up survey 1 month after beginning internship. In the survey, 68.9% (82/119) of interns indicated they were more aware of how to identify potential hazards facing hospitalized patients as a result of the simulation. Furthermore, 52.1% (62/119) of interns had taken action during internship to reduce a potential hazard that was present in the simulation.

While many GME orientations include simulation and safety training, this study is the first of its kind to incorporate low-value care from Choosing Wisely™ recommendations into simulated training. It is concerning that interns identified significantly fewer low-value hazards than safety hazards in the simulation. In some cases, no interns identified the low-value hazard. For example, while almost all interns identified the hand hygiene hazard, not one could identify the unnecessary stress ulcer prophylaxis. Furthermore, interns who self-reported as confident in their ability to identify hazards did not perform any better in the simulation. Interns entering less procedural-intensive specialties identified more safety hazards overall.

The simulation was well received by interns. Many commented that the experience was engaging, challenging, and effective in cultivating situational awareness towards low-value care. Our follow-up survey demonstrated the majority of interns reported taking action during their first month of internship to reduce a hazard included in the simulation. Most interns also reported a greater awareness of how to identify hospital hazards as a result of the simulation. These findings suggest that a brief simulation-based experience has the potential to create a lasting retention of situational awareness and behavior change.

It is worth exploring why interns identified significantly fewer low-value hazards than safety hazards in the simulation. One hypothesis is that interns were less attuned to low-value hazards, which may reflect a lacking emphasis on value-based care in undergraduate medical education (UME). It is especially concerning that so few interns identified the catheter-associated urinary tract infection (CAUTI) risk, as interns are primarily responsible for recognizing and removing an unnecessary catheter. Although the risks of low-value care should be apparent to most trainees, the process of recognizing and deliberately stopping or avoiding low-value care can be challenging for young clinicians.¹⁹ To promote value-based thinking among entering residents, UME programs should teach students to question the utility of the interventions their patients are receiving. One promising framework for doing so is the Subjective, Objective, Assessment, Plan- (SOAP)-V, in which a V for “Value” is added to the traditional SOAP note.²⁰ SOAP-V notes serve as a cognitive forcing function that requires students to pause and assess the value and cost-consciousness of their patients’ care.²⁰

The results from the “Room of Horrors” simulation can also guide health leaders and educators in identifying institutional areas of focus towards providing high-value and safe care. For example, at the University of Chicago we launched an initiative to improve the inappropriate use of urinary catheters after learning that few of our incoming interns recognized this during the simulation. Institutions could use this model to raise awareness of initiatives and redirect resources from areas that trainees perform well in (eg, hand hygiene) to areas that need improvement (eg, recognition of low-value care). Given the simulation’s low cost and minimal material requirements, it could be easily integrated into existing training programs with the support of an institution’s simulation center.

This study’s limitations include its conduction at single-institution, although the participants represented graduates of 60 different institutions. Furthermore, while the 12 hazards included in the simulation represent patient safety and value initiatives from a wide array of medical societies, they were not intended to be comprehensive and were not tailored to specific specialties. The simulation included only 4 low-value hazards, and future iterations of this exercise should aim to include an equal number of safety and low-value hazards. Furthermore, the evaluation of interns’ prior hospital safety training relied on self-reporting, and the specific context and content of each interns’ training was not examined. Finally, at this point we are unable to provide objective longitudinal data assessing the simulation’s impact on clinical practice and patient outcomes. Subsequent work will assess the sustained impact of the simulation by correlating with institutional data on measurable occurrences of low-value care.

In conclusion, interns identified significantly fewer low-value hazards than safety hazards in an inpatient simulation designed to promote situational awareness. Our results suggest that interns are on the lookout for errors of omission (eg, absence of hand hygiene, absence of isolation precautions) but are often blinded to errors of commission, such that when patients are started on therapies there is an assumption that the therapies are correct and necessary (eg, blood transfusions, stress ulcer prophylaxis). These findings suggest poor awareness of low-value care among incoming interns and highlight the need for additional training in both UME and GME to place a greater emphasis on preventing low-value care.

Disclosure

Dr. Arora is a member of the American Board of Medicine Board of Directors and has received grant funding from ABIM Foundation via Costs of Care for the Teaching Value Choosing Wisely™ Challenge. Dr. Farnan, Dr. Arora, and Ms. Hirsch receive grant funds from Accreditation Council of Graduate Medical Education as part of the Pursuing Excellence Initiative. Dr. Arora and Dr. Farnan also receive grant funds from the American Medical Association Accelerating Change in Medical Education initiative. Kathleen Wiest and Lukas Matern were funded through matching funds of the Pritzker Summer Research Program for NIA T35AG029795.

In recent years, the American Board of Internal Medicine (ABIM) Foundation’s Choosing Wisely™ campaign has advanced the dialogue on cost-consciousness by identifying potential examples of overuse in clinical practice.¹ Eliminating low-value care can decrease costs, improve quality, and potentially decrease patient harm.² In fact, there is growing consensus among health leaders and educators on the need for a physician workforce that is conscious of high-value care.^3,4 The Institute of Medicine has issued a call-to-action for graduate medical education (GME) to emphasize value-based care,⁵ and the Accreditation Council for Graduate Medical Education has outlined expectations that residents receive formal and experiential training on overuse as a part of its Clinical Learning Environment Review.⁶

However, recent reports highlight a lack of emphasis on value-based care in medical education.⁷ For example, few residency program directors believe that residents are prepared to incorporate value and cost into their medical decisions.⁸ In 2012, only 15% of medicine residencies reported having formal curricula addressing value, although many were developing one.⁸ Of the curricula reported, most were didactic in nature and did not include an assessment component.⁸

Experiential learning through simulation is one promising method to teach clinicians-in-training to practice value-based care. Simulation-based training promotes situational awareness (defined as being cognizant of one’s working environment), a concept that is crucial for recognizing both low-value and unsafe care.^9,10 Simulated training exercises are often included in GME orientation “boot-camps,” which have typically addressed safety.¹¹ The incorporation of value into existing GME boot-camp exercises could provide a promising model for the addition of value-based training to GME.

At the University of Chicago, we had successfully implemented the “Room of Horrors,” a simulation for entering interns to promote the detection of patient safety hazards.¹¹ Here, we describe a modification to this simulation to embed low-value hazards in addition to traditional patient safety hazards. The aim of this study is to assess the entering interns’ recognition of low-value care and their ability to recognize unsafe care in a simulation designed to promote situational awareness.

Setting and Participants

The simulation was conducted during GME orientation at a large, urban academic medical institution. One hundred and twenty-five entering postgraduate year one (PGY1) interns participated in the simulation, which was a required component of a multiday orientation “boot-camp” experience. All eligible interns participated in the simulation, representing 13 specialty programs and 60 medical schools. Interns entering into pathology were excluded because of infrequent patient contact. Participating interns were divided into 7 specialty groups for analysis in order to preserve the anonymity of interns in smaller residency programs (surgical subspecialties combined with general surgery, medicine-pediatrics grouped with internal medicine). The University of Chicago Institutional Review Board deemed this study exempt from review.

Program Description

A simulation of an inpatient hospital room, known as the “Room of Horrors,” was constructed in collaboration with the University of Chicago Simulation Center and adapted from a previous version of the exercise.¹¹ The simulation consisted of a mock door chart highlighting the patient had been admitted for diarrhea (Clostridium difficile positive) following a recent hospitalization for pneumonia. A clinical scenario was constructed by using a patient mannequin and an accompanying door chart that listed information on the patient’s hospital course, allergies, and medications. In addition to the 8 patient safety hazards utilized in the prior version, our team selected 4 low-value hazards to be included in the simulation.

The 8 safety hazards have been detailed in a prior study and were previously selected from Medicare’s Hospital-Acquired Conditions (HAC) Reduction Program and Agency for Healthcare Research and Quality (AHRQ) Patient Safety Indicators.^11-13 Each of the hazards was represented either physically in the simulation room and/or was indicated on the patient’s chart. For example, the latex allergy hazard was represented by latex gloves at the bedside despite an allergy indicated on the patient’s chart and wristband. A complete list of the 8 safety hazards and their representations in the simulation is shown in Table 1.

The Choosing Wisely™ lists were reviewed to identify low-value hazards for addition to the simulation.¹⁴ Our team selected 3 low-value hazards from the Society of Hospital Medicine (SHM) list,¹⁵ including (1) arbitrary blood transfusion despite the patient’s stable hemoglobin level of 8.0 g/dL and absence of cardiac symptoms,¹⁶ (2) addition of a proton pump inhibitor (PPI) for stress ulcer prophylaxis in a patient without high risk for gastrointestinal (GI) complications who was not on a PPI prior to admission, and (3) placement of a urinary catheter without medical indication. We had originally selected continuous telemetry monitoring as a fourth hazard from the SHM list, but were unable to operationalize, as it was difficult to simulate continuous telemetry on a mannequin. Because many inpatients are older than 65 years, we reviewed the American Geriatrics Society list¹⁷ and selected our fourth low-value hazard: (4) unnecessary use of physical restraints to manage behavioral symptoms in a hospitalized patient with delirium. Several of these hazards were also quality and safety priorities at our institution, including the overuse of urinary catheters, physical restraints, and blood transfusions. All 4 low-value hazards were referenced in the patient’s door chart, and 3 were also physically represented in the room via presence of a hanging unit of blood, Foley catheter, and upper-arm restraints (Table 1). See Appendix for a photograph of the simulation setup.

Each intern was allowed 10 minutes inside the simulation room. During this time, they were instructed to read the 1-page door chart, inspect the simulation room, and write down as many potential low-value and safety hazards as they could identify on a free-response form (see Appendix). Upon exiting the room, they were allotted 5 additional minutes to complete their free-response answers and provide written feedback on the simulation. The simulation was conducted in 3 simulated hospital rooms over the course of 2 days, and the correct answers were provided via e-mail after all interns had completed the exercise.

To assess prior training and safety knowledge, interns were asked to complete a 3-question preassessment on a ScanTron^TM (Tustin, CA) form. The preassessment asked interns whether they had received training on hospital safety during medical school (yes, no, or unsure), if they were satisfied with the hospital safety training they received during medical school (strongly disagree to strongly agree on a Likert scale), and if they were confident in their ability to identify potential hazards in a hospital setting (strongly disagree to strongly agree). Interns were also given the opportunity to provide feedback on the simulation experience on the ScanTron^TM (Tustin, CA) form.

One month after participating in the simulation, interns were asked to complete an online follow-up survey on MedHub^TM (Ann Arbor, MI), which included 2 Likert-scale questions (strongly disagree to strongly agree) assessing the simulation’s impact on their experience mitigating hospital hazards during the first month of internship.

Data Analysis

Interns’ free-response answers were manually coded, and descriptive statistics were used to summarize the mean percent correct for each hazard. A paired t test was used to compare intern identification of low-value vs safety hazards. T tests were used to compare hazard identification for interns entering highly procedural-intensive specialties (ie, surgical specialties, emergency medicine, anesthesia, obstetrics/gynecology) and those entering less procedural-intensive specialties (ie, internal medicine, pediatrics, psychiatry), as well as among those graduating from “Top 30” medical schools (based on US News & World Report Medical School Rankings¹⁸) and our own institution. One-way analysis of variance (ANOVA) calculations were used to test for differences in hazard identification based on interns’ prior hospital safety training, with interns who rated their satisfaction with prior training or confidence in identifying hazards as a “4” or a “5” considered “satisfied” and “confident,” respectively. Responses to the MedHub^TM (Ann Arbor, MI) survey were dichotomized with “strongly agree” and “agree” considered positive responses. Statistical significance was defined at P < .05. All data analysis was conducted using Stata 14^TM software (College Station, TX).

Intern Characteristics

One hundred twenty-five entering PGY1 interns participated in the simulation, representing 60 medical schools and 7 different specialty groups (Table 2). Thirty-five percent (44/125) were graduates from “Top 30” medical schools, and 8.8% (11/125) graduated from our own institution. Seventy-four percent (89/121) had received prior hospital safety training during medical school, and 62.9% (56/89) were satisfied with their training. A majority of interns (64.2%, 79/123) felt confident in their ability to identify potential hazards in a hospital setting, although confidence was much higher among those with prior safety training (71.9%, 64/89) compared to those without prior training or who were unsure about their training (40.6%, 13/32; P = .09, t test).

Identification of Hazards

The mean percentage of hazards correctly identified by interns during the simulation was 50.4% (standard deviation [SD] 11.8%), with a normal distribution (Figure 1). Interns identified a significantly lower percentage of low-value hazards than safety hazards in the simulation (mean 19.2% [SD 18.6%] vs 66.0% [SD 16.0%], respectively; P < .001, paired t test). Interns also identified significantly more room-based errors than chart-based errors (mean 58.6% [SD 13.4%] vs 9.6% [SD 19.8%], respectively; P < .001, paired t test). The 3 most commonly identified hazards were unavailability of hand hygiene (120/125, 96.0%), presence of latex gloves despite the patient’s allergy (111/125, 88.8%), and fall risk due to the lowered bed rail (107/125, 85.6%). More than half of interns identified the incorrect name on the patient’s wristband and IV bag (91/125, 72.8%), a lack of isolation precautions (90/125, 72.0%), administration of penicillin despite the patient’s allergy (67/125, 53.6%), and unnecessary restraints (64/125, 51.2%). Less than half of interns identified the wrong medication being administered (50/125, 40.0%), unnecessary Foley catheter (25/125, 20.0%), and absence of venous thromboembolism (VTE) prophylaxis (24/125, 19.2%). Few interns identified the unnecessary blood transfusion (7/125, 5.6%), and no one identified the unnecessary stress ulcer prophylaxis (0/125, 0.0%; Figure 2).

Predictors of Hazard Identification

Interns who self-reported as confident in their ability to identify hazards were not any more likely to correctly identify hazards than those who were not confident (50.9% overall hazard identification vs 49.6%, respectively; P = .56, t test). Interns entering into less procedural-intensive specialties identified significantly more safety hazards than those entering highly procedural-intensive specialties (mean 69.1% [SD 16.9%] vs 61.8% [SD 13.7%], respectively; P = .01, t test). However, there was no statistically significant difference in their identification of low-value hazards (mean 19.8% [SD 18.3%] for less procedural-intensive vs 18.4% [SD 19.1%] for highly procedural-intensive; P = .68, t test). There was no statistically significant difference in hazard identification among graduates of “Top 30” medical schools or graduates of our own institution. Prior hospital safety training had no significant impact on interns’ ability to identify safety or low-value hazards. Overall, interns who were satisfied with their prior training identified a mean of 51.8% of hazards present (SD 11.8%), interns who were not satisfied with their prior training identified 51.5% (SD 12.7%), interns with no prior training identified 48.7% (SD 11.7%), and interns who were unsure about their prior training identified 47.4% (SD 11.5%) [F(3,117) = .79; P = .51, ANOVA]. There was also no significant association between prior training and the identification of any one of the 12 specific hazards (chi-square tests, all P values > .1).

Intern Feedback and Follow-Up Survey

Debriefing revealed that most interns passively assumed the patient’s chart was correct and did not think they should question the patient’s current care regimen. For example, many interns commented that they did not think to consider the patient’s blood transfusion as unnecessary, even though they were aware of the recommended hemoglobin cutoffs for stable patients.

Interns also provided formal feedback on the simulation through open-ended comments on their ScanTron^TM (Tustin, CA) form. For example, one intern wrote that they would “inherently approach every patient room ‘looking’ for safety issues, probably directly because of this exercise.” Another commented that the simulation was “more difficult than I expected, but very necessary to facilitate discussion and learning.” One intern wrote that “I wish I had done this earlier in my career.”

Ninety-six percent of participating interns (120/125) completed an online follow-up survey 1 month after beginning internship. In the survey, 68.9% (82/119) of interns indicated they were more aware of how to identify potential hazards facing hospitalized patients as a result of the simulation. Furthermore, 52.1% (62/119) of interns had taken action during internship to reduce a potential hazard that was present in the simulation.

While many GME orientations include simulation and safety training, this study is the first of its kind to incorporate low-value care from Choosing Wisely™ recommendations into simulated training. It is concerning that interns identified significantly fewer low-value hazards than safety hazards in the simulation. In some cases, no interns identified the low-value hazard. For example, while almost all interns identified the hand hygiene hazard, not one could identify the unnecessary stress ulcer prophylaxis. Furthermore, interns who self-reported as confident in their ability to identify hazards did not perform any better in the simulation. Interns entering less procedural-intensive specialties identified more safety hazards overall.

The simulation was well received by interns. Many commented that the experience was engaging, challenging, and effective in cultivating situational awareness towards low-value care. Our follow-up survey demonstrated the majority of interns reported taking action during their first month of internship to reduce a hazard included in the simulation. Most interns also reported a greater awareness of how to identify hospital hazards as a result of the simulation. These findings suggest that a brief simulation-based experience has the potential to create a lasting retention of situational awareness and behavior change.

It is worth exploring why interns identified significantly fewer low-value hazards than safety hazards in the simulation. One hypothesis is that interns were less attuned to low-value hazards, which may reflect a lacking emphasis on value-based care in undergraduate medical education (UME). It is especially concerning that so few interns identified the catheter-associated urinary tract infection (CAUTI) risk, as interns are primarily responsible for recognizing and removing an unnecessary catheter. Although the risks of low-value care should be apparent to most trainees, the process of recognizing and deliberately stopping or avoiding low-value care can be challenging for young clinicians.¹⁹ To promote value-based thinking among entering residents, UME programs should teach students to question the utility of the interventions their patients are receiving. One promising framework for doing so is the Subjective, Objective, Assessment, Plan- (SOAP)-V, in which a V for “Value” is added to the traditional SOAP note.²⁰ SOAP-V notes serve as a cognitive forcing function that requires students to pause and assess the value and cost-consciousness of their patients’ care.²⁰

The results from the “Room of Horrors” simulation can also guide health leaders and educators in identifying institutional areas of focus towards providing high-value and safe care. For example, at the University of Chicago we launched an initiative to improve the inappropriate use of urinary catheters after learning that few of our incoming interns recognized this during the simulation. Institutions could use this model to raise awareness of initiatives and redirect resources from areas that trainees perform well in (eg, hand hygiene) to areas that need improvement (eg, recognition of low-value care). Given the simulation’s low cost and minimal material requirements, it could be easily integrated into existing training programs with the support of an institution’s simulation center.

This study’s limitations include its conduction at single-institution, although the participants represented graduates of 60 different institutions. Furthermore, while the 12 hazards included in the simulation represent patient safety and value initiatives from a wide array of medical societies, they were not intended to be comprehensive and were not tailored to specific specialties. The simulation included only 4 low-value hazards, and future iterations of this exercise should aim to include an equal number of safety and low-value hazards. Furthermore, the evaluation of interns’ prior hospital safety training relied on self-reporting, and the specific context and content of each interns’ training was not examined. Finally, at this point we are unable to provide objective longitudinal data assessing the simulation’s impact on clinical practice and patient outcomes. Subsequent work will assess the sustained impact of the simulation by correlating with institutional data on measurable occurrences of low-value care.

In conclusion, interns identified significantly fewer low-value hazards than safety hazards in an inpatient simulation designed to promote situational awareness. Our results suggest that interns are on the lookout for errors of omission (eg, absence of hand hygiene, absence of isolation precautions) but are often blinded to errors of commission, such that when patients are started on therapies there is an assumption that the therapies are correct and necessary (eg, blood transfusions, stress ulcer prophylaxis). These findings suggest poor awareness of low-value care among incoming interns and highlight the need for additional training in both UME and GME to place a greater emphasis on preventing low-value care.

Disclosure

Dr. Arora is a member of the American Board of Medicine Board of Directors and has received grant funding from ABIM Foundation via Costs of Care for the Teaching Value Choosing Wisely™ Challenge. Dr. Farnan, Dr. Arora, and Ms. Hirsch receive grant funds from Accreditation Council of Graduate Medical Education as part of the Pursuing Excellence Initiative. Dr. Arora and Dr. Farnan also receive grant funds from the American Medical Association Accelerating Change in Medical Education initiative. Kathleen Wiest and Lukas Matern were funded through matching funds of the Pritzker Summer Research Program for NIA T35AG029795.

The VA and the Department of Energy (DoE) have formed a new partnership focused on secure analysis of “big data.” The VA-DoE Big Data Science Initiative will use digital health and genomic data from the Million Veteran Program (MVP), the VA’s electronic health records system, DoD, Centers for Medicare and Medicaid Services, and the CDC’s National Death Index.

The partnership is based in DoE’s National Laboratory system, one of the world’s top resources for supercomputing, where machines are capable of millions of billions of calculations per second. The partnership will allow thousands of researchers access to this unprecedented data resource over time in a secure environment, said VA Secretary David J. Shulkin, MD.

An initial suite of specific studies is already being planned, the VA says. One group of researchers will build algorithms to generate “highly tailored” risk scores for suicide, which could help VA clinicians and researchers predict which patients are at highest risk and evaluate prevention strategies.

Other projects include one to find new ways to distinguish lethal from nonlethal prostate cancer and another to determine which risk factors best predict certain forms of cardiovascular disease.

“The transformative science that will be developed through this partnership,” Shulkin says, “will improve health care for veterans and all Americans.”

The VA and the Department of Energy (DoE) have formed a new partnership focused on secure analysis of “big data.” The VA-DoE Big Data Science Initiative will use digital health and genomic data from the Million Veteran Program (MVP), the VA’s electronic health records system, DoD, Centers for Medicare and Medicaid Services, and the CDC’s National Death Index.

The partnership is based in DoE’s National Laboratory system, one of the world’s top resources for supercomputing, where machines are capable of millions of billions of calculations per second. The partnership will allow thousands of researchers access to this unprecedented data resource over time in a secure environment, said VA Secretary David J. Shulkin, MD.

An initial suite of specific studies is already being planned, the VA says. One group of researchers will build algorithms to generate “highly tailored” risk scores for suicide, which could help VA clinicians and researchers predict which patients are at highest risk and evaluate prevention strategies.

Other projects include one to find new ways to distinguish lethal from nonlethal prostate cancer and another to determine which risk factors best predict certain forms of cardiovascular disease.

“The transformative science that will be developed through this partnership,” Shulkin says, “will improve health care for veterans and all Americans.”

The VA and the Department of Energy (DoE) have formed a new partnership focused on secure analysis of “big data.” The VA-DoE Big Data Science Initiative will use digital health and genomic data from the Million Veteran Program (MVP), the VA’s electronic health records system, DoD, Centers for Medicare and Medicaid Services, and the CDC’s National Death Index.

The partnership is based in DoE’s National Laboratory system, one of the world’s top resources for supercomputing, where machines are capable of millions of billions of calculations per second. The partnership will allow thousands of researchers access to this unprecedented data resource over time in a secure environment, said VA Secretary David J. Shulkin, MD.

An initial suite of specific studies is already being planned, the VA says. One group of researchers will build algorithms to generate “highly tailored” risk scores for suicide, which could help VA clinicians and researchers predict which patients are at highest risk and evaluate prevention strategies.

Other projects include one to find new ways to distinguish lethal from nonlethal prostate cancer and another to determine which risk factors best predict certain forms of cardiovascular disease.

“The transformative science that will be developed through this partnership,” Shulkin says, “will improve health care for veterans and all Americans.”

When military and VA nurses work side by side—learning from and teaching each other—they benefit and so do their patients. A “unique partnership” between the DoD and VA is proving that at Captain James A. Lovell Federal Health Care Center outside Chicago, Illinois.

The first of its kind facility serves nearly 67,000 active-duty military, military retirees, family members, and veterans. In an article for Health.mil News, U.S. Navy Lt. Nathan Aranas, an active-duty registered nurse (RN) and assistant nurse manager in the emergency department (ED), says, “We learn from local trauma, mental health, and pediatrics and birthing centers, exposing me more to how medicine outside of the military is practiced. It gives me a bigger perspective of how the rest of the country operates as a health care institution.”

Christine Barassi-Jackson, a VA civilian RN, nurse manager in the ED, says “having a combined organization is a great balance that pulls out the best parts of both the Navy and VA.” She leans on Aranas, the article says, to serve as an interpreter with some of the patients. “Knowing more of the Navy culture helps break down walls with the patients and other providers.” Aranas also believes that former active-duty patients may be more at ease with a uniformed nurse “because they understand the lingo.”

Overall, Aranas says, “It’s a great experience for young, active-duty clinicians to have."

When military and VA nurses work side by side—learning from and teaching each other—they benefit and so do their patients. A “unique partnership” between the DoD and VA is proving that at Captain James A. Lovell Federal Health Care Center outside Chicago, Illinois.

The first of its kind facility serves nearly 67,000 active-duty military, military retirees, family members, and veterans. In an article for Health.mil News, U.S. Navy Lt. Nathan Aranas, an active-duty registered nurse (RN) and assistant nurse manager in the emergency department (ED), says, “We learn from local trauma, mental health, and pediatrics and birthing centers, exposing me more to how medicine outside of the military is practiced. It gives me a bigger perspective of how the rest of the country operates as a health care institution.”

Christine Barassi-Jackson, a VA civilian RN, nurse manager in the ED, says “having a combined organization is a great balance that pulls out the best parts of both the Navy and VA.” She leans on Aranas, the article says, to serve as an interpreter with some of the patients. “Knowing more of the Navy culture helps break down walls with the patients and other providers.” Aranas also believes that former active-duty patients may be more at ease with a uniformed nurse “because they understand the lingo.”

Overall, Aranas says, “It’s a great experience for young, active-duty clinicians to have."

When military and VA nurses work side by side—learning from and teaching each other—they benefit and so do their patients. A “unique partnership” between the DoD and VA is proving that at Captain James A. Lovell Federal Health Care Center outside Chicago, Illinois.

The first of its kind facility serves nearly 67,000 active-duty military, military retirees, family members, and veterans. In an article for Health.mil News, U.S. Navy Lt. Nathan Aranas, an active-duty registered nurse (RN) and assistant nurse manager in the emergency department (ED), says, “We learn from local trauma, mental health, and pediatrics and birthing centers, exposing me more to how medicine outside of the military is practiced. It gives me a bigger perspective of how the rest of the country operates as a health care institution.”

Christine Barassi-Jackson, a VA civilian RN, nurse manager in the ED, says “having a combined organization is a great balance that pulls out the best parts of both the Navy and VA.” She leans on Aranas, the article says, to serve as an interpreter with some of the patients. “Knowing more of the Navy culture helps break down walls with the patients and other providers.” Aranas also believes that former active-duty patients may be more at ease with a uniformed nurse “because they understand the lingo.”

Overall, Aranas says, “It’s a great experience for young, active-duty clinicians to have."

In-hospital mobility (walking and transferring) is an important modifiable factor for posthospital functional outcomes and mortality among older adults.^1-4 In fact, daily mobility assessment has been considered for a standard clinical evaluation of the hospitalized older adult.^5,6 This would provide a ready source for targeting patients at risk for mobility impairment and identifying strategies to prevent in-hospital mobility limitation and posthospital functional decline. Despite their potential importance, mobility assessment tools have not been readily adopted in the hospital setting.

There are various ways to assess mobility in hospital settings. Mobility tracking technology (radar and accelerometers) has demonstrated older adults have extremely low mobility during hospitalization. Although these objective methods provide an unbiased way to monitor physical activity level and track in-hospital mobility change,^6-8 and have provided important information about mobility in the hospital, they are largely impractical in real-world settings.

While mobility technology appears to be advancing, there is a potential to assess in-hospital mobility using commonly administered and inexpensive tools. Many hospitals ask staff to regularly rate physical function (Braden and Morse score) as part of their standard-of-care procedures. The rating scales used have the potential to provide valuable information about mobility variations without using special equipment or burdening patients. The Braden Scale for Predicting Pressure Sore Risk is a good example of a validated assessment instrument that is better than nurses’ judgment, which is often confounded by nursing experience.⁹ This scale, which has 6 subscales (Sensory Perception, Moisture, Activity, Mobility, Nutrition, Friction and Shear), has shown high sensitivity in detecting patient condition changes in the clinical setting.¹⁰ The scale typically is used holistically to evaluate pressure ulcer risk, but the Activity subscale, which assesses mobility, could serve as a useful tool for predicting posthospital recovery and identifying needs for posthospital mobility interventions.

We conducted a study to evaluate the prognostic value of using the Braden Activity subscale (BAS) to identify in-hospital incident mobility impairment and recovery for predicting mortality and discharge status among hospitalized older adults.

The University of Florida Gainesville Health Science Center Institutional Review Board reviewed and approved the study protocol as exempt from human subjects’ research.

Design and Setting

The design followed a retrospective cohort study in which hospitalized patients were evaluated at admission (baseline) and assessed throughout their stay for incident mobility impairment and recovery. Data were collected in older adults (≥65 years old) hospitalized at UF Health Shands Hospital (University of Florida), an 852-bed level I trauma center in Gainesville, Florida.

Data Sources

Patient data from electronic medical records were warehoused in an integrated data repository (IDR) between January 1, 2009 and April 20, 2014. The IDR aggregates clinical and administrative system data, which can subsequently be used for research. The data were compiled in a de-identified longitudinal dataset that included demographics, Charlson Comorbidity Index,¹¹ hospital length of stay, BAS scores (at admission, during hospitalization, at discharge), discharge disposition (including in-hospital death), and mortality after hospitalization (from the national Social Security Death Index).

Patients

The study population consisted of 19,769 older adults (≥65 years old) hospitalized between January 1, 2009 and April 20, 2014.

Outcomes

The major outcomes were patients’ primary discharge disposition and posthospital mortality over 4.5-year follow-up. Discharge dispositions were divided into 9 categories: expired in hospital, other hospital admission, home, home care, hospice, rehabilitation, skilled nursing home, healthcare facility, or other, which included psychiatric facilities, court, or law enforcement.

Predictors

The BAS was used to identify incident mobility impairment and incident mobility recovery during hospitalization and subsequently was used to predict discharge disposition and mortality. The Braden scale,¹² which is commonly administered to predict pressure sores, has 6 subscales: Sensory Perception, Moisture, Activity, Mobility, Nutrition, and Friction and Shear. Each subscale has a score of 1 to 4, with higher scores representing higher activity levels. In particular, the BAS measures the mobility (walking and transferring) level of the hospitalized patient with a score of 1 (“patient is confined to bed”), 2 (“severely limited or nonexistent ability to walk; patient cannot bear his own weight and/or must be assisted into chair or wheelchair”), 3 (“patient walks occasionally during the day, but for very short distances, with or without assistance; he spends majority of each shift in bed or chair”), or 4 (“patient walks outside the room at least twice a day and inside the room at least once every 2 hours during waking hours”). The BAS is correlated with the total Braden scale¹⁰ and has shown excellent interrater reliability (interclass correlation coefficient, 0.96) among hospital staff.¹³ Analysis of the current dataset revealed excellent rater agreement across 3 working shifts (κ = 0.76 for first day of hospitalization in those hospitalized <3 days; κ = 0.70 for first day in those hospitalized ≥3 days).

UF Health Shands Hospital nursing staff administered the BAS at each shift change during a hospital stay (~3 times/d). Mobility scores were averaged across an entire day to reduce potential interrater variation. A daily average BAS score cutpoint was chosen to capture an absorbing mobility state. Average BAS score ≥3 was selected, as it indicates a patient is mobile most of the day, whereas average BAS score <3 indicates significant mobility impairment most of the day. The average daily score was calculated with a minimum of 3 determinations per day. Incident mobility impairment was defined as first transition from “being able to walk occasionally or twice a day outside or at least once every 2 hours during waking hours” to “severely limited or nonexistent ability to walk or confined to bed.” Numerically speaking, daily average BAS score transition from ≥3 at admission to <3 during hospitalization constituted a mobility impairment event. Incident mobility recovery was evaluated in those patient hospital observations that were “severely limited or nonexistent ability to walk or confined to bed” at admission. Incident mobility recovery was defined as first transition to “ability to walk occasionally or twice a day outside or at least once every 2 hours during waking hours.” A mobility recovery event was operationally defined as daily average BAS score transition from <3 at admission to daily average of ≥3 during hospitalization.

Data Analysis

Patient baseline characteristics are reported as counts, means, or medians. Chi-square statistics were used to test group differences for categorical variables, and analysis of variance was performed for continuous variables. Posthospital outcomes were evaluated descriptively and with time-to-event analyses. Kaplan-Meier curves and Wilcoxon P were also used to compare the survival probability for the mobility impairment and recovery groups. Although Cox proportional hazard regression is appropriate for these data, we found the proportionality assumption tenuous. As an alternative, logistic regression was used to model the probability of impairment/recovery outcomes. In addition, a survival time estimate that is robust to the proportionality assumption was derived according to Royston and Parmar^14,15 and Zhao et al.¹⁶ This approach reports the difference between 2 survival curves using the restricted mean—a measure of average survival using the area under the survival curve from time point zero to last observed follow-up time. All models were adjusted for age, sex, race, and hospital length of stay. Analyses were performed with R 3.1.1.¹⁷ All analyses were 2-tailed, and an α of 0.05 was considered statistically significant.

Table 1 lists the baseline characteristics of the hospitalized patients: 10,717 (54%) with normal mobility at admission and 9052 (46%) admitted with impaired mobility. Compared with patients admitted with normal mobility, those with impaired mobility at admission were older, mean (SD) 75.73 (7.84) years versus 73.73 (7.00) years; spent more days in the hospital, median 5 days versus 3 days; and had a higher Charlson Comorbidity Index, mean (SD) 2.59 (2.34) versus 2.22 (2.31). Patients with impaired mobility at admission had a significantly higher prevalence of myocardial infarction, congestive heart failure, peripheral vascular disease, cerebrovascular disease, dementia, and diabetes. However, cancer was significantly more prevalent among patients admitted with normal mobility compared with those admitted with impaired mobility.

Of the 10,717 patients with normal mobility at admission, 2218 (20.7%) had incident mobility impairment over a median follow-up of 3 days (interquartile range, 2-5 days). Of the 9052 patients admitted with impaired mobility, 4734 (52.3%) recovered from their impairment over a median follow-up of 5 days (interquartile range, 3-9 days).

In this study, we evaluated the predictive value of the BAS in assessing incident mobility impairment and recovery during hospitalization among older adults. Patients admitted with impaired mobility were older, spent more days in the hospital, and had more comorbidities than those admitted with normal mobility. Compared with older adults who did not develop incident mobility impairment during hospitalization, those who became mobility impaired had a higher posthospital mortality risk and a higher prevalence of in-hospital death and hospice discharge. In addition, compared with older adults who did not recover mobility in the hospital, those who recovered mobility had a lower posthospital mortality risk and a higher prevalence of home discharge. It is interesting that incident in the hospital appears to have a finite effect. The association was largely erased 6 months after discharge. This was also observed in patients who recovered their mobility in the hospital, but to a lesser extent. Overall, the results suggest that developing mobility impairment or recovering from mobility impairment in the hospital is an important predictor of discharge status and posthospital mortality.

The large number of patient observations and repeated evaluation of in-hospital mobility made this analysis possible. To our knowledge, this is the first large-scale study to evaluate the predictive value of the BAS in assessing mobility impairment and recovery during hospitalization among older adults. Such a test provides a simple and efficient assessment of in-hospital mobility changes that are sensitive to discharge locations and posthospital mortality risk.

Poor mobility in the hospital is associated with higher posthospital mortality. Kasotakis et al.¹⁸ evaluated the predictive value of a nursing staff–assessed clinical mobility score for surgical critically ill patients whose functional mobility was unimpaired on presentation. The Surgical Intensive Care Unit Optimal Mobility Score has been shown to be a reliable and valid tool for predicting mortality in a relatively young population (average age, 60 years). Using accelerometer technology with older adults, Ostir et al.⁷ found that each 100-step increase was associated with 2% and 3% lower risk of death over 2 years in the first and last 24 hours of hospitalization, respectively. The present mortality results show that mobility patterns in the hospital are crucially important for patients’ health the first 6 months after discharge. This finding suggests that developing mobility impairment in the hospital is a sign for significant and rapid health decline. It also suggests that interventions need to be started relatively early in order to reduce the risk of death. In contrast, patients who recover mobility in the hospital obtain a substantial mortality risk reduction. In-hospital interventions to enhance mobility recovery and prevent mobility impairment could have a large impact on posthospital adverse events, particularly for older patients, who are susceptible to disease complications.

Regarding discharge disposition, Sommerfeld and von Arbin¹⁹ found that the ability to rise from a chair (a component of mobility) during hospitalization was a strong predictor of early discharge home. Similarly, Vochteloo et al.²⁰ found that limited mobility as assessed with a questionnaire was associated with discharge to a location other than home among patients with hip fracture. We utilized existing information, collected at a relatively high resolution (3 times per day) that is often readily available without added patient burden. This is particularly important in the hospital setting, where added assessments in frail older adults and in those with multimorbid conditions is challenging. Although our approach is appealing, we should note that BAS scores were modified to reduce interrater variation and capture more absorbing mobility states over a hospitalized day, and that a similar approach would be required to replicate these results and provide clinical value to the BAS as a prognostic indicator of posthospital mortality.

Despite the strengths of this study, it had notable limitations. Pooling BAS scores could have modified the interpretation and clinical implications of the results. Although we had a large number of patient observations, this retrospective analysis may have had biases that were not completely considered. In addition, the results of this single-center study cannot be generalized across all hospital systems. The Braden activity sub score has demonstrated good validity and reliability for activity changes¹³, but this measure was not objectively ascertained as demonstrated by others using accelerometers^6-7. Moreover, the medical records used did not provide prehospital patient mobility status, limiting adjustments for prehospital mobility function. Despite these limitations, this study represents an important initial step in validating a simple and efficient clinical tool for identifying in-hospital mobility impairment and recovery and predicting posthospital adverse outcomes.

BAS assessment of incident mobility impairment and recovery in the hospital setting has prognostic value in predicting discharge disposition, in-hospital death, and posthospital mortality risk. That the majority of the effect appears to occur within the first 6 months after discharge suggests that interventions to improve mobility should be started during hospitalization or expeditiously after discharge. Overall, this study’s results showed that a simple and efficient mobility status assessment can become a valuable clinical and administrative tool for targeting and improving mobility in the hospital and after discharge in older adults.

Acknowledgments

This work was supported by the National Institutes of Health and the National Center for Advancing Translational Sciences (NIH/NCATS) Clinical and Translational Science Award to the University of Florida (UL1 TR000064) and by the University of Florida’s Claude D. Pepper Center (P30AG028740-R6, significant contributions from the Data and Applied Science Core and Biostatistical Core).

Disclosure

Nothing to report.

In-hospital mobility (walking and transferring) is an important modifiable factor for posthospital functional outcomes and mortality among older adults.^1-4 In fact, daily mobility assessment has been considered for a standard clinical evaluation of the hospitalized older adult.^5,6 This would provide a ready source for targeting patients at risk for mobility impairment and identifying strategies to prevent in-hospital mobility limitation and posthospital functional decline. Despite their potential importance, mobility assessment tools have not been readily adopted in the hospital setting.

There are various ways to assess mobility in hospital settings. Mobility tracking technology (radar and accelerometers) has demonstrated older adults have extremely low mobility during hospitalization. Although these objective methods provide an unbiased way to monitor physical activity level and track in-hospital mobility change,^6-8 and have provided important information about mobility in the hospital, they are largely impractical in real-world settings.

While mobility technology appears to be advancing, there is a potential to assess in-hospital mobility using commonly administered and inexpensive tools. Many hospitals ask staff to regularly rate physical function (Braden and Morse score) as part of their standard-of-care procedures. The rating scales used have the potential to provide valuable information about mobility variations without using special equipment or burdening patients. The Braden Scale for Predicting Pressure Sore Risk is a good example of a validated assessment instrument that is better than nurses’ judgment, which is often confounded by nursing experience.⁹ This scale, which has 6 subscales (Sensory Perception, Moisture, Activity, Mobility, Nutrition, Friction and Shear), has shown high sensitivity in detecting patient condition changes in the clinical setting.¹⁰ The scale typically is used holistically to evaluate pressure ulcer risk, but the Activity subscale, which assesses mobility, could serve as a useful tool for predicting posthospital recovery and identifying needs for posthospital mobility interventions.

We conducted a study to evaluate the prognostic value of using the Braden Activity subscale (BAS) to identify in-hospital incident mobility impairment and recovery for predicting mortality and discharge status among hospitalized older adults.

The University of Florida Gainesville Health Science Center Institutional Review Board reviewed and approved the study protocol as exempt from human subjects’ research.

Design and Setting

The design followed a retrospective cohort study in which hospitalized patients were evaluated at admission (baseline) and assessed throughout their stay for incident mobility impairment and recovery. Data were collected in older adults (≥65 years old) hospitalized at UF Health Shands Hospital (University of Florida), an 852-bed level I trauma center in Gainesville, Florida.

Data Sources

Patient data from electronic medical records were warehoused in an integrated data repository (IDR) between January 1, 2009 and April 20, 2014. The IDR aggregates clinical and administrative system data, which can subsequently be used for research. The data were compiled in a de-identified longitudinal dataset that included demographics, Charlson Comorbidity Index,¹¹ hospital length of stay, BAS scores (at admission, during hospitalization, at discharge), discharge disposition (including in-hospital death), and mortality after hospitalization (from the national Social Security Death Index).

Patients

The study population consisted of 19,769 older adults (≥65 years old) hospitalized between January 1, 2009 and April 20, 2014.

Outcomes

The major outcomes were patients’ primary discharge disposition and posthospital mortality over 4.5-year follow-up. Discharge dispositions were divided into 9 categories: expired in hospital, other hospital admission, home, home care, hospice, rehabilitation, skilled nursing home, healthcare facility, or other, which included psychiatric facilities, court, or law enforcement.

Predictors

The BAS was used to identify incident mobility impairment and incident mobility recovery during hospitalization and subsequently was used to predict discharge disposition and mortality. The Braden scale,¹² which is commonly administered to predict pressure sores, has 6 subscales: Sensory Perception, Moisture, Activity, Mobility, Nutrition, and Friction and Shear. Each subscale has a score of 1 to 4, with higher scores representing higher activity levels. In particular, the BAS measures the mobility (walking and transferring) level of the hospitalized patient with a score of 1 (“patient is confined to bed”), 2 (“severely limited or nonexistent ability to walk; patient cannot bear his own weight and/or must be assisted into chair or wheelchair”), 3 (“patient walks occasionally during the day, but for very short distances, with or without assistance; he spends majority of each shift in bed or chair”), or 4 (“patient walks outside the room at least twice a day and inside the room at least once every 2 hours during waking hours”). The BAS is correlated with the total Braden scale¹⁰ and has shown excellent interrater reliability (interclass correlation coefficient, 0.96) among hospital staff.¹³ Analysis of the current dataset revealed excellent rater agreement across 3 working shifts (κ = 0.76 for first day of hospitalization in those hospitalized <3 days; κ = 0.70 for first day in those hospitalized ≥3 days).

UF Health Shands Hospital nursing staff administered the BAS at each shift change during a hospital stay (~3 times/d). Mobility scores were averaged across an entire day to reduce potential interrater variation. A daily average BAS score cutpoint was chosen to capture an absorbing mobility state. Average BAS score ≥3 was selected, as it indicates a patient is mobile most of the day, whereas average BAS score <3 indicates significant mobility impairment most of the day. The average daily score was calculated with a minimum of 3 determinations per day. Incident mobility impairment was defined as first transition from “being able to walk occasionally or twice a day outside or at least once every 2 hours during waking hours” to “severely limited or nonexistent ability to walk or confined to bed.” Numerically speaking, daily average BAS score transition from ≥3 at admission to <3 during hospitalization constituted a mobility impairment event. Incident mobility recovery was evaluated in those patient hospital observations that were “severely limited or nonexistent ability to walk or confined to bed” at admission. Incident mobility recovery was defined as first transition to “ability to walk occasionally or twice a day outside or at least once every 2 hours during waking hours.” A mobility recovery event was operationally defined as daily average BAS score transition from <3 at admission to daily average of ≥3 during hospitalization.

Data Analysis

Patient baseline characteristics are reported as counts, means, or medians. Chi-square statistics were used to test group differences for categorical variables, and analysis of variance was performed for continuous variables. Posthospital outcomes were evaluated descriptively and with time-to-event analyses. Kaplan-Meier curves and Wilcoxon P were also used to compare the survival probability for the mobility impairment and recovery groups. Although Cox proportional hazard regression is appropriate for these data, we found the proportionality assumption tenuous. As an alternative, logistic regression was used to model the probability of impairment/recovery outcomes. In addition, a survival time estimate that is robust to the proportionality assumption was derived according to Royston and Parmar^14,15 and Zhao et al.¹⁶ This approach reports the difference between 2 survival curves using the restricted mean—a measure of average survival using the area under the survival curve from time point zero to last observed follow-up time. All models were adjusted for age, sex, race, and hospital length of stay. Analyses were performed with R 3.1.1.¹⁷ All analyses were 2-tailed, and an α of 0.05 was considered statistically significant.

Table 1 lists the baseline characteristics of the hospitalized patients: 10,717 (54%) with normal mobility at admission and 9052 (46%) admitted with impaired mobility. Compared with patients admitted with normal mobility, those with impaired mobility at admission were older, mean (SD) 75.73 (7.84) years versus 73.73 (7.00) years; spent more days in the hospital, median 5 days versus 3 days; and had a higher Charlson Comorbidity Index, mean (SD) 2.59 (2.34) versus 2.22 (2.31). Patients with impaired mobility at admission had a significantly higher prevalence of myocardial infarction, congestive heart failure, peripheral vascular disease, cerebrovascular disease, dementia, and diabetes. However, cancer was significantly more prevalent among patients admitted with normal mobility compared with those admitted with impaired mobility.

Of the 10,717 patients with normal mobility at admission, 2218 (20.7%) had incident mobility impairment over a median follow-up of 3 days (interquartile range, 2-5 days). Of the 9052 patients admitted with impaired mobility, 4734 (52.3%) recovered from their impairment over a median follow-up of 5 days (interquartile range, 3-9 days).

In this study, we evaluated the predictive value of the BAS in assessing incident mobility impairment and recovery during hospitalization among older adults. Patients admitted with impaired mobility were older, spent more days in the hospital, and had more comorbidities than those admitted with normal mobility. Compared with older adults who did not develop incident mobility impairment during hospitalization, those who became mobility impaired had a higher posthospital mortality risk and a higher prevalence of in-hospital death and hospice discharge. In addition, compared with older adults who did not recover mobility in the hospital, those who recovered mobility had a lower posthospital mortality risk and a higher prevalence of home discharge. It is interesting that incident in the hospital appears to have a finite effect. The association was largely erased 6 months after discharge. This was also observed in patients who recovered their mobility in the hospital, but to a lesser extent. Overall, the results suggest that developing mobility impairment or recovering from mobility impairment in the hospital is an important predictor of discharge status and posthospital mortality.

The large number of patient observations and repeated evaluation of in-hospital mobility made this analysis possible. To our knowledge, this is the first large-scale study to evaluate the predictive value of the BAS in assessing mobility impairment and recovery during hospitalization among older adults. Such a test provides a simple and efficient assessment of in-hospital mobility changes that are sensitive to discharge locations and posthospital mortality risk.

Poor mobility in the hospital is associated with higher posthospital mortality. Kasotakis et al.¹⁸ evaluated the predictive value of a nursing staff–assessed clinical mobility score for surgical critically ill patients whose functional mobility was unimpaired on presentation. The Surgical Intensive Care Unit Optimal Mobility Score has been shown to be a reliable and valid tool for predicting mortality in a relatively young population (average age, 60 years). Using accelerometer technology with older adults, Ostir et al.⁷ found that each 100-step increase was associated with 2% and 3% lower risk of death over 2 years in the first and last 24 hours of hospitalization, respectively. The present mortality results show that mobility patterns in the hospital are crucially important for patients’ health the first 6 months after discharge. This finding suggests that developing mobility impairment in the hospital is a sign for significant and rapid health decline. It also suggests that interventions need to be started relatively early in order to reduce the risk of death. In contrast, patients who recover mobility in the hospital obtain a substantial mortality risk reduction. In-hospital interventions to enhance mobility recovery and prevent mobility impairment could have a large impact on posthospital adverse events, particularly for older patients, who are susceptible to disease complications.

Regarding discharge disposition, Sommerfeld and von Arbin¹⁹ found that the ability to rise from a chair (a component of mobility) during hospitalization was a strong predictor of early discharge home. Similarly, Vochteloo et al.²⁰ found that limited mobility as assessed with a questionnaire was associated with discharge to a location other than home among patients with hip fracture. We utilized existing information, collected at a relatively high resolution (3 times per day) that is often readily available without added patient burden. This is particularly important in the hospital setting, where added assessments in frail older adults and in those with multimorbid conditions is challenging. Although our approach is appealing, we should note that BAS scores were modified to reduce interrater variation and capture more absorbing mobility states over a hospitalized day, and that a similar approach would be required to replicate these results and provide clinical value to the BAS as a prognostic indicator of posthospital mortality.

Despite the strengths of this study, it had notable limitations. Pooling BAS scores could have modified the interpretation and clinical implications of the results. Although we had a large number of patient observations, this retrospective analysis may have had biases that were not completely considered. In addition, the results of this single-center study cannot be generalized across all hospital systems. The Braden activity sub score has demonstrated good validity and reliability for activity changes¹³, but this measure was not objectively ascertained as demonstrated by others using accelerometers^6-7. Moreover, the medical records used did not provide prehospital patient mobility status, limiting adjustments for prehospital mobility function. Despite these limitations, this study represents an important initial step in validating a simple and efficient clinical tool for identifying in-hospital mobility impairment and recovery and predicting posthospital adverse outcomes.

BAS assessment of incident mobility impairment and recovery in the hospital setting has prognostic value in predicting discharge disposition, in-hospital death, and posthospital mortality risk. That the majority of the effect appears to occur within the first 6 months after discharge suggests that interventions to improve mobility should be started during hospitalization or expeditiously after discharge. Overall, this study’s results showed that a simple and efficient mobility status assessment can become a valuable clinical and administrative tool for targeting and improving mobility in the hospital and after discharge in older adults.

Acknowledgments

This work was supported by the National Institutes of Health and the National Center for Advancing Translational Sciences (NIH/NCATS) Clinical and Translational Science Award to the University of Florida (UL1 TR000064) and by the University of Florida’s Claude D. Pepper Center (P30AG028740-R6, significant contributions from the Data and Applied Science Core and Biostatistical Core).

Disclosure

Nothing to report.

In-hospital mobility (walking and transferring) is an important modifiable factor for posthospital functional outcomes and mortality among older adults.^1-4 In fact, daily mobility assessment has been considered for a standard clinical evaluation of the hospitalized older adult.^5,6 This would provide a ready source for targeting patients at risk for mobility impairment and identifying strategies to prevent in-hospital mobility limitation and posthospital functional decline. Despite their potential importance, mobility assessment tools have not been readily adopted in the hospital setting.

There are various ways to assess mobility in hospital settings. Mobility tracking technology (radar and accelerometers) has demonstrated older adults have extremely low mobility during hospitalization. Although these objective methods provide an unbiased way to monitor physical activity level and track in-hospital mobility change,^6-8 and have provided important information about mobility in the hospital, they are largely impractical in real-world settings.

While mobility technology appears to be advancing, there is a potential to assess in-hospital mobility using commonly administered and inexpensive tools. Many hospitals ask staff to regularly rate physical function (Braden and Morse score) as part of their standard-of-care procedures. The rating scales used have the potential to provide valuable information about mobility variations without using special equipment or burdening patients. The Braden Scale for Predicting Pressure Sore Risk is a good example of a validated assessment instrument that is better than nurses’ judgment, which is often confounded by nursing experience.⁹ This scale, which has 6 subscales (Sensory Perception, Moisture, Activity, Mobility, Nutrition, Friction and Shear), has shown high sensitivity in detecting patient condition changes in the clinical setting.¹⁰ The scale typically is used holistically to evaluate pressure ulcer risk, but the Activity subscale, which assesses mobility, could serve as a useful tool for predicting posthospital recovery and identifying needs for posthospital mobility interventions.

We conducted a study to evaluate the prognostic value of using the Braden Activity subscale (BAS) to identify in-hospital incident mobility impairment and recovery for predicting mortality and discharge status among hospitalized older adults.

The University of Florida Gainesville Health Science Center Institutional Review Board reviewed and approved the study protocol as exempt from human subjects’ research.

Design and Setting

The design followed a retrospective cohort study in which hospitalized patients were evaluated at admission (baseline) and assessed throughout their stay for incident mobility impairment and recovery. Data were collected in older adults (≥65 years old) hospitalized at UF Health Shands Hospital (University of Florida), an 852-bed level I trauma center in Gainesville, Florida.

Data Sources

Patient data from electronic medical records were warehoused in an integrated data repository (IDR) between January 1, 2009 and April 20, 2014. The IDR aggregates clinical and administrative system data, which can subsequently be used for research. The data were compiled in a de-identified longitudinal dataset that included demographics, Charlson Comorbidity Index,¹¹ hospital length of stay, BAS scores (at admission, during hospitalization, at discharge), discharge disposition (including in-hospital death), and mortality after hospitalization (from the national Social Security Death Index).

Patients

The study population consisted of 19,769 older adults (≥65 years old) hospitalized between January 1, 2009 and April 20, 2014.

Outcomes

The major outcomes were patients’ primary discharge disposition and posthospital mortality over 4.5-year follow-up. Discharge dispositions were divided into 9 categories: expired in hospital, other hospital admission, home, home care, hospice, rehabilitation, skilled nursing home, healthcare facility, or other, which included psychiatric facilities, court, or law enforcement.

Predictors

The BAS was used to identify incident mobility impairment and incident mobility recovery during hospitalization and subsequently was used to predict discharge disposition and mortality. The Braden scale,¹² which is commonly administered to predict pressure sores, has 6 subscales: Sensory Perception, Moisture, Activity, Mobility, Nutrition, and Friction and Shear. Each subscale has a score of 1 to 4, with higher scores representing higher activity levels. In particular, the BAS measures the mobility (walking and transferring) level of the hospitalized patient with a score of 1 (“patient is confined to bed”), 2 (“severely limited or nonexistent ability to walk; patient cannot bear his own weight and/or must be assisted into chair or wheelchair”), 3 (“patient walks occasionally during the day, but for very short distances, with or without assistance; he spends majority of each shift in bed or chair”), or 4 (“patient walks outside the room at least twice a day and inside the room at least once every 2 hours during waking hours”). The BAS is correlated with the total Braden scale¹⁰ and has shown excellent interrater reliability (interclass correlation coefficient, 0.96) among hospital staff.¹³ Analysis of the current dataset revealed excellent rater agreement across 3 working shifts (κ = 0.76 for first day of hospitalization in those hospitalized <3 days; κ = 0.70 for first day in those hospitalized ≥3 days).

UF Health Shands Hospital nursing staff administered the BAS at each shift change during a hospital stay (~3 times/d). Mobility scores were averaged across an entire day to reduce potential interrater variation. A daily average BAS score cutpoint was chosen to capture an absorbing mobility state. Average BAS score ≥3 was selected, as it indicates a patient is mobile most of the day, whereas average BAS score <3 indicates significant mobility impairment most of the day. The average daily score was calculated with a minimum of 3 determinations per day. Incident mobility impairment was defined as first transition from “being able to walk occasionally or twice a day outside or at least once every 2 hours during waking hours” to “severely limited or nonexistent ability to walk or confined to bed.” Numerically speaking, daily average BAS score transition from ≥3 at admission to <3 during hospitalization constituted a mobility impairment event. Incident mobility recovery was evaluated in those patient hospital observations that were “severely limited or nonexistent ability to walk or confined to bed” at admission. Incident mobility recovery was defined as first transition to “ability to walk occasionally or twice a day outside or at least once every 2 hours during waking hours.” A mobility recovery event was operationally defined as daily average BAS score transition from <3 at admission to daily average of ≥3 during hospitalization.

Data Analysis

Patient baseline characteristics are reported as counts, means, or medians. Chi-square statistics were used to test group differences for categorical variables, and analysis of variance was performed for continuous variables. Posthospital outcomes were evaluated descriptively and with time-to-event analyses. Kaplan-Meier curves and Wilcoxon P were also used to compare the survival probability for the mobility impairment and recovery groups. Although Cox proportional hazard regression is appropriate for these data, we found the proportionality assumption tenuous. As an alternative, logistic regression was used to model the probability of impairment/recovery outcomes. In addition, a survival time estimate that is robust to the proportionality assumption was derived according to Royston and Parmar^14,15 and Zhao et al.¹⁶ This approach reports the difference between 2 survival curves using the restricted mean—a measure of average survival using the area under the survival curve from time point zero to last observed follow-up time. All models were adjusted for age, sex, race, and hospital length of stay. Analyses were performed with R 3.1.1.¹⁷ All analyses were 2-tailed, and an α of 0.05 was considered statistically significant.

Table 1 lists the baseline characteristics of the hospitalized patients: 10,717 (54%) with normal mobility at admission and 9052 (46%) admitted with impaired mobility. Compared with patients admitted with normal mobility, those with impaired mobility at admission were older, mean (SD) 75.73 (7.84) years versus 73.73 (7.00) years; spent more days in the hospital, median 5 days versus 3 days; and had a higher Charlson Comorbidity Index, mean (SD) 2.59 (2.34) versus 2.22 (2.31). Patients with impaired mobility at admission had a significantly higher prevalence of myocardial infarction, congestive heart failure, peripheral vascular disease, cerebrovascular disease, dementia, and diabetes. However, cancer was significantly more prevalent among patients admitted with normal mobility compared with those admitted with impaired mobility.

Of the 10,717 patients with normal mobility at admission, 2218 (20.7%) had incident mobility impairment over a median follow-up of 3 days (interquartile range, 2-5 days). Of the 9052 patients admitted with impaired mobility, 4734 (52.3%) recovered from their impairment over a median follow-up of 5 days (interquartile range, 3-9 days).

In this study, we evaluated the predictive value of the BAS in assessing incident mobility impairment and recovery during hospitalization among older adults. Patients admitted with impaired mobility were older, spent more days in the hospital, and had more comorbidities than those admitted with normal mobility. Compared with older adults who did not develop incident mobility impairment during hospitalization, those who became mobility impaired had a higher posthospital mortality risk and a higher prevalence of in-hospital death and hospice discharge. In addition, compared with older adults who did not recover mobility in the hospital, those who recovered mobility had a lower posthospital mortality risk and a higher prevalence of home discharge. It is interesting that incident in the hospital appears to have a finite effect. The association was largely erased 6 months after discharge. This was also observed in patients who recovered their mobility in the hospital, but to a lesser extent. Overall, the results suggest that developing mobility impairment or recovering from mobility impairment in the hospital is an important predictor of discharge status and posthospital mortality.

The large number of patient observations and repeated evaluation of in-hospital mobility made this analysis possible. To our knowledge, this is the first large-scale study to evaluate the predictive value of the BAS in assessing mobility impairment and recovery during hospitalization among older adults. Such a test provides a simple and efficient assessment of in-hospital mobility changes that are sensitive to discharge locations and posthospital mortality risk.

Poor mobility in the hospital is associated with higher posthospital mortality. Kasotakis et al.¹⁸ evaluated the predictive value of a nursing staff–assessed clinical mobility score for surgical critically ill patients whose functional mobility was unimpaired on presentation. The Surgical Intensive Care Unit Optimal Mobility Score has been shown to be a reliable and valid tool for predicting mortality in a relatively young population (average age, 60 years). Using accelerometer technology with older adults, Ostir et al.⁷ found that each 100-step increase was associated with 2% and 3% lower risk of death over 2 years in the first and last 24 hours of hospitalization, respectively. The present mortality results show that mobility patterns in the hospital are crucially important for patients’ health the first 6 months after discharge. This finding suggests that developing mobility impairment in the hospital is a sign for significant and rapid health decline. It also suggests that interventions need to be started relatively early in order to reduce the risk of death. In contrast, patients who recover mobility in the hospital obtain a substantial mortality risk reduction. In-hospital interventions to enhance mobility recovery and prevent mobility impairment could have a large impact on posthospital adverse events, particularly for older patients, who are susceptible to disease complications.

Regarding discharge disposition, Sommerfeld and von Arbin¹⁹ found that the ability to rise from a chair (a component of mobility) during hospitalization was a strong predictor of early discharge home. Similarly, Vochteloo et al.²⁰ found that limited mobility as assessed with a questionnaire was associated with discharge to a location other than home among patients with hip fracture. We utilized existing information, collected at a relatively high resolution (3 times per day) that is often readily available without added patient burden. This is particularly important in the hospital setting, where added assessments in frail older adults and in those with multimorbid conditions is challenging. Although our approach is appealing, we should note that BAS scores were modified to reduce interrater variation and capture more absorbing mobility states over a hospitalized day, and that a similar approach would be required to replicate these results and provide clinical value to the BAS as a prognostic indicator of posthospital mortality.

Despite the strengths of this study, it had notable limitations. Pooling BAS scores could have modified the interpretation and clinical implications of the results. Although we had a large number of patient observations, this retrospective analysis may have had biases that were not completely considered. In addition, the results of this single-center study cannot be generalized across all hospital systems. The Braden activity sub score has demonstrated good validity and reliability for activity changes¹³, but this measure was not objectively ascertained as demonstrated by others using accelerometers^6-7. Moreover, the medical records used did not provide prehospital patient mobility status, limiting adjustments for prehospital mobility function. Despite these limitations, this study represents an important initial step in validating a simple and efficient clinical tool for identifying in-hospital mobility impairment and recovery and predicting posthospital adverse outcomes.

BAS assessment of incident mobility impairment and recovery in the hospital setting has prognostic value in predicting discharge disposition, in-hospital death, and posthospital mortality risk. That the majority of the effect appears to occur within the first 6 months after discharge suggests that interventions to improve mobility should be started during hospitalization or expeditiously after discharge. Overall, this study’s results showed that a simple and efficient mobility status assessment can become a valuable clinical and administrative tool for targeting and improving mobility in the hospital and after discharge in older adults.

Acknowledgments

This work was supported by the National Institutes of Health and the National Center for Advancing Translational Sciences (NIH/NCATS) Clinical and Translational Science Award to the University of Florida (UL1 TR000064) and by the University of Florida’s Claude D. Pepper Center (P30AG028740-R6, significant contributions from the Data and Applied Science Core and Biostatistical Core).

Disclosure

Nothing to report.