Using methodology created by the Centers for Medicare & Medicaid Services (CMS), the Department of Veterans Affairs (VA) calculates and reports hospital performance measures for several key conditions, including acute myocardial infarction (AMI), heart failure (HF), and pneumonia.¹ These measures are designed to benchmark individual hospitals against how average hospitals perform when caring for a similar case-mix index. Because readmissions to the hospital within 30-days of discharge are common and costly, this metric has garnered extensive attention in recent years.

To summarize the 30-day readmission metric, the VA utilizes the Strategic Analytics for Improvement and Learning (SAIL) system to present internally its findings to VA practitioners and leadership.² The VA provides these data as a means to drive quality improvement and allow for comparison of individual hospitals’ performance across measures throughout the VA healthcare system. Since 2010, the VA began using and publicly reporting the CMS-derived 30-day Risk-Stratified Readmission Rate (RSRR) on the Hospital Compare website.³ Similar to CMS, the VA uses three years of combined data so that patients, providers, and other stakeholders can compare individual hospitals’ performance across these measures.¹ In response to this, hospitals and healthcare organizations have implemented quality improvement and large-scale programmatic interventions in an attempt to improve quality around readmissions.^4-6 A recent assessment on how hospitals within the Medicare fee-for-service program have responded to such reporting found large degrees of variability, with more than half of the participating institutions facing penalties due to greater-than-expected readmission rates.⁵ Although the VA utilizes the same CMS-derived model in its assessments and reporting, the variability and distribution around this metric are not publicly reported—thus making it difficult to ascertain how individual VA hospitals compare with one another. Without such information, individual facilities may not know how to benchmark the quality of their care to others, nor would the VA recognize which interventions addressing readmissions are working, and which are not. Although previous assessments of interinstitutional variance have been performed in Medicare populations,⁷ a focused analysis of such variance within the VA has yet to be performed.

In this study, we performed a multiyear assessment of the CMS-derived 30-day RSRR metric for AMI, HF, and pneumonia as a useful measure to drive VA quality improvement or distinguish VA facility performance based on its ability to detect interfacility variability.

Data Source

We used VA administrative and Medicare claims data from 2010 to 2012. After identifying index hospitalizations to VA hospitals, we obtained patients’ respective inpatient Medicare claims data from the Medicare Provider Analysis and Review (MedPAR) and Outpatient files. All Medicare records were linked to VA records via scrambled Social Security numbers and were provided by the VA Information Resource Center. This study was approved by the San Francisco VA Medical Center Institutional Review Board.

Study Sample

Our cohort consisted of hospitalized VA beneficiary and Medicare fee-for-service patients who were aged ≥65 years and admitted to and discharged from a VA acute care center with a primary discharge diagnosis of AMI, HF, or pneumonia. These comorbidities were chosen as they are publicly reported and frequently used for interfacility comparisons. Because studies have found that inclusion of secondary payer data (ie, CMS data) may affect hospital-profiling outcomes, we included Medicare data on all available patients.⁸ We excluded hospitalizations that resulted in a transfer to another acute care facility and those admitted to observation status at their index admission. To ensure a full year of data for risk adjustment, beneficiaries were included only if they were enrolled in Medicare for 12 months prior to and including the date of the index admission.

Index hospitalizations were first identified using VA-only inpatient data similar to methods outlined by the CMS and endorsed by the National Quality Forum for Hospital Profiling.⁹ An index hospitalization was defined as an acute inpatient discharge between 2010 and 2012 in which the principal diagnosis was AMI, HF, or pneumonia. We excluded in-hospital deaths, discharges against medical advice, and--for the AMI cohort only--discharges on the same day as admission. Patients may have multiple admissions per year, but only admissions after 30 days of discharge from an index admission were eligible to be included as an additional index admission.

Outcomes

A readmission was defined as any unplanned rehospitalization to either non-VA or VA acute care facilities for any cause within 30 days of discharge from the index hospitalization. Readmissions to observation status or nonacute or rehabilitation units, such as skilled nursing facilities, were not included. Planned readmissions for elective procedures, such as elective chemotherapy and revascularization following an AMI index admission, were not considered as an outcome event.

Risk Standardization for 30-day Readmission

Using approaches developed by CMS,^10-12 we calculated hospital-specific 30-day RSRRs for each VA. Briefly, the RSRR is a ratio of the number of predicted readmissions within 30 days of discharge to the expected number of readmissions within 30 days of hospital discharge, multiplied by the national unadjusted 30-day readmission rate. This measure calculates hospital-specific RSRRs using hierarchical logistic regression models, which account for clustering of patients within hospitals and risk-adjusting for differences in case-mix, during the assessed time periods.¹³ This approach simultaneously models two levels (patient and hospital) to account for the variance in patient outcomes within and between hospitals.¹⁴ At the patient level, the model uses the log odds of readmissions as the dependent variable and age and selected comorbidities as the independent variables. The second level models the hospital-specific intercepts. According to CMS guidelines, the analysis was limited to facilities with at least 25 patient admissions annually for each condition. All readmissions were attributed to the hospital that initially discharged the patient to a nonacute setting.

Analysis

We examined and reported the distribution of patient and clinical characteristics at the hospital level. For each condition, we determined the number of hospitals that had a sufficient number of admissions (n ≥ 25) to be included in the analyses. We calculated the mean, median, and interquartile range for the observed unadjusted readmission rates across all included hospitals.

Similar to methods used by CMS, we used one year of data in the VA to assess hospital quality and variation in facility performance. First, we calculated the 30-day RSRRs using one year (2012) of data. To assess how variability changed with higher facility volume (ie, more years included in the analysis), we also calculated the 30-day RSRRs using two and three years of data. For this, we identified and quantified the number of hospitals whose RSRRs were calculated as being above or below the national VA average (mean ± 95% CI). Specifically, we calculated the number and percentage of hospitals that were classified as either above (+95% CI) or below the national average (−95% CI) using data from all three time periods. All analyses were conducted using SAS Enterprise Guide, Version 7.1. The SAS statistical packages made available by the CMS Measure Team were used to calculate RSRRs.

Patient Characteristics

Patients were predominantly older males (98.3%). Among those hospitalized for AMI, most of them had a history of previous coronary artery bypass graft (CABG) (69.1%), acute coronary syndrome (ACS; 66.2%), or documented coronary atherosclerosis (89.8%). Similarly, patients admitted for HF had high rates of CABG (71.3%) and HF (94.6%), in addition to cardiac arrhythmias (69.3%) and diabetes (60.8%). Patients admitted with a diagnosis of pneumonia had high rates of CABG (61.9%), chronic obstructive pulmonary disease (COPD; 58.1%), and previous diagnosis of pneumonia (78.8%; Table 1). Patient characteristics for two and three years of data are presented in Supplementary Table 1.

VA Hospitals with Sufficient Volume to Be Included in Profiling Assessments

There were 146 acute-care hospitals in the VA. In 2012, 56 (38%) VA hospitals had at least 25 admissions for AMI, 102 (70%) hospitals had at least 25 admissions for CHF, and 106 (73%) hospitals had at least 25 admissions for pneumonia (Table 1) and therefore qualified for analysis based on CMS criteria for 30-day RSRR calculation. The study sample included 3,571 patients with AMI, 10,609 patients with CHF, and 10,191 patients with pneumonia.

30-Day Readmission Rates

The mean observed readmission rates in 2012 were 20% (95% CI 19%-21%) among patients admitted for AMI, 20% (95% CI 19%-20%) for patients admitted with CHF, and 15% (95% CI 15%-16%) for patients admitted with pneumonia. No significant variation from these rates was noted following risk standardization across hospitals (Table 2). Observed and risk-standardized rates were also calculated for two and three years of data (Supplementary Table 2) but were not found to be grossly different when utilizing a single year of data.

In 2012, two hospitals (2%) exhibited HF RSRRs worse than the national average (+95% CI), whereas no hospital demonstrated worse-than-average rates (+95% CI) for AMI or pneumonia (Table 3, Figure 1). Similarly, in 2012, only three hospitals had RSRRs better than the national average (−95% CI) for HF and pneumonia.

We found that the CMS-derived 30-day risk-stratified readmission metric for AMI, HF, and pneumonia showed little variation among VA hospitals. The lack of institutional 30-day readmission volume appears to be a fundamental limitation that subsequently requires multiple years of data to make this metric clinically meaningful. As the largest integrated healthcare system in the United States, the VA relies upon and makes large-scale programmatic decisions based on such performance data. The inability to detect meaningful interhospital variation in a timely manner suggests that the CMS-derived 30-day RSRR may not be a sensitive metric to distinguish facility performance or drive quality improvement initiatives within the VA.

First, we found it notable that among the 146 VA medical centers available for analysis,¹⁵ between 38% and 77% of hospitals qualified for evaluation when using CMS-based participation criteria—which excludes institutions with fewer than 25 episodes per year. Although this low degree of qualification for profiling was most dramatic when using one year of data (range: 38%-72%), we noted that it did not dramatically improve when we combined three years of data (range: 52%-77%). These findings act to highlight the population and systems differences between CMS and VA populations¹⁶ and further support the idea that CMS-derived models may not be optimized for use in the VA healthcare system.

Our findings are particularly relevant within the VA given the quarterly rate with which these data are reported within the VA SAIL scorecard.² The VA designed SAIL for internal benchmarking to spotlight successful strategies of top performing institutions and promote high-quality, value-based care. Using one year of data, the minimum required to utilize CMS models, showed that quarterly feedback (ie, three months of data) may not be informative or useful given that few hospitals are able to differentiate themselves from the mean (±95% CI). Although the capacity to distinguish between high and low performers does improve by combining hospital admissions over three years, this is not a reasonable timeline for institutions to wait for quality comparisons. Furthermore, although the VA does present its data on CMS’s Hospital Compare website using three years of combined data, the variability and distribution of such results are not supplied.³

This lack of discriminability raises concerns about the ability to compare hospital performance between low- and high-volume institutions. Although these models function well in CMS settings with large patient volumes in which greater variability exists,⁵ they lose their capacity to discriminate when applied to low-volume settings such as the VA. Given that several hospitals in the US are small community hospitals with low patient volumes,¹⁷ this issue probably occurs in other non-VA settings. Although our study focuses on the VA, others have been able to compare VA and non-VA settings’ variation and distribution. For example, Nuti et al. explored the differences in 30-day RSRRs among hospitalized patients with AMI, HF, and pneumonia and similarly showed little variation, narrow distributions, and few outliers in the VA setting compared to those in the non-VA setting. For small patient volume institutions, including the VA, a focus on high-volume services, outcomes, and measures (ie, blood pressure control, medication reconciliation, etc.) may offer more discriminability between high- and low-performing facilities. For example, Patel et al. found that VA process measures in patients with HF (ie, beta-blocker and ACE-inhibitor use) can be used as valid quality measures as they exhibited consistent reliability over time and validity with adjusted mortality rates, whereas the 30-day RSRR did not.¹⁸

Our findings may have substantial financial, resource, and policy implications. Automatically developing and reporting measures created for the Medicare program in the VA may not be a good use of VA resources. In addition, facilities may react to these reported outcomes and expend local resources and finances to implement interventions to improve on a performance outcome whose measure is statistically no different than the vast majority of its comparators. Such events have been highlighted in the public media and have pointed to the fact that small changes in quality, or statistical errors themselves, can have large ramifications within the VA’s hospital rating system.¹⁹

These findings may also add to the discussion on whether public reporting of health and quality outcomes improves patient care. Since the CMS began public reporting on RSRRs in 2009, these rates have fallen for all three examined conditions (AMI, HF, and pneumonia),^7,20,21 in addition to several other health outcomes.¹⁷ Although recent studies have suggested that these decreased rates have been driven by the CMS-sponsored Hospital Readmissions Reduction Program (HRRP),²² others have suggested that these findings are consistent with ongoing secular trends toward decreased readmissions and may not be completely explained by public reporting alone.²³ Moreover, prior work has also found that readmissions may be strongly impacted by factors external to the hospital setting, such as patients’ social demographics (ie, household income, social isolation), that are not currently captured in risk-prediction models.²⁴ Given the small variability we see in our data, public reporting within the VA is probably not beneficial, as only a small number of facilities are outliers based on RSRR.

Our study has several limitations. First, although we adapted the CMS model to the VA, we did not include gender in the model because >99% of all patient admissions were male. Second, we assessed only three medical conditions that were being tracked by both CMS and VA during this time period, and these outcomes may not be representative of other aspects of care and cannot be generalized to other medical conditions. Finally, more contemporary data could lead to differing results – though we note that no large-scale structural or policy changes addressing readmission rates have been implemented within the VA since our study period.

The results of this study suggest that the CMS-derived 30-day risk-stratified readmission metric for AMI, HF, and pneumonia may not have the capacity to properly detect interfacility variance and thus may not be an optimal quality indicator within the VA. As the VA and other healthcare systems continually strive to improve the quality of care they provide, they will require more accurate and timely metrics for which to index their performance.

Disclosures

The authors have nothing to disclose

Using methodology created by the Centers for Medicare & Medicaid Services (CMS), the Department of Veterans Affairs (VA) calculates and reports hospital performance measures for several key conditions, including acute myocardial infarction (AMI), heart failure (HF), and pneumonia.¹ These measures are designed to benchmark individual hospitals against how average hospitals perform when caring for a similar case-mix index. Because readmissions to the hospital within 30-days of discharge are common and costly, this metric has garnered extensive attention in recent years.

To summarize the 30-day readmission metric, the VA utilizes the Strategic Analytics for Improvement and Learning (SAIL) system to present internally its findings to VA practitioners and leadership.² The VA provides these data as a means to drive quality improvement and allow for comparison of individual hospitals’ performance across measures throughout the VA healthcare system. Since 2010, the VA began using and publicly reporting the CMS-derived 30-day Risk-Stratified Readmission Rate (RSRR) on the Hospital Compare website.³ Similar to CMS, the VA uses three years of combined data so that patients, providers, and other stakeholders can compare individual hospitals’ performance across these measures.¹ In response to this, hospitals and healthcare organizations have implemented quality improvement and large-scale programmatic interventions in an attempt to improve quality around readmissions.^4-6 A recent assessment on how hospitals within the Medicare fee-for-service program have responded to such reporting found large degrees of variability, with more than half of the participating institutions facing penalties due to greater-than-expected readmission rates.⁵ Although the VA utilizes the same CMS-derived model in its assessments and reporting, the variability and distribution around this metric are not publicly reported—thus making it difficult to ascertain how individual VA hospitals compare with one another. Without such information, individual facilities may not know how to benchmark the quality of their care to others, nor would the VA recognize which interventions addressing readmissions are working, and which are not. Although previous assessments of interinstitutional variance have been performed in Medicare populations,⁷ a focused analysis of such variance within the VA has yet to be performed.

In this study, we performed a multiyear assessment of the CMS-derived 30-day RSRR metric for AMI, HF, and pneumonia as a useful measure to drive VA quality improvement or distinguish VA facility performance based on its ability to detect interfacility variability.

Data Source

We used VA administrative and Medicare claims data from 2010 to 2012. After identifying index hospitalizations to VA hospitals, we obtained patients’ respective inpatient Medicare claims data from the Medicare Provider Analysis and Review (MedPAR) and Outpatient files. All Medicare records were linked to VA records via scrambled Social Security numbers and were provided by the VA Information Resource Center. This study was approved by the San Francisco VA Medical Center Institutional Review Board.

Study Sample

Our cohort consisted of hospitalized VA beneficiary and Medicare fee-for-service patients who were aged ≥65 years and admitted to and discharged from a VA acute care center with a primary discharge diagnosis of AMI, HF, or pneumonia. These comorbidities were chosen as they are publicly reported and frequently used for interfacility comparisons. Because studies have found that inclusion of secondary payer data (ie, CMS data) may affect hospital-profiling outcomes, we included Medicare data on all available patients.⁸ We excluded hospitalizations that resulted in a transfer to another acute care facility and those admitted to observation status at their index admission. To ensure a full year of data for risk adjustment, beneficiaries were included only if they were enrolled in Medicare for 12 months prior to and including the date of the index admission.

Index hospitalizations were first identified using VA-only inpatient data similar to methods outlined by the CMS and endorsed by the National Quality Forum for Hospital Profiling.⁹ An index hospitalization was defined as an acute inpatient discharge between 2010 and 2012 in which the principal diagnosis was AMI, HF, or pneumonia. We excluded in-hospital deaths, discharges against medical advice, and--for the AMI cohort only--discharges on the same day as admission. Patients may have multiple admissions per year, but only admissions after 30 days of discharge from an index admission were eligible to be included as an additional index admission.

Outcomes

A readmission was defined as any unplanned rehospitalization to either non-VA or VA acute care facilities for any cause within 30 days of discharge from the index hospitalization. Readmissions to observation status or nonacute or rehabilitation units, such as skilled nursing facilities, were not included. Planned readmissions for elective procedures, such as elective chemotherapy and revascularization following an AMI index admission, were not considered as an outcome event.

Risk Standardization for 30-day Readmission

Using approaches developed by CMS,^10-12 we calculated hospital-specific 30-day RSRRs for each VA. Briefly, the RSRR is a ratio of the number of predicted readmissions within 30 days of discharge to the expected number of readmissions within 30 days of hospital discharge, multiplied by the national unadjusted 30-day readmission rate. This measure calculates hospital-specific RSRRs using hierarchical logistic regression models, which account for clustering of patients within hospitals and risk-adjusting for differences in case-mix, during the assessed time periods.¹³ This approach simultaneously models two levels (patient and hospital) to account for the variance in patient outcomes within and between hospitals.¹⁴ At the patient level, the model uses the log odds of readmissions as the dependent variable and age and selected comorbidities as the independent variables. The second level models the hospital-specific intercepts. According to CMS guidelines, the analysis was limited to facilities with at least 25 patient admissions annually for each condition. All readmissions were attributed to the hospital that initially discharged the patient to a nonacute setting.

Analysis

We examined and reported the distribution of patient and clinical characteristics at the hospital level. For each condition, we determined the number of hospitals that had a sufficient number of admissions (n ≥ 25) to be included in the analyses. We calculated the mean, median, and interquartile range for the observed unadjusted readmission rates across all included hospitals.

Similar to methods used by CMS, we used one year of data in the VA to assess hospital quality and variation in facility performance. First, we calculated the 30-day RSRRs using one year (2012) of data. To assess how variability changed with higher facility volume (ie, more years included in the analysis), we also calculated the 30-day RSRRs using two and three years of data. For this, we identified and quantified the number of hospitals whose RSRRs were calculated as being above or below the national VA average (mean ± 95% CI). Specifically, we calculated the number and percentage of hospitals that were classified as either above (+95% CI) or below the national average (−95% CI) using data from all three time periods. All analyses were conducted using SAS Enterprise Guide, Version 7.1. The SAS statistical packages made available by the CMS Measure Team were used to calculate RSRRs.

Patient Characteristics

Patients were predominantly older males (98.3%). Among those hospitalized for AMI, most of them had a history of previous coronary artery bypass graft (CABG) (69.1%), acute coronary syndrome (ACS; 66.2%), or documented coronary atherosclerosis (89.8%). Similarly, patients admitted for HF had high rates of CABG (71.3%) and HF (94.6%), in addition to cardiac arrhythmias (69.3%) and diabetes (60.8%). Patients admitted with a diagnosis of pneumonia had high rates of CABG (61.9%), chronic obstructive pulmonary disease (COPD; 58.1%), and previous diagnosis of pneumonia (78.8%; Table 1). Patient characteristics for two and three years of data are presented in Supplementary Table 1.

VA Hospitals with Sufficient Volume to Be Included in Profiling Assessments

There were 146 acute-care hospitals in the VA. In 2012, 56 (38%) VA hospitals had at least 25 admissions for AMI, 102 (70%) hospitals had at least 25 admissions for CHF, and 106 (73%) hospitals had at least 25 admissions for pneumonia (Table 1) and therefore qualified for analysis based on CMS criteria for 30-day RSRR calculation. The study sample included 3,571 patients with AMI, 10,609 patients with CHF, and 10,191 patients with pneumonia.

30-Day Readmission Rates

The mean observed readmission rates in 2012 were 20% (95% CI 19%-21%) among patients admitted for AMI, 20% (95% CI 19%-20%) for patients admitted with CHF, and 15% (95% CI 15%-16%) for patients admitted with pneumonia. No significant variation from these rates was noted following risk standardization across hospitals (Table 2). Observed and risk-standardized rates were also calculated for two and three years of data (Supplementary Table 2) but were not found to be grossly different when utilizing a single year of data.

In 2012, two hospitals (2%) exhibited HF RSRRs worse than the national average (+95% CI), whereas no hospital demonstrated worse-than-average rates (+95% CI) for AMI or pneumonia (Table 3, Figure 1). Similarly, in 2012, only three hospitals had RSRRs better than the national average (−95% CI) for HF and pneumonia.

We found that the CMS-derived 30-day risk-stratified readmission metric for AMI, HF, and pneumonia showed little variation among VA hospitals. The lack of institutional 30-day readmission volume appears to be a fundamental limitation that subsequently requires multiple years of data to make this metric clinically meaningful. As the largest integrated healthcare system in the United States, the VA relies upon and makes large-scale programmatic decisions based on such performance data. The inability to detect meaningful interhospital variation in a timely manner suggests that the CMS-derived 30-day RSRR may not be a sensitive metric to distinguish facility performance or drive quality improvement initiatives within the VA.

First, we found it notable that among the 146 VA medical centers available for analysis,¹⁵ between 38% and 77% of hospitals qualified for evaluation when using CMS-based participation criteria—which excludes institutions with fewer than 25 episodes per year. Although this low degree of qualification for profiling was most dramatic when using one year of data (range: 38%-72%), we noted that it did not dramatically improve when we combined three years of data (range: 52%-77%). These findings act to highlight the population and systems differences between CMS and VA populations¹⁶ and further support the idea that CMS-derived models may not be optimized for use in the VA healthcare system.

Our findings are particularly relevant within the VA given the quarterly rate with which these data are reported within the VA SAIL scorecard.² The VA designed SAIL for internal benchmarking to spotlight successful strategies of top performing institutions and promote high-quality, value-based care. Using one year of data, the minimum required to utilize CMS models, showed that quarterly feedback (ie, three months of data) may not be informative or useful given that few hospitals are able to differentiate themselves from the mean (±95% CI). Although the capacity to distinguish between high and low performers does improve by combining hospital admissions over three years, this is not a reasonable timeline for institutions to wait for quality comparisons. Furthermore, although the VA does present its data on CMS’s Hospital Compare website using three years of combined data, the variability and distribution of such results are not supplied.³

This lack of discriminability raises concerns about the ability to compare hospital performance between low- and high-volume institutions. Although these models function well in CMS settings with large patient volumes in which greater variability exists,⁵ they lose their capacity to discriminate when applied to low-volume settings such as the VA. Given that several hospitals in the US are small community hospitals with low patient volumes,¹⁷ this issue probably occurs in other non-VA settings. Although our study focuses on the VA, others have been able to compare VA and non-VA settings’ variation and distribution. For example, Nuti et al. explored the differences in 30-day RSRRs among hospitalized patients with AMI, HF, and pneumonia and similarly showed little variation, narrow distributions, and few outliers in the VA setting compared to those in the non-VA setting. For small patient volume institutions, including the VA, a focus on high-volume services, outcomes, and measures (ie, blood pressure control, medication reconciliation, etc.) may offer more discriminability between high- and low-performing facilities. For example, Patel et al. found that VA process measures in patients with HF (ie, beta-blocker and ACE-inhibitor use) can be used as valid quality measures as they exhibited consistent reliability over time and validity with adjusted mortality rates, whereas the 30-day RSRR did not.¹⁸

Our findings may have substantial financial, resource, and policy implications. Automatically developing and reporting measures created for the Medicare program in the VA may not be a good use of VA resources. In addition, facilities may react to these reported outcomes and expend local resources and finances to implement interventions to improve on a performance outcome whose measure is statistically no different than the vast majority of its comparators. Such events have been highlighted in the public media and have pointed to the fact that small changes in quality, or statistical errors themselves, can have large ramifications within the VA’s hospital rating system.¹⁹

These findings may also add to the discussion on whether public reporting of health and quality outcomes improves patient care. Since the CMS began public reporting on RSRRs in 2009, these rates have fallen for all three examined conditions (AMI, HF, and pneumonia),^7,20,21 in addition to several other health outcomes.¹⁷ Although recent studies have suggested that these decreased rates have been driven by the CMS-sponsored Hospital Readmissions Reduction Program (HRRP),²² others have suggested that these findings are consistent with ongoing secular trends toward decreased readmissions and may not be completely explained by public reporting alone.²³ Moreover, prior work has also found that readmissions may be strongly impacted by factors external to the hospital setting, such as patients’ social demographics (ie, household income, social isolation), that are not currently captured in risk-prediction models.²⁴ Given the small variability we see in our data, public reporting within the VA is probably not beneficial, as only a small number of facilities are outliers based on RSRR.

Our study has several limitations. First, although we adapted the CMS model to the VA, we did not include gender in the model because >99% of all patient admissions were male. Second, we assessed only three medical conditions that were being tracked by both CMS and VA during this time period, and these outcomes may not be representative of other aspects of care and cannot be generalized to other medical conditions. Finally, more contemporary data could lead to differing results – though we note that no large-scale structural or policy changes addressing readmission rates have been implemented within the VA since our study period.

The results of this study suggest that the CMS-derived 30-day risk-stratified readmission metric for AMI, HF, and pneumonia may not have the capacity to properly detect interfacility variance and thus may not be an optimal quality indicator within the VA. As the VA and other healthcare systems continually strive to improve the quality of care they provide, they will require more accurate and timely metrics for which to index their performance.

Disclosures

The authors have nothing to disclose

Using methodology created by the Centers for Medicare & Medicaid Services (CMS), the Department of Veterans Affairs (VA) calculates and reports hospital performance measures for several key conditions, including acute myocardial infarction (AMI), heart failure (HF), and pneumonia.¹ These measures are designed to benchmark individual hospitals against how average hospitals perform when caring for a similar case-mix index. Because readmissions to the hospital within 30-days of discharge are common and costly, this metric has garnered extensive attention in recent years.

To summarize the 30-day readmission metric, the VA utilizes the Strategic Analytics for Improvement and Learning (SAIL) system to present internally its findings to VA practitioners and leadership.² The VA provides these data as a means to drive quality improvement and allow for comparison of individual hospitals’ performance across measures throughout the VA healthcare system. Since 2010, the VA began using and publicly reporting the CMS-derived 30-day Risk-Stratified Readmission Rate (RSRR) on the Hospital Compare website.³ Similar to CMS, the VA uses three years of combined data so that patients, providers, and other stakeholders can compare individual hospitals’ performance across these measures.¹ In response to this, hospitals and healthcare organizations have implemented quality improvement and large-scale programmatic interventions in an attempt to improve quality around readmissions.^4-6 A recent assessment on how hospitals within the Medicare fee-for-service program have responded to such reporting found large degrees of variability, with more than half of the participating institutions facing penalties due to greater-than-expected readmission rates.⁵ Although the VA utilizes the same CMS-derived model in its assessments and reporting, the variability and distribution around this metric are not publicly reported—thus making it difficult to ascertain how individual VA hospitals compare with one another. Without such information, individual facilities may not know how to benchmark the quality of their care to others, nor would the VA recognize which interventions addressing readmissions are working, and which are not. Although previous assessments of interinstitutional variance have been performed in Medicare populations,⁷ a focused analysis of such variance within the VA has yet to be performed.

In this study, we performed a multiyear assessment of the CMS-derived 30-day RSRR metric for AMI, HF, and pneumonia as a useful measure to drive VA quality improvement or distinguish VA facility performance based on its ability to detect interfacility variability.

Data Source

We used VA administrative and Medicare claims data from 2010 to 2012. After identifying index hospitalizations to VA hospitals, we obtained patients’ respective inpatient Medicare claims data from the Medicare Provider Analysis and Review (MedPAR) and Outpatient files. All Medicare records were linked to VA records via scrambled Social Security numbers and were provided by the VA Information Resource Center. This study was approved by the San Francisco VA Medical Center Institutional Review Board.

Study Sample

Our cohort consisted of hospitalized VA beneficiary and Medicare fee-for-service patients who were aged ≥65 years and admitted to and discharged from a VA acute care center with a primary discharge diagnosis of AMI, HF, or pneumonia. These comorbidities were chosen as they are publicly reported and frequently used for interfacility comparisons. Because studies have found that inclusion of secondary payer data (ie, CMS data) may affect hospital-profiling outcomes, we included Medicare data on all available patients.⁸ We excluded hospitalizations that resulted in a transfer to another acute care facility and those admitted to observation status at their index admission. To ensure a full year of data for risk adjustment, beneficiaries were included only if they were enrolled in Medicare for 12 months prior to and including the date of the index admission.

Index hospitalizations were first identified using VA-only inpatient data similar to methods outlined by the CMS and endorsed by the National Quality Forum for Hospital Profiling.⁹ An index hospitalization was defined as an acute inpatient discharge between 2010 and 2012 in which the principal diagnosis was AMI, HF, or pneumonia. We excluded in-hospital deaths, discharges against medical advice, and--for the AMI cohort only--discharges on the same day as admission. Patients may have multiple admissions per year, but only admissions after 30 days of discharge from an index admission were eligible to be included as an additional index admission.

Outcomes

A readmission was defined as any unplanned rehospitalization to either non-VA or VA acute care facilities for any cause within 30 days of discharge from the index hospitalization. Readmissions to observation status or nonacute or rehabilitation units, such as skilled nursing facilities, were not included. Planned readmissions for elective procedures, such as elective chemotherapy and revascularization following an AMI index admission, were not considered as an outcome event.

Risk Standardization for 30-day Readmission

Using approaches developed by CMS,^10-12 we calculated hospital-specific 30-day RSRRs for each VA. Briefly, the RSRR is a ratio of the number of predicted readmissions within 30 days of discharge to the expected number of readmissions within 30 days of hospital discharge, multiplied by the national unadjusted 30-day readmission rate. This measure calculates hospital-specific RSRRs using hierarchical logistic regression models, which account for clustering of patients within hospitals and risk-adjusting for differences in case-mix, during the assessed time periods.¹³ This approach simultaneously models two levels (patient and hospital) to account for the variance in patient outcomes within and between hospitals.¹⁴ At the patient level, the model uses the log odds of readmissions as the dependent variable and age and selected comorbidities as the independent variables. The second level models the hospital-specific intercepts. According to CMS guidelines, the analysis was limited to facilities with at least 25 patient admissions annually for each condition. All readmissions were attributed to the hospital that initially discharged the patient to a nonacute setting.

Analysis

We examined and reported the distribution of patient and clinical characteristics at the hospital level. For each condition, we determined the number of hospitals that had a sufficient number of admissions (n ≥ 25) to be included in the analyses. We calculated the mean, median, and interquartile range for the observed unadjusted readmission rates across all included hospitals.

Similar to methods used by CMS, we used one year of data in the VA to assess hospital quality and variation in facility performance. First, we calculated the 30-day RSRRs using one year (2012) of data. To assess how variability changed with higher facility volume (ie, more years included in the analysis), we also calculated the 30-day RSRRs using two and three years of data. For this, we identified and quantified the number of hospitals whose RSRRs were calculated as being above or below the national VA average (mean ± 95% CI). Specifically, we calculated the number and percentage of hospitals that were classified as either above (+95% CI) or below the national average (−95% CI) using data from all three time periods. All analyses were conducted using SAS Enterprise Guide, Version 7.1. The SAS statistical packages made available by the CMS Measure Team were used to calculate RSRRs.

Patient Characteristics

Patients were predominantly older males (98.3%). Among those hospitalized for AMI, most of them had a history of previous coronary artery bypass graft (CABG) (69.1%), acute coronary syndrome (ACS; 66.2%), or documented coronary atherosclerosis (89.8%). Similarly, patients admitted for HF had high rates of CABG (71.3%) and HF (94.6%), in addition to cardiac arrhythmias (69.3%) and diabetes (60.8%). Patients admitted with a diagnosis of pneumonia had high rates of CABG (61.9%), chronic obstructive pulmonary disease (COPD; 58.1%), and previous diagnosis of pneumonia (78.8%; Table 1). Patient characteristics for two and three years of data are presented in Supplementary Table 1.

VA Hospitals with Sufficient Volume to Be Included in Profiling Assessments

There were 146 acute-care hospitals in the VA. In 2012, 56 (38%) VA hospitals had at least 25 admissions for AMI, 102 (70%) hospitals had at least 25 admissions for CHF, and 106 (73%) hospitals had at least 25 admissions for pneumonia (Table 1) and therefore qualified for analysis based on CMS criteria for 30-day RSRR calculation. The study sample included 3,571 patients with AMI, 10,609 patients with CHF, and 10,191 patients with pneumonia.

30-Day Readmission Rates

The mean observed readmission rates in 2012 were 20% (95% CI 19%-21%) among patients admitted for AMI, 20% (95% CI 19%-20%) for patients admitted with CHF, and 15% (95% CI 15%-16%) for patients admitted with pneumonia. No significant variation from these rates was noted following risk standardization across hospitals (Table 2). Observed and risk-standardized rates were also calculated for two and three years of data (Supplementary Table 2) but were not found to be grossly different when utilizing a single year of data.

In 2012, two hospitals (2%) exhibited HF RSRRs worse than the national average (+95% CI), whereas no hospital demonstrated worse-than-average rates (+95% CI) for AMI or pneumonia (Table 3, Figure 1). Similarly, in 2012, only three hospitals had RSRRs better than the national average (−95% CI) for HF and pneumonia.

We found that the CMS-derived 30-day risk-stratified readmission metric for AMI, HF, and pneumonia showed little variation among VA hospitals. The lack of institutional 30-day readmission volume appears to be a fundamental limitation that subsequently requires multiple years of data to make this metric clinically meaningful. As the largest integrated healthcare system in the United States, the VA relies upon and makes large-scale programmatic decisions based on such performance data. The inability to detect meaningful interhospital variation in a timely manner suggests that the CMS-derived 30-day RSRR may not be a sensitive metric to distinguish facility performance or drive quality improvement initiatives within the VA.

First, we found it notable that among the 146 VA medical centers available for analysis,¹⁵ between 38% and 77% of hospitals qualified for evaluation when using CMS-based participation criteria—which excludes institutions with fewer than 25 episodes per year. Although this low degree of qualification for profiling was most dramatic when using one year of data (range: 38%-72%), we noted that it did not dramatically improve when we combined three years of data (range: 52%-77%). These findings act to highlight the population and systems differences between CMS and VA populations¹⁶ and further support the idea that CMS-derived models may not be optimized for use in the VA healthcare system.

Our findings are particularly relevant within the VA given the quarterly rate with which these data are reported within the VA SAIL scorecard.² The VA designed SAIL for internal benchmarking to spotlight successful strategies of top performing institutions and promote high-quality, value-based care. Using one year of data, the minimum required to utilize CMS models, showed that quarterly feedback (ie, three months of data) may not be informative or useful given that few hospitals are able to differentiate themselves from the mean (±95% CI). Although the capacity to distinguish between high and low performers does improve by combining hospital admissions over three years, this is not a reasonable timeline for institutions to wait for quality comparisons. Furthermore, although the VA does present its data on CMS’s Hospital Compare website using three years of combined data, the variability and distribution of such results are not supplied.³

This lack of discriminability raises concerns about the ability to compare hospital performance between low- and high-volume institutions. Although these models function well in CMS settings with large patient volumes in which greater variability exists,⁵ they lose their capacity to discriminate when applied to low-volume settings such as the VA. Given that several hospitals in the US are small community hospitals with low patient volumes,¹⁷ this issue probably occurs in other non-VA settings. Although our study focuses on the VA, others have been able to compare VA and non-VA settings’ variation and distribution. For example, Nuti et al. explored the differences in 30-day RSRRs among hospitalized patients with AMI, HF, and pneumonia and similarly showed little variation, narrow distributions, and few outliers in the VA setting compared to those in the non-VA setting. For small patient volume institutions, including the VA, a focus on high-volume services, outcomes, and measures (ie, blood pressure control, medication reconciliation, etc.) may offer more discriminability between high- and low-performing facilities. For example, Patel et al. found that VA process measures in patients with HF (ie, beta-blocker and ACE-inhibitor use) can be used as valid quality measures as they exhibited consistent reliability over time and validity with adjusted mortality rates, whereas the 30-day RSRR did not.¹⁸

Our findings may have substantial financial, resource, and policy implications. Automatically developing and reporting measures created for the Medicare program in the VA may not be a good use of VA resources. In addition, facilities may react to these reported outcomes and expend local resources and finances to implement interventions to improve on a performance outcome whose measure is statistically no different than the vast majority of its comparators. Such events have been highlighted in the public media and have pointed to the fact that small changes in quality, or statistical errors themselves, can have large ramifications within the VA’s hospital rating system.¹⁹

These findings may also add to the discussion on whether public reporting of health and quality outcomes improves patient care. Since the CMS began public reporting on RSRRs in 2009, these rates have fallen for all three examined conditions (AMI, HF, and pneumonia),^7,20,21 in addition to several other health outcomes.¹⁷ Although recent studies have suggested that these decreased rates have been driven by the CMS-sponsored Hospital Readmissions Reduction Program (HRRP),²² others have suggested that these findings are consistent with ongoing secular trends toward decreased readmissions and may not be completely explained by public reporting alone.²³ Moreover, prior work has also found that readmissions may be strongly impacted by factors external to the hospital setting, such as patients’ social demographics (ie, household income, social isolation), that are not currently captured in risk-prediction models.²⁴ Given the small variability we see in our data, public reporting within the VA is probably not beneficial, as only a small number of facilities are outliers based on RSRR.

Our study has several limitations. First, although we adapted the CMS model to the VA, we did not include gender in the model because >99% of all patient admissions were male. Second, we assessed only three medical conditions that were being tracked by both CMS and VA during this time period, and these outcomes may not be representative of other aspects of care and cannot be generalized to other medical conditions. Finally, more contemporary data could lead to differing results – though we note that no large-scale structural or policy changes addressing readmission rates have been implemented within the VA since our study period.

The results of this study suggest that the CMS-derived 30-day risk-stratified readmission metric for AMI, HF, and pneumonia may not have the capacity to properly detect interfacility variance and thus may not be an optimal quality indicator within the VA. As the VA and other healthcare systems continually strive to improve the quality of care they provide, they will require more accurate and timely metrics for which to index their performance.

Disclosures

The authors have nothing to disclose

Access to life-sustaining treatment (LST) became a mainstay in hospitals across the United States in the 1970s. This has raised complex ethical questions surrounding the use of these therapies, particularly in the face of a poor prognosis or significant morbidity. The Society for Critical Care Medicine formed a consensus panel in 1989 to construct ethical guidelines regarding the initiation, continuation, and withdrawal of intensive care.¹ These guidelines emphasized that withdrawing and withholding are not only permissible but may be necessary to preserve the balance between quantity and quality of life. Nevertheless, an increasing number of Americans are dying after aggressive LST in the hospital, and greater than one in five deaths occur after admission to the ICU.² Understanding the factors associated with decisions to withhold or withdraw LST are important to policy makers, ethicists, and healthcare leaders because they affect resources used at the end of life and the need for palliative care and hospice in the ICU setting.

Several studies have characterized the patient characteristics, incidence, and variability associated with limitation of LST in various populations of critically ill patients in the US. We are unaware of another systematic review of the literature that has examined data from these studies in order to understand the process and outcomes of LST limitations. We defined limitations of LST as decisions to withdraw or withhold cardiopulmonary resuscitation through Do Not Resuscitate (DNR) orders, mechanical ventilation, renal replacement therapy, intravenous blood pressure support, or artificial nutrition (enteric or intravenous).

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement was used for reporting. A comprehensive literature search was performed by a medical librarian (TWE) in Ovid MEDLINE, PubMed, Embase, the full Cochrane Library, CINAHL, PsycINFO, the Philosopher’s Index, Scopus, Web of Science, and Google Scholar. PubMed was limited to non-MEDLINE records in order to complement the Ovid results. The Georgetown Bioethics Research Library at the Kennedy Institute (https://bioethics.georgetown.edu/) was also searched for any unpublished literature. Initial searches were conducted in December 2014, and an update was performed in April 2017. All databases were searched from inception, and bibliographies of relevant studies were reviewed for additional references (Appendix 1).

Database-specific subject headings and keyword variants for each of the five main concepts—intensive care, end-of-life, decision making, limitation of treatment, and death—were identified and combined. Results were limited to the adult population and to the English language.

Two authors independently reviewed article titles and abstracts (KM, AMT). The full text of potentially eligible studies was then reviewed for inclusion. All disputes were discussed and resolved by consensus. The criteria for inclusion were reporting of patient-level data, critical care patients only (or reported separately from other unit types), US setting, and reporting of data on limitations of LST. The exclusion criteria were studies published only as research abstracts, surveys of physicians or families, organ donors, studies of brain death, surveys, patients less than 18 years old, and long-term intensive care settings (ie, long-term acute care hospitals, long-term respiratory units). Also excluded were studies in which an intervention was performed; as a result, all included studies were observational. Research abstracts were excluded because they lacked sufficient detail from which to abstract study quality or results. Studies of organ donation, brain death, and pediatrics were excluded due to differences in the decision-making context that would make it difficult to draw conclusions about adult ICU care. Studies which included an intervention were excluded to avoid affecting the rate of limitation of LST as a result of the intervention, since our goal was to quantify the number of limitations of LST in usual medical practice.

For each article, we abstracted the number of patients who experienced a limitation of LST out of the total population and factors associated with the limitation. If a multivariable analysis was performed, we reported only variables that remained significant in this analysis. We also reported the number of patients who died, and of those, the number of decedents who underwent a limitation of LST before death. In some cases, this proportion was not reported in the manuscript but could be calculated based on the data presented. This number was calculated based on the number of deaths that were preceded by a limitation in life-sustaining care divided by the total number of deaths. Patients with brain death were not counted as having had a “limitation” if support was withdrawn after the declaration of brain death. We were unable to conduct a meta-analysis of the findings because of the wide variation in study populations and criteria used to define limitations of care.

To assess risk of bias in individual studies, the two raters independently made a yes/no determination regarding several quality metrics established at the outset of the review: clarity of the eligibility criteria for participant inclusion, whether a power or sample size calculation was done, adequacy of the description of the sampling approach and recruitment, and generalizability. Disagreements were resolved by consensus.

Study Selection

A total of 2,460 references were identified, and after removal of 578 duplicates, 1,882 unique titles and abstracts were reviewed. One hundred thirteen titles met the inclusion criteria. After review of complete texts, 83 were excluded based on the above criteria (Appendix). This led to a final number of 36 studies included for analysis.

Fifteen articles were prospective, observational studies. The rest were retrospective analyses of patient-level data. Seven were large, multicenter studies with greater than 20 centers involved (including Project IMPACT); six such studies included medical and surgical patients. The remaining large, multicenter study examined a surgical trauma cohort.

Historical Trends

Examination of the three studies that looked at >20 regionally diverse ICUs revealed a trend over time toward increased limitation prior to death (Figure). Jayes looked at the number of DNR orders preceding death from 1979 to 1980 then compared that to a cohort from 1988 to 1990; Prendergast included withholding/withdrawing of LST prior to death from 1994 to 1995;and Quill used the IMPACT database to examine limitations prior to death from 2001 to 2009.^3-5

Effect of Unit Specialty

Twelve studies were mixed (surgical/medical or medical/neuro) ICUs, 11 were medical/cardiac units, five were neurologic units, and six were surgical/trauma units only. Two studies did not report unit specialty. Four studies that compared surgical and medical ICUs found that surgical patients were more likely to die with full intervention.^4-7 In all of these studies, medical patients were more likely to have limitations of LST preceding death. Quill, et al. further detailed that emergency surgery was more likely to be associated with limitation than elective surgery.⁵

Patient Factors

In 15 studies, older age was associated with an increased likelihood of limitations on LST.^3,5-18 In one study, advanced age was associated with early versus late withdrawal.¹⁹ Poor performance status and multiple medical comorbidities were also associated with limitations of LST. The largest population-based study by Quill et al. found that being fully dependent on others upon admission to the ICU was associated with an increased likelihood of limiting LST.⁵ Sise et al. found, in an analysis performed over 10 years in one trauma center, that increased age, comorbidities, and a fall as the reason for trauma admission were associated with limitation of LST.⁹ Salottolo et al. found that if the reason for trauma admission was a fall, there was an increased odds ratio of DNR status.¹⁸ Many studies found that having medical comorbidities prior to admission was associated with increased likelihood of limiting LST in both medical and surgical patients.^{3,7,9,13,15,18}

Five studies found a statistically significant difference between women and men in the likelihood of limitation of LST,^3,5,9,14,16 and another study reported that women who were trauma patients had an increased odds ratio of changing to DNR code status.¹⁸ Only one study found that males were associated with an increased likelihood of limiting aggressive treatment.²⁰

White race was associated with increased limitation of LST in nine studies.^{4,5,10,11,14-16,21,22} One study in neurocritical care patients found that both white and Hispanic races were correlated with a higher likelihood of limitations.²³ Muni et al. found that nonwhite patients had a statistically significantly lower likelihood of having comfort measures and DNR orders written prior to death, but discussion of prognosis was more likely to be documented in nonwhite patients.²¹

In summary, white race, female gender, and older age were the most frequent factors associated with a higher likelihood of limiting LST.

Factors Related to Critical Illness

There were several illness severity indicators that were associated with limitations. The Acute Physiology and Chronic Health Evaluation (APACHE) scores were the most common for medical patients and Glasgow Coma Scale (GCS) was the most common for patients with neurologic injury. Eight studies reported that a higher APACHE score was associated with an increased likelihood of limitations.^{3,7,10,15,17,20,22,24} Similar associations were found based on the Sepsis Related Organ Failure Assessment score in one study and a scoring system developed by the author in a second study.^25,26

Seven studies, consisting of three neurologic, two medical-surgical, and two trauma cohorts, reported that a lower GCS score increased the likelihood that the patient would have limited LST.^{5,10,11,13,14,18,22} Additionally, Geocadin and colleagues discussed the difficulty with neurological prognostication in clinical practice; they reported that the cortical evoked potential (CEP) was correlated with the time to withdrawLST if the CEP was malignant, and the time to withdraw LST was less in malignant than in benign CEP.²⁷

Mortality and End Effects of Limiting LST

Chen and colleagues used propensity scores to control for mortality differences between patients who had full interventions versus those with limitations and found that higher mortality correlated with the decision to withhold or withdraw LST.¹⁰ Weimer and colleagues used modeling to predict the probable outcome of patients who experienced an intracranial hemorrhage who had limitation of LST. Based on this model, nearly all the patients in their study would have died or had severe disability at 12 months despite having maximal therapy; they concluded that withdrawal of LST may not have been a self-fulfilling prophecy as others have proposed.²⁸ Mulder and colleagues reported that in a small cohort of out-of-hospital cardiac arrest survivors admitted to the hospital, over one-third had good neurological outcomes after coding after 72 hours.²⁹ The study highlighted the importance of timing in neurological prognostication.

Variation in Limitation Rates among Centers

In the 36 studies, we found an overall range of DNR orders from 5.4%⁷ to 82.0%.³⁰ For other limitations, the rates ranged from 6.3%¹³ to 80.4%.³¹ Hart reported a low rate of limitations (4.8%) at the time of ICU admission.¹⁶ Four large, multicenter studies drew attention to the large variability between critical care centers and the limitation of end-of-life care.^3-5,14 Jayes first described this phenomenon when examining the frequency of DNR orders from 1979 to 1980 and 1988 to 1990.³ This study found a range from 1.5% to 22%. Later, in another large, multicenter study, Prendergast et al. looked at 131 ICUs at 110 different institutions in 38 states that participated in postgraduate training and found variability in CPR attempts prior to death between 4% and 79%.⁴ In 2008, Nathens et al. reported significant variation in DNR rates across trauma centers; they found a higher incidence of DNR orders when there was an open ICU structure.¹⁴

Overall, there was wide variation in the proportion of deaths preceded by limitation of LST, ranging from 29.5% in one study of trauma patients⁸ to 92% in another study of trauma patients whose death occurred after 24 hours of care.⁹ In the largest study to date by Quill and colleagues utilizing the IMPACT database, they found large variability in the number of deaths preceded by full intervention based on differences in practice patterns of critical care centers.⁵

Bias

All studies indicated clear eligibility criteria for inclusion and described their sampling approach in adequate detail. All but one stated their method of participant recruitment, and the one remaining study was a secondary analysis and referenced the earlier manuscript.³⁰ No study provided a power or sample size calculation, and sample sizes varied widely. Generalizability was most affected by the fact that many studies were conducted in a single ICU.

This systematic review of LST in US ICUs found several patient and illness factors that were associated with limitation of LST. The association of preadmission functional status and comorbidities with limitation of LST suggest that prior health is a factor in decision making. Further, ICU severity of illness, as measured by several commonly used indices, was associated with limitations.

Although variations in study design precluded meta-analysis, examination of the largest studies suggests that limitations are becoming more frequent over time. Also, early studies generally addressed DNR status, while later studies included withdrawal or withholding of LST, most commonly artificial ventilation. These findings reflect the current consensus in US medicine that it is ethically acceptable to limit LSTs in cases when they no longer benefit the patient or the patient would no longer want them.^32,33

Some studies found variability by unit type, suggesting that decision making may differ among surgical, medical, and neurologic illness. Mayerand Kossoff concluded, in study of a cohort of neurocritical care ICU patients, that medical patients often have issues of physiologic futility and imminent death, whereas neurologic patients more often confront issues of quality of life. They also note that there is a difference in how patients with differing illnesses die; medical patients will have limitation of hemodialysis or vasopressors, whereas neurologic surrogate decision makers often confront decisions around terminal extubation.²³

Some patient-level factors, such as race or ethnicity, may point to cultural differences in preferences for LST at the end of life. Other authors have documented that African American patients are more likely to choose end-of-life care for themselves or their family members, which may be due to cultural or religious factors as well as to a history of unequal access to medical care.³⁴ Reasons for the finding that women are more likely to have limitations has not been as well described. Further research could explore whether this is due to differences in patient preferences by gender or to other factors.

Even when examining patient-level factors, illness severity and type of ICU, the wide variability in end-of-life care in critical care units across the country is still large. A worldwide review also found a high degree of variability, even within geographical regions.³⁵ More research is needed to understand the factors associated with this wide variability, as this seems to indicate that approaches to end-of-life care may vary based on the ICU as much as individual patient preferences or clinical factors.

These findings can inform clinicians about variables that are important in the decision-making process. Patient age and race are factors to consider in the likelihood of reaching a decision to set limitations. Information about patients’ health status prior to critical illness, as well as ICU illness severity, are also important considerations.

The limitations of this review include the wide variety of LSTs assessed, including code status change, ventilator withdrawal, removal of pressors, and cessation of renal replacement therapy. Also, there was variation in sample size and the number of included units. There was also significant heterogeneity in the outcomes addressed and the variety of methods used in the included studies. We attempted to address this with an analysis of the quality of the studies, but given the wide variability, we were unable to account for all of the differences; unfortunately, this is a standard issue within studies that utilize systematic reviews, as well as similar concepts such as meta-analyses.

In conclusion, the increase in the frequency of limitations of LST in critically ill patients and a change in the nature of limitations from DNR order to withdrawal or withholding of LST suggests a trend toward growing acceptance of limiting treatments in critical illness. The wide variation in withdrawal of care in US ICUs does not seem fully explained by patient variables including preferences, illness type, or changes over time. Factors such as poor prefunctional status, a higher number of comorbid conditions prior to critical illness, and the severity of critical illness are likely important for surrogates and clinicians to consider during goals of care discussions. Further research is needed to explore why patients may receive very different types of care at the end of life depending the institution and ICU in which they receive their care.

Disclosures

The authors have no conflicts of interest to disclose. This work was performed at the Indiana University School of Medicine.

Funding

Financial support for Dr. Torke was provided by a Midcareer Investigator Award in Patient Oriented Research from the National Institute on Aging (K24AG053794). Dr. McPherson was supported by the Indiana University Department of Medicine.

Access to life-sustaining treatment (LST) became a mainstay in hospitals across the United States in the 1970s. This has raised complex ethical questions surrounding the use of these therapies, particularly in the face of a poor prognosis or significant morbidity. The Society for Critical Care Medicine formed a consensus panel in 1989 to construct ethical guidelines regarding the initiation, continuation, and withdrawal of intensive care.¹ These guidelines emphasized that withdrawing and withholding are not only permissible but may be necessary to preserve the balance between quantity and quality of life. Nevertheless, an increasing number of Americans are dying after aggressive LST in the hospital, and greater than one in five deaths occur after admission to the ICU.² Understanding the factors associated with decisions to withhold or withdraw LST are important to policy makers, ethicists, and healthcare leaders because they affect resources used at the end of life and the need for palliative care and hospice in the ICU setting.

Several studies have characterized the patient characteristics, incidence, and variability associated with limitation of LST in various populations of critically ill patients in the US. We are unaware of another systematic review of the literature that has examined data from these studies in order to understand the process and outcomes of LST limitations. We defined limitations of LST as decisions to withdraw or withhold cardiopulmonary resuscitation through Do Not Resuscitate (DNR) orders, mechanical ventilation, renal replacement therapy, intravenous blood pressure support, or artificial nutrition (enteric or intravenous).

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement was used for reporting. A comprehensive literature search was performed by a medical librarian (TWE) in Ovid MEDLINE, PubMed, Embase, the full Cochrane Library, CINAHL, PsycINFO, the Philosopher’s Index, Scopus, Web of Science, and Google Scholar. PubMed was limited to non-MEDLINE records in order to complement the Ovid results. The Georgetown Bioethics Research Library at the Kennedy Institute (https://bioethics.georgetown.edu/) was also searched for any unpublished literature. Initial searches were conducted in December 2014, and an update was performed in April 2017. All databases were searched from inception, and bibliographies of relevant studies were reviewed for additional references (Appendix 1).

Database-specific subject headings and keyword variants for each of the five main concepts—intensive care, end-of-life, decision making, limitation of treatment, and death—were identified and combined. Results were limited to the adult population and to the English language.

Two authors independently reviewed article titles and abstracts (KM, AMT). The full text of potentially eligible studies was then reviewed for inclusion. All disputes were discussed and resolved by consensus. The criteria for inclusion were reporting of patient-level data, critical care patients only (or reported separately from other unit types), US setting, and reporting of data on limitations of LST. The exclusion criteria were studies published only as research abstracts, surveys of physicians or families, organ donors, studies of brain death, surveys, patients less than 18 years old, and long-term intensive care settings (ie, long-term acute care hospitals, long-term respiratory units). Also excluded were studies in which an intervention was performed; as a result, all included studies were observational. Research abstracts were excluded because they lacked sufficient detail from which to abstract study quality or results. Studies of organ donation, brain death, and pediatrics were excluded due to differences in the decision-making context that would make it difficult to draw conclusions about adult ICU care. Studies which included an intervention were excluded to avoid affecting the rate of limitation of LST as a result of the intervention, since our goal was to quantify the number of limitations of LST in usual medical practice.

For each article, we abstracted the number of patients who experienced a limitation of LST out of the total population and factors associated with the limitation. If a multivariable analysis was performed, we reported only variables that remained significant in this analysis. We also reported the number of patients who died, and of those, the number of decedents who underwent a limitation of LST before death. In some cases, this proportion was not reported in the manuscript but could be calculated based on the data presented. This number was calculated based on the number of deaths that were preceded by a limitation in life-sustaining care divided by the total number of deaths. Patients with brain death were not counted as having had a “limitation” if support was withdrawn after the declaration of brain death. We were unable to conduct a meta-analysis of the findings because of the wide variation in study populations and criteria used to define limitations of care.

To assess risk of bias in individual studies, the two raters independently made a yes/no determination regarding several quality metrics established at the outset of the review: clarity of the eligibility criteria for participant inclusion, whether a power or sample size calculation was done, adequacy of the description of the sampling approach and recruitment, and generalizability. Disagreements were resolved by consensus.

Study Selection

A total of 2,460 references were identified, and after removal of 578 duplicates, 1,882 unique titles and abstracts were reviewed. One hundred thirteen titles met the inclusion criteria. After review of complete texts, 83 were excluded based on the above criteria (Appendix). This led to a final number of 36 studies included for analysis.

Fifteen articles were prospective, observational studies. The rest were retrospective analyses of patient-level data. Seven were large, multicenter studies with greater than 20 centers involved (including Project IMPACT); six such studies included medical and surgical patients. The remaining large, multicenter study examined a surgical trauma cohort.

Historical Trends

Examination of the three studies that looked at >20 regionally diverse ICUs revealed a trend over time toward increased limitation prior to death (Figure). Jayes looked at the number of DNR orders preceding death from 1979 to 1980 then compared that to a cohort from 1988 to 1990; Prendergast included withholding/withdrawing of LST prior to death from 1994 to 1995;and Quill used the IMPACT database to examine limitations prior to death from 2001 to 2009.^3-5

Effect of Unit Specialty

Twelve studies were mixed (surgical/medical or medical/neuro) ICUs, 11 were medical/cardiac units, five were neurologic units, and six were surgical/trauma units only. Two studies did not report unit specialty. Four studies that compared surgical and medical ICUs found that surgical patients were more likely to die with full intervention.^4-7 In all of these studies, medical patients were more likely to have limitations of LST preceding death. Quill, et al. further detailed that emergency surgery was more likely to be associated with limitation than elective surgery.⁵

Patient Factors

In 15 studies, older age was associated with an increased likelihood of limitations on LST.^3,5-18 In one study, advanced age was associated with early versus late withdrawal.¹⁹ Poor performance status and multiple medical comorbidities were also associated with limitations of LST. The largest population-based study by Quill et al. found that being fully dependent on others upon admission to the ICU was associated with an increased likelihood of limiting LST.⁵ Sise et al. found, in an analysis performed over 10 years in one trauma center, that increased age, comorbidities, and a fall as the reason for trauma admission were associated with limitation of LST.⁹ Salottolo et al. found that if the reason for trauma admission was a fall, there was an increased odds ratio of DNR status.¹⁸ Many studies found that having medical comorbidities prior to admission was associated with increased likelihood of limiting LST in both medical and surgical patients.^{3,7,9,13,15,18}

Five studies found a statistically significant difference between women and men in the likelihood of limitation of LST,^3,5,9,14,16 and another study reported that women who were trauma patients had an increased odds ratio of changing to DNR code status.¹⁸ Only one study found that males were associated with an increased likelihood of limiting aggressive treatment.²⁰

White race was associated with increased limitation of LST in nine studies.^{4,5,10,11,14-16,21,22} One study in neurocritical care patients found that both white and Hispanic races were correlated with a higher likelihood of limitations.²³ Muni et al. found that nonwhite patients had a statistically significantly lower likelihood of having comfort measures and DNR orders written prior to death, but discussion of prognosis was more likely to be documented in nonwhite patients.²¹

In summary, white race, female gender, and older age were the most frequent factors associated with a higher likelihood of limiting LST.

Factors Related to Critical Illness

There were several illness severity indicators that were associated with limitations. The Acute Physiology and Chronic Health Evaluation (APACHE) scores were the most common for medical patients and Glasgow Coma Scale (GCS) was the most common for patients with neurologic injury. Eight studies reported that a higher APACHE score was associated with an increased likelihood of limitations.^{3,7,10,15,17,20,22,24} Similar associations were found based on the Sepsis Related Organ Failure Assessment score in one study and a scoring system developed by the author in a second study.^25,26

Seven studies, consisting of three neurologic, two medical-surgical, and two trauma cohorts, reported that a lower GCS score increased the likelihood that the patient would have limited LST.^{5,10,11,13,14,18,22} Additionally, Geocadin and colleagues discussed the difficulty with neurological prognostication in clinical practice; they reported that the cortical evoked potential (CEP) was correlated with the time to withdrawLST if the CEP was malignant, and the time to withdraw LST was less in malignant than in benign CEP.²⁷

Mortality and End Effects of Limiting LST

Chen and colleagues used propensity scores to control for mortality differences between patients who had full interventions versus those with limitations and found that higher mortality correlated with the decision to withhold or withdraw LST.¹⁰ Weimer and colleagues used modeling to predict the probable outcome of patients who experienced an intracranial hemorrhage who had limitation of LST. Based on this model, nearly all the patients in their study would have died or had severe disability at 12 months despite having maximal therapy; they concluded that withdrawal of LST may not have been a self-fulfilling prophecy as others have proposed.²⁸ Mulder and colleagues reported that in a small cohort of out-of-hospital cardiac arrest survivors admitted to the hospital, over one-third had good neurological outcomes after coding after 72 hours.²⁹ The study highlighted the importance of timing in neurological prognostication.

Variation in Limitation Rates among Centers

In the 36 studies, we found an overall range of DNR orders from 5.4%⁷ to 82.0%.³⁰ For other limitations, the rates ranged from 6.3%¹³ to 80.4%.³¹ Hart reported a low rate of limitations (4.8%) at the time of ICU admission.¹⁶ Four large, multicenter studies drew attention to the large variability between critical care centers and the limitation of end-of-life care.^3-5,14 Jayes first described this phenomenon when examining the frequency of DNR orders from 1979 to 1980 and 1988 to 1990.³ This study found a range from 1.5% to 22%. Later, in another large, multicenter study, Prendergast et al. looked at 131 ICUs at 110 different institutions in 38 states that participated in postgraduate training and found variability in CPR attempts prior to death between 4% and 79%.⁴ In 2008, Nathens et al. reported significant variation in DNR rates across trauma centers; they found a higher incidence of DNR orders when there was an open ICU structure.¹⁴

Overall, there was wide variation in the proportion of deaths preceded by limitation of LST, ranging from 29.5% in one study of trauma patients⁸ to 92% in another study of trauma patients whose death occurred after 24 hours of care.⁹ In the largest study to date by Quill and colleagues utilizing the IMPACT database, they found large variability in the number of deaths preceded by full intervention based on differences in practice patterns of critical care centers.⁵

Bias

All studies indicated clear eligibility criteria for inclusion and described their sampling approach in adequate detail. All but one stated their method of participant recruitment, and the one remaining study was a secondary analysis and referenced the earlier manuscript.³⁰ No study provided a power or sample size calculation, and sample sizes varied widely. Generalizability was most affected by the fact that many studies were conducted in a single ICU.

This systematic review of LST in US ICUs found several patient and illness factors that were associated with limitation of LST. The association of preadmission functional status and comorbidities with limitation of LST suggest that prior health is a factor in decision making. Further, ICU severity of illness, as measured by several commonly used indices, was associated with limitations.

Although variations in study design precluded meta-analysis, examination of the largest studies suggests that limitations are becoming more frequent over time. Also, early studies generally addressed DNR status, while later studies included withdrawal or withholding of LST, most commonly artificial ventilation. These findings reflect the current consensus in US medicine that it is ethically acceptable to limit LSTs in cases when they no longer benefit the patient or the patient would no longer want them.^32,33

Some studies found variability by unit type, suggesting that decision making may differ among surgical, medical, and neurologic illness. Mayerand Kossoff concluded, in study of a cohort of neurocritical care ICU patients, that medical patients often have issues of physiologic futility and imminent death, whereas neurologic patients more often confront issues of quality of life. They also note that there is a difference in how patients with differing illnesses die; medical patients will have limitation of hemodialysis or vasopressors, whereas neurologic surrogate decision makers often confront decisions around terminal extubation.²³

Some patient-level factors, such as race or ethnicity, may point to cultural differences in preferences for LST at the end of life. Other authors have documented that African American patients are more likely to choose end-of-life care for themselves or their family members, which may be due to cultural or religious factors as well as to a history of unequal access to medical care.³⁴ Reasons for the finding that women are more likely to have limitations has not been as well described. Further research could explore whether this is due to differences in patient preferences by gender or to other factors.

Even when examining patient-level factors, illness severity and type of ICU, the wide variability in end-of-life care in critical care units across the country is still large. A worldwide review also found a high degree of variability, even within geographical regions.³⁵ More research is needed to understand the factors associated with this wide variability, as this seems to indicate that approaches to end-of-life care may vary based on the ICU as much as individual patient preferences or clinical factors.

These findings can inform clinicians about variables that are important in the decision-making process. Patient age and race are factors to consider in the likelihood of reaching a decision to set limitations. Information about patients’ health status prior to critical illness, as well as ICU illness severity, are also important considerations.

The limitations of this review include the wide variety of LSTs assessed, including code status change, ventilator withdrawal, removal of pressors, and cessation of renal replacement therapy. Also, there was variation in sample size and the number of included units. There was also significant heterogeneity in the outcomes addressed and the variety of methods used in the included studies. We attempted to address this with an analysis of the quality of the studies, but given the wide variability, we were unable to account for all of the differences; unfortunately, this is a standard issue within studies that utilize systematic reviews, as well as similar concepts such as meta-analyses.

In conclusion, the increase in the frequency of limitations of LST in critically ill patients and a change in the nature of limitations from DNR order to withdrawal or withholding of LST suggests a trend toward growing acceptance of limiting treatments in critical illness. The wide variation in withdrawal of care in US ICUs does not seem fully explained by patient variables including preferences, illness type, or changes over time. Factors such as poor prefunctional status, a higher number of comorbid conditions prior to critical illness, and the severity of critical illness are likely important for surrogates and clinicians to consider during goals of care discussions. Further research is needed to explore why patients may receive very different types of care at the end of life depending the institution and ICU in which they receive their care.

Disclosures

The authors have no conflicts of interest to disclose. This work was performed at the Indiana University School of Medicine.

Funding

Financial support for Dr. Torke was provided by a Midcareer Investigator Award in Patient Oriented Research from the National Institute on Aging (K24AG053794). Dr. McPherson was supported by the Indiana University Department of Medicine.

Access to life-sustaining treatment (LST) became a mainstay in hospitals across the United States in the 1970s. This has raised complex ethical questions surrounding the use of these therapies, particularly in the face of a poor prognosis or significant morbidity. The Society for Critical Care Medicine formed a consensus panel in 1989 to construct ethical guidelines regarding the initiation, continuation, and withdrawal of intensive care.¹ These guidelines emphasized that withdrawing and withholding are not only permissible but may be necessary to preserve the balance between quantity and quality of life. Nevertheless, an increasing number of Americans are dying after aggressive LST in the hospital, and greater than one in five deaths occur after admission to the ICU.² Understanding the factors associated with decisions to withhold or withdraw LST are important to policy makers, ethicists, and healthcare leaders because they affect resources used at the end of life and the need for palliative care and hospice in the ICU setting.

Several studies have characterized the patient characteristics, incidence, and variability associated with limitation of LST in various populations of critically ill patients in the US. We are unaware of another systematic review of the literature that has examined data from these studies in order to understand the process and outcomes of LST limitations. We defined limitations of LST as decisions to withdraw or withhold cardiopulmonary resuscitation through Do Not Resuscitate (DNR) orders, mechanical ventilation, renal replacement therapy, intravenous blood pressure support, or artificial nutrition (enteric or intravenous).

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement was used for reporting. A comprehensive literature search was performed by a medical librarian (TWE) in Ovid MEDLINE, PubMed, Embase, the full Cochrane Library, CINAHL, PsycINFO, the Philosopher’s Index, Scopus, Web of Science, and Google Scholar. PubMed was limited to non-MEDLINE records in order to complement the Ovid results. The Georgetown Bioethics Research Library at the Kennedy Institute (https://bioethics.georgetown.edu/) was also searched for any unpublished literature. Initial searches were conducted in December 2014, and an update was performed in April 2017. All databases were searched from inception, and bibliographies of relevant studies were reviewed for additional references (Appendix 1).

Database-specific subject headings and keyword variants for each of the five main concepts—intensive care, end-of-life, decision making, limitation of treatment, and death—were identified and combined. Results were limited to the adult population and to the English language.

Two authors independently reviewed article titles and abstracts (KM, AMT). The full text of potentially eligible studies was then reviewed for inclusion. All disputes were discussed and resolved by consensus. The criteria for inclusion were reporting of patient-level data, critical care patients only (or reported separately from other unit types), US setting, and reporting of data on limitations of LST. The exclusion criteria were studies published only as research abstracts, surveys of physicians or families, organ donors, studies of brain death, surveys, patients less than 18 years old, and long-term intensive care settings (ie, long-term acute care hospitals, long-term respiratory units). Also excluded were studies in which an intervention was performed; as a result, all included studies were observational. Research abstracts were excluded because they lacked sufficient detail from which to abstract study quality or results. Studies of organ donation, brain death, and pediatrics were excluded due to differences in the decision-making context that would make it difficult to draw conclusions about adult ICU care. Studies which included an intervention were excluded to avoid affecting the rate of limitation of LST as a result of the intervention, since our goal was to quantify the number of limitations of LST in usual medical practice.

For each article, we abstracted the number of patients who experienced a limitation of LST out of the total population and factors associated with the limitation. If a multivariable analysis was performed, we reported only variables that remained significant in this analysis. We also reported the number of patients who died, and of those, the number of decedents who underwent a limitation of LST before death. In some cases, this proportion was not reported in the manuscript but could be calculated based on the data presented. This number was calculated based on the number of deaths that were preceded by a limitation in life-sustaining care divided by the total number of deaths. Patients with brain death were not counted as having had a “limitation” if support was withdrawn after the declaration of brain death. We were unable to conduct a meta-analysis of the findings because of the wide variation in study populations and criteria used to define limitations of care.

To assess risk of bias in individual studies, the two raters independently made a yes/no determination regarding several quality metrics established at the outset of the review: clarity of the eligibility criteria for participant inclusion, whether a power or sample size calculation was done, adequacy of the description of the sampling approach and recruitment, and generalizability. Disagreements were resolved by consensus.

Study Selection

A total of 2,460 references were identified, and after removal of 578 duplicates, 1,882 unique titles and abstracts were reviewed. One hundred thirteen titles met the inclusion criteria. After review of complete texts, 83 were excluded based on the above criteria (Appendix). This led to a final number of 36 studies included for analysis.

Fifteen articles were prospective, observational studies. The rest were retrospective analyses of patient-level data. Seven were large, multicenter studies with greater than 20 centers involved (including Project IMPACT); six such studies included medical and surgical patients. The remaining large, multicenter study examined a surgical trauma cohort.

Historical Trends

Examination of the three studies that looked at >20 regionally diverse ICUs revealed a trend over time toward increased limitation prior to death (Figure). Jayes looked at the number of DNR orders preceding death from 1979 to 1980 then compared that to a cohort from 1988 to 1990; Prendergast included withholding/withdrawing of LST prior to death from 1994 to 1995;and Quill used the IMPACT database to examine limitations prior to death from 2001 to 2009.^3-5

Effect of Unit Specialty

Twelve studies were mixed (surgical/medical or medical/neuro) ICUs, 11 were medical/cardiac units, five were neurologic units, and six were surgical/trauma units only. Two studies did not report unit specialty. Four studies that compared surgical and medical ICUs found that surgical patients were more likely to die with full intervention.^4-7 In all of these studies, medical patients were more likely to have limitations of LST preceding death. Quill, et al. further detailed that emergency surgery was more likely to be associated with limitation than elective surgery.⁵

Patient Factors

In 15 studies, older age was associated with an increased likelihood of limitations on LST.^3,5-18 In one study, advanced age was associated with early versus late withdrawal.¹⁹ Poor performance status and multiple medical comorbidities were also associated with limitations of LST. The largest population-based study by Quill et al. found that being fully dependent on others upon admission to the ICU was associated with an increased likelihood of limiting LST.⁵ Sise et al. found, in an analysis performed over 10 years in one trauma center, that increased age, comorbidities, and a fall as the reason for trauma admission were associated with limitation of LST.⁹ Salottolo et al. found that if the reason for trauma admission was a fall, there was an increased odds ratio of DNR status.¹⁸ Many studies found that having medical comorbidities prior to admission was associated with increased likelihood of limiting LST in both medical and surgical patients.^{3,7,9,13,15,18}

Five studies found a statistically significant difference between women and men in the likelihood of limitation of LST,^3,5,9,14,16 and another study reported that women who were trauma patients had an increased odds ratio of changing to DNR code status.¹⁸ Only one study found that males were associated with an increased likelihood of limiting aggressive treatment.²⁰

White race was associated with increased limitation of LST in nine studies.^{4,5,10,11,14-16,21,22} One study in neurocritical care patients found that both white and Hispanic races were correlated with a higher likelihood of limitations.²³ Muni et al. found that nonwhite patients had a statistically significantly lower likelihood of having comfort measures and DNR orders written prior to death, but discussion of prognosis was more likely to be documented in nonwhite patients.²¹

In summary, white race, female gender, and older age were the most frequent factors associated with a higher likelihood of limiting LST.

Factors Related to Critical Illness

There were several illness severity indicators that were associated with limitations. The Acute Physiology and Chronic Health Evaluation (APACHE) scores were the most common for medical patients and Glasgow Coma Scale (GCS) was the most common for patients with neurologic injury. Eight studies reported that a higher APACHE score was associated with an increased likelihood of limitations.^{3,7,10,15,17,20,22,24} Similar associations were found based on the Sepsis Related Organ Failure Assessment score in one study and a scoring system developed by the author in a second study.^25,26

Seven studies, consisting of three neurologic, two medical-surgical, and two trauma cohorts, reported that a lower GCS score increased the likelihood that the patient would have limited LST.^{5,10,11,13,14,18,22} Additionally, Geocadin and colleagues discussed the difficulty with neurological prognostication in clinical practice; they reported that the cortical evoked potential (CEP) was correlated with the time to withdrawLST if the CEP was malignant, and the time to withdraw LST was less in malignant than in benign CEP.²⁷

Mortality and End Effects of Limiting LST

Chen and colleagues used propensity scores to control for mortality differences between patients who had full interventions versus those with limitations and found that higher mortality correlated with the decision to withhold or withdraw LST.¹⁰ Weimer and colleagues used modeling to predict the probable outcome of patients who experienced an intracranial hemorrhage who had limitation of LST. Based on this model, nearly all the patients in their study would have died or had severe disability at 12 months despite having maximal therapy; they concluded that withdrawal of LST may not have been a self-fulfilling prophecy as others have proposed.²⁸ Mulder and colleagues reported that in a small cohort of out-of-hospital cardiac arrest survivors admitted to the hospital, over one-third had good neurological outcomes after coding after 72 hours.²⁹ The study highlighted the importance of timing in neurological prognostication.

Variation in Limitation Rates among Centers

In the 36 studies, we found an overall range of DNR orders from 5.4%⁷ to 82.0%.³⁰ For other limitations, the rates ranged from 6.3%¹³ to 80.4%.³¹ Hart reported a low rate of limitations (4.8%) at the time of ICU admission.¹⁶ Four large, multicenter studies drew attention to the large variability between critical care centers and the limitation of end-of-life care.^3-5,14 Jayes first described this phenomenon when examining the frequency of DNR orders from 1979 to 1980 and 1988 to 1990.³ This study found a range from 1.5% to 22%. Later, in another large, multicenter study, Prendergast et al. looked at 131 ICUs at 110 different institutions in 38 states that participated in postgraduate training and found variability in CPR attempts prior to death between 4% and 79%.⁴ In 2008, Nathens et al. reported significant variation in DNR rates across trauma centers; they found a higher incidence of DNR orders when there was an open ICU structure.¹⁴

Overall, there was wide variation in the proportion of deaths preceded by limitation of LST, ranging from 29.5% in one study of trauma patients⁸ to 92% in another study of trauma patients whose death occurred after 24 hours of care.⁹ In the largest study to date by Quill and colleagues utilizing the IMPACT database, they found large variability in the number of deaths preceded by full intervention based on differences in practice patterns of critical care centers.⁵

Bias

All studies indicated clear eligibility criteria for inclusion and described their sampling approach in adequate detail. All but one stated their method of participant recruitment, and the one remaining study was a secondary analysis and referenced the earlier manuscript.³⁰ No study provided a power or sample size calculation, and sample sizes varied widely. Generalizability was most affected by the fact that many studies were conducted in a single ICU.

This systematic review of LST in US ICUs found several patient and illness factors that were associated with limitation of LST. The association of preadmission functional status and comorbidities with limitation of LST suggest that prior health is a factor in decision making. Further, ICU severity of illness, as measured by several commonly used indices, was associated with limitations.

Although variations in study design precluded meta-analysis, examination of the largest studies suggests that limitations are becoming more frequent over time. Also, early studies generally addressed DNR status, while later studies included withdrawal or withholding of LST, most commonly artificial ventilation. These findings reflect the current consensus in US medicine that it is ethically acceptable to limit LSTs in cases when they no longer benefit the patient or the patient would no longer want them.^32,33

Some studies found variability by unit type, suggesting that decision making may differ among surgical, medical, and neurologic illness. Mayerand Kossoff concluded, in study of a cohort of neurocritical care ICU patients, that medical patients often have issues of physiologic futility and imminent death, whereas neurologic patients more often confront issues of quality of life. They also note that there is a difference in how patients with differing illnesses die; medical patients will have limitation of hemodialysis or vasopressors, whereas neurologic surrogate decision makers often confront decisions around terminal extubation.²³

Some patient-level factors, such as race or ethnicity, may point to cultural differences in preferences for LST at the end of life. Other authors have documented that African American patients are more likely to choose end-of-life care for themselves or their family members, which may be due to cultural or religious factors as well as to a history of unequal access to medical care.³⁴ Reasons for the finding that women are more likely to have limitations has not been as well described. Further research could explore whether this is due to differences in patient preferences by gender or to other factors.

Even when examining patient-level factors, illness severity and type of ICU, the wide variability in end-of-life care in critical care units across the country is still large. A worldwide review also found a high degree of variability, even within geographical regions.³⁵ More research is needed to understand the factors associated with this wide variability, as this seems to indicate that approaches to end-of-life care may vary based on the ICU as much as individual patient preferences or clinical factors.

These findings can inform clinicians about variables that are important in the decision-making process. Patient age and race are factors to consider in the likelihood of reaching a decision to set limitations. Information about patients’ health status prior to critical illness, as well as ICU illness severity, are also important considerations.

The limitations of this review include the wide variety of LSTs assessed, including code status change, ventilator withdrawal, removal of pressors, and cessation of renal replacement therapy. Also, there was variation in sample size and the number of included units. There was also significant heterogeneity in the outcomes addressed and the variety of methods used in the included studies. We attempted to address this with an analysis of the quality of the studies, but given the wide variability, we were unable to account for all of the differences; unfortunately, this is a standard issue within studies that utilize systematic reviews, as well as similar concepts such as meta-analyses.

In conclusion, the increase in the frequency of limitations of LST in critically ill patients and a change in the nature of limitations from DNR order to withdrawal or withholding of LST suggests a trend toward growing acceptance of limiting treatments in critical illness. The wide variation in withdrawal of care in US ICUs does not seem fully explained by patient variables including preferences, illness type, or changes over time. Factors such as poor prefunctional status, a higher number of comorbid conditions prior to critical illness, and the severity of critical illness are likely important for surrogates and clinicians to consider during goals of care discussions. Further research is needed to explore why patients may receive very different types of care at the end of life depending the institution and ICU in which they receive their care.

Disclosures

The authors have no conflicts of interest to disclose. This work was performed at the Indiana University School of Medicine.

Funding

Financial support for Dr. Torke was provided by a Midcareer Investigator Award in Patient Oriented Research from the National Institute on Aging (K24AG053794). Dr. McPherson was supported by the Indiana University Department of Medicine.

Inspired by the ABIM Foundation's Choosing Wisely campaign, the “Things We Do for No Reason” (TWDFNR) series reviews practices that have become common parts of hospital care but may provide little value to our patients. Practices reviewed in the TWDFNR series do not represent “black and white” conclusions or clinical practice standards but are meant as a starting place for research and active discussions among hospitalists and patients. We invite you to be part of that discussion. https://www.choosingwisely.org/

A 74-year-old man with Alzheimer’s dementia and chronic dysphagia with a history of aspiration pneumonia presents with urinary tract infection, hypovolemia, and hypernatremia. He has been on thickened liquids at home for the past several months. As his overall condition improves with intravenous fluids and antibiotics, he requests to drink thin liquids.

Dysphagia is defined as difficulty or discomfort with feeding or swallowing¹ and is a common clinical problem facing hospitalists. The prevalence of swallowing difficulties is estimated to affect 13 million people in the United States, which is likely to increase as the population ages.² Dysphagia often results in inadequate fluid consumption, resulting in complications such as dehydration.¹ However, the most dreaded complication is pneumonia from aspiration. Aspiration, the entry of material from the oropharynx or the gastrointestinal tract into the larynx and lungs, can be problematic since it is often colonized with pathogens.^3-5 It constitutes 5%-15% of the four and a half million cases of community-acquired pneumonia per year with a mortality rate as high as 21%.^5,6

Dysphagia is a clinical diagnosis, and assessment tools are available to help establish the mechanism and severity.³ For example, the bedside swallow evaluation uses the administration of water by the clinician to the patient to assess for the presence and severity of dysphagia.^1,7 The evaluation is performed by making the patient sit upright at up at 90° and administering either single sips of ≤20 ml of water, consecutive sips with intake up to 100 ml of water, or progressively increasing volumes of water. The clinician then observes for clinical signs of aspiration such as choking or coughing. This evaluation is inexpensive, noninvasive, and time-efficient with a sensitivity as high as 91%, if conducted using the consecutive sips technique.⁷ A video fluoroscopic swallowing exam (VFSE) includes the administration of various barium consistencies that may be helpful in determining the precise mechanism of dysphagia, particularly in the pharyngeal stage of swallowing.^3,8 VFSE is often considered as the standard for dysphagia evaluation, although it is expensive, time-consuming, exposes the patient to radiation, and its translation to functional ability to safely eat and drink is unproven.⁸

Modifying oral liquid intake using thickened liquids has been the cornerstone of clinical practice in treating adults with dysphagia.^4,9-11 Water, a thin liquid with a low viscosity, flows rapidly from the mouth into the oropharynx. The rapid rate may be too fast for the patient’s pharyngeal muscles to compensate, thus allowing aspiration.¹⁰ Thickening the liquids is meant to slow the flow of liquids to allow more time for airway closure, which could potentially reduce the risk of aspiration.^10,11

The strongest evidence for thickened liquids originates from a study based on videofluoroscopy findings. Clave et al. studied patients with stroke or traumatic brain injury, patients with neurodegenerative diseases, and healthy volunteers using videofluoroscopy while swallowing liquid, nectar, and pudding boluses.¹¹ Of the 46 patients with stroke or traumatic brain injury, 21.6% had aspiration of liquid into the airway, but this incidence was reduced to 10.5% and 5.3% when the diet was modified to nectar and pudding, respectively. Of the 46 patients with neurodegenerative diseases, 16.2% had aspiration of liquid into the airway, which was reduced to 8.3% and 2.9% when given nectar and pudding boluses, respectively. Thus, thickened liquids significantly improved the videofluoroscopy results, leading to a presumptive decrease in the rate of respiratory complications. Other authors have reached similar conclusions in different settings and selected patient populations.⁹ These results, although mostly based on imaging findings and in only narrow populations, have been widely extrapolated to routine clinical practice.^1,9,12

Evidence against thickened liquids dates back to 1994, when a comparative effectiveness trial of stroke patients found that family instruction on appropriate compensatory swallowing techniques without the use of thickened liquids carried no increased risk of pneumonia, dehydration, malnutrition, or death when compared with thickened liquids.¹³ Recent evidence has established the risk for harm with thickened liquids. Specifically, patients assigned to thickened liquids in one study had a higher rate of dehydration (6%-2%), fever (4%-2%), and urinary tract infections (6%-3%) than those assigned to thin liquids.¹⁴ This is presumed to be related to poor fluid and nutritional intake resulting from the thickened liquids.^1,9,14

Patients’ perceived quality of life is also lower when on thickened liquids. Studies typically measured this using the validated Swallowing Quality of Life (SWAL-QOL), which is a quality-of-life and quality-of-care outcomes tool designed for patients with oropharyngeal dysphagia.^1,15 One study found that those started on thickened liquids had a significant reduction in their SWAL-QOL score by nearly 14 points (P < .05).¹⁵ Perhaps because of this reduced quality of life, patient compliance has been reported to be as low as 35% at five days.¹⁶

Several systematic reviews support allowing access to free water rather than limiting patients to thickened liquids in the setting of dysphagia. Gillman et al., Kaneoka et al., and Loeb et al. found no statistical difference in the risk of developing aspiration pneumonia in patients granted access to free water when compared to those with thickened liquids.^1,9,12,15 In the meta-analysis of Gillman et al. of 206 patients, there was no significant increase in the odds of having lung complications when allowing patients access to free water in comparison to thickened liquids (odds ratio 1.51, 95% confidence interval 0.2-100.03).¹ The meta-analysis of Kaneoka et al. showed no significant difference in the odds of developing pneumonia in patients with access to free water compared with thickened liquids in a sample of 135 patients (odds ratio 0.82, 95% confidence interval 0.05-13.42).¹² However, the systematic reviews of Gillman et al. and Kaneoka et al. included studies with stringent exclusion criteria, including impaired cognition and mobility limitations, which limits their applicability.^1,12

In patients who have extreme choking with water intake, restricting access to oral water may be reasonable to avoid the physical stress of coughing. Similarly, in end-of-life situations, if coughing is so bothersome to patients or families as to be inconsistent with goals of care, then thickened liquids for comfort measures may be reasonable. Finally, Foley et al. found that combining thickened liquids with texture-modified diets and intensive training sessions with speech-language pathologists focused on swallowing techniques led to a reduced risk for aspiration pneumonia during the first seven days following an acute stroke. Since risk reduction did not persist after seven days, prolonged modification is likely not helpful.⁴

Access to free water is important for hydration, quality of life, and delirium prevention. A collaborative approach with nurses, speech therapists, and caretakers should be employed to focus on strategies to prevent aspiration pneumonia via positioning, oral hygiene, and patient and family education. Postural adjustment with the chin-down posture alters the flow of the bolus during the pharyngeal phase of the swallow.^14,17 This technique has shown superior safety when directly compared with thickened liquids without any difference in aspiration pneumonia rates.¹⁴ In addition, oral hygiene for patients who cannot perform oral care themselves should be implemented to decrease the amount of pathogenic bacteria in secretions.^1,15 Finally, ensuring that patients and families understand the risks and benefits of access to free water is paramount.

Tube feeding (eg, nasogastric and gastric tubes) allows for reliable delivery of enteral nutrition and medications. Tube feeding does not decrease aspiration events compared with oral diets. Moreover, the risk of developing aspiration pneumonia appears to be similar among gastrostomy, nasogastric, and postpyloric feeding tubes.⁵ This approach may be preferable, though, when the dysphagia is the result of a structural abnormality such as stroke deficit, neoplastic changes, or surgical alteration of the larynx.

Free water protocols use an interdisciplinary approach to safely improve access to water for patients with dysphagia. Free water protocols involve screening high-risk populations such as the elderly, confused, or stroke patients with a bedside swallow evaluation. Those with difficulty following directions, who are unable to limit their drinking to manageable-sized sips, or with excessive cough are restricted to supervised water drinking with access to water only between meals (30 minutes after a meal) and with aggressive oral hygiene. Posturing techniques with the chin-down position may be employed. Patients and their families must be educated on protocol implementation and rationale.^1,9,12

Overall, free water protocols have demonstrated an improvement in quality of life, no change in adverse events, and improved water intake. SWAL-QOL scores were significantly improved by nearly three points (P < .05).¹⁵ There was no significant difference in the odds of developing aspiration pneumonia when comparing those on thickened liquids to those with access to free water.^1,9,12 Furthermore, one study by Loeb et al. even found that those allocated to a thickened liquid group were more likely to develop aspiration pneumonia, although this difference was not statistically significant.⁹ Finally, those given access to free water had higher amounts of fluid intake by a mean of 180 ml.¹

• Allow patients with dysphagia access to free water
• Initiate protocols to ensure adequate oral hygiene, patient and family education, and optimization of positioning strategies

Our patient is assessed with a bedside swallow evaluation and has issues with minor coughing. Despite this, he repeatedly requests access to free water, and these requests are upsetting to his family. The risks of potential aspiration are explained to him, and he and his family express understanding. He is given supervised access to water between meals and is encouraged to sit upright and brush his teeth prior to drinking. He continues to improve throughout the hospitalization and at the time of discharge, his sodium level is within normal limits and he is delighted to be drinking regular water.

Patients with dysphagia are often restricted to thickened liquids. This approach does alter the liquid flow throughout the oropharynx and minimal clinical evidence supports this practice as a method to reduce aspiration pneumonia. Given the potential harm and the reduced quality of life, we recommend against thickened liquids in this setting. Taken as a whole, available evidence suggests that protocols to facilitate safe access to water,¹ family information and education,¹³ and positioning techniques¹⁴ are safe, effective, and preferable to thickened liquids.^1,12

Do you think this is a low-value practice? Is this truly a “Thing We Do for No Reason?” Share what you do in your practice and join in the conversation online by retweeting it on Twitter (#TWDFNR) and liking it on Facebook. We invite you to propose ideas for other “Things We Do for No Reason” topics by emailing [email protected].

Disclosures

The authors have nothing to disclose.

Inspired by the ABIM Foundation's Choosing Wisely campaign, the “Things We Do for No Reason” (TWDFNR) series reviews practices that have become common parts of hospital care but may provide little value to our patients. Practices reviewed in the TWDFNR series do not represent “black and white” conclusions or clinical practice standards but are meant as a starting place for research and active discussions among hospitalists and patients. We invite you to be part of that discussion. https://www.choosingwisely.org/

A 74-year-old man with Alzheimer’s dementia and chronic dysphagia with a history of aspiration pneumonia presents with urinary tract infection, hypovolemia, and hypernatremia. He has been on thickened liquids at home for the past several months. As his overall condition improves with intravenous fluids and antibiotics, he requests to drink thin liquids.

Dysphagia is defined as difficulty or discomfort with feeding or swallowing¹ and is a common clinical problem facing hospitalists. The prevalence of swallowing difficulties is estimated to affect 13 million people in the United States, which is likely to increase as the population ages.² Dysphagia often results in inadequate fluid consumption, resulting in complications such as dehydration.¹ However, the most dreaded complication is pneumonia from aspiration. Aspiration, the entry of material from the oropharynx or the gastrointestinal tract into the larynx and lungs, can be problematic since it is often colonized with pathogens.^3-5 It constitutes 5%-15% of the four and a half million cases of community-acquired pneumonia per year with a mortality rate as high as 21%.^5,6

Dysphagia is a clinical diagnosis, and assessment tools are available to help establish the mechanism and severity.³ For example, the bedside swallow evaluation uses the administration of water by the clinician to the patient to assess for the presence and severity of dysphagia.^1,7 The evaluation is performed by making the patient sit upright at up at 90° and administering either single sips of ≤20 ml of water, consecutive sips with intake up to 100 ml of water, or progressively increasing volumes of water. The clinician then observes for clinical signs of aspiration such as choking or coughing. This evaluation is inexpensive, noninvasive, and time-efficient with a sensitivity as high as 91%, if conducted using the consecutive sips technique.⁷ A video fluoroscopic swallowing exam (VFSE) includes the administration of various barium consistencies that may be helpful in determining the precise mechanism of dysphagia, particularly in the pharyngeal stage of swallowing.^3,8 VFSE is often considered as the standard for dysphagia evaluation, although it is expensive, time-consuming, exposes the patient to radiation, and its translation to functional ability to safely eat and drink is unproven.⁸

Modifying oral liquid intake using thickened liquids has been the cornerstone of clinical practice in treating adults with dysphagia.^4,9-11 Water, a thin liquid with a low viscosity, flows rapidly from the mouth into the oropharynx. The rapid rate may be too fast for the patient’s pharyngeal muscles to compensate, thus allowing aspiration.¹⁰ Thickening the liquids is meant to slow the flow of liquids to allow more time for airway closure, which could potentially reduce the risk of aspiration.^10,11

The strongest evidence for thickened liquids originates from a study based on videofluoroscopy findings. Clave et al. studied patients with stroke or traumatic brain injury, patients with neurodegenerative diseases, and healthy volunteers using videofluoroscopy while swallowing liquid, nectar, and pudding boluses.¹¹ Of the 46 patients with stroke or traumatic brain injury, 21.6% had aspiration of liquid into the airway, but this incidence was reduced to 10.5% and 5.3% when the diet was modified to nectar and pudding, respectively. Of the 46 patients with neurodegenerative diseases, 16.2% had aspiration of liquid into the airway, which was reduced to 8.3% and 2.9% when given nectar and pudding boluses, respectively. Thus, thickened liquids significantly improved the videofluoroscopy results, leading to a presumptive decrease in the rate of respiratory complications. Other authors have reached similar conclusions in different settings and selected patient populations.⁹ These results, although mostly based on imaging findings and in only narrow populations, have been widely extrapolated to routine clinical practice.^1,9,12

Evidence against thickened liquids dates back to 1994, when a comparative effectiveness trial of stroke patients found that family instruction on appropriate compensatory swallowing techniques without the use of thickened liquids carried no increased risk of pneumonia, dehydration, malnutrition, or death when compared with thickened liquids.¹³ Recent evidence has established the risk for harm with thickened liquids. Specifically, patients assigned to thickened liquids in one study had a higher rate of dehydration (6%-2%), fever (4%-2%), and urinary tract infections (6%-3%) than those assigned to thin liquids.¹⁴ This is presumed to be related to poor fluid and nutritional intake resulting from the thickened liquids.^1,9,14

Patients’ perceived quality of life is also lower when on thickened liquids. Studies typically measured this using the validated Swallowing Quality of Life (SWAL-QOL), which is a quality-of-life and quality-of-care outcomes tool designed for patients with oropharyngeal dysphagia.^1,15 One study found that those started on thickened liquids had a significant reduction in their SWAL-QOL score by nearly 14 points (P < .05).¹⁵ Perhaps because of this reduced quality of life, patient compliance has been reported to be as low as 35% at five days.¹⁶

Several systematic reviews support allowing access to free water rather than limiting patients to thickened liquids in the setting of dysphagia. Gillman et al., Kaneoka et al., and Loeb et al. found no statistical difference in the risk of developing aspiration pneumonia in patients granted access to free water when compared to those with thickened liquids.^1,9,12,15 In the meta-analysis of Gillman et al. of 206 patients, there was no significant increase in the odds of having lung complications when allowing patients access to free water in comparison to thickened liquids (odds ratio 1.51, 95% confidence interval 0.2-100.03).¹ The meta-analysis of Kaneoka et al. showed no significant difference in the odds of developing pneumonia in patients with access to free water compared with thickened liquids in a sample of 135 patients (odds ratio 0.82, 95% confidence interval 0.05-13.42).¹² However, the systematic reviews of Gillman et al. and Kaneoka et al. included studies with stringent exclusion criteria, including impaired cognition and mobility limitations, which limits their applicability.^1,12

In patients who have extreme choking with water intake, restricting access to oral water may be reasonable to avoid the physical stress of coughing. Similarly, in end-of-life situations, if coughing is so bothersome to patients or families as to be inconsistent with goals of care, then thickened liquids for comfort measures may be reasonable. Finally, Foley et al. found that combining thickened liquids with texture-modified diets and intensive training sessions with speech-language pathologists focused on swallowing techniques led to a reduced risk for aspiration pneumonia during the first seven days following an acute stroke. Since risk reduction did not persist after seven days, prolonged modification is likely not helpful.⁴

Access to free water is important for hydration, quality of life, and delirium prevention. A collaborative approach with nurses, speech therapists, and caretakers should be employed to focus on strategies to prevent aspiration pneumonia via positioning, oral hygiene, and patient and family education. Postural adjustment with the chin-down posture alters the flow of the bolus during the pharyngeal phase of the swallow.^14,17 This technique has shown superior safety when directly compared with thickened liquids without any difference in aspiration pneumonia rates.¹⁴ In addition, oral hygiene for patients who cannot perform oral care themselves should be implemented to decrease the amount of pathogenic bacteria in secretions.^1,15 Finally, ensuring that patients and families understand the risks and benefits of access to free water is paramount.

Tube feeding (eg, nasogastric and gastric tubes) allows for reliable delivery of enteral nutrition and medications. Tube feeding does not decrease aspiration events compared with oral diets. Moreover, the risk of developing aspiration pneumonia appears to be similar among gastrostomy, nasogastric, and postpyloric feeding tubes.⁵ This approach may be preferable, though, when the dysphagia is the result of a structural abnormality such as stroke deficit, neoplastic changes, or surgical alteration of the larynx.

Free water protocols use an interdisciplinary approach to safely improve access to water for patients with dysphagia. Free water protocols involve screening high-risk populations such as the elderly, confused, or stroke patients with a bedside swallow evaluation. Those with difficulty following directions, who are unable to limit their drinking to manageable-sized sips, or with excessive cough are restricted to supervised water drinking with access to water only between meals (30 minutes after a meal) and with aggressive oral hygiene. Posturing techniques with the chin-down position may be employed. Patients and their families must be educated on protocol implementation and rationale.^1,9,12

Overall, free water protocols have demonstrated an improvement in quality of life, no change in adverse events, and improved water intake. SWAL-QOL scores were significantly improved by nearly three points (P < .05).¹⁵ There was no significant difference in the odds of developing aspiration pneumonia when comparing those on thickened liquids to those with access to free water.^1,9,12 Furthermore, one study by Loeb et al. even found that those allocated to a thickened liquid group were more likely to develop aspiration pneumonia, although this difference was not statistically significant.⁹ Finally, those given access to free water had higher amounts of fluid intake by a mean of 180 ml.¹

• Allow patients with dysphagia access to free water
• Initiate protocols to ensure adequate oral hygiene, patient and family education, and optimization of positioning strategies

Our patient is assessed with a bedside swallow evaluation and has issues with minor coughing. Despite this, he repeatedly requests access to free water, and these requests are upsetting to his family. The risks of potential aspiration are explained to him, and he and his family express understanding. He is given supervised access to water between meals and is encouraged to sit upright and brush his teeth prior to drinking. He continues to improve throughout the hospitalization and at the time of discharge, his sodium level is within normal limits and he is delighted to be drinking regular water.

Patients with dysphagia are often restricted to thickened liquids. This approach does alter the liquid flow throughout the oropharynx and minimal clinical evidence supports this practice as a method to reduce aspiration pneumonia. Given the potential harm and the reduced quality of life, we recommend against thickened liquids in this setting. Taken as a whole, available evidence suggests that protocols to facilitate safe access to water,¹ family information and education,¹³ and positioning techniques¹⁴ are safe, effective, and preferable to thickened liquids.^1,12

Do you think this is a low-value practice? Is this truly a “Thing We Do for No Reason?” Share what you do in your practice and join in the conversation online by retweeting it on Twitter (#TWDFNR) and liking it on Facebook. We invite you to propose ideas for other “Things We Do for No Reason” topics by emailing [email protected].

Disclosures

The authors have nothing to disclose.

Inspired by the ABIM Foundation's Choosing Wisely campaign, the “Things We Do for No Reason” (TWDFNR) series reviews practices that have become common parts of hospital care but may provide little value to our patients. Practices reviewed in the TWDFNR series do not represent “black and white” conclusions or clinical practice standards but are meant as a starting place for research and active discussions among hospitalists and patients. We invite you to be part of that discussion. https://www.choosingwisely.org/

A 74-year-old man with Alzheimer’s dementia and chronic dysphagia with a history of aspiration pneumonia presents with urinary tract infection, hypovolemia, and hypernatremia. He has been on thickened liquids at home for the past several months. As his overall condition improves with intravenous fluids and antibiotics, he requests to drink thin liquids.

Dysphagia is defined as difficulty or discomfort with feeding or swallowing¹ and is a common clinical problem facing hospitalists. The prevalence of swallowing difficulties is estimated to affect 13 million people in the United States, which is likely to increase as the population ages.² Dysphagia often results in inadequate fluid consumption, resulting in complications such as dehydration.¹ However, the most dreaded complication is pneumonia from aspiration. Aspiration, the entry of material from the oropharynx or the gastrointestinal tract into the larynx and lungs, can be problematic since it is often colonized with pathogens.^3-5 It constitutes 5%-15% of the four and a half million cases of community-acquired pneumonia per year with a mortality rate as high as 21%.^5,6

Dysphagia is a clinical diagnosis, and assessment tools are available to help establish the mechanism and severity.³ For example, the bedside swallow evaluation uses the administration of water by the clinician to the patient to assess for the presence and severity of dysphagia.^1,7 The evaluation is performed by making the patient sit upright at up at 90° and administering either single sips of ≤20 ml of water, consecutive sips with intake up to 100 ml of water, or progressively increasing volumes of water. The clinician then observes for clinical signs of aspiration such as choking or coughing. This evaluation is inexpensive, noninvasive, and time-efficient with a sensitivity as high as 91%, if conducted using the consecutive sips technique.⁷ A video fluoroscopic swallowing exam (VFSE) includes the administration of various barium consistencies that may be helpful in determining the precise mechanism of dysphagia, particularly in the pharyngeal stage of swallowing.^3,8 VFSE is often considered as the standard for dysphagia evaluation, although it is expensive, time-consuming, exposes the patient to radiation, and its translation to functional ability to safely eat and drink is unproven.⁸

Modifying oral liquid intake using thickened liquids has been the cornerstone of clinical practice in treating adults with dysphagia.^4,9-11 Water, a thin liquid with a low viscosity, flows rapidly from the mouth into the oropharynx. The rapid rate may be too fast for the patient’s pharyngeal muscles to compensate, thus allowing aspiration.¹⁰ Thickening the liquids is meant to slow the flow of liquids to allow more time for airway closure, which could potentially reduce the risk of aspiration.^10,11

The strongest evidence for thickened liquids originates from a study based on videofluoroscopy findings. Clave et al. studied patients with stroke or traumatic brain injury, patients with neurodegenerative diseases, and healthy volunteers using videofluoroscopy while swallowing liquid, nectar, and pudding boluses.¹¹ Of the 46 patients with stroke or traumatic brain injury, 21.6% had aspiration of liquid into the airway, but this incidence was reduced to 10.5% and 5.3% when the diet was modified to nectar and pudding, respectively. Of the 46 patients with neurodegenerative diseases, 16.2% had aspiration of liquid into the airway, which was reduced to 8.3% and 2.9% when given nectar and pudding boluses, respectively. Thus, thickened liquids significantly improved the videofluoroscopy results, leading to a presumptive decrease in the rate of respiratory complications. Other authors have reached similar conclusions in different settings and selected patient populations.⁹ These results, although mostly based on imaging findings and in only narrow populations, have been widely extrapolated to routine clinical practice.^1,9,12

Evidence against thickened liquids dates back to 1994, when a comparative effectiveness trial of stroke patients found that family instruction on appropriate compensatory swallowing techniques without the use of thickened liquids carried no increased risk of pneumonia, dehydration, malnutrition, or death when compared with thickened liquids.¹³ Recent evidence has established the risk for harm with thickened liquids. Specifically, patients assigned to thickened liquids in one study had a higher rate of dehydration (6%-2%), fever (4%-2%), and urinary tract infections (6%-3%) than those assigned to thin liquids.¹⁴ This is presumed to be related to poor fluid and nutritional intake resulting from the thickened liquids.^1,9,14

Patients’ perceived quality of life is also lower when on thickened liquids. Studies typically measured this using the validated Swallowing Quality of Life (SWAL-QOL), which is a quality-of-life and quality-of-care outcomes tool designed for patients with oropharyngeal dysphagia.^1,15 One study found that those started on thickened liquids had a significant reduction in their SWAL-QOL score by nearly 14 points (P < .05).¹⁵ Perhaps because of this reduced quality of life, patient compliance has been reported to be as low as 35% at five days.¹⁶

Several systematic reviews support allowing access to free water rather than limiting patients to thickened liquids in the setting of dysphagia. Gillman et al., Kaneoka et al., and Loeb et al. found no statistical difference in the risk of developing aspiration pneumonia in patients granted access to free water when compared to those with thickened liquids.^1,9,12,15 In the meta-analysis of Gillman et al. of 206 patients, there was no significant increase in the odds of having lung complications when allowing patients access to free water in comparison to thickened liquids (odds ratio 1.51, 95% confidence interval 0.2-100.03).¹ The meta-analysis of Kaneoka et al. showed no significant difference in the odds of developing pneumonia in patients with access to free water compared with thickened liquids in a sample of 135 patients (odds ratio 0.82, 95% confidence interval 0.05-13.42).¹² However, the systematic reviews of Gillman et al. and Kaneoka et al. included studies with stringent exclusion criteria, including impaired cognition and mobility limitations, which limits their applicability.^1,12

In patients who have extreme choking with water intake, restricting access to oral water may be reasonable to avoid the physical stress of coughing. Similarly, in end-of-life situations, if coughing is so bothersome to patients or families as to be inconsistent with goals of care, then thickened liquids for comfort measures may be reasonable. Finally, Foley et al. found that combining thickened liquids with texture-modified diets and intensive training sessions with speech-language pathologists focused on swallowing techniques led to a reduced risk for aspiration pneumonia during the first seven days following an acute stroke. Since risk reduction did not persist after seven days, prolonged modification is likely not helpful.⁴

Access to free water is important for hydration, quality of life, and delirium prevention. A collaborative approach with nurses, speech therapists, and caretakers should be employed to focus on strategies to prevent aspiration pneumonia via positioning, oral hygiene, and patient and family education. Postural adjustment with the chin-down posture alters the flow of the bolus during the pharyngeal phase of the swallow.^14,17 This technique has shown superior safety when directly compared with thickened liquids without any difference in aspiration pneumonia rates.¹⁴ In addition, oral hygiene for patients who cannot perform oral care themselves should be implemented to decrease the amount of pathogenic bacteria in secretions.^1,15 Finally, ensuring that patients and families understand the risks and benefits of access to free water is paramount.

Tube feeding (eg, nasogastric and gastric tubes) allows for reliable delivery of enteral nutrition and medications. Tube feeding does not decrease aspiration events compared with oral diets. Moreover, the risk of developing aspiration pneumonia appears to be similar among gastrostomy, nasogastric, and postpyloric feeding tubes.⁵ This approach may be preferable, though, when the dysphagia is the result of a structural abnormality such as stroke deficit, neoplastic changes, or surgical alteration of the larynx.

Free water protocols use an interdisciplinary approach to safely improve access to water for patients with dysphagia. Free water protocols involve screening high-risk populations such as the elderly, confused, or stroke patients with a bedside swallow evaluation. Those with difficulty following directions, who are unable to limit their drinking to manageable-sized sips, or with excessive cough are restricted to supervised water drinking with access to water only between meals (30 minutes after a meal) and with aggressive oral hygiene. Posturing techniques with the chin-down position may be employed. Patients and their families must be educated on protocol implementation and rationale.^1,9,12

Overall, free water protocols have demonstrated an improvement in quality of life, no change in adverse events, and improved water intake. SWAL-QOL scores were significantly improved by nearly three points (P < .05).¹⁵ There was no significant difference in the odds of developing aspiration pneumonia when comparing those on thickened liquids to those with access to free water.^1,9,12 Furthermore, one study by Loeb et al. even found that those allocated to a thickened liquid group were more likely to develop aspiration pneumonia, although this difference was not statistically significant.⁹ Finally, those given access to free water had higher amounts of fluid intake by a mean of 180 ml.¹

• Allow patients with dysphagia access to free water
• Initiate protocols to ensure adequate oral hygiene, patient and family education, and optimization of positioning strategies

Our patient is assessed with a bedside swallow evaluation and has issues with minor coughing. Despite this, he repeatedly requests access to free water, and these requests are upsetting to his family. The risks of potential aspiration are explained to him, and he and his family express understanding. He is given supervised access to water between meals and is encouraged to sit upright and brush his teeth prior to drinking. He continues to improve throughout the hospitalization and at the time of discharge, his sodium level is within normal limits and he is delighted to be drinking regular water.

Patients with dysphagia are often restricted to thickened liquids. This approach does alter the liquid flow throughout the oropharynx and minimal clinical evidence supports this practice as a method to reduce aspiration pneumonia. Given the potential harm and the reduced quality of life, we recommend against thickened liquids in this setting. Taken as a whole, available evidence suggests that protocols to facilitate safe access to water,¹ family information and education,¹³ and positioning techniques¹⁴ are safe, effective, and preferable to thickened liquids.^1,12

Do you think this is a low-value practice? Is this truly a “Thing We Do for No Reason?” Share what you do in your practice and join in the conversation online by retweeting it on Twitter (#TWDFNR) and liking it on Facebook. We invite you to propose ideas for other “Things We Do for No Reason” topics by emailing [email protected].

Disclosures

The authors have nothing to disclose.

We spoke with medical social worker Laura Shea, MSW, LICSW on her role at our tertiary care hospital. Laura’s reflections on the struggles and rewards of her job may resonate with those of us who search for balance and meaning in work.

Laura, tell us about yourself. What made you want to be a social worker?

I couldn’t really picture doing anything else. I got a degree in psychology and loved counseling. Social work was a natural fit because of the social justice component and the look into larger systems. I knew I had the skill set for this, and for those most marginalized, to be a supportive person for someone who doesn’t have that.

I also have a family member with major mental illness and chronic suicidality who I supported for a very long time. In many ways, I was a personal social worker advocating on their behalf while growing up. I remember being in high school when they overdosed, and going to the ER in the middle of the night. The next morning, I was back at school. I was a total do-gooder—President of the student council and on top of my grades. I tried dealing with this while keeping up the appearance that everything was ok, even though it wasn’t.

As I got older, there were middle-of-the-night phone calls professing suicidality which were so painful. I learned a lot about compartmentalizing and resiliency. It has given me an incredible amount of empathy for family members of patients. I have learned that it’s not always simple, and decisions aren’t easy, and solutions are complicated and can feel incomplete. We often hear, “Why hasn’t the family stepped in?” Well these issues are hard for families too, I know from firsthand experience.

At the end of the day, as challenging as the work is, I get something from it. I feel honored to bear witness to some of people’s darkest moments and also some of the most beautiful moments—the joys of coming out the other side of their process and journey.

How much of your personal story do you reveal to your patients?

I rarely do. However, to some families that are particularly devastated, I do share some of my family story. I try to affirm their challenge and acknowledge that family and friends can’t always “solve this.”

We have a culture that reveres going above and beyond, however I really honor those family members who can set boundaries. Sometimes caregivers need space, that doesn’t make you a bad person. It’s actually brave and really hard to do. You can’t give from an empty well.

Laura, tell us about your typical day.

Well, it begins with responding to e-mails. Then I meet with patients and obtain collateral to prep for multidisciplinary rounds (with physicians, RNs, case managers). I usually consult on 20-30 patients a day. In the afternoon, it varies -- maybe three patients are leaving that may need my help with things like providing substance use information or shelter resources. Typically, I’ll have a few complicated long-term patients, who may have challenging family dynamics, ongoing goals of care discussions, or behavioral difficulties. These patients keep me just as busy, it’s not quite as time sensitive but I have to keep chipping away at the work.

Seems like a busy day. Do you get a break at all?

When possible, I take a walk in the woods behind the hospital on my lunch break. There’s a beautiful path, it’s an important part of my day -- getting outside and taking a step back. I bring my pager, so I am still connected.

I used to feel like I didn’t have time to take a break, and I would work through lunch. But now I find if I take a break, I am more productive the rest of the day because it makes me more mindful. It quiets me a little, gives me perspective on the stress and stressors of working in the hospital and allows me to better connect to my job and others around me.

What does a successful day look like?

Well, one involved a homeless gentleman and a search for his family. He was in his 40s, though he looked much older, and recently had been assaulted at a shelter. He presented to either the ER or was admitted to various hospitals 14 times over the past month – typically for intoxication and hypothermia. He kept saying “I just need to find my brother” though no one was taking this request too seriously. We spent a lot of time looking for his brother with the Office of Public Guardian’s help, and we actually found him! The patient hadn’t seen his brother in four years and as it turns out was searching for him too. The brother thought the patient had passed away. With his brother’s support, the patient is now housed, going to alcohol treatment, reunited with his family, and taking his medications. His whole life changed. So that was amazing, and a reminder of how rewarding this job can be.

What is most challenging about your work?

The biggest challenge is grappling with the limitations of the system, and discharging someone to the community when the community has limited resources for these patients.

Though it’s not just the limitation of resources, some patients have been through the system so many times that as a coping mechanism and to protect themselves they do everything possible to push you away. They have walls firmly up, because of prior negative experiences with providers. I am not fazed by being yelled at, but it’s hard trying to connect with someone who has learned not to let you in. These are often the patients that need the support the most, and yet I want to respect their ability to have control or to say no. It is a tough balance.

What’s fun about your job?

I love meeting new people. I met a woman a few weeks ago who was talking about being a hippie in the ‘60s in San Francisco, and how great it was and how soft millennials are. She actually put meth in her coffee because she needed a pick-me-up to clean her house. You can’t make this stuff up! It’s just really fascinating how people live their lives, and to have a window into their world and perspective is a privilege.

Do you take work home with you or do you disconnect?

I try to disconnect, however there are days when something sticks with you and you really worry and wonder about a patient. As I mentioned, you can’t give from an empty well—so I try to acknowledge this. I find that trying to have a rich life outside of work is an important part of self-care as well. Social work is a big part of my identity but it’s not entirely who I am. I focus on friends, family, travel, yoga, and things that sustain me. I can’t do my job effectively if I am not taking a step back regularly.

What advice do you have for other providers and for patients?

The hospital is so overwhelming for our patients, more so than some providers realize. I could be in the room with a patient for 45 minutes and six different providers may come in. I try to maintain that this is the patient’s bedroom I’m walking into. It’s a private, and a sacred space for them. That’s where they sleep. This is where they are trying to recover and grapple with what brought them into the hospital.

Laura, thank you so much for telling us about your work. Anything else you’d like to share with us?

Some days I’ll go home completely exhausted and wiped out, and at first, I don’t feel like I did a single solitary thing. Some of the things that I’m trying to help people work through ...it never occurred to me that someone could, for whatever reason, find themselves in such challenging situations. I don’t have a magic wand to provide someone with housing or sobriety, but maybe in that moment I can begin to make a connection. When I just listen, I am beginning to build relationships – which for some patients is something they haven’t had in a long time. It’s in these moments of being present, without an agenda, walking with them in their challenges, that I feel most connected to the work.

Thanks, Laura.

We spoke with medical social worker Laura Shea, MSW, LICSW on her role at our tertiary care hospital. Laura’s reflections on the struggles and rewards of her job may resonate with those of us who search for balance and meaning in work.

Laura, tell us about yourself. What made you want to be a social worker?

I couldn’t really picture doing anything else. I got a degree in psychology and loved counseling. Social work was a natural fit because of the social justice component and the look into larger systems. I knew I had the skill set for this, and for those most marginalized, to be a supportive person for someone who doesn’t have that.

I also have a family member with major mental illness and chronic suicidality who I supported for a very long time. In many ways, I was a personal social worker advocating on their behalf while growing up. I remember being in high school when they overdosed, and going to the ER in the middle of the night. The next morning, I was back at school. I was a total do-gooder—President of the student council and on top of my grades. I tried dealing with this while keeping up the appearance that everything was ok, even though it wasn’t.

As I got older, there were middle-of-the-night phone calls professing suicidality which were so painful. I learned a lot about compartmentalizing and resiliency. It has given me an incredible amount of empathy for family members of patients. I have learned that it’s not always simple, and decisions aren’t easy, and solutions are complicated and can feel incomplete. We often hear, “Why hasn’t the family stepped in?” Well these issues are hard for families too, I know from firsthand experience.

At the end of the day, as challenging as the work is, I get something from it. I feel honored to bear witness to some of people’s darkest moments and also some of the most beautiful moments—the joys of coming out the other side of their process and journey.

How much of your personal story do you reveal to your patients?

I rarely do. However, to some families that are particularly devastated, I do share some of my family story. I try to affirm their challenge and acknowledge that family and friends can’t always “solve this.”

We have a culture that reveres going above and beyond, however I really honor those family members who can set boundaries. Sometimes caregivers need space, that doesn’t make you a bad person. It’s actually brave and really hard to do. You can’t give from an empty well.

Laura, tell us about your typical day.

Well, it begins with responding to e-mails. Then I meet with patients and obtain collateral to prep for multidisciplinary rounds (with physicians, RNs, case managers). I usually consult on 20-30 patients a day. In the afternoon, it varies -- maybe three patients are leaving that may need my help with things like providing substance use information or shelter resources. Typically, I’ll have a few complicated long-term patients, who may have challenging family dynamics, ongoing goals of care discussions, or behavioral difficulties. These patients keep me just as busy, it’s not quite as time sensitive but I have to keep chipping away at the work.

Seems like a busy day. Do you get a break at all?

When possible, I take a walk in the woods behind the hospital on my lunch break. There’s a beautiful path, it’s an important part of my day -- getting outside and taking a step back. I bring my pager, so I am still connected.

I used to feel like I didn’t have time to take a break, and I would work through lunch. But now I find if I take a break, I am more productive the rest of the day because it makes me more mindful. It quiets me a little, gives me perspective on the stress and stressors of working in the hospital and allows me to better connect to my job and others around me.

What does a successful day look like?

Well, one involved a homeless gentleman and a search for his family. He was in his 40s, though he looked much older, and recently had been assaulted at a shelter. He presented to either the ER or was admitted to various hospitals 14 times over the past month – typically for intoxication and hypothermia. He kept saying “I just need to find my brother” though no one was taking this request too seriously. We spent a lot of time looking for his brother with the Office of Public Guardian’s help, and we actually found him! The patient hadn’t seen his brother in four years and as it turns out was searching for him too. The brother thought the patient had passed away. With his brother’s support, the patient is now housed, going to alcohol treatment, reunited with his family, and taking his medications. His whole life changed. So that was amazing, and a reminder of how rewarding this job can be.

What is most challenging about your work?

The biggest challenge is grappling with the limitations of the system, and discharging someone to the community when the community has limited resources for these patients.

Though it’s not just the limitation of resources, some patients have been through the system so many times that as a coping mechanism and to protect themselves they do everything possible to push you away. They have walls firmly up, because of prior negative experiences with providers. I am not fazed by being yelled at, but it’s hard trying to connect with someone who has learned not to let you in. These are often the patients that need the support the most, and yet I want to respect their ability to have control or to say no. It is a tough balance.

What’s fun about your job?

I love meeting new people. I met a woman a few weeks ago who was talking about being a hippie in the ‘60s in San Francisco, and how great it was and how soft millennials are. She actually put meth in her coffee because she needed a pick-me-up to clean her house. You can’t make this stuff up! It’s just really fascinating how people live their lives, and to have a window into their world and perspective is a privilege.

Do you take work home with you or do you disconnect?

I try to disconnect, however there are days when something sticks with you and you really worry and wonder about a patient. As I mentioned, you can’t give from an empty well—so I try to acknowledge this. I find that trying to have a rich life outside of work is an important part of self-care as well. Social work is a big part of my identity but it’s not entirely who I am. I focus on friends, family, travel, yoga, and things that sustain me. I can’t do my job effectively if I am not taking a step back regularly.

What advice do you have for other providers and for patients?

The hospital is so overwhelming for our patients, more so than some providers realize. I could be in the room with a patient for 45 minutes and six different providers may come in. I try to maintain that this is the patient’s bedroom I’m walking into. It’s a private, and a sacred space for them. That’s where they sleep. This is where they are trying to recover and grapple with what brought them into the hospital.

Laura, thank you so much for telling us about your work. Anything else you’d like to share with us?

Some days I’ll go home completely exhausted and wiped out, and at first, I don’t feel like I did a single solitary thing. Some of the things that I’m trying to help people work through ...it never occurred to me that someone could, for whatever reason, find themselves in such challenging situations. I don’t have a magic wand to provide someone with housing or sobriety, but maybe in that moment I can begin to make a connection. When I just listen, I am beginning to build relationships – which for some patients is something they haven’t had in a long time. It’s in these moments of being present, without an agenda, walking with them in their challenges, that I feel most connected to the work.

Thanks, Laura.

We spoke with medical social worker Laura Shea, MSW, LICSW on her role at our tertiary care hospital. Laura’s reflections on the struggles and rewards of her job may resonate with those of us who search for balance and meaning in work.

Laura, tell us about yourself. What made you want to be a social worker?

I couldn’t really picture doing anything else. I got a degree in psychology and loved counseling. Social work was a natural fit because of the social justice component and the look into larger systems. I knew I had the skill set for this, and for those most marginalized, to be a supportive person for someone who doesn’t have that.

I also have a family member with major mental illness and chronic suicidality who I supported for a very long time. In many ways, I was a personal social worker advocating on their behalf while growing up. I remember being in high school when they overdosed, and going to the ER in the middle of the night. The next morning, I was back at school. I was a total do-gooder—President of the student council and on top of my grades. I tried dealing with this while keeping up the appearance that everything was ok, even though it wasn’t.

As I got older, there were middle-of-the-night phone calls professing suicidality which were so painful. I learned a lot about compartmentalizing and resiliency. It has given me an incredible amount of empathy for family members of patients. I have learned that it’s not always simple, and decisions aren’t easy, and solutions are complicated and can feel incomplete. We often hear, “Why hasn’t the family stepped in?” Well these issues are hard for families too, I know from firsthand experience.

At the end of the day, as challenging as the work is, I get something from it. I feel honored to bear witness to some of people’s darkest moments and also some of the most beautiful moments—the joys of coming out the other side of their process and journey.

How much of your personal story do you reveal to your patients?

I rarely do. However, to some families that are particularly devastated, I do share some of my family story. I try to affirm their challenge and acknowledge that family and friends can’t always “solve this.”

We have a culture that reveres going above and beyond, however I really honor those family members who can set boundaries. Sometimes caregivers need space, that doesn’t make you a bad person. It’s actually brave and really hard to do. You can’t give from an empty well.

Laura, tell us about your typical day.

Well, it begins with responding to e-mails. Then I meet with patients and obtain collateral to prep for multidisciplinary rounds (with physicians, RNs, case managers). I usually consult on 20-30 patients a day. In the afternoon, it varies -- maybe three patients are leaving that may need my help with things like providing substance use information or shelter resources. Typically, I’ll have a few complicated long-term patients, who may have challenging family dynamics, ongoing goals of care discussions, or behavioral difficulties. These patients keep me just as busy, it’s not quite as time sensitive but I have to keep chipping away at the work.

Seems like a busy day. Do you get a break at all?

When possible, I take a walk in the woods behind the hospital on my lunch break. There’s a beautiful path, it’s an important part of my day -- getting outside and taking a step back. I bring my pager, so I am still connected.

I used to feel like I didn’t have time to take a break, and I would work through lunch. But now I find if I take a break, I am more productive the rest of the day because it makes me more mindful. It quiets me a little, gives me perspective on the stress and stressors of working in the hospital and allows me to better connect to my job and others around me.

What does a successful day look like?

Well, one involved a homeless gentleman and a search for his family. He was in his 40s, though he looked much older, and recently had been assaulted at a shelter. He presented to either the ER or was admitted to various hospitals 14 times over the past month – typically for intoxication and hypothermia. He kept saying “I just need to find my brother” though no one was taking this request too seriously. We spent a lot of time looking for his brother with the Office of Public Guardian’s help, and we actually found him! The patient hadn’t seen his brother in four years and as it turns out was searching for him too. The brother thought the patient had passed away. With his brother’s support, the patient is now housed, going to alcohol treatment, reunited with his family, and taking his medications. His whole life changed. So that was amazing, and a reminder of how rewarding this job can be.

What is most challenging about your work?

The biggest challenge is grappling with the limitations of the system, and discharging someone to the community when the community has limited resources for these patients.

Though it’s not just the limitation of resources, some patients have been through the system so many times that as a coping mechanism and to protect themselves they do everything possible to push you away. They have walls firmly up, because of prior negative experiences with providers. I am not fazed by being yelled at, but it’s hard trying to connect with someone who has learned not to let you in. These are often the patients that need the support the most, and yet I want to respect their ability to have control or to say no. It is a tough balance.

What’s fun about your job?

I love meeting new people. I met a woman a few weeks ago who was talking about being a hippie in the ‘60s in San Francisco, and how great it was and how soft millennials are. She actually put meth in her coffee because she needed a pick-me-up to clean her house. You can’t make this stuff up! It’s just really fascinating how people live their lives, and to have a window into their world and perspective is a privilege.

Do you take work home with you or do you disconnect?

I try to disconnect, however there are days when something sticks with you and you really worry and wonder about a patient. As I mentioned, you can’t give from an empty well—so I try to acknowledge this. I find that trying to have a rich life outside of work is an important part of self-care as well. Social work is a big part of my identity but it’s not entirely who I am. I focus on friends, family, travel, yoga, and things that sustain me. I can’t do my job effectively if I am not taking a step back regularly.

What advice do you have for other providers and for patients?

The hospital is so overwhelming for our patients, more so than some providers realize. I could be in the room with a patient for 45 minutes and six different providers may come in. I try to maintain that this is the patient’s bedroom I’m walking into. It’s a private, and a sacred space for them. That’s where they sleep. This is where they are trying to recover and grapple with what brought them into the hospital.

Laura, thank you so much for telling us about your work. Anything else you’d like to share with us?

Some days I’ll go home completely exhausted and wiped out, and at first, I don’t feel like I did a single solitary thing. Some of the things that I’m trying to help people work through ...it never occurred to me that someone could, for whatever reason, find themselves in such challenging situations. I don’t have a magic wand to provide someone with housing or sobriety, but maybe in that moment I can begin to make a connection. When I just listen, I am beginning to build relationships – which for some patients is something they haven’t had in a long time. It’s in these moments of being present, without an agenda, walking with them in their challenges, that I feel most connected to the work.

Thanks, Laura.

Unplanned hospital admissions and readmissions have become a major focus of efforts to improve the value of healthcare given that these potentially preventable events exert substantial burden on patients, caregivers, health systems, and the economy.¹ The percentage of patients who are rehospitalized within 30 days have decreased from 20%-21% at the start of the Accountable Care Act and readmission penalties to approximately 18%.^2-5 Rehospitalization rates are 33% at 90 days and approach 40% at six months.^6,7 Readmissions cost Medicare more than $26 billion annually,⁴ with one in five Medicare beneficiaries readmitted within 30 days of hospital discharge.⁸ Centers for Medicare and Medicaid Services and other payers use condition-specific and all-cause 30-day unplanned readmission rates and potentially preventable admissions among patients with complex or multiple comorbidities for public reporting, value-based purchasing, and performance-based reimbursement.^9,10 Consequently, medical groups and hospitals have begun to place an increasing emphasis on improving the transitions of care following hospitalization with the goal of reducing unplanned readmissions.¹¹ Care transitions programs have been shown to decrease readmission rates, mortality, and emergency department (ED) visits.¹²

Care transitions programs vary greatly in their scope of intervention and target groups, as well as in their efficacy in reducing readmissions.^13,14 The Mayo Clinic Care Transition Program, hereafter referred to as CTP, was launched in 2011. This program was modeled after other successful programs and involves home visits by a nurse practitioner (NP) and telephonic support and triage provided by a registered nurse (RN). It is offered to high-risk community-dwelling patients during their hospitalization and begins within a week of hospital discharge.

Although the CTP reduces 30-day readmissions from 20% to 17%,⁷ it is a highly resource-intensive, multimodal, multidisciplinary program. Moreover, whether some components of the CTP are more critical than others remains unknown. Prior studies that examined the individual components of successful CTPs have suggested that a multipronged approach that includes close patient and caregiver support is most predictive of program efficacy.¹³ Long-term program sustainability would benefit from optimization of the most critical components of the program while reducing or eliminating resource-intensive factors that have negligible effects on program success. We therefore examined our CTP to identify whether and which program components are most critical for preventing 30-day readmissions and whether any patient characteristics contribute risk within this complex population.

Study Design and Setting

This study is a retrospective cohort study of patients who were enrolled in the care transitions program of Mayo Clinic Rochester during the period January 1, 2010 to June 30, 2013. Patient demographic and clinical data were obtained from electronic health records (EHR), and information regarding CTP processes and interventions was obtained from a prospectively maintained program database. The study complied with the principles of the Declaration of Helsinki and was approved by the Mayo Clinic Institutional Review Board.

Objectives

The study aimed to describe the performance and utilization of a multidisciplinary care transitions program that has been successful in reducing readmissions for high-risk patients. The study also sought to identify patient and/or program factors associated with failure to prevent readmission within 30 days of program enrollment.

Population

Patients who were enrolled in the CTP following hospital discharge and seen for a posthospital in-home visit prior to hospital readmission (for those readmitted) were included. Patients discharged to a skilled nursing facility were excluded. Patients were eligible for CTP enrollment if they were hospitalized for any cause, community dwelling (including assisted living) prior to hospitalization, and ≥60 years old with an Elder Risk Assessment (ERA) score ≥16.⁷ The ERA incorporates information regarding previous hospital days, age, and comorbid health burden and has been shown to predict 30-day readmissions, mortality, and critical illness (

Intervention

Detailed descriptions of the CTP have been previously published.^7,17 Patients meeting enrollment criteria are enrolled into the CTP by a RN prior to or immediately after hospital discharge. The patient is then seen at home within one to five business days of discharge and again the following week by a NP who performs medication reconciliation; chronic illness management; and acute illness, mobility, safety, and cognition assessments. The NP also provides patient education on self-care and advance care planning. Patient and caregiver support and liaisons with community resources are provided. Home visits by an NP or MD are continued as needed for at least one month. A RN case manager performs weekly phone calls to assess changes in the patient’s clinical status and is available for phone triage of acute health issues. An interdisciplinary team composed of MDs, NPs, RNs, and pharmacists review patient management at weekly meetings. Although after-hours or weekend coverage for home visits are unavailable, an on-call primary care physician is available by phone at all times.

Primary Outcome

The primary outcome was all-cause hospital readmission within 30 days of the first CTP home visit, indicating successful program enrollment. Hospitalization was determined on the basis of billing codes from Mayo Clinic hospitals; this approach is 99% reliable in detecting readmissions for this population.¹⁸

Secondary Outcome Measures

Secondary outcome measures included six-month mortality and hospitalizations, as well as the number of hospital and ICU days and home, ED, primary care, and specialty office visits within 180 days after index hospitalizations as per the EHR. ED visits were counted only when they did not result in a hospital admission.

Independent Variables

Patient characteristics and clinical variables were retrieved from the EHR and included patient age, sex, and marital status. Comorbidities, ERA score,¹⁹ and Charlson comorbidity index (CCI)²⁰ within two years of program enrollment were determined by using ICD-9 billing codes. The frequencies of primary care and specialty visits within six months of the index hospitalization were also ascertained using the EHR. Mobility limitations and cognitive impairment were categorized as binary variables (yes/no) and were assessed at the first home visit by the NP. The presence of mobility limitations was defined as a Barthel’s score of <75^21,22 or Timed up and Go time of >20 seconds.²³ Cognitive impairment was established as Kokmen below the normal cutoff for patient’s age group,²⁴ Mini-Cog ≤2,²⁵or AD8 ≥2.²⁶ If these measures were not specifically documented during the first visit, clinical notes were queried for the description of pertinent cognitive and/or mobility limitations. Dementia diagnosis billing codes (ICD9 Code 290.*) were also included. High medication use was defined as >14 given the reported average medication number ranges from 8-13 in this population.²⁷

As previously published, fidelity measures were abstracted from clinical notes by a trained nurse abstractor within 30 days of program enrollment and prior to a readmission.⁷ The five program fidelity measures included medication reconciliation, home service evaluation, advanced directives discussion, action plan for acute and chronic disease, safety plan, and discussion of community resources. The presence of advanced care planning was determined on the basis of visit medical notes and/or change of code status within the EHR, the identification or scanning of written advanced directives or “provider order for life-sustaining treatment,” and documentation of the discussion of resuscitation status. It was abstracted in duplicate by a nurse abstractor with physician adjudication for disagreement. Moreover, whether the initial visit met the goal of being within five days of discharge was determined by using billing data.

Analysis

The contribution of each independent variable to 30-day readmission was first directly assessed by using a univariate logistic regression model. Five patients died within 30 days without being admitted. These deaths, however, were not censored given that home death (as opposed to hospital death) was considered a positive outcome of the CTP. Multivariable modeling was performed through log rank test with backwards elimination and included all independent variables with P < .05. Variables with P values between .05 and >.1 were tested for interaction with age and sex. Age was categorized as <80 or ≥80 years. The length of hospital stay was categorized as <3 days (not qualifying for a Medicare skilled nursing facility), 3-13 days, or ≥14 days.

This study had 30% power to detect a reduction of 5% in the rates of hospital admissions; 5% is the median absolute risk reduction reported by previous randomized studies on care transitions programs previously reported.¹⁰ All analyses were performed using SAS 6.01 (SAS Inc., Cary, North Carolina).

Study Population

The study cohort included 315 patients who met the inclusion criteria (Fig 1). The demographic and clinical characteristics of the participants were ascertained at the time of CTP enrollment and are shown in Table 1. Patients were, on average, 82.5 (SD, 8.2) years old and had multiple comorbidities with a mean CCI score of 6.2 and ERA score of 18.5. Almost half of the patients (43.2%) exhibited cognitive impairment and more than half (51.7%) had mobility limitations. Among the patients, 42.9% had been hospitalized at least once in the 180 days prior to their CTP-qualifying hospitalization and 14.2% had ≥2 hospitalizations prior to their CTP-qualifying hospitalization. Similarly, 32.4% had at least one emergency department (ED) visit, and 3.5% had ≥3 ED visits. The majority of patients had frequent outpatient visits, with 30.8% having ≥4 office visits in primary care and 32.4% having ≥4 specialty office visits in the preceding six months.

Readmissions, Mortality, ED, and Outpatient Visits

Of the 315 patients, 54 (17.1%) had a readmission within 30 days and seven (2%) had >1 readmission. Among the patients, 126 (40.0%) were readmitted at least once within 180 days with 55 (17.5%) having more than one readmission. A total of 41 patients (13.1%) died during the six-month follow-up period. The need for both office and ED visits was reduced compared to the 180 days prior to admission with the biggest difference in ED visits: 72 (22.9%) of patients needed visits within 180 days of enrollment, as opposed to 102 (32.4%) before enrollment.

Impact of Patient Clinical Variables on Readmission Risk

Readmitted patients were less likely to exhibit cognitive impairment (29.6% vs 46.0%; P = .03) and were more likely to have high medication use (59.3% vs 44.4%; P = .047) than patients without readmission (Table 1). Readmitted patients had a higher frequency of visits to primary care (4.0 vs 3.0; P =.02) in the six months prior to admission and more hospital days in the prior year (4.6 vs 2.5; P = .04) than those without readmission.

Multivariable analysis, which included the cognitive status of the patient; the high use of medication; and the number of ED visits, primary care visits, and hospital days in the previous six months, provided a C statistic of 0.665. After backwards elimination, only the cognitive status of the patient and number of ED visits remained predictive of readmission risk.

Impact of Program Interventions on Readmission Risk

The completion of the CTP fidelity measures drastically varied with completion rates between 29.5% (community resource evaluation) and 87.0% (home visit within five days of hospital discharge; Table 2). Only 12.1% of patients received all components of the CTP at the first home visit. Readmission rates among patients who received all program components (13.2%) were lower than those among patients who did not receive all program components. This difference, however, failed to reach statistical significance. No single program component significantly reduced readmission risk. The completion rate of program fidelity measures increased with time (Figure 2). The present findings did not change even after performing sensitivity analysis that excluded the first program year. The overall agreement between chart abstractors on determining whether advance care planning occurred was 69.5% but the Cohens Kappa was only 18.4. This result was largely ascribed to the following: One abstractor counted the presence of a shorthand template used to document the delivery of an advance care planning document as discussion, whereas the other abstractor required further documentation or corroborating evidence (ie, change of code status). The majority of patients required multiple home visits to address ongoing medical needs (mean 2.7; SD = 1.3) over the first 30 days. Among these patients, only 17.1% received one visit, and 54.6% of patients received ≥3 visits. Eleven (3.5%) patients transitioned to a palliative homebound program that we began offering toward the end of this study to meet patient needs.²⁸

The present study met our objective of identifying individual patient factors that are predictive of the success of our CTP. Cognitively impaired patients were less likely to be readmitted than cognitively intact patients. This finding is particularly important because patients with dementia constitute a subgroup that is at an increased risk of readmission after hospitalization²⁹ and often suffer burdensome transitions at the end of life.^30,31 High medication use and high number of visits to primary care and number of hospital days in the six months leading up to enrollment increase the likelihood of readmission and are plausible measures of disease severity or multi-morbidity that have been identified in previous studies.^32,33 No one program intervention was found to be significantly associated with readmission. This result is consistent with prior works that demonstrated the need for multifaceted and intensive interventions to reduce readmission risk among highly complex and multimorbid patients.^13,14

Our findings suggest that the provision of an alternative to stressful hospitalization to cognitively impaired patients and their caregivers may be an important benefit of care transitions programs. Having a trusted team to consult in acute situations may have enabled early intervention and crisis avoidance. Avoiding hospitalizations and ED visits may also have been in line with their goals of care.^34,35 Given that program intensity varied on the basis of the discretion of the clinical team, patients with cognitive impairment and their caregivers may also have received more intensive support than cognitively intact patients.

In contrast to recent systematic reviews, our study did not find that advance directive discussion had significant effects on reductions in readmission.^36,37 The lack of discussion surrounding the goals of care for patients with serious illnesses was also listed as one of four factors that are strongly associated with preventability in a national cohort of readmitted general medicine patients.³⁸ The lack of power and incomplete documentation may have contributed to our null findings. Trust building must also occur before any meaningful discussion of the goals of care could be achieved, and follow-up time may have to be extended. Toward the end of this study, we developed an extension of our program for patients with limited life expectancy and conservative goals of care. In this extension, reductions in hospitalizations were observed among patients who had multiple goals of care discussions.²⁸

Previous studies have shown that readmissions reduced with timely follow up among patients with heart failure.³⁹ Our results showed no difference in readmission rate based on whether or not our patients were visited within five days from discharge, but we may have been underpowered to detect this difference. In addition, we may have missed readmissions that occurred before the enrollment visit.

The elements of the CTP were evidence based. Fidelity to program goals improved over time and reached high levels with program maturity. Only 12% of the patients received all program components at the first home visit. Patients that had all pillars addressed and documented showed a nonsignificant trend toward reduced readmission rates. NPs were given discretion as to how many visits were required to stabilize a patient and achieve program objectives. Heart failure management was driven by protocol with input from cardiology. Medication reconciliation and clinical assessment with action plan were prioritized at the first visit and thus allowed for the completion of other goals at a subsequent visit if time was insufficient. These decisions were deliberated at weekly physician-led multidisciplinary meetings. This variability allowed the team to meet chronic and urgent needs but further confounded the interpretation of our results. One possible way to interpret the lack of significant predictors of success is that through clinical assessment and flexibility, we were able to tailor our program to meet the needs of this complex multi-morbid population.

This study has important limitations. Given that it is a retrospective cohort study, we were unable to include patients who were enrolled but were either readmitted or dropped out before the first program visit. In addition, because of our study’s limited sample size and readmission rate, we had limited power to detect other potential predictor variables and test for confounding and interaction. While we included numerous variables in our analyses, we lacked information on mental health and the social determinants of health, which are known to influence readmission risk.^40,41 Similarly, we lacked patient self-reported measures of health and information regarding caregiver support, which are important.^42,43 Several of our predictive measures (cognitive impairment, mobility limitations, and program objective completion) were dependent on supplementing billing codes with heterogeneous data abstracted from usual clinical care as opposed to standardized research protocols. Neither method is completely accurate, nor can the combination of the two be assumed to be without inaccuracies. Failure to adequately document the clinical interventions performed by the clinical team is possibly a major confounder as evidenced by the considerable lack of agreement by our trained abstractors in determining whether advance care planning took place. The generalizability of our results is also a concern because the local population is largely white and highly educated, although our experience tells us that many of our program patients have limited means and thus may more closely resemble the general US population.⁴⁴ The strength of our study is that it uses real, practice-based data that can be directly translated to practice.

This study focused on a successful high-intensity CTP. Results showed that compared with patients without dementia, patients with dementia were more likely to avoid hospitalizations as a result of enrollment in the investigated CTP. This study, however, failed to identify specific programmatic components critical for the success of the CTP. These findings support the current hypothesis that multidisciplinary, multimodal, and highly intensive interventions are necessary to care for complex and multi-morbid patients. They also suggest that compared with cognitively functional patients, cognitively impaired patients with conservative goals of care may be more likely to avoid burdensome hospitalizations when provided with early intervention in their home.

Acknowledgments

B.T. conceived and designed the study, interpreted the data, drafted and provided final revisions to the manuscript. P.Y.T, N.D.S., and J.M.N obtained funding, contributed to the conception and design of the study, analysis, and interpretation of the data, and provided critical revisions to the manuscript. P.A.R., R.G.M, and G.J.H., contributed to the conception and design of the study, analysis, and interpretation of the data, and provided critical revisions to the manuscript. S.M.P. Assisted with data acquisition and interpretation, performed the data analysis, and drafted parts of the manuscript. C.Y.Y.C, L.J.H., A.L, A.C., L.B., and R.H. helped with methodologic questions and data interpretation, and provided critical revisions to the manuscript.

All authors read and approved the final manuscript and the decision to submit the manuscript for publication.

We thank Donna Lawson, RN for her help with data abstraction and Annika Beck and Anna Jones in Mayo Clinic Biomedical Ethics Research Program for her help in preparing this manuscript for publication.

Disclosures

The authors declare no conflicts of interest.

Funding

This publication was supported by the Mayo Clinic, Robert D and Patricia E. Center for the Science of Health Care Delivery (B.T., R.H., R.G.M, L.J.H), by the Extramural Grant Program by Satellite Healthcare, a not-for-profit renal care provider (L.J.H., B.T.), and by the National Institute of Health (NIH) National Institute Of Diabetes And Digestive And Kidney Diseases grant K23 DK109134 (L.J.H.) K23DK114497 (RGM) and National Institute on Aging grant K23 AG051679 (B.T.). Additional support was provided by the National Center for Advancing Translational Sciences grant UL1 TR000135. Study contents are the sole responsibility of the authors and do not necessarily represent the official views of NIH.

The sponsors had no role in the design, execution, or reporting of this study.

Prior Presentations

Part of this data was presented in poster format at the American Geriatrics Society meeting in Washington DC 2015.

Unplanned hospital admissions and readmissions have become a major focus of efforts to improve the value of healthcare given that these potentially preventable events exert substantial burden on patients, caregivers, health systems, and the economy.¹ The percentage of patients who are rehospitalized within 30 days have decreased from 20%-21% at the start of the Accountable Care Act and readmission penalties to approximately 18%.^2-5 Rehospitalization rates are 33% at 90 days and approach 40% at six months.^6,7 Readmissions cost Medicare more than $26 billion annually,⁴ with one in five Medicare beneficiaries readmitted within 30 days of hospital discharge.⁸ Centers for Medicare and Medicaid Services and other payers use condition-specific and all-cause 30-day unplanned readmission rates and potentially preventable admissions among patients with complex or multiple comorbidities for public reporting, value-based purchasing, and performance-based reimbursement.^9,10 Consequently, medical groups and hospitals have begun to place an increasing emphasis on improving the transitions of care following hospitalization with the goal of reducing unplanned readmissions.¹¹ Care transitions programs have been shown to decrease readmission rates, mortality, and emergency department (ED) visits.¹²

Care transitions programs vary greatly in their scope of intervention and target groups, as well as in their efficacy in reducing readmissions.^13,14 The Mayo Clinic Care Transition Program, hereafter referred to as CTP, was launched in 2011. This program was modeled after other successful programs and involves home visits by a nurse practitioner (NP) and telephonic support and triage provided by a registered nurse (RN). It is offered to high-risk community-dwelling patients during their hospitalization and begins within a week of hospital discharge.

Although the CTP reduces 30-day readmissions from 20% to 17%,⁷ it is a highly resource-intensive, multimodal, multidisciplinary program. Moreover, whether some components of the CTP are more critical than others remains unknown. Prior studies that examined the individual components of successful CTPs have suggested that a multipronged approach that includes close patient and caregiver support is most predictive of program efficacy.¹³ Long-term program sustainability would benefit from optimization of the most critical components of the program while reducing or eliminating resource-intensive factors that have negligible effects on program success. We therefore examined our CTP to identify whether and which program components are most critical for preventing 30-day readmissions and whether any patient characteristics contribute risk within this complex population.

Study Design and Setting

This study is a retrospective cohort study of patients who were enrolled in the care transitions program of Mayo Clinic Rochester during the period January 1, 2010 to June 30, 2013. Patient demographic and clinical data were obtained from electronic health records (EHR), and information regarding CTP processes and interventions was obtained from a prospectively maintained program database. The study complied with the principles of the Declaration of Helsinki and was approved by the Mayo Clinic Institutional Review Board.

Objectives

The study aimed to describe the performance and utilization of a multidisciplinary care transitions program that has been successful in reducing readmissions for high-risk patients. The study also sought to identify patient and/or program factors associated with failure to prevent readmission within 30 days of program enrollment.

Population

Patients who were enrolled in the CTP following hospital discharge and seen for a posthospital in-home visit prior to hospital readmission (for those readmitted) were included. Patients discharged to a skilled nursing facility were excluded. Patients were eligible for CTP enrollment if they were hospitalized for any cause, community dwelling (including assisted living) prior to hospitalization, and ≥60 years old with an Elder Risk Assessment (ERA) score ≥16.⁷ The ERA incorporates information regarding previous hospital days, age, and comorbid health burden and has been shown to predict 30-day readmissions, mortality, and critical illness (

Intervention

Detailed descriptions of the CTP have been previously published.^7,17 Patients meeting enrollment criteria are enrolled into the CTP by a RN prior to or immediately after hospital discharge. The patient is then seen at home within one to five business days of discharge and again the following week by a NP who performs medication reconciliation; chronic illness management; and acute illness, mobility, safety, and cognition assessments. The NP also provides patient education on self-care and advance care planning. Patient and caregiver support and liaisons with community resources are provided. Home visits by an NP or MD are continued as needed for at least one month. A RN case manager performs weekly phone calls to assess changes in the patient’s clinical status and is available for phone triage of acute health issues. An interdisciplinary team composed of MDs, NPs, RNs, and pharmacists review patient management at weekly meetings. Although after-hours or weekend coverage for home visits are unavailable, an on-call primary care physician is available by phone at all times.

Primary Outcome

The primary outcome was all-cause hospital readmission within 30 days of the first CTP home visit, indicating successful program enrollment. Hospitalization was determined on the basis of billing codes from Mayo Clinic hospitals; this approach is 99% reliable in detecting readmissions for this population.¹⁸

Secondary Outcome Measures

Secondary outcome measures included six-month mortality and hospitalizations, as well as the number of hospital and ICU days and home, ED, primary care, and specialty office visits within 180 days after index hospitalizations as per the EHR. ED visits were counted only when they did not result in a hospital admission.

Independent Variables

Patient characteristics and clinical variables were retrieved from the EHR and included patient age, sex, and marital status. Comorbidities, ERA score,¹⁹ and Charlson comorbidity index (CCI)²⁰ within two years of program enrollment were determined by using ICD-9 billing codes. The frequencies of primary care and specialty visits within six months of the index hospitalization were also ascertained using the EHR. Mobility limitations and cognitive impairment were categorized as binary variables (yes/no) and were assessed at the first home visit by the NP. The presence of mobility limitations was defined as a Barthel’s score of <75^21,22 or Timed up and Go time of >20 seconds.²³ Cognitive impairment was established as Kokmen below the normal cutoff for patient’s age group,²⁴ Mini-Cog ≤2,²⁵or AD8 ≥2.²⁶ If these measures were not specifically documented during the first visit, clinical notes were queried for the description of pertinent cognitive and/or mobility limitations. Dementia diagnosis billing codes (ICD9 Code 290.*) were also included. High medication use was defined as >14 given the reported average medication number ranges from 8-13 in this population.²⁷

As previously published, fidelity measures were abstracted from clinical notes by a trained nurse abstractor within 30 days of program enrollment and prior to a readmission.⁷ The five program fidelity measures included medication reconciliation, home service evaluation, advanced directives discussion, action plan for acute and chronic disease, safety plan, and discussion of community resources. The presence of advanced care planning was determined on the basis of visit medical notes and/or change of code status within the EHR, the identification or scanning of written advanced directives or “provider order for life-sustaining treatment,” and documentation of the discussion of resuscitation status. It was abstracted in duplicate by a nurse abstractor with physician adjudication for disagreement. Moreover, whether the initial visit met the goal of being within five days of discharge was determined by using billing data.

Analysis

The contribution of each independent variable to 30-day readmission was first directly assessed by using a univariate logistic regression model. Five patients died within 30 days without being admitted. These deaths, however, were not censored given that home death (as opposed to hospital death) was considered a positive outcome of the CTP. Multivariable modeling was performed through log rank test with backwards elimination and included all independent variables with P < .05. Variables with P values between .05 and >.1 were tested for interaction with age and sex. Age was categorized as <80 or ≥80 years. The length of hospital stay was categorized as <3 days (not qualifying for a Medicare skilled nursing facility), 3-13 days, or ≥14 days.

This study had 30% power to detect a reduction of 5% in the rates of hospital admissions; 5% is the median absolute risk reduction reported by previous randomized studies on care transitions programs previously reported.¹⁰ All analyses were performed using SAS 6.01 (SAS Inc., Cary, North Carolina).

Study Population

The study cohort included 315 patients who met the inclusion criteria (Fig 1). The demographic and clinical characteristics of the participants were ascertained at the time of CTP enrollment and are shown in Table 1. Patients were, on average, 82.5 (SD, 8.2) years old and had multiple comorbidities with a mean CCI score of 6.2 and ERA score of 18.5. Almost half of the patients (43.2%) exhibited cognitive impairment and more than half (51.7%) had mobility limitations. Among the patients, 42.9% had been hospitalized at least once in the 180 days prior to their CTP-qualifying hospitalization and 14.2% had ≥2 hospitalizations prior to their CTP-qualifying hospitalization. Similarly, 32.4% had at least one emergency department (ED) visit, and 3.5% had ≥3 ED visits. The majority of patients had frequent outpatient visits, with 30.8% having ≥4 office visits in primary care and 32.4% having ≥4 specialty office visits in the preceding six months.

Readmissions, Mortality, ED, and Outpatient Visits

Of the 315 patients, 54 (17.1%) had a readmission within 30 days and seven (2%) had >1 readmission. Among the patients, 126 (40.0%) were readmitted at least once within 180 days with 55 (17.5%) having more than one readmission. A total of 41 patients (13.1%) died during the six-month follow-up period. The need for both office and ED visits was reduced compared to the 180 days prior to admission with the biggest difference in ED visits: 72 (22.9%) of patients needed visits within 180 days of enrollment, as opposed to 102 (32.4%) before enrollment.

Impact of Patient Clinical Variables on Readmission Risk

Readmitted patients were less likely to exhibit cognitive impairment (29.6% vs 46.0%; P = .03) and were more likely to have high medication use (59.3% vs 44.4%; P = .047) than patients without readmission (Table 1). Readmitted patients had a higher frequency of visits to primary care (4.0 vs 3.0; P =.02) in the six months prior to admission and more hospital days in the prior year (4.6 vs 2.5; P = .04) than those without readmission.

Multivariable analysis, which included the cognitive status of the patient; the high use of medication; and the number of ED visits, primary care visits, and hospital days in the previous six months, provided a C statistic of 0.665. After backwards elimination, only the cognitive status of the patient and number of ED visits remained predictive of readmission risk.

Impact of Program Interventions on Readmission Risk

The completion of the CTP fidelity measures drastically varied with completion rates between 29.5% (community resource evaluation) and 87.0% (home visit within five days of hospital discharge; Table 2). Only 12.1% of patients received all components of the CTP at the first home visit. Readmission rates among patients who received all program components (13.2%) were lower than those among patients who did not receive all program components. This difference, however, failed to reach statistical significance. No single program component significantly reduced readmission risk. The completion rate of program fidelity measures increased with time (Figure 2). The present findings did not change even after performing sensitivity analysis that excluded the first program year. The overall agreement between chart abstractors on determining whether advance care planning occurred was 69.5% but the Cohens Kappa was only 18.4. This result was largely ascribed to the following: One abstractor counted the presence of a shorthand template used to document the delivery of an advance care planning document as discussion, whereas the other abstractor required further documentation or corroborating evidence (ie, change of code status). The majority of patients required multiple home visits to address ongoing medical needs (mean 2.7; SD = 1.3) over the first 30 days. Among these patients, only 17.1% received one visit, and 54.6% of patients received ≥3 visits. Eleven (3.5%) patients transitioned to a palliative homebound program that we began offering toward the end of this study to meet patient needs.²⁸

The present study met our objective of identifying individual patient factors that are predictive of the success of our CTP. Cognitively impaired patients were less likely to be readmitted than cognitively intact patients. This finding is particularly important because patients with dementia constitute a subgroup that is at an increased risk of readmission after hospitalization²⁹ and often suffer burdensome transitions at the end of life.^30,31 High medication use and high number of visits to primary care and number of hospital days in the six months leading up to enrollment increase the likelihood of readmission and are plausible measures of disease severity or multi-morbidity that have been identified in previous studies.^32,33 No one program intervention was found to be significantly associated with readmission. This result is consistent with prior works that demonstrated the need for multifaceted and intensive interventions to reduce readmission risk among highly complex and multimorbid patients.^13,14

Our findings suggest that the provision of an alternative to stressful hospitalization to cognitively impaired patients and their caregivers may be an important benefit of care transitions programs. Having a trusted team to consult in acute situations may have enabled early intervention and crisis avoidance. Avoiding hospitalizations and ED visits may also have been in line with their goals of care.^34,35 Given that program intensity varied on the basis of the discretion of the clinical team, patients with cognitive impairment and their caregivers may also have received more intensive support than cognitively intact patients.

In contrast to recent systematic reviews, our study did not find that advance directive discussion had significant effects on reductions in readmission.^36,37 The lack of discussion surrounding the goals of care for patients with serious illnesses was also listed as one of four factors that are strongly associated with preventability in a national cohort of readmitted general medicine patients.³⁸ The lack of power and incomplete documentation may have contributed to our null findings. Trust building must also occur before any meaningful discussion of the goals of care could be achieved, and follow-up time may have to be extended. Toward the end of this study, we developed an extension of our program for patients with limited life expectancy and conservative goals of care. In this extension, reductions in hospitalizations were observed among patients who had multiple goals of care discussions.²⁸

Previous studies have shown that readmissions reduced with timely follow up among patients with heart failure.³⁹ Our results showed no difference in readmission rate based on whether or not our patients were visited within five days from discharge, but we may have been underpowered to detect this difference. In addition, we may have missed readmissions that occurred before the enrollment visit.

The elements of the CTP were evidence based. Fidelity to program goals improved over time and reached high levels with program maturity. Only 12% of the patients received all program components at the first home visit. Patients that had all pillars addressed and documented showed a nonsignificant trend toward reduced readmission rates. NPs were given discretion as to how many visits were required to stabilize a patient and achieve program objectives. Heart failure management was driven by protocol with input from cardiology. Medication reconciliation and clinical assessment with action plan were prioritized at the first visit and thus allowed for the completion of other goals at a subsequent visit if time was insufficient. These decisions were deliberated at weekly physician-led multidisciplinary meetings. This variability allowed the team to meet chronic and urgent needs but further confounded the interpretation of our results. One possible way to interpret the lack of significant predictors of success is that through clinical assessment and flexibility, we were able to tailor our program to meet the needs of this complex multi-morbid population.

This study has important limitations. Given that it is a retrospective cohort study, we were unable to include patients who were enrolled but were either readmitted or dropped out before the first program visit. In addition, because of our study’s limited sample size and readmission rate, we had limited power to detect other potential predictor variables and test for confounding and interaction. While we included numerous variables in our analyses, we lacked information on mental health and the social determinants of health, which are known to influence readmission risk.^40,41 Similarly, we lacked patient self-reported measures of health and information regarding caregiver support, which are important.^42,43 Several of our predictive measures (cognitive impairment, mobility limitations, and program objective completion) were dependent on supplementing billing codes with heterogeneous data abstracted from usual clinical care as opposed to standardized research protocols. Neither method is completely accurate, nor can the combination of the two be assumed to be without inaccuracies. Failure to adequately document the clinical interventions performed by the clinical team is possibly a major confounder as evidenced by the considerable lack of agreement by our trained abstractors in determining whether advance care planning took place. The generalizability of our results is also a concern because the local population is largely white and highly educated, although our experience tells us that many of our program patients have limited means and thus may more closely resemble the general US population.⁴⁴ The strength of our study is that it uses real, practice-based data that can be directly translated to practice.

This study focused on a successful high-intensity CTP. Results showed that compared with patients without dementia, patients with dementia were more likely to avoid hospitalizations as a result of enrollment in the investigated CTP. This study, however, failed to identify specific programmatic components critical for the success of the CTP. These findings support the current hypothesis that multidisciplinary, multimodal, and highly intensive interventions are necessary to care for complex and multi-morbid patients. They also suggest that compared with cognitively functional patients, cognitively impaired patients with conservative goals of care may be more likely to avoid burdensome hospitalizations when provided with early intervention in their home.

Acknowledgments

B.T. conceived and designed the study, interpreted the data, drafted and provided final revisions to the manuscript. P.Y.T, N.D.S., and J.M.N obtained funding, contributed to the conception and design of the study, analysis, and interpretation of the data, and provided critical revisions to the manuscript. P.A.R., R.G.M, and G.J.H., contributed to the conception and design of the study, analysis, and interpretation of the data, and provided critical revisions to the manuscript. S.M.P. Assisted with data acquisition and interpretation, performed the data analysis, and drafted parts of the manuscript. C.Y.Y.C, L.J.H., A.L, A.C., L.B., and R.H. helped with methodologic questions and data interpretation, and provided critical revisions to the manuscript.

All authors read and approved the final manuscript and the decision to submit the manuscript for publication.

We thank Donna Lawson, RN for her help with data abstraction and Annika Beck and Anna Jones in Mayo Clinic Biomedical Ethics Research Program for her help in preparing this manuscript for publication.

Disclosures

The authors declare no conflicts of interest.

Funding

This publication was supported by the Mayo Clinic, Robert D and Patricia E. Center for the Science of Health Care Delivery (B.T., R.H., R.G.M, L.J.H), by the Extramural Grant Program by Satellite Healthcare, a not-for-profit renal care provider (L.J.H., B.T.), and by the National Institute of Health (NIH) National Institute Of Diabetes And Digestive And Kidney Diseases grant K23 DK109134 (L.J.H.) K23DK114497 (RGM) and National Institute on Aging grant K23 AG051679 (B.T.). Additional support was provided by the National Center for Advancing Translational Sciences grant UL1 TR000135. Study contents are the sole responsibility of the authors and do not necessarily represent the official views of NIH.

The sponsors had no role in the design, execution, or reporting of this study.

Prior Presentations

Part of this data was presented in poster format at the American Geriatrics Society meeting in Washington DC 2015.

Unplanned hospital admissions and readmissions have become a major focus of efforts to improve the value of healthcare given that these potentially preventable events exert substantial burden on patients, caregivers, health systems, and the economy.¹ The percentage of patients who are rehospitalized within 30 days have decreased from 20%-21% at the start of the Accountable Care Act and readmission penalties to approximately 18%.^2-5 Rehospitalization rates are 33% at 90 days and approach 40% at six months.^6,7 Readmissions cost Medicare more than $26 billion annually,⁴ with one in five Medicare beneficiaries readmitted within 30 days of hospital discharge.⁸ Centers for Medicare and Medicaid Services and other payers use condition-specific and all-cause 30-day unplanned readmission rates and potentially preventable admissions among patients with complex or multiple comorbidities for public reporting, value-based purchasing, and performance-based reimbursement.^9,10 Consequently, medical groups and hospitals have begun to place an increasing emphasis on improving the transitions of care following hospitalization with the goal of reducing unplanned readmissions.¹¹ Care transitions programs have been shown to decrease readmission rates, mortality, and emergency department (ED) visits.¹²

Care transitions programs vary greatly in their scope of intervention and target groups, as well as in their efficacy in reducing readmissions.^13,14 The Mayo Clinic Care Transition Program, hereafter referred to as CTP, was launched in 2011. This program was modeled after other successful programs and involves home visits by a nurse practitioner (NP) and telephonic support and triage provided by a registered nurse (RN). It is offered to high-risk community-dwelling patients during their hospitalization and begins within a week of hospital discharge.

Although the CTP reduces 30-day readmissions from 20% to 17%,⁷ it is a highly resource-intensive, multimodal, multidisciplinary program. Moreover, whether some components of the CTP are more critical than others remains unknown. Prior studies that examined the individual components of successful CTPs have suggested that a multipronged approach that includes close patient and caregiver support is most predictive of program efficacy.¹³ Long-term program sustainability would benefit from optimization of the most critical components of the program while reducing or eliminating resource-intensive factors that have negligible effects on program success. We therefore examined our CTP to identify whether and which program components are most critical for preventing 30-day readmissions and whether any patient characteristics contribute risk within this complex population.

Study Design and Setting

This study is a retrospective cohort study of patients who were enrolled in the care transitions program of Mayo Clinic Rochester during the period January 1, 2010 to June 30, 2013. Patient demographic and clinical data were obtained from electronic health records (EHR), and information regarding CTP processes and interventions was obtained from a prospectively maintained program database. The study complied with the principles of the Declaration of Helsinki and was approved by the Mayo Clinic Institutional Review Board.

Objectives

The study aimed to describe the performance and utilization of a multidisciplinary care transitions program that has been successful in reducing readmissions for high-risk patients. The study also sought to identify patient and/or program factors associated with failure to prevent readmission within 30 days of program enrollment.

Population

Patients who were enrolled in the CTP following hospital discharge and seen for a posthospital in-home visit prior to hospital readmission (for those readmitted) were included. Patients discharged to a skilled nursing facility were excluded. Patients were eligible for CTP enrollment if they were hospitalized for any cause, community dwelling (including assisted living) prior to hospitalization, and ≥60 years old with an Elder Risk Assessment (ERA) score ≥16.⁷ The ERA incorporates information regarding previous hospital days, age, and comorbid health burden and has been shown to predict 30-day readmissions, mortality, and critical illness (

Intervention

Detailed descriptions of the CTP have been previously published.^7,17 Patients meeting enrollment criteria are enrolled into the CTP by a RN prior to or immediately after hospital discharge. The patient is then seen at home within one to five business days of discharge and again the following week by a NP who performs medication reconciliation; chronic illness management; and acute illness, mobility, safety, and cognition assessments. The NP also provides patient education on self-care and advance care planning. Patient and caregiver support and liaisons with community resources are provided. Home visits by an NP or MD are continued as needed for at least one month. A RN case manager performs weekly phone calls to assess changes in the patient’s clinical status and is available for phone triage of acute health issues. An interdisciplinary team composed of MDs, NPs, RNs, and pharmacists review patient management at weekly meetings. Although after-hours or weekend coverage for home visits are unavailable, an on-call primary care physician is available by phone at all times.

Primary Outcome

The primary outcome was all-cause hospital readmission within 30 days of the first CTP home visit, indicating successful program enrollment. Hospitalization was determined on the basis of billing codes from Mayo Clinic hospitals; this approach is 99% reliable in detecting readmissions for this population.¹⁸

Secondary Outcome Measures

Secondary outcome measures included six-month mortality and hospitalizations, as well as the number of hospital and ICU days and home, ED, primary care, and specialty office visits within 180 days after index hospitalizations as per the EHR. ED visits were counted only when they did not result in a hospital admission.

Independent Variables

Patient characteristics and clinical variables were retrieved from the EHR and included patient age, sex, and marital status. Comorbidities, ERA score,¹⁹ and Charlson comorbidity index (CCI)²⁰ within two years of program enrollment were determined by using ICD-9 billing codes. The frequencies of primary care and specialty visits within six months of the index hospitalization were also ascertained using the EHR. Mobility limitations and cognitive impairment were categorized as binary variables (yes/no) and were assessed at the first home visit by the NP. The presence of mobility limitations was defined as a Barthel’s score of <75^21,22 or Timed up and Go time of >20 seconds.²³ Cognitive impairment was established as Kokmen below the normal cutoff for patient’s age group,²⁴ Mini-Cog ≤2,²⁵or AD8 ≥2.²⁶ If these measures were not specifically documented during the first visit, clinical notes were queried for the description of pertinent cognitive and/or mobility limitations. Dementia diagnosis billing codes (ICD9 Code 290.*) were also included. High medication use was defined as >14 given the reported average medication number ranges from 8-13 in this population.²⁷

As previously published, fidelity measures were abstracted from clinical notes by a trained nurse abstractor within 30 days of program enrollment and prior to a readmission.⁷ The five program fidelity measures included medication reconciliation, home service evaluation, advanced directives discussion, action plan for acute and chronic disease, safety plan, and discussion of community resources. The presence of advanced care planning was determined on the basis of visit medical notes and/or change of code status within the EHR, the identification or scanning of written advanced directives or “provider order for life-sustaining treatment,” and documentation of the discussion of resuscitation status. It was abstracted in duplicate by a nurse abstractor with physician adjudication for disagreement. Moreover, whether the initial visit met the goal of being within five days of discharge was determined by using billing data.

Analysis

The contribution of each independent variable to 30-day readmission was first directly assessed by using a univariate logistic regression model. Five patients died within 30 days without being admitted. These deaths, however, were not censored given that home death (as opposed to hospital death) was considered a positive outcome of the CTP. Multivariable modeling was performed through log rank test with backwards elimination and included all independent variables with P < .05. Variables with P values between .05 and >.1 were tested for interaction with age and sex. Age was categorized as <80 or ≥80 years. The length of hospital stay was categorized as <3 days (not qualifying for a Medicare skilled nursing facility), 3-13 days, or ≥14 days.

This study had 30% power to detect a reduction of 5% in the rates of hospital admissions; 5% is the median absolute risk reduction reported by previous randomized studies on care transitions programs previously reported.¹⁰ All analyses were performed using SAS 6.01 (SAS Inc., Cary, North Carolina).

Study Population

The study cohort included 315 patients who met the inclusion criteria (Fig 1). The demographic and clinical characteristics of the participants were ascertained at the time of CTP enrollment and are shown in Table 1. Patients were, on average, 82.5 (SD, 8.2) years old and had multiple comorbidities with a mean CCI score of 6.2 and ERA score of 18.5. Almost half of the patients (43.2%) exhibited cognitive impairment and more than half (51.7%) had mobility limitations. Among the patients, 42.9% had been hospitalized at least once in the 180 days prior to their CTP-qualifying hospitalization and 14.2% had ≥2 hospitalizations prior to their CTP-qualifying hospitalization. Similarly, 32.4% had at least one emergency department (ED) visit, and 3.5% had ≥3 ED visits. The majority of patients had frequent outpatient visits, with 30.8% having ≥4 office visits in primary care and 32.4% having ≥4 specialty office visits in the preceding six months.

Readmissions, Mortality, ED, and Outpatient Visits

Of the 315 patients, 54 (17.1%) had a readmission within 30 days and seven (2%) had >1 readmission. Among the patients, 126 (40.0%) were readmitted at least once within 180 days with 55 (17.5%) having more than one readmission. A total of 41 patients (13.1%) died during the six-month follow-up period. The need for both office and ED visits was reduced compared to the 180 days prior to admission with the biggest difference in ED visits: 72 (22.9%) of patients needed visits within 180 days of enrollment, as opposed to 102 (32.4%) before enrollment.

Impact of Patient Clinical Variables on Readmission Risk

Readmitted patients were less likely to exhibit cognitive impairment (29.6% vs 46.0%; P = .03) and were more likely to have high medication use (59.3% vs 44.4%; P = .047) than patients without readmission (Table 1). Readmitted patients had a higher frequency of visits to primary care (4.0 vs 3.0; P =.02) in the six months prior to admission and more hospital days in the prior year (4.6 vs 2.5; P = .04) than those without readmission.

Multivariable analysis, which included the cognitive status of the patient; the high use of medication; and the number of ED visits, primary care visits, and hospital days in the previous six months, provided a C statistic of 0.665. After backwards elimination, only the cognitive status of the patient and number of ED visits remained predictive of readmission risk.

Impact of Program Interventions on Readmission Risk

The completion of the CTP fidelity measures drastically varied with completion rates between 29.5% (community resource evaluation) and 87.0% (home visit within five days of hospital discharge; Table 2). Only 12.1% of patients received all components of the CTP at the first home visit. Readmission rates among patients who received all program components (13.2%) were lower than those among patients who did not receive all program components. This difference, however, failed to reach statistical significance. No single program component significantly reduced readmission risk. The completion rate of program fidelity measures increased with time (Figure 2). The present findings did not change even after performing sensitivity analysis that excluded the first program year. The overall agreement between chart abstractors on determining whether advance care planning occurred was 69.5% but the Cohens Kappa was only 18.4. This result was largely ascribed to the following: One abstractor counted the presence of a shorthand template used to document the delivery of an advance care planning document as discussion, whereas the other abstractor required further documentation or corroborating evidence (ie, change of code status). The majority of patients required multiple home visits to address ongoing medical needs (mean 2.7; SD = 1.3) over the first 30 days. Among these patients, only 17.1% received one visit, and 54.6% of patients received ≥3 visits. Eleven (3.5%) patients transitioned to a palliative homebound program that we began offering toward the end of this study to meet patient needs.²⁸

The present study met our objective of identifying individual patient factors that are predictive of the success of our CTP. Cognitively impaired patients were less likely to be readmitted than cognitively intact patients. This finding is particularly important because patients with dementia constitute a subgroup that is at an increased risk of readmission after hospitalization²⁹ and often suffer burdensome transitions at the end of life.^30,31 High medication use and high number of visits to primary care and number of hospital days in the six months leading up to enrollment increase the likelihood of readmission and are plausible measures of disease severity or multi-morbidity that have been identified in previous studies.^32,33 No one program intervention was found to be significantly associated with readmission. This result is consistent with prior works that demonstrated the need for multifaceted and intensive interventions to reduce readmission risk among highly complex and multimorbid patients.^13,14

Our findings suggest that the provision of an alternative to stressful hospitalization to cognitively impaired patients and their caregivers may be an important benefit of care transitions programs. Having a trusted team to consult in acute situations may have enabled early intervention and crisis avoidance. Avoiding hospitalizations and ED visits may also have been in line with their goals of care.^34,35 Given that program intensity varied on the basis of the discretion of the clinical team, patients with cognitive impairment and their caregivers may also have received more intensive support than cognitively intact patients.

In contrast to recent systematic reviews, our study did not find that advance directive discussion had significant effects on reductions in readmission.^36,37 The lack of discussion surrounding the goals of care for patients with serious illnesses was also listed as one of four factors that are strongly associated with preventability in a national cohort of readmitted general medicine patients.³⁸ The lack of power and incomplete documentation may have contributed to our null findings. Trust building must also occur before any meaningful discussion of the goals of care could be achieved, and follow-up time may have to be extended. Toward the end of this study, we developed an extension of our program for patients with limited life expectancy and conservative goals of care. In this extension, reductions in hospitalizations were observed among patients who had multiple goals of care discussions.²⁸

Previous studies have shown that readmissions reduced with timely follow up among patients with heart failure.³⁹ Our results showed no difference in readmission rate based on whether or not our patients were visited within five days from discharge, but we may have been underpowered to detect this difference. In addition, we may have missed readmissions that occurred before the enrollment visit.

The elements of the CTP were evidence based. Fidelity to program goals improved over time and reached high levels with program maturity. Only 12% of the patients received all program components at the first home visit. Patients that had all pillars addressed and documented showed a nonsignificant trend toward reduced readmission rates. NPs were given discretion as to how many visits were required to stabilize a patient and achieve program objectives. Heart failure management was driven by protocol with input from cardiology. Medication reconciliation and clinical assessment with action plan were prioritized at the first visit and thus allowed for the completion of other goals at a subsequent visit if time was insufficient. These decisions were deliberated at weekly physician-led multidisciplinary meetings. This variability allowed the team to meet chronic and urgent needs but further confounded the interpretation of our results. One possible way to interpret the lack of significant predictors of success is that through clinical assessment and flexibility, we were able to tailor our program to meet the needs of this complex multi-morbid population.

This study has important limitations. Given that it is a retrospective cohort study, we were unable to include patients who were enrolled but were either readmitted or dropped out before the first program visit. In addition, because of our study’s limited sample size and readmission rate, we had limited power to detect other potential predictor variables and test for confounding and interaction. While we included numerous variables in our analyses, we lacked information on mental health and the social determinants of health, which are known to influence readmission risk.^40,41 Similarly, we lacked patient self-reported measures of health and information regarding caregiver support, which are important.^42,43 Several of our predictive measures (cognitive impairment, mobility limitations, and program objective completion) were dependent on supplementing billing codes with heterogeneous data abstracted from usual clinical care as opposed to standardized research protocols. Neither method is completely accurate, nor can the combination of the two be assumed to be without inaccuracies. Failure to adequately document the clinical interventions performed by the clinical team is possibly a major confounder as evidenced by the considerable lack of agreement by our trained abstractors in determining whether advance care planning took place. The generalizability of our results is also a concern because the local population is largely white and highly educated, although our experience tells us that many of our program patients have limited means and thus may more closely resemble the general US population.⁴⁴ The strength of our study is that it uses real, practice-based data that can be directly translated to practice.

This study focused on a successful high-intensity CTP. Results showed that compared with patients without dementia, patients with dementia were more likely to avoid hospitalizations as a result of enrollment in the investigated CTP. This study, however, failed to identify specific programmatic components critical for the success of the CTP. These findings support the current hypothesis that multidisciplinary, multimodal, and highly intensive interventions are necessary to care for complex and multi-morbid patients. They also suggest that compared with cognitively functional patients, cognitively impaired patients with conservative goals of care may be more likely to avoid burdensome hospitalizations when provided with early intervention in their home.

Acknowledgments

B.T. conceived and designed the study, interpreted the data, drafted and provided final revisions to the manuscript. P.Y.T, N.D.S., and J.M.N obtained funding, contributed to the conception and design of the study, analysis, and interpretation of the data, and provided critical revisions to the manuscript. P.A.R., R.G.M, and G.J.H., contributed to the conception and design of the study, analysis, and interpretation of the data, and provided critical revisions to the manuscript. S.M.P. Assisted with data acquisition and interpretation, performed the data analysis, and drafted parts of the manuscript. C.Y.Y.C, L.J.H., A.L, A.C., L.B., and R.H. helped with methodologic questions and data interpretation, and provided critical revisions to the manuscript.

All authors read and approved the final manuscript and the decision to submit the manuscript for publication.

We thank Donna Lawson, RN for her help with data abstraction and Annika Beck and Anna Jones in Mayo Clinic Biomedical Ethics Research Program for her help in preparing this manuscript for publication.

Disclosures

The authors declare no conflicts of interest.

Funding

This publication was supported by the Mayo Clinic, Robert D and Patricia E. Center for the Science of Health Care Delivery (B.T., R.H., R.G.M, L.J.H), by the Extramural Grant Program by Satellite Healthcare, a not-for-profit renal care provider (L.J.H., B.T.), and by the National Institute of Health (NIH) National Institute Of Diabetes And Digestive And Kidney Diseases grant K23 DK109134 (L.J.H.) K23DK114497 (RGM) and National Institute on Aging grant K23 AG051679 (B.T.). Additional support was provided by the National Center for Advancing Translational Sciences grant UL1 TR000135. Study contents are the sole responsibility of the authors and do not necessarily represent the official views of NIH.

The sponsors had no role in the design, execution, or reporting of this study.

Prior Presentations

Part of this data was presented in poster format at the American Geriatrics Society meeting in Washington DC 2015.

Research evidence is critical for strengthening the value, quality, and safety of patient care. Learning healthcare systems (LHS) can support the delivery of evidence-based healthcare by establishing organizational processes that support three activities (Figure).^1-3

• Knowledge: Identifying and synthesizing evidence to address clinical challenges
• Practice: Applying knowledge in the process of care delivery
• Data: Assessing performance and creating a feedback cycle for learning and improvement

The systematic implementation of evidence into practice continues to be a challenge for many healthcare organizations^4-7 due to limited resources, expertise, and culture.^5,8-12 Missing opportunities for translating knowledge into practice not only results in low-value care (ie, waste) but also in harm.¹

The AHRQ (Agency for Healthcare Research and Quality) Evidence-based Practice Center (EPC) Program was established in 1997, with the goal of synthesizing research to inform evidence-based healthcare. The national impact of this program has been significant. Since the American Recovery and Reinvestment Act of 2009, EPC program reports have been used to inform over 95 clinical practice guidelines from societies such as the American College of Physicians, 16 health coverage decisions from payers such as the Centers for Medicare & Medicaid Services, and 24 government policies and program planning efforts, such as the National Institutes of Health Pathways to Prevention Program.¹³

The EPC program recognizes that evidence awareness is not sufficient to change practice and improve clinical outcomes. As such, the EPC program also embarked on initiatives to facilitate the translation of evidence into clinical practice and to measure and monitor how changes in practice impact health outcomes. AHRQ has historically worked with professional organizations to translate systematic reviews into clinical practice guidelines as well as federal agencies to inform payer decisions and program planning. Recently, the EPC program has increased collaborative efforts with hospitals and healthcare systems to understand how they use evidence and to partner with them to identify methods to improve the uptake of evidence into practice.^9,12

In this perspective, we describe the AHRQ EPC Program’s work to address the three phases of the LHS cycle (knowledge, practice, and data) to support high-value care, using the topic of preventing and treating Clostridium difficile colitis as a relevant example to the hospital medicine field (Figure 2). By sharing this work, we hope it can serve as a model to illustrate how partnerships between organizations and AHRQ can lead to improvements in healthcare.

Knowledge: Identifying and Synthesizing Evidence to Address Clinical Challenges

Systematic reviews use carefully formulated questions to summarize the literature results using specific and established methods.¹⁴ Given that individual studies can have disparate results, it is critical to summarize and synthesize findings across studies, so we know what the overall evidence suggests, and whether we can be confident in the findings. To date, the EPC program has developed more than 500 evidence synthesis reports. An example relevant to the field of hospital medicine is the 2016 review that examined the effects of interventions to prevent and treat Clostridium difficile colitis in adults.¹⁵

The review examined the best available evidence, including data from randomized controlled trials and observational studies, on diagnosing, preventing, and treating Clostridium difficile colitis. Major findings included the following: vancomycin is more effective than metronidazole for treating the first occurrence of Clostridium difficile colitis (high-strength evidence), fecal transplantation may have a significant benefit in the treatment of recurrent Clostridium difficile colitis (low-strength evidence), and institutional preventive interventions such as antibiotic stewardship practices, transmission interruption through terminal room cleaning, and handwashing campaigns reduce the incidence of Clostridium difficile colitis (low-strength evidence). The report results provided the most recent review of the evidence and were particularly important as they suggested a need for significant practice changes in the treatment of Clostridium difficile colitis based on the new evidence available. Previous to this report, the 2010 guidelines from the Infectious Diseases Society of America (IDSA) recommended metronidazole over vancomycin for the treatment of the first occurrence of Clostridium difficile colitis.¹⁶ Subsequently, the newly released 2018 IDSA guideline provides recommendations consistent with the findings in this AHRQ report.¹⁷

Practice: Applying Knowledge in the Process of Care Delivery

AHRQ recognizes there are many interim steps between having the results from a systematic review and changing practice and improving care. In 2017, the EPC program began piloting approaches to make it easier for healthcare systems and hospitals to use its reports to improve the delivery of patient care and clinical outcomes. A pilot project conducted by the ECRI Institute - Penn Medicine EPC evaluated the feasibility of using an existing clinical pathway development and dissemination framework¹⁸ to translate findings from the 2016 AHRQ EPC report on Clostridium difficile colitis into a pathway for Clostridium difficile colitis treatment in the acute care setting.

To develop a Clostridium difficile colitis treatment pathway, the ECRI-Penn EPC team recruited a representative stakeholder group from Penn Medicine to review the EPC report as well as existing society guidelines. The clinical pathway was subsequently developed and approved by the stakeholders and disseminated through the Penn Medicine cloud-based pathways repository beginning on April 16, 2018.¹⁹ Most recently, the pathway became available in the electronic health record (EHR; 2018 Epic Systems Corporation) to facilitate provider review during care. Specifically, hyperlinks to the pathway are embedded within the ordering screens for those antibiotics used to treat Clostridium difficile colitis (ie, oral and rectal vancomycin, fidaxomicin, and metronidazole). Upon clicking the link in the ordering screen, the pathway launches a floating internet explorer window. The pathway is now publicly available on the AHRQ’s Clinical Decision Support (CDS) Connect Project (https://cds.ahrq.gov/), which is a resource to share pathway artifacts for other healthcare systems to use.

Data: Assessing Performance and Creating a Feedback Cycle for Learning and Improvement

The last step in the LHS cycle is to identify the impact of interventions on practice change and clinical outcomes, to understand how local results compare to peer institutions, and to inform future research and knowledge.

For the ECRI Institute-Penn Medicine EPC pilot project, both qualitative and quantitative outcomes were assessed. The initial qualitative analysis focused on the feasibility of using the AHRQ report in an existing pathway development and dissemination framework.¹⁸ It was found that clinical stakeholders identified the EPC report as trustworthy and more current than the society guidelines available at the time of development, particularly regarding the finding that vancomycin was more effective than metronidazole for the first occurrence of Clostridium difficile colitis. Additional qualitative analysis will be conducted to understand provider satisfaction with the pathway and practice impact. The quantitative analysis focused on pathway use (clicks over time) and found that as of September 16, 2018, the pathway had been viewed by providers 403 times. Future analysis will evaluate the impact of the pathway on the use of oral vancomycin for the first occurrences of Clostridium difficile colitis.

Patient registries can also help clinicians and healthcare systems to complete the feedback cycle and evaluate outcomes. Patient registries collect data from clinical and other sources in a standardized way in order to evaluate specific outcomes for various populations.²⁰ AHRQ has created a registry handbook, including best practices for how to create, operate, and evaluate registries.²⁰ This handbook enables the development of high-quality registries with data that can be leveraged for both research and improvement.

In the example of the ECRI Institute-Penn Medicine EPC pilot project, one way that a learning healthcare system, such as Penn Medicine, might measure the impact of the clinical pathway is to develop a quality improvement registry, which might be developed with information from their electronic health record, to examine the impact on the use of vancomycin for first occurrences of Clostridium difficile colitis. This information could help drive improvement in the implementation of the clinical pathway.

Registries can also be used as a source for research data. The NIH-funded American Gastroenterological Association (AGA) Fecal Microbiota Transplantation National Registry is an example of a research registry that collects data on outcomes and adverse events associated with fecal transplants to fill gaps in existing research. The 2016 AHRQ EPC review found low-strength evidence on fecal transplant for treatment of recurrent Clostridium difficile colitis. When designing the protocol for this registry, the researchers used the AHRQ handbook to inform the design. Given that this is a research registry, it can be used by researchers to examine trends and outcomes of fecal transplant to treat Clostridium difficile colitis. Publications that use the registry as its source of data may be used in future systematic reviews, thus completing the cycle of learning.

The EPC program recognizes that gaps remain in the evidence to practice translation process and that more support is needed. Some upcoming activities of the AHRQ EPC Program to address these gaps and make its evidence reports more actionable for healthcare systems include:

• Projects to Disseminate EPC Reports into Clinical Practice. In addition to the ECRI Institute - Penn Medicine EPC pilot dissemination project, other pilot projects are aimed at helping systems apply evidence to practice and include new ways to visualize evidence to make it more actionable and usable; creating other dissemination products, such as evidence summaries and presentations for decision makers; and other implementation tools, such as decision aids. These products and summary reports are available on the AHRQ Effective Health Care Program website at https://effectivehealthcare.ahrq.gov/topics/health-systems-use-evidence/overview.
• Healthcare Systems Stakeholder Panel. Starting in Fall 2018, the AHRQ EPC Program will be convening a panel of healthcare system leaders to help make its reports and products more useful and responsive to the needs of healthcare systems and promote the use of evidence in clinical practice.
• Rapid Evidence Products. AHRQ understands that healthcare systems need information rapidly and cannot wait a year or more for a traditional systematic review to be completed. Therefore, AHRQ is applying its methods work on rapid reviews^21-24 to pilot new report types that systematically identify and summarize the evidence quickly for healthcare systems and quality improvement efforts.²⁵
• Data Integration. Originally launched in 2012, the Systematic Review Data Repository (SRDR) is an AHRQ-supported online open-access repository of abstracted data from individual studies from systematic reviews. The goal is to enable more efficient updates of systematic reviews through data reuse. An updated version of the SRDR is scheduled to launch in 2020. With the new version, future sharing of summary data from systematic reviews digitally in a computable and portable format may allow integration into CDS tools and clinical practice guideline development and dissemination, facilitating the use of evidence in clinical practice.

The AHRQ EPC program supports initiatives to make evidence more actionable and provide resources and tools throughout all the phases of the learning healthcare system cycle. This case study on C. difficile is one example of how the EPC program is helping hospitals and healthcare systems improve clinical care delivery and its derivative value.

Disclosures

Dr. Umscheid reports grants from AHRQ, during the conduct of the study; serves on the Advisory Board of DynaMed, and founded and directed a hospital-based evidence-based practice center. All other authors have nothing to disclose.

Disclaimer

The findings and conclusions in this document are those of the author(s), who are responsible for its content, and do not necessarily represent the views of AHRQ. No statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.

Research evidence is critical for strengthening the value, quality, and safety of patient care. Learning healthcare systems (LHS) can support the delivery of evidence-based healthcare by establishing organizational processes that support three activities (Figure).^1-3

• Knowledge: Identifying and synthesizing evidence to address clinical challenges
• Practice: Applying knowledge in the process of care delivery
• Data: Assessing performance and creating a feedback cycle for learning and improvement

The systematic implementation of evidence into practice continues to be a challenge for many healthcare organizations^4-7 due to limited resources, expertise, and culture.^5,8-12 Missing opportunities for translating knowledge into practice not only results in low-value care (ie, waste) but also in harm.¹

The AHRQ (Agency for Healthcare Research and Quality) Evidence-based Practice Center (EPC) Program was established in 1997, with the goal of synthesizing research to inform evidence-based healthcare. The national impact of this program has been significant. Since the American Recovery and Reinvestment Act of 2009, EPC program reports have been used to inform over 95 clinical practice guidelines from societies such as the American College of Physicians, 16 health coverage decisions from payers such as the Centers for Medicare & Medicaid Services, and 24 government policies and program planning efforts, such as the National Institutes of Health Pathways to Prevention Program.¹³

The EPC program recognizes that evidence awareness is not sufficient to change practice and improve clinical outcomes. As such, the EPC program also embarked on initiatives to facilitate the translation of evidence into clinical practice and to measure and monitor how changes in practice impact health outcomes. AHRQ has historically worked with professional organizations to translate systematic reviews into clinical practice guidelines as well as federal agencies to inform payer decisions and program planning. Recently, the EPC program has increased collaborative efforts with hospitals and healthcare systems to understand how they use evidence and to partner with them to identify methods to improve the uptake of evidence into practice.^9,12

In this perspective, we describe the AHRQ EPC Program’s work to address the three phases of the LHS cycle (knowledge, practice, and data) to support high-value care, using the topic of preventing and treating Clostridium difficile colitis as a relevant example to the hospital medicine field (Figure 2). By sharing this work, we hope it can serve as a model to illustrate how partnerships between organizations and AHRQ can lead to improvements in healthcare.

Knowledge: Identifying and Synthesizing Evidence to Address Clinical Challenges

Systematic reviews use carefully formulated questions to summarize the literature results using specific and established methods.¹⁴ Given that individual studies can have disparate results, it is critical to summarize and synthesize findings across studies, so we know what the overall evidence suggests, and whether we can be confident in the findings. To date, the EPC program has developed more than 500 evidence synthesis reports. An example relevant to the field of hospital medicine is the 2016 review that examined the effects of interventions to prevent and treat Clostridium difficile colitis in adults.¹⁵

The review examined the best available evidence, including data from randomized controlled trials and observational studies, on diagnosing, preventing, and treating Clostridium difficile colitis. Major findings included the following: vancomycin is more effective than metronidazole for treating the first occurrence of Clostridium difficile colitis (high-strength evidence), fecal transplantation may have a significant benefit in the treatment of recurrent Clostridium difficile colitis (low-strength evidence), and institutional preventive interventions such as antibiotic stewardship practices, transmission interruption through terminal room cleaning, and handwashing campaigns reduce the incidence of Clostridium difficile colitis (low-strength evidence). The report results provided the most recent review of the evidence and were particularly important as they suggested a need for significant practice changes in the treatment of Clostridium difficile colitis based on the new evidence available. Previous to this report, the 2010 guidelines from the Infectious Diseases Society of America (IDSA) recommended metronidazole over vancomycin for the treatment of the first occurrence of Clostridium difficile colitis.¹⁶ Subsequently, the newly released 2018 IDSA guideline provides recommendations consistent with the findings in this AHRQ report.¹⁷

Practice: Applying Knowledge in the Process of Care Delivery

AHRQ recognizes there are many interim steps between having the results from a systematic review and changing practice and improving care. In 2017, the EPC program began piloting approaches to make it easier for healthcare systems and hospitals to use its reports to improve the delivery of patient care and clinical outcomes. A pilot project conducted by the ECRI Institute - Penn Medicine EPC evaluated the feasibility of using an existing clinical pathway development and dissemination framework¹⁸ to translate findings from the 2016 AHRQ EPC report on Clostridium difficile colitis into a pathway for Clostridium difficile colitis treatment in the acute care setting.

To develop a Clostridium difficile colitis treatment pathway, the ECRI-Penn EPC team recruited a representative stakeholder group from Penn Medicine to review the EPC report as well as existing society guidelines. The clinical pathway was subsequently developed and approved by the stakeholders and disseminated through the Penn Medicine cloud-based pathways repository beginning on April 16, 2018.¹⁹ Most recently, the pathway became available in the electronic health record (EHR; 2018 Epic Systems Corporation) to facilitate provider review during care. Specifically, hyperlinks to the pathway are embedded within the ordering screens for those antibiotics used to treat Clostridium difficile colitis (ie, oral and rectal vancomycin, fidaxomicin, and metronidazole). Upon clicking the link in the ordering screen, the pathway launches a floating internet explorer window. The pathway is now publicly available on the AHRQ’s Clinical Decision Support (CDS) Connect Project (https://cds.ahrq.gov/), which is a resource to share pathway artifacts for other healthcare systems to use.

Data: Assessing Performance and Creating a Feedback Cycle for Learning and Improvement

The last step in the LHS cycle is to identify the impact of interventions on practice change and clinical outcomes, to understand how local results compare to peer institutions, and to inform future research and knowledge.

For the ECRI Institute-Penn Medicine EPC pilot project, both qualitative and quantitative outcomes were assessed. The initial qualitative analysis focused on the feasibility of using the AHRQ report in an existing pathway development and dissemination framework.¹⁸ It was found that clinical stakeholders identified the EPC report as trustworthy and more current than the society guidelines available at the time of development, particularly regarding the finding that vancomycin was more effective than metronidazole for the first occurrence of Clostridium difficile colitis. Additional qualitative analysis will be conducted to understand provider satisfaction with the pathway and practice impact. The quantitative analysis focused on pathway use (clicks over time) and found that as of September 16, 2018, the pathway had been viewed by providers 403 times. Future analysis will evaluate the impact of the pathway on the use of oral vancomycin for the first occurrences of Clostridium difficile colitis.

Patient registries can also help clinicians and healthcare systems to complete the feedback cycle and evaluate outcomes. Patient registries collect data from clinical and other sources in a standardized way in order to evaluate specific outcomes for various populations.²⁰ AHRQ has created a registry handbook, including best practices for how to create, operate, and evaluate registries.²⁰ This handbook enables the development of high-quality registries with data that can be leveraged for both research and improvement.

In the example of the ECRI Institute-Penn Medicine EPC pilot project, one way that a learning healthcare system, such as Penn Medicine, might measure the impact of the clinical pathway is to develop a quality improvement registry, which might be developed with information from their electronic health record, to examine the impact on the use of vancomycin for first occurrences of Clostridium difficile colitis. This information could help drive improvement in the implementation of the clinical pathway.

Registries can also be used as a source for research data. The NIH-funded American Gastroenterological Association (AGA) Fecal Microbiota Transplantation National Registry is an example of a research registry that collects data on outcomes and adverse events associated with fecal transplants to fill gaps in existing research. The 2016 AHRQ EPC review found low-strength evidence on fecal transplant for treatment of recurrent Clostridium difficile colitis. When designing the protocol for this registry, the researchers used the AHRQ handbook to inform the design. Given that this is a research registry, it can be used by researchers to examine trends and outcomes of fecal transplant to treat Clostridium difficile colitis. Publications that use the registry as its source of data may be used in future systematic reviews, thus completing the cycle of learning.

The EPC program recognizes that gaps remain in the evidence to practice translation process and that more support is needed. Some upcoming activities of the AHRQ EPC Program to address these gaps and make its evidence reports more actionable for healthcare systems include:

• Projects to Disseminate EPC Reports into Clinical Practice. In addition to the ECRI Institute - Penn Medicine EPC pilot dissemination project, other pilot projects are aimed at helping systems apply evidence to practice and include new ways to visualize evidence to make it more actionable and usable; creating other dissemination products, such as evidence summaries and presentations for decision makers; and other implementation tools, such as decision aids. These products and summary reports are available on the AHRQ Effective Health Care Program website at https://effectivehealthcare.ahrq.gov/topics/health-systems-use-evidence/overview.
• Healthcare Systems Stakeholder Panel. Starting in Fall 2018, the AHRQ EPC Program will be convening a panel of healthcare system leaders to help make its reports and products more useful and responsive to the needs of healthcare systems and promote the use of evidence in clinical practice.
• Rapid Evidence Products. AHRQ understands that healthcare systems need information rapidly and cannot wait a year or more for a traditional systematic review to be completed. Therefore, AHRQ is applying its methods work on rapid reviews^21-24 to pilot new report types that systematically identify and summarize the evidence quickly for healthcare systems and quality improvement efforts.²⁵
• Data Integration. Originally launched in 2012, the Systematic Review Data Repository (SRDR) is an AHRQ-supported online open-access repository of abstracted data from individual studies from systematic reviews. The goal is to enable more efficient updates of systematic reviews through data reuse. An updated version of the SRDR is scheduled to launch in 2020. With the new version, future sharing of summary data from systematic reviews digitally in a computable and portable format may allow integration into CDS tools and clinical practice guideline development and dissemination, facilitating the use of evidence in clinical practice.

The AHRQ EPC program supports initiatives to make evidence more actionable and provide resources and tools throughout all the phases of the learning healthcare system cycle. This case study on C. difficile is one example of how the EPC program is helping hospitals and healthcare systems improve clinical care delivery and its derivative value.

Disclosures

Dr. Umscheid reports grants from AHRQ, during the conduct of the study; serves on the Advisory Board of DynaMed, and founded and directed a hospital-based evidence-based practice center. All other authors have nothing to disclose.

Disclaimer

The findings and conclusions in this document are those of the author(s), who are responsible for its content, and do not necessarily represent the views of AHRQ. No statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.

Research evidence is critical for strengthening the value, quality, and safety of patient care. Learning healthcare systems (LHS) can support the delivery of evidence-based healthcare by establishing organizational processes that support three activities (Figure).^1-3

• Knowledge: Identifying and synthesizing evidence to address clinical challenges
• Practice: Applying knowledge in the process of care delivery
• Data: Assessing performance and creating a feedback cycle for learning and improvement

The systematic implementation of evidence into practice continues to be a challenge for many healthcare organizations^4-7 due to limited resources, expertise, and culture.^5,8-12 Missing opportunities for translating knowledge into practice not only results in low-value care (ie, waste) but also in harm.¹

The AHRQ (Agency for Healthcare Research and Quality) Evidence-based Practice Center (EPC) Program was established in 1997, with the goal of synthesizing research to inform evidence-based healthcare. The national impact of this program has been significant. Since the American Recovery and Reinvestment Act of 2009, EPC program reports have been used to inform over 95 clinical practice guidelines from societies such as the American College of Physicians, 16 health coverage decisions from payers such as the Centers for Medicare & Medicaid Services, and 24 government policies and program planning efforts, such as the National Institutes of Health Pathways to Prevention Program.¹³

The EPC program recognizes that evidence awareness is not sufficient to change practice and improve clinical outcomes. As such, the EPC program also embarked on initiatives to facilitate the translation of evidence into clinical practice and to measure and monitor how changes in practice impact health outcomes. AHRQ has historically worked with professional organizations to translate systematic reviews into clinical practice guidelines as well as federal agencies to inform payer decisions and program planning. Recently, the EPC program has increased collaborative efforts with hospitals and healthcare systems to understand how they use evidence and to partner with them to identify methods to improve the uptake of evidence into practice.^9,12

In this perspective, we describe the AHRQ EPC Program’s work to address the three phases of the LHS cycle (knowledge, practice, and data) to support high-value care, using the topic of preventing and treating Clostridium difficile colitis as a relevant example to the hospital medicine field (Figure 2). By sharing this work, we hope it can serve as a model to illustrate how partnerships between organizations and AHRQ can lead to improvements in healthcare.

Knowledge: Identifying and Synthesizing Evidence to Address Clinical Challenges

Systematic reviews use carefully formulated questions to summarize the literature results using specific and established methods.¹⁴ Given that individual studies can have disparate results, it is critical to summarize and synthesize findings across studies, so we know what the overall evidence suggests, and whether we can be confident in the findings. To date, the EPC program has developed more than 500 evidence synthesis reports. An example relevant to the field of hospital medicine is the 2016 review that examined the effects of interventions to prevent and treat Clostridium difficile colitis in adults.¹⁵

The review examined the best available evidence, including data from randomized controlled trials and observational studies, on diagnosing, preventing, and treating Clostridium difficile colitis. Major findings included the following: vancomycin is more effective than metronidazole for treating the first occurrence of Clostridium difficile colitis (high-strength evidence), fecal transplantation may have a significant benefit in the treatment of recurrent Clostridium difficile colitis (low-strength evidence), and institutional preventive interventions such as antibiotic stewardship practices, transmission interruption through terminal room cleaning, and handwashing campaigns reduce the incidence of Clostridium difficile colitis (low-strength evidence). The report results provided the most recent review of the evidence and were particularly important as they suggested a need for significant practice changes in the treatment of Clostridium difficile colitis based on the new evidence available. Previous to this report, the 2010 guidelines from the Infectious Diseases Society of America (IDSA) recommended metronidazole over vancomycin for the treatment of the first occurrence of Clostridium difficile colitis.¹⁶ Subsequently, the newly released 2018 IDSA guideline provides recommendations consistent with the findings in this AHRQ report.¹⁷

Practice: Applying Knowledge in the Process of Care Delivery

AHRQ recognizes there are many interim steps between having the results from a systematic review and changing practice and improving care. In 2017, the EPC program began piloting approaches to make it easier for healthcare systems and hospitals to use its reports to improve the delivery of patient care and clinical outcomes. A pilot project conducted by the ECRI Institute - Penn Medicine EPC evaluated the feasibility of using an existing clinical pathway development and dissemination framework¹⁸ to translate findings from the 2016 AHRQ EPC report on Clostridium difficile colitis into a pathway for Clostridium difficile colitis treatment in the acute care setting.

To develop a Clostridium difficile colitis treatment pathway, the ECRI-Penn EPC team recruited a representative stakeholder group from Penn Medicine to review the EPC report as well as existing society guidelines. The clinical pathway was subsequently developed and approved by the stakeholders and disseminated through the Penn Medicine cloud-based pathways repository beginning on April 16, 2018.¹⁹ Most recently, the pathway became available in the electronic health record (EHR; 2018 Epic Systems Corporation) to facilitate provider review during care. Specifically, hyperlinks to the pathway are embedded within the ordering screens for those antibiotics used to treat Clostridium difficile colitis (ie, oral and rectal vancomycin, fidaxomicin, and metronidazole). Upon clicking the link in the ordering screen, the pathway launches a floating internet explorer window. The pathway is now publicly available on the AHRQ’s Clinical Decision Support (CDS) Connect Project (https://cds.ahrq.gov/), which is a resource to share pathway artifacts for other healthcare systems to use.

Data: Assessing Performance and Creating a Feedback Cycle for Learning and Improvement

The last step in the LHS cycle is to identify the impact of interventions on practice change and clinical outcomes, to understand how local results compare to peer institutions, and to inform future research and knowledge.

For the ECRI Institute-Penn Medicine EPC pilot project, both qualitative and quantitative outcomes were assessed. The initial qualitative analysis focused on the feasibility of using the AHRQ report in an existing pathway development and dissemination framework.¹⁸ It was found that clinical stakeholders identified the EPC report as trustworthy and more current than the society guidelines available at the time of development, particularly regarding the finding that vancomycin was more effective than metronidazole for the first occurrence of Clostridium difficile colitis. Additional qualitative analysis will be conducted to understand provider satisfaction with the pathway and practice impact. The quantitative analysis focused on pathway use (clicks over time) and found that as of September 16, 2018, the pathway had been viewed by providers 403 times. Future analysis will evaluate the impact of the pathway on the use of oral vancomycin for the first occurrences of Clostridium difficile colitis.

Patient registries can also help clinicians and healthcare systems to complete the feedback cycle and evaluate outcomes. Patient registries collect data from clinical and other sources in a standardized way in order to evaluate specific outcomes for various populations.²⁰ AHRQ has created a registry handbook, including best practices for how to create, operate, and evaluate registries.²⁰ This handbook enables the development of high-quality registries with data that can be leveraged for both research and improvement.

In the example of the ECRI Institute-Penn Medicine EPC pilot project, one way that a learning healthcare system, such as Penn Medicine, might measure the impact of the clinical pathway is to develop a quality improvement registry, which might be developed with information from their electronic health record, to examine the impact on the use of vancomycin for first occurrences of Clostridium difficile colitis. This information could help drive improvement in the implementation of the clinical pathway.

Registries can also be used as a source for research data. The NIH-funded American Gastroenterological Association (AGA) Fecal Microbiota Transplantation National Registry is an example of a research registry that collects data on outcomes and adverse events associated with fecal transplants to fill gaps in existing research. The 2016 AHRQ EPC review found low-strength evidence on fecal transplant for treatment of recurrent Clostridium difficile colitis. When designing the protocol for this registry, the researchers used the AHRQ handbook to inform the design. Given that this is a research registry, it can be used by researchers to examine trends and outcomes of fecal transplant to treat Clostridium difficile colitis. Publications that use the registry as its source of data may be used in future systematic reviews, thus completing the cycle of learning.

The EPC program recognizes that gaps remain in the evidence to practice translation process and that more support is needed. Some upcoming activities of the AHRQ EPC Program to address these gaps and make its evidence reports more actionable for healthcare systems include:

• Projects to Disseminate EPC Reports into Clinical Practice. In addition to the ECRI Institute - Penn Medicine EPC pilot dissemination project, other pilot projects are aimed at helping systems apply evidence to practice and include new ways to visualize evidence to make it more actionable and usable; creating other dissemination products, such as evidence summaries and presentations for decision makers; and other implementation tools, such as decision aids. These products and summary reports are available on the AHRQ Effective Health Care Program website at https://effectivehealthcare.ahrq.gov/topics/health-systems-use-evidence/overview.
• Healthcare Systems Stakeholder Panel. Starting in Fall 2018, the AHRQ EPC Program will be convening a panel of healthcare system leaders to help make its reports and products more useful and responsive to the needs of healthcare systems and promote the use of evidence in clinical practice.
• Rapid Evidence Products. AHRQ understands that healthcare systems need information rapidly and cannot wait a year or more for a traditional systematic review to be completed. Therefore, AHRQ is applying its methods work on rapid reviews^21-24 to pilot new report types that systematically identify and summarize the evidence quickly for healthcare systems and quality improvement efforts.²⁵
• Data Integration. Originally launched in 2012, the Systematic Review Data Repository (SRDR) is an AHRQ-supported online open-access repository of abstracted data from individual studies from systematic reviews. The goal is to enable more efficient updates of systematic reviews through data reuse. An updated version of the SRDR is scheduled to launch in 2020. With the new version, future sharing of summary data from systematic reviews digitally in a computable and portable format may allow integration into CDS tools and clinical practice guideline development and dissemination, facilitating the use of evidence in clinical practice.

The AHRQ EPC program supports initiatives to make evidence more actionable and provide resources and tools throughout all the phases of the learning healthcare system cycle. This case study on C. difficile is one example of how the EPC program is helping hospitals and healthcare systems improve clinical care delivery and its derivative value.

Disclosures

Dr. Umscheid reports grants from AHRQ, during the conduct of the study; serves on the Advisory Board of DynaMed, and founded and directed a hospital-based evidence-based practice center. All other authors have nothing to disclose.

Disclaimer

The findings and conclusions in this document are those of the author(s), who are responsible for its content, and do not necessarily represent the views of AHRQ. No statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.

Hyperkalemia is common in hospitalized patients, with an estimated prevalence of 1%-10%.^1,2 Hyperkalemia can lead to life-threatening cardiac arrhythmias. The risk of arrhythmias increases with serum potassium values >6.5 mmol/L, and hyperkalemia is associated with increased in-hospital mortality.³ Treatment for hyperkalemia is indicated by a combination of the absolute serum potassium level, the rate of change of potassium level, and the presence of electrocardiogram abnormalities.

Intravenous insulin stimulates the sodium/potassium-ATP pump, leading to intracellular uptake of potassium. Recommendations vary regarding the optimal dosing of insulin and dextrose for the treatment of hyperkalemia.⁴

Hypoglycemia is a common complication following treatment of hyperkalemia with insulin/dextrose. The reported incidence in hospitalized patients ranges from 6% to 75% depending on the population studied, the doses of insulin/dextrose administered, and the definition of hypoglycemia.^5-8 Hypoglycemia itself is associated with increased morbidity and mortality in hospitalized patients.⁹

The aims of this study were to describe the incidence of hypoglycemia following hyperkalemia treatment with intravenous insulin/dextrose in inpatients in a large (900-bed) UK teaching hospital and to determine the risk factors predisposing to hypoglycemia.

We conducted a retrospective, single-center cohort study reviewing the Electronic Patient Records (EPR) of all adult (aged ≥18 years) inpatients (excluding critical care) prescribed treatment for hyperkalemia with intravenous insulin from January 1, 2013, to March 1, 2017. Local hyperkalemia treatment guidelines included administration of 10 units of insulin and 100 ml of 20% glucose intravenously in accordance with national guidelines.¹⁰ The study received local approval.

Episodes occurring before May 1, 2015, were excluded because modification to the hyperkalemia prescription care bundle was implemented in April 2015 recommending standardized simultaneous insulin and dextrose administration and hourly capillary blood glucose (CBG) measurement for six hours following treatment. Episodes where no dextrose was prescribed or administered (n = 435) or where no CBG value was recorded within six hours after treatment were excluded (n = 63). All patients included in the analysis received the same insulin/dextrose treatment confirmed by electronic signature of the prescription.

Data extracted included patient demographics, laboratory values, and treatment and administration details. Pretreatment and posttreatment potassium measurements were taken within four hours before and after insulin/dextrose administration, respectively. Serum creatinine and estimated glomerular filtration rate (eGFR) measurements were taken within six hours prior to treatment. Pretreatment CBG levels were measured within two hours of insulin/dextrose administration, and the lowest value within six hours after treatment was used for the analysis. We collected data on length of stay and mortality during one-year follow-up.

Hypoglycemia and severe hypoglycemia were defined as CBG ≤3.9 mmol/l (70 mg/dL) and <3.0 mmol/l (54 mg/dL) in line with definitions used in the National Diabetes Inpatient Audit.¹¹

Descriptive statistics are reported as mean (±SD) or median (interquartile range [IQR]) values for continuous data or numbers and percentages for categorical data. All P values are two-tailed, and P values <.05 were considered to indicate statistical significance. Chi-squared test and Student t test were used to assess differences for categorical and continuous variables between groups. The statistical analysis was performed using the SPSS software, version 25 (IBM).

A total of 662 episodes of hyperkalemia treatment with insulin/dextrose were included in the analysis. These episodes occurred over 445 admissions in 415 individuals. The median number of treatments/patient admission was 1.0 (range 1-11); 108 patients received more than one insulin/dextrose treatment during their admission. Mean pretreatment serum potassium level was 6.4 ± 0.5 mmol/l, and treatment reduced the serum potassium level by 0.6 ± 0.6 mmol/l.

Patient Demographics

Median age of the patients was 67 years (IQR 55.0-79.0), and 39.3% of episodes occurred in females (Table 1). Median weight of the patients was 76.6 kg (IQR 62.1-95.0). Diabetes was present in 31.1% of episodes. Renal impairment was common, with median creatinine levels being 166 µmol/l (IQR 113-256) and 1.9 mg/dL (IQR 1.3-2.9) and eGFR being 29.0 ml/min/1.73 m² (IQR 19.0-45.0), and 11% of episodes occurred in patients requiring acute or chronic dialysis. Median length of stay was 19.5 days (IQR 9.8-49.1). Inpatient mortality was 13%, and one-year mortality was 19.4%.

Incidence

Hypoglycemia occurred following 116 of 662 hyperkalemia treatments administered (17.5%), and severe hypoglycemia occurred after 47 of 662 treatments (7.1%).

Risk Factors

The median age of patients with hypoglycemia was significantly greater than that of patients without hypoglycemia (71.0 years [54.8-83.5] vs 67.0 years [55.0-77.0]; P = .023) (Table 2). There were no significant differences in gender, degree of renal impairment, or requirement for renal replacement therapy between the groups with and without hypoglycemia.

Hypoglycemia occurred in patients who were, on average, 15 kg lighter than those who did not have hypoglycemia (median body weight 66.1 kg [55.4-72.5] vs 81.0 kg [63.1-96.0]; P < .001).

Pretreatment CBG was lower in those who had hypoglycemia following treatment, the levels being 5.8 mmol/l (5.0-7.3), 104 mg/dL (90-131) vs 8.7 mmol/l (6.4-11.4), 157 mg/dL (115-205; P < .001).

There was a nonsignificant trend toward an increased prevalence of diabetes in patients without hypoglycemia (32.6% vs 24.1%; P = .074).

This study reports the incidence of iatrogenic hypoglycemia following intravenous insulin treatment for hyperkalemia in a large cohort of general medical and surgical inpatients and describes the risk factors predisposing to this important complication.

The incidence rates of hypoglycemia and severe hypoglycemia in our study were 17.5% and 7.1%, respectively. These rates are greater than previously observed in a smaller US study undertaken in a similar population, which reported an incidence of 8.7% of hypoglycemia and 2.3% of severe hypoglycemia, although it included five different insulin/dextrose prescriptions.⁸ A similar incidence of hypoglycemia (17%) was reported in patients treated for hyperkalemia in an Emergency Department in the United States.¹²

Variables that increased the risk of hypoglycemia in the present study included older age, lower body weight, and lower pretreatment CBG level. These risk factors have been reported in previous studies, although inconsistently. Pretreatment CBG is an important predictor of hypoglycemia following treatment for hyperkalemia and has been observed in patients in the emergency department¹² and in patients with renal impairment.^6,7 In the present study, lower body weight was observed in patients with hypoglycemia compared with those without hypoglycemia. Weight-based insulin dosing (0.1 units/kg) for hyperkalemia has been associated with less hypoglycemia compared with fixed insulin doses (10 units) without affecting the potassium-lowering effect.¹³

The degree of renal impairment did not affect the risk of hypoglycemia. Chronic kidney disease is associated with increased insulin resistance, which may attenuate the hypoglycemic response to insulin.¹⁴ Patients with renal impairment may experience delayed hypoglycemia, which may not have been captured although the posttreatment blood glucose values extended to six hours.

The increased risk of hypoglycemia in older patients treated for hyperkalemia is of concern given the lack of counterregulatory response and reduced symptoms of hypoglycemia in older adults. This association has not been reported in other studies;^8,12 however, the average age of subjects in our cohort was higher than that in other studies (67 vs 57 years). Although age did not correlate with weight, older adults may have reduced carbohydrate intake and mild renal impairment affecting insulin clearance.

Hospitalized patients treated for hyperkalemia in the present study had a greater inpatient mortality rate (13%) than the general inpatient population at the same hospital (3%).¹⁵ Hyperkalemia often occurs in individuals with comorbidities and is associated with an increased risk of all-cause mortality^.3

Mean pretreatment potassium level was 6.4 mmol/l in this study. Evidence-based criteria for treatment thresholds are lacking. Acute potassium increases are associated with cardiac mortality, whereas elevated potassium levels in patients with chronic kidney disease taking renin-angiotensin system drugs are often not treated as emergencies unless significant electrocardiogram (ECG) changes are apparent. It is likely that some hyperkalemia treatments are unnecessary, which is important given the high risk of treatment-related complications.

To our knowledge, this is the largest analysis of hypoglycemic episodes following treatment of hyperkalemia in medical and surgical inpatients. This is a single-center study, thereby limiting its generalizability; however, the patient characteristics are likely similar to those reported from other large, urban institutions.

Treatment prescriptions in our study were consistent due to the electronic prescribing care bundle, allowing us to compare the risk factors for hypoglycemia with standardized prescriptions. Point-of-care CBG measurements can be less accurate at low blood glucose levels; however, laboratory glucose levels are obtained much less frequently, underestimating incidence, due to the need for prompt hypoglycemia treatment without delaying for laboratory glucose measurement.

The dataset was not complete and depended on healthcare professionals entering some data. We did not assess the clinical consequences of hypoglycemia.

An increasing proportion of hospitals are utilizing Electronic Prescribing Systems with the potential to improve patient safety by standardizing prescriptions. However, despite standardized prescriptions, 37% of prescriptions for concurrent dextrose were not administered with insulin and 8.6% of patients had no CBG monitoring within six hours after insulin administration.

We propose to integrate the risk factors identified in this study into a decision support tool embedded within electronic prescriptions and medication administration. This will auto-populate with data from the EPR and identify individuals at high risk of hypoglycemia following hyperkalemia treatment. The decision support tool will then advise a prescription with a higher volume of dextrose and/or lower insulin dose to mitigate this risk.

Hyperkalemia treatment with insulin was associated with high incidence of hypoglycemia. Decision support tools highlighting individuals at high risk of hypoglycemia may reduce this incidence.

Disclosures

The authors have nothing to disclose.

Hyperkalemia is common in hospitalized patients, with an estimated prevalence of 1%-10%.^1,2 Hyperkalemia can lead to life-threatening cardiac arrhythmias. The risk of arrhythmias increases with serum potassium values >6.5 mmol/L, and hyperkalemia is associated with increased in-hospital mortality.³ Treatment for hyperkalemia is indicated by a combination of the absolute serum potassium level, the rate of change of potassium level, and the presence of electrocardiogram abnormalities.

Intravenous insulin stimulates the sodium/potassium-ATP pump, leading to intracellular uptake of potassium. Recommendations vary regarding the optimal dosing of insulin and dextrose for the treatment of hyperkalemia.⁴

Hypoglycemia is a common complication following treatment of hyperkalemia with insulin/dextrose. The reported incidence in hospitalized patients ranges from 6% to 75% depending on the population studied, the doses of insulin/dextrose administered, and the definition of hypoglycemia.^5-8 Hypoglycemia itself is associated with increased morbidity and mortality in hospitalized patients.⁹

The aims of this study were to describe the incidence of hypoglycemia following hyperkalemia treatment with intravenous insulin/dextrose in inpatients in a large (900-bed) UK teaching hospital and to determine the risk factors predisposing to hypoglycemia.

We conducted a retrospective, single-center cohort study reviewing the Electronic Patient Records (EPR) of all adult (aged ≥18 years) inpatients (excluding critical care) prescribed treatment for hyperkalemia with intravenous insulin from January 1, 2013, to March 1, 2017. Local hyperkalemia treatment guidelines included administration of 10 units of insulin and 100 ml of 20% glucose intravenously in accordance with national guidelines.¹⁰ The study received local approval.

Episodes occurring before May 1, 2015, were excluded because modification to the hyperkalemia prescription care bundle was implemented in April 2015 recommending standardized simultaneous insulin and dextrose administration and hourly capillary blood glucose (CBG) measurement for six hours following treatment. Episodes where no dextrose was prescribed or administered (n = 435) or where no CBG value was recorded within six hours after treatment were excluded (n = 63). All patients included in the analysis received the same insulin/dextrose treatment confirmed by electronic signature of the prescription.

Data extracted included patient demographics, laboratory values, and treatment and administration details. Pretreatment and posttreatment potassium measurements were taken within four hours before and after insulin/dextrose administration, respectively. Serum creatinine and estimated glomerular filtration rate (eGFR) measurements were taken within six hours prior to treatment. Pretreatment CBG levels were measured within two hours of insulin/dextrose administration, and the lowest value within six hours after treatment was used for the analysis. We collected data on length of stay and mortality during one-year follow-up.

Hypoglycemia and severe hypoglycemia were defined as CBG ≤3.9 mmol/l (70 mg/dL) and <3.0 mmol/l (54 mg/dL) in line with definitions used in the National Diabetes Inpatient Audit.¹¹

Descriptive statistics are reported as mean (±SD) or median (interquartile range [IQR]) values for continuous data or numbers and percentages for categorical data. All P values are two-tailed, and P values <.05 were considered to indicate statistical significance. Chi-squared test and Student t test were used to assess differences for categorical and continuous variables between groups. The statistical analysis was performed using the SPSS software, version 25 (IBM).

A total of 662 episodes of hyperkalemia treatment with insulin/dextrose were included in the analysis. These episodes occurred over 445 admissions in 415 individuals. The median number of treatments/patient admission was 1.0 (range 1-11); 108 patients received more than one insulin/dextrose treatment during their admission. Mean pretreatment serum potassium level was 6.4 ± 0.5 mmol/l, and treatment reduced the serum potassium level by 0.6 ± 0.6 mmol/l.

Patient Demographics

Median age of the patients was 67 years (IQR 55.0-79.0), and 39.3% of episodes occurred in females (Table 1). Median weight of the patients was 76.6 kg (IQR 62.1-95.0). Diabetes was present in 31.1% of episodes. Renal impairment was common, with median creatinine levels being 166 µmol/l (IQR 113-256) and 1.9 mg/dL (IQR 1.3-2.9) and eGFR being 29.0 ml/min/1.73 m² (IQR 19.0-45.0), and 11% of episodes occurred in patients requiring acute or chronic dialysis. Median length of stay was 19.5 days (IQR 9.8-49.1). Inpatient mortality was 13%, and one-year mortality was 19.4%.

Incidence

Hypoglycemia occurred following 116 of 662 hyperkalemia treatments administered (17.5%), and severe hypoglycemia occurred after 47 of 662 treatments (7.1%).

Risk Factors

The median age of patients with hypoglycemia was significantly greater than that of patients without hypoglycemia (71.0 years [54.8-83.5] vs 67.0 years [55.0-77.0]; P = .023) (Table 2). There were no significant differences in gender, degree of renal impairment, or requirement for renal replacement therapy between the groups with and without hypoglycemia.

Hypoglycemia occurred in patients who were, on average, 15 kg lighter than those who did not have hypoglycemia (median body weight 66.1 kg [55.4-72.5] vs 81.0 kg [63.1-96.0]; P < .001).

Pretreatment CBG was lower in those who had hypoglycemia following treatment, the levels being 5.8 mmol/l (5.0-7.3), 104 mg/dL (90-131) vs 8.7 mmol/l (6.4-11.4), 157 mg/dL (115-205; P < .001).

There was a nonsignificant trend toward an increased prevalence of diabetes in patients without hypoglycemia (32.6% vs 24.1%; P = .074).

This study reports the incidence of iatrogenic hypoglycemia following intravenous insulin treatment for hyperkalemia in a large cohort of general medical and surgical inpatients and describes the risk factors predisposing to this important complication.

The incidence rates of hypoglycemia and severe hypoglycemia in our study were 17.5% and 7.1%, respectively. These rates are greater than previously observed in a smaller US study undertaken in a similar population, which reported an incidence of 8.7% of hypoglycemia and 2.3% of severe hypoglycemia, although it included five different insulin/dextrose prescriptions.⁸ A similar incidence of hypoglycemia (17%) was reported in patients treated for hyperkalemia in an Emergency Department in the United States.¹²

Variables that increased the risk of hypoglycemia in the present study included older age, lower body weight, and lower pretreatment CBG level. These risk factors have been reported in previous studies, although inconsistently. Pretreatment CBG is an important predictor of hypoglycemia following treatment for hyperkalemia and has been observed in patients in the emergency department¹² and in patients with renal impairment.^6,7 In the present study, lower body weight was observed in patients with hypoglycemia compared with those without hypoglycemia. Weight-based insulin dosing (0.1 units/kg) for hyperkalemia has been associated with less hypoglycemia compared with fixed insulin doses (10 units) without affecting the potassium-lowering effect.¹³

The degree of renal impairment did not affect the risk of hypoglycemia. Chronic kidney disease is associated with increased insulin resistance, which may attenuate the hypoglycemic response to insulin.¹⁴ Patients with renal impairment may experience delayed hypoglycemia, which may not have been captured although the posttreatment blood glucose values extended to six hours.

The increased risk of hypoglycemia in older patients treated for hyperkalemia is of concern given the lack of counterregulatory response and reduced symptoms of hypoglycemia in older adults. This association has not been reported in other studies;^8,12 however, the average age of subjects in our cohort was higher than that in other studies (67 vs 57 years). Although age did not correlate with weight, older adults may have reduced carbohydrate intake and mild renal impairment affecting insulin clearance.

Hospitalized patients treated for hyperkalemia in the present study had a greater inpatient mortality rate (13%) than the general inpatient population at the same hospital (3%).¹⁵ Hyperkalemia often occurs in individuals with comorbidities and is associated with an increased risk of all-cause mortality^.3

Mean pretreatment potassium level was 6.4 mmol/l in this study. Evidence-based criteria for treatment thresholds are lacking. Acute potassium increases are associated with cardiac mortality, whereas elevated potassium levels in patients with chronic kidney disease taking renin-angiotensin system drugs are often not treated as emergencies unless significant electrocardiogram (ECG) changes are apparent. It is likely that some hyperkalemia treatments are unnecessary, which is important given the high risk of treatment-related complications.

To our knowledge, this is the largest analysis of hypoglycemic episodes following treatment of hyperkalemia in medical and surgical inpatients. This is a single-center study, thereby limiting its generalizability; however, the patient characteristics are likely similar to those reported from other large, urban institutions.

Treatment prescriptions in our study were consistent due to the electronic prescribing care bundle, allowing us to compare the risk factors for hypoglycemia with standardized prescriptions. Point-of-care CBG measurements can be less accurate at low blood glucose levels; however, laboratory glucose levels are obtained much less frequently, underestimating incidence, due to the need for prompt hypoglycemia treatment without delaying for laboratory glucose measurement.

The dataset was not complete and depended on healthcare professionals entering some data. We did not assess the clinical consequences of hypoglycemia.

An increasing proportion of hospitals are utilizing Electronic Prescribing Systems with the potential to improve patient safety by standardizing prescriptions. However, despite standardized prescriptions, 37% of prescriptions for concurrent dextrose were not administered with insulin and 8.6% of patients had no CBG monitoring within six hours after insulin administration.

We propose to integrate the risk factors identified in this study into a decision support tool embedded within electronic prescriptions and medication administration. This will auto-populate with data from the EPR and identify individuals at high risk of hypoglycemia following hyperkalemia treatment. The decision support tool will then advise a prescription with a higher volume of dextrose and/or lower insulin dose to mitigate this risk.

Hyperkalemia treatment with insulin was associated with high incidence of hypoglycemia. Decision support tools highlighting individuals at high risk of hypoglycemia may reduce this incidence.

Disclosures

The authors have nothing to disclose.

Hyperkalemia is common in hospitalized patients, with an estimated prevalence of 1%-10%.^1,2 Hyperkalemia can lead to life-threatening cardiac arrhythmias. The risk of arrhythmias increases with serum potassium values >6.5 mmol/L, and hyperkalemia is associated with increased in-hospital mortality.³ Treatment for hyperkalemia is indicated by a combination of the absolute serum potassium level, the rate of change of potassium level, and the presence of electrocardiogram abnormalities.

Intravenous insulin stimulates the sodium/potassium-ATP pump, leading to intracellular uptake of potassium. Recommendations vary regarding the optimal dosing of insulin and dextrose for the treatment of hyperkalemia.⁴

Hypoglycemia is a common complication following treatment of hyperkalemia with insulin/dextrose. The reported incidence in hospitalized patients ranges from 6% to 75% depending on the population studied, the doses of insulin/dextrose administered, and the definition of hypoglycemia.^5-8 Hypoglycemia itself is associated with increased morbidity and mortality in hospitalized patients.⁹

The aims of this study were to describe the incidence of hypoglycemia following hyperkalemia treatment with intravenous insulin/dextrose in inpatients in a large (900-bed) UK teaching hospital and to determine the risk factors predisposing to hypoglycemia.

We conducted a retrospective, single-center cohort study reviewing the Electronic Patient Records (EPR) of all adult (aged ≥18 years) inpatients (excluding critical care) prescribed treatment for hyperkalemia with intravenous insulin from January 1, 2013, to March 1, 2017. Local hyperkalemia treatment guidelines included administration of 10 units of insulin and 100 ml of 20% glucose intravenously in accordance with national guidelines.¹⁰ The study received local approval.

Episodes occurring before May 1, 2015, were excluded because modification to the hyperkalemia prescription care bundle was implemented in April 2015 recommending standardized simultaneous insulin and dextrose administration and hourly capillary blood glucose (CBG) measurement for six hours following treatment. Episodes where no dextrose was prescribed or administered (n = 435) or where no CBG value was recorded within six hours after treatment were excluded (n = 63). All patients included in the analysis received the same insulin/dextrose treatment confirmed by electronic signature of the prescription.

Data extracted included patient demographics, laboratory values, and treatment and administration details. Pretreatment and posttreatment potassium measurements were taken within four hours before and after insulin/dextrose administration, respectively. Serum creatinine and estimated glomerular filtration rate (eGFR) measurements were taken within six hours prior to treatment. Pretreatment CBG levels were measured within two hours of insulin/dextrose administration, and the lowest value within six hours after treatment was used for the analysis. We collected data on length of stay and mortality during one-year follow-up.

Hypoglycemia and severe hypoglycemia were defined as CBG ≤3.9 mmol/l (70 mg/dL) and <3.0 mmol/l (54 mg/dL) in line with definitions used in the National Diabetes Inpatient Audit.¹¹

Descriptive statistics are reported as mean (±SD) or median (interquartile range [IQR]) values for continuous data or numbers and percentages for categorical data. All P values are two-tailed, and P values <.05 were considered to indicate statistical significance. Chi-squared test and Student t test were used to assess differences for categorical and continuous variables between groups. The statistical analysis was performed using the SPSS software, version 25 (IBM).

A total of 662 episodes of hyperkalemia treatment with insulin/dextrose were included in the analysis. These episodes occurred over 445 admissions in 415 individuals. The median number of treatments/patient admission was 1.0 (range 1-11); 108 patients received more than one insulin/dextrose treatment during their admission. Mean pretreatment serum potassium level was 6.4 ± 0.5 mmol/l, and treatment reduced the serum potassium level by 0.6 ± 0.6 mmol/l.

Patient Demographics

Median age of the patients was 67 years (IQR 55.0-79.0), and 39.3% of episodes occurred in females (Table 1). Median weight of the patients was 76.6 kg (IQR 62.1-95.0). Diabetes was present in 31.1% of episodes. Renal impairment was common, with median creatinine levels being 166 µmol/l (IQR 113-256) and 1.9 mg/dL (IQR 1.3-2.9) and eGFR being 29.0 ml/min/1.73 m² (IQR 19.0-45.0), and 11% of episodes occurred in patients requiring acute or chronic dialysis. Median length of stay was 19.5 days (IQR 9.8-49.1). Inpatient mortality was 13%, and one-year mortality was 19.4%.

Incidence

Hypoglycemia occurred following 116 of 662 hyperkalemia treatments administered (17.5%), and severe hypoglycemia occurred after 47 of 662 treatments (7.1%).

Risk Factors

The median age of patients with hypoglycemia was significantly greater than that of patients without hypoglycemia (71.0 years [54.8-83.5] vs 67.0 years [55.0-77.0]; P = .023) (Table 2). There were no significant differences in gender, degree of renal impairment, or requirement for renal replacement therapy between the groups with and without hypoglycemia.

Hypoglycemia occurred in patients who were, on average, 15 kg lighter than those who did not have hypoglycemia (median body weight 66.1 kg [55.4-72.5] vs 81.0 kg [63.1-96.0]; P < .001).

Pretreatment CBG was lower in those who had hypoglycemia following treatment, the levels being 5.8 mmol/l (5.0-7.3), 104 mg/dL (90-131) vs 8.7 mmol/l (6.4-11.4), 157 mg/dL (115-205; P < .001).

There was a nonsignificant trend toward an increased prevalence of diabetes in patients without hypoglycemia (32.6% vs 24.1%; P = .074).

This study reports the incidence of iatrogenic hypoglycemia following intravenous insulin treatment for hyperkalemia in a large cohort of general medical and surgical inpatients and describes the risk factors predisposing to this important complication.

The incidence rates of hypoglycemia and severe hypoglycemia in our study were 17.5% and 7.1%, respectively. These rates are greater than previously observed in a smaller US study undertaken in a similar population, which reported an incidence of 8.7% of hypoglycemia and 2.3% of severe hypoglycemia, although it included five different insulin/dextrose prescriptions.⁸ A similar incidence of hypoglycemia (17%) was reported in patients treated for hyperkalemia in an Emergency Department in the United States.¹²

Variables that increased the risk of hypoglycemia in the present study included older age, lower body weight, and lower pretreatment CBG level. These risk factors have been reported in previous studies, although inconsistently. Pretreatment CBG is an important predictor of hypoglycemia following treatment for hyperkalemia and has been observed in patients in the emergency department¹² and in patients with renal impairment.^6,7 In the present study, lower body weight was observed in patients with hypoglycemia compared with those without hypoglycemia. Weight-based insulin dosing (0.1 units/kg) for hyperkalemia has been associated with less hypoglycemia compared with fixed insulin doses (10 units) without affecting the potassium-lowering effect.¹³

The degree of renal impairment did not affect the risk of hypoglycemia. Chronic kidney disease is associated with increased insulin resistance, which may attenuate the hypoglycemic response to insulin.¹⁴ Patients with renal impairment may experience delayed hypoglycemia, which may not have been captured although the posttreatment blood glucose values extended to six hours.

The increased risk of hypoglycemia in older patients treated for hyperkalemia is of concern given the lack of counterregulatory response and reduced symptoms of hypoglycemia in older adults. This association has not been reported in other studies;^8,12 however, the average age of subjects in our cohort was higher than that in other studies (67 vs 57 years). Although age did not correlate with weight, older adults may have reduced carbohydrate intake and mild renal impairment affecting insulin clearance.

Hospitalized patients treated for hyperkalemia in the present study had a greater inpatient mortality rate (13%) than the general inpatient population at the same hospital (3%).¹⁵ Hyperkalemia often occurs in individuals with comorbidities and is associated with an increased risk of all-cause mortality^.3

Mean pretreatment potassium level was 6.4 mmol/l in this study. Evidence-based criteria for treatment thresholds are lacking. Acute potassium increases are associated with cardiac mortality, whereas elevated potassium levels in patients with chronic kidney disease taking renin-angiotensin system drugs are often not treated as emergencies unless significant electrocardiogram (ECG) changes are apparent. It is likely that some hyperkalemia treatments are unnecessary, which is important given the high risk of treatment-related complications.

To our knowledge, this is the largest analysis of hypoglycemic episodes following treatment of hyperkalemia in medical and surgical inpatients. This is a single-center study, thereby limiting its generalizability; however, the patient characteristics are likely similar to those reported from other large, urban institutions.

Treatment prescriptions in our study were consistent due to the electronic prescribing care bundle, allowing us to compare the risk factors for hypoglycemia with standardized prescriptions. Point-of-care CBG measurements can be less accurate at low blood glucose levels; however, laboratory glucose levels are obtained much less frequently, underestimating incidence, due to the need for prompt hypoglycemia treatment without delaying for laboratory glucose measurement.

The dataset was not complete and depended on healthcare professionals entering some data. We did not assess the clinical consequences of hypoglycemia.

An increasing proportion of hospitals are utilizing Electronic Prescribing Systems with the potential to improve patient safety by standardizing prescriptions. However, despite standardized prescriptions, 37% of prescriptions for concurrent dextrose were not administered with insulin and 8.6% of patients had no CBG monitoring within six hours after insulin administration.

We propose to integrate the risk factors identified in this study into a decision support tool embedded within electronic prescriptions and medication administration. This will auto-populate with data from the EPR and identify individuals at high risk of hypoglycemia following hyperkalemia treatment. The decision support tool will then advise a prescription with a higher volume of dextrose and/or lower insulin dose to mitigate this risk.

Hyperkalemia treatment with insulin was associated with high incidence of hypoglycemia. Decision support tools highlighting individuals at high risk of hypoglycemia may reduce this incidence.

Disclosures

The authors have nothing to disclose.

Computed tomography pulmonary angiography (CTPA) is often used in the evaluation of suspected pulmonary embolism (PE). The detection of incidental findings that require follow-up is common; in just over 50% of cases, those incidental findings are pulmonary nodules.¹ Although the majority of these nodules are benign, Fleischner Society guidelines² recommend that patients with nodules at high risk for malignancy should undergo follow-up CT imaging within 3-12 months, with patients who smoke and have large nodules requiring closer follow up.

The failure to follow-up on abnormal test results is known to contribute to diagnostic error and can lead to patient harm.³ We sought to determine the proportion of high-risk pulmonary nodules on CTPA which did not undergo the recommended follow-up imaging.

Study Setting and Design

This retrospective cohort study included all patients who underwent CTPA in the emergency department (ED) and inpatient settings at three academic health centers (Mount Sinai Hospital, Toronto General Hospital, and Toronto Western Hospital) in Toronto, Canada between September 1, 2014, and August 31, 2015.

We examined the proportion of patients with pulmonary nodules requiring follow up who received repeat CT imaging within six weeks of the time frame recommended by the radiologist. Since we were interested in measuring the rate of an important test result that is missed (rather than accuracy of the test itself), we defined “requiring follow up” as the inclusion of explicit recommendations for follow up in the radiology report.

Montage (Philadelphia, Pennsylvania), a natural language processing software, was applied to a linked radiology information system (RIS) to identify all CTPAs that contained pulmonary nodules. We conducted manual chart review to confirm software accuracy. We initially searched the RIS for all CTPAs that were completed within the study period, resulting in the identification of 1932 imaging studies. Following a review of these 1,932 studies, we excluded 22 as they were not CTPAs. We then applied the search term, “nodule-” to 1,910 confirmed CTPAs, resulting in the identification of 836 imaging studies. Following a review of these 836 studies, we excluded 10 as they were duplicate studies. We also excluded 152 studies where the term “nodule-” did not identify a pulmonary nodule but instead referred to a radiologist reporting the absence of pulmonary nodules (eg “there were no pulmonary nodules found”) or the presence of non-lung nodules (eg thyroid nodules). This resulted in the identification of 674 CTPAs containing pulmonary nodules (Figure 1).

Thereafter, we generated a cohort with possible new lung malignancy eligible for follow-up imaging by reviewing available health records and applying the following prespecified exclusion criteria: (1) patients who died, (2) left against medical advice, (3) were critically ill during the follow-up period, (4) lived outside the hospital catchment area (Greater Toronto Area), (5) were seen in the outpatient setting, (6) identified as palliative, (7) had an active malignancy, (8) had nodules that were already being followed, or (9) had nodules with characteristics suggestive of alternate diagnoses to lung malignancy (such as infection or inflammation) with no follow up recommended as reported by the radiologist. For patients with multiple CTPAs, we included only the first study. For each eligible patient, we determined whether follow-up imaging was completed by manually reviewing the linked RIS. We reviewed available health records to determine whether the pulmonary nodule findings had been discussed with the patient and whether the patient had attended an outpatient follow-up visit. In patients for whom recommended follow-up imaging was not confirmed, we notified the ordering physician by e-mail.

Each radiology department followed the same protocol adherent to the 2005 Fleischner guidelines for identifying nodules requiring follow up.² Virtually all CTPAs at the three study institutions are read and reported within 72 hours. The ordering physician is sometimes called at the discretion of the reading radiologist when the findings are judged to be urgent and time-sensitive in nature. For example, the ordering physician may be contacted if a CTPA is positive for segmental PE but is not typically called for incidental pulmonary nodules. It is not common practice for ordering physicians to be notified of incidental findings above and beyond the radiology report. Primary care physicians are not typically copied on radiology reports and usually do not use the same electronic health record.

Statistical Analysis

We calculated simple descriptive statistics for all results. Mean values were compared using two-tailed t-tests, categorical groups using chi-square tests, and median values using Mann-Whitney U tests. We performed all analyses using Microsoft Excel version 16.14.1 (Redmond, Washington).

Ethics Approval

This study was approved by each institution’s research ethics board.

Follow Up of Incidental High-Risk Pulmonary Nodules

Of the 1910 CTPAs performed over the study period (Figure), 674 (35.3%) contained pulmonary nodules. Of the 259 patients with new pulmonary nodules eligible for follow-up imaging, 194 (74.9%) did not have an explicit suggestion for follow up by the radiologist. Ninety-five percent of radiologists (184 out of 194) provided an explanation for not recommending follow up in the radiology report; the two most common reasons were small nodule size (often described as “tiny”) and no interval change compared with the prior imaging study.² Of the 65 patients who did receive an explicit suggestion for follow up by radiology, 35 (53.8%) did not receive repeat imaging within the recommended time frame, allowing for a six-week grace period. Of these 35 patients, 10 eventually went on to receive delayed repeat imaging. The median follow-up time for the 30 patients who received timely repeat imaging was four months (IQR 2-6 months); in contrast, the median follow-up time for the 10 patients who received delayed repeat imaging was seven months (IQR 6-8 months), P = .01.

Of the 65 patients for whom follow up was recommended, the medical record showed evidence that there was a discussion between the medical team and the patient regarding patient preference for or against follow up in 55.4% (36 out of 65) of the patients. Notably, all 36 patients showed interest in receiving follow up; no patient indicated a preference for no follow up.

Furthermore, of the 65 patients that had follow up recommended, two patients were eventually diagnosed with lung cancer (one via lung biopsy, the other via positron emission tomography imaging); both patients did not receive timely follow-up imaging. While we did not include nodule size as an exclusion criterion, not one of the 65 patients included in the final cohort had nodules larger than 3 cm.

Physician Notification

In circumstances where we could not confirm that followed up had occurred, we notified the ordering physician by e-mail. Since 10 of the 35 patients who did not receive timely follow-up imaging went on to receive delayed repeat imaging, we notified 25 physicians. Of the 25 physicians that we e-mailed, 24 acknowledged receipt of the information. Of these 24 physicians, 14 reported conducting a detailed review of the chart, from which the following additional information was obtained: one patient expired, and five physicians notified the corresponding primary care physicians (two of whom were unaware of the nodule, and subsequently arranged further follow up with the patient).

Characteristics Associated with Timely Follow Up

Explicit mention that follow up was required in the discharge summary (P = .03), attending an outpatient follow-up visit (P < .001), and younger age (P = .03) were associated with receiving timely follow up; patient sex, smoking history, history of chronic obstructive pulmonary disease, lung nodule count, recommended follow-up time, and hospital department (defined as the discharging service) were not (Table).

In this multicenter cohort study, over 50% of patients with new high-risk pulmonary nodules detected incidentally on CTPA did not receive timely follow-up imaging. Including follow-up recommendations in the discharge summary, attending an outpatient follow-up visit, and younger age were associated with timely follow-up imaging.

Few studies have assessed the follow up of incidental nodules identified on CTPA. In a retrospective cohort study of ED patients in the United States, Blagev et al. found that only 29% received timely follow up.⁴ Our study contributes to the literature in several ways. First, our study included all hospitalized patients, not only those in the ED. Notably, most of our cohort were inpatients, a group of patients not previously described. Second, we examined factors associated with timely follow up, which may help to inform future quality improvement initiatives and interventions. Third, we included data from three different hospitals, which may improve generalization. Lastly, our study draws on contemporary Canadian data. Most of the studies investigating test result follow up have been conducted in the US^5,6 and Europe,⁷ with few empirical studies describing this phenomenon within the Canadian healthcare setting. We believe that our work contributes to the existing evidence that missed test results occur across diverse healthcare systems and have yet to be solved.^5-7

Our study had limitations. First, we defined follow up as repeat imaging and did not include office visits or biopsy in this definition. Second, we may have missed repeat imaging and outpatient follow-up visits that occurred outside the study hospitals. Although we were able to determine if repeat imaging and outpatient follow-up visits (eg, pulmonology or thoracic surgery clinics) had occurred within the study hospitals, we did not have access to follow-up encounters that occurred outside of the study hospitals (eg primary care clinics). We are unaware of any published regional data on the rate of outpatient follow up at the index facility following discharge. However, we know from provincial data of patients discharged from the ED with a new cardiac diagnosis that just under half are seen by a family physician, cardiologist, or internist within seven days, with just under 80% seen within 30 days.⁸ Third, although we attempted to capture patient preference for or against repeat imaging using chart review, the absence of documentation of patient preference did not confirm that a discussion regarding patient preferences had not occurred. Fourth, while we did exclude patients that had an active malignancy, we did not exclude patients who were younger than 35 years or were immunocompromised, which may have led to an overestimation of the percentage of patients who did not receive follow up.

Incidental findings detected on acute diagnostic tests requiring handoffs for chronic follow up are at risk of falling through the cracks. The inclusion of follow-up recommendations in discharge summaries has been shown to increase the likelihood of follow-up completion.⁹ Our study provides additional evidence of the urgent need for interventions aimed at closing the loop on test result follow up.^5,6

Disclosures

None of the authors have any conflicts of interest to disclose in reference to this study.

Funding

JLK is supported by the Mount Sinai Hospital Department of Medicine Research Fund. PC is supported by a K24 award from the National Institute of Arthritis and Musculoskeletal and Skin Diseases (AR062133).

Computed tomography pulmonary angiography (CTPA) is often used in the evaluation of suspected pulmonary embolism (PE). The detection of incidental findings that require follow-up is common; in just over 50% of cases, those incidental findings are pulmonary nodules.¹ Although the majority of these nodules are benign, Fleischner Society guidelines² recommend that patients with nodules at high risk for malignancy should undergo follow-up CT imaging within 3-12 months, with patients who smoke and have large nodules requiring closer follow up.

The failure to follow-up on abnormal test results is known to contribute to diagnostic error and can lead to patient harm.³ We sought to determine the proportion of high-risk pulmonary nodules on CTPA which did not undergo the recommended follow-up imaging.

Study Setting and Design

This retrospective cohort study included all patients who underwent CTPA in the emergency department (ED) and inpatient settings at three academic health centers (Mount Sinai Hospital, Toronto General Hospital, and Toronto Western Hospital) in Toronto, Canada between September 1, 2014, and August 31, 2015.

We examined the proportion of patients with pulmonary nodules requiring follow up who received repeat CT imaging within six weeks of the time frame recommended by the radiologist. Since we were interested in measuring the rate of an important test result that is missed (rather than accuracy of the test itself), we defined “requiring follow up” as the inclusion of explicit recommendations for follow up in the radiology report.

Montage (Philadelphia, Pennsylvania), a natural language processing software, was applied to a linked radiology information system (RIS) to identify all CTPAs that contained pulmonary nodules. We conducted manual chart review to confirm software accuracy. We initially searched the RIS for all CTPAs that were completed within the study period, resulting in the identification of 1932 imaging studies. Following a review of these 1,932 studies, we excluded 22 as they were not CTPAs. We then applied the search term, “nodule-” to 1,910 confirmed CTPAs, resulting in the identification of 836 imaging studies. Following a review of these 836 studies, we excluded 10 as they were duplicate studies. We also excluded 152 studies where the term “nodule-” did not identify a pulmonary nodule but instead referred to a radiologist reporting the absence of pulmonary nodules (eg “there were no pulmonary nodules found”) or the presence of non-lung nodules (eg thyroid nodules). This resulted in the identification of 674 CTPAs containing pulmonary nodules (Figure 1).

Thereafter, we generated a cohort with possible new lung malignancy eligible for follow-up imaging by reviewing available health records and applying the following prespecified exclusion criteria: (1) patients who died, (2) left against medical advice, (3) were critically ill during the follow-up period, (4) lived outside the hospital catchment area (Greater Toronto Area), (5) were seen in the outpatient setting, (6) identified as palliative, (7) had an active malignancy, (8) had nodules that were already being followed, or (9) had nodules with characteristics suggestive of alternate diagnoses to lung malignancy (such as infection or inflammation) with no follow up recommended as reported by the radiologist. For patients with multiple CTPAs, we included only the first study. For each eligible patient, we determined whether follow-up imaging was completed by manually reviewing the linked RIS. We reviewed available health records to determine whether the pulmonary nodule findings had been discussed with the patient and whether the patient had attended an outpatient follow-up visit. In patients for whom recommended follow-up imaging was not confirmed, we notified the ordering physician by e-mail.

Each radiology department followed the same protocol adherent to the 2005 Fleischner guidelines for identifying nodules requiring follow up.² Virtually all CTPAs at the three study institutions are read and reported within 72 hours. The ordering physician is sometimes called at the discretion of the reading radiologist when the findings are judged to be urgent and time-sensitive in nature. For example, the ordering physician may be contacted if a CTPA is positive for segmental PE but is not typically called for incidental pulmonary nodules. It is not common practice for ordering physicians to be notified of incidental findings above and beyond the radiology report. Primary care physicians are not typically copied on radiology reports and usually do not use the same electronic health record.

Statistical Analysis

We calculated simple descriptive statistics for all results. Mean values were compared using two-tailed t-tests, categorical groups using chi-square tests, and median values using Mann-Whitney U tests. We performed all analyses using Microsoft Excel version 16.14.1 (Redmond, Washington).

Ethics Approval

This study was approved by each institution’s research ethics board.

Follow Up of Incidental High-Risk Pulmonary Nodules

Of the 1910 CTPAs performed over the study period (Figure), 674 (35.3%) contained pulmonary nodules. Of the 259 patients with new pulmonary nodules eligible for follow-up imaging, 194 (74.9%) did not have an explicit suggestion for follow up by the radiologist. Ninety-five percent of radiologists (184 out of 194) provided an explanation for not recommending follow up in the radiology report; the two most common reasons were small nodule size (often described as “tiny”) and no interval change compared with the prior imaging study.² Of the 65 patients who did receive an explicit suggestion for follow up by radiology, 35 (53.8%) did not receive repeat imaging within the recommended time frame, allowing for a six-week grace period. Of these 35 patients, 10 eventually went on to receive delayed repeat imaging. The median follow-up time for the 30 patients who received timely repeat imaging was four months (IQR 2-6 months); in contrast, the median follow-up time for the 10 patients who received delayed repeat imaging was seven months (IQR 6-8 months), P = .01.

Of the 65 patients for whom follow up was recommended, the medical record showed evidence that there was a discussion between the medical team and the patient regarding patient preference for or against follow up in 55.4% (36 out of 65) of the patients. Notably, all 36 patients showed interest in receiving follow up; no patient indicated a preference for no follow up.

Furthermore, of the 65 patients that had follow up recommended, two patients were eventually diagnosed with lung cancer (one via lung biopsy, the other via positron emission tomography imaging); both patients did not receive timely follow-up imaging. While we did not include nodule size as an exclusion criterion, not one of the 65 patients included in the final cohort had nodules larger than 3 cm.

Physician Notification

In circumstances where we could not confirm that followed up had occurred, we notified the ordering physician by e-mail. Since 10 of the 35 patients who did not receive timely follow-up imaging went on to receive delayed repeat imaging, we notified 25 physicians. Of the 25 physicians that we e-mailed, 24 acknowledged receipt of the information. Of these 24 physicians, 14 reported conducting a detailed review of the chart, from which the following additional information was obtained: one patient expired, and five physicians notified the corresponding primary care physicians (two of whom were unaware of the nodule, and subsequently arranged further follow up with the patient).

Characteristics Associated with Timely Follow Up

Explicit mention that follow up was required in the discharge summary (P = .03), attending an outpatient follow-up visit (P < .001), and younger age (P = .03) were associated with receiving timely follow up; patient sex, smoking history, history of chronic obstructive pulmonary disease, lung nodule count, recommended follow-up time, and hospital department (defined as the discharging service) were not (Table).

In this multicenter cohort study, over 50% of patients with new high-risk pulmonary nodules detected incidentally on CTPA did not receive timely follow-up imaging. Including follow-up recommendations in the discharge summary, attending an outpatient follow-up visit, and younger age were associated with timely follow-up imaging.

Few studies have assessed the follow up of incidental nodules identified on CTPA. In a retrospective cohort study of ED patients in the United States, Blagev et al. found that only 29% received timely follow up.⁴ Our study contributes to the literature in several ways. First, our study included all hospitalized patients, not only those in the ED. Notably, most of our cohort were inpatients, a group of patients not previously described. Second, we examined factors associated with timely follow up, which may help to inform future quality improvement initiatives and interventions. Third, we included data from three different hospitals, which may improve generalization. Lastly, our study draws on contemporary Canadian data. Most of the studies investigating test result follow up have been conducted in the US^5,6 and Europe,⁷ with few empirical studies describing this phenomenon within the Canadian healthcare setting. We believe that our work contributes to the existing evidence that missed test results occur across diverse healthcare systems and have yet to be solved.^5-7

Our study had limitations. First, we defined follow up as repeat imaging and did not include office visits or biopsy in this definition. Second, we may have missed repeat imaging and outpatient follow-up visits that occurred outside the study hospitals. Although we were able to determine if repeat imaging and outpatient follow-up visits (eg, pulmonology or thoracic surgery clinics) had occurred within the study hospitals, we did not have access to follow-up encounters that occurred outside of the study hospitals (eg primary care clinics). We are unaware of any published regional data on the rate of outpatient follow up at the index facility following discharge. However, we know from provincial data of patients discharged from the ED with a new cardiac diagnosis that just under half are seen by a family physician, cardiologist, or internist within seven days, with just under 80% seen within 30 days.⁸ Third, although we attempted to capture patient preference for or against repeat imaging using chart review, the absence of documentation of patient preference did not confirm that a discussion regarding patient preferences had not occurred. Fourth, while we did exclude patients that had an active malignancy, we did not exclude patients who were younger than 35 years or were immunocompromised, which may have led to an overestimation of the percentage of patients who did not receive follow up.

Incidental findings detected on acute diagnostic tests requiring handoffs for chronic follow up are at risk of falling through the cracks. The inclusion of follow-up recommendations in discharge summaries has been shown to increase the likelihood of follow-up completion.⁹ Our study provides additional evidence of the urgent need for interventions aimed at closing the loop on test result follow up.^5,6

Disclosures

None of the authors have any conflicts of interest to disclose in reference to this study.

Funding

JLK is supported by the Mount Sinai Hospital Department of Medicine Research Fund. PC is supported by a K24 award from the National Institute of Arthritis and Musculoskeletal and Skin Diseases (AR062133).

Computed tomography pulmonary angiography (CTPA) is often used in the evaluation of suspected pulmonary embolism (PE). The detection of incidental findings that require follow-up is common; in just over 50% of cases, those incidental findings are pulmonary nodules.¹ Although the majority of these nodules are benign, Fleischner Society guidelines² recommend that patients with nodules at high risk for malignancy should undergo follow-up CT imaging within 3-12 months, with patients who smoke and have large nodules requiring closer follow up.

The failure to follow-up on abnormal test results is known to contribute to diagnostic error and can lead to patient harm.³ We sought to determine the proportion of high-risk pulmonary nodules on CTPA which did not undergo the recommended follow-up imaging.

Study Setting and Design

This retrospective cohort study included all patients who underwent CTPA in the emergency department (ED) and inpatient settings at three academic health centers (Mount Sinai Hospital, Toronto General Hospital, and Toronto Western Hospital) in Toronto, Canada between September 1, 2014, and August 31, 2015.

We examined the proportion of patients with pulmonary nodules requiring follow up who received repeat CT imaging within six weeks of the time frame recommended by the radiologist. Since we were interested in measuring the rate of an important test result that is missed (rather than accuracy of the test itself), we defined “requiring follow up” as the inclusion of explicit recommendations for follow up in the radiology report.

Montage (Philadelphia, Pennsylvania), a natural language processing software, was applied to a linked radiology information system (RIS) to identify all CTPAs that contained pulmonary nodules. We conducted manual chart review to confirm software accuracy. We initially searched the RIS for all CTPAs that were completed within the study period, resulting in the identification of 1932 imaging studies. Following a review of these 1,932 studies, we excluded 22 as they were not CTPAs. We then applied the search term, “nodule-” to 1,910 confirmed CTPAs, resulting in the identification of 836 imaging studies. Following a review of these 836 studies, we excluded 10 as they were duplicate studies. We also excluded 152 studies where the term “nodule-” did not identify a pulmonary nodule but instead referred to a radiologist reporting the absence of pulmonary nodules (eg “there were no pulmonary nodules found”) or the presence of non-lung nodules (eg thyroid nodules). This resulted in the identification of 674 CTPAs containing pulmonary nodules (Figure 1).

Thereafter, we generated a cohort with possible new lung malignancy eligible for follow-up imaging by reviewing available health records and applying the following prespecified exclusion criteria: (1) patients who died, (2) left against medical advice, (3) were critically ill during the follow-up period, (4) lived outside the hospital catchment area (Greater Toronto Area), (5) were seen in the outpatient setting, (6) identified as palliative, (7) had an active malignancy, (8) had nodules that were already being followed, or (9) had nodules with characteristics suggestive of alternate diagnoses to lung malignancy (such as infection or inflammation) with no follow up recommended as reported by the radiologist. For patients with multiple CTPAs, we included only the first study. For each eligible patient, we determined whether follow-up imaging was completed by manually reviewing the linked RIS. We reviewed available health records to determine whether the pulmonary nodule findings had been discussed with the patient and whether the patient had attended an outpatient follow-up visit. In patients for whom recommended follow-up imaging was not confirmed, we notified the ordering physician by e-mail.

Each radiology department followed the same protocol adherent to the 2005 Fleischner guidelines for identifying nodules requiring follow up.² Virtually all CTPAs at the three study institutions are read and reported within 72 hours. The ordering physician is sometimes called at the discretion of the reading radiologist when the findings are judged to be urgent and time-sensitive in nature. For example, the ordering physician may be contacted if a CTPA is positive for segmental PE but is not typically called for incidental pulmonary nodules. It is not common practice for ordering physicians to be notified of incidental findings above and beyond the radiology report. Primary care physicians are not typically copied on radiology reports and usually do not use the same electronic health record.

Statistical Analysis

We calculated simple descriptive statistics for all results. Mean values were compared using two-tailed t-tests, categorical groups using chi-square tests, and median values using Mann-Whitney U tests. We performed all analyses using Microsoft Excel version 16.14.1 (Redmond, Washington).

Ethics Approval

This study was approved by each institution’s research ethics board.

Follow Up of Incidental High-Risk Pulmonary Nodules

Of the 1910 CTPAs performed over the study period (Figure), 674 (35.3%) contained pulmonary nodules. Of the 259 patients with new pulmonary nodules eligible for follow-up imaging, 194 (74.9%) did not have an explicit suggestion for follow up by the radiologist. Ninety-five percent of radiologists (184 out of 194) provided an explanation for not recommending follow up in the radiology report; the two most common reasons were small nodule size (often described as “tiny”) and no interval change compared with the prior imaging study.² Of the 65 patients who did receive an explicit suggestion for follow up by radiology, 35 (53.8%) did not receive repeat imaging within the recommended time frame, allowing for a six-week grace period. Of these 35 patients, 10 eventually went on to receive delayed repeat imaging. The median follow-up time for the 30 patients who received timely repeat imaging was four months (IQR 2-6 months); in contrast, the median follow-up time for the 10 patients who received delayed repeat imaging was seven months (IQR 6-8 months), P = .01.

Of the 65 patients for whom follow up was recommended, the medical record showed evidence that there was a discussion between the medical team and the patient regarding patient preference for or against follow up in 55.4% (36 out of 65) of the patients. Notably, all 36 patients showed interest in receiving follow up; no patient indicated a preference for no follow up.

Furthermore, of the 65 patients that had follow up recommended, two patients were eventually diagnosed with lung cancer (one via lung biopsy, the other via positron emission tomography imaging); both patients did not receive timely follow-up imaging. While we did not include nodule size as an exclusion criterion, not one of the 65 patients included in the final cohort had nodules larger than 3 cm.

Physician Notification

In circumstances where we could not confirm that followed up had occurred, we notified the ordering physician by e-mail. Since 10 of the 35 patients who did not receive timely follow-up imaging went on to receive delayed repeat imaging, we notified 25 physicians. Of the 25 physicians that we e-mailed, 24 acknowledged receipt of the information. Of these 24 physicians, 14 reported conducting a detailed review of the chart, from which the following additional information was obtained: one patient expired, and five physicians notified the corresponding primary care physicians (two of whom were unaware of the nodule, and subsequently arranged further follow up with the patient).

Characteristics Associated with Timely Follow Up

Explicit mention that follow up was required in the discharge summary (P = .03), attending an outpatient follow-up visit (P < .001), and younger age (P = .03) were associated with receiving timely follow up; patient sex, smoking history, history of chronic obstructive pulmonary disease, lung nodule count, recommended follow-up time, and hospital department (defined as the discharging service) were not (Table).

In this multicenter cohort study, over 50% of patients with new high-risk pulmonary nodules detected incidentally on CTPA did not receive timely follow-up imaging. Including follow-up recommendations in the discharge summary, attending an outpatient follow-up visit, and younger age were associated with timely follow-up imaging.

Few studies have assessed the follow up of incidental nodules identified on CTPA. In a retrospective cohort study of ED patients in the United States, Blagev et al. found that only 29% received timely follow up.⁴ Our study contributes to the literature in several ways. First, our study included all hospitalized patients, not only those in the ED. Notably, most of our cohort were inpatients, a group of patients not previously described. Second, we examined factors associated with timely follow up, which may help to inform future quality improvement initiatives and interventions. Third, we included data from three different hospitals, which may improve generalization. Lastly, our study draws on contemporary Canadian data. Most of the studies investigating test result follow up have been conducted in the US^5,6 and Europe,⁷ with few empirical studies describing this phenomenon within the Canadian healthcare setting. We believe that our work contributes to the existing evidence that missed test results occur across diverse healthcare systems and have yet to be solved.^5-7

Our study had limitations. First, we defined follow up as repeat imaging and did not include office visits or biopsy in this definition. Second, we may have missed repeat imaging and outpatient follow-up visits that occurred outside the study hospitals. Although we were able to determine if repeat imaging and outpatient follow-up visits (eg, pulmonology or thoracic surgery clinics) had occurred within the study hospitals, we did not have access to follow-up encounters that occurred outside of the study hospitals (eg primary care clinics). We are unaware of any published regional data on the rate of outpatient follow up at the index facility following discharge. However, we know from provincial data of patients discharged from the ED with a new cardiac diagnosis that just under half are seen by a family physician, cardiologist, or internist within seven days, with just under 80% seen within 30 days.⁸ Third, although we attempted to capture patient preference for or against repeat imaging using chart review, the absence of documentation of patient preference did not confirm that a discussion regarding patient preferences had not occurred. Fourth, while we did exclude patients that had an active malignancy, we did not exclude patients who were younger than 35 years or were immunocompromised, which may have led to an overestimation of the percentage of patients who did not receive follow up.

Incidental findings detected on acute diagnostic tests requiring handoffs for chronic follow up are at risk of falling through the cracks. The inclusion of follow-up recommendations in discharge summaries has been shown to increase the likelihood of follow-up completion.⁹ Our study provides additional evidence of the urgent need for interventions aimed at closing the loop on test result follow up.^5,6

Disclosures

None of the authors have any conflicts of interest to disclose in reference to this study.

Funding

JLK is supported by the Mount Sinai Hospital Department of Medicine Research Fund. PC is supported by a K24 award from the National Institute of Arthritis and Musculoskeletal and Skin Diseases (AR062133).

Pulmonary nodules are common, and their identification is increasing as a result of the use of more sensitive chest imaging modalities.¹ Pulmonary nodules are defined on imaging as small (≤30 mm), well-defined lesions, completely surrounded by pulmonary parenchyma.² Most of the pulmonary nodules detected incidentally (ie, in asymptomatic patients outside the context of chest CT screening for lung cancer) are benign.¹ Lesions >30 mm are defined as masses and have higher risks of malignancy.²

Because the majority of patients will not benefit from the identification of incidental pulmonary nodules (IPNs), improving the benefits and minimizing the harms of IPN follow-up are critical. The Fleischner Society³ published their first guideline on the management of solid IPNs in 2005,⁴ which was supplemented in 2013 with specific guidance for the management of subsolid IPNs.⁵ In 2017, both guidelines were combined in a single update.⁶ The Fleischner Society recommendations for imaging surveillance and tissue sampling are based on nodule type (solid vs subsolid), number (single vs multiple), size, appearance, and patient risk for malignancy.

For IPNs identified in the hospital, management may be particularly challenging. For one, the provider initially ordering the chest imaging may not be the provider coordinating the patient’s discharge, leading to a lack of knowledge that the IPN even exists. The hospitalist to primary care provider (PCP) handoff may also have vulnerabilities, including the lack of inclusion of the IPN follow-up in the discharge summary and the nonreceipt of the discharge summary by the PCP. Moreover, because a patient’s acute medical problems often take precedence during a hospitalization, inpatients may not even be made aware of identified IPNs and the need for follow-up. Thus, the absence of standardized approaches to managing IPNs is a threat to patient safety, as well as a legal liability for providers and their institutions.

To better understand the current state of IPN management in our own institution, we examined the management of IPNs identified by chest computed tomographies (CTs) performed for inpatients on our general medicine services over a two-year period.⁷ Among the 50 inpatients identified with IPNs requiring follow-up, 78% had no follow-up imaging documented. Moreover, 40% had no mention of the IPN in their hospital summary or discharge instructions.

To inform our approach to addressing this challenge, we sought to examine the practices of hospitalist physicians nationally regarding the management of IPNs, including hospitalists’ familiarity with the Fleischner Society guidelines.

We developed a 14-item survey to assess hospitalists’ exposure to and management of IPNs. The survey targeted attendees of the 2016 Society of Hospital Medicine (SHM) annual conference and was available for completion on a tablet at the conference registration desk, the SHM kiosk in the exhibit hall, and at the entrance and exit of the morning plenary sessions. Following the annual conference, the survey was e-mailed to conference attendees, with one follow-up e-mailed to nonresponders.

Analyses were descriptive and included proportions for categorical variables and median and mean values and standard deviations for continuous variables. In addition, we examined the association between survey items and a response of “yes” to the question “Are you familiar with the Fleischner Society guidelines for the management of incidental pulmonary nodules?”

Associations between familiarity with the Fleischner Society guidelines and survey items were examined using Pearson’s chi-square test for categorical variables, Fisher’s exact test for categorical variables with small sample sizes, the Cochran–Armitage test for trend for ordinal variables, and the t-test for continuous variables. The associations between categorical items were measured by odds ratios with 95% confidence intervals. Statistical tests were two-sided using a P =.05 level for statistical significance. All analyses were performed using R version 3.4.4 (R Foundation for Statistical Computing, Vienna, Austria), with the R packages MASS, stats, and Publish. Institutional review board exemption was granted.

We received 174 responses from a total of 3,954 conference attendees. The majority were identified as hospitalist physicians, and most of them were internists (Table 1). About half practiced at a university or a teaching hospital, and more than half supervised trainees and practiced for more than five years. Respondents were involved in direct patient care (whether a teaching or a nonteaching service) for a median of 28 weeks annually (mean 31.2 weeks, standard deviation 13.5), and practice regions were geographically diverse. All respondents reported seeing at least one IPN case in the past six months, with most seeing three or more cases (Table 2). Despite this exposure, 42% were unfamiliar with the Fleischner Society guidelines. When determining the need for IPN follow-up, most of them utilized radiology report recommendations or consulted national or international guidelines, and a third spoke with radiologists directly. About a third agreed that determining the need for follow-up was challenging, with 39% citing patient factors (eg, lack of insurance, poor access to healthcare), and 30% citing scheduling of follow-up imaging. Few reported the availability of an automated tracking system at their institution, although most of them desired automatic notifications of results requiring follow-up.

Unadjusted analyses revealed that supervision of trainees and seeing more IPN cases significantly increased the odds of a survey respondent being familiar with the Fleischner Society guidelines (OR 1.96, 95% CI 1.04-3.68, P =.05, and OR 1.55, 95% CI 1.12-2.18, P =.008, respectively; Supplementary Table 1).

To our knowledge, the survey reported here is the first to examine hospitalists’ knowledge of the Fleischner Society guidelines and their approach to management of IPNs. Although our data suggest that hospitalists are less familiar with the Fleischner Society recommendations than pulmonologists⁸ and radiologists,^8-10 the majority of hospitalists in our study rely on radiology report recommendations to inform follow-up. This suggests that embedding the Fleischner Society recommendations into radiology reports is an effective method to promote adherence to these recommendations, which has been demonstrated in previous research.^11-13 Our study also suggests that hospitalists with more IPN exposure and those who supervise trainees are more likely to be aware of the Fleischner Society recommendations, which is similar to findings from studies examining radiologists and pulmonologists.^8-9

Our findings highlight other opportunities for quality improvement in IPN management. Almost a quarter of hospitalists reported formally consulting pulmonologists for IPN management. Hospitalist groups wishing to improve value could partner with their radiology departments and embed the Fleischner Society recommendations into their imaging reports to potentially reduce unnecessary pulmonary consultations. Among the 59 hospitalists who agreed that IPN management was challenging, a majority cited the scheduling process (30%) as a barrier. Redesigning the scheduling process for follow-up imaging could be a focus in local efforts to improve IPN management. Strengthening communication between hospitalists and PCPs may provide additional opportunities for improved IPN follow-up, given the centrality of PCPs to ensuring such follow-up. This might include enhancing direct communication between hospitalists and PCPs for high-risk patients, or creating systems to ensure robust indirect communication, such as the implementation of standardized discharge summaries that uniformly include essential follow-up information.

At our institution, given the large volume of high-risk patients and imaging performed, and the available resources, we have established an IPN consult team to improve follow-up for inpatients with IPNs identified by chest CTs on Medicine services. The team includes a nurse practitioner (NP) and a pulmonologist who consult by default, to notify patients of their findings and recommended follow-up, and communicate results to their PCPs. The IPN consult team also sees patients for follow-up in the ambulatory IPN clinic. This initiative has addressed the most frequently cited challenges identified in our nationwide hospitalist survey by taking the communication and follow-up out of the hospitalists’ hands. To ensure identification of all IPNs by the NP, our radiology department has created a structured template for radiology attendings to document follow-up for all chest CTs reviewed based on the Fleischner Society guidelines. Compliance with use of the template by radiologists is followed monthly. After a run-in period, almost 100% of chest CT reports use the structured template, consistent with published findings from similar initiatives,¹⁴ and 100% of patients with new IPNs identified on the inpatient Medicine services have had an IPN consult.

The major limitation of our survey study is the response rate. It is difficult to determine in what direction this could bias our results, as those with and without experience in managing IPNs may have been equally likely to complete the survey. Despite the low response rate, our sample targeted the general cohort of conference attendees (rather than specific forums such as audiences interested in quality or imaging), and the descriptive characteristics of our convenience sample align well with the overall conference attendee demographics (eg, conference attendees were 77% hospitalist attendings and 9% advanced practice providers, as compared with 82% and 7% of survey respondents, respectively), suggesting that our respondents were representative of conference attendees as a whole.

Next steps for this work at our institution include developing systems to ensure appropriate follow-up for those with IPNs identified on chest CTs performed for Medicine outpatients. In addition, our institution is collaborating on a national study to compare outcomes resulting from following the traditional Fleischner Society recommendations compared to the new 2017 recommendations, which recommend more lenient follow-up.¹⁵

Acknowledgments

The authors acknowledge Vivek Ahya, Eduardo Barbosa Jr., Tammy Tursi, and Anil Vachani for their leadership of the local quality improvement initiatives described in our Discussion, namely, the development and implementation of the structured templates for radiology reports and the incidental pulmonary nodule consult team.
 

Disclosures

Dr. Cook reports relevant financial activity outside the submitted work, including royalties from Osler Institute and grants from ACRIN and Beryl Institute. All other authors report no potential conflicts of interest relevant to this study. There was no financial support for this study.

Previous Presentations

Presented as a poster at the 2017 Society of Hospital Medicine Annual Conference, Las Vegas, NV: Wilen J, Garin M, Umscheid CA, Cook TS, Myers JS. Follow-up of incidental pulmonary nodules: a survey of hospitalists nationwide [abstract]. Journal of Hospital Medicine. 2017; 12 (Suppl 2). Available at: https://www.shmabstracts.com/abstract/follow-up-of-incidental-pulmonary-nodules-a-survey-of-hospitalists-nationwide/. Accessed March 18, 2018.

Pulmonary nodules are common, and their identification is increasing as a result of the use of more sensitive chest imaging modalities.¹ Pulmonary nodules are defined on imaging as small (≤30 mm), well-defined lesions, completely surrounded by pulmonary parenchyma.² Most of the pulmonary nodules detected incidentally (ie, in asymptomatic patients outside the context of chest CT screening for lung cancer) are benign.¹ Lesions >30 mm are defined as masses and have higher risks of malignancy.²

Because the majority of patients will not benefit from the identification of incidental pulmonary nodules (IPNs), improving the benefits and minimizing the harms of IPN follow-up are critical. The Fleischner Society³ published their first guideline on the management of solid IPNs in 2005,⁴ which was supplemented in 2013 with specific guidance for the management of subsolid IPNs.⁵ In 2017, both guidelines were combined in a single update.⁶ The Fleischner Society recommendations for imaging surveillance and tissue sampling are based on nodule type (solid vs subsolid), number (single vs multiple), size, appearance, and patient risk for malignancy.

For IPNs identified in the hospital, management may be particularly challenging. For one, the provider initially ordering the chest imaging may not be the provider coordinating the patient’s discharge, leading to a lack of knowledge that the IPN even exists. The hospitalist to primary care provider (PCP) handoff may also have vulnerabilities, including the lack of inclusion of the IPN follow-up in the discharge summary and the nonreceipt of the discharge summary by the PCP. Moreover, because a patient’s acute medical problems often take precedence during a hospitalization, inpatients may not even be made aware of identified IPNs and the need for follow-up. Thus, the absence of standardized approaches to managing IPNs is a threat to patient safety, as well as a legal liability for providers and their institutions.

To better understand the current state of IPN management in our own institution, we examined the management of IPNs identified by chest computed tomographies (CTs) performed for inpatients on our general medicine services over a two-year period.⁷ Among the 50 inpatients identified with IPNs requiring follow-up, 78% had no follow-up imaging documented. Moreover, 40% had no mention of the IPN in their hospital summary or discharge instructions.

To inform our approach to addressing this challenge, we sought to examine the practices of hospitalist physicians nationally regarding the management of IPNs, including hospitalists’ familiarity with the Fleischner Society guidelines.

We developed a 14-item survey to assess hospitalists’ exposure to and management of IPNs. The survey targeted attendees of the 2016 Society of Hospital Medicine (SHM) annual conference and was available for completion on a tablet at the conference registration desk, the SHM kiosk in the exhibit hall, and at the entrance and exit of the morning plenary sessions. Following the annual conference, the survey was e-mailed to conference attendees, with one follow-up e-mailed to nonresponders.

Analyses were descriptive and included proportions for categorical variables and median and mean values and standard deviations for continuous variables. In addition, we examined the association between survey items and a response of “yes” to the question “Are you familiar with the Fleischner Society guidelines for the management of incidental pulmonary nodules?”

Associations between familiarity with the Fleischner Society guidelines and survey items were examined using Pearson’s chi-square test for categorical variables, Fisher’s exact test for categorical variables with small sample sizes, the Cochran–Armitage test for trend for ordinal variables, and the t-test for continuous variables. The associations between categorical items were measured by odds ratios with 95% confidence intervals. Statistical tests were two-sided using a P =.05 level for statistical significance. All analyses were performed using R version 3.4.4 (R Foundation for Statistical Computing, Vienna, Austria), with the R packages MASS, stats, and Publish. Institutional review board exemption was granted.

We received 174 responses from a total of 3,954 conference attendees. The majority were identified as hospitalist physicians, and most of them were internists (Table 1). About half practiced at a university or a teaching hospital, and more than half supervised trainees and practiced for more than five years. Respondents were involved in direct patient care (whether a teaching or a nonteaching service) for a median of 28 weeks annually (mean 31.2 weeks, standard deviation 13.5), and practice regions were geographically diverse. All respondents reported seeing at least one IPN case in the past six months, with most seeing three or more cases (Table 2). Despite this exposure, 42% were unfamiliar with the Fleischner Society guidelines. When determining the need for IPN follow-up, most of them utilized radiology report recommendations or consulted national or international guidelines, and a third spoke with radiologists directly. About a third agreed that determining the need for follow-up was challenging, with 39% citing patient factors (eg, lack of insurance, poor access to healthcare), and 30% citing scheduling of follow-up imaging. Few reported the availability of an automated tracking system at their institution, although most of them desired automatic notifications of results requiring follow-up.

Unadjusted analyses revealed that supervision of trainees and seeing more IPN cases significantly increased the odds of a survey respondent being familiar with the Fleischner Society guidelines (OR 1.96, 95% CI 1.04-3.68, P =.05, and OR 1.55, 95% CI 1.12-2.18, P =.008, respectively; Supplementary Table 1).

To our knowledge, the survey reported here is the first to examine hospitalists’ knowledge of the Fleischner Society guidelines and their approach to management of IPNs. Although our data suggest that hospitalists are less familiar with the Fleischner Society recommendations than pulmonologists⁸ and radiologists,^8-10 the majority of hospitalists in our study rely on radiology report recommendations to inform follow-up. This suggests that embedding the Fleischner Society recommendations into radiology reports is an effective method to promote adherence to these recommendations, which has been demonstrated in previous research.^11-13 Our study also suggests that hospitalists with more IPN exposure and those who supervise trainees are more likely to be aware of the Fleischner Society recommendations, which is similar to findings from studies examining radiologists and pulmonologists.^8-9

Our findings highlight other opportunities for quality improvement in IPN management. Almost a quarter of hospitalists reported formally consulting pulmonologists for IPN management. Hospitalist groups wishing to improve value could partner with their radiology departments and embed the Fleischner Society recommendations into their imaging reports to potentially reduce unnecessary pulmonary consultations. Among the 59 hospitalists who agreed that IPN management was challenging, a majority cited the scheduling process (30%) as a barrier. Redesigning the scheduling process for follow-up imaging could be a focus in local efforts to improve IPN management. Strengthening communication between hospitalists and PCPs may provide additional opportunities for improved IPN follow-up, given the centrality of PCPs to ensuring such follow-up. This might include enhancing direct communication between hospitalists and PCPs for high-risk patients, or creating systems to ensure robust indirect communication, such as the implementation of standardized discharge summaries that uniformly include essential follow-up information.

At our institution, given the large volume of high-risk patients and imaging performed, and the available resources, we have established an IPN consult team to improve follow-up for inpatients with IPNs identified by chest CTs on Medicine services. The team includes a nurse practitioner (NP) and a pulmonologist who consult by default, to notify patients of their findings and recommended follow-up, and communicate results to their PCPs. The IPN consult team also sees patients for follow-up in the ambulatory IPN clinic. This initiative has addressed the most frequently cited challenges identified in our nationwide hospitalist survey by taking the communication and follow-up out of the hospitalists’ hands. To ensure identification of all IPNs by the NP, our radiology department has created a structured template for radiology attendings to document follow-up for all chest CTs reviewed based on the Fleischner Society guidelines. Compliance with use of the template by radiologists is followed monthly. After a run-in period, almost 100% of chest CT reports use the structured template, consistent with published findings from similar initiatives,¹⁴ and 100% of patients with new IPNs identified on the inpatient Medicine services have had an IPN consult.

The major limitation of our survey study is the response rate. It is difficult to determine in what direction this could bias our results, as those with and without experience in managing IPNs may have been equally likely to complete the survey. Despite the low response rate, our sample targeted the general cohort of conference attendees (rather than specific forums such as audiences interested in quality or imaging), and the descriptive characteristics of our convenience sample align well with the overall conference attendee demographics (eg, conference attendees were 77% hospitalist attendings and 9% advanced practice providers, as compared with 82% and 7% of survey respondents, respectively), suggesting that our respondents were representative of conference attendees as a whole.

Next steps for this work at our institution include developing systems to ensure appropriate follow-up for those with IPNs identified on chest CTs performed for Medicine outpatients. In addition, our institution is collaborating on a national study to compare outcomes resulting from following the traditional Fleischner Society recommendations compared to the new 2017 recommendations, which recommend more lenient follow-up.¹⁵

Acknowledgments

The authors acknowledge Vivek Ahya, Eduardo Barbosa Jr., Tammy Tursi, and Anil Vachani for their leadership of the local quality improvement initiatives described in our Discussion, namely, the development and implementation of the structured templates for radiology reports and the incidental pulmonary nodule consult team.
 

Disclosures

Dr. Cook reports relevant financial activity outside the submitted work, including royalties from Osler Institute and grants from ACRIN and Beryl Institute. All other authors report no potential conflicts of interest relevant to this study. There was no financial support for this study.

Previous Presentations

Presented as a poster at the 2017 Society of Hospital Medicine Annual Conference, Las Vegas, NV: Wilen J, Garin M, Umscheid CA, Cook TS, Myers JS. Follow-up of incidental pulmonary nodules: a survey of hospitalists nationwide [abstract]. Journal of Hospital Medicine. 2017; 12 (Suppl 2). Available at: https://www.shmabstracts.com/abstract/follow-up-of-incidental-pulmonary-nodules-a-survey-of-hospitalists-nationwide/. Accessed March 18, 2018.

Pulmonary nodules are common, and their identification is increasing as a result of the use of more sensitive chest imaging modalities.¹ Pulmonary nodules are defined on imaging as small (≤30 mm), well-defined lesions, completely surrounded by pulmonary parenchyma.² Most of the pulmonary nodules detected incidentally (ie, in asymptomatic patients outside the context of chest CT screening for lung cancer) are benign.¹ Lesions >30 mm are defined as masses and have higher risks of malignancy.²

Because the majority of patients will not benefit from the identification of incidental pulmonary nodules (IPNs), improving the benefits and minimizing the harms of IPN follow-up are critical. The Fleischner Society³ published their first guideline on the management of solid IPNs in 2005,⁴ which was supplemented in 2013 with specific guidance for the management of subsolid IPNs.⁵ In 2017, both guidelines were combined in a single update.⁶ The Fleischner Society recommendations for imaging surveillance and tissue sampling are based on nodule type (solid vs subsolid), number (single vs multiple), size, appearance, and patient risk for malignancy.

For IPNs identified in the hospital, management may be particularly challenging. For one, the provider initially ordering the chest imaging may not be the provider coordinating the patient’s discharge, leading to a lack of knowledge that the IPN even exists. The hospitalist to primary care provider (PCP) handoff may also have vulnerabilities, including the lack of inclusion of the IPN follow-up in the discharge summary and the nonreceipt of the discharge summary by the PCP. Moreover, because a patient’s acute medical problems often take precedence during a hospitalization, inpatients may not even be made aware of identified IPNs and the need for follow-up. Thus, the absence of standardized approaches to managing IPNs is a threat to patient safety, as well as a legal liability for providers and their institutions.

To better understand the current state of IPN management in our own institution, we examined the management of IPNs identified by chest computed tomographies (CTs) performed for inpatients on our general medicine services over a two-year period.⁷ Among the 50 inpatients identified with IPNs requiring follow-up, 78% had no follow-up imaging documented. Moreover, 40% had no mention of the IPN in their hospital summary or discharge instructions.

To inform our approach to addressing this challenge, we sought to examine the practices of hospitalist physicians nationally regarding the management of IPNs, including hospitalists’ familiarity with the Fleischner Society guidelines.

We developed a 14-item survey to assess hospitalists’ exposure to and management of IPNs. The survey targeted attendees of the 2016 Society of Hospital Medicine (SHM) annual conference and was available for completion on a tablet at the conference registration desk, the SHM kiosk in the exhibit hall, and at the entrance and exit of the morning plenary sessions. Following the annual conference, the survey was e-mailed to conference attendees, with one follow-up e-mailed to nonresponders.

Analyses were descriptive and included proportions for categorical variables and median and mean values and standard deviations for continuous variables. In addition, we examined the association between survey items and a response of “yes” to the question “Are you familiar with the Fleischner Society guidelines for the management of incidental pulmonary nodules?”

Associations between familiarity with the Fleischner Society guidelines and survey items were examined using Pearson’s chi-square test for categorical variables, Fisher’s exact test for categorical variables with small sample sizes, the Cochran–Armitage test for trend for ordinal variables, and the t-test for continuous variables. The associations between categorical items were measured by odds ratios with 95% confidence intervals. Statistical tests were two-sided using a P =.05 level for statistical significance. All analyses were performed using R version 3.4.4 (R Foundation for Statistical Computing, Vienna, Austria), with the R packages MASS, stats, and Publish. Institutional review board exemption was granted.

We received 174 responses from a total of 3,954 conference attendees. The majority were identified as hospitalist physicians, and most of them were internists (Table 1). About half practiced at a university or a teaching hospital, and more than half supervised trainees and practiced for more than five years. Respondents were involved in direct patient care (whether a teaching or a nonteaching service) for a median of 28 weeks annually (mean 31.2 weeks, standard deviation 13.5), and practice regions were geographically diverse. All respondents reported seeing at least one IPN case in the past six months, with most seeing three or more cases (Table 2). Despite this exposure, 42% were unfamiliar with the Fleischner Society guidelines. When determining the need for IPN follow-up, most of them utilized radiology report recommendations or consulted national or international guidelines, and a third spoke with radiologists directly. About a third agreed that determining the need for follow-up was challenging, with 39% citing patient factors (eg, lack of insurance, poor access to healthcare), and 30% citing scheduling of follow-up imaging. Few reported the availability of an automated tracking system at their institution, although most of them desired automatic notifications of results requiring follow-up.

Unadjusted analyses revealed that supervision of trainees and seeing more IPN cases significantly increased the odds of a survey respondent being familiar with the Fleischner Society guidelines (OR 1.96, 95% CI 1.04-3.68, P =.05, and OR 1.55, 95% CI 1.12-2.18, P =.008, respectively; Supplementary Table 1).

To our knowledge, the survey reported here is the first to examine hospitalists’ knowledge of the Fleischner Society guidelines and their approach to management of IPNs. Although our data suggest that hospitalists are less familiar with the Fleischner Society recommendations than pulmonologists⁸ and radiologists,^8-10 the majority of hospitalists in our study rely on radiology report recommendations to inform follow-up. This suggests that embedding the Fleischner Society recommendations into radiology reports is an effective method to promote adherence to these recommendations, which has been demonstrated in previous research.^11-13 Our study also suggests that hospitalists with more IPN exposure and those who supervise trainees are more likely to be aware of the Fleischner Society recommendations, which is similar to findings from studies examining radiologists and pulmonologists.^8-9

Our findings highlight other opportunities for quality improvement in IPN management. Almost a quarter of hospitalists reported formally consulting pulmonologists for IPN management. Hospitalist groups wishing to improve value could partner with their radiology departments and embed the Fleischner Society recommendations into their imaging reports to potentially reduce unnecessary pulmonary consultations. Among the 59 hospitalists who agreed that IPN management was challenging, a majority cited the scheduling process (30%) as a barrier. Redesigning the scheduling process for follow-up imaging could be a focus in local efforts to improve IPN management. Strengthening communication between hospitalists and PCPs may provide additional opportunities for improved IPN follow-up, given the centrality of PCPs to ensuring such follow-up. This might include enhancing direct communication between hospitalists and PCPs for high-risk patients, or creating systems to ensure robust indirect communication, such as the implementation of standardized discharge summaries that uniformly include essential follow-up information.

At our institution, given the large volume of high-risk patients and imaging performed, and the available resources, we have established an IPN consult team to improve follow-up for inpatients with IPNs identified by chest CTs on Medicine services. The team includes a nurse practitioner (NP) and a pulmonologist who consult by default, to notify patients of their findings and recommended follow-up, and communicate results to their PCPs. The IPN consult team also sees patients for follow-up in the ambulatory IPN clinic. This initiative has addressed the most frequently cited challenges identified in our nationwide hospitalist survey by taking the communication and follow-up out of the hospitalists’ hands. To ensure identification of all IPNs by the NP, our radiology department has created a structured template for radiology attendings to document follow-up for all chest CTs reviewed based on the Fleischner Society guidelines. Compliance with use of the template by radiologists is followed monthly. After a run-in period, almost 100% of chest CT reports use the structured template, consistent with published findings from similar initiatives,¹⁴ and 100% of patients with new IPNs identified on the inpatient Medicine services have had an IPN consult.

The major limitation of our survey study is the response rate. It is difficult to determine in what direction this could bias our results, as those with and without experience in managing IPNs may have been equally likely to complete the survey. Despite the low response rate, our sample targeted the general cohort of conference attendees (rather than specific forums such as audiences interested in quality or imaging), and the descriptive characteristics of our convenience sample align well with the overall conference attendee demographics (eg, conference attendees were 77% hospitalist attendings and 9% advanced practice providers, as compared with 82% and 7% of survey respondents, respectively), suggesting that our respondents were representative of conference attendees as a whole.

Next steps for this work at our institution include developing systems to ensure appropriate follow-up for those with IPNs identified on chest CTs performed for Medicine outpatients. In addition, our institution is collaborating on a national study to compare outcomes resulting from following the traditional Fleischner Society recommendations compared to the new 2017 recommendations, which recommend more lenient follow-up.¹⁵

Acknowledgments

The authors acknowledge Vivek Ahya, Eduardo Barbosa Jr., Tammy Tursi, and Anil Vachani for their leadership of the local quality improvement initiatives described in our Discussion, namely, the development and implementation of the structured templates for radiology reports and the incidental pulmonary nodule consult team.
 

Disclosures

Dr. Cook reports relevant financial activity outside the submitted work, including royalties from Osler Institute and grants from ACRIN and Beryl Institute. All other authors report no potential conflicts of interest relevant to this study. There was no financial support for this study.

Previous Presentations

Presented as a poster at the 2017 Society of Hospital Medicine Annual Conference, Las Vegas, NV: Wilen J, Garin M, Umscheid CA, Cook TS, Myers JS. Follow-up of incidental pulmonary nodules: a survey of hospitalists nationwide [abstract]. Journal of Hospital Medicine. 2017; 12 (Suppl 2). Available at: https://www.shmabstracts.com/abstract/follow-up-of-incidental-pulmonary-nodules-a-survey-of-hospitalists-nationwide/. Accessed March 18, 2018.

People often do not receive the kind of care they want at the end of their lives.^1,2 Although most people say they do not wish to have aggressive interventions if they are dying,^3-5 nearly one in five dies in the hospital and one in seven dies in the intensive care unit (ICU), where aggressive care is usually provided.⁶ Coming demographic shifts will put this phenomenon in relief. The US Census Bureau estimates the number of people over age 85 will balloon to 20 million by 2050.⁷

A proposed strategy for reducing this mismatch is to expand shared decision making for people facing life-sustaining treatment decisions.^8-10 Patient decision aids are tools that help people make informed healthcare decisions in light of their values and preferences, facilitating shared decision making.^8,11 Decision aids can take many forms: paper-based, audio/video-based, or online. They can be intended for the clinical encounter (used in partnership with a physician, nurse, or other clinician), independent patient use, or peer-to-peer use.⁸ In a 2017 review, Stacey and colleagues found that patient decision aids improve knowledge, clarify values, encourage more active decision making, and improve risk perception, across a variety of treatment and screening decisions.¹² They also concluded that decision aids might help people make decisions that are more aligned with their values, without affecting health outcomes negatively. ¹²

The number of available patient decision aids for people making life-sustaining treatment choices during serious illness near death is currently unknown. A 2014 review of all advanced care planning decision aids, including those for people who are healthy and people who are seriously ill, found 16 published studies in the peer-reviewed literature that tested patient decision aids for advanced care planning, but they did not systematically search the Internet and query key informants.¹³

Given the frequency of serious illness and death in hospital settings, awareness of potentially useful tools, their quality, and their use may be of interest to practicing hospitalists. This awareness may inform their decision making around whether or not to use decision aids in their own practice.

Study Aims and Design

With our systematic environmental scan, we aimed to identify all decision aids available to seriously ill people near death facing choices about life-sustaining treatments, developed by both academic researchers and private organizations. We set out to articulate their quality and the degree to which they are used.

Protocol

We developed four research questions to address our study objectives. Our questions were as follows: (1) What English-language patient decision aids are available? (2) What are the characteristics of these patient decision aids? (3) What is the quality of these patient decision aids, including readability? (4) What organizations use these patient decision aids in routine care (exploratory)? ^14-16 See protocol: doi: 10.1007/s40271-017-0268-2.¹⁷

Decision Aid Search Strategy

We searched for patient decision aids among published systematic reviews, Internet search results (Google.com), and app stores (Google Play and Apple App Store). To identify previously published systematic reviews, we searched MEDLINE via PubMed, with the date range from inception to 2017. We chose not to include other academic databases because the unit of observation for this environmental scan was the decision aids themselves, not the published articles. Additionally, we were aware of systematic reviews concerning this issue and felt that adding additional databases would not appreciably improve our likelihood of identifying eligible decision aids. We conducted searches using Google.com on November 30, 2016, and January 26, 2017, and included the first 100 search results. We also contacted shared decision-making and palliative care experts using a previously established list, via an online survey and one-on-one interviews between April 17, 2017, and August 30, 2017.

Published Reviews

Using a search strategy developed with a librarian, we identified reviews of decision aids that met our inclusion criteria using the MEDLINE database.¹⁷ The primary reviewer (CHS) examined the results of the search, identifying reviews appropriate for further investigation and the secondary reviewer (KP) extracted patient decision aids potentially eligible for our study. See Appendix Table 1 and our published protocol.¹⁷ Notably, given that the decision aids themselves, not published articles, were the unit of observation for our environmental scan, we did not perform dual coding on the MEDLINE extraction.

Google and App Stores

Two reviewers (CHS and MAD) performed the Google and application screening, including both the Apple App Store and Google Play.¹⁷ Using Google Advanced Search, we ran the queries detailed in Appendix Table 2. We disabled cookies and limited our search to English.

The primary reviewer ran each Google search and app store search, archiving the first 100 results of Google searches and first 50 results of app store searches.¹⁸ Then, the primary reviewer opened each page and scanned for patient decision aids or references to patient decision aids, marking those that met our inclusion criteria, those that might meet our inclusion criteria with further research, and those that were not appropriate. We documented specific reasons for exclusion. The secondary reviewer assessed a randomly-selected, 10% subsample. We calculated interrater reliability using a Cohen’s Kappa statistic.

Key Informants

To identify decision aids that did not appear in our online search, we surveyed 187 key informants who work in or study issues related to aging, death and dying and shared decision making.¹⁹ We developed a questionnaire for these informants and deployed it using the online survey software Qualtrics (see Appendix 1. Key Informant Survey). We used a snowball approach, asking participants for other individuals they thought we should speak with about other relevant decision aids. We corresponded with individuals who suggested decision aids that were not already in our decision aid database.

Decision Aid Selection Criteria

We included patient decision aids designed to help seriously ill people near death or their caregivers make decisions about life-sustaining treatments. See Appendix Table 1 for an explanation of terms. We saved decision aids that met our inclusion criteria in an online database, organizing them by target user or index decision(s). When identified decision aids were unavailable online, we e-mailed developers three times to ask for access to the decision aid. If after three queries, we did not receive access to the decision aid, we excluded the tool from our review. Similarly, if developers explicitly refused to participate in the study, we excluded them.

Once we banked and organized the decision aids, one reviewer (KP) systematically collected information about decision aid characteristics using a data collection form (see Appendix 2. Table 3). The data we collected for decision aids from all sources included (1) the index decision, (2) secondary decision(s), (3) the disease/condition, (4) availability (whether the decision aids are available publicly or proprietorially), and (5) use, ie, whether we learned anything about routine use in clinical environments.

Decision Aid Quality Grading Methods

At least two or three reviewers (C.H.S., K.P., M.A.D.), independently assessed the quality of each included patient decision aid, using the NQF standards. Before assessing the quality of each decision aid, we tested an NQF quality assessment form on five decision aids. We subsequently added specificity to the NQF quality criteria for this review. At least two of three reviewers (CHS, KP, MAD) assessed the quality of all included patient decision aids. We calculated interrater reliability using both Cohen’s Kappa statistic for individual quality categories and Spearman’s correlations for overall scores.

Notably, one of the NQF items concerns plain language. We assessed plain language using average readability scores, generated via Readable.io. If readability scores were below seventh-grade level, we considered them plain language. When we could not assess readability using an average score, ie, in the case of video decision aids, the researchers made a qualitative judgment about the plain language criteria.

Statistical Analysis

Our primary outcome was the number and variety of decision aids available for seriously ill individuals near death facing choices about life-sustaining treatments. Secondary outcomes included the quality, actual availability, and use of the available decision aids. We used Stata 13 to synthesize our results. We also reported overall quality and use. We conducted subgroup analyses, including quality, availability, and use of decision aids by category.

Decision Aid Selection Process

We identified 608 links with information about potential decision aids from our Google search. The two raters had substantial interrater reliability according to Cohen’s Kappa statistic (K = 0.64).²⁰ We did not detect any possible decision aids with our app store searches. We identified 31 studies from our MEDLINE search with information about potential decision aids eligible for inclusion. We received 60 responses to our expert survey from the 187 administered (a 32% response rate).

Altogether, we identified 105 potential decision aids from these sources. We excluded 22/105 potential decision aids from our analysis because they were not publicly accessible, and we could not successfully obtain them from the developers. It remains unknown whether these tools would have qualified for inclusion in our review. We excluded 55/105 tools for not meeting one of the following criteria: 1) not being decision aids according to the NQF criteria 2) not concerning life-sustaining treatments 3) not being targeted at people with serious illness near death. A majority of decision aids for life-sustaining treatment decisions are intended for people who do not yet have an advanced serious illness or are not near death. There were 27 decision aids in our final review (Figure 1).

Characteristics of Included Decision Aids

Of the 27 decision aids we included in our review, 14 (52%) were tailored to seriously ill individuals with specific conditions. Eleven decision aids (41%) concerned specific life-sustaining treatments. Two decision aids concerned general treatment approaches, such as life-sustaining care versus palliative care (Table 1).

The decision aids were of variable length and approach. Some were text only, while others were image heavy. The mean length of decision aids was 19 pages, while the median length was 10 pages. Included decision aids offered interventions meant to return patients to health, as well as palliative interventions and comfort care.

Notably, most of the decision aids we included in our review (25 decision aids; 93%) were freely available online. Three (11%) were not. Seventeen (63%) decision aids were developed in the U.S., eight (30%) in Canada, two (7%) in Australia, and one (4%) in the Netherlands (in Dutch, translated using Google Translate). Additionally, there were 22 potentially eligible decision aids that we could not access to review and therefore could not include.

Quality of Included Decision Aids

The overall correlation of scores between the two reviewers was high (0.85). Agreement was high for both reviewers for all categories (balanced 90%, K = 0.0; outcome probabilities 86%, K = 0.7; publication date 93%, K = 0.8; update policy 93%, K = 0.7; funding sources 96%, K = 0.8), except the category concerning the rigor of the decision aid development process (66%, K = 0.2) and the evidence sources used (79%, K = 0.6) categories.

The quality of the decision aids was high in some categories. Of 27 decision aids, most presented options in a balanced way (24, 89%) and identified funding sources (23, 85%). They also reported publication dates most of the time (19, 70%). Readability of the included decision aids was mixed. The average readability grade level was 7.5, with a low score of 4.1 and a high score of 10.7. Eleven decision aids (41%) had readability levels less than seventh grade (Table 2). Thirteen had plain language, including video decision aids that we agreed used plain language.

The decision aids also had consistently low scores in some categories. Of 27, only 11 listed their evidence sources (41%), 11 reported a rigorous evidence-synthesis method (41%), six stated their competing interests (22%), and three offered an update policy (11%). There were no notable differences in the quality of the decision aids in each of the three category types (condition-specific, treatment-specific, general).

Use of Included and Excluded Decision Aids (exploratory)

We received 60 of 187 responses to our key informant survey. We asked every respondent if they were aware of any relevant decision aids. Of the 60 respondents, 45 (75%) said they were aware of decision aids, but only 38 (63%) offered the names of potential tools. Twenty-six respondents (43%) said they were aware of institutions that used the decision aids in routine and sustained care. Twenty-four respondents (40%) offered names of organizations, but most of the suggestions concerned decision aids that did not qualify for inclusion in our review or care that was not routine or sustained. In this preliminary use estimation, we found evidence for the use of only three decision aids or similar tools in routine care, two of which we included in our review.

We found many decision aids of varying quality for people with serious illnesses facing decisions about life-sustaining treatments. Most available decision aids are customized for people with particular diseases or conditions, like cancer or heart failure, with few generalized tools. This may make it difficult for practicing clinicians to find tools that are appropriate for their patients. It could also contribute to the gap between their availability and use in routine care, which is an essential but exploratory finding of this systematic environmental scan. Even if seriously ill people or those who cared for them wanted to obtain and use a decision aid independently, a large proportion of them are not publicly accessible.

Concerning the quality of decision aids, they were usually balanced and listed their funding sources, but other quality areas we often missing concerning their development, content, and disclosures. These deficiencies may affect the trustworthiness of decision aids, which may make practicing clinicians less likely to use them in hospital settings. Reporting of outcome probabilities was particularly weak. Reporting outcome probabilities in ways that people who are ill and their relatives can understand, especially during times of heightened emotion, is critically important. Therefore, it is a cause for concern that the available decision aids often neglect to use evidence-based techniques for conveying outcome information.

Our work built on Butler and colleagues’ “state of the science” review in 2014.¹³ Focusing specifically on proximal life-sustaining treatment decisions, we found many more decision aids by expanding our search beyond the peer-reviewed literature to include the Internet and experts.¹³ We also identified an important gap worthy of further exploration between the decision aids available and their usage in real-world clinical environments.

Our review confirms that implementation of decision aids in routine care is a continued challenge, especially for seriously ill people facing life-sustaining treatment decisions.⁵³ Why tools that are efficacious in controlled trial environments have failed to gain acceptance in real-world settings remains unanswered for this population.⁵⁴ For decision aids in general, researchers have reported barriers concerning clinician awareness, perception, and comfort, as well as usability issues.^55,56 Additionally, systems-level barriers exist, like culture and priorities, difficulty incorporating decision aids into the workflow, resistance from parties who favor other interventions, and the costs associated with implementation.⁵⁶ There may also be particular barriers related to the topics of death and dying.A strength of this work is thatwe applied the rigor of the systematic review method to the environmental scan, a newer method that answers different questions, such as “How many?”, “How much?”, and “How often?” We hope our use of the word systematic will reinforce perception among the scientific community that the environmental scan method is thorough, valid and worthwhile. We believe this method unearthed more decision aids than a traditional systematic review limited to the academic literature would have revealed. Another strength of our review was the rigor of screening and assessment.

A limitation of our work is the challenge of defining serious illness. We worked with palliative care physicians to make these judgments as grounded in clinical practice as possible. The preliminary nature and selection of experts for our sustained—use survey are limitations as well. Despite our efforts to conduct a comprehensive review of a vast environment of tools, we may have missed some decision aids that met our inclusion criteria. An additional limitation of our work is that due to the exploratory nature of our sustained-use survey, we cannot determine with accuracy how often these tools are used, although we have provided the first preliminary assessment of use, to our knowledge.

The gap between prolific patient decision aid development and real-world usage is puzzling. It is possible that using a tool at all is inappropriate for the complex, emotionally-laden decision-making process associated with death and dying. Alternatively, the tools may be inappropriate for serious illness, due to their design, their content, or some other characteristics. Perhaps the existing tools are too tailored for specific conditions and interventions―less appropriate for generalized use. Indeed, only two decision aids included in our final review addressed general care pathways, like life-sustaining care, palliative care, and hospice care. The others were highly specific, concerning particular diseases like kidney disease and particular interventions, like CPR. We know that most people die with comultimorbidities, meaning such specificity may paradoxically make it more difficult for individuals and their families to identify with the content in the materials.^57,58 Without having data from real-world use, we cannot know whether any particular tool is suited or helpful for hospital practice.

It is essential for practicing hospitalists to know whether patient decision aids are appropriate for use in routine care. We hope that our review will help clinicians and health systems find appropriate tools to use with their patients. We also believe there should be mechanisms for providing feedback on whether decision aids are feasible and acceptable to hospitalized people and their caregivers and to practicing hospitalists and what leads to their sustained implementation.^55,56 This can be explored with on-the-ground observational research or through health system quality improvement efforts.

Acknowledgments

Pamela J. Bagley provided search strategy support. Meredith MacMartin provided clinical counsel. Amber Barnato provided comments and insight as an advisor and a new member of Catherine’s Ph.D. committee.

Author contributions

Catherine H. Saunders designed the study, with support from Marie-Anne Durand, Glyn Elwyn, and Kathryn Kirkland. Catherine H. Saunders conducted all screening, with support from Marie-Anne Durand. Khusbu Patel managed the inventory of decision aids. Catherine H. Saunders designed and distributed the key informant survey, with support from Marie-Anne Durand. Hyunkyung Kang and Catherine H. Saunders managed follow-up with key informants. Khusbu Patel and Catherine H. Saunders conducted the decision aid quality review. Catherine H. Saunders, Marie-Anne Durand, and Kathryn Kirkland screened decision aids to determine appropriateness for people with serious illness. Catherine H. Saunders drafted the manuscript, and all authors reviewed and approved it.

Ethical approval

The Dartmouth College Committee for the Protection of Human Subjects designated this project as exempt from further review. All survey participants confirmed their consent via an online form.

Disclosures

Ms. Saunders, Ms. Patel, Ms. Kang, and Dr. Kirkland have nothing to disclose. Dr. Elwyn reports personal fees from ACCESS Community Health Network, personal fees from EBSCO Health, personal fees from Chicago (Federally Qualified Medical Centers), outside the submitted work, and as Director of &think LLC, which owns the registered trademark for OptionGrids(TM) patient decision aids. He owns copyright in measures of shared decision making and care integration, namely collaboRATE, integRATE, Observer OPTION-5, and Observer OPTION-12, which are freely available for use. He is codeveloper of the OptionGrid patient decision aids, which are licensed to EBSCO Health. He has received reimbursement for travel, accommodations, and expenses from EBSCO Health, ACCESS Community Health Network, and Chicago (Federally Qualified Medical Centers). Dr. Durand reports personal fees from ACCESS Community Health Network, personal fees from EBSCO Health, outside the submitted work, and as codeveloper of the OptionGrid patient decision aids, which are licensed to EBSCO Health. She has received reimbursement for travel, accommodations, and expenses from EBSCO Health and ACCESS Community Health Network.

Financial conflicts of interest

Glyn Elwyn (GE) and Marie-Anne Durand (M-A D) have developed the Option Grid patient decision aids, which are licensed to EBSCO Health. They receive consulting income from EBSCO Health and may receive royalties in the future. M-A D is a consultant for ACCESS Community Health Network. No other competing interests declared.

Funding

The authors did not receive funding for this research.

Published protocol linked here: https://www.ncbi.nlm.nih.gov/pubmed/28825182

People often do not receive the kind of care they want at the end of their lives.^1,2 Although most people say they do not wish to have aggressive interventions if they are dying,^3-5 nearly one in five dies in the hospital and one in seven dies in the intensive care unit (ICU), where aggressive care is usually provided.⁶ Coming demographic shifts will put this phenomenon in relief. The US Census Bureau estimates the number of people over age 85 will balloon to 20 million by 2050.⁷

A proposed strategy for reducing this mismatch is to expand shared decision making for people facing life-sustaining treatment decisions.^8-10 Patient decision aids are tools that help people make informed healthcare decisions in light of their values and preferences, facilitating shared decision making.^8,11 Decision aids can take many forms: paper-based, audio/video-based, or online. They can be intended for the clinical encounter (used in partnership with a physician, nurse, or other clinician), independent patient use, or peer-to-peer use.⁸ In a 2017 review, Stacey and colleagues found that patient decision aids improve knowledge, clarify values, encourage more active decision making, and improve risk perception, across a variety of treatment and screening decisions.¹² They also concluded that decision aids might help people make decisions that are more aligned with their values, without affecting health outcomes negatively. ¹²

The number of available patient decision aids for people making life-sustaining treatment choices during serious illness near death is currently unknown. A 2014 review of all advanced care planning decision aids, including those for people who are healthy and people who are seriously ill, found 16 published studies in the peer-reviewed literature that tested patient decision aids for advanced care planning, but they did not systematically search the Internet and query key informants.¹³

Given the frequency of serious illness and death in hospital settings, awareness of potentially useful tools, their quality, and their use may be of interest to practicing hospitalists. This awareness may inform their decision making around whether or not to use decision aids in their own practice.

Study Aims and Design

With our systematic environmental scan, we aimed to identify all decision aids available to seriously ill people near death facing choices about life-sustaining treatments, developed by both academic researchers and private organizations. We set out to articulate their quality and the degree to which they are used.

Protocol

We developed four research questions to address our study objectives. Our questions were as follows: (1) What English-language patient decision aids are available? (2) What are the characteristics of these patient decision aids? (3) What is the quality of these patient decision aids, including readability? (4) What organizations use these patient decision aids in routine care (exploratory)? ^14-16 See protocol: doi: 10.1007/s40271-017-0268-2.¹⁷

Decision Aid Search Strategy

We searched for patient decision aids among published systematic reviews, Internet search results (Google.com), and app stores (Google Play and Apple App Store). To identify previously published systematic reviews, we searched MEDLINE via PubMed, with the date range from inception to 2017. We chose not to include other academic databases because the unit of observation for this environmental scan was the decision aids themselves, not the published articles. Additionally, we were aware of systematic reviews concerning this issue and felt that adding additional databases would not appreciably improve our likelihood of identifying eligible decision aids. We conducted searches using Google.com on November 30, 2016, and January 26, 2017, and included the first 100 search results. We also contacted shared decision-making and palliative care experts using a previously established list, via an online survey and one-on-one interviews between April 17, 2017, and August 30, 2017.

Published Reviews

Using a search strategy developed with a librarian, we identified reviews of decision aids that met our inclusion criteria using the MEDLINE database.¹⁷ The primary reviewer (CHS) examined the results of the search, identifying reviews appropriate for further investigation and the secondary reviewer (KP) extracted patient decision aids potentially eligible for our study. See Appendix Table 1 and our published protocol.¹⁷ Notably, given that the decision aids themselves, not published articles, were the unit of observation for our environmental scan, we did not perform dual coding on the MEDLINE extraction.

Google and App Stores

Two reviewers (CHS and MAD) performed the Google and application screening, including both the Apple App Store and Google Play.¹⁷ Using Google Advanced Search, we ran the queries detailed in Appendix Table 2. We disabled cookies and limited our search to English.

The primary reviewer ran each Google search and app store search, archiving the first 100 results of Google searches and first 50 results of app store searches.¹⁸ Then, the primary reviewer opened each page and scanned for patient decision aids or references to patient decision aids, marking those that met our inclusion criteria, those that might meet our inclusion criteria with further research, and those that were not appropriate. We documented specific reasons for exclusion. The secondary reviewer assessed a randomly-selected, 10% subsample. We calculated interrater reliability using a Cohen’s Kappa statistic.

Key Informants

To identify decision aids that did not appear in our online search, we surveyed 187 key informants who work in or study issues related to aging, death and dying and shared decision making.¹⁹ We developed a questionnaire for these informants and deployed it using the online survey software Qualtrics (see Appendix 1. Key Informant Survey). We used a snowball approach, asking participants for other individuals they thought we should speak with about other relevant decision aids. We corresponded with individuals who suggested decision aids that were not already in our decision aid database.

Decision Aid Selection Criteria

We included patient decision aids designed to help seriously ill people near death or their caregivers make decisions about life-sustaining treatments. See Appendix Table 1 for an explanation of terms. We saved decision aids that met our inclusion criteria in an online database, organizing them by target user or index decision(s). When identified decision aids were unavailable online, we e-mailed developers three times to ask for access to the decision aid. If after three queries, we did not receive access to the decision aid, we excluded the tool from our review. Similarly, if developers explicitly refused to participate in the study, we excluded them.

Once we banked and organized the decision aids, one reviewer (KP) systematically collected information about decision aid characteristics using a data collection form (see Appendix 2. Table 3). The data we collected for decision aids from all sources included (1) the index decision, (2) secondary decision(s), (3) the disease/condition, (4) availability (whether the decision aids are available publicly or proprietorially), and (5) use, ie, whether we learned anything about routine use in clinical environments.

Decision Aid Quality Grading Methods

At least two or three reviewers (C.H.S., K.P., M.A.D.), independently assessed the quality of each included patient decision aid, using the NQF standards. Before assessing the quality of each decision aid, we tested an NQF quality assessment form on five decision aids. We subsequently added specificity to the NQF quality criteria for this review. At least two of three reviewers (CHS, KP, MAD) assessed the quality of all included patient decision aids. We calculated interrater reliability using both Cohen’s Kappa statistic for individual quality categories and Spearman’s correlations for overall scores.

Notably, one of the NQF items concerns plain language. We assessed plain language using average readability scores, generated via Readable.io. If readability scores were below seventh-grade level, we considered them plain language. When we could not assess readability using an average score, ie, in the case of video decision aids, the researchers made a qualitative judgment about the plain language criteria.

Statistical Analysis

Our primary outcome was the number and variety of decision aids available for seriously ill individuals near death facing choices about life-sustaining treatments. Secondary outcomes included the quality, actual availability, and use of the available decision aids. We used Stata 13 to synthesize our results. We also reported overall quality and use. We conducted subgroup analyses, including quality, availability, and use of decision aids by category.

Decision Aid Selection Process

We identified 608 links with information about potential decision aids from our Google search. The two raters had substantial interrater reliability according to Cohen’s Kappa statistic (K = 0.64).²⁰ We did not detect any possible decision aids with our app store searches. We identified 31 studies from our MEDLINE search with information about potential decision aids eligible for inclusion. We received 60 responses to our expert survey from the 187 administered (a 32% response rate).

Altogether, we identified 105 potential decision aids from these sources. We excluded 22/105 potential decision aids from our analysis because they were not publicly accessible, and we could not successfully obtain them from the developers. It remains unknown whether these tools would have qualified for inclusion in our review. We excluded 55/105 tools for not meeting one of the following criteria: 1) not being decision aids according to the NQF criteria 2) not concerning life-sustaining treatments 3) not being targeted at people with serious illness near death. A majority of decision aids for life-sustaining treatment decisions are intended for people who do not yet have an advanced serious illness or are not near death. There were 27 decision aids in our final review (Figure 1).

Characteristics of Included Decision Aids

Of the 27 decision aids we included in our review, 14 (52%) were tailored to seriously ill individuals with specific conditions. Eleven decision aids (41%) concerned specific life-sustaining treatments. Two decision aids concerned general treatment approaches, such as life-sustaining care versus palliative care (Table 1).

The decision aids were of variable length and approach. Some were text only, while others were image heavy. The mean length of decision aids was 19 pages, while the median length was 10 pages. Included decision aids offered interventions meant to return patients to health, as well as palliative interventions and comfort care.

Notably, most of the decision aids we included in our review (25 decision aids; 93%) were freely available online. Three (11%) were not. Seventeen (63%) decision aids were developed in the U.S., eight (30%) in Canada, two (7%) in Australia, and one (4%) in the Netherlands (in Dutch, translated using Google Translate). Additionally, there were 22 potentially eligible decision aids that we could not access to review and therefore could not include.

Quality of Included Decision Aids

The overall correlation of scores between the two reviewers was high (0.85). Agreement was high for both reviewers for all categories (balanced 90%, K = 0.0; outcome probabilities 86%, K = 0.7; publication date 93%, K = 0.8; update policy 93%, K = 0.7; funding sources 96%, K = 0.8), except the category concerning the rigor of the decision aid development process (66%, K = 0.2) and the evidence sources used (79%, K = 0.6) categories.

The quality of the decision aids was high in some categories. Of 27 decision aids, most presented options in a balanced way (24, 89%) and identified funding sources (23, 85%). They also reported publication dates most of the time (19, 70%). Readability of the included decision aids was mixed. The average readability grade level was 7.5, with a low score of 4.1 and a high score of 10.7. Eleven decision aids (41%) had readability levels less than seventh grade (Table 2). Thirteen had plain language, including video decision aids that we agreed used plain language.

The decision aids also had consistently low scores in some categories. Of 27, only 11 listed their evidence sources (41%), 11 reported a rigorous evidence-synthesis method (41%), six stated their competing interests (22%), and three offered an update policy (11%). There were no notable differences in the quality of the decision aids in each of the three category types (condition-specific, treatment-specific, general).

Use of Included and Excluded Decision Aids (exploratory)

We received 60 of 187 responses to our key informant survey. We asked every respondent if they were aware of any relevant decision aids. Of the 60 respondents, 45 (75%) said they were aware of decision aids, but only 38 (63%) offered the names of potential tools. Twenty-six respondents (43%) said they were aware of institutions that used the decision aids in routine and sustained care. Twenty-four respondents (40%) offered names of organizations, but most of the suggestions concerned decision aids that did not qualify for inclusion in our review or care that was not routine or sustained. In this preliminary use estimation, we found evidence for the use of only three decision aids or similar tools in routine care, two of which we included in our review.

We found many decision aids of varying quality for people with serious illnesses facing decisions about life-sustaining treatments. Most available decision aids are customized for people with particular diseases or conditions, like cancer or heart failure, with few generalized tools. This may make it difficult for practicing clinicians to find tools that are appropriate for their patients. It could also contribute to the gap between their availability and use in routine care, which is an essential but exploratory finding of this systematic environmental scan. Even if seriously ill people or those who cared for them wanted to obtain and use a decision aid independently, a large proportion of them are not publicly accessible.

Concerning the quality of decision aids, they were usually balanced and listed their funding sources, but other quality areas we often missing concerning their development, content, and disclosures. These deficiencies may affect the trustworthiness of decision aids, which may make practicing clinicians less likely to use them in hospital settings. Reporting of outcome probabilities was particularly weak. Reporting outcome probabilities in ways that people who are ill and their relatives can understand, especially during times of heightened emotion, is critically important. Therefore, it is a cause for concern that the available decision aids often neglect to use evidence-based techniques for conveying outcome information.

Our work built on Butler and colleagues’ “state of the science” review in 2014.¹³ Focusing specifically on proximal life-sustaining treatment decisions, we found many more decision aids by expanding our search beyond the peer-reviewed literature to include the Internet and experts.¹³ We also identified an important gap worthy of further exploration between the decision aids available and their usage in real-world clinical environments.

Our review confirms that implementation of decision aids in routine care is a continued challenge, especially for seriously ill people facing life-sustaining treatment decisions.⁵³ Why tools that are efficacious in controlled trial environments have failed to gain acceptance in real-world settings remains unanswered for this population.⁵⁴ For decision aids in general, researchers have reported barriers concerning clinician awareness, perception, and comfort, as well as usability issues.^55,56 Additionally, systems-level barriers exist, like culture and priorities, difficulty incorporating decision aids into the workflow, resistance from parties who favor other interventions, and the costs associated with implementation.⁵⁶ There may also be particular barriers related to the topics of death and dying.A strength of this work is thatwe applied the rigor of the systematic review method to the environmental scan, a newer method that answers different questions, such as “How many?”, “How much?”, and “How often?” We hope our use of the word systematic will reinforce perception among the scientific community that the environmental scan method is thorough, valid and worthwhile. We believe this method unearthed more decision aids than a traditional systematic review limited to the academic literature would have revealed. Another strength of our review was the rigor of screening and assessment.

A limitation of our work is the challenge of defining serious illness. We worked with palliative care physicians to make these judgments as grounded in clinical practice as possible. The preliminary nature and selection of experts for our sustained—use survey are limitations as well. Despite our efforts to conduct a comprehensive review of a vast environment of tools, we may have missed some decision aids that met our inclusion criteria. An additional limitation of our work is that due to the exploratory nature of our sustained-use survey, we cannot determine with accuracy how often these tools are used, although we have provided the first preliminary assessment of use, to our knowledge.

The gap between prolific patient decision aid development and real-world usage is puzzling. It is possible that using a tool at all is inappropriate for the complex, emotionally-laden decision-making process associated with death and dying. Alternatively, the tools may be inappropriate for serious illness, due to their design, their content, or some other characteristics. Perhaps the existing tools are too tailored for specific conditions and interventions―less appropriate for generalized use. Indeed, only two decision aids included in our final review addressed general care pathways, like life-sustaining care, palliative care, and hospice care. The others were highly specific, concerning particular diseases like kidney disease and particular interventions, like CPR. We know that most people die with comultimorbidities, meaning such specificity may paradoxically make it more difficult for individuals and their families to identify with the content in the materials.^57,58 Without having data from real-world use, we cannot know whether any particular tool is suited or helpful for hospital practice.

It is essential for practicing hospitalists to know whether patient decision aids are appropriate for use in routine care. We hope that our review will help clinicians and health systems find appropriate tools to use with their patients. We also believe there should be mechanisms for providing feedback on whether decision aids are feasible and acceptable to hospitalized people and their caregivers and to practicing hospitalists and what leads to their sustained implementation.^55,56 This can be explored with on-the-ground observational research or through health system quality improvement efforts.

Acknowledgments

Pamela J. Bagley provided search strategy support. Meredith MacMartin provided clinical counsel. Amber Barnato provided comments and insight as an advisor and a new member of Catherine’s Ph.D. committee.

Author contributions

Catherine H. Saunders designed the study, with support from Marie-Anne Durand, Glyn Elwyn, and Kathryn Kirkland. Catherine H. Saunders conducted all screening, with support from Marie-Anne Durand. Khusbu Patel managed the inventory of decision aids. Catherine H. Saunders designed and distributed the key informant survey, with support from Marie-Anne Durand. Hyunkyung Kang and Catherine H. Saunders managed follow-up with key informants. Khusbu Patel and Catherine H. Saunders conducted the decision aid quality review. Catherine H. Saunders, Marie-Anne Durand, and Kathryn Kirkland screened decision aids to determine appropriateness for people with serious illness. Catherine H. Saunders drafted the manuscript, and all authors reviewed and approved it.

Ethical approval

The Dartmouth College Committee for the Protection of Human Subjects designated this project as exempt from further review. All survey participants confirmed their consent via an online form.

Disclosures

Ms. Saunders, Ms. Patel, Ms. Kang, and Dr. Kirkland have nothing to disclose. Dr. Elwyn reports personal fees from ACCESS Community Health Network, personal fees from EBSCO Health, personal fees from Chicago (Federally Qualified Medical Centers), outside the submitted work, and as Director of &think LLC, which owns the registered trademark for OptionGrids(TM) patient decision aids. He owns copyright in measures of shared decision making and care integration, namely collaboRATE, integRATE, Observer OPTION-5, and Observer OPTION-12, which are freely available for use. He is codeveloper of the OptionGrid patient decision aids, which are licensed to EBSCO Health. He has received reimbursement for travel, accommodations, and expenses from EBSCO Health, ACCESS Community Health Network, and Chicago (Federally Qualified Medical Centers). Dr. Durand reports personal fees from ACCESS Community Health Network, personal fees from EBSCO Health, outside the submitted work, and as codeveloper of the OptionGrid patient decision aids, which are licensed to EBSCO Health. She has received reimbursement for travel, accommodations, and expenses from EBSCO Health and ACCESS Community Health Network.

Financial conflicts of interest

Glyn Elwyn (GE) and Marie-Anne Durand (M-A D) have developed the Option Grid patient decision aids, which are licensed to EBSCO Health. They receive consulting income from EBSCO Health and may receive royalties in the future. M-A D is a consultant for ACCESS Community Health Network. No other competing interests declared.

Funding

The authors did not receive funding for this research.

Published protocol linked here: https://www.ncbi.nlm.nih.gov/pubmed/28825182

People often do not receive the kind of care they want at the end of their lives.^1,2 Although most people say they do not wish to have aggressive interventions if they are dying,^3-5 nearly one in five dies in the hospital and one in seven dies in the intensive care unit (ICU), where aggressive care is usually provided.⁶ Coming demographic shifts will put this phenomenon in relief. The US Census Bureau estimates the number of people over age 85 will balloon to 20 million by 2050.⁷

A proposed strategy for reducing this mismatch is to expand shared decision making for people facing life-sustaining treatment decisions.^8-10 Patient decision aids are tools that help people make informed healthcare decisions in light of their values and preferences, facilitating shared decision making.^8,11 Decision aids can take many forms: paper-based, audio/video-based, or online. They can be intended for the clinical encounter (used in partnership with a physician, nurse, or other clinician), independent patient use, or peer-to-peer use.⁸ In a 2017 review, Stacey and colleagues found that patient decision aids improve knowledge, clarify values, encourage more active decision making, and improve risk perception, across a variety of treatment and screening decisions.¹² They also concluded that decision aids might help people make decisions that are more aligned with their values, without affecting health outcomes negatively. ¹²

The number of available patient decision aids for people making life-sustaining treatment choices during serious illness near death is currently unknown. A 2014 review of all advanced care planning decision aids, including those for people who are healthy and people who are seriously ill, found 16 published studies in the peer-reviewed literature that tested patient decision aids for advanced care planning, but they did not systematically search the Internet and query key informants.¹³

Given the frequency of serious illness and death in hospital settings, awareness of potentially useful tools, their quality, and their use may be of interest to practicing hospitalists. This awareness may inform their decision making around whether or not to use decision aids in their own practice.

Study Aims and Design

With our systematic environmental scan, we aimed to identify all decision aids available to seriously ill people near death facing choices about life-sustaining treatments, developed by both academic researchers and private organizations. We set out to articulate their quality and the degree to which they are used.

Protocol

We developed four research questions to address our study objectives. Our questions were as follows: (1) What English-language patient decision aids are available? (2) What are the characteristics of these patient decision aids? (3) What is the quality of these patient decision aids, including readability? (4) What organizations use these patient decision aids in routine care (exploratory)? ^14-16 See protocol: doi: 10.1007/s40271-017-0268-2.¹⁷

Decision Aid Search Strategy

We searched for patient decision aids among published systematic reviews, Internet search results (Google.com), and app stores (Google Play and Apple App Store). To identify previously published systematic reviews, we searched MEDLINE via PubMed, with the date range from inception to 2017. We chose not to include other academic databases because the unit of observation for this environmental scan was the decision aids themselves, not the published articles. Additionally, we were aware of systematic reviews concerning this issue and felt that adding additional databases would not appreciably improve our likelihood of identifying eligible decision aids. We conducted searches using Google.com on November 30, 2016, and January 26, 2017, and included the first 100 search results. We also contacted shared decision-making and palliative care experts using a previously established list, via an online survey and one-on-one interviews between April 17, 2017, and August 30, 2017.

Published Reviews

Using a search strategy developed with a librarian, we identified reviews of decision aids that met our inclusion criteria using the MEDLINE database.¹⁷ The primary reviewer (CHS) examined the results of the search, identifying reviews appropriate for further investigation and the secondary reviewer (KP) extracted patient decision aids potentially eligible for our study. See Appendix Table 1 and our published protocol.¹⁷ Notably, given that the decision aids themselves, not published articles, were the unit of observation for our environmental scan, we did not perform dual coding on the MEDLINE extraction.

Google and App Stores

Two reviewers (CHS and MAD) performed the Google and application screening, including both the Apple App Store and Google Play.¹⁷ Using Google Advanced Search, we ran the queries detailed in Appendix Table 2. We disabled cookies and limited our search to English.

The primary reviewer ran each Google search and app store search, archiving the first 100 results of Google searches and first 50 results of app store searches.¹⁸ Then, the primary reviewer opened each page and scanned for patient decision aids or references to patient decision aids, marking those that met our inclusion criteria, those that might meet our inclusion criteria with further research, and those that were not appropriate. We documented specific reasons for exclusion. The secondary reviewer assessed a randomly-selected, 10% subsample. We calculated interrater reliability using a Cohen’s Kappa statistic.

Key Informants

To identify decision aids that did not appear in our online search, we surveyed 187 key informants who work in or study issues related to aging, death and dying and shared decision making.¹⁹ We developed a questionnaire for these informants and deployed it using the online survey software Qualtrics (see Appendix 1. Key Informant Survey). We used a snowball approach, asking participants for other individuals they thought we should speak with about other relevant decision aids. We corresponded with individuals who suggested decision aids that were not already in our decision aid database.

Decision Aid Selection Criteria

We included patient decision aids designed to help seriously ill people near death or their caregivers make decisions about life-sustaining treatments. See Appendix Table 1 for an explanation of terms. We saved decision aids that met our inclusion criteria in an online database, organizing them by target user or index decision(s). When identified decision aids were unavailable online, we e-mailed developers three times to ask for access to the decision aid. If after three queries, we did not receive access to the decision aid, we excluded the tool from our review. Similarly, if developers explicitly refused to participate in the study, we excluded them.

Once we banked and organized the decision aids, one reviewer (KP) systematically collected information about decision aid characteristics using a data collection form (see Appendix 2. Table 3). The data we collected for decision aids from all sources included (1) the index decision, (2) secondary decision(s), (3) the disease/condition, (4) availability (whether the decision aids are available publicly or proprietorially), and (5) use, ie, whether we learned anything about routine use in clinical environments.

Decision Aid Quality Grading Methods

At least two or three reviewers (C.H.S., K.P., M.A.D.), independently assessed the quality of each included patient decision aid, using the NQF standards. Before assessing the quality of each decision aid, we tested an NQF quality assessment form on five decision aids. We subsequently added specificity to the NQF quality criteria for this review. At least two of three reviewers (CHS, KP, MAD) assessed the quality of all included patient decision aids. We calculated interrater reliability using both Cohen’s Kappa statistic for individual quality categories and Spearman’s correlations for overall scores.

Notably, one of the NQF items concerns plain language. We assessed plain language using average readability scores, generated via Readable.io. If readability scores were below seventh-grade level, we considered them plain language. When we could not assess readability using an average score, ie, in the case of video decision aids, the researchers made a qualitative judgment about the plain language criteria.

Statistical Analysis

Our primary outcome was the number and variety of decision aids available for seriously ill individuals near death facing choices about life-sustaining treatments. Secondary outcomes included the quality, actual availability, and use of the available decision aids. We used Stata 13 to synthesize our results. We also reported overall quality and use. We conducted subgroup analyses, including quality, availability, and use of decision aids by category.

Decision Aid Selection Process

We identified 608 links with information about potential decision aids from our Google search. The two raters had substantial interrater reliability according to Cohen’s Kappa statistic (K = 0.64).²⁰ We did not detect any possible decision aids with our app store searches. We identified 31 studies from our MEDLINE search with information about potential decision aids eligible for inclusion. We received 60 responses to our expert survey from the 187 administered (a 32% response rate).

Altogether, we identified 105 potential decision aids from these sources. We excluded 22/105 potential decision aids from our analysis because they were not publicly accessible, and we could not successfully obtain them from the developers. It remains unknown whether these tools would have qualified for inclusion in our review. We excluded 55/105 tools for not meeting one of the following criteria: 1) not being decision aids according to the NQF criteria 2) not concerning life-sustaining treatments 3) not being targeted at people with serious illness near death. A majority of decision aids for life-sustaining treatment decisions are intended for people who do not yet have an advanced serious illness or are not near death. There were 27 decision aids in our final review (Figure 1).

Characteristics of Included Decision Aids

Of the 27 decision aids we included in our review, 14 (52%) were tailored to seriously ill individuals with specific conditions. Eleven decision aids (41%) concerned specific life-sustaining treatments. Two decision aids concerned general treatment approaches, such as life-sustaining care versus palliative care (Table 1).

The decision aids were of variable length and approach. Some were text only, while others were image heavy. The mean length of decision aids was 19 pages, while the median length was 10 pages. Included decision aids offered interventions meant to return patients to health, as well as palliative interventions and comfort care.

Notably, most of the decision aids we included in our review (25 decision aids; 93%) were freely available online. Three (11%) were not. Seventeen (63%) decision aids were developed in the U.S., eight (30%) in Canada, two (7%) in Australia, and one (4%) in the Netherlands (in Dutch, translated using Google Translate). Additionally, there were 22 potentially eligible decision aids that we could not access to review and therefore could not include.

Quality of Included Decision Aids

The overall correlation of scores between the two reviewers was high (0.85). Agreement was high for both reviewers for all categories (balanced 90%, K = 0.0; outcome probabilities 86%, K = 0.7; publication date 93%, K = 0.8; update policy 93%, K = 0.7; funding sources 96%, K = 0.8), except the category concerning the rigor of the decision aid development process (66%, K = 0.2) and the evidence sources used (79%, K = 0.6) categories.

The quality of the decision aids was high in some categories. Of 27 decision aids, most presented options in a balanced way (24, 89%) and identified funding sources (23, 85%). They also reported publication dates most of the time (19, 70%). Readability of the included decision aids was mixed. The average readability grade level was 7.5, with a low score of 4.1 and a high score of 10.7. Eleven decision aids (41%) had readability levels less than seventh grade (Table 2). Thirteen had plain language, including video decision aids that we agreed used plain language.

The decision aids also had consistently low scores in some categories. Of 27, only 11 listed their evidence sources (41%), 11 reported a rigorous evidence-synthesis method (41%), six stated their competing interests (22%), and three offered an update policy (11%). There were no notable differences in the quality of the decision aids in each of the three category types (condition-specific, treatment-specific, general).

Use of Included and Excluded Decision Aids (exploratory)

We received 60 of 187 responses to our key informant survey. We asked every respondent if they were aware of any relevant decision aids. Of the 60 respondents, 45 (75%) said they were aware of decision aids, but only 38 (63%) offered the names of potential tools. Twenty-six respondents (43%) said they were aware of institutions that used the decision aids in routine and sustained care. Twenty-four respondents (40%) offered names of organizations, but most of the suggestions concerned decision aids that did not qualify for inclusion in our review or care that was not routine or sustained. In this preliminary use estimation, we found evidence for the use of only three decision aids or similar tools in routine care, two of which we included in our review.

We found many decision aids of varying quality for people with serious illnesses facing decisions about life-sustaining treatments. Most available decision aids are customized for people with particular diseases or conditions, like cancer or heart failure, with few generalized tools. This may make it difficult for practicing clinicians to find tools that are appropriate for their patients. It could also contribute to the gap between their availability and use in routine care, which is an essential but exploratory finding of this systematic environmental scan. Even if seriously ill people or those who cared for them wanted to obtain and use a decision aid independently, a large proportion of them are not publicly accessible.

Concerning the quality of decision aids, they were usually balanced and listed their funding sources, but other quality areas we often missing concerning their development, content, and disclosures. These deficiencies may affect the trustworthiness of decision aids, which may make practicing clinicians less likely to use them in hospital settings. Reporting of outcome probabilities was particularly weak. Reporting outcome probabilities in ways that people who are ill and their relatives can understand, especially during times of heightened emotion, is critically important. Therefore, it is a cause for concern that the available decision aids often neglect to use evidence-based techniques for conveying outcome information.

Our work built on Butler and colleagues’ “state of the science” review in 2014.¹³ Focusing specifically on proximal life-sustaining treatment decisions, we found many more decision aids by expanding our search beyond the peer-reviewed literature to include the Internet and experts.¹³ We also identified an important gap worthy of further exploration between the decision aids available and their usage in real-world clinical environments.

Our review confirms that implementation of decision aids in routine care is a continued challenge, especially for seriously ill people facing life-sustaining treatment decisions.⁵³ Why tools that are efficacious in controlled trial environments have failed to gain acceptance in real-world settings remains unanswered for this population.⁵⁴ For decision aids in general, researchers have reported barriers concerning clinician awareness, perception, and comfort, as well as usability issues.^55,56 Additionally, systems-level barriers exist, like culture and priorities, difficulty incorporating decision aids into the workflow, resistance from parties who favor other interventions, and the costs associated with implementation.⁵⁶ There may also be particular barriers related to the topics of death and dying.A strength of this work is thatwe applied the rigor of the systematic review method to the environmental scan, a newer method that answers different questions, such as “How many?”, “How much?”, and “How often?” We hope our use of the word systematic will reinforce perception among the scientific community that the environmental scan method is thorough, valid and worthwhile. We believe this method unearthed more decision aids than a traditional systematic review limited to the academic literature would have revealed. Another strength of our review was the rigor of screening and assessment.

A limitation of our work is the challenge of defining serious illness. We worked with palliative care physicians to make these judgments as grounded in clinical practice as possible. The preliminary nature and selection of experts for our sustained—use survey are limitations as well. Despite our efforts to conduct a comprehensive review of a vast environment of tools, we may have missed some decision aids that met our inclusion criteria. An additional limitation of our work is that due to the exploratory nature of our sustained-use survey, we cannot determine with accuracy how often these tools are used, although we have provided the first preliminary assessment of use, to our knowledge.

The gap between prolific patient decision aid development and real-world usage is puzzling. It is possible that using a tool at all is inappropriate for the complex, emotionally-laden decision-making process associated with death and dying. Alternatively, the tools may be inappropriate for serious illness, due to their design, their content, or some other characteristics. Perhaps the existing tools are too tailored for specific conditions and interventions―less appropriate for generalized use. Indeed, only two decision aids included in our final review addressed general care pathways, like life-sustaining care, palliative care, and hospice care. The others were highly specific, concerning particular diseases like kidney disease and particular interventions, like CPR. We know that most people die with comultimorbidities, meaning such specificity may paradoxically make it more difficult for individuals and their families to identify with the content in the materials.^57,58 Without having data from real-world use, we cannot know whether any particular tool is suited or helpful for hospital practice.

It is essential for practicing hospitalists to know whether patient decision aids are appropriate for use in routine care. We hope that our review will help clinicians and health systems find appropriate tools to use with their patients. We also believe there should be mechanisms for providing feedback on whether decision aids are feasible and acceptable to hospitalized people and their caregivers and to practicing hospitalists and what leads to their sustained implementation.^55,56 This can be explored with on-the-ground observational research or through health system quality improvement efforts.

Acknowledgments

Pamela J. Bagley provided search strategy support. Meredith MacMartin provided clinical counsel. Amber Barnato provided comments and insight as an advisor and a new member of Catherine’s Ph.D. committee.

Author contributions

Catherine H. Saunders designed the study, with support from Marie-Anne Durand, Glyn Elwyn, and Kathryn Kirkland. Catherine H. Saunders conducted all screening, with support from Marie-Anne Durand. Khusbu Patel managed the inventory of decision aids. Catherine H. Saunders designed and distributed the key informant survey, with support from Marie-Anne Durand. Hyunkyung Kang and Catherine H. Saunders managed follow-up with key informants. Khusbu Patel and Catherine H. Saunders conducted the decision aid quality review. Catherine H. Saunders, Marie-Anne Durand, and Kathryn Kirkland screened decision aids to determine appropriateness for people with serious illness. Catherine H. Saunders drafted the manuscript, and all authors reviewed and approved it.

Ethical approval

The Dartmouth College Committee for the Protection of Human Subjects designated this project as exempt from further review. All survey participants confirmed their consent via an online form.

Disclosures

Ms. Saunders, Ms. Patel, Ms. Kang, and Dr. Kirkland have nothing to disclose. Dr. Elwyn reports personal fees from ACCESS Community Health Network, personal fees from EBSCO Health, personal fees from Chicago (Federally Qualified Medical Centers), outside the submitted work, and as Director of &think LLC, which owns the registered trademark for OptionGrids(TM) patient decision aids. He owns copyright in measures of shared decision making and care integration, namely collaboRATE, integRATE, Observer OPTION-5, and Observer OPTION-12, which are freely available for use. He is codeveloper of the OptionGrid patient decision aids, which are licensed to EBSCO Health. He has received reimbursement for travel, accommodations, and expenses from EBSCO Health, ACCESS Community Health Network, and Chicago (Federally Qualified Medical Centers). Dr. Durand reports personal fees from ACCESS Community Health Network, personal fees from EBSCO Health, outside the submitted work, and as codeveloper of the OptionGrid patient decision aids, which are licensed to EBSCO Health. She has received reimbursement for travel, accommodations, and expenses from EBSCO Health and ACCESS Community Health Network.

Financial conflicts of interest

Glyn Elwyn (GE) and Marie-Anne Durand (M-A D) have developed the Option Grid patient decision aids, which are licensed to EBSCO Health. They receive consulting income from EBSCO Health and may receive royalties in the future. M-A D is a consultant for ACCESS Community Health Network. No other competing interests declared.

Funding

The authors did not receive funding for this research.

Published protocol linked here: https://www.ncbi.nlm.nih.gov/pubmed/28825182